Compare commits

..
1395 Commits
Author SHA1 Message Date
Victor Shyba 5718be1c61 log top memory usage on status if tracemalloc enabled 2019-09-16 15:00:35 -03:00
Lex Berezhny 5c151a3a09 v0.41.0 2019-09-16 13:24:11 -04:00
Lex Berezhny 3c46c6cbb9 fix mypy type issue 2019-09-16 13:21:45 -04:00
Lex Berezhny d31f09c174 client wallet db versioning 2019-09-16 13:21:45 -04:00
Lex Berezhny 063eeb34ec fix channel import test to wait for address subscription tasks to finish before looking up channel 2019-09-16 13:21:45 -04:00
Lex Berezhny f06192601c dont fail when claim is not decodable 2019-09-16 13:21:45 -04:00
Lex Berezhny 091221ba2c added wallet/constants.py 2019-09-16 13:21:45 -04:00
Lex Berezhny 7c22111ae0 channel certificate and channel claim can be different accounts, also JSON RPC responses now include a field showing if the channel certificate is available 2019-09-16 13:21:45 -04:00
Alex Grin daac3eed09 Merge pull request #2458 from lbryio/wallet_server_default_coin
default wallet server to LBC coin
2019-09-16 12:34:54 -04:00
Alex Grintsvayg 32ee7feea8 download wallet snapshot on first run 2019-09-16 12:24:44 -04:00
Alex Grintsvayg 2026848a77 default wallet server to LBC coin 2019-09-16 12:01:38 -04:00
Jack Robison a7b205603c Merge pull request #2447 from lbryio/file_encoder
use json_file_encoder for managed streams (file list)
2019-09-13 11:21:36 -04:00
Victor Shyba 291d14f2d4 move as_dict to json_encoder 2019-09-12 14:08:37 -03:00
Victor Shyba 4a68137cfa remove dead code 2019-09-12 14:08:37 -03:00
Jack Robison b2f0374426 Merge pull request #2452 from lbryio/fix-sqlite-misuse
Fix SQLITE_MISUSE
2019-09-12 11:57:23 -04:00
Jack Robison 5fcca6ed0c feedback
-add comment for the fetchall() that if removed will cause SQLITE_MISUSE
2019-09-10 15:03:17 -04:00
Jack Robison a05c44ad1e fix SQLITE_MISUSE
fixes https://github.com/lbryio/lbry-sdk/issues/2164
2019-09-09 22:38:53 -04:00
Jack Robison f9880c8931 test sqlite binding error 2019-09-09 21:46:04 -04:00
Lex Berezhny b876896628 v0.40.1 2019-09-09 13:00:12 -04:00
Victor Shyba 375c2a3363 fix rare edge cases on network 2019-09-09 12:25:16 -04:00
Lex Berezhny 5dbc749000 v0.40.0 2019-09-03 12:36:00 -04:00
Lex Berezhny 36fbd91280 generate docs 2019-09-03 12:00:29 -04:00
Lex Berezhny d2ad3e7ff1 initial kv store 2019-09-03 12:00:29 -04:00
shyba 072f1f112e Merge pull request #2418 from lbryio/no_chdir
sync and connection issues
2019-09-03 11:53:30 -03:00
Victor Shyba 6c79f98d3c connect directly on test server 2019-09-03 11:29:02 -03:00
Victor Shyba 8edebbb684 fix tests for transaction show 2019-09-02 15:43:33 -03:00
Victor Shyba 4839ca631a make restricted default 2019-09-02 15:10:16 -03:00
Victor Shyba 5018fe90de can also be protocolerror 2019-09-01 00:02:07 -03:00
Victor Shyba 379144bcfe fix restriction rules, minor refactor, translate error from transaction not found into returning none 2019-08-31 17:39:03 -03:00
Victor Shyba d2cd0ece5f restore multi server, improve sync concurrency 2019-08-31 06:37:29 -03:00
Victor Shyba 10b7ccaa92 fix unordered history inconsistency 2019-08-31 06:37:29 -03:00
Victor Shyba 925eb618de failing test for unordered mempool 2019-08-31 06:37:29 -03:00
Victor Shyba 96800de052 switch tests and fixes + support for multiple servers 2019-08-31 06:37:29 -03:00
Victor Shyba 9df659b647 no chdir 2019-08-31 06:37:29 -03:00
Lex Berezhny aec0bd5d11 fix intermittent bug where claim metadata was cleared out 2019-08-30 09:44:30 -04:00
Lex Berezhny c1c9c2c578 regenerate api.json 2019-08-28 12:55:19 -04:00
Lex Berezhny c739fa3ca1 fixed typo in daemon API doc 2019-08-26 13:22:42 -04:00
Lex Berezhny 355dfcb6c1 v0.39.3 2019-08-26 13:06:26 -04:00
Oleg Silkin 8f7c291e33 Adds comment_hide functionality 2019-08-26 12:43:07 -04:00
Jack Robison 71fbb3df24 Merge pull request #2414 from lbryio/fix-connection-errors-reloaded
Switch to the fastest wallet server and fix connection errors
2019-08-26 11:13:20 -04:00
Victor Shyba 9cfa9b2c57 use only the main session, dont cancel on connectionerror 2019-08-26 11:17:39 -03:00
Victor Shyba b9b8178e30 use a semaphore to avoid false timeouts on large syncs 2019-08-26 11:17:39 -03:00
Victor Shyba 089cefb77b make it debug for now, there are false positives 2019-08-26 11:17:39 -03:00
Victor Shyba ee9ea1faab get history only from main server 2019-08-26 11:17:39 -03:00
Victor Shyba 22b2cf8b0c warn when out of sync 2019-08-26 11:17:39 -03:00
Victor Shyba 10bd3c9dda fix misc reconnect scenarios 2019-08-26 11:17:39 -03:00
Victor Shyba 6c3147fcf4 simple address subs instead of batch 2019-08-26 11:17:39 -03:00
Jack Robison 5e8d169c50 debug 2019-08-26 11:17:39 -03:00
Jack Robison 8657a35fd7 switch_to_fastest 2019-08-26 11:17:39 -03:00
Jack Robison 97c0b60a22 logging 2019-08-26 11:17:39 -03:00
Lex Berezhny 8a284d6c49 metrics reporting fix for wallet servers 2019-08-26 09:39:42 -04:00
Jack Robison ff4b2f2dec Merge pull request #2411 from lbryio/fix-offline-connection-errors
Fix misc errors when going offline/online
2019-08-22 13:48:02 -04:00
Jack Robison aedce256af fix wallet status silently breaking when offline
-update wallet status to use real latency values and the full list of connected servers
2019-08-22 13:11:17 -04:00
Jack Robison 48b4b97132 fix uncaught exchange rate manager errors when going offline 2019-08-22 13:11:17 -04:00
Jack Robison 6b1f88df66 fix status breaking (and loggly) when the internet connection is lost 2019-08-22 13:11:17 -04:00
shyba d246c46227 Merge pull request #2397 from lbryio/unpack_leftovers
fix unpack error known leftovers
2019-08-22 13:41:25 -03:00
Victor Shyba 7fdd9dc293 fix annoying assertEquals warning 2019-08-21 22:03:24 -03:00
Victor Shyba 202861a900 fix unpack leftovers 2019-08-21 22:03:24 -03:00
Victor Shyba 4c41e7cb29 check headers bounds on access 2019-08-21 22:03:24 -03:00
Jack Robison 92823dcea9 Merge pull request #2407 from lbryio/temporary_speedup_ci
travis: remount /tmp as tmpfs
2019-08-21 20:08:27 -04:00
Victor Shyba 795a5227bc travis: remount /tmp as tmpfs 2019-08-21 20:05:39 -03:00
Alex Grintsvayg bbd4bb99b7 committing my claim search test script 2019-08-21 13:28:30 -04:00
Alex Grin f7443061ef less verbose logging 2019-08-21 09:54:10 -04:00
Jack Robison 416c3c16ab v0.39.2 2019-08-20 20:37:20 -04:00
Jack Robison a11956ece0 Merge pull request #2398 from lbryio/retriable_batcheable_networking
Improve wallet server selection
2019-08-20 20:10:05 -04:00
Jack Robison c4e0c659a7 fix missing perf_counter 2019-08-20 19:22:22 -04:00
Jack Robison f6ba09b171 mypy 2019-08-20 15:41:27 -04:00
Jack Robison b827881024 mypy 2019-08-20 15:35:06 -04:00
Jack Robison 89e395b5f4 pylint 2019-08-20 15:26:00 -04:00
Jack Robison c590954eb9 include connection latency in server selection 2019-08-20 15:11:01 -04:00
Jack Robison a5af3866fd time.time -> time.perf_counter 2019-08-20 14:56:13 -04:00
Jack Robison 1bef56f030 disconnect from duplicate servers (same ip, different dns) 2019-08-20 14:55:24 -04:00
Jack Robison 9045737504 track response time for server.version only 2019-08-20 14:54:55 -04:00
Jack Robison 0e48772c84 track connection latency 2019-08-20 14:45:53 -04:00
Jack Robison 72690b5cff raise asyncio.TimeoutError 2019-08-20 14:19:50 -04:00
Jack Robison 07bae26fd3 close connection and reraise on CancelledError 2019-08-20 14:16:56 -04:00
Victor Shyba 7b0e4617d3 fix db test stability 2019-08-20 04:11:04 -03:00
Victor Shyba af797296ed rolling averages on response time 2019-08-19 16:50:24 -03:00
Victor Shyba c826c7da0d take load into consideration when picking fastest 2019-08-19 14:02:30 -03:00
Victor Shyba 20af1396b9 refactor stop 2019-08-19 14:02:30 -03:00
Victor Shyba f567aca532 retry and batch requests, fix some loose ends 2019-08-19 14:02:30 -03:00
Lex Berezhny c60b658443 v0.39.1 2019-08-19 13:02:16 -04:00
Jack Robison fde8c34088 Merge pull request #2401 from lbryio/check-blob-peers-pingable
UDP ping blob peers to prioritize those who can be connected to
2019-08-19 13:01:40 -04:00
Jack Robison deb6a07160 fix test 2019-08-19 12:51:17 -04:00
Jack Robison 0a66e1b24c logging 2019-08-19 12:29:13 -04:00
Jack Robison 819a551b77 Merge pull request #2402 from lbryio/aioupnp-0.0.14
bump `aioupnp` requirement to 0.0.14, fixes upnp errors blocking startup
2019-08-19 12:08:22 -04:00
Jack Robison 5119b71035 prevent attempting to download from oneself 2019-08-19 12:08:17 -04:00
Jack Robison 33945e14d3 use _value_producer in peer_list to match the downloader behavior 2019-08-19 12:08:17 -04:00
Jack Robison 1d9705fb17 reuse blob protocol object instead of transport 2019-08-19 12:08:17 -04:00
Jack Robison 1949937b5e bump aioupnp requirement to 0.0.14 2019-08-19 10:51:20 -04:00
Lex Berezhny 956e21269e monitor wallet server block transitions and report if new block is excessively delayed 2019-08-19 09:52:37 -04:00
Alex Grin ff73418fc1 Merge pull request #2371 from lbryio/basenetwork_refactor
refactor basenetwork so each session takes care of itself
2019-08-16 10:19:53 -04:00
shyba 76d40f44d8 Merge pull request #2395 from lbryio/lbrycrd17
add support for lbrycrd 17
2019-08-14 11:41:13 -03:00
Victor Shyba ab6944b7b0 wallet server: dont fail to start on segwit txs 2019-08-14 10:26:02 -03:00
Victor Shyba 4d741bd200 update lbrycrd to 17, add option to disable segwit for now 2019-08-13 18:07:15 -03:00
Victor Shyba f3a163b382 fix names and types 2019-08-12 18:16:58 -03:00
Victor Shyba 011b7f090f add urgent reconnect when api is called to bypass retry delay 2019-08-12 18:16:58 -03:00
Victor Shyba 4ead92cfbe fixes from review 2019-08-12 18:16:58 -03:00
Victor Shyba 9ee2f30df4 timeout instead of cancel + minor fixes 2019-08-12 18:16:58 -03:00
Victor Shyba 4a749f6c38 on timeout, restore latency to max 2019-08-12 18:16:58 -03:00
Victor Shyba cf924373f6 make stream test an asyncio test 2019-08-12 18:16:58 -03:00
Victor Shyba d1f5b25418 merge repeated subscription events 2019-08-12 18:16:57 -03:00
Victor Shyba 822f53c888 remove unused things 2019-08-12 18:16:57 -03:00
Victor Shyba f04c5ee902 fix test reliability 2019-08-12 18:16:57 -03:00
Victor Shyba 022d60ec2c both exceptions can be raised, depending on when the call happens 2019-08-12 18:16:57 -03:00
Victor Shyba 8fce374cae race condition where the request is sent and connection lost right after 2019-08-12 18:16:57 -03:00
Victor Shyba e78e2738ef basenet refactor 2019-08-12 18:16:57 -03:00
Victor Shyba 5728493abb refactor basenetwork so each session takes care of itself 2019-08-12 18:16:57 -03:00
Lex Berezhny af1c8ec35c adding missing comma in wallet_server_monitor 2019-08-12 14:16:47 -04:00
Lex Berezhny cb84abfb88 v0.39.0 2019-08-12 13:01:16 -04:00
Jack Robison df1e349470 Merge pull request #2385 from lbryio/blob-exchange-timeouts
Blob exchange server timeouts
2019-08-12 12:29:40 -04:00
Jack Robison 357ce881a6 more blob exchange tests 2019-08-12 10:47:28 -04:00
Jack Robison aa417f5410 test max request size 2019-08-12 10:47:28 -04:00
Jack Robison 0ca29b9518 test idle timeout 2019-08-12 10:47:28 -04:00
Jack Robison 22db04068e add server side timeouts for idle blob exchange connections and slow transfers 2019-08-12 10:47:28 -04:00
Jack Robison 65e74281e0 max request size 2019-08-12 10:47:28 -04:00
Lex Berezhny 46d89929ba fixup balance tests and other stuff 2019-08-12 10:22:14 -04:00
Lex Berezhny 1c941a355b regenerate api docs 2019-08-12 10:22:14 -04:00
Lex Berezhny 0d6248fbeb fix torba tests 2019-08-12 10:22:14 -04:00
Lex Berezhny 98d4d00f96 most commands now work across all accounts 2019-08-12 10:22:14 -04:00
Jack Robison e2d618f472 formatting in time_to_first_byte.py
-pending claim_type being usable in claim_search, don't try downloading channel claims in time_to_first_byte.py
2019-08-12 00:49:33 -04:00
Jack Robison 79d643db6c Merge pull request #2384 from lbryio/faster_file_list
Faster file list
2019-08-11 22:41:15 -04:00
Victor Shyba 13891330d9 check blob hashes from completed_blob_hashes 2019-08-11 19:11:13 -03:00
Jack Robison 6fd0c2bf77 Merge pull request #2379 from lbryio/add-wallet-server-to-ttfb-analytics
Add `connected_servers` field to wallet status and improve analytics for wallet related download errors
2019-08-09 14:18:03 -04:00
Jack Robison b8b42a1fd2 add error_message to ttfb analytics, update tests 2019-08-09 12:55:01 -04:00
Jack Robison fba1b28615 add connected_servers field to wallet status
-add `wallet_server` to time to first bytes analytics event
2019-08-09 12:54:38 -04:00
Lex Berezhny 98828387e3 slow queries saved to metabase 2019-08-06 20:41:14 -04:00
Lex Berezhny 1c12247b71 updated wallet_server_monitor to update slack 2019-08-06 17:44:31 -04:00
Lex Berezhny ee10ebd5d7 renamed lbry/scripts/wallet_server_metabase_analytics.py -> lbry/scripts/wallet_server_monitor.py 2019-08-06 16:07:45 -04:00
Lex Berezhny 9937ad6e85 v0.38.7 2019-08-05 13:40:49 -04:00
Jack Robison 60ce393531 remove comment 2019-08-05 13:39:38 -04:00
Jack Robison bc33cad999 add test for connection failure analytics 2019-08-05 13:39:38 -04:00
Jack Robison 4261b50d9d include connection failure count in ttfb analytics 2019-08-05 13:39:38 -04:00
Jack Robison e2b3141536 typing 2019-08-05 13:39:38 -04:00
Thomas Zarebczan 266aa1b15d update delete>abandon 2019-08-05 12:42:25 -04:00
Oleg Silkin b57a7475c5 Fixes comment docs & adds wallet component requirement on comment funcs 2019-08-05 12:42:25 -04:00
Victor Shyba 8cf48eb5f7 move granular balances logic to account method 2019-08-05 11:08:33 -04:00
Victor Shyba 35541bbbb9 new api format, hide subtotals by default 2019-08-05 11:08:33 -04:00
Victor Shyba 59424a3e73 add query to get tips summary for granular balances 2019-08-05 11:08:33 -04:00
Victor Shyba 2056888667 implement tips_received tips_sent, wip, in python for now 2019-08-05 11:08:33 -04:00
Victor Shyba 1e54785742 add granular balances 2019-08-05 11:08:33 -04:00
Lex Berezhny a58ed6f331 wallet server metabase analytics script 2019-08-05 09:23:47 -04:00
Lex Berezhny 9a79b71d9f removed dart 2019-08-04 20:57:05 -04:00
Lex Berezhny e9098887fb script to check channel signature 2019-08-01 17:27:54 -04:00
Jack Robison 9e8a3ef7fc Merge pull request #2359 from lbryio/fix-error-on-null-fee
Fix dust transaction error on zero amount content fee
2019-07-30 11:53:04 -04:00
Jack Robison 221585908a re-raise unexpected resolve error during download as a ResolveError 2019-07-29 17:56:30 -04:00
Jack Robison f4c111f04c handle null content fee 2019-07-29 17:56:24 -04:00
Alex Grin de12e25450 v0.38.6 2019-07-29 15:11:18 -04:00
Oleg Silkin 2545ebbece Adds unittests & fixes comment_client functions 2019-07-29 14:55:42 -04:00
Oleg Silkin 14ec6ce43b Adds unit-testing for delete comment signatures 2019-07-29 14:55:42 -04:00
Oleg Silkin 0cac1e65aa Imports comment_client and uses dot notation for rest of modules 2019-07-29 14:55:42 -04:00
Oleg Silkin 1e4d57ab83 Simplifies signing function into one to make API symmetric 2019-07-29 14:55:42 -04:00
Oleg Silkin 49b42b7e03 Simplifies 2019-07-29 14:55:42 -04:00
Oleg Silkin 89c0b5dbdf adds comment abandon function 2019-07-29 14:55:42 -04:00
Lex Berezhny 8138675808 added index for claim_type searches 2019-07-29 14:55:07 -04:00
Jack Robison 726129836a Merge pull request #2323 from lbryio/refactor_ttfb
refactor and fix ttfb for new api
2019-07-28 22:28:02 -04:00
Jack Robison 356b3c85e5 use app default order_by 2019-07-28 21:42:04 -04:00
Jack Robison 1a0680ead9 update claim_search_performance.py, add new test cases shown by time_to_first_byte.py 2019-07-28 21:22:07 -04:00
Jack Robison c876d891fa use perf_counter 2019-07-28 21:18:29 -04:00
Jack Robison a08232f164 fix claim search kwargs in time to first byte script 2019-07-28 21:18:13 -04:00
Victor Shyba 0f8cf46626 refactor and fix ttfb for new api 2019-07-28 17:33:37 -04:00
Alex Grin 95017b5fc8 v0.38.5 2019-07-25 13:22:54 -04:00
Jack Robison 1374d337d8 Merge pull request #2348 from lbryio/probemon
Probe if peers can be connected to during blob downloads to more quickly skip bad peers
2019-07-25 13:21:42 -04:00
Jack Robison 35e7cc685f fix logging error 2019-07-24 22:12:51 -04:00
Victor Shyba 2ac4883bda cleanup and fix prober 2019-07-24 22:32:20 -03:00
Victor Shyba 5a7212aebb gotta probe 'em all 2019-07-24 20:26:38 -03:00
Lex Berezhny 3f39cc2955 v0.38.4 2019-07-24 18:02:03 -04:00
Lex Berezhny 8a5bfe884a save channel keys into wallet before broadcasting 2019-07-24 18:00:06 -04:00
Lex Berezhny 046dd53913 add index to filter by fee_amount and order by release_time 2019-07-24 14:27:59 -04:00
Alex Grintsvayg d2f174635c add script to monitor wallet servers for slow queries 2019-07-24 13:36:22 -04:00
Lex Berezhny 01a760d8f0 v0.38.3 2019-07-23 17:44:38 -04:00
Victor Shyba 9215d2c9f9 log that checkpoint succeeded 2019-07-23 17:44:13 -04:00
Victor Shyba 0bb09ae91e use checkpoint to speed up 2019-07-23 17:44:13 -04:00
Victor Shyba afeff61268 fix repair and tests 2019-07-23 17:44:13 -04:00
Victor Shyba 6f5a595975 check from final size instead of partials 2019-07-23 17:44:13 -04:00
Victor Shyba 8a69ec062e download headers to disk and repair instead of connecting each 2019-07-23 17:44:13 -04:00
Victor Shyba 45fb673e49 dont fail when s3 is out of reach 2019-07-23 17:44:13 -04:00
Lex Berezhny 93b5f1809e revert to v0.38.2 so we can release v0.38.3 again 2019-07-23 17:43:04 -04:00
Alex Grin ff05eaae48 v0.38.3 2019-07-23 11:14:20 -04:00
Alex Grintsvayg b48b67ed10 fix aws key 2019-07-23 11:14:00 -04:00
Alex Grintsvayg 3ead6f63b6 add lbry-ci bot auth token 2019-07-23 11:04:10 -04:00
Alex Grin 77235f0ad8 v0.38.2 2019-07-22 17:45:43 -04:00
Alex Grintsvayg f6a80ba929 Revert "v0.38.2"
This reverts commit 295ceb90d5.
2019-07-22 17:45:12 -04:00
Alex Grin 295ceb90d5 v0.38.2 2019-07-22 13:31:20 -04:00
Victor Shyba df41a82b23 fix integration test 2019-07-22 11:31:45 -04:00
Victor Shyba 34500e07c6 change picking logic to probe the session before considering it a good session 2019-07-22 11:31:45 -04:00
Alex Grintsvayg 1e6097e3b2 more spv dns entries 2019-07-22 11:24:12 -04:00
Lex Berezhny 521facedc4 script to find max server load 2019-07-22 00:15:05 -04:00
Lex Berezhny de80a2ba97 fixed 0 results when only filtering by lots of common tags 2019-07-21 19:44:10 -04:00
Lex Berezhny 3afd615447 dont run common tag optimization strategy if there are 5 or more common tags provided 2019-07-20 15:45:15 -04:00
Lex Berezhny 87e52bec0a successfuly run indexes after full wallet server sync 2019-07-20 15:45:15 -04:00
Lex Berezhny b89f04b1f1 fix claim integration tests 2019-07-20 15:45:15 -04:00
Lex Berezhny e4f802e5f4 when resolving inside channel break a tie of claims with same effective_amount by using height 2019-07-20 15:45:15 -04:00
Lex Berezhny 16fc2c456f moved executing create index queries outside of a tx 2019-07-20 15:45:15 -04:00
Lex Berezhny ef7656ae98 removed some dead code 2019-07-20 15:45:15 -04:00
Lex Berezhny ad2df4871e popular tags are indexed and use different search strategy than less popular tags 2019-07-20 15:45:15 -04:00
Jack Robison 8d5f50b695 improve claim search test script to show sql for nearly interrupted queries 2019-07-19 14:17:21 -04:00
Lex Berezhny bbc056eef0 added better stats to sqlite_perf_test.py 2019-07-19 11:26:50 -04:00
Jack Robison 82fd3c632e another claim search performance test case 2019-07-19 10:50:37 -04:00
Jack Robison dbae939808 claim search performance test 2019-07-19 10:42:38 -04:00
Jack Robison c2dcacb0de more test cases 2019-07-19 10:37:15 -04:00
Lex Berezhny 8f45618cd0 added improved effective_amount index covering release_time 2019-07-18 23:24:00 -04:00
Lex Berezhny 29a87c850c fix sqlite_perf_test 2019-07-18 19:54:38 -04:00
Lex Berezhny 395518ff76 ver updates to new metrics 2019-07-18 19:54:38 -04:00
Lex Berezhny 08da763327 more metrics cleanup 2019-07-18 19:54:38 -04:00
Lex Berezhny d7ec3c182d tuples 2019-07-18 19:54:38 -04:00
Lex Berezhny ffe6418392 fix metrics json response test 2019-07-18 19:54:38 -04:00
Lex Berezhny 88adf3ee27 oops 2019-07-18 19:54:38 -04:00
Lex Berezhny f406db85dc pylint again 2019-07-18 19:54:38 -04:00
Lex Berezhny d5f8af471a pylint 2019-07-18 19:54:38 -04:00
Lex Berezhny 046459a72d refactored websockets and metrics 2019-07-18 19:54:38 -04:00
Lex Berezhny 49b0f59790 added metrics module 2019-07-18 19:54:38 -04:00
Lex Berezhny 71551058ce query interpolation 2019-07-18 19:54:38 -04:00
Jack Robison 2a9e6473eb Merge pull request #2310 from lbryio/2293
set file status to completed only when all bytes were written
2019-07-18 13:21:41 -04:00
Victor Shyba 576ef9c4ed remove unused variable 2019-07-17 18:54:21 -03:00
Victor Shyba 4d64fca9de fix memory blob showing as completed/verified 2019-07-17 18:54:21 -03:00
Victor Shyba 8b1307e1ca file status completed set only if all bytes were written 2019-07-17 18:54:21 -03:00
Jack Robison eae4ed7b27 Merge pull request #2317 from lbryio/new_2282
torba client networking: improve network timeout handling and fix spurious logs on shutdown
2019-07-17 17:49:59 -04:00
Victor Shyba 6997ea608d fix reconnect and timeouts 2019-07-16 23:59:28 -03:00
Alex Grintsvayg d97d738880 remove RCs from release tool 2019-07-16 21:35:21 -04:00
Lex Berezhny f72b07d0f4 fixup 2019-07-16 21:28:36 -04:00
Lex Berezhny b89916a3d7 guarded dict access 2019-07-16 21:15:39 -04:00
Lex Berezhny e4aeb66257 claim_list doesnt blow up 2019-07-16 21:08:51 -04:00
Jack Robison 578f8f4062 Merge pull request #2320 from lbryio/interrupt-slow-queries
Interrupt and log slow wallet server queries
2019-07-16 20:50:37 -04:00
Jack Robison 8fbf82f4ea filter mutually exclusive tags 2019-07-16 20:05:49 -04:00
Jack Robison 478ae7e657 add claim search performance script 2019-07-16 20:05:49 -04:00
Jack Robison d319761483 interrupt slow queries and log them for debugging
- add QUERY_TIMEOUT_MS env variable

- return sqlite timeout errors, jsonrpc error code -32000
2019-07-16 20:05:45 -04:00
Lex Berezhny 16fb716872 thicker chart line 2019-07-16 17:10:02 -04:00
Lex Berezhny b8d9d097d8 graph taller 2019-07-16 15:23:45 -04:00
Lex Berezhny ec79a1d0e8 improved graph ui 2019-07-16 15:21:41 -04:00
Lex Berezhny 31284e0c5e dont run empty queries 2019-07-16 13:57:22 -04:00
Lex Berezhny 8e64b1840d added errors key to metrics tracking 2019-07-16 13:39:17 -04:00
Lex Berezhny dd9bf04e35 default to HOST for WEBSOCKET_HOST if not set 2019-07-16 12:05:43 -04:00
Thomas Zarebczan 76a27fdef1 str > list for a few fields 2019-07-16 00:59:06 -04:00
Victor Shyba f96443515a optimize header check 2019-07-16 00:58:36 -04:00
Lex Berezhny 0f1bde2274 pylint 2019-07-16 00:57:53 -04:00
Lex Berezhny 6fb94e445f ver gets cache hit info 2019-07-16 00:57:53 -04:00
Lex Berezhny e02a4e44c0 working resolve/search caching 2019-07-16 00:57:53 -04:00
Lex Berezhny 71dc882d5d indexes update 2019-07-16 00:57:53 -04:00
Lex Berezhny ae4a29d41c prevent duplicate tags 2019-07-16 00:57:53 -04:00
Lex Berezhny d27a3b6421 support for UN M.49 region codes in language 2019-07-16 00:57:53 -04:00
Lex Berezhny 03a58cf50a added lrucache to search/resolve 2019-07-16 00:57:53 -04:00
Lex Berezhny f43e370e89 refactor 2019-07-16 00:57:53 -04:00
Lex Berezhny 45bdb9023a ui to add servers 2019-07-16 00:57:53 -04:00
Lex Berezhny 6d7aa93157 sqlite_perf_test uses lbry.wallet.server.db.reader 2019-07-16 00:57:53 -04:00
Lex Berezhny d083910255 refactored metrics gathering for wallet server 2019-07-16 00:57:53 -04:00
Lex Berezhny db3f9af174 refactored metrics gathering for wallet server 2019-07-16 00:57:53 -04:00
Lex Berezhny 7d2ad5e4ed added TRACK_METRICS env variable to torba wallet server 2019-07-16 00:57:53 -04:00
Lex Berezhny 84d3771984 more detailed stress testing metrics and graphs 2019-07-16 00:57:53 -04:00
Lex Berezhny 1eebe006ae pylint 2019-07-16 00:57:53 -04:00
Lex Berezhny 0327ce53a6 uncomment imports 2019-07-16 00:57:53 -04:00
Lex Berezhny 85442c1d07 websocket admin connection to get query metric data 2019-07-16 00:57:53 -04:00
Lex Berezhny a6ed51b7a7 websocket admin connection to get query metric data 2019-07-16 00:57:53 -04:00
Lex Berezhny dc8b4b5252 instrument wallet.server.db.reader with timer metrics 2019-07-16 00:57:53 -04:00
Lex Berezhny 8ca74d3de2 env variables to set websocket host and port 2019-07-16 00:57:53 -04:00
LBRY bot 1923e6f152 v0.38.1 2019-07-15 12:02:18 -04:00
Lex Berezhny 0d621e085b v0.38.1rc1 2019-07-15 11:58:13 -04:00
Victor Shyba 8749c78622 close daemon connector on server close to avoid errors on tests 2019-07-14 00:33:57 -04:00
Victor Shyba 510d88fff1 remove unused checksum, add typing 2019-07-13 01:21:26 -04:00
Victor Shyba d9460dcd9e refactor and clean up 2019-07-13 01:21:26 -04:00
Victor Shyba d0607b6fec make header component use torba headers 2019-07-13 01:21:26 -04:00
Victor Shyba 2481305a1a make integrity check faster 2019-07-13 01:21:26 -04:00
Victor Shyba 9990889cce add repair ability to baseheaders 2019-07-13 01:21:26 -04:00
Victor Shyba 14f743c493 show status where expected 2019-07-13 01:21:26 -04:00
Victor Shyba df8fe2e3ef bump header checksum 2019-07-13 01:21:26 -04:00
Victor Shyba 906fea972e remove lost warning message 2019-07-13 01:21:26 -04:00
Victor Shyba 04a82d5c23 add make test 2019-07-13 01:21:26 -04:00
Victor Shyba f2944fd33b if no headers, best block hash is genesis 2019-07-13 01:21:26 -04:00
Victor Shyba dcb1e53aba remove unused network command 2019-07-13 01:21:26 -04:00
Victor Shyba a4d8508b40 update remote height on demand 2019-07-13 01:21:26 -04:00
Victor Shyba f109aa1f73 make header access synchronous for now 2019-07-13 01:21:26 -04:00
Victor Shyba f09c18539b add test for concurrent header access 2019-07-13 01:21:26 -04:00
Lex Berezhny c5aeac6898 sqlite performance script measures sql time inside each process individually 2019-07-11 20:10:28 -04:00
Lex Berezhny b81c801f2d sqlite perf test script 2019-07-11 19:20:58 -04:00
Lex Berezhny f3cedf2f44 added windows build for ver 2019-07-11 17:20:50 -04:00
Jack Robison 910999e7e4 fix dart bug
fixes `The argument type 'Utf8Decoder' can't be assigned to the parameter type 'StreamTransformer<Uint8List, dynamic>'.` on `flutter run`
2019-07-11 17:15:51 -04:00
Lex Berezhny 5bceb06fc5 load tester supports changing host, port and payload 2019-07-11 16:13:16 -04:00
Lex Berezhny cc098f3090 refactored wallet server db reader to support thread pool and process pool 2019-07-11 14:04:49 -04:00
Lex Berezhny a73c5aec27 pylint 2019-07-11 14:04:49 -04:00
Lex Berezhny 68b29d9529 multi-process query executor 2019-07-11 14:04:49 -04:00
Lex Berezhny 57f069d750 added ability to customize SessionManager to torba 2019-07-11 14:04:49 -04:00
Lex Berezhny 67e12bc590 ver app updates 2019-07-11 14:04:49 -04:00
Lex Berezhny ff2a08747b added lbry dart 2019-07-11 14:04:49 -04:00
Lex Berezhny 3791d22713 added ver 2019-07-11 14:04:49 -04:00
Lex Berezhny 638ecafb69 no longer using virtualenv for travis windows build 2019-07-09 18:42:34 -04:00
Alex Grintsvayg 5176feed81 let daemon be shut down before it has fully started. fixes #2231 2019-07-09 16:55:48 -04:00
Lex Berezhny 8c63033461 tx parsing speedup and faster wallet sync 2019-07-09 02:20:56 -04:00
Lex Berezhny 8c878f8e25 claim search uses different tag filtering approaches depeding on query type 2019-07-08 16:33:24 -04:00
Victor Shyba c1c7e30ec9 test boundaries, make limit a constant 2019-07-08 12:03:24 -04:00
Victor Shyba 28ae7f81e9 closes 2281 - fix too many sql variables + tests 2019-07-08 12:03:24 -04:00
hackrush 71de4d769d Merge pull request #2299 from lbryio/gitignore-update-mac
added .DS_Store to .gitignore
2019-07-08 20:50:04 +05:30
hackrush f502944c5b added .DS_Store to .gitignore 2019-07-08 19:40:39 +05:30
Lex Berezhny 99d21e93f3 added --no_totals argument to claim_search 2019-07-07 23:51:32 -04:00
Lex Berezhny 350e610f15 handle invalid fee in old JSON claims 2019-07-06 20:53:03 -04:00
Thomas Zarebczan 4f208b6b31 Merge pull request #2286 from osilkin98/2279-fix
Fixes corner case on comment validation for pending channel
2019-07-03 14:14:47 -04:00
Thomas Zarebczan dcb781a57e Merge pull request #2287 from lbryio/aiohttp
use utils.aiohttp for comments
2019-07-03 14:12:22 -04:00
Thomas Zarebczan 09db71ae7f Update comment_client.py 2019-07-02 23:39:27 -04:00
Thomas Zarebczan 60dc43340d use util.aiohttp
Fix certificate errors
2019-07-02 18:38:43 -04:00
Oleg Silkin 256b10bd7d Merge branch 'master' into 2279-fix 2019-07-02 12:01:00 -04:00
Oleg Silkin 77adcaeb50 Fixes corner case where channel is pending confirmation 2019-07-02 11:58:24 -04:00
Jack Robison abf2ca40a2 Merge pull request #2273 from lbryio/fix-session-bloat
Fix wallet server session bloat from unhandled socket errors
2019-07-01 12:18:57 -04:00
Jack Robison d5183123fa fix uvloop being ignored 2019-07-01 11:42:16 -04:00
Jack Robison 8b0d96acba reuse lbrycrd connector 2019-07-01 11:40:13 -04:00
Jack Robison 66122a45ed remove zombie sessions on uncaught socket errors 2019-07-01 11:36:26 -04:00
Jack Robison a2953ac9ac test session bloat from timed out sockets 2019-07-01 11:36:26 -04:00
Jack Robison c51db5294f typing 2019-07-01 11:36:26 -04:00
Jack Robison 8cba43bfed add method_counts field to getinfo 2019-07-01 11:36:23 -04:00
Jack Robison 4b1c7aa6c5 fix getinfo command 2019-07-01 10:41:36 -04:00
Jack Robison 14c1e95841 Merge pull request #2275 from lbryio/disable-bandwidth-tracking
add `track_bandwidth` config setting to disable bandwidth usage in `status`, add `total_sent` and `total_received` fields to connection status
2019-07-01 10:39:50 -04:00
Jack Robison 4f5a4d9ce0 add track_bandwidth config setting to disable bandwidth usage in status 2019-07-01 09:54:06 -04:00
Lex Berezhny 82f2ad21e1 added IGNORE to insert into trend query 2019-06-28 16:02:54 -04:00
Jack Robison 8a5b803788 add first 32260 blocks of bitcoin headers for torba unit tests 2019-06-28 15:09:49 -04:00
Lex Berezhny 86b46ebf19 fix for has_channel_signature in claim_search 2019-06-26 19:19:10 -04:00
Lex Berezhny ca999053ae update lbrycrd to v0.12.4.1 2019-06-26 12:40:26 -04:00
Lex Berezhny 8b152f3c96 check for valid fee.currency and fee.amount before processing 2019-06-26 10:44:33 -04:00
Lex Berezhny c68bfbdf2e v0.38.0 2019-06-25 16:27:01 -04:00
Jack Robison 7e8180fefc fix attribute error if availability_response field is missing 2019-06-25 16:22:02 -04:00
Jack Robison 305226a3bd fix database migration bugs 2019-06-25 16:09:24 -04:00
Jack Robison c8752b03c1 travis checksums 2019-06-25 16:09:24 -04:00
Lex Berezhny 8195deebe3 updated generated docs 2019-06-25 15:34:32 -04:00
Lex Berezhny 88f2156aeb update install to new testing tools and directory structure 2019-06-25 13:49:54 -04:00
Lex Berezhny b1982a81dc updated release script to add --start-tag option 2019-06-25 12:50:13 -04:00
Lex Berezhny 18e19213f4 fix fee_amount default on wallet server 2019-06-25 11:04:32 -04:00
Lex Berezhny 6fb67b86f5 added spv4-spv6 2019-06-25 10:42:59 -04:00
Lex Berezhny a4c15769d0 v0.38.0rc10 2019-06-25 02:01:42 -04:00
Lex Berezhny 894e86d3ee fix intermittent test_queries failure in torba due to non-unique TX generated 2019-06-25 01:33:04 -04:00
Lex Berezhny 9712ef8abf cache Homebrew 2019-06-25 01:33:04 -04:00
Lex Berezhny 908107d24f cache virtualenv directory on windows 2019-06-25 01:33:04 -04:00
Lex Berezhny d667efa9de remove docker wine based windows build process scripts 2019-06-25 01:33:04 -04:00
Lex Berezhny e649d7219b stage: build 2019-06-25 01:33:04 -04:00
Lex Berezhny 1f5698ec86 build using travis windows container 2019-06-25 01:33:04 -04:00
Jack Robison 15341aa80e fix setting the external ip from upnp 2019-06-24 15:39:31 -04:00
Lex Berezhny 3f4895bd40 v0.38.0rc9 2019-06-23 23:42:29 -04:00
Jack Robison 103c93a06c update aioupnp to 0.0.13 2019-06-23 23:39:45 -04:00
Lex Berezhny 21f02d3e6d fix torba test_sync test reliability 2019-06-23 22:40:41 -04:00
Lex Berezhny 9ce93a1da6 added --not_channel_ids and --has_channel_signature 2019-06-23 20:24:35 -04:00
Lex Berezhny d0e1468acd channel migration better handles missing or invalid certificates 2019-06-23 13:46:02 -04:00
Lex Berezhny 8a5bb239c5 fix working directory in windows build 2019-06-23 11:31:46 -04:00
Lex Berezhny 816cde32a9 fix support reorg error 2019-06-23 10:56:59 -04:00
Lex Berezhny 2b514a9df7 fix github release upload 2019-06-23 00:38:02 -04:00
Lex Berezhny ef39ba26a0 v0.38.0rc8 2019-06-23 00:08:41 -04:00
Lex Berezhny 59ca3d3d16 take_over_height fix 2019-06-22 23:24:56 -04:00
Lex Berezhny 80ff60f0aa added take_over_height to claim search / resolve output 2019-06-22 23:24:56 -04:00
Lex Berezhny 488c2e2f17 searching by fee_amount and fee_currency 2019-06-22 23:24:56 -04:00
Lex Berezhny 9e11fc1e99 dont error when claim search gets invalid channel 2019-06-22 23:24:56 -04:00
Lex Berezhny a8b590ecbb add tag normalization on storage and on search 2019-06-22 23:24:56 -04:00
Lex Berezhny 8ca8591ed6 wrong test method reverted 2019-06-22 23:24:56 -04:00
Lex Berezhny e393314423 fix for claims_in_channel not updating on abandoned claim 2019-06-22 23:24:56 -04:00
Lex Berezhny c54373bd9c moved enum into script to avoid weird import conflict with Python built-in enum module 2019-06-22 23:24:56 -04:00
Alex Grin f459b6833a Update INSTALL.md 2019-06-21 14:13:14 -04:00
Lex Berezhny 0a6eb76542 cleaned up make idea 2019-06-21 11:39:35 -04:00
Lex Berezhny e6d42b5319 .idea directory now requires running make idea first 2019-06-21 11:26:58 -04:00
Lex Berezhny 0d2804ce55 travis windows build path fixes 2019-06-21 03:27:22 -04:00
Lex Berezhny 8efdd9d29a popd directory before windows build 2019-06-21 03:27:22 -04:00
Lex Berezhny 27c36a4dce fix get_spendable_utxos 2019-06-21 03:27:22 -04:00
Lex Berezhny 2f74611844 pylint 2019-06-21 03:27:22 -04:00
Lex Berezhny d8265add2d added coin selection strategy for both prefer_confirmed and only_confirmed 2019-06-21 03:27:22 -04:00
Lex Berezhny 01cd02e4c5 import fix 2019-06-21 03:27:22 -04:00
Lex Berezhny 2b98f85961 fix module paths in .idea/lbry-sdk.iml 2019-06-21 03:27:22 -04:00
Alex Grintsvayg 338d1bd8e7 add coin selection strategy config 2019-06-21 03:27:22 -04:00
Lex Berezhny 7dedaeeef7 remaining lbry -> lbry-sdk changes 2019-06-20 23:13:57 -04:00
Lex Berezhny 54830b8968 renamed repo to lbry-sdk 2019-06-20 23:03:48 -04:00
Lex Berezhny 41b866c1ee fix linux/windows build 2019-06-20 22:50:39 -04:00
Lex Berezhny fe7224e537 residual fixes after lbrynet -> lbry move 2019-06-20 22:50:39 -04:00
Lex Berezhny f5e38f6e23 updated imports after moving lbrynet -> lbry 2019-06-20 22:50:39 -04:00
Lex Berezhny 063335377e updated lbry.iml 2019-06-20 22:50:39 -04:00
Lex Berezhny ee6f511126 moved lbrynet -> lbry 2019-06-20 22:50:39 -04:00
Lex Berezhny 75574486d3 simplify lbry.iml 2019-06-20 22:50:39 -04:00
Lex Berezhny 03a09f0f42 ignore changes to files in .idea 2019-06-20 22:50:39 -04:00
Lex Berezhny 075b3ec3d2 added default .idea files 2019-06-20 22:50:39 -04:00
Lex Berezhny 3d5aa6a44b travis fixes 2019-06-20 22:50:39 -04:00
Lex Berezhny 8350b42c1b require python 3.7 2019-06-20 22:50:39 -04:00
Lex Berezhny 7061a62e88 set correct directory in Docker build 2019-06-20 22:50:39 -04:00
Lex Berezhny 133fceeee5 fixing docker build 2019-06-20 22:50:39 -04:00
Lex Berezhny 38331b8918 travis fixups 2019-06-20 22:50:39 -04:00
Lex Berezhny 5bbba0b65a updated windows build 2019-06-20 22:50:39 -04:00
Lex Berezhny 07a63b2a0a moved lbry.png into lbry 2019-06-20 22:50:39 -04:00
Lex Berezhny 9ddc70fc81 deleted .github templates 2019-06-20 22:50:39 -04:00
Lex Berezhny 2b6f5fde13 moved more lbry files into lbry dir 2019-06-20 22:50:39 -04:00
Lex Berezhny d4b006656d torba integration test 2019-06-20 22:50:39 -04:00
Lex Berezhny b1e70723c7 travis coverage combine fix 2019-06-20 22:50:39 -04:00
Lex Berezhny 1ec1ad19f8 pin to mypy==0.701 2019-06-20 22:50:39 -04:00
Lex Berezhny b96b27ed91 travis ci install mypy 2019-06-20 22:50:39 -04:00
Lex Berezhny 8b145cb1d8 combined .travis.yml 2019-06-20 22:50:39 -04:00
Lex Berezhny 85d7adc0b5 moved lbrynet into lbry directory 2019-06-20 22:50:39 -04:00
Lex Berezhny d530599b5d Merge remote-tracking branch 'torba/move-to-lbry' 2019-06-20 14:04:30 -04:00
Lex Berezhny 16215d5b96 preparing for moving torba into lbry sdk repo 2019-06-20 13:45:46 -04:00
Jack Robison db685e5807 bump torba requirement 2019-06-19 14:23:38 -04:00
Jack Robison 190304344c v0.5.7 2019-06-19 10:55:57 -04:00
Alex Grintsvayg 24e8a3c62e more accurate coin selection strategy name 2019-06-19 10:39:11 -04:00
Lex Berezhny 571b02f381 v0.38.0rc7 2019-06-19 06:44:26 -04:00
Lex Berezhny 36e3f1b3ec integration test fix 2019-06-19 06:33:12 -04:00
Lex Berezhny 564d6bdc5d more refactoring 2019-06-19 06:33:12 -04:00
Victor Shyba 1e2b040f75 simplify validation, move to fee.update 2019-06-19 06:33:12 -04:00
Victor Shyba 5876d5c295 cover more edge cases, add tests 2019-06-19 06:33:12 -04:00
Victor Shyba 54e53932bd fix bug: fee address defaults respect replace 2019-06-19 06:33:12 -04:00
Victor Shyba c9b3c2e6aa change test to use preview=True on broadcast 2019-06-19 06:33:12 -04:00
Victor Shyba 912c102d9a fix updating fee address to claim address, fix testing for malformed claims 2019-06-19 06:33:12 -04:00
Victor Shyba 2b7cbf9759 get without fee address defaults pays to claim address 2019-06-19 06:33:12 -04:00
Oleg Silkin 1cc6b2e31e More integration unittests and mock server fixes 2019-06-19 06:25:26 -04:00
Oleg Silkin 7fa797712a Adds comment signature unit testing 2019-06-19 06:25:26 -04:00
Oleg Silkin 636f36ebad Moves get_channel,get_stream out of Class into their own functions 2019-06-19 06:25:26 -04:00
Oleg Silkin 8e0888b846 Fixes comment functions 2019-06-19 06:25:26 -04:00
Oleg Silkin ce5dd5022b Adds Comment Signing and Validation 2019-06-19 06:25:26 -04:00
Lex Berezhny 4a24cc0095 pylint & mypint fixes 2019-06-19 06:12:57 -04:00
Alex Grintsvayg f2bd0edc51 add option to only use confirmed utxos 2019-06-19 06:12:57 -04:00
Thomas Zarebczan 756d17bada update wallet servers
Not sure where spv3 will be yet. We can probably repurpose lbryumx2/3/4 once we go live, so we'll make sure to add those before release.
2019-06-19 02:53:41 -04:00
Lex Berezhny 81e061af30 added missing raise SystemExit() 2019-06-19 02:53:22 -04:00
Alex Grintsvayg a60cd1d475 separate try/except for media server 2019-06-19 02:53:22 -04:00
Lex Berezhny f78323e0fb update torba version 2019-06-19 02:17:08 -04:00
Lex Berezhny fd6cbe0f6c reorgs do not hang the wallet server 2019-06-19 02:17:08 -04:00
Lex Berezhny c30e905e19 undo backup_txs method signature change 2019-06-19 01:49:47 -04:00
Lex Berezhny f5650c1b13 minor refactor of reorg integration test 2019-06-19 01:49:47 -04:00
Lex Berezhny ecebbc86c6 fixed reorg issue with headers 2019-06-19 01:49:47 -04:00
Jack Robison bfeed40bfe Merge pull request #2247 from lbryio/fix-unable-to-serialize-response
paginated deterministically shuffled find_value
2019-06-18 23:25:12 -04:00
Jack Robison 2ad22d7d19 paginated deterministically shuffled find_value
- fixes https://github.com/lbryio/lbry/issues/2244
- reduce the max DHT datagram size to 1400 bytes
- truncate `contacts` field of find_value response datagrams to the k closest (8)
- truncate peers in find_node response datagrams to the 2k closest (16)
- remove `contacts` field from find_value responses beyond `page` 0 (the first/default)
- deterministically shuffle the peers for a blob in a find_value response
- add optional `page` argument to `find_value` and `p` field to find_value responses containing the number of pages of k peers for the blob
- test one blob being announced by 150 different peers to one peer
- speed up pylint and remove some disabled checks
2019-06-18 22:52:39 -04:00
Lex Berezhny 262735c219 --blocking 2019-06-18 19:32:49 -04:00
Alex Grin 3abc98acc8 Update PULL_REQUEST_TEMPLATE.md 2019-06-18 09:17:54 -04:00
Alex Grin eaa8ba814e Update ISSUE_TEMPLATE.md 2019-06-17 14:06:01 -04:00
Jack Robison 588568752b Merge pull request #2217 from lbryio/chunked-reflector-response
fix handling chunked reflector server responses
2019-06-11 14:05:06 -04:00
Lex Berezhny 794eac0519 0.5.6 2019-06-10 23:49:04 -04:00
Victor Shyba b75139662b refactor pick_fastest_session 2019-06-10 23:25:38 -04:00
Victor Shyba 9e24f4ca54 fix mypy, add types 2019-06-10 23:25:38 -04:00
Victor Shyba 1b5281c23e connect to multiple servers 2019-06-10 23:25:38 -04:00
Lex Berezhny be15601a67 pylint 2019-06-10 01:44:07 -04:00
Lex Berezhny 35cd42bfea ledger.wait() accept timeout argument 2019-06-10 01:44:07 -04:00
Jack Robison a30e2daf05 handle chunked responses from reflector server 2019-06-07 17:26:31 -04:00
Jack Robison adb7edd3b2 test chunked reflector server response 2019-06-07 17:26:31 -04:00
Jack Robison 06a9770504 Merge pull request #2216 from lbryio/fix-connection-status-startup-error
fix connections status raising an error during startup
2019-06-07 13:55:54 -04:00
Jack Robison 0da43d3e05 fix connections status raising an error during startup 2019-06-07 12:14:05 -04:00
Lex Berezhny 077dbb3138 better handle claims which fail to decode 2019-06-07 08:42:17 -04:00
Jack Robison 3b8fd65fa9 Merge pull request #2212 from lbryio/add-torba-version-to-version-rpc
update `version` docstring, add `torba_version`
2019-06-05 20:59:58 -04:00
Jack Robison d0ddaa5518 update version docstring, add torba_version 2019-06-05 20:33:21 -04:00
Jack Robison a60ab97552 Merge pull request #2211 from lbryio/pin-aioupnp
pin aioupnp requirement to 0.0.12
2019-06-05 19:41:45 -04:00
Jack Robison 078d09976c pin aioupnp requirement 2019-06-05 18:50:36 -04:00
Jack Robison 3770558694 Merge pull request #2210 from lbryio/fix-wine-build
fix torba version on travis
2019-06-05 18:46:24 -04:00
Jack Robison 4409b8c4f8 fix torba version on travis 2019-06-05 18:19:56 -04:00
Lex Berezhny 03e48c0968 0.5.5 2019-06-05 17:07:36 -04:00
Lex Berezhny f5b7aaac18 v0.38.0rc6 2019-06-04 22:32:33 -04:00
Lex Berezhny 6dbb6b9079 fix claim expiration calculation 2019-06-04 22:28:35 -04:00
Lex Berezhny 16b774a69e add normalized to claims_in_channel index to make update claim faster 2019-06-04 22:28:35 -04:00
Lex Berezhny 72ea19532e added index specifically for claim_in_channel computation 2019-06-04 22:28:35 -04:00
Lex Berezhny ad24946af3 add improved profiling to validate_channel_signatures 2019-06-04 22:28:35 -04:00
Lex Berezhny b676907fd7 added index for claim expiration_height 2019-06-04 22:28:35 -04:00
Lex Berezhny 6965971613 delete claims and re-calculate winner when claim is expired 2019-06-04 22:28:35 -04:00
Lex Berezhny 6651111522 pylint 2019-06-04 18:44:21 -04:00
Lex Berezhny d8630a02d9 Strings setting works like Server from CLI 2019-06-04 18:44:21 -04:00
Lex Berezhny 69345b4b0e integrationt tests fixed 2019-06-04 16:04:46 -04:00
Lex Berezhny f23aea9951 better handling for claims in abandoned channels 2019-06-04 16:04:46 -04:00
Lex Berezhny f96c46e84a claim search fixes for filtering with valid_channel_signatures 2019-06-04 16:04:46 -04:00
Victor Shyba 3ea92c0d6e pylint 2019-06-04 11:04:47 -04:00
Victor Shyba 8e43e40810 fix reconnect and improve test 2019-06-04 11:04:47 -04:00
Alex Grintsvayg bae2939943 change a few instances of lbry.io to lbry.com. fixes #2135 2019-06-04 10:23:04 -04:00
Alex Grintsvayg 7063138e3c small improvements from our cli investigation 2019-06-04 09:39:33 -04:00
Lex Berezhny 798ba9acf8 v0.38.0rc5 2019-06-04 01:20:32 -04:00
Jack Robison c8dd1987e6 add connection status 2019-06-04 01:20:05 -04:00
Lex Berezhny 8d2c9e5785 claim_type/stream_type/media_type claim search integration test 2019-06-04 00:51:25 -04:00
Lex Berezhny 9dbf47916b add public_key_id to channel integration tests 2019-06-03 21:56:17 -04:00
Lex Berezhny 565ef5e454 docopt doc string fix 2019-06-03 21:56:17 -04:00
Lex Berezhny 9f1f2b1781 claim search with --public_key_id 2019-06-03 21:56:17 -04:00
Lex Berezhny 76df239d1a start command accepts server lists consistent with other commands 2019-06-03 21:22:24 -04:00
Thomas Zarebczan 86ee6a78f5 add MKV to mime types 2019-06-03 19:51:48 -04:00
Victor Shyba a14b91cdda refactor test and make it faster 2019-06-03 16:04:15 -04:00
Victor Shyba 08609e9760 remove unnecessary call to ledger stop on client tests 2019-06-03 16:04:15 -04:00
Victor Shyba ca0a3dbf9e minor cleanups on test spurious exceptions 2019-06-03 16:04:15 -04:00
Victor Shyba 9d0b9f0ded improve server picking tests 2019-06-03 16:04:15 -04:00
Victor Shyba 8911af58cf pick fastest server on client start 2019-06-03 16:04:15 -04:00
Lex Berezhny 4fb6f2bb27 review fixes 2019-06-03 06:51:09 -04:00
Oleg Silkin ec039825e8 Fixes non-subscribtable object error 2019-06-03 06:51:09 -04:00
Oleg Silkin a10f09d169 Updates signature return field 2019-06-03 06:51:09 -04:00
Oleg Silkin 76409600e1 Fixes stuff up 2019-06-03 06:51:09 -04:00
Oleg Silkin d06a0f876a Patches comment server interface in the SDK 2019-06-03 06:51:09 -04:00
Oleg Silkin 8e857de62a Fixes non-subscribtable object error 2019-06-03 06:51:09 -04:00
Oleg Silkin 3d37f7c735 Updates signature return field 2019-06-03 06:51:09 -04:00
Oleg Silkin 552b94bc65 Fixes stuff up 2019-06-03 06:51:09 -04:00
Oleg Silkin 17164601a1 Patches comment server interface in the SDK 2019-06-03 06:51:09 -04:00
Lex Berezhny 26dd6d2ac7 claim search test fix 2019-05-30 21:57:49 -04:00
Lex Berezhny 5d5e36a2c2 short_url and canonical_url returned from wallet server use original claim name instead of normalized 2019-05-30 21:32:35 -04:00
Lex Berezhny 77afbcded6 wallet server, skip table join for count(*) query 2019-05-30 21:07:07 -04:00
Lex Berezhny ae11d72f66 added claim.effective_amount index on wallet server 2019-05-30 16:13:46 -04:00
Lex Berezhny d2773f9713 v0.38.0rc4 2019-05-29 22:48:28 -04:00
Lex Berezhny a7fba569cd update tests to use valid_channel_signatures argument 2019-05-29 22:47:33 -04:00
Lex Berezhny 0559c64038 fixed claim search bug related to signed claims 2019-05-29 22:47:33 -04:00
Lex Berezhny 5ab2754bc6 shutdown daemon during tests teardown 2019-05-29 19:27:05 -04:00
Lex Berezhny 6bfb8384e7 channel import/export working tests 2019-05-29 19:27:05 -04:00
Lex Berezhny 0156aa8b00 channel import/export wip 2019-05-29 19:27:05 -04:00
Lex Berezhny ad411cc157 trying to fix rebase 2019-05-29 19:27:05 -04:00
hackrush 6d462ad6d8 channel export import 2019-05-29 19:27:05 -04:00
Lex Berezhny d23920cc61 added get_public_key methods 2019-05-29 01:38:53 -04:00
Lex Berezhny 1b35cef77a fix wallet.server.test_sqldb test 2019-05-28 20:51:22 -04:00
Lex Berezhny a357208a77 cleaned up with passing tests 2019-05-28 20:51:22 -04:00
hackrush 7e1f8a56d5 Changed txo.private_key to be an object 2019-05-28 20:51:22 -04:00
hackrush 37665f2df7 Review Fixes 2019-05-28 20:51:22 -04:00
hackrush 8f3b596385 Add consolidated field to migration result 2019-05-28 20:51:22 -04:00
hackrush e45c6fe42c Only store pubkey-private key pairs 2019-05-28 20:51:22 -04:00
hackrush 56edf62cb9 Temporary fix for github.com/PyCQA/pylint/issues/2820 2019-05-28 20:51:22 -04:00
hackrush d367ff6ac1 Update tests and code 2019-05-28 20:51:22 -04:00
hackrush 7a78d49e13 Store public-private mappings in wallet 2019-05-28 20:51:22 -04:00
Lex Berezhny 4dae1ae05f regenerate docs 2019-05-27 23:18:26 -04:00
Lex Berezhny 9004f240e6 bug fixes 2019-05-27 23:18:26 -04:00
Lex Berezhny 1848462ad9 v0.38.0rc3 2019-05-26 14:04:45 -04:00
Lex Berezhny 786efa06dd fixed resolve to work with partial claim_ids 2019-05-26 14:01:06 -04:00
Lex Berezhny b6811d55cf no longer log skipped claim insertions on wallet server, happens too often 2019-05-26 14:01:06 -04:00
Lex Berezhny 42dc0eccd6 wallet server channel key change error 2019-05-26 14:01:06 -04:00
Lex Berezhny 29bd936181 wallet server tracks claim signature 2019-05-26 14:01:06 -04:00
Lex Berezhny 9d97b90ff4 pylint 2019-05-26 14:01:06 -04:00
Lex Berezhny 370b34f860 canonical_url takes into account whether channel signature is valid 2019-05-26 14:01:06 -04:00
Lex Berezhny affa46e0f6 initial import of canonical url via the new sql wallet server 2019-05-26 14:01:06 -04:00
Jack Robison 6b93705599 Merge pull request #2181 from lbryio/improve-set-build
improve set_build.py, pin torba version
2019-05-24 19:00:38 -04:00
Jack Robison 925e39c5a8 improve set_build.py, pin torba version
- adds BUILD_COMMIT to lbrynet.build_type
2019-05-24 18:02:55 -04:00
Jack Robison 1efa2c9406 Merge pull request #2180 from lbryio/fix-completed-false-negative
fix false negative with file `completed` field
2019-05-24 17:18:04 -04:00
Jack Robison 4f8bcfcd18 fix false negative with file completed field 2019-05-24 16:47:25 -04:00
Jack Robison 3ed72a215e Merge pull request #2179 from lbryio/fix-range-request-start-offset
start returning bytes for a range request at the requested starting position
2019-05-24 16:46:21 -04:00
Jack Robison 85d94d4ca3 start returning range request bytes at the requested offset 2019-05-24 15:32:40 -04:00
Jack Robison b922ea8e7e Merge pull request #2178 from lbryio/lru-cached-read-blob
add a lru cache for decrypted blobs to minimize redownloading them, add `blob_lru_cache_size` to the config to set the cache size
2019-05-24 13:56:59 -04:00
Jack Robison cb6ddaf276 add blob_lru_cache_size config setting, defaults to 32 2019-05-24 12:02:58 -04:00
Lex Berezhny 1c2029cddf fix startup error when single key account present in wallet 2019-05-22 18:31:49 -04:00
Alex Grintsvayg d8d136f1d8 uniform spacing on docstring 2019-05-22 16:51:14 -04:00
Victor Shyba fedc0e5bc3 add checktrie.py script 2019-05-22 12:20:33 -04:00
Lex Berezhny ee353ca776 0.5.4 2019-05-22 09:35:23 -04:00
Alex Grintsvayg 4fbccb9336 uniform spacing on docstring 2019-05-22 09:23:31 -04:00
Lex Berezhny ef5dd5593f v0.38.0rc2 2019-05-20 21:39:16 -04:00
Lex Berezhny baed974be0 several minor wallet server fixes 2019-05-20 21:35:20 -04:00
Lex Berezhny b2b3509b61 regenerate protobufs 2019-05-20 15:19:14 -04:00
Lex Berezhny 51720d9b73 add MAX() to claimtrie GROUP BY algo to make results consistent across SQLite versions 2019-05-20 15:19:14 -04:00
Lex Berezhny 6abb569ec8 v0.38.0rc1 2019-05-19 23:03:25 -04:00
Lex Berezhny c2fbbadc69 start trending calculation as wallet server sync nears best height 2019-05-19 21:16:19 -04:00
Lex Berezhny b8897223ec added zscore trending in wallet server 2019-05-19 21:16:19 -04:00
hackrush c2c184b4ef Updated to include the environment variables needed in mac 2019-05-18 21:44:49 -04:00
Lex Berezhny 07dbdef3fe dont run test_resolve_lru_cache_doesnt_persist_errors 2019-05-18 21:43:46 -04:00
Lex Berezhny b841fb2f0c fix docopt parser issue 2019-05-18 21:43:46 -04:00
Lex Berezhny 2b75982446 disable trending until its fixed 2019-05-18 21:43:46 -04:00
Lex Berezhny 021ca3efd2 updated doc strings and fixed wallet server claimtrie implementation 2019-05-18 21:43:46 -04:00
Lex Berezhny e56b72cf04 pylint and use torba master 2019-05-18 21:43:46 -04:00
Lex Berezhny 342f485ddf good stuff 2019-05-18 21:43:46 -04:00
Lex Berezhny 8fe16fbfc9 performance 2019-05-18 21:43:46 -04:00
Lex Berezhny 1e3035029e timer fix 2019-05-18 21:43:46 -04:00
Lex Berezhny 5f4c02f836 wallet server sync performance improvements 2019-05-18 21:43:46 -04:00
Lex Berezhny 03a1185e0f pylint 2019-05-18 21:43:46 -04:00
Lex Berezhny 0e1705d19e instrumented most wallet server claim related functions with timer 2019-05-18 21:43:46 -04:00
Lex Berezhny 82c739daa6 abstract _apply_constraints_for_array_attributes for wallet server 2019-05-18 21:43:46 -04:00
Lex Berezhny 4702bea9f0 claim_search --not_tags feature + integration tests 2019-05-18 21:43:46 -04:00
Lex Berezhny 75211b7e84 updated claim_search doc string 2019-05-18 21:43:46 -04:00
Lex Berezhny 9f9273bd82 working claim_search with tags 2019-05-18 21:43:46 -04:00
Lex Berezhny d2608bdf43 fixed wallet/server/test_sqldb.py tests 2019-05-18 21:43:46 -04:00
Lex Berezhny 091bef1ead sequence/amount_order resolve 2019-05-18 21:43:46 -04:00
Lex Berezhny 998ee9feba resolve integration tests fixed 2019-05-18 21:43:46 -04:00
Lex Berezhny 587f0b2ea8 integration tests for jsonrpc_transaction_show fixed 2019-05-18 21:43:46 -04:00
Lex Berezhny 048aa07e80 test_file_commands integration tests fixed 2019-05-18 21:43:46 -04:00
Lex Berezhny 7ca01511f4 fixing integration tests 2019-05-18 21:43:46 -04:00
Lex Berezhny 8dee83aac0 disable cache for integration tests while using custom torba branch 2019-05-18 21:43:46 -04:00
Lex Berezhny 879ae5cc72 .travis.yml 2019-05-18 21:43:46 -04:00
Lex Berezhny 74a9e17178 .travis.yml 2019-05-18 21:43:46 -04:00
Lex Berezhny 93d2173fad chris45 integration test passing 2019-05-18 21:43:46 -04:00
Lex Berezhny a459ee34be test_url unit test fix 2019-05-18 21:43:46 -04:00
Lex Berezhny 2e6297f834 pylint 2019-05-18 21:43:46 -04:00
Lex Berezhny 9accbfcf8b move exception handling into is_signed_by 2019-05-18 21:43:46 -04:00
Lex Berezhny 6946860521 added result schema types 2019-05-18 21:43:46 -04:00
Lex Berezhny 9b8be8682c massive progress 2019-05-18 21:43:46 -04:00
Lex Berezhny 03455310ae rewrote URL parser 2019-05-18 21:43:46 -04:00
Lex Berezhny e5edaed677 removed test code from wallet server daemon.py 2019-05-18 21:43:46 -04:00
Lex Berezhny 5e97215eaa integration tests should clone sqltrie torba branch 2019-05-18 21:43:46 -04:00
Lex Berezhny 266f94bc95 pylint 2019-05-18 21:43:46 -04:00
Lex Berezhny d26caded53 use sqltrie branch of torba 2019-05-18 21:43:46 -04:00
Lex Berezhny 2260608bb6 working sql claimtrie 2019-05-18 21:43:46 -04:00
Lex Berezhny bba4c6e4f4 added __gte sql criteria and pass header to wallet server advance_txs 2019-05-18 00:03:23 -04:00
Lex Berezhny 1d126913e9 added __or and __and to sql query generator 2019-05-18 00:03:23 -04:00
Lex Berezhny b3afee2f86 ignore too-many-statements pylint 2019-05-18 00:03:23 -04:00
Lex Berezhny 31983fbb8e support in db for constraints that reference the same column by appending a # followed by some unique string 2019-05-18 00:03:23 -04:00
Lex Berezhny ba8870823c added network.get_transaction_height 2019-05-18 00:03:23 -04:00
Lex Berezhny 5f1bc01636 swap advance_txs arguments 2019-05-18 00:03:23 -04:00
Lex Berezhny ffa9e508e9 block_processor passes height on to advance_txs 2019-05-18 00:03:23 -04:00
Lex Berezhny cfb051396a sql __in uses proper escaping instead of putting values directly in SQL 2019-05-18 00:03:23 -04:00
Lex Berezhny bb1f42057e v0.37.1 2019-05-17 23:01:47 -04:00
Lex Berezhny 695f6658a0 v0.37.1rc2 2019-05-17 21:43:37 -04:00
Jack Robison eca677a720 Merge pull request #2148 from lbryio/lru-cache-concurrent
Fix resolve internals caching and persisting errors
2019-05-17 19:26:22 -04:00
Jack Robison 51f301785f fix test, add lru_cache_concurrent decorator 2019-05-17 18:57:07 -04:00
Jack Robison 41a6e288aa add test_resolve_lru_cache_doesnt_persist_errors 2019-05-17 18:56:28 -04:00
Jack Robison 97d8d0ef4c Merge pull request #2133 from lbryio/more-wallets
Add missing wallet server
2019-05-17 15:50:10 -04:00
Niko Storni 5baeb5f538 fix examples 2019-05-17 15:19:11 -04:00
Niko Storni cc49c8407c add missing wallet server
switch to .com domain
2019-05-17 15:19:11 -04:00
Jack Robison 3f37963f7f Merge pull request #2139 from lbryio/android-platform-name
fix system name on android
2019-05-17 14:39:20 -04:00
Jack Robison 12eb83f773 fix system name on android 2019-05-17 14:00:25 -04:00
Jack Robison ca1d135ba4 Merge pull request #2138 from lbryio/fix-file-save-doc
fix save_file doc
2019-05-17 13:44:22 -04:00
Jack Robison 107e57e3ff fix docs 2019-05-17 13:11:04 -04:00
Jack Robison 928689a10d Merge pull request #2130 from lbryio/fix-datastore
Fix updating refreshed announcements in the datastore
2019-05-17 12:58:11 -04:00
Jack Robison 6ecc22f2c7 fix refreshing peers in the datastore 2019-05-17 12:22:47 -04:00
Lex Berezhny 079c81f298 v0.37.1rc1 2019-05-17 08:59:10 -04:00
Victor Shyba 8afe178d62 remove supports from resolve 2019-05-17 08:55:39 -04:00
Jack Robison 8777dcee7c v0.37.0 2019-05-13 23:39:22 -04:00
Jack Robison 1cca4a91dd Merge pull request #2128 from lbryio/add-streaming-server-settings
add `streaming_server` and `streaming_get` settings to configure the interface/port of the media server and if it supports the `/get` endpoint
2019-05-13 22:51:53 -04:00
Jack Robison 41abfbdd9b add streaming_server and streaming_get settings
-split rpc runner from range request runner
2019-05-13 22:27:48 -04:00
Jack Robison 46ee65c8f2 Merge pull request #2127 from lbryio/add-streaming-url
add `streaming_url` field to `File` api response type
2019-05-13 21:05:52 -04:00
Jack Robison fd19daad9e add streaming_url to file api response type 2019-05-13 19:05:26 -04:00
Jack Robison 26d183cbab Merge pull request #2125 from lbryio/dht_wip
Remove search junctions, make search retry, some refactoring
2019-05-13 18:02:56 -04:00
Victor Shyba 71a4be02dd use loop to create task 2019-05-13 15:57:03 -03:00
Victor Shyba 124fdb7d31 fixes from review 2019-05-13 14:57:58 -03:00
Victor Shyba 2439743804 probe peer instead of scheduling ping for removal 2019-05-13 14:51:03 -03:00
Victor Shyba 6f79875135 fix test 2019-05-13 14:27:49 -03:00
Victor Shyba b91d2190f4 disable infinite peer search, cleanup logging, tune scores to slow connections 2019-05-13 14:24:13 -03:00
Victor Shyba d024433d1b simplify accumulate task 2019-05-13 13:34:49 -03:00
Victor Shyba 7c4eb416d6 dont clearbanned if there are working peers, dont remove node immediately during search 2019-05-13 13:14:08 -03:00
hackrush 397ebe8428 Added DS_Store to .gitignore 2019-05-13 09:54:19 -04:00
Victor Shyba fb457c820a reorganizing iterative find code 2019-05-13 04:34:39 -03:00
Victor Shyba 18af2dcd4e remove peer immediately instead of probing when its bad during search 2019-05-13 02:52:24 -03:00
Victor Shyba 0075dcc2c0 iterative find probes bad peers + test case for edge scenario Tom found 2019-05-13 02:40:04 -03:00
Victor Shyba 6e327d729b fix test 2019-05-12 04:28:58 -03:00
Victor Shyba 20022d49c1 less verbose dht tests 2019-05-12 04:13:58 -03:00
Victor Shyba 32d1563885 fix handling of invalid token 2019-05-12 04:12:12 -03:00
Victor Shyba f5cef14d8d test get token on announce 2019-05-12 03:55:07 -03:00
Victor Shyba bac7d99b8a add ability to re-join network on disconnect + tests 2019-05-12 03:39:11 -03:00
Victor Shyba 6f06026511 announcer goes idle if no peers in routing table 2019-05-12 01:10:05 -03:00
Victor Shyba f02df86709 fix announce loop when there are no peers to announce 2019-05-12 00:42:19 -03:00
Victor Shyba b7d76fd09f add integration tests, fix bug that made refresh not exclude bad peers 2019-05-11 23:30:05 -03:00
Victor Shyba e675f1387c remove peer junction and some refactoring 2019-05-11 04:58:56 -03:00
Victor Shyba 0da2827c78 fix dht deadlock 2019-05-10 19:54:59 -03:00
Jack Robison 309cfc840c v0.37.0rc7 2019-05-10 15:34:38 -04:00
Jack Robison 68ee31b62a Merge pull request #2123 from lbryio/stop-timed-out-downloads
Stop timed out downloads
2019-05-10 15:29:33 -04:00
Jack Robison ad2da24d64 improve test 2019-05-10 14:57:04 -04:00
Jack Robison 648dbcc36b add timeout to blob downloads after the head blob, stop timed out streams 2019-05-10 14:50:01 -04:00
Victor Shyba e59e96190e fix peer search using inexistent variable, fix test to cover it 2019-05-10 13:49:35 -04:00
Jack Robison 7d2f592bb5 Merge pull request #2107 from lbryio/fix-update-file-we-dont-have
fix stream_update failing on metadata only updates for files we don't have
2019-05-10 12:28:53 -04:00
Jack Robison 1a182bc9bd bytes/str 2019-05-10 11:50:20 -04:00
Jack Robison 324cbdcdb2 only save the content claim if stream_hash isn't None
-stream_hash being none means we don't have the file in the database
2019-05-10 11:50:20 -04:00
Lex Berezhny 6c0c0ee322 added logging to publish command 2019-05-10 11:43:34 -04:00
Jack Robison 1e54271d8a Merge pull request #2105 from lbryio/fix-store-content-fee
fix store content fee
2019-05-09 18:47:30 -04:00
Jack Robison 54b9ed6cb2 fix store content fee 2019-05-09 18:14:54 -04:00
Lex Berezhny 2626a28e6d 0.5.3 2019-05-09 18:10:27 -04:00
Lex Berezhny fb2d7cbe28 pylint 2019-05-09 18:05:17 -04:00
Lex Berezhny 874b3faf8c use a cache when loading TXs in db.get_txos() 2019-05-09 18:05:17 -04:00
Lex Berezhny 558a02d187 correctly format lat/long and fee amounts in JSON RPC response 2019-05-09 16:19:49 -04:00
Jack Robison 3567b331c0 v0.37.0rc6 2019-05-09 11:52:34 -04:00
Jack Robison 9e4cf3c995 Merge pull request #2048 from lbryio/timeout_settings
Adjust default max_connections_per_download and peer_connect_timeout settings
2019-05-09 11:14:59 -04:00
Thomas Zarebczan db1d547b19 adjust the default max_connections_per_download and peer_connect_timeout 2019-05-09 10:38:32 -04:00
Jack Robison c31efd780b docs 2019-05-09 10:32:07 -04:00
Jack Robison 7cdb7987e3 Merge pull request #2102 from lbryio/shutdown_fast
Set a shutdown timeout for TCPSite to avoid hanging on exit
2019-05-09 10:24:13 -04:00
Victor Shyba ec07037e88 set shutdown_timeout on TCPSite 2019-05-09 01:01:07 -03:00
Lex Berezhny 9bb7b0571e fix integrat tests 2019-05-09 00:00:39 -04:00
Lex Berezhny d98162a118 stream_update --replace also clears the channel 2019-05-09 00:00:39 -04:00
Jack Robison c5ae59bff7 Merge pull request #2100 from lbryio/junction_fixes
DHT junction fixes for cases that can lock up
2019-05-08 23:52:54 -04:00
Victor Shyba efbf2f49a9 fix generator junction 2019-05-08 23:02:38 -03:00
Victor Shyba 77c69f661d add locking cases scenarios 2019-05-08 23:02:38 -03:00
Victor Shyba 2e09d2adfa close search on transport close 2019-05-08 23:02:38 -03:00
Jack Robison 8620a0e424 Merge pull request #2095 from lbryio/add-content-fee-and-saved-file-to-db
add saved_file and content_fee columns to file table
2019-05-08 20:37:56 -04:00
Jack Robison e2c123456b write decrypted blob data from run_in_executor 2019-05-08 16:07:32 -04:00
Jack Robison c9be79f062 prevent re-migrating 10 to 11 2019-05-08 15:13:07 -04:00
Jack Robison a2a1ef34ab fix tests 2019-05-08 14:55:09 -04:00
Jack Robison 761a2b235c set written_bytes, download_path, file_name, and download_directory to None if the file doesn't exist 2019-05-08 14:11:44 -04:00
Jack Robison 3ca2fee592 use stream size from claim if available 2019-05-08 14:11:39 -04:00
Jack Robison a480b2d25f fix race when closing blob client protocol after getting a response 2019-05-08 12:21:11 -04:00
Jack Robison d7032b12d7 add saved_file and content_fee columns to file table
-drop not null constraints for file_name and download_directory

-add migrator
2019-05-08 10:26:06 -04:00
Jack Robison 84381ff76c Merge pull request #2093 from lbryio/fix_announce_on_storage
Refactor announcer + fixups for last db migrator code
2019-05-07 10:46:12 -04:00
Victor Shyba 007dd43861 re-raise cancelled errors 2019-05-07 11:10:49 -03:00
Victor Shyba 1bc4e4b702 typo on logging, announcements logging is info, not warning 2019-05-07 11:10:49 -03:00
Victor Shyba b91f27219d use fetchall as there is an update while iterating 2019-05-07 11:10:49 -03:00
Victor Shyba dc4560cc9a refactor announcer 2019-05-07 11:10:49 -03:00
Victor Shyba 84b471d486 limit batch size on announcer test so it covers batching logic 2019-05-07 11:10:49 -03:00
Victor Shyba c6a3c05a0c add missing migrator line 2019-05-07 11:10:49 -03:00
Jack Robison dd665a758d Merge pull request #2065 from lbryio/streaming-bug-fixes
Streaming bug fixes
2019-05-07 10:10:39 -04:00
Jack Robison 32dc1297ec fix infinite loop 2019-05-06 18:46:26 -04:00
Jack Robison bef244ba77 dont make duplicate files 2019-05-06 18:46:26 -04:00
Jack Robison 3c231d6215 handle get with file_name/download_directory/save_file arg for a stream that is already running 2019-05-06 18:46:26 -04:00
Jack Robison a212cf6ba4 set content_fee attribute 2019-05-06 18:46:26 -04:00
Jack Robison bc060f1378 pay fee after downloading the sd blob/adding to the db 2019-05-06 18:46:26 -04:00
Jack Robison 22c701fd50 fix completed field for an item in file list 2019-05-06 18:46:26 -04:00
Jack Robison 4f0e8fce6e add optional download_directory argument to get 2019-05-06 18:46:26 -04:00
Jack Robison f642cfe9dd fix resuming save tasks on startup 2019-05-06 18:46:26 -04:00
Jack Robison 24e073680b add connection id workaround 2019-05-06 18:46:26 -04:00
Jack Robison b2f63a1545 fix tests 2019-05-06 18:46:26 -04:00
Jack Robison f506b3e6d4 tests 2019-05-06 18:46:26 -04:00
Jack Robison f7412514b3 fix partial content 0 padding 2019-05-06 18:46:26 -04:00
Jack Robison bd6a609b30 fix logging 2019-05-06 18:46:26 -04:00
Jack Robison 4e32b69d1d don't set running streams as stopped on startup 2019-05-06 18:46:26 -04:00
Jack Robison 1116c7f29e logging 2019-05-06 18:46:26 -04:00
Jack Robison d80c671cf2 look victor 2019-05-06 18:46:26 -04:00
Jack Robison 14b12cbea2 block /stream and /get requests on the stream manager starting 2019-05-06 18:46:26 -04:00
Jack Robison 1f7feafb67 force close open streaming requests and api calls on shutdown 2019-05-06 18:46:26 -04:00
Jack Robison cbe689ea7f logging 2019-05-06 18:46:26 -04:00
Jack Robison 9099ee2e8e fix/refactor starting and stopping files
-move partial content handling into ManagedStream
-add delayed stop test
2019-05-06 18:46:26 -04:00
Jack Robison b134e0c9c9 fix blob_get 2019-05-06 18:46:26 -04:00
Jack Robison f8c0e80cfc add __slots__ to BlobInfo, StreamDescriptor, and KademliaPeer 2019-05-06 18:46:26 -04:00
Jack Robison 1f266ebdad add logging for streams we already have 2019-05-06 18:46:26 -04:00
Jack Robison b09e1f8825 fix written_bytes never being 0 2019-05-06 18:46:26 -04:00
Jack Robison c496278a5b Merge pull request #2092 from lbryio/fix_announce_on_storage
Set should_announce=1 to head and descriptor blobs when saving a stream to database
2019-05-06 18:45:36 -04:00
Victor Shyba cacc65d45e fix edge case from test for single blob stream 2019-05-06 18:18:12 -03:00
Victor Shyba 2247b354b3 integration test for announcing both on publish and download 2019-05-06 17:27:12 -03:00
Victor Shyba 43111f0497 bump db revision 2019-05-06 17:19:40 -03:00
Victor Shyba 6c142b4477 add migrator for previous bugs 2019-05-06 17:12:04 -03:00
Victor Shyba e8683a24bd fix should_announce being set on stream creation 2019-05-06 16:49:52 -03:00
Victor Shyba a4d17d1705 re-enable tests for announcing on stream creation 2019-05-06 16:49:03 -03:00
Baltazar Gomez a030e13548 fix incorrect media-type of .cbr archives 2019-05-05 20:57:36 -04:00
Jack Robison 32b8601c38 Merge pull request #2083 from lbryio/fix_get_params
make save_file a parameter instead of a flag so it obeys conf
2019-05-05 19:44:29 -04:00
Victor Shyba f5bd28bfdb make save_file a parameter instead of a flag so it obeys conf 2019-05-05 19:46:42 -03:00
hackrush a54a11dca0 Making tests work on OS X 2019-05-05 16:16:00 -04:00
Lex Berezhny 6cbbe23c7c account_list defaults to 0 confirmations for balance 2019-05-04 18:08:33 -04:00
Lex Berezhny d7c2e4e9b3 fixed bug where lbrynet failed to start with single-key accounts in the wallet 2019-05-04 17:56:39 -04:00
Jack Robison d61ddbb950 Merge pull request #2076 from lbryio/download_improv
make ttfb configurable, fix race condition on writers, ban on time scales, remove idle code for getting unstuck on long downloads, fix foreign key error and improve downloader loop checks
2019-05-02 17:18:16 -04:00
Victor Shyba cf7bb6a391 fix tests and stream being deleted on data timeout 2019-05-02 16:41:48 -03:00
Victor Shyba 68d7328918 always close the write handle 2019-05-02 12:20:49 -03:00
Victor Shyba ff2985cc80 close protocol if transport reuse brings responses before requests 2019-05-02 12:13:33 -03:00
Victor Shyba ac69fcbc5f fix possible division by zero 2019-05-02 11:52:09 -03:00
Victor Shyba 609cf42868 cleanup downloader code 2019-05-01 18:04:45 -03:00
Victor Shyba e43b29fcd1 if the current writer is closed, its fine to open a new one 2019-05-01 15:09:11 -03:00
Victor Shyba 6594b7c50c test concurrency through races on getting a writer 2019-05-01 15:08:47 -03:00
Victor Shyba 971f2a1cef log download speed only if downloading full streams 2019-05-01 14:31:11 -03:00
Victor Shyba fc88261cc6 add head_blob_only to ttfb 2019-05-01 13:59:45 -03:00
Victor Shyba 895b3992c9 set type for stall download timeout 2019-05-01 13:26:51 -03:00
Victor Shyba 41951d8ec2 improve ttfb script confs 2019-05-01 13:17:13 -03:00
Victor Shyba ae2121c5c4 fix tests and new test 2019-05-01 13:16:45 -03:00
Victor Shyba 771fe935f4 test get-delete-get respects existing files 2019-05-01 13:16:32 -03:00
Victor Shyba 60f7766cf7 small adjustment to new_peer_or_finished 2019-05-01 02:55:51 -03:00
Victor Shyba 63bd983012 pylint 2019-05-01 00:42:51 -03:00
Victor Shyba 7ffce1040c fix foreign key error 2019-04-30 19:51:19 -03:00
Victor Shyba ac6835fa60 dont get stuck 2019-04-30 19:51:02 -03:00
Victor Shyba 3cf5c536c0 improve ban timing and downloader loop friction 2019-04-30 17:56:29 -03:00
Lex Berezhny 85cc1a4763 v0.37.0rc5 2019-04-29 17:22:47 -04:00
Lex Berezhny 180cfcfb72 fix chris45 tests 2019-04-29 16:01:59 -04:00
Lex Berezhny 864c1dd901 fix test_claim_commands integration tests 2019-04-29 16:01:59 -04:00
Lex Berezhny aeaa7ba986 switched from sha256 for unencrypted file hash to sha384 2019-04-29 16:01:59 -04:00
Lex Berezhny e7beb6b1a7 fix for adding new account from private_key_string 2019-04-29 14:14:31 -04:00
Lex Berezhny b8325c1f89 regenerated docs 2019-04-29 09:49:39 -04:00
Lex Berezhny a36936b5ff v0.37.0rc4 2019-04-29 00:27:53 -04:00
Victor Shyba 7cf245a0e9 encode timestamp and confirmations into file list 2019-04-29 00:26:14 -04:00
Victor Shyba 9f83cce574 test confirmations and timestamp on file list, being updated on resolve 2019-04-29 00:26:14 -04:00
Victor Shyba 30b113d64a depth -> confirmations 2019-04-29 00:26:14 -04:00
Victor Shyba ccd2023d37 add timestamp field to resolve 2019-04-29 00:26:14 -04:00
Victor Shyba 3127cd11cf test for timestamp field on resolve 2019-04-29 00:26:14 -04:00
Victor Shyba 6e483cba8d tests: add confirm parameters, like the other calls 2019-04-28 22:53:43 -04:00
Victor Shyba 46c03c0cb2 refactor: move generate to blockchain call 2019-04-28 22:53:43 -04:00
Victor Shyba c2ca0a9467 tests: replace sleep with the proper generate call 2019-04-28 22:53:43 -04:00
Victor Shyba b350e74aca fix stream update callback to update the main dict 2019-04-28 22:53:43 -04:00
Victor Shyba de9473f75a test resolve updates file list 2019-04-28 22:53:43 -04:00
Lex Berezhny bfc107fdb3 removed blockchain.block.get_block from wallet server API 2019-04-27 18:03:35 -04:00
Lex Berezhny 6a2efa54ab dropped block_show command from API, users should run a full node if they need access to blocks 2019-04-27 18:03:35 -04:00
Lex Berezhny 72f0e1ca14 added timestamp field to transaction output results from JSON RPC calls 2019-04-27 17:10:49 -04:00
Lex Berezhny 9a1262d557 regenerate docs 2019-04-27 13:13:20 -04:00
Lex Berezhny 706a8e52dc v0.37.0rc3 2019-04-27 13:12:27 -04:00
Lex Berezhny f3c8776878 fixed --preview mode on stream_create/update 2019-04-27 13:11:32 -04:00
Lex Berezhny 539ea04039 fixed signing streams using --channel_name argument 2019-04-27 11:52:29 -04:00
Lex Berezhny 8f96b07559 updating stream claim with new file now correctly updates file list removing old file 2019-04-27 10:55:20 -04:00
Lex Berezhny 0a1e97fe23 cli test fix 2019-04-26 23:37:10 -04:00
Lex Berezhny da637b9c0e added --replace option to stream_update and channel_update, publish defaults to --replace 2019-04-26 23:37:10 -04:00
Lex Berezhny 3618de1227 added --clear_fee ability to stream update command 2019-04-26 17:52:29 -04:00
Victor Shyba 3cb8a63bf9 add channel_name parameter on claim search 2019-04-26 17:30:21 -04:00
Victor Shyba bad99de5b3 add tests for channel_name, which should work for urls too 2019-04-26 17:30:21 -04:00
Lex Berezhny 3d1efbc9bb v0.37.0rc2 2019-04-25 20:40:07 -04:00
Victor Shyba 7bdef3146b fix ttfb 2019-04-25 20:38:33 -04:00
Victor Shyba 2145bfcabf install uvloop on wallet server docker image and default to it 2019-04-25 20:07:10 -04:00
Lex Berezhny b88832d0a9 regenerated docs/api.json 2019-04-24 23:03:30 -04:00
Lex Berezhny db08882f20 v0.37.0rc1 2019-04-24 21:35:59 -04:00
Jack Robison 6f52d36b22 Merge pull request #2020 from lbryio/download-range-requests
Support HTTP 206 partial content requests for streaming downloads
2019-04-24 14:21:41 -04:00
Jack Robison 056a629e87 remove unused property 2019-04-24 13:39:06 -04:00
Jack Robison 1ffb99f9f9 rename streaming_only setting -> save_files 2019-04-24 12:44:12 -04:00
Jack Robison a8e612773d test 2019-04-24 09:48:04 -04:00
Jack Robison 67088b2ca2 test_blob_writers_concurrency 2019-04-24 09:48:04 -04:00
Jack Robison dab49e812f set writer result or error before calling close_handle 2019-04-24 09:48:04 -04:00
Jack Robison 0f3f38d8a3 pylint 2019-04-24 09:48:04 -04:00
Jack Robison 7ba433c2e5 fix setting blob verified event and canceling blob writers
-raise OSError if multiple writers for one peer are opened for a blob
2019-04-24 09:48:04 -04:00
Jack Robison bb7649e8ad test file_save stopped before finished writing with streaming_only on/off 2019-04-24 09:48:04 -04:00
Jack Robison 832537a5cf set deleted downloads as streaming mode on startup 2019-04-24 09:48:04 -04:00
Jack Robison 3234d70270 add peer address/port to hash blob writer 2019-04-24 09:48:04 -04:00
Jack Robison 432fe444f8 fix cache_concurrent decorator 2019-04-24 09:48:04 -04:00
Jack Robison 861c25716e add unit tests for cache_concurrent decorator 2019-04-24 09:48:04 -04:00
Victor Shyba 9ba6653475 raise proper error when sd is not available 2019-04-24 09:48:04 -04:00
Jack Robison 8107e97304 remove debugging /streams endpoint 2019-04-24 09:48:04 -04:00
Jack Robison e2663f6d57 remove unnecessary except/raise 2019-04-24 09:48:04 -04:00
Jack Robison 35479dcbd2 add file_save integration tests 2019-04-24 09:48:04 -04:00
Jack Robison 3a997277aa reduce the amount of Blob objects created 2019-04-24 09:48:04 -04:00
Jack Robison a7d94f4514 add file_save 2019-04-24 09:48:04 -04:00
Jack Robison a0e3338177 only re-reflect streams if the sd blob is readable 2019-04-24 09:48:04 -04:00
Jack Robison d5a353066c missing await 2019-04-24 09:48:04 -04:00
Jack Robison bc29fc4465 type 2019-04-24 09:48:04 -04:00
Jack Robison 987f175fcc logging 2019-04-24 09:48:04 -04:00
Jack Robison 5b1398b588 rename test_streaming.py 2019-04-24 09:48:04 -04:00
Jack Robison 873b58efc9 update example_daemon_settings.yml 2019-04-24 09:48:04 -04:00
Jack Robison 01a81fceed more streaming_only + save_blobs integration testing 2019-04-24 09:48:04 -04:00
Jack Robison a14b0aa573 tests 2019-04-24 09:48:04 -04:00
Jack Robison a1abb7c8fb test range requests 2019-04-24 09:48:04 -04:00
Jack Robison 46b166952b non async blob_completed callback, tests 2019-04-24 09:48:04 -04:00
Jack Robison 1375153fda fix publish updates not being saved to the db, don't query the db for the stream hash 2019-04-24 09:48:04 -04:00
Jack Robison 1a14157d37 fix unawaited task 2019-04-24 09:48:04 -04:00
Jack Robison 40b9fa38b0 update mock dht 2019-04-24 09:48:04 -04:00
Jack Robison 4d0e9e0428 relative http redirect from /get -> /stream 2019-04-24 09:48:04 -04:00
Jack Robison 3a53ef5690 catch TransportNotConnected 2019-04-24 09:48:04 -04:00
Jack Robison 33a68b5cef handle dht is_closing 2019-04-24 09:48:04 -04:00
Jack Robison 63bf084a6a cancel dht response future 2019-04-24 09:48:04 -04:00
Jack Robison a882f2e8c5 merge conflicts 2019-04-24 09:48:04 -04:00
Jack Robison 119e0ec408 cancelled error 2019-04-24 09:48:04 -04:00
Jack Robison 7d782d516a track and close blob readers 2019-04-24 09:48:04 -04:00
Jack Robison d3eac48035 tests, add __slots__ to AbstractBlob 2019-04-24 09:48:04 -04:00
Jack Robison 60c9ae64b4 endpoint to call get and redirect to the partial content stream 2019-04-24 09:48:04 -04:00
Jack Robison 2d314dce60 add save_file argument to get 2019-04-24 09:48:04 -04:00
Jack Robison 287b89db66 fix tests 2019-04-24 09:48:04 -04:00
Jack Robison 7a160f7335 fix tests 2019-04-24 09:48:04 -04:00
Jack Robison 04ddee7d89 blob tests 2019-04-24 09:48:04 -04:00
Jack Robison 3a916a8e8e tests 2019-04-24 09:48:04 -04:00
Jack Robison f125468ebf fix claim not being saved to the db 2019-04-24 09:48:04 -04:00
Jack Robison f0e17cff9a support streaming downloads / range requests 2019-04-24 09:48:04 -04:00
Jack Robison 93267efe0b remove StreamAssembler, refactor ManagedStream and StreamDownloader (WIP) 2019-04-24 09:48:04 -04:00
Jack Robison c663e5a3cf add cache_concurrent decorator 2019-04-24 09:48:04 -04:00
Jack Robison 676f0015aa refactor BlobFile into AbstractBlob, BlobFile, and BlobBuffer classes 2019-04-24 09:48:04 -04:00
Jack Robison d44a79ada2 cache resolve_host, add async_timed_cache decorator 2019-04-24 09:48:04 -04:00
Jack Robison d7ed8ffaa6 add streaming_only and save_blobs settings 2019-04-24 09:48:04 -04:00
Jack Robison b230d8fbcb rename BlobFileManager -> BlobManager 2019-04-24 09:48:04 -04:00
Lex Berezhny 30edd53f3e 0.5.2 2019-04-24 00:11:14 -04:00
Lex Berezhny 7d2428fd17 regenerate api.json 2019-04-23 23:49:39 -04:00
Lex Berezhny 4157045a41 block_show command example 2019-04-23 23:49:39 -04:00
Lex Berezhny 58987857cf v0.36.0 2019-04-22 18:47:58 -04:00
Lex Berezhny fa8858325c add lbry:// to permanent_url response field 2019-04-22 18:41:11 -04:00
Lex Berezhny 3357a419ab ability to overide the file name, hash and size when updating a stream claim 2019-04-22 18:05:49 -04:00
Lex Berezhny a782adea75 regenerate api doc 2019-04-22 14:06:42 -04:00
Lex Berezhny e8991e6d52 improve account locking/unlocing CLI docs 2019-04-22 14:06:42 -04:00
Lex Berezhny 13bd81c2b6 v0.36.0rc4 2019-04-21 23:11:23 -04:00
Victor Shyba 21241b98a4 workaround for having new encoder fields on resolve so app can use them 2019-04-21 22:51:48 -04:00
Lex Berezhny bf467e06c5 added create stream image detection test 2019-04-21 22:12:02 -04:00
Lex Berezhny 886ed6415a added examples which include all metadata passed to create stream and create channel 2019-04-21 21:34:13 -04:00
Lex Berezhny 97db241109 v0.36.0rc3 2019-04-21 20:20:53 -04:00
Lex Berezhny 216379918b renamed txo sub_type to value_type in txo json encoder output 2019-04-21 17:40:40 -04:00
Lex Berezhny fcab4eb9bd make sure to use python3 when running set_build.py 2019-04-21 16:31:46 -04:00
Lex Berezhny 19f300cc45 added protobuf to get and file_list commands 2019-04-21 14:28:43 -04:00
Lex Berezhny 26fec098e4 fix stream unit tests 2019-04-21 14:28:43 -04:00
Lex Berezhny 7824de655e updated stream manager to pull protobuf instead of hex from resolve response 2019-04-21 14:28:43 -04:00
Lex Berezhny 6ef1b1dd59 resolve command also returns protobuf 2019-04-21 14:28:43 -04:00
Lex Berezhny 9fae8f5153 added --include_protobuf to add hexlified protobuf to RPC responses 2019-04-21 14:28:43 -04:00
Lex Berezhny f660691e78 v0.36.0rc2 2019-04-21 00:50:27 -04:00
Lex Berezhny bf60683ba6 update to protobuf change where ClaimList defaults to collection list type 2019-04-21 00:45:51 -04:00
Lex Berezhny 68215699f7 fix chri45 test 2019-04-21 00:45:51 -04:00
Lex Berezhny 1644ffbf22 regenerated docs 2019-04-21 00:45:51 -04:00
Lex Berezhny 71f5061848 exposed new protobuf fields in APIs and updated tests 2019-04-21 00:45:51 -04:00
Lex Berezhny 61cadb5cbe v0.36.0rc1 2019-04-20 02:34:27 -04:00
Lex Berezhny 2034104cfc test fixes 2019-04-20 02:33:41 -04:00
Lex Berezhny 633f725634 stream manager test fix 2019-04-20 02:33:41 -04:00
Lex Berezhny 8bf2e41c95 --contact_email -> --email and --homepage_url -> --website_url 2019-04-20 02:33:41 -04:00
Lex Berezhny fa9a4a75c6 update to an even newer protobufs 2019-04-20 02:33:41 -04:00
Lex Berezhny 90bef98bc3 use TXORef.hash instead of TXORef.id when signing claim with channel 2019-04-20 02:33:41 -04:00
Lex Berezhny 35c43647aa use TXORef.hash instead of TXORef.id when signing claim with channel 2019-04-20 02:33:41 -04:00
Lex Berezhny 449a583286 default to claimType == 1 when it is missing for old claim protobufs 2019-04-20 02:33:41 -04:00
Lex Berezhny b1f813782a regenerated protobuf python code 2019-04-20 02:33:41 -04:00
Lex Berezhny 60080cbd19 updating to new protobufs 2019-04-20 02:33:41 -04:00
Victor Shyba 665dba62c7 simplify save from resolve, deduplicate items 2019-04-20 01:19:40 -04:00
Victor Shyba 76dea9e89b batch save supports 2019-04-20 01:19:40 -04:00
Victor Shyba ebc9ed01c5 make lru_cache a power of two, following py docs 2019-04-20 01:19:40 -04:00
Victor Shyba 55116763e9 cache transactions 2019-04-20 01:19:40 -04:00
Victor Shyba e1a6b4e3d6 fix dict handling mistake 2019-04-20 01:19:40 -04:00
Victor Shyba 911ca8c37e cache and share txs fetches while validating concurrently on batches 2019-04-20 01:19:40 -04:00
Lex Berezhny fe446862c2 txo.hash added 2019-04-19 08:42:12 -04:00
jessop 7f5deaf6c8 adds api examples to docs 2019-04-16 14:18:23 -04:00
binaryflesh 6788e09ae9 PEP-257 docstrings
Signed-off-by: binaryflesh <logan.campos123@gmail.com>
2019-04-16 06:24:08 -04:00
Lex Berezhny 0200756218 put generated commands in groups 2019-04-12 14:24:16 -04:00
ポール ウェッブ a63442f843 Fixes and updated gitignore 2019-04-12 11:53:37 -04:00
Oleg Silkin 74b3da79aa Adds integration tests 2019-04-07 15:27:59 -04:00
Oleg Silkin b6a4589da5 Adds comment group to lbrynet with commands list and create 2019-04-07 15:27:59 -04:00
Oleg Silkin fbbb0a9931 Adds server URL to configuration settings 2019-04-07 15:27:59 -04:00
Oleg Silkin a94d65d522 Adds utility functions for making batch jsonrpc requests to comment server 2019-04-07 15:27:59 -04:00
Oleg Silkin fa2ead002e Fixes incompatible attrs issue when running pip install -e . 2019-04-06 18:43:34 -04:00
Thomas Zarebczan 8dfee4867a Update python3 > python, add mac instructions 2019-04-06 17:10:32 -04:00
Thomas Zarebczan 6614ab8e79 updated with new venv command 2019-04-06 17:10:32 -04:00
Thomas Zarebczan d32c1dfbb3 update install for windowz
Didn't need to install any build tools
Also, venv commands are a bit different.
2019-04-06 17:10:32 -04:00
Lex Berezhny 9f42b958fa run examples fix 2019-04-06 17:08:15 -04:00
Lex Berezhny 574a912f7d run examples as part of ci 2019-04-06 17:08:15 -04:00
Lex Berezhny 2cfcc3cd45 test fix 2019-04-06 17:08:15 -04:00
Lex Berezhny 9af625e627 test fixes 2019-04-06 17:08:15 -04:00
Lex Berezhny 1871c94003 new examples 2019-04-06 17:08:15 -04:00
Lex Berezhny 93220c3874 daemon json rpc return type docs 2019-04-06 17:08:15 -04:00
Lex Berezhny 362ed3f1c8 remove unused import 2019-04-06 17:08:15 -04:00
Lex Berezhny 62a182b8bb automated example generator 2019-04-06 17:08:15 -04:00
Lex Berezhny 6f76d9e24a moved integration testcase.py into lbrynet so that it can be used from scripts 2019-04-06 17:08:15 -04:00
Lex Berezhny 55e83fe05c regenerate api.json 2019-04-06 17:08:15 -04:00
Lex Berezhny 2f0c7d6a3f added title to examples 2019-04-06 17:08:15 -04:00
Lex Berezhny 59e1df9efa generate api examples 2019-04-06 17:08:15 -04:00
Lex Berezhny f0c2d16749 changed account list to use is_default instead of is_default_account and improved blockchain deamon port settings 2019-04-06 15:27:58 -04:00
Victor Shyba fd56dc68cd fix depth on resolve, add tests 2019-04-05 09:40:26 -04:00
Victor Shyba a0a810edc3 make permanent url name#claim_id on torba too 2019-04-05 09:40:26 -04:00
Victor Shyba c1d3b5984d fix old hash reverse confusion 2019-04-05 09:40:26 -04:00
Victor Shyba 1fb719dd61 fix valid_at_height not being shown for name resolutions, which fixes tests 2019-04-05 09:40:26 -04:00
Victor Shyba b87316fb02 make permanent url name#id only for now 2019-04-05 09:40:26 -04:00
Victor Shyba 65f3d78084 fix invalid signatures for missing certificates 2019-04-05 09:40:26 -04:00
Victor Shyba 47bc1526f7 test invalid signature to be False, not accepting None 2019-04-05 09:40:26 -04:00
Victor Shyba 3acabec4bb fixes from review 2019-04-03 00:51:05 -04:00
Victor Shyba 36a62d3756 port orchstr8 fix to our test case as its lbrynet specific 2019-04-03 00:51:05 -04:00
Victor Shyba 2e22473ce3 pylint and smaller fixes 2019-04-03 00:51:05 -04:00
Victor Shyba 0b0fd64642 further refactor resolve.py 2019-04-03 00:51:05 -04:00
Victor Shyba ca17d8597d move resolve from ledger to resolve.py 2019-04-03 00:51:05 -04:00
Victor Shyba d16e2716c8 handle invalid data 2019-04-03 00:51:05 -04:00
Victor Shyba 89c80cedfd test resolve for invalid data 2019-04-03 00:51:05 -04:00
Thomas Zarebczan 021e30ffde fix: extension check
disregard the first period, otherwise the `filename.` test fails.
2019-04-01 22:56:40 -04:00
Thomas Zarebczan eb2cd8fec1 stream type mapping
Mapping extensions to stream types. I went with document / model / binary for the ones we were missing. This can be adjusted later on if not correct, but at least we'll have the main ones. Added extensions for lbry, cbz, cbr also.
2019-04-01 22:56:40 -04:00
Thomas Zarebczan 6356818557 fix: time to first byte script
Fix with for metadata
2019-04-01 16:14:53 -04:00
YULIUS KURNIAWAN KRISTIANTO 8bf724a78b Update CONTRIBUTING.md 2019-04-01 16:12:49 -04:00
YULIUS KURNIAWAN KRISTIANTO 65d0a924c1 Update README.md 2019-04-01 16:12:49 -04:00
Lex Berezhny ef7bd00f22 round USD fees to nearest penny 2019-03-31 18:14:13 -04:00
Lex Berezhny 3c1fdb5114 v0.35.0 2019-03-29 21:51:05 -04:00
Lex Berezhny 83411acfbd split claim_abandon into stream_abandon and channel_abandon 2019-03-29 21:48:55 -04:00
Lex Berezhny 4230812f82 added integration test to make sure publishing the same stream re-signs it with the original channel 2019-03-29 21:14:20 -04:00
Lex Berezhny 7b57ba96dd fix docopt parsing and stream manager tests 2019-03-29 21:14:20 -04:00
Lex Berezhny 668840921f typo fix 2019-03-29 21:14:20 -04:00
Lex Berezhny a3bd335f7f --languages, --locations fully support all attributes, add BTC 2019-03-29 21:14:20 -04:00
Lex Berezhny 4e6bca1389 support_abandon sends new --keep support to same address as the first support being abandoned 2019-03-29 21:14:20 -04:00
Lex Berezhny c4b15920ff dont show supports to foreign address if the inputs are also foreign 2019-03-29 21:14:20 -04:00
Lex Berezhny 2a55e583e7 added to docstring that a channel_id is a channel claim id 2019-03-29 21:14:20 -04:00
Lex Berezhny 2c4fde06c5 --new-signing-key -> --new_signing_key 2019-03-29 21:14:20 -04:00
Lex Berezhny eca83d7f47 --clear-tags -> --clear_tags 2019-03-29 21:14:20 -04:00
Lex Berezhny 9b21bd1a6f 0.5.1 2019-03-29 21:11:13 -04:00
Lex Berezhny 169ef00b9e + txo.pubkey_hash property 2019-03-29 17:59:18 -04:00
Lex Berezhny 87e818ac39 v0.35.0rc2 2019-03-27 22:20:09 -04:00
Lex Berezhny adab7bd7b1 fix file command tests 2019-03-27 21:57:22 -04:00
Lex Berezhny 750c0e4f4f fix cli test 2019-03-27 21:57:22 -04:00
Lex Berezhny 5f8482bdb7 hash -> sd_hash, encoding fixes, update to new protobufs 2019-03-27 21:57:22 -04:00
Lex Berezhny cd266477fb stop regenerating v1 protobufs 2019-03-27 21:57:22 -04:00
Lex Berezhny 7860b956ff fixes and tests for publish command 2019-03-27 21:57:22 -04:00
Thomas Zarebczan 148481a5ca doh 2019-03-27 21:54:19 -04:00
Thomas Zarebczan 93faa276a5 update BIP32 2019-03-27 21:54:19 -04:00
Victor Shyba 53e67a6e81 fixes from rebase 2019-03-27 10:31:37 -04:00
Victor Shyba 3ad6808d87 remove unused claim_id_hash 2019-03-27 10:31:37 -04:00
Victor Shyba 071ab32ab7 fix hash references 2019-03-27 10:31:37 -04:00
Victor Shyba ab87588250 use torba parsing for addresses too, erase old code 2019-03-27 10:31:37 -04:00
Victor Shyba 22db29ee1d use torba tx parsing on wallet server 2019-03-27 10:31:37 -04:00
Lex Berezhny 66bf6c61f0 v0.35.0rc1 2019-03-25 23:57:59 -04:00
Lex Berezhny 9abf40d21f stream_type fix 2019-03-25 23:45:10 -04:00
Lex Berezhny ea4444edbe fix internal transaction api test 2019-03-25 22:54:08 -04:00
Lex Berezhny 51bb8fd283 pylint 2019-03-25 22:54:08 -04:00
Lex Berezhny 183e3ac1c7 claim search 2019-03-25 22:54:08 -04:00
Lex Berezhny 8087b457c6 reorganized commands and finished support_abandon with --keep 2019-03-25 22:54:08 -04:00
Lex Berezhny fedb7c23d0 bring back blocks_behind in wallet status 2019-03-25 22:54:08 -04:00
Lex Berezhny e2964f9728 test cli fix, take 2 2019-03-25 22:54:08 -04:00
Lex Berezhny e0cea60c87 fix cli test 2019-03-25 22:54:08 -04:00
Lex Berezhny c708c1db07 audio/video/image stream types 2019-03-25 22:54:08 -04:00
Lex Berezhny 1cb696fb5c fix test_claim_show 2019-03-25 22:54:08 -04:00
Lex Berezhny 7cd0213eb9 fixed chris45 again 2019-03-25 22:54:08 -04:00
Lex Berezhny 2007c48af3 fixed tests for setting claim values 2019-03-25 22:54:08 -04:00
Lex Berezhny 50721786d2 skip test_claim_list for now 2019-03-25 22:54:08 -04:00
Lex Berezhny 6d7eb259cb test normalization fixes 2019-03-25 22:54:08 -04:00
Lex Berezhny cc5852a9b9 update pyyaml due to security alert 2019-03-25 22:54:08 -04:00
Lex Berezhny 0592cee567 claim integration test fixes 2019-03-25 22:54:08 -04:00
Lex Berezhny 814eaca524 cli tests fixed 2019-03-25 22:54:08 -04:00
Lex Berezhny e06916f5ae fixing unit tests 2019-03-25 22:54:08 -04:00
Victor Shyba d61e1e9d8c fix proto v1 integration test 2019-03-25 22:54:08 -04:00
Lex Berezhny 25436893bc pylint and a few other fixes 2019-03-25 22:54:08 -04:00
Lex Berezhny 5b9d41fd5c regeneate script after modifying protobufs 2019-03-25 22:54:08 -04:00
Victor Shyba b1720d1659 fix setting sd hash 2019-03-25 22:54:08 -04:00
Lex Berezhny 4b3a6ab875 big support tests passes 2019-03-25 22:54:08 -04:00
Lex Berezhny 67dcede2d8 lots of stuff deleted 2019-03-25 22:54:08 -04:00
Victor Shyba 0872a6fea3 partial test_claim_list_by_channel 2019-03-25 22:54:08 -04:00
Victor Shyba a3aed3219f fix test_abandoned_channel_with_signed_claims 2019-03-25 22:54:08 -04:00
Lex Berezhny af73e1a2f5 chris45 passing again 2019-03-25 22:54:08 -04:00
Lex Berezhny 55bbf5aa74 progress on chris45 2019-03-25 22:54:08 -04:00
Lex Berezhny c1fbb02bfe dont validate inputs in json_response_encoder 2019-03-25 22:54:08 -04:00
Lex Berezhny d47575e8e0 progress 2019-03-25 22:54:08 -04:00
Victor Shyba 8fc4f4aa73 cleanup recent test changes 2019-03-25 22:54:08 -04:00
Victor Shyba fe9aa666e9 fix name is not part of old sigs 2019-03-25 22:54:08 -04:00
Victor Shyba a9e0eeba7e add tests and fix verification of old signatures 2019-03-25 22:54:08 -04:00
Victor Shyba 0830917afb delete unused test method 2019-03-25 22:54:08 -04:00
Victor Shyba 1ed086e90f bring back test for picking oldest on dupe claims 2019-03-25 22:54:08 -04:00
Victor Shyba 56db631626 fix signed channel tests, WIP on tampering 2019-03-25 22:54:08 -04:00
Victor Shyba aa029955ee fix tx api tests 2019-03-25 22:54:08 -04:00
Victor Shyba f2236eb7a4 resolve: check if there is a cert before fetching a tx 2019-03-25 22:54:08 -04:00
Lex Berezhny df649341cd json output formatting fix for Claim 2019-03-25 22:54:08 -04:00
Lex Berezhny a3c3143868 fixed remaining references to lbrynet.extras.wallet -> lbrynet.wallet 2019-03-25 22:54:08 -04:00
Lex Berezhny d0201fe7c7 base integration test class should default to VERBOSITY=WARN, not DEBUG 2019-03-25 22:54:08 -04:00
Lex Berezhny 664a2a324d fix import error on stream unit test 2019-03-25 22:54:08 -04:00
Lex Berezhny ea09aca175 pylint fixes and ClaimDict->Claim 2019-03-25 22:54:08 -04:00
Lex Berezhny ca92fddecf pylint 2019-03-25 22:54:08 -04:00
Lex Berezhny 8259d7c680 txo.claim_hash and other fixes related to proto3 refactor 2019-03-25 22:54:08 -04:00
Victor Shyba 6653da2973 all 2019-03-25 22:54:08 -04:00
Lex Berezhny e74998209d for victor 2019-03-25 22:54:08 -04:00
Lex Berezhny 2bea0f697a pylint 2019-03-25 22:54:08 -04:00
Lex Berezhny 1d68bef6f2 client side pretty much done 2019-03-25 22:54:08 -04:00
Lex Berezhny 22bacc907b moved test_channel_signing.py from tests/unit/schema to tests/unit/wallet 2019-03-25 22:54:08 -04:00
Lex Berezhny 1e8ad900c7 renamed test_signing.py to test_channel_signing.py 2019-03-25 22:54:08 -04:00
Lex Berezhny cd2b535afb moved claim signing to TXO 2019-03-25 22:54:08 -04:00
Lex Berezhny 9dfe7bbcf6 imports updated after moving lbrynet.extras.wallet to lbrynet.wallet 2019-03-25 22:54:08 -04:00
Lex Berezhny 120618ffd5 updated imports due to lbrynet.extras.wallet -> lbrynet.wallet move 2019-03-25 22:54:08 -04:00
Lex Berezhny e8abcbb06e moved lbrynet.extras.wallet -> lbrynet.wallet 2019-03-25 22:54:08 -04:00
Lex Berezhny 1ec8f0b0b4 wip 2019-03-25 22:54:08 -04:00
Lex Berezhny 323694fb12 removed address.py 2019-03-25 22:54:08 -04:00
Lex Berezhny d5e37acd8f minor Fee changes 2019-03-25 22:54:08 -04:00
Lex Berezhny 129a886f2b + fee.amount 2019-03-25 22:54:08 -04:00
Lex Berezhny cd15230a92 getting close to a nice model 2019-03-25 22:54:08 -04:00
Lex Berezhny c1b7fdc68e deleted bunch of stuff 2019-03-25 22:54:08 -04:00
Lex Berezhny 9fd67aaad6 removed legacy_schema_v1 2019-03-25 22:54:08 -04:00
Lex Berezhny ce9dc1836e moved schema/current_schema/claim.py into schema/claim.py 2019-03-25 22:54:08 -04:00
Lex Berezhny 2e557d3a36 converted everyting in schema/legacy/* into schema/jsonschema.py 2019-03-25 22:54:08 -04:00
Lex Berezhny 5df05cb37e make schema.types a module 2019-03-25 22:54:08 -04:00
Lex Berezhny 326b62dc57 moved proto2/proto3 to types/v1 and types/v2 respectively 2019-03-25 22:54:08 -04:00
Lex Berezhny 25cc7ba284 updated schema to latest types: Claim uses oneof type for Channel/Stream 2019-03-25 22:54:08 -04:00
Victor Shyba 550f064302 baseschema 2019-03-25 22:54:08 -04:00
Victor Shyba 11086b93f1 migrate other fields 2019-03-25 22:54:08 -04:00
Victor Shyba df5662136d migrate cert + adapt to latest types repo 2019-03-25 22:54:08 -04:00
Victor Shyba 291d65070d move common schema values to constants.py 2019-03-25 22:54:08 -04:00
Victor Shyba 9b64ee33dc fix imports: schema.schema -> schema.legacy_schema_v1 2019-03-25 22:54:08 -04:00
Victor Shyba 5404d26f2f schema.schema -> schema.legacy_schema_v1 2019-03-25 22:54:08 -04:00
Victor Shyba 26ade1f228 bump merged proto3 2019-03-25 22:54:08 -04:00
Victor Shyba b49f1f1b54 fix imports 2019-03-25 22:54:08 -04:00
Victor Shyba 3bafa2ad22 regenerate protos 2019-03-25 22:54:08 -04:00
Victor Shyba 378ec44f3c update make to generate proto2 and proto3 2019-03-25 22:54:08 -04:00
Victor Shyba 9d0d17b422 rename proto proto2 2019-03-25 22:54:08 -04:00
Lex Berezhny aa93b94b61 fix unit tests 2019-03-25 22:27:52 -04:00
Lex Berezhny d7dadc2f0e + __not_like db condition 2019-03-25 22:04:48 -04:00
Lex Berezhny 33ac9eea13 fix is_valid_address 2019-03-24 21:53:52 -04:00
Lex Berezhny 9ff41f748e upgrades 2019-03-24 17:09:55 -04:00
Lex Berezhny 9589ac6d02 include tx in db get_txos() 2019-03-22 03:03:48 -04:00
Lex Berezhny cfa74985e0 Revert "add hash property to txoref"
This reverts commit 38ae660a68.
2019-03-20 14:17:14 -04:00
Victor Shyba 38ae660a68 add hash property to txoref 2019-03-20 14:02:14 -04:00
Lex Berezhny ca4a07f766 ledger.is_valid_address added 2019-03-20 01:36:30 -04:00
Victor Shyba f7c8145283 fix travis not to build wallet on PR 2019-03-19 12:22:00 -04:00
Lex Berezhny 9281bdb61b lazy parse scripts 2019-03-18 23:25:00 -04:00
Lex Berezhny 80b65ae7e7 OP_0 can be an OP but also an empty string, figure this out contextually in parser 2019-03-18 18:13:49 -04:00
Lex Berezhny 0de17730ca v0.34.0 2019-03-15 16:21:29 -04:00
Jack Robison 58743ba19b Merge pull request #1987 from lbryio/more-dht-tests
improve dht unit tests
2019-03-15 15:38:51 -04:00
Jack Robison f2fefbe287 add lbrynet.dht.blob_announcer unit tests 2019-03-15 14:29:02 -04:00
Jack Robison 60a24f0e6e add lbrynet.dht.protocol.data_store unit tests 2019-03-15 12:44:55 -04:00
Jack Robison 664f91bfab add lbrynet.dht.protocol.distance unit tests 2019-03-15 12:44:41 -04:00
Jack Robison 44f92a271f move dht tests to reflect the real directory structure 2019-03-15 12:01:21 -04:00
Jack Robison 6565ca8558 improve lbrynet.dht.serialization unit tests 2019-03-14 18:45:56 -04:00
Jack Robison 3b0ba3e534 v0.34.0rc2 2019-03-14 16:03:39 -04:00
Jack Robison 8106eb1fa5 Merge pull request #1986 from lbryio/fix-timeout-no-peers
fix default download timeout not being set
2019-03-14 15:47:12 -04:00
Jack Robison c9b2829227 fix/test fixed_peer_delay and added_fixed_peers 2019-03-14 15:08:26 -04:00
Jack Robison 4f2f8b16f3 fix pending downloads not being removed 2019-03-14 13:52:00 -04:00
Jack Robison 2ab2b846b1 fix default download timeout not being set 2019-03-14 13:07:11 -04:00
Jack Robison 2df7284c29 Merge pull request #1985 from lbryio/fix-ttfb-analytics
fix the time to the head blob including the time to the sd blob in analytics
2019-03-14 11:17:37 -04:00
Jack Robison ab5d71ff61 fix the time to the head blob including the time to the sd blob in analytics 2019-03-13 16:24:23 -04:00
Lex Berezhny 99b289f135 v0.34.0rc1 2019-03-12 20:29:10 -04:00
Lex Berezhny 6db02e4362 show build type setting when building 2019-03-12 20:23:10 -04:00
Lex Berezhny 98d8b7d463 sync uses new apply APIs in torba 2019-03-12 20:22:41 -04:00
Lex Berezhny 96f1883fc4 check modified_on 2019-03-12 20:22:41 -04:00
Lex Berezhny a914de155a working sync 2019-03-12 20:22:41 -04:00
Lex Berezhny 2dee0ff0bf wip account manifest command 2019-03-12 20:22:41 -04:00
Jack Robison 993cb43e5b fix tests
-fix cleanup of a failed download
2019-03-12 20:22:01 -04:00
Jack Robison 5d212a0f82 time to first bytes analytics 2019-03-12 20:22:01 -04:00
Lex Berezhny 61a254f580 0.4.0 2019-03-12 15:36:24 -04:00
Jack Robison 0cc602ca53 Merge pull request #1978 from lbryio/sigint
handle sigint
2019-03-12 15:06:08 -04:00
Lex Berezhny 4ba90a727b make apply not abstract 2019-03-12 15:05:07 -04:00
Lex Berezhny 27558ffd89 implement abstract method for pylint 2019-03-12 15:05:07 -04:00
Lex Berezhny c44de48481 + account.apply and addressmanager.apply 2019-03-12 15:05:07 -04:00
Victor Shyba f390ff144d handle sigint 2019-03-12 13:43:12 -04:00
Jack Robison b266cde511 Merge pull request #1980 from lbryio/batched-channel-name-initialization
use _batched_select to populate channel names when initializing files
2019-03-12 13:41:58 -04:00
Jack Robison d0a1541d0a fix test 2019-03-12 13:00:43 -04:00
Jack Robison 4cbde1b08f use _batched_select to populate channel names when initializing files 2019-03-12 09:58:19 -04:00
Lex Berezhny a021ddeffd + account.modified_on 2019-03-11 22:50:39 -04:00
Victor Shyba cdd25e3b0f pylint 2019-03-11 22:50:39 -04:00
Victor Shyba 1286417aec apply scrypt as kdf, add tests 2019-03-11 22:50:39 -04:00
Lex Berezhny fcac4f1488 fix encrypt 2019-03-11 22:50:39 -04:00
Lex Berezhny f655fefe52 typing fix 2019-03-11 22:50:39 -04:00
Lex Berezhny 7979f0a683 typing fix 2019-03-11 22:50:39 -04:00
Lex Berezhny d9df2ead84 pylint fixes 2019-03-11 22:50:39 -04:00
Lex Berezhny b07ca20e0e working sync 2019-03-11 22:50:39 -04:00
Lex Berezhny c6740d8d70 v0.33.0 2019-03-08 20:00:45 -05:00
Lex Berezhny 46358142ed v0.33.0rc1 2019-03-06 22:18:04 -05:00
Lex Berezhny f0f651fc77 rely on torba cffi explicit version 2019-03-05 09:32:43 -05:00
Lex Berezhny 5d428138ed try older cffi on windows build 2019-03-05 09:32:43 -05:00
Brannon King 8d9da61c7d reverted temporary code for full local stack from 2019-03-05 09:32:43 -05:00
Brannon King c326a9382c changed to pull down the newer lbrycrd 2019-03-05 09:32:43 -05:00
Brannon King 5bdd87c904 removed name-to-claim lookup table
added unit test for Greek and made it work


pylint: revert bad move


revert claim_sequence changes


fixed broken test
2019-03-05 09:32:43 -05:00
Jack Robison ad134fe8bb v0.32.4 2019-03-01 17:24:55 -05:00
Jack Robison 2f3c616acd v0.32.4rc1 2019-03-01 16:33:59 -05:00
Jack Robison 3caa2a5039 Merge pull request #1968 from lbryio/analytics
restore old download analytics
2019-03-01 16:08:18 -05:00
Jack Robison e38a0f7a95 logging 2019-03-01 15:26:45 -05:00
Jack Robison 9de3f3ff09 don't block on sending download analytics events 2019-03-01 15:22:19 -05:00
Jack Robison df8934d472 restore old download analytics and update tests 2019-03-01 14:50:06 -05:00
Jack Robison be4a96c1b3 Merge pull request #1965 from lbryio/certifi
fix ssl errors, use certifi to make ssl contexts
2019-02-28 13:14:17 -05:00
Jack Robison d97d978961 use certifi for all operating systems 2019-02-28 12:45:56 -05:00
Jack Robison f52e44b56a typing 2019-02-28 12:42:23 -05:00
Jack Robison 10d73fe480 use certifi 2019-02-28 12:40:11 -05:00
Jack Robison 4fd2d994f2 Merge pull request #1964 from lbryio/update-aiohttp
update aiohttp and pylint
2019-02-28 12:38:17 -05:00
Jack Robison 3a1cd8985c Merge pull request #70 from lbryio/update-aiohttp
update aiohttp and astroid requirements
2019-02-28 12:17:08 -05:00
Jack Robison d95903d219 update aiohttp and pylint 2019-02-28 12:04:07 -05:00
Jack Robison a79be0513b pylint 2019-02-28 12:00:44 -05:00
Jack Robison 569ac502ed update aiohttp and astroid requirement
pylint 2.3.0 fixes aiohttp failure
2019-02-28 11:56:39 -05:00
Jack Robison 591e66bfbb Merge pull request #1952 from lbryio/no_trap
stop trapping CancelledError
2019-02-27 11:06:02 -05:00
Victor Shyba e047748961 log dropped api request instead of stack trace 2019-02-26 17:42:45 -05:00
Victor Shyba 9f071bbe36 pylint 2019-02-26 17:42:45 -05:00
Victor Shyba 2d8ebe25ed refactor so we can stop trapping CancelledError everywhere 2019-02-26 17:42:45 -05:00
Victor Shyba 7a96e742f2 stop trapping CancelledError 2019-02-26 17:42:45 -05:00
Jack Robison 13c7026b20 Merge pull request #1961 from lbryio/pin-requirements
Pin requirements
2019-02-26 17:42:13 -05:00
Jack Robison 876a3c68dd pin requirements 2019-02-26 17:20:16 -05:00
Lex Berezhny b86d2e6e73 use older astroid for pylint 2019-02-26 15:46:47 -05:00
Lex Berezhny f8d24d3402 older aiohttp but newer astroid 2019-02-26 15:43:20 -05:00
Lex Berezhny 589768864a try latest mypy and aiohttp 2019-02-26 15:38:37 -05:00
Lex Berezhny bd2512c27e pin dependency versions 2019-02-26 15:33:14 -05:00
Jack Robison 573617d6b6 Merge pull request #1957 from lbryio/dht_memory
add LRU to KademliaPeer creation
2019-02-26 10:10:33 -05:00
Victor Shyba e8c9abb944 add LRU to KademliaPeer creation 2019-02-26 09:42:32 -05:00
Jack Robison 8cee627dfa Merge pull request #1959 from lbryio/pin-jsonschema
pin jsonschema version to 2.6.0
2019-02-26 09:41:20 -05:00
Jack Robison ab4a5cee01 pin jsonschema version 2019-02-26 09:23:41 -05:00
Lex Berezhny 9e4c1120fa 0.3.2 2019-02-22 19:26:03 -05:00
Lex Berezhny 4eff0d2660 only install plyvel on linux 2019-02-22 19:25:27 -05:00
Jack Robison 09726876ad pylint 2019-02-13 22:50:43 -05:00
Jack Robison 45c6ad3f6d remove commits after pragma 2019-02-13 22:50:43 -05:00
Jack Robison 97e4d51c29 tests and feedback 2019-02-13 22:50:43 -05:00
Jack Robison f134bc264b add run_nofk to AIOSQLite to run without foreign keys 2019-02-13 22:50:43 -05:00
Victor Shyba 57eb56b92d timeout earlier on connect 2019-02-12 21:02:25 -05:00
Victor Shyba 6c68ba1d22 add timeout->retry tests] 2019-02-12 21:02:25 -05:00
Brannon King cabc58feb4 fix decoding error in torba log 2019-02-11 16:11:01 -05:00
Lex Berezhny 364f95612a 0.3.0 2019-02-10 23:11:53 -05:00
Victor Shyba 45669dad33 simplify add_keys 2019-01-29 13:17:35 -05:00
Victor Shyba 19e326c140 simplify generate_keys 2019-01-29 13:17:35 -05:00
Victor Shyba e1c0caa357 add executemany and translate batches to it 2019-01-29 13:17:35 -05:00
Victor Shyba 9d3e1af69b [wallet server] stop executor after stopping server and wait for shutdown event 2019-01-25 17:02:09 -05:00
Victor Shyba 662f2bff4c [wallet server] take control of executor 2019-01-25 11:32:08 -05:00
Victor Shyba 9a9d37cc50 [wallet server] improved/stabler signal handlers 2019-01-25 11:32:08 -05:00
Victor Shyba 3e4c9f25a8 signal handlers cant call async functions 2019-01-25 11:32:08 -05:00
Lex Berezhny 2cb9d5c636 orchstr service transfer command works without wallet availability 2019-01-16 13:28:12 -05:00
Lex Berezhny a53b60fca4 use correct env variable for setting SPV hostname 2019-01-16 13:05:23 -05:00
Lex Berezhny 09b77972d9 convert localhost to 127.0.0.1 bypassing /etc/hosts which resolves to ::1 and fails on default docker setup 2019-01-15 15:24:08 -05:00
Lex Berezhny 7a8c2fc98b removed hard coded ipv4 and replaced it with being able to provide hostname to --spv 2019-01-15 00:22:55 -05:00
Lex Berezhny 355989d6d1 test_sync.py fixes 2019-01-13 22:59:57 -05:00
Lex Berezhny 08814ad811 test_sync.py fixes 2019-01-13 22:52:49 -05:00
Lex Berezhny 6a00cf9942 mypy fixes 2019-01-13 22:44:46 -05:00
Lex Berezhny e79202de3c support for linking between nodes 2019-01-13 22:38:36 -05:00
Lex Berezhny 527e288568 explicitely specify ipv4 2019-01-13 14:50:24 -05:00
Lex Berezhny ed32d17936 use guarded stop methods in orchstr8/node.py 2019-01-13 11:44:32 -05:00
Lex Berezhny 3a8e1cd4a5 initial push of workbench 2019-01-13 02:59:54 -05:00
Lex Berezhny 186de045b3 pylint 2019-01-10 17:17:12 -05:00
Lex Berezhny feee534829 silence sql transaction errors 2019-01-10 17:03:09 -05:00
Jack Robison f4bf260cdb fix improperly handled CancelledError on shutdown 2019-01-09 12:15:14 -05:00
Lex Berezhny 442138ef36 added AdvanceTimeTestCase 2019-01-07 03:02:15 -05:00
Lex Berezhny b9b411ec30 instal older aiohttp to make pylint happy 2019-01-06 16:05:52 -05:00
Lex Berezhny d72a152195 run transaction interactions now return a result 2018-12-15 15:53:39 -05:00
Lex Berezhny 390226d2ef pylint 2018-12-15 15:53:39 -05:00
Lex Berezhny a16495d06f await self.server.stop() 2018-12-15 15:53:39 -05:00
Lex Berezhny 7092f40701 improved shutdown for torba.server and related test setup code 2018-12-15 15:53:39 -05:00
Victor Shyba 458189366f typo on session.py 2018-12-12 20:13:34 -03:00
Victor Shyba 9d772a45d5 wait the shutdown event instead of running forever 2018-12-12 20:13:08 -03:00
Victor Shyba 1c860e8578 set logging to info 2018-12-12 20:12:35 -03:00
Lex Berezhny 50fc027289 initial version of torba/server/cli.py 2018-12-12 12:54:38 -05:00
Victor Shyba e310601266 keep connection alive for idling peers 2018-12-11 16:07:48 -05:00
Lex Berezhny 78fd434686 asyncio.sleep() -> sleep() and asyncio.Event() to Event() in order to reduce overall diff after dropping curio.py 2018-12-11 15:59:15 -05:00
Lex Berezhny a6de3a9642 completely dropped curio.py 2018-12-11 15:59:15 -05:00
Lex Berezhny 5ed478ed11 dropped everything in curio.py except TaskGroup so far, TaskGroup is next 2018-12-11 15:59:15 -05:00
Lex Berezhny b93f9d4c94 removing dependence on curio abstraction 2018-12-11 15:59:15 -05:00
Victor Shyba f4ec20a2e2 tentatively enable test_reorg for travis 2018-12-10 13:48:23 -05:00
Victor Shyba 8bc07fad32 fast and furious 2018-12-07 08:47:30 -05:00
Lex Berezhny 612066165a improved integration test for tx synchronization 2018-12-06 13:38:56 -05:00
Lex Berezhny b273d58a80 0.2.5 2018-12-06 00:19:35 -05:00
Lex Berezhny b2544139e6 stress test 2018-12-05 23:45:35 -05:00
Lex Berezhny 367ffdea1f may be the sync fix 2018-12-05 23:42:16 -05:00
Lex Berezhny 4c854dd5a7 0.2.4 2018-12-05 11:50:11 -05:00
Lex Berezhny e96b2f702d 0.2.3 2018-12-05 11:49:22 -05:00
Lex Berezhny cff22a8951 pylint line too long 2018-12-05 11:48:19 -05:00
Lex Berezhny 1b7c5a1373 update_history serially handles unique addresses 2018-12-05 11:48:19 -05:00
Lex Berezhny 2bd60021aa pylint 2018-12-05 11:48:19 -05:00
Lex Berezhny 2b8c06bda7 wip sync fixing 2018-12-05 11:48:19 -05:00
Lex Berezhny c36b4626a9 added missing imports 2018-12-05 01:11:55 -05:00
Lex Berezhny b5f245d9b6 made pylint and mypy happy 2018-12-05 01:11:55 -05:00
Lex Berezhny 899a6f0d4a merged aiorpcx into torba.rpc 2018-12-05 01:11:55 -05:00
Lex Berezhny ba5fc2a627 allow 3.6 for now 2018-11-30 10:40:03 -05:00
Lex Berezhny 9525bd6ca0 do not run 3.6 tests anymore 2018-11-29 16:39:59 -05:00
Lex Berezhny 6c4f6dd59a do not install plyvel on windows 2018-11-29 16:28:38 -05:00
Jack Robison 997b573aa4 Merge pull request #42 from lbryio/catch-decrypt-errors
catch decryption errors and verify seed is valid upon decrypt
2018-11-26 14:53:33 -05:00
Jack Robison ca67c56222 catch decryption errors and verify seed is valid upon decrypt 2018-11-26 14:40:59 -05:00
Victor Shyba 56fc003a96 log errors from wallet servers 2018-11-26 00:32:20 -05:00
Lex Berezhny 8cb811ca1d force older astroid 2018-11-25 20:54:59 -05:00
Lex Berezhny 9e5246c619 drop old style classes in torba.server 2018-11-25 20:51:41 -05:00
Lex Berezhny b2bcf39872 0.2.2 2018-11-25 11:08:49 -05:00
Lex Berezhny 0fc864c2d5 update ignore path for bitcoin_headers 2018-11-25 11:08:13 -05:00
Lex Berezhny 4b39aad07e drop [test,server] extras from travis and tox.ini 2018-11-20 22:59:58 -05:00
Lex Berezhny 10c1f8e1cf drop requirements: mock, requests. merge [server] into the main requirement section 2018-11-20 22:57:59 -05:00
Lex Berezhny ed880be3e5 0.2.1 2018-11-19 19:27:07 -05:00
Lex Berezhny e5714dc1fc test sync integration test and other fixes 2018-11-19 18:14:57 -05:00
Lex Berezhny 1732167af1 use assertBalance in reconnect tests 2018-11-19 18:14:57 -05:00
Lex Berezhny ee28bbc19b pylint and mypy fixes 2018-11-19 18:14:57 -05:00
Lex Berezhny 345d4f8ab1 refactored tx sync 2018-11-19 18:14:57 -05:00
Jack Robison e745b6c16e fix wallet encryption
fixes https://github.com/lbryio/lbry/issues/1623
2018-11-19 17:15:09 -05:00
Victor Shyba ee57af8e99 restore skips for Travis 2018-11-07 16:46:47 -05:00
Victor Shyba 700f73e8e9 make tox run it all together 2018-11-07 16:46:47 -05:00
Victor Shyba fb6a7442a6 fix wallet server shutdown for tests 2018-11-07 16:46:47 -05:00
Lex Berezhny b8a3b62f0f moved import to make pylint happy, take 2 2018-11-07 15:10:27 -05:00
Lex Berezhny 78b84ba698 moved import to make pylint happy 2018-11-07 15:07:36 -05:00
Lex Berezhny fdf479241a added ledger.wait(tx) to support reliably waiting for TX to be in ledger 2018-11-07 14:42:17 -05:00
Lex Berezhny 8fcbf48bdf attempt UTXO check 5 times when UTXOs are less than the fee 2018-11-06 19:27:48 -05:00
Lex Berezhny 268e599423 handle creating transactions where input is consumed by fees and there is no output by creating an additional input which will require some change output 2018-11-06 19:16:27 -05:00
Lex Berezhny ae8fece9bb python3.6 tests pass again 2018-11-04 01:42:27 -05:00
Lex Berezhny c91c69540b orchstr8 command for orchestration 2018-11-04 01:09:16 -05:00
Lex Berezhny c497385bc7 setup.py fixes 2018-11-04 01:04:14 -05:00
Lex Berezhny 37468cad59 fix pylint 2018-11-04 01:01:03 -05:00
Lex Berezhny 5f70ae9923 reorganization 2018-11-04 01:55:50 -04:00
Lex Berezhny c1b96a36a7 skip running 3.6 tests 2018-11-03 20:52:26 -04:00
Lex Berezhny 5cdda0f3a8 fix AsyncTestCase imports 2018-11-03 20:52:26 -04:00
Lex Berezhny f0e1fc1a49 more pylint and mypy fixes 2018-11-03 20:52:26 -04:00
Lex Berezhny 77d8c68b13 pylint disable workbench check for now 2018-11-03 20:52:26 -04:00
Lex Berezhny 48d7bfd225 pylint and mypy fixes 2018-11-03 20:52:26 -04:00
Lex Berezhny f60435c878 moving and refactoring electrumx into torba 2018-11-03 20:52:26 -04:00
Victor Shyba 41332f22c3 bump aiorpcx + adapt basenetwork 2018-11-03 11:00:23 -04:00
Lex Berezhny a6ff18c872 pin aiorpcx to 0.8.2 until we can upgrade to latest 2018-10-26 16:20:07 -04:00
Lex Berezhny b90378d739 0.1.6 2018-10-25 17:49:17 -04:00
Victor Shyba a5a0089daa fix subscribe 2018-10-25 17:47:45 -04:00
Lex Berezhny 82c71e4255 v0.1.5 2018-10-21 11:53:13 -04:00
Victor Shyba dce5541c3a revert insert many due to lbrynet 2018-10-20 16:48:02 -04:00
Victor Shyba 8bbda75865 save_transaction_io back to a transaction 2018-10-20 16:48:02 -04:00
Nikita Titov e650562758 Use README as long description on PyPI 2018-10-20 10:17:02 -04:00
Lex Berezhny d3d60071ae v0.1.4 2018-10-19 11:01:35 -04:00
Victor Shyba 85503cf92b retry with a delay increasing from 0 up to 10 seconds 2018-10-19 10:59:27 -04:00
Lex Berezhny f67b872bd1 0.1.3 2018-10-18 18:21:39 -04:00
Victor Shyba c497e9884a pylint 2018-10-18 18:20:09 -04:00
Victor Shyba 3b021af1bd adds a simple sqlite asyncio wrapper 2018-10-18 18:20:09 -04:00
Lex Berezhny 6fe648b9a9 v0.1.2 2018-10-17 21:21:21 -04:00
Lex Berezhny 03594dab6a exception message correction 2018-10-17 21:20:21 -04:00
Victor Shyba b830f4c3ca fixes from code review 2018-10-17 19:43:26 -04:00
Victor Shyba 28b638f0aa prefetch and remove manual update calls 2018-10-17 19:43:26 -04:00
Victor Shyba 4d0ecd1fb4 fix history hash 2018-10-17 19:43:26 -04:00
Victor Shyba b62c5c57af re-enable test_reconnect on tox 2018-10-17 14:12:08 -04:00
Victor Shyba c7b647b1eb fix test_reconnect 2018-10-17 14:12:08 -04:00
Victor Shyba 63563c412c cancel pending requests and close connection on server switch 2018-10-17 14:12:08 -04:00
Victor Shyba 9fed3fce3e unlock networking limits 2018-10-17 14:01:54 -04:00
Lex Berezhny a8f8ff30a2 account list --show_seed fix 2018-10-17 10:57:24 -04:00
Lex Berezhny d270e84810 v0.1.1. 2018-10-17 10:39:21 -04:00
Lex Berezhny 0b8c67f2f0 move db.commit() 2018-10-16 21:56:50 -04:00
Lex Berezhny 2b73d3d109 pylint fixes 2018-10-16 21:53:23 -04:00
Lex Berezhny a7838b80c3 removed extraneous db.commit() 2018-10-16 21:49:49 -04:00
Lex Berezhny f2f2a12ea1 added commit() after insert/update operations 2018-10-16 21:45:59 -04:00
Lex Berezhny 382ebb55b9 log formatting fix 2018-10-16 11:56:53 -04:00
Lex Berezhny c6b32fa946 basedatabase.py test coverage 2018-10-15 17:15:05 -04:00
Lex Berezhny ae0e1bfaef support python 3.6 2018-10-15 17:15:05 -04:00
Lex Berezhny aa73cb9bbf elctrumx is needed by orchstr8 even in unittests 2018-10-15 17:15:05 -04:00
Lex Berezhny d60b98236b orchstr8 is needed by unittests now as well 2018-10-15 17:15:05 -04:00
Lex Berezhny 2cd1947ef5 fixing tests 2018-10-15 17:15:05 -04:00
Lex Berezhny e10f1df321 pylint and mypy fixes 2018-10-15 17:15:05 -04:00
Lex Berezhny 2c5fd4aade twisted -> asyncio 2018-10-15 17:15:05 -04:00
Victor Shyba 0ce4b9a7de tentatively refactor prefetch 2018-10-11 17:48:55 -04:00
Victor Shyba 44b1178cfe fix prefetch proofs for unconfirmed and separate prefetch logic 2018-10-11 17:48:55 -04:00
Victor Shyba 7bdcaa6737 sync: fetch proofs and txs in batches 2018-10-11 17:48:55 -04:00
Victor Shyba cab8981716 avoid reconstructing the history during loop 2018-10-11 17:48:55 -04:00
Victor Shyba 36ad834d43 add reconnection test 2018-10-11 17:45:54 -04:00
Victor Shyba b999abb9ea improve request cancelling and timeouts during switches 2018-10-11 17:45:54 -04:00
Victor Shyba 55f5c7491c fixes from review for readability 2018-10-11 17:41:26 -04:00
Victor Shyba 52b7bb0fce add test for checking db records when generating keys above the batch threshold 2018-10-11 17:41:26 -04:00
Victor Shyba b19fd8ed85 lower batch size so it works on restricted platforms
Windows and maybe some other platforms have a lower limit on sqlite variables.
2018-10-11 17:41:26 -04:00
Victor Shyba 6bdd9c9d78 optimize insert key loop 2018-10-11 17:41:26 -04:00
Victor Shyba 87c210952b enable WAL 2018-10-11 17:41:26 -04:00
Victor Shyba 4b9c3e8409 generate keys in batches of 2000s 2018-10-11 17:41:26 -04:00
Lex Berezhny c6f2d8b6fc fix pylint 2018-10-10 21:39:36 -04:00
Lex Berezhny b23481bd21 add height to TXRef 2018-10-10 21:29:29 -04:00
Lex Berezhny 64a626cbb5 v0.1.0 2018-10-09 23:59:57 -04:00
Lex Berezhny d4f964ef8e short circuit adding annotations when get_transaction does not return anything 2018-10-09 23:57:41 -04:00
Lex Berezhny 7f09405335 txi (txoid) index added 2018-10-09 23:39:00 -04:00
Lex Berezhny e37da81dc6 added database indexes 2018-10-09 22:52:43 -04:00
Lex Berezhny febadc5dd6 fix get_addresses where it was returned all addresses instead of for specific account 2018-10-09 20:36:58 -04:00
Lex Berezhny 4aea68ecd1 + account.get_*_count() methods 2018-10-09 19:29:20 -04:00
Lex Berezhny 6a97f96745 fixed database unit test 2018-10-09 16:49:37 -04:00
Lex Berezhny 3ed03c267a v0.0.9 2018-10-09 14:44:49 -04:00
Lex Berezhny c60e17b341 fix LIMIT OFFSET syntax error 2018-10-09 14:44:26 -04:00
Lex Berezhny 689c578081 get_or_create_usable_address returns a random choice when returning an existing unused address 2018-10-09 13:13:46 -04:00
Lex Berezhny 2723a55766 start() should not return until ledger has finished starting, which means self.join_network() needs to be yielded on the first connection 2018-10-08 23:27:32 -04:00
Lex Berezhny d02da2f3a5 fix test method name to make sure it gets run 2018-10-08 23:26:45 -04:00
Lex Berezhny 462fd8bad1 get_address test fixes 2018-10-08 23:26:11 -04:00
Lex Berezhny 75dd93436c more get_address fixes 2018-10-08 22:49:50 -04:00
Lex Berezhny eebfe2be08 get_address fix 2018-10-08 22:44:30 -04:00
Lex Berezhny 3a264c1eec drop ledger.get_transaction() in favor of just using ledger.db.get_transaction() directly 2018-10-08 18:50:45 -04:00
Victor Shyba 9c4dfa7f84 attach subscriptions to on_connected and stop service before discarding the object 2018-10-08 17:11:50 -04:00
Lex Berezhny d9c21ce37b v0.0.8 2018-10-08 14:53:57 -04:00
Lex Berezhny 1d84440b74 select_txos needs a join on tx table 2018-10-08 10:53:53 -04:00
Lex Berezhny 9aa22e1071 sql: pop offset/limit/order_by when doing simple count(*) query 2018-10-08 10:37:52 -04:00
Lex Berezhny 9cea8d2725 db.get_utxos_count -> db.get_utxo_count 2018-10-07 20:14:27 -04:00
Lex Berezhny 944238d542 removed unused method and minor pylint fix 2018-10-07 14:59:27 -04:00
Lex Berezhny 95cbdcb131 refactored sql queries, added full pagination support: limit, offset and count 2018-10-07 14:53:44 -04:00
Lex Berezhny 38b1fb5fba fix get_transaction unit test 2018-10-04 19:42:29 -04:00
Lex Berezhny fe59ed0105 remove unused import 2018-10-04 19:33:14 -04:00
Lex Berezhny 356ab9666f refactored get_transactions and get_txos 2018-10-04 19:27:39 -04:00
Lex Berezhny 0960762694 sql constraints support table.column dot delimited column names 2018-10-03 11:51:42 -04:00
Lex Berezhny 1f4a9cff26 ignore too-many-branches pylint issue 2018-10-03 09:41:52 -04:00
Lex Berezhny a6f97dfbde refactored queries 2018-10-03 09:41:52 -04:00
Lex Berezhny 04aa559037 simplified .gitignore 2018-10-02 09:53:23 -04:00
Lex Berezhny a55feee960 0.0.5 2018-10-01 22:03:48 -04:00
Lex Berezhny 29c0e3f15f fix for get_transaction 2018-09-25 23:28:06 -04:00
Victor Shyba b0bc06ae00 improve reorgs 2018-09-25 23:07:32 -04:00
Lex Berezhny cdbb34e976 pylint fix 2018-09-25 22:54:18 -04:00
Lex Berezhny 5c5b0ace53 mypy related fix 2018-09-25 22:46:22 -04:00
Lex Berezhny a879c822c2 check if txi.txo_ref.txo is not None before using it 2018-09-25 22:39:20 -04:00
Lex Berezhny 841d49554c fix unit tests 2018-09-25 18:21:42 -04:00
Lex Berezhny 0d22b8c23c typing and pylint fixes 2018-09-25 18:14:28 -04:00
Lex Berezhny c29b4c476d + tx.position, + tx.net_account_balance, + txo.is_my_account 2018-09-25 18:02:50 -04:00
Lex Berezhny 5977f42a7e remove unused import 2018-09-24 23:20:14 -04:00
Lex Berezhny 58d2c04b9f simplified and cleaned up typing for encryp / decrypt 2018-09-24 23:20:14 -04:00
Jack Robison c45c792657 add account encryption tests, update aes tests 2018-09-24 23:20:14 -04:00
Jack Robison c0e0b4b745 review feedback and pylint 2018-09-24 23:20:14 -04:00
Jack Robison aa47f10602 add encrypted_on_disk to BaseAccount
-encrypt the seed and private key upon to_dict() if encrypted_on_disk is true
2018-09-24 23:20:14 -04:00
Jack Robison 9268b6ca13 fix double-double sha 2018-09-24 23:20:14 -04:00
Jack Robison 4832526477 fix aes_encrypt not prepending the iv
-allow random iv to be overridden
2018-09-24 23:20:14 -04:00
Lex Berezhny 53e0fe9a69 pylint fix 2018-09-21 22:26:07 -04:00
Lex Berezhny 8ed0791b26 improved db.get_transactions() 2018-09-21 22:18:30 -04:00
Victor Shyba 8a87195f55 ecdsa -> coincurve 2018-09-21 20:05:59 -04:00
Lex Berezhny 0b75bb4052 + added get_transactions 2018-09-21 09:47:31 -04:00
Lex Berezhny 543f533a68 update bitcoin to 0.16.3 as 0.16.0 is no longer downloadable 2018-09-20 13:52:00 -04:00
Lex Berezhny 89f4ec12e9 pylint cleanup 2018-09-20 13:52:00 -04:00
Thomas Zarebczan d76758bd1d am I doing this right?
Copied from https://github.com/lbryio/lbry/commit/f1e36823025199461836733e2a1b2e22db35596e, looks like it could work, but didn't test locally.
2018-09-20 13:52:00 -04:00
Lex Berezhny 18629c93a2 defer.returnValue -> plain return per twisted support for Py3 plain return 2018-09-20 13:00:38 -04:00
Lex Berezhny b3f3fdeb1d explicit import of english words 2018-09-18 23:06:56 -04:00
Lex Berezhny 71d99046d1 dont perform pylint check in words directory 2018-09-17 22:59:24 -04:00
Lex Berezhny eedc9bf9ed store mnemonic words in python files instead of txt, this avoids packaging issues 2018-09-17 22:59:24 -04:00
Victor Shyba 94b920ab2a fix reconnect 2018-09-17 14:34:23 -04:00
Victor Shyba 873437e6a9 check for None on height to allow for genesis 2018-08-31 10:46:03 -04:00
Lex Berezhny 57abad2181 removed unnecessary address subscribe call in integration tests 2018-08-30 15:06:10 -04:00
Lex Berezhny 91d95bd42e have to explicitly subscribe accounts 2018-08-30 14:40:44 -04:00
Lex Berezhny b102afd68a - extraneous constants import 2018-08-30 12:46:51 -04:00
Lex Berezhny 9e55694512 + get_detailed_accounts 2018-08-30 11:50:11 -04:00
Lex Berezhny 00cf8131ec removed excessive white space around logo 2018-08-24 17:02:15 -04:00
Lex Berezhny 026bcd4d4d pylint fix 2018-08-23 02:56:25 -04:00
Lex Berezhny a27f89157e fixed bug where txis were duplicated in the database 2018-08-23 02:53:14 -04:00
Lex Berezhny 9a2e9d68e7 + PUSH_INTEGER 2018-08-20 22:04:22 -04:00
Lex Berezhny c961dd609a pylint fixes 2018-08-16 21:46:02 -04:00
Lex Berezhny dabedc17d0 improvements to reorg support 2018-08-16 21:41:22 -04:00
Lex Berezhny e7dcd9216b fixing tests 2018-08-16 01:21:45 -04:00
Lex Berezhny 95a22831aa unit test fixes 2018-08-16 01:11:03 -04:00
Lex Berezhny ac2fe3476a fixed unit tests in test_transaction.py 2018-08-16 01:03:18 -04:00
Lex Berezhny df251def26 pylint and mypy fixes 2018-08-16 00:56:46 -04:00
Lex Berezhny 133a86cd89 blockchain reorg handling and overall header refactor 2018-08-16 00:43:38 -04:00
Lex Berezhny 1a5654d50b Seed version 2018-08-15 17:26:09 -04:00
Lex Berezhny 5b28e77ab5 name conflict 2018-08-07 21:47:30 -04:00
Lex Berezhny f8c1790558 pylint and added type info 2018-08-07 21:36:44 -04:00
Lex Berezhny e2537e43b5 gave accounts reference to wallet so that they can save themselves after being modified 2018-08-07 21:31:29 -04:00
Lex Berezhny d4107600a5 from_dict() now also handles from_seed() 2018-08-06 02:52:52 -04:00
Lex Berezhny e996717b60 oops, basemanager shouldnt create default accounts, moved that into lbrynet manager 2018-08-06 00:26:46 -04:00
Lex Berezhny 75ddf209e2 create default account when missing and new new account.fund() to support transfer of funds between accounts 2018-08-06 00:15:19 -04:00
Lex Berezhny f5515e5e77 general purpose constraints builder 2018-08-03 21:26:53 -04:00
Lex Berezhny 1eb375ba80 add tx.fee property back 2018-08-03 12:29:02 -04:00
Lex Berezhny d3a171e638 + closest_match step between branch_and_bound and random_draw coin selections, + Transaction.create() single method instead of .pay()/.liquidate() also its more powerful 2018-08-03 10:41:40 -04:00
Lex Berezhny e154c83ffe pylint and mypy fixes 2018-07-29 14:34:56 -04:00
Lex Berezhny 69ad8e384a customizable address generator support 2018-07-29 13:13:40 -04:00
Lex Berezhny cfeb7b249b enable caching 2018-07-28 23:49:51 -04:00
Lex Berezhny 3a35951b46 travis yaml w/ anchors 2018-07-28 23:43:17 -04:00
Lex Berezhny 420ba43d5a travis yaml parse error fix 2018-07-28 23:02:12 -04:00
Lex Berezhny 2a42bda0a0 .travis.yml 2018-07-28 22:49:16 -04:00
Lex Berezhny 6be95dceab split unit tests and integration tests 2018-07-28 21:08:04 -04:00
Lex Berezhny a20c060931 install pylint from git 2018-07-28 20:57:50 -04:00
Lex Berezhny b0bd0b1fc0 + reserving outpoints should no longer have race conditions
+ converted all comment type annotations to python 3 style syntax annotations
+ pylint & mypy
2018-07-28 20:52:54 -04:00
Lex Berezhny 746aef474a hash.py dropping py2 backwards compat 2018-07-26 14:35:44 -04:00
Lex Berezhny 5a2a3727b2 + Account.get_max_gap() 2018-07-25 23:29:41 -04:00
Lex Berezhny 40651322d0 add account id to account balance listing 2018-07-25 12:05:25 -04:00
Lex Berezhny c2a9027c19 python 3.6 is also supported 2018-07-24 04:02:53 -04:00
Lex Berezhny 25046124c0 fix balance with confirmations and header event reporting 2018-07-22 22:52:21 -04:00
Lex Berezhny 2f83ebb3b5 tox fix 2018-07-21 16:12:25 -04:00
Lex Berezhny a03069bf89 travis-ci sudo:true 2018-07-21 16:02:38 -04:00
Lex Berezhny 93555cad3c ask travis to use xenial 2018-07-21 15:57:26 -04:00
Lex Berezhny 0a2d916931 py37 2018-07-21 15:06:20 -04:00
Lex Berezhny a4519d3add 0.0.4 2018-07-21 14:43:47 -04:00
Lex Berezhny 67dc297c97 use floats() instead of Decimal() when we need to serialized to JSON (while providing satoshis for accuracy) 2018-07-21 14:39:53 -04:00
Lex Berezhny 7c39e55089 callLater breaks determinism 2018-07-17 00:09:02 -04:00
Lex Berezhny 3f17dd336b 0.0.3 2018-07-17 00:01:22 -04:00
Lex Berezhny 10fd654edc improved account balance support 2018-07-16 23:58:29 -04:00
Victor Shyba 686eb7b1f0 fix a failure for channels where we catch the wrong error 2018-07-15 17:41:52 -03:00
Lex Berezhny 97532fb300 don't delete lock if it's not in locks dict and other fun things 2018-07-15 16:04:11 -04:00
Lex Berezhny e36e4c6354 pubkey should stay as blob 2018-07-15 00:40:46 -04:00
Lex Berezhny fcc7177a9a unit tests for bcd_data_stream 2018-07-14 22:38:18 -04:00
Lex Berezhny 0f786a8c43 removed unused methods 2018-07-14 22:16:39 -04:00
Lex Berezhny 184d3a5910 switched from blob to text for most columns, added a new TXRef/TXORef referencing scheme 2018-07-14 21:34:07 -04:00
Lex Berezhny 9370b1d2fa switching most columns from blob to text 2018-07-14 17:47:51 -04:00
Lex Berezhny f46550262f added SingleKey account support 2018-07-14 17:47:18 -04:00
Lex Berezhny adec715010 + ledger.get_transaction() 2018-07-12 12:14:04 -04:00
Lex Berezhny 1ce5da80d6 unit test fixes take 2 2018-07-11 22:49:54 -04:00
Lex Berezhny 0ffec82e35 unit test fixes 2018-07-11 22:47:44 -04:00
Lex Berezhny 32ba594e55 ledger API cleanup 2018-07-11 22:37:15 -04:00
Lex Berezhny feac754925 reduce verbosity for integration test 2018-07-10 21:23:13 -04:00
Lex Berezhny 565b095a8d have to keep using old electrumx until we upgrade everywhere 2018-07-10 21:17:07 -04:00
Lex Berezhny 5db792e99c .travis.yml: electrumx master now contains the extensibility feature 2018-07-10 21:09:49 -04:00
Lex Berezhny c9c3585e1e test for Transaction.liquidate 2018-07-09 22:02:18 -04:00
Lex Berezhny 4a0d461446 use extensible branch of electrumx 2018-07-09 20:57:09 -04:00
Lex Berezhny 630a805c49 improved test coverage for bip32 2018-07-09 20:22:18 -04:00
Lex Berezhny 5eb276655d + Transaction.liquidate, balance/get_utxos support constraints 2018-07-09 20:22:04 -04:00
Lex Berezhny 2006ad68ee get_balance supports custom constraints 2018-07-09 09:55:59 -04:00
Lex Berezhny ae56b81a08 test manager.get_balance() 2018-07-04 21:30:38 -04:00
Lex Berezhny e2296d4491 improving code coverage 2018-07-04 21:17:45 -04:00
Lex Berezhny dd40d2e6ce explicit env for tox 2018-07-04 20:51:18 -04:00
Lex Berezhny c3cb297205 unit test fixes and reducing support for py2 2018-07-04 20:43:25 -04:00
Lex Berezhny 6bb81e4d6f wip4victor 2018-07-01 17:20:17 -04:00
Lex Berezhny c58a1c9ead BaseLedger.stop() should also close the db 2018-06-26 20:41:03 -04:00
Lex Berezhny da265198f1 cleanup integration test 2018-06-26 20:22:35 -04:00
Lex Berezhny 1dfc18683d fixing unit tests 2018-06-26 18:31:42 -04:00
Lex Berezhny 9a467f8840 * on_transaction now produces TransactionEvents with useful info and lots of other changes 2018-06-26 17:22:05 -04:00
Lex Berezhny aba3ed7ca0 SPV support 2018-06-25 09:54:35 -04:00
Lex Berezhny 22b897db2e unit test fix 2018-06-17 23:29:23 -04:00
Lex Berezhny 833ef98ff5 refactoring and added basic test for reading/writing wallet file 2018-06-17 23:22:15 -04:00
Lex Berezhny 43cd9c4100 removed unused msqr module 2018-06-14 16:27:17 -04:00
Lex Berezhny 4dee59a49d better customization during testing 2018-06-14 15:17:59 -04:00
Lex Berezhny 52846a3b24 switch to master branch of orchstr8 2018-06-14 00:03:52 -04:00
Lex Berezhny ac0c88768d moved orchstr8 download into tox.ini instead of .travis.yml 2018-06-13 23:00:53 -05:00
Lex Berezhny 9977a398af orchstr8 download needs electrumx installed 2018-06-13 23:00:53 -05:00
Lex Berezhny 643998332e orchstr8 download needs torba installed 2018-06-13 23:00:53 -05:00
Lex Berezhny 13444ae6c3 try downloading blockchain binary before tests run to prevent timeout during testing 2018-06-13 23:00:53 -05:00
Lex Berezhny 05ec13394c only run py36 and bitcoinsegwit integration tests until we work out the kinks 2018-06-13 23:00:53 -05:00
Lex Berezhny b7cbe16159 spelling fix and cleaned up tox.ini 2018-06-13 23:00:53 -05:00
Lex Berezhny f0ecf58752 use wip branch of orchstr8 2018-06-13 23:00:53 -05:00
Lex Berezhny e9fbc875e0 passing tests 2018-06-13 23:00:53 -05:00
Lex Berezhny 99ae39012f tests/__init__.py 2018-06-13 23:00:53 -05:00
Lex Berezhny fbd1b2ec30 monday progress 2018-06-13 23:00:53 -05:00
Lex Berezhny eb6781481a progress since thursday 2018-06-13 23:00:53 -05:00
Lex Berezhny 78e4135159 wip 2018-06-13 23:00:53 -05:00
Lex Berezhny 54350bcbc4 BaseOutputAmountEstimator 2018-06-03 20:13:30 -04:00
Lex Berezhny 09a32174c8 handle tuples in network RPC request data 2018-06-03 20:07:40 -04:00
Lex Berezhny 740af0465e send_amount_to_address 2018-06-03 20:06:55 -04:00
Lex Berezhny 08f3ecbf25 initial readme 2018-06-01 01:52:51 -04:00
Lex Berezhny 02768a8181 shorter icon 2018-06-01 01:34:30 -04:00
Lex Berezhny c86113b288 torba icon 2018-06-01 01:32:12 -04:00
Lex Berezhny 246cc234c4 test fixes to get it running on travis 2018-05-25 13:19:54 -04:00
Lex Berezhny 4e4c4b5f96 fixing travis 2018-05-25 11:11:28 -04:00
Lex Berezhny ca6d5937ba stream.where and stream.async_where 2018-05-25 09:54:01 -04:00
Lex Berezhny ece2db08da initial import 2018-05-25 02:03:25 -04:00
471 changed files with 60957 additions and 16431 deletions

No files matched your search

-74
View File
@@ -1,74 +0,0 @@
<!--
Thanks for reporting an issue to LBRY and helping us improve!
To make it possible for us to help you, please fill out below information carefully.
Before reporting any issues, please make sure that you're using the latest version.
- App: https://github.com/lbryio/lbry-desktop/releases
- Daemon: https://github.com/lbryio/lbry/releases
We are also available on Discord at https://chat.lbry.io
-->
## The Issue
In order to <achieve some value>,
as a <type of user>,
I want <some functionality>.
### Steps to reproduce
1.
2.
3.
### Expected behaviour
Tell us what should happen
### Actual behaviour
Tell us what happens instead
## System Configuration
<!-- For the app, this info is in the About section at the bottom of the Help page.
You can include a screenshot instead of typing it out -->
<!-- For the daemon, run:
curl 'http://localhost:5279' --data '{"method":"version"}'
and include the full output -->
- LBRY Daemon version:
- LBRY App version:
- LBRY Installation ID:
- Operating system:
## Anything Else
<!-- Include anything else that does not fit into the above sections -->
## Screenshots
<!-- If a screenshot would help explain the bug, please include one or two here -->
## Internal Use
### Acceptance Criteria
1.
2.
3.
### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed
-38
View File
@@ -1,38 +0,0 @@
## PR Checklist
Please check all that apply to this PR using "x":
- [ ] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
- [ ] I have checked that this PR does not introduce a breaking change
- [ ] This PR introduces breaking changes and I have provided a detailed explanation below
## PR Type
What kind of change does this PR introduce?
Why is this change necessary?
<!-- Please check all that apply to this PR using "x". -->
- [ ] Bugfix
- [ ] Feature
- [ ] Breaking changes (bugfix or feature that introduces breaking changes)
- [ ] Code style update (formatting)
- [ ] Refactoring (no functional changes)
- [ ] Documentation changes
- [ ] Other - Please describe:
## Fixes
Issue Number: N/A
## What is the current behavior?
## What is the new behavior?
## Other information
<!-- If this PR contains a breaking change, please describe the impact and solution strategy for existing applications below. -->
+1 -8
View File
@@ -1,9 +1,2 @@
/build
/dist
/.tox
/.idea
/.coverage
lbrynet.egg-info
__pycache__
_trial_temp/
/.DS_Store
+82 -66
View File
@@ -7,124 +7,140 @@ jobs:
include:
- stage: code quality
name: "pylint lbrynet"
name: "pylint & mypy"
install:
- pip install astroid==2.0.4 aiohttp==3.4.4
# newer astroid and aiohttp fails in pylint so we pre-install older version
- pip install pylint
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .
script: pylint lbrynet
- make install
script: make lint
- stage: test
name: "Unit Tests"
name: "LBRY Unit Tests"
install:
- pip install coverage
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .
- make install
script:
- HOME=/tmp coverage run -p --source=lbrynet -m unittest discover -vv tests.unit
- cd lbry && HOME=/tmp coverage run -p --source=lbry -m unittest discover -vv tests.unit
after_success:
- coverage combine
- coverage combine lbry/
- bash <(curl -s https://codecov.io/bash)
- name: "Integration Tests"
- name: "LBRY Integration Tests"
install:
- pip install tox-travis coverage
- pushd .. && git clone https://github.com/lbryio/torba.git && popd
script: tox
- pip install coverage tox-travis
- sudo mount -o mode=1777,nosuid,nodev -t tmpfs tmpfs /tmp
script: cd lbry && tox
after_success:
- coverage combine tests/
- coverage combine lbry
- bash <(curl -s https://codecov.io/bash)
- stage: build
name: "Windows"
language: generic
services:
- docker
- &torba-tests
name: "Torba Unit Tests"
env: TESTTYPE=unit
install:
- docker pull lbry/pyinstaller34_32bits:py371
- pip install coverage tox-travis
script: cd torba && tox
after_success:
- coverage combine torba/tests
- bash <(curl -s https://codecov.io/bash)
- <<: *torba-tests
name: "Torba Integration Tests"
env: TESTTYPE=integration
- name: "Run Examples"
install:
- make install
script:
- python scripts/set_build.py
- docker run -v "$(pwd):/src/lbry" lbry/pyinstaller34_32bits:py371 lbry/scripts/wine_build.sh
- sudo zip -j dist/lbrynet-windows.zip dist/lbrynet.exe
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-windows.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-windows.zip
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- cd lbry && HOME=/tmp coverage run -p --source=lbry scripts/generate_json_api.py
after_success:
- coverage combine lbry
- bash <(curl -s https://codecov.io/bash)
- &build
stage: build
name: "Linux"
env: OS=linux
install:
- pip3 install pyinstaller
- pip3 install git+https://github.com/lbryio/torba.git
- pip install pyinstaller awscli
- cd torba && pip install -e . && cd ..
- cd lbry
- python scripts/set_build.py
- pip3 install -e .
- pip install -e .
script:
- pyinstaller -F -n lbrynet lbrynet/extras/cli.py
- chmod +x dist/lbrynet
- zip -j dist/lbrynet-${OS}.zip dist/lbrynet
- ./dist/lbrynet --version
- pyinstaller -F -n lbrynet lbry/extras/cli.py
- cd dist
- chmod +x lbrynet
- zip -j lbrynet-${OS}.zip lbrynet
- shasum -a 256 -b lbrynet-${OS}.zip
- ./lbrynet --version
after_success:
- aws configure set aws_access_key_id $ARTIFACTS_KEY
- aws configure set aws_secret_access_key $ARTIFACTS_SECRET
- aws configure set region us-east-1
- export S3_PATH="daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})"
- aws s3 cp lbrynet-${OS}.zip s3://build.lbry.io/${S3_PATH}/lbrynet-${OS}.zip
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-${OS}.zip
api_key:
secure: "unnR+aSJ1937Cl1PyBBZzGuZvV5W5TGcXELhXTgyOeeI6FgO/j80qmbNxJDA7qdFH/hvVicQFWoflhZu2dxN5rYP5BQJW3q3XoOLY3XAc1s1vicFkwqn3TIfdFiJTz+/D9eBUBBhHKeYFxm3M+thvklTLgjKl6fflh14NfGuNTevK9yQke8wewW3f9UmFTo1qNOPF1OsTZRbwua6oQYa59P+KukoPt4Dsu1VtILtTkj7hfEsUL79cjotwO3gkhYftxbl/xeDSZWOt+9Nhb8ZKmQG/uDx4JiTMm5lWRk4QB7pUujZ1CftxCYWz/lJx9nuJpdCOgP624tcHymErNlD+vGLwMTNslcXGYkAJH6xvGyxBJ+Obc8vRVnZbRM26BfH34TcPK1ueRxHSrDUbzMIIUsgcoZAxBuim8uDPp+K7bGqiygzSs2vQfr9U5Jhe9/F8sPdtNctfJZEfgmthNTeVFjyNsGIfIt754uGSfACqM7wDLh6fbKx7M+FHlNyOdvYCrbKUOAYXmTikYIpVDvlaaeMO+N+uW8Rhvm1j+JU7CVwhMavLySaPVc6Dt5OxiMMmxw9mVrjW9bBPjS5AkrS5MOA13T5wapoLzH6+gE92U4HzA6ilMcwRaQPSFnK2JU7tzyt2Wy1PH4MjHowXI2WyICG1x510dD3tX1P/1px8ro="
file: lbrynet-${OS}.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-${OS}.zip
# artifact uploader thinks lbrynet is a directory, https://github.com/travis-ci/artifacts/issues/78
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- <<: *build
name: "Mac"
os: osx
osx_image: xcode8.3
language: generic
language: shell
env: OS=mac
cache: false
before_install:
- brew upgrade python || true
- brew upgrade python || true
- pip3 install --user --upgrade pip virtualenv
- /Users/travis/Library/Python/3.7/bin/virtualenv $HOME/venv
- source $HOME/venv/bin/activate
before_cache:
- brew cleanup
- <<: *build
name: "Windows"
os: windows
language: shell
env:
- OS=windows
- PATH=/c/Python37:/c/Python37/Scripts:/C/Windows/System32/downlevel:$PATH
before_install:
- choco install python --x86
- python -m pip install --upgrade pip
- pip install pywin32
script:
- pyinstaller --additional-hooks-dir=scripts/. --icon=icons/lbry256.ico -F -n lbrynet lbry/extras/cli.py
- cd dist
- 7z a -tzip lbrynet-windows.zip lbrynet.exe
- sha256sum -b lbrynet-windows.zip
- ./lbrynet.exe --version
- if: tag IS present
stage: build
name: "Wallet Server Docker Image - Tagged Release"
script:
- set -e
- echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
- travis_retry docker build -t lbry/wallet-server:$TRAVIS_TAG -f scripts/Dockerfile.wallet_server .
- travis_retry docker build -t lbry/wallet-server:$TRAVIS_TAG -f lbry/scripts/Dockerfile.wallet_server .
- docker push lbry/wallet-server:$TRAVIS_TAG
- if: tag IS blank AND branch = master
- if: tag IS blank AND branch = master AND NOT type IN (pull_request)
stage: build
name: "Wallet Server Docker Image - Master"
script:
- set -e
- echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
- travis_retry docker build -t lbry/wallet-server:master -f scripts/Dockerfile.wallet_server .
- travis_retry docker build -t lbry/wallet-server:master -f lbry/scripts/Dockerfile.wallet_server .
- docker push lbry/wallet-server:master
cache:
directories:
- $HOME/venv
- $HOME/.cache/pip
- $HOME/Library/Caches/pip
- $HOME/Library/Caches/Homebrew
- $TRAVIS_BUILD_DIR/.tox
+1 -1
View File
@@ -1,3 +1,3 @@
## Contributing to LBRY
https://lbry.io/faq/contributing
https://lbry.tech/contribute
-6
View File
@@ -1,6 +0,0 @@
# Add a CHANGELOG entry for app changes
has_app_changes = !git.modified_files.grep(/lbrynet/).empty?
if !git.modified_files.include?("CHANGELOG.md") && has_app_changes
fail("Please include a CHANGELOG entry.")
message "See http://keepachangelog.com/en/0.3.0/ for details on good changelog guidelines"
end
+56 -22
View File
@@ -9,35 +9,34 @@ Here's a video walkthrough of this setup, which is itself hosted by the LBRY net
## Prerequisites
Running `lbrynet` from source requires Python 3.6 or higher (3.7 is preferred). Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/)
Running `lbrynet` from source requires Python 3.7 or higher. Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/)
After installing python 3, you'll need to install some additional libraries depending on your operating system.
### Virtualenv
Once python 3 is installed run `python3 -m pip install virtualenv` to install virtualenv.
### Windows
Windows users will need to install `Visual C++ Build Tools`, which can be installed by [Microsoft Build Tools](Microsoft Build Tools 2015)
### macOS
macOS users will need to install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
These environment variables also need to be set
1. PYTHONUNBUFFERED=1
2. EVENT_NOKQUEUE=1
Remaining dependencies can then be installed by running:
```
brew install python3 protobuf
brew install python protobuf
```
Assistance installing Python3: https://docs.python-guide.org/starting/install3/osx/
### Linux
On Ubuntu (we recommend 18.04), install the following:
On Ubuntu (16.04 minimum, we recommend 18.04), install the following:
```
sudo apt-get install build-essential python3.7 python3.7-dev git python-virtualenv libssl-dev python-protobuf
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install build-essential python3.7 python3.7-dev git python3.7-venv libssl-dev python-protobuf
```
On Raspbian, you will also need to install `python-pyparsing`.
@@ -46,26 +45,61 @@ If you're running another Linux distro, install the equivalent of the above pack
## Installation
To install:
### Linux/Mac
To install on Linux/Mac:
```
git clone https://github.com/lbryio/lbry.git
cd lbry
Clone the repository:
$ git clone https://github.com/lbryio/lbry-sdk.git
$ cd lbry-sdk
virtualenv lbry-venv --python=python3.7
source lbry-venv/bin/activate
Create a Python virtual environment for lbry-sdk:
$ python3.7 -m venv lbry-venv
Activating lbry-sdk virtual environment:
$ source lbry-venv/bin/activate
Make sure you're on Python 3.7+ (as the default Python in virtual environment):
$ python --version
python --version # Python 2 is not supported. Make sure you're on Python 3.7
Install packages:
$ make install
pip install -e .
If you are on Linux and using PyCharm, generates initial configs:
$ make idea
```
To verify your installation, `which lbrynet` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
To verify your installation, `which lbrynet` should return a path inside of the `lbry-venv` folder created by the `python3.7 -m venv lbry-venv` command.
### Windows
To install on Windows:
```
Clone the repository:
> git clone https://github.com/lbryio/lbry-sdk.git
> cd lbry-sdk
Create a Python virtual environment for lbry-sdk:
> python -m venv lbry-venv
Activating lbry-sdk virtual environment:
> lbry-venv\Scripts\activate
Install packages:
> cd torba
> pip install -e .
> cd ../lbry
> pip install -e .
```
## Run the tests
To run the unit tests from the repo directory:
```
trial --reactor=asyncio tests.unit
python -m unittest discover -s lbry tests.unit
```
## Usage
+18
View File
@@ -0,0 +1,18 @@
install:
cd torba && pip install -e .
cd lbry && pip install -e .
pip install mypy==0.701
pip install coverage astroid pylint
lint:
cd lbry && pylint lbry
cd torba && pylint --rcfile=setup.cfg torba
cd torba && mypy --ignore-missing-imports torba
test:
cd lbry && tox
cd torba && tox
idea:
mkdir -p .idea
cp -r lbry/scripts/idea/* .idea
+11 -11
View File
@@ -1,19 +1,19 @@
# <img src="https://raw.githubusercontent.com/lbryio/lbry/master/lbry.png" alt="LBRY" width="48" height="36" /> LBRY SDK [![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry) [![Test Coverage](https://codecov.io/gh/lbryio/lbry/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry)
# <img src="https://raw.githubusercontent.com/lbryio/lbry-sdk/master/lbry/lbry.png" alt="LBRY" width="48" height="36" /> LBRY SDK [![Build Status](https://travis-ci.org/lbryio/lbry-sdk.svg?branch=master)](https://travis-ci.org/lbryio/lbry-sdk) [![Test Coverage](https://codecov.io/gh/lbryio/lbry-sdk/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry-sdk)
LBRY is a decentralized peer-to-peer protocol for publishing and accessing digital content. It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, rights and access rules. LBRY also provides a data network that consists of peers (seeders) uploading and downloading data from other peers, possibly in exchange for payments, as well as a distributed hash table used by peers to discover other peers.
LBRY SDK for Python is currently the most full featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components:
* Built on Python 3.7+ and `asyncio`.
* Kademlia DHT (Distributed Hash Table) implementation for finding peers to download from and announcing to peers what we have to host ([lbrynet.dht](https://github.com/lbryio/lbry/tree/master/lbrynet/dht)).
* Blob exchange protocol for transferring encrypted blobs of content and negotiating payments ([lbrynet.blob_exchange](https://github.com/lbryio/lbry/tree/master/lbrynet/blob_exchange)).
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([lbrynet.schema](https://github.com/lbryio/lbry/tree/master/lbrynet/schema)).
* Wallet implementation for the LBRY blockchain ([lbrynet.extras.wallet](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/wallet)).
* Daemon with a JSON-RPC API to ease building end user applications in any language and for automating various tasks ([lbrynet.extras.daemon](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/daemon)).
* Kademlia DHT (Distributed Hash Table) implementation for finding peers to download from and announcing to peers what we have to host ([lbry.dht](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/dht)).
* Blob exchange protocol for transferring encrypted blobs of content and negotiating payments ([lbry.blob_exchange](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/blob_exchange)).
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([lbry.schema](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/schema)).
* Wallet implementation for the LBRY blockchain ([lbry.wallet](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/wallet)).
* Daemon with a JSON-RPC API to ease building end user applications in any language and for automating various tasks ([lbry.extras.daemon](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/extras/daemon)).
## Installation
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
Our [releases page](https://github.com/lbryio/lbry-sdk/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
## Usage
@@ -33,7 +33,7 @@ Installing from source is also relatively painless. Full instructions are in [IN
## Contributing
Contributions to this project are welcome, encouraged, and compensated. For more details, please check [this](https://lbry.io/faq/contributing) link.
Contributions to this project are welcome, encouraged, and compensated. For more details, please check [this](https://lbry.tech/contribute) link.
## License
@@ -41,11 +41,11 @@ This project is MIT licensed. For the full license, see [LICENSE](LICENSE).
## Security
We take security seriously. Please contact security@lbry.io regarding any security issues. [Our GPG key is here](https://lbry.io/faq/gpg-key) if you need it.
We take security seriously. Please contact security@lbry.com regarding any security issues. [Our GPG key is here](https://lbry.com/faq/gpg-key) if you need it.
## Contact
The primary contact for this project is [@eukreign](mailto:lex@lbry.io).
The primary contact for this project is [@eukreign](mailto:lex@lbry.com).
## Additional information and links
@@ -53,4 +53,4 @@ The documentation for the API can be found [here](https://lbry.tech/api/sdk).
Daemon defaults, ports, and other settings are documented [here](https://lbry.tech/resources/daemon-settings).
Settings can be configured using a daemon-settings.yml file. An example can be found [here](https://github.com/lbryio/lbry/blob/master/example_daemon_settings.yml).
Settings can be configured using a daemon-settings.yml file. An example can be found [here](https://github.com/lbryio/lbry-sdk/blob/master/lbry/example_daemon_settings.yml).
-1389
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -2,6 +2,6 @@
.tox
__pycache__
dist
lbrynet.egg-info
lbry.egg-info
docs
tests
File renamed without changes.
+13
View File
@@ -0,0 +1,13 @@
.DS_Store
/build
/dist
/.tox
/.idea
/.coverage
/lbry-venv
lbry.egg-info
__pycache__
_trial_temp/
/tests/integration/files
+1 -5
View File
@@ -24,7 +24,7 @@ persistent=yes
load-plugins=
# Use multiple processes to speed up Pylint.
jobs=1
jobs=4
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
@@ -77,8 +77,6 @@ disable=
dangerous-default-value,
duplicate-code,
fixme,
global-statement,
inherit-non-class,
invalid-name,
len-as-condition,
locally-disabled,
@@ -111,13 +109,11 @@ disable=
unnecessary-lambda,
unused-argument,
unused-variable,
wildcard-import,
wrong-import-order,
wrong-import-position,
deprecated-lambda,
simplifiable-if-statement,
unidiomatic-typecheck,
global-at-module-level,
inconsistent-return-statements,
keyword-arg-before-vararg,
assignment-from-no-return,
+56
View File
@@ -0,0 +1,56 @@
# <img src="https://raw.githubusercontent.com/lbryio/lbry-sdk/master/lbry/lbry.png" alt="LBRY" width="48" height="36" /> LBRY SDK [![Build Status](https://travis-ci.org/lbryio/lbry-sdk.svg?branch=master)](https://travis-ci.org/lbryio/lbry-sdk) [![Test Coverage](https://codecov.io/gh/lbryio/lbry-sdk/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry-sdk)
LBRY is a decentralized peer-to-peer protocol for publishing and accessing digital content. It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, rights and access rules. LBRY also provides a data network that consists of peers (seeders) uploading and downloading data from other peers, possibly in exchange for payments, as well as a distributed hash table used by peers to discover other peers.
LBRY SDK for Python is currently the most full featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components:
* Built on Python 3.7+ and `asyncio`.
* Kademlia DHT (Distributed Hash Table) implementation for finding peers to download from and announcing to peers what we have to host ([lbry.dht](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/dht)).
* Blob exchange protocol for transferring encrypted blobs of content and negotiating payments ([lbry.blob_exchange](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/blob_exchange)).
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([lbry.schema](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/schema)).
* Wallet implementation for the LBRY blockchain ([lbry.wallet](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/wallet)).
* Daemon with a JSON-RPC API to ease building end user applications in any language and for automating various tasks ([lbry.extras.daemon](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/extras/daemon)).
## Installation
Our [releases page](https://github.com/lbryio/lbry-sdk/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
## Usage
Run `lbrynet start` to launch the API server.
By default, `lbrynet` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
Our [quickstart guide](https://lbry.tech/playground) provides a simple walkthrough and examples for learning.
With the daemon running, `lbrynet commands` will show you a list of commands.
The full API is documented [here](https://lbry.tech/api/sdk).
## Running from source
Installing from source is also relatively painless. Full instructions are in [INSTALL.md](INSTALL.md)
## Contributing
Contributions to this project are welcome, encouraged, and compensated. For more details, please check [this](https://lbry.tech/contribute) link.
## License
This project is MIT licensed. For the full license, see [LICENSE](LICENSE).
## Security
We take security seriously. Please contact security@lbry.com regarding any security issues. [Our GPG key is here](https://lbry.com/faq/gpg-key) if you need it.
## Contact
The primary contact for this project is [@eukreign](mailto:lex@lbry.com).
## Additional information and links
The documentation for the API can be found [here](https://lbry.tech/api/sdk).
Daemon defaults, ports, and other settings are documented [here](https://lbry.tech/resources/daemon-settings).
Settings can be configured using a daemon-settings.yml file. An example can be found [here](https://github.com/lbryio/lbry-sdk/blob/master/lbry/example_daemon_settings.yml).
File renamed without changes.
+3142
View File
File diff suppressed because it is too large. Load diff
File renamed without changes.

Before

Width:  |  Height:  |  Size: 521 B

After

Width:  |  Height:  |  Size: 521 B

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.4 KiB

After

Width:  |  Height:  |  Size: 1.4 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.2 KiB

After

Width:  |  Height:  |  Size: 1.2 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.6 KiB

After

Width:  |  Height:  |  Size: 1.6 KiB

@@ -1409,7 +1409,7 @@ Returns:
'amount': (float) claim amount,
'effective_amount': (float) claim amount including supports,
'claim_id': (str) claim id,
'claim_sequence': (int) claim sequence number,
'claim_sequence': (int) claim sequence number (or -1 if unknown),
'decoded_claim': (bool) whether or not the claim value was decoded,
'height': (int) claim height,
'depth': (int) claim depth,
@@ -1954,7 +1954,6 @@ Returns:
'amount': (float) claim amount,
'effective_amount': (float) claim amount including supports,
'claim_id': (str) claim id,
'claim_sequence': (int) claim sequence number,
'decoded_claim': (bool) whether or not the claim value was decoded,
'height': (int) claim height,
'depth': (int) claim depth,
@@ -1979,7 +1978,6 @@ Returns:
'amount': (float) claim amount,
'effective_amount': (float) claim amount including supports,
'claim_id': (str) claim id,
'claim_sequence': (int) claim sequence number,
'decoded_claim': (bool) whether or not the claim value was decoded,
'height': (int) claim height,
'depth': (int) claim depth,
@@ -2586,4 +2584,4 @@ Returns:
</body>
</html>
</html>
+2 -4
View File
@@ -1339,7 +1339,7 @@ Returns:
'amount': (float) claim amount,
'effective_amount': (float) claim amount including supports,
'claim_id': (str) claim id,
'claim_sequence': (int) claim sequence number,
'claim_sequence': (int) claim sequence number (or -1 if unknown),
'decoded_claim': (bool) whether or not the claim value was decoded,
'height': (int) claim height,
'depth': (int) claim depth,
@@ -1786,7 +1786,6 @@ Returns:
'amount': (float) claim amount,
'effective_amount': (float) claim amount including supports,
'claim_id': (str) claim id,
'claim_sequence': (int) claim sequence number,
'decoded_claim': (bool) whether or not the claim value was decoded,
'height': (int) claim height,
'depth': (int) claim depth,
@@ -1811,7 +1810,6 @@ Returns:
'amount': (float) claim amount,
'effective_amount': (float) claim amount including supports,
'claim_id': (str) claim id,
'claim_sequence': (int) claim sequence number,
'decoded_claim': (bool) whether or not the claim value was decoded,
'height': (int) claim height,
'depth': (int) claim depth,
@@ -2312,4 +2310,4 @@ Returns:
</body>
</html>
</html>
File renamed without changes.
@@ -4,20 +4,22 @@
share_usage_data: True
lbryum_servers:
- lbryumx1.lbry.io:50001
- lbryumx2.lbry.io:50001
- lbryumx1.lbry.com:50001
- lbryumx2.lbry.com:50001
- lbryumx4.lbry.com:50001
blockchain_name: lbrycrd_main
data_dir: /home/lbry/.lbrynet
download_directory: /home/lbry/downloads
delete_blobs_on_remove: True
save_blobs: true
save_files: false
dht_node_port: 4444
peer_port: 3333
use_upnp: True
use_upnp: true
#components_to_skip:
# - peer_protocol_server
# - hash_announcer
# - blob_server
# - dht
File renamed without changes.

Before

Width:  |  Height:  |  Size: 7.4 KiB

After

Width:  |  Height:  |  Size: 7.4 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 19 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.2 KiB

After

Width:  |  Height:  |  Size: 1.2 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 2.6 KiB

After

Width:  |  Height:  |  Size: 2.6 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 6.1 KiB

After

Width:  |  Height:  |  Size: 6.1 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 97 KiB

After

Width:  |  Height:  |  Size: 97 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 1.1 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 361 KiB

After

Width:  |  Height:  |  Size: 361 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 5.3 KiB

After

Width:  |  Height:  |  Size: 5.3 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 15 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 31 KiB

View File
File renamed without changes.

Before

Width:  |  Height:  |  Size: 2.6 KiB

After

Width:  |  Height:  |  Size: 2.6 KiB

+4
View File
@@ -0,0 +1,4 @@
__name__ = "lbry"
__version__ = "0.41.0"
version = tuple(__version__.split('.'))
@@ -1,4 +1,4 @@
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbry.cryptoutils import get_lbry_hash_obj
MAX_BLOB_SIZE = 2 * 2 ** 20
+335
View File
@@ -0,0 +1,335 @@
import os
import re
import asyncio
import binascii
import logging
import typing
import contextlib
from io import BytesIO
from cryptography.hazmat.primitives.ciphers import Cipher, modes
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from cryptography.hazmat.primitives.padding import PKCS7
from lbry.cryptoutils import backend, get_lbry_hash_obj
from lbry.error import DownloadCancelledError, InvalidBlobHashError, InvalidDataError
from lbry.blob import MAX_BLOB_SIZE, blobhash_length
from lbry.blob.blob_info import BlobInfo
from lbry.blob.writer import HashBlobWriter
log = logging.getLogger(__name__)
_hexmatch = re.compile("^[a-f,0-9]+$")
def is_valid_blobhash(blobhash: str) -> bool:
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: True/False
"""
return len(blobhash) == blobhash_length and _hexmatch.match(blobhash)
def encrypt_blob_bytes(key: bytes, iv: bytes, unencrypted: bytes) -> typing.Tuple[bytes, str]:
cipher = Cipher(AES(key), modes.CBC(iv), backend=backend)
padder = PKCS7(AES.block_size).padder()
encryptor = cipher.encryptor()
encrypted = encryptor.update(padder.update(unencrypted) + padder.finalize()) + encryptor.finalize()
digest = get_lbry_hash_obj()
digest.update(encrypted)
return encrypted, digest.hexdigest()
def decrypt_blob_bytes(data: bytes, length: int, key: bytes, iv: bytes) -> bytes:
if len(data) != length:
raise ValueError("unexpected length")
cipher = Cipher(AES(key), modes.CBC(iv), backend=backend)
unpadder = PKCS7(AES.block_size).unpadder()
decryptor = cipher.decryptor()
return unpadder.update(decryptor.update(data) + decryptor.finalize()) + unpadder.finalize()
class AbstractBlob:
"""
A chunk of data (up to 2MB) available on the network which is specified by a sha384 hash
This class is non-io specific
"""
__slots__ = [
'loop',
'blob_hash',
'length',
'blob_completed_callback',
'blob_directory',
'writers',
'verified',
'writing',
'readers'
]
def __init__(self, loop: asyncio.AbstractEventLoop, blob_hash: str, length: typing.Optional[int] = None,
blob_completed_callback: typing.Optional[typing.Callable[['AbstractBlob'], asyncio.Task]] = None,
blob_directory: typing.Optional[str] = None):
self.loop = loop
self.blob_hash = blob_hash
self.length = length
self.blob_completed_callback = blob_completed_callback
self.blob_directory = blob_directory
self.writers: typing.Dict[typing.Tuple[typing.Optional[str], typing.Optional[int]], HashBlobWriter] = {}
self.verified: asyncio.Event = asyncio.Event(loop=self.loop)
self.writing: asyncio.Event = asyncio.Event(loop=self.loop)
self.readers: typing.List[typing.BinaryIO] = []
if not is_valid_blobhash(blob_hash):
raise InvalidBlobHashError(blob_hash)
def __del__(self):
if self.writers or self.readers:
log.warning("%s not closed before being garbage collected", self.blob_hash)
self.close()
@contextlib.contextmanager
def _reader_context(self) -> typing.ContextManager[typing.BinaryIO]:
raise NotImplementedError()
@contextlib.contextmanager
def reader_context(self) -> typing.ContextManager[typing.BinaryIO]:
if not self.is_readable():
raise OSError(f"{str(type(self))} not readable, {len(self.readers)} readers {len(self.writers)} writers")
with self._reader_context() as reader:
try:
self.readers.append(reader)
yield reader
finally:
if reader in self.readers:
self.readers.remove(reader)
def _write_blob(self, blob_bytes: bytes):
raise NotImplementedError()
def set_length(self, length) -> None:
if self.length is not None and length == self.length:
return
if self.length is None and 0 <= length <= MAX_BLOB_SIZE:
self.length = length
return
log.warning("Got an invalid length. Previous length: %s, Invalid length: %s", self.length, length)
def get_length(self) -> typing.Optional[int]:
return self.length
def get_is_verified(self) -> bool:
return self.verified.is_set()
def is_readable(self) -> bool:
return self.verified.is_set()
def is_writeable(self) -> bool:
return not self.writing.is_set()
def write_blob(self, blob_bytes: bytes):
if not self.is_writeable():
raise OSError("cannot open blob for writing")
try:
self.writing.set()
self._write_blob(blob_bytes)
finally:
self.writing.clear()
def close(self):
while self.writers:
peer, writer = self.writers.popitem()
if writer and writer.finished and not writer.finished.done() and not self.loop.is_closed():
writer.finished.cancel()
while self.readers:
reader = self.readers.pop()
if reader:
reader.close()
def delete(self):
self.close()
self.verified.clear()
self.length = None
async def sendfile(self, writer: asyncio.StreamWriter) -> int:
"""
Read and send the file to the writer and return the number of bytes sent
"""
if not self.is_readable():
raise OSError('blob files cannot be read')
with self.reader_context() as handle:
return await self.loop.sendfile(writer.transport, handle, count=self.get_length())
def decrypt(self, key: bytes, iv: bytes) -> bytes:
"""
Decrypt a BlobFile to plaintext bytes
"""
with self.reader_context() as reader:
return decrypt_blob_bytes(reader.read(), self.length, key, iv)
@classmethod
async def create_from_unencrypted(
cls, loop: asyncio.AbstractEventLoop, blob_dir: typing.Optional[str], key: bytes, iv: bytes,
unencrypted: bytes, blob_num: int,
blob_completed_callback: typing.Optional[typing.Callable[['AbstractBlob'], None]] = None) -> BlobInfo:
"""
Create an encrypted BlobFile from plaintext bytes
"""
blob_bytes, blob_hash = encrypt_blob_bytes(key, iv, unencrypted)
length = len(blob_bytes)
blob = cls(loop, blob_hash, length, blob_completed_callback, blob_dir)
writer = blob.get_blob_writer()
writer.write(blob_bytes)
await blob.verified.wait()
return BlobInfo(blob_num, length, binascii.hexlify(iv).decode(), blob_hash)
def save_verified_blob(self, verified_bytes: bytes):
if self.verified.is_set():
return
if self.is_writeable():
self._write_blob(verified_bytes)
self.verified.set()
if self.blob_completed_callback:
self.blob_completed_callback(self)
def get_blob_writer(self, peer_address: typing.Optional[str] = None,
peer_port: typing.Optional[int] = None) -> HashBlobWriter:
if (peer_address, peer_port) in self.writers and not self.writers[(peer_address, peer_port)].closed():
raise OSError(f"attempted to download blob twice from {peer_address}:{peer_port}")
fut = asyncio.Future(loop=self.loop)
writer = HashBlobWriter(self.blob_hash, self.get_length, fut)
self.writers[(peer_address, peer_port)] = writer
def remove_writer(_):
if (peer_address, peer_port) in self.writers:
del self.writers[(peer_address, peer_port)]
fut.add_done_callback(remove_writer)
def writer_finished_callback(finished: asyncio.Future):
try:
err = finished.exception()
if err:
raise err
verified_bytes = finished.result()
while self.writers:
_, other = self.writers.popitem()
if other is not writer:
other.close_handle()
self.save_verified_blob(verified_bytes)
except (InvalidBlobHashError, InvalidDataError) as error:
log.warning("writer error downloading %s: %s", self.blob_hash[:8], str(error))
except (DownloadCancelledError, asyncio.CancelledError, asyncio.TimeoutError):
pass
fut.add_done_callback(writer_finished_callback)
return writer
class BlobBuffer(AbstractBlob):
"""
An in-memory only blob
"""
def __init__(self, loop: asyncio.AbstractEventLoop, blob_hash: str, length: typing.Optional[int] = None,
blob_completed_callback: typing.Optional[typing.Callable[['AbstractBlob'], asyncio.Task]] = None,
blob_directory: typing.Optional[str] = None):
self._verified_bytes: typing.Optional[BytesIO] = None
super().__init__(loop, blob_hash, length, blob_completed_callback, blob_directory)
@contextlib.contextmanager
def _reader_context(self) -> typing.ContextManager[typing.BinaryIO]:
if not self.is_readable():
raise OSError("cannot open blob for reading")
try:
yield self._verified_bytes
finally:
if self._verified_bytes:
self._verified_bytes.close()
self._verified_bytes = None
self.verified.clear()
def _write_blob(self, blob_bytes: bytes):
if self._verified_bytes:
raise OSError("already have bytes for blob")
self._verified_bytes = BytesIO(blob_bytes)
def delete(self):
if self._verified_bytes:
self._verified_bytes.close()
self._verified_bytes = None
return super().delete()
def __del__(self):
super().__del__()
if self._verified_bytes:
self.delete()
class BlobFile(AbstractBlob):
"""
A blob existing on the local file system
"""
def __init__(self, loop: asyncio.AbstractEventLoop, blob_hash: str, length: typing.Optional[int] = None,
blob_completed_callback: typing.Optional[typing.Callable[['AbstractBlob'], asyncio.Task]] = None,
blob_directory: typing.Optional[str] = None):
super().__init__(loop, blob_hash, length, blob_completed_callback, blob_directory)
if not blob_directory or not os.path.isdir(blob_directory):
raise OSError(f"invalid blob directory '{blob_directory}'")
self.file_path = os.path.join(self.blob_directory, self.blob_hash)
if self.file_exists:
file_size = int(os.stat(self.file_path).st_size)
if length and length != file_size:
log.warning("expected %s to be %s bytes, file has %s", self.blob_hash, length, file_size)
self.delete()
else:
self.length = file_size
self.verified.set()
@property
def file_exists(self):
return os.path.isfile(self.file_path)
def is_writeable(self) -> bool:
return super().is_writeable() and not os.path.isfile(self.file_path)
def get_blob_writer(self, peer_address: typing.Optional[str] = None,
peer_port: typing.Optional[str] = None) -> HashBlobWriter:
if self.file_exists:
raise OSError(f"File already exists '{self.file_path}'")
return super().get_blob_writer(peer_address, peer_port)
@contextlib.contextmanager
def _reader_context(self) -> typing.ContextManager[typing.BinaryIO]:
handle = open(self.file_path, 'rb')
try:
yield handle
finally:
handle.close()
def _write_blob(self, blob_bytes: bytes):
with open(self.file_path, 'wb') as f:
f.write(blob_bytes)
def delete(self):
if os.path.isfile(self.file_path):
os.remove(self.file_path)
return super().delete()
@classmethod
async def create_from_unencrypted(
cls, loop: asyncio.AbstractEventLoop, blob_dir: typing.Optional[str], key: bytes, iv: bytes,
unencrypted: bytes, blob_num: int,
blob_completed_callback: typing.Optional[typing.Callable[['AbstractBlob'],
asyncio.Task]] = None) -> BlobInfo:
if not blob_dir or not os.path.isdir(blob_dir):
raise OSError(f"cannot create blob in directory: '{blob_dir}'")
return await super().create_from_unencrypted(
loop, blob_dir, key, iv, unencrypted, blob_num, blob_completed_callback
)
@@ -2,6 +2,13 @@ import typing
class BlobInfo:
__slots__ = [
'blob_hash',
'blob_num',
'length',
'iv',
]
def __init__(self, blob_num: int, length: int, iv: str, blob_hash: typing.Optional[str] = None):
self.blob_hash = blob_hash
self.blob_num = blob_num
+136
View File
@@ -0,0 +1,136 @@
import os
import typing
import asyncio
import logging
from lbry.utils import LRUCache
from lbry.blob.blob_file import is_valid_blobhash, BlobFile, BlobBuffer, AbstractBlob
from lbry.stream.descriptor import StreamDescriptor
from lbry.connection_manager import ConnectionManager
if typing.TYPE_CHECKING:
from lbry.conf import Config
from lbry.dht.protocol.data_store import DictDataStore
from lbry.extras.daemon.storage import SQLiteStorage
log = logging.getLogger(__name__)
class BlobManager:
def __init__(self, loop: asyncio.AbstractEventLoop, blob_dir: str, storage: 'SQLiteStorage', config: 'Config',
node_data_store: typing.Optional['DictDataStore'] = None):
"""
This class stores blobs on the hard disk
blob_dir - directory where blobs are stored
storage - SQLiteStorage object
"""
self.loop = loop
self.blob_dir = blob_dir
self.storage = storage
self._node_data_store = node_data_store
self.completed_blob_hashes: typing.Set[str] = set() if not self._node_data_store\
else self._node_data_store.completed_blobs
self.blobs: typing.Dict[str, AbstractBlob] = {}
self.config = config
self.decrypted_blob_lru_cache = None if not self.config.blob_lru_cache_size else LRUCache(
self.config.blob_lru_cache_size)
self.connection_manager = ConnectionManager(loop)
def _get_blob(self, blob_hash: str, length: typing.Optional[int] = None):
if self.config.save_blobs:
return BlobFile(
self.loop, blob_hash, length, self.blob_completed, self.blob_dir
)
else:
if is_valid_blobhash(blob_hash) and os.path.isfile(os.path.join(self.blob_dir, blob_hash)):
return BlobFile(
self.loop, blob_hash, length, self.blob_completed, self.blob_dir
)
return BlobBuffer(
self.loop, blob_hash, length, self.blob_completed, self.blob_dir
)
def get_blob(self, blob_hash, length: typing.Optional[int] = None):
if blob_hash in self.blobs:
if self.config.save_blobs and isinstance(self.blobs[blob_hash], BlobBuffer):
buffer = self.blobs.pop(blob_hash)
if blob_hash in self.completed_blob_hashes:
self.completed_blob_hashes.remove(blob_hash)
self.blobs[blob_hash] = self._get_blob(blob_hash, length)
if buffer.is_readable():
with buffer.reader_context() as reader:
self.blobs[blob_hash].write_blob(reader.read())
if length and self.blobs[blob_hash].length is None:
self.blobs[blob_hash].set_length(length)
else:
self.blobs[blob_hash] = self._get_blob(blob_hash, length)
return self.blobs[blob_hash]
def is_blob_verified(self, blob_hash: str, length: typing.Optional[int] = None) -> bool:
if not is_valid_blobhash(blob_hash):
raise ValueError(blob_hash)
if not os.path.isfile(os.path.join(self.blob_dir, blob_hash)):
return False
if blob_hash in self.blobs:
return self.blobs[blob_hash].get_is_verified()
return self._get_blob(blob_hash, length).get_is_verified()
async def setup(self) -> bool:
def get_files_in_blob_dir() -> typing.Set[str]:
if not self.blob_dir:
return set()
return {
item.name for item in os.scandir(self.blob_dir) if is_valid_blobhash(item.name)
}
in_blobfiles_dir = await self.loop.run_in_executor(None, get_files_in_blob_dir)
to_add = await self.storage.sync_missing_blobs(in_blobfiles_dir)
if to_add:
self.completed_blob_hashes.update(to_add)
if self.config.track_bandwidth:
self.connection_manager.start()
return True
def stop(self):
self.connection_manager.stop()
while self.blobs:
_, blob = self.blobs.popitem()
blob.close()
self.completed_blob_hashes.clear()
def get_stream_descriptor(self, sd_hash):
return StreamDescriptor.from_stream_descriptor_blob(self.loop, self.blob_dir, self.get_blob(sd_hash))
def blob_completed(self, blob: AbstractBlob) -> asyncio.Task:
if blob.blob_hash is None:
raise Exception("Blob hash is None")
if not blob.length:
raise Exception("Blob has a length of 0")
if isinstance(blob, BlobFile):
if blob.blob_hash not in self.completed_blob_hashes:
self.completed_blob_hashes.add(blob.blob_hash)
return self.loop.create_task(self.storage.add_blobs((blob.blob_hash, blob.length), finished=True))
else:
return self.loop.create_task(self.storage.add_blobs((blob.blob_hash, blob.length), finished=False))
def check_completed_blobs(self, blob_hashes: typing.List[str]) -> typing.List[str]:
"""Returns of the blobhashes_to_check, which are valid"""
return [blob_hash for blob_hash in blob_hashes if self.is_blob_verified(blob_hash)]
def delete_blob(self, blob_hash: str):
if not is_valid_blobhash(blob_hash):
raise Exception("invalid blob hash to delete")
if blob_hash not in self.blobs:
if self.blob_dir and os.path.isfile(os.path.join(self.blob_dir, blob_hash)):
os.remove(os.path.join(self.blob_dir, blob_hash))
else:
self.blobs.pop(blob_hash).delete()
if blob_hash in self.completed_blob_hashes:
self.completed_blob_hashes.remove(blob_hash)
async def delete_blobs(self, blob_hashes: typing.List[str], delete_from_db: typing.Optional[bool] = True):
for blob_hash in blob_hashes:
self.delete_blob(blob_hash)
if delete_from_db:
await self.storage.delete_blobs_from_db(blob_hashes)
@@ -2,8 +2,8 @@ import typing
import logging
import asyncio
from io import BytesIO
from lbrynet.error import InvalidBlobHashError, InvalidDataError
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbry.error import InvalidBlobHashError, InvalidDataError
from lbry.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
@@ -44,16 +44,15 @@ class HashBlobWriter:
self._hashsum.update(data)
self.len_so_far += len(data)
if self.len_so_far > expected_length:
self.close_handle()
self.finished.set_exception(InvalidDataError(
f'Length so far is greater than the expected length. {self.len_so_far} to {expected_length}'
))
self.close_handle()
return
self.buffer.write(data)
if self.len_so_far == expected_length:
blob_hash = self.calculate_blob_hash()
if blob_hash != self.expected_blob_hash:
self.close_handle()
self.finished.set_exception(InvalidBlobHashError(
f"blob hash is {blob_hash} vs expected {self.expected_blob_hash}"
))
@@ -62,6 +61,8 @@ class HashBlobWriter:
self.close_handle()
def close_handle(self):
if not self.finished.done():
self.finished.cancel()
if self.buffer is not None:
self.buffer.close()
self.buffer = None
File renamed without changes.
@@ -1,40 +1,55 @@
import asyncio
import time
import logging
import typing
import binascii
from lbrynet.error import InvalidBlobHashError, InvalidDataError
from lbrynet.blob_exchange.serialization import BlobResponse, BlobRequest
from typing import Optional
from lbry.error import InvalidBlobHashError, InvalidDataError
from lbry.blob_exchange.serialization import BlobResponse, BlobRequest
from lbry.utils import cache_concurrent
if typing.TYPE_CHECKING:
from lbrynet.blob.blob_file import BlobFile
from lbrynet.blob.writer import HashBlobWriter
from lbry.blob.blob_file import AbstractBlob
from lbry.blob.writer import HashBlobWriter
from lbry.connection_manager import ConnectionManager
log = logging.getLogger(__name__)
class BlobExchangeClientProtocol(asyncio.Protocol):
def __init__(self, loop: asyncio.BaseEventLoop, peer_timeout: typing.Optional[float] = 10):
def __init__(self, loop: asyncio.AbstractEventLoop, peer_timeout: typing.Optional[float] = 10,
connection_manager: typing.Optional['ConnectionManager'] = None):
self.loop = loop
self.peer_port: typing.Optional[int] = None
self.peer_address: typing.Optional[str] = None
self.transport: typing.Optional[asyncio.Transport] = None
self.peer_timeout = peer_timeout
self.transport: asyncio.Transport = None
self.writer: 'HashBlobWriter' = None
self.blob: 'BlobFile' = None
self.connection_manager = connection_manager
self.writer: typing.Optional['HashBlobWriter'] = None
self.blob: typing.Optional['AbstractBlob'] = None
self._blob_bytes_received = 0
self._response_fut: asyncio.Future = None
self._response_fut: typing.Optional[asyncio.Future] = None
self.buf = b''
# this is here to handle the race when the downloader is closed right as response_fut gets a result
self.closed = asyncio.Event(loop=self.loop)
def data_received(self, data: bytes):
log.debug("%s:%d -- got %s bytes -- %s bytes on buffer -- %s blob bytes received",
self.peer_address, self.peer_port, len(data), len(self.buf), self._blob_bytes_received)
if self.connection_manager:
if not self.peer_address:
addr_info = self.transport.get_extra_info('peername')
self.peer_address, self.peer_port = addr_info
# assert self.peer_address is not None
self.connection_manager.received_data(f"{self.peer_address}:{self.peer_port}", len(data))
if not self.transport or self.transport.is_closing():
log.warning("transport closing, but got more bytes from %s:%i\n%s", self.peer_address, self.peer_port,
binascii.hexlify(data))
if self._response_fut and not self._response_fut.done():
self._response_fut.cancel()
return
if not self._response_fut:
log.warning("Protocol received data before expected, probable race on keep alive. Closing transport.")
return self.close()
if self._blob_bytes_received and not self.writer.closed():
return self._write(data)
@@ -52,7 +67,8 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
self.blob.set_length(blob_response.length)
elif blob_response and not blob_response.error and self.blob.blob_hash != blob_response.blob_hash:
# the server started sending a blob we didn't request
log.warning("mismatch with self.blob %s", self.blob.blob_hash)
log.warning("%s started sending blob we didnt request %s instead of %s", self.peer_address,
blob_response.blob_hash, self.blob.blob_hash)
return
if response.responses:
log.debug("got response from %s:%i <- %s", self.peer_address, self.peer_port, response.to_dict())
@@ -63,8 +79,7 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
# write blob bytes if we're writing a blob and have blob bytes to write
self._write(response.blob_data)
def _write(self, data):
def _write(self, data: bytes):
if len(data) > (self.blob.get_length() - self._blob_bytes_received):
data = data[:(self.blob.get_length() - self._blob_bytes_received)]
log.warning("got more than asked from %s:%d, probable sendfile bug", self.peer_address, self.peer_port)
@@ -77,35 +92,51 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
log.error("error downloading blob from %s:%i: %s", self.peer_address, self.peer_port, err)
if self._response_fut and not self._response_fut.done():
self._response_fut.set_exception(err)
except (asyncio.TimeoutError) as err: # TODO: is this needed?
except asyncio.TimeoutError as err:
log.error("%s downloading blob from %s:%i", str(err), self.peer_address, self.peer_port)
if self._response_fut and not self._response_fut.done():
self._response_fut.set_exception(err)
async def _download_blob(self) -> typing.Tuple[int, typing.Optional[asyncio.Transport]]:
async def _download_blob(self) -> typing.Tuple[int, Optional['BlobExchangeClientProtocol']]:
"""
:return: download success (bool), keep connection (bool)
:return: download success (bool), connected protocol (BlobExchangeClientProtocol)
"""
start_time = time.perf_counter()
request = BlobRequest.make_request_for_blob_hash(self.blob.blob_hash)
blob_hash = self.blob.blob_hash
if not self.peer_address:
addr_info = self.transport.get_extra_info('peername')
self.peer_address, self.peer_port = addr_info
try:
msg = request.serialize()
log.debug("send request to %s:%i -> %s", self.peer_address, self.peer_port, msg.decode())
self.transport.write(msg)
if self.connection_manager:
self.connection_manager.sent_data(f"{self.peer_address}:{self.peer_port}", len(msg))
response: BlobResponse = await asyncio.wait_for(self._response_fut, self.peer_timeout, loop=self.loop)
availability_response = response.get_availability_response()
price_response = response.get_price_response()
blob_response = response.get_blob_response()
if self.closed.is_set():
msg = f"cancelled blob request for {blob_hash} immediately after we got a response"
log.warning(msg)
raise asyncio.CancelledError(msg)
if (not blob_response or blob_response.error) and\
(not availability_response or not availability_response.available_blobs):
log.warning("%s not in availability response from %s:%i", self.blob.blob_hash, self.peer_address,
self.peer_port)
log.warning(response.to_dict())
return self._blob_bytes_received, self.close()
elif availability_response.available_blobs and \
elif availability_response and availability_response.available_blobs and \
availability_response.available_blobs != [self.blob.blob_hash]:
log.warning("blob availability response doesn't match our request from %s:%i",
self.peer_address, self.peer_port)
return self._blob_bytes_received, self.close()
elif not availability_response:
log.warning("response from %s:%i did not include an availability response (we requested %s)",
self.peer_address, self.peer_port, blob_hash)
return self._blob_bytes_received, self.close()
if not price_response or price_response.blob_data_payment_rate != 'RATE_ACCEPTED':
log.warning("data rate rejected by %s:%i", self.peer_address, self.peer_port)
return self._blob_bytes_received, self.close()
@@ -123,9 +154,11 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
log.debug(msg)
msg = f"downloaded {self.blob.blob_hash[:8]} from {self.peer_address}:{self.peer_port}"
await asyncio.wait_for(self.writer.finished, self.peer_timeout, loop=self.loop)
log.info(msg)
log.info("%s at %fMB/s", msg,
round((float(self._blob_bytes_received) /
float(time.perf_counter() - start_time)) / 1000000.0, 2))
# await self.blob.finished_writing.wait() not necessary, but a dangerous change. TODO: is it needed?
return self._blob_bytes_received, self.transport
return self._blob_bytes_received, self
except asyncio.TimeoutError:
return self._blob_bytes_received, self.close()
except (InvalidBlobHashError, InvalidDataError):
@@ -133,6 +166,7 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
return self._blob_bytes_received, self.close()
def close(self):
self.closed.set()
if self._response_fut and not self._response_fut.done():
self._response_fut.cancel()
if self.writer and not self.writer.closed():
@@ -145,18 +179,21 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
self.transport = None
self.buf = b''
async def download_blob(self, blob: 'BlobFile') -> typing.Tuple[int, typing.Optional[asyncio.Transport]]:
if blob.get_is_verified() or blob.file_exists or blob.blob_write_lock.locked():
return 0, self.transport
async def download_blob(self, blob: 'AbstractBlob') -> typing.Tuple[int, Optional['BlobExchangeClientProtocol']]:
self.closed.clear()
blob_hash = blob.blob_hash
if blob.get_is_verified() or not blob.is_writeable():
return 0, self
try:
self.blob, self.writer, self._blob_bytes_received = blob, blob.open_for_writing(), 0
self._blob_bytes_received = 0
self.blob, self.writer = blob, blob.get_blob_writer(self.peer_address, self.peer_port)
self._response_fut = asyncio.Future(loop=self.loop)
return await self._download_blob()
except OSError as e:
log.error("race happened downloading from %s:%i", self.peer_address, self.peer_port)
except OSError:
# i'm not sure how to fix this race condition - jack
log.exception(e)
return self._blob_bytes_received, self.transport
log.warning("race happened downloading %s from %s:%s", blob_hash, self.peer_address, self.peer_port)
# return self._blob_bytes_received, self.transport
raise
except asyncio.TimeoutError:
if self._response_fut and not self._response_fut.done():
self._response_fut.cancel()
@@ -165,40 +202,54 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
except asyncio.CancelledError:
self.close()
raise
finally:
if self.writer and not self.writer.closed():
self.writer.close_handle()
self.writer = None
def connection_made(self, transport: asyncio.Transport):
addr = transport.get_extra_info('peername')
self.peer_address, self.peer_port = addr[0], addr[1]
self.transport = transport
self.peer_address, self.peer_port = self.transport.get_extra_info('peername')
if self.connection_manager:
self.connection_manager.connection_made(f"{self.peer_address}:{self.peer_port}")
log.debug("connection made to %s:%i", self.peer_address, self.peer_port)
def connection_lost(self, reason):
if self.connection_manager:
self.connection_manager.outgoing_connection_lost(f"{self.peer_address}:{self.peer_port}")
log.debug("connection lost to %s:%i (reason: %s, %s)", self.peer_address, self.peer_port, str(reason),
str(type(reason)))
self.close()
async def request_blob(loop: asyncio.BaseEventLoop, blob: 'BlobFile', address: str, tcp_port: int,
peer_connect_timeout: float, blob_download_timeout: float,
connected_transport: asyncio.Transport = None)\
-> typing.Tuple[int, typing.Optional[asyncio.Transport]]:
@cache_concurrent
async def request_blob(loop: asyncio.AbstractEventLoop, blob: Optional['AbstractBlob'], address: str,
tcp_port: int, peer_connect_timeout: float, blob_download_timeout: float,
connected_protocol: Optional['BlobExchangeClientProtocol'] = None,
connection_id: int = 0, connection_manager: Optional['ConnectionManager'] = None)\
-> typing.Tuple[int, Optional['BlobExchangeClientProtocol']]:
"""
Returns [<downloaded blob>, <keep connection>]
Returns [<amount of bytes received>, <client protocol if connected>]
"""
protocol = BlobExchangeClientProtocol(loop, blob_download_timeout)
if connected_transport and not connected_transport.is_closing():
connected_transport.set_protocol(protocol)
protocol.connection_made(connected_transport)
log.debug("reusing connection for %s:%d", address, tcp_port)
protocol = connected_protocol
if not connected_protocol or not connected_protocol.transport or connected_protocol.transport.is_closing():
connected_protocol = None
protocol = BlobExchangeClientProtocol(
loop, blob_download_timeout, connection_manager
)
else:
connected_transport = None
log.debug("reusing connection for %s:%d", address, tcp_port)
try:
if not connected_transport:
if not connected_protocol:
await asyncio.wait_for(loop.create_connection(lambda: protocol, address, tcp_port),
peer_connect_timeout, loop=loop)
if blob.get_is_verified() or blob.file_exists:
connected_protocol = protocol
if blob is None or blob.get_is_verified() or not blob.is_writeable():
# blob is None happens when we are just opening a connection
# file exists but not verified means someone is writing right now, give it time, come back later
return 0, connected_transport
return await protocol.download_blob(blob)
return 0, connected_protocol
return await connected_protocol.download_blob(blob)
except (asyncio.TimeoutError, ConnectionRefusedError, ConnectionAbortedError, OSError):
return 0, None
+137
View File
@@ -0,0 +1,137 @@
import asyncio
import typing
import logging
from lbry.utils import cache_concurrent
from lbry.blob_exchange.client import request_blob
if typing.TYPE_CHECKING:
from lbry.conf import Config
from lbry.dht.node import Node
from lbry.dht.peer import KademliaPeer
from lbry.blob.blob_manager import BlobManager
from lbry.blob.blob_file import AbstractBlob
from lbry.blob_exchange.client import BlobExchangeClientProtocol
log = logging.getLogger(__name__)
class BlobDownloader:
BAN_FACTOR = 2.0 # fixme: when connection manager gets implemented, move it out from here
def __init__(self, loop: asyncio.AbstractEventLoop, config: 'Config', blob_manager: 'BlobManager',
peer_queue: asyncio.Queue):
self.loop = loop
self.config = config
self.blob_manager = blob_manager
self.peer_queue = peer_queue
self.active_connections: typing.Dict['KademliaPeer', asyncio.Task] = {} # active request_blob calls
self.ignored: typing.Dict['KademliaPeer', int] = {}
self.scores: typing.Dict['KademliaPeer', int] = {}
self.failures: typing.Dict['KademliaPeer', int] = {}
self.connection_failures: typing.List['KademliaPeer'] = []
self.connections: typing.Dict['KademliaPeer', 'BlobExchangeClientProtocol'] = {}
self.is_running = asyncio.Event(loop=self.loop)
def should_race_continue(self, blob: 'AbstractBlob'):
max_probes = self.config.max_connections_per_download * (1 if self.connections else 10)
if len(self.active_connections) >= max_probes:
return False
return not (blob.get_is_verified() or not blob.is_writeable())
async def request_blob_from_peer(self, blob: 'AbstractBlob', peer: 'KademliaPeer', connection_id: int = 0,
just_probe: bool = False):
if blob.get_is_verified():
return
start = self.loop.time()
bytes_received, protocol = await request_blob(
self.loop, blob if not just_probe else None, peer.address, peer.tcp_port, self.config.peer_connect_timeout,
self.config.blob_download_timeout, connected_protocol=self.connections.get(peer),
connection_id=connection_id, connection_manager=self.blob_manager.connection_manager
)
if not bytes_received and not protocol and peer not in self.connection_failures:
self.connection_failures.append(peer)
if not protocol and peer not in self.ignored:
self.ignored[peer] = self.loop.time()
log.debug("drop peer %s:%i", peer.address, peer.tcp_port)
self.failures[peer] = self.failures.get(peer, 0) + 1
if peer in self.connections:
del self.connections[peer]
elif protocol:
log.debug("keep peer %s:%i", peer.address, peer.tcp_port)
self.failures[peer] = 0
self.connections[peer] = protocol
elapsed = self.loop.time() - start
self.scores[peer] = bytes_received / elapsed if bytes_received and elapsed else 1
async def new_peer_or_finished(self):
active_tasks = list(self.active_connections.values()) + [asyncio.sleep(1)]
await asyncio.wait(active_tasks, loop=self.loop, return_when='FIRST_COMPLETED')
def cleanup_active(self):
if not self.active_connections and not self.connections:
self.clearbanned()
to_remove = [peer for (peer, task) in self.active_connections.items() if task.done()]
for peer in to_remove:
del self.active_connections[peer]
def clearbanned(self):
now = self.loop.time()
self.ignored = dict((
(peer, when) for (peer, when) in self.ignored.items()
if (now - when) < min(30.0, (self.failures.get(peer, 0) ** self.BAN_FACTOR))
))
@cache_concurrent
async def download_blob(self, blob_hash: str, length: typing.Optional[int] = None,
connection_id: int = 0) -> 'AbstractBlob':
blob = self.blob_manager.get_blob(blob_hash, length)
if blob.get_is_verified():
return blob
self.is_running.set()
try:
while not blob.get_is_verified() and self.is_running.is_set():
batch: typing.Set['KademliaPeer'] = set()
while not self.peer_queue.empty():
batch.update(self.peer_queue.get_nowait())
if batch:
self.peer_queue.put_nowait(list(batch))
log.debug(
"running, %d peers, %d ignored, %d active, %s connections",
len(batch), len(self.ignored), len(self.active_connections), len(self.connections)
)
for peer in sorted(batch, key=lambda peer: self.scores.get(peer, 0), reverse=True):
if peer in self.ignored or peer in self.active_connections:
continue
if not self.should_race_continue(blob):
break
log.debug("request %s from %s:%i", blob_hash[:8], peer.address, peer.tcp_port)
just_probe = len(self.connections) == 0
t = self.loop.create_task(self.request_blob_from_peer(blob, peer, connection_id, just_probe))
self.active_connections[peer] = t
await self.new_peer_or_finished()
self.cleanup_active()
log.debug("downloaded %s", blob_hash[:8])
return blob
finally:
blob.close()
def close(self):
self.connection_failures.clear()
self.scores.clear()
self.ignored.clear()
self.is_running.clear()
for protocol in self.connections.values():
protocol.close()
async def download_blob(loop, config: 'Config', blob_manager: 'BlobManager', node: 'Node',
blob_hash: str) -> 'AbstractBlob':
search_queue = asyncio.Queue(loop=loop, maxsize=config.max_connections_per_download)
search_queue.put_nowait(blob_hash)
peer_queue, accumulate_task = node.accumulate_peers(search_queue)
downloader = BlobDownloader(loop, config, blob_manager, peer_queue)
try:
return await downloader.download_blob(blob_hash)
finally:
if accumulate_task and not accumulate_task.done():
accumulate_task.cancel()
downloader.close()
+178
View File
@@ -0,0 +1,178 @@
import asyncio
import binascii
import logging
import typing
from json.decoder import JSONDecodeError
from lbry.blob_exchange.serialization import BlobResponse, BlobRequest, blob_response_types
from lbry.blob_exchange.serialization import BlobAvailabilityResponse, BlobPriceResponse, BlobDownloadResponse, \
BlobPaymentAddressResponse
if typing.TYPE_CHECKING:
from lbry.blob.blob_manager import BlobManager
log = logging.getLogger(__name__)
# a standard request will be 295 bytes
MAX_REQUEST_SIZE = 1200
class BlobServerProtocol(asyncio.Protocol):
def __init__(self, loop: asyncio.AbstractEventLoop, blob_manager: 'BlobManager', lbrycrd_address: str,
idle_timeout: float = 30.0, transfer_timeout: float = 60.0):
self.loop = loop
self.blob_manager = blob_manager
self.idle_timeout = idle_timeout
self.transfer_timeout = transfer_timeout
self.server_task: typing.Optional[asyncio.Task] = None
self.started_listening = asyncio.Event(loop=self.loop)
self.buf = b''
self.transport: typing.Optional[asyncio.Transport] = None
self.lbrycrd_address = lbrycrd_address
self.peer_address_and_port: typing.Optional[str] = None
self.started_transfer = asyncio.Event(loop=self.loop)
self.transfer_finished = asyncio.Event(loop=self.loop)
self.close_on_idle_task: typing.Optional[asyncio.Task] = None
async def close_on_idle(self):
while self.transport:
try:
await asyncio.wait_for(self.started_transfer.wait(), self.idle_timeout, loop=self.loop)
except asyncio.TimeoutError:
log.debug("closing idle connection from %s", self.peer_address_and_port)
return self.close()
self.started_transfer.clear()
await self.transfer_finished.wait()
self.transfer_finished.clear()
def close(self):
if self.transport:
self.transport.close()
def connection_made(self, transport):
self.transport = transport
self.close_on_idle_task = self.loop.create_task(self.close_on_idle())
self.peer_address_and_port = "%s:%i" % self.transport.get_extra_info('peername')
self.blob_manager.connection_manager.connection_received(self.peer_address_and_port)
log.debug("received connection from %s", self.peer_address_and_port)
def connection_lost(self, exc: typing.Optional[Exception]) -> None:
log.debug("lost connection from %s", self.peer_address_and_port)
self.blob_manager.connection_manager.incoming_connection_lost(self.peer_address_and_port)
self.transport = None
if self.close_on_idle_task and not self.close_on_idle_task.done():
self.close_on_idle_task.cancel()
self.close_on_idle_task = None
def send_response(self, responses: typing.List[blob_response_types]):
to_send = []
while responses:
to_send.append(responses.pop())
serialized = BlobResponse(to_send).serialize()
self.transport.write(serialized)
self.blob_manager.connection_manager.sent_data(self.peer_address_and_port, len(serialized))
async def handle_request(self, request: BlobRequest):
addr = self.transport.get_extra_info('peername')
peer_address, peer_port = addr
responses = []
address_request = request.get_address_request()
if address_request:
responses.append(BlobPaymentAddressResponse(lbrycrd_address=self.lbrycrd_address))
availability_request = request.get_availability_request()
if availability_request:
responses.append(BlobAvailabilityResponse(available_blobs=list(set((
filter(lambda blob_hash: blob_hash in self.blob_manager.completed_blob_hashes,
availability_request.requested_blobs)
)))))
price_request = request.get_price_request()
if price_request:
responses.append(BlobPriceResponse(blob_data_payment_rate='RATE_ACCEPTED'))
download_request = request.get_blob_request()
if download_request:
blob = self.blob_manager.get_blob(download_request.requested_blob)
if blob.get_is_verified():
incoming_blob = {'blob_hash': blob.blob_hash, 'length': blob.length}
responses.append(BlobDownloadResponse(incoming_blob=incoming_blob))
self.send_response(responses)
bh = blob.blob_hash[:8]
log.debug("send %s to %s:%i", bh, peer_address, peer_port)
self.started_transfer.set()
try:
sent = await asyncio.wait_for(blob.sendfile(self), self.transfer_timeout, loop=self.loop)
self.blob_manager.connection_manager.sent_data(self.peer_address_and_port, sent)
log.info("sent %s (%i bytes) to %s:%i", bh, sent, peer_address, peer_port)
except (ConnectionResetError, BrokenPipeError, RuntimeError, OSError, asyncio.TimeoutError) as err:
if isinstance(err, asyncio.TimeoutError):
log.debug("timed out sending blob %s to %s", bh, peer_address)
else:
log.debug("stopped sending %s to %s:%i", bh, peer_address, peer_port)
self.close()
finally:
self.transfer_finished.set()
if responses:
self.send_response(responses)
def data_received(self, data):
request = None
if len(self.buf) + len(data or b'') >= MAX_REQUEST_SIZE:
log.warning("request from %s is too large", self.peer_address_and_port)
self.close()
return
if data:
self.blob_manager.connection_manager.received_data(self.peer_address_and_port, len(data))
message, separator, remainder = data.rpartition(b'}')
if not separator:
self.buf += data
return
try:
request = BlobRequest.deserialize(self.buf + data)
self.buf = remainder
except JSONDecodeError:
log.error("request from %s is not valid json (%i bytes): %s", self.peer_address_and_port,
len(self.buf + data), '' if not data else binascii.hexlify(self.buf + data).decode())
self.close()
return
if not request.requests:
log.error("failed to decode request from %s (%i bytes): %s", self.peer_address_and_port,
len(self.buf + data), '' if not data else binascii.hexlify(self.buf + data).decode())
self.close()
return
self.loop.create_task(self.handle_request(request))
class BlobServer:
def __init__(self, loop: asyncio.AbstractEventLoop, blob_manager: 'BlobManager', lbrycrd_address: str,
idle_timeout: float = 30.0, transfer_timeout: float = 60.0):
self.loop = loop
self.blob_manager = blob_manager
self.server_task: typing.Optional[asyncio.Task] = None
self.started_listening = asyncio.Event(loop=self.loop)
self.lbrycrd_address = lbrycrd_address
self.idle_timeout = idle_timeout
self.transfer_timeout = transfer_timeout
self.server_protocol_class = BlobServerProtocol
def start_server(self, port: int, interface: typing.Optional[str] = '0.0.0.0'):
if self.server_task is not None:
raise Exception("already running")
async def _start_server():
server = await self.loop.create_server(
lambda: self.server_protocol_class(self.loop, self.blob_manager, self.lbrycrd_address,
self.idle_timeout, self.transfer_timeout),
interface, port
)
self.started_listening.set()
log.info("Blob server listening on TCP %s:%i", interface, port)
async with server:
await server.serve_forever()
self.server_task = self.loop.create_task(_start_server())
def stop_server(self):
if self.server_task:
self.server_task.cancel()
self.server_task = None
log.info("Stopped blob server")
@@ -1,2 +1,3 @@
# don't touch this. CI server changes this during build/deployment
BUILD = "dev"
BUILD_COMMIT = "source installation"
+86 -43
View File
@@ -7,8 +7,9 @@ import yaml
from argparse import ArgumentParser
from contextlib import contextmanager
from appdirs import user_data_dir, user_config_dir
from lbrynet.error import InvalidCurrencyError
from lbrynet.dht import constants
from lbry.error import InvalidCurrencyError
from lbry.dht import constants
from torba.client.coinselection import STRATEGIES
log = logging.getLogger(__name__)
@@ -21,21 +22,6 @@ CURRENCIES = {
'LBC': {'type': 'crypto'},
'USD': {'type': 'fiat'},
}
SLACK_WEBHOOK = (
'nUE0pUZ6Yl9bo29epl5moTSwnl5wo20ip2IlqzywMKZiIQSFZR5'
'AHx4mY0VmF0WQZ1ESEP9kMHZlp1WzJwWOoKN3ImR1M2yUAaMyqGZ='
)
HEADERS_FILE_SHA256_CHECKSUM = (
366295, 'b0c8197153a33ccbc52fb81a279588b6015b68b7726f73f6a2b81f7e25bfe4b9'
)
if sys.platform == "win32":
PIPE_NAME = r'\\.\pipe\lbrypipe'
else:
PIPE_NAME = './lbrypipe'
IS_THE_SPECIAL_CYTHON_PR_MERGED = True
class Setting(typing.Generic[T]):
@@ -133,7 +119,7 @@ class Toggle(Setting[bool]):
)
parser.add_argument(
self.no_cli_name,
help=f"Opposite of --{self.cli_name}",
help=f"Opposite of {self.cli_name}",
dest=self.name,
action="store_false",
default=NOT_SET
@@ -204,7 +190,36 @@ class MaxKeyFee(Setting[dict]):
)
class Servers(Setting[list]):
class StringChoice(String):
def __init__(self, doc: str, valid_values: typing.List[str], default: str, *args, **kwargs):
super().__init__(doc, default, *args, **kwargs)
if not valid_values:
raise ValueError("No valid values provided")
if default not in valid_values:
raise ValueError(f"Default value must be one of: {', '.join(valid_values)}")
self.valid_values = valid_values
def validate(self, val):
super().validate(val)
if val not in self.valid_values:
raise ValueError(f"Setting '{self.name}' value must be one of: {', '.join(self.valid_values)}")
class ListSetting(Setting[list]):
def validate(self, val):
assert isinstance(val, (tuple, list)), \
f"Setting '{self.name}' must be a tuple or list."
def contribute_to_argparse(self, parser: ArgumentParser):
parser.add_argument(
self.cli_name,
help=self.doc,
action='append'
)
class Servers(ListSetting):
def validate(self, val):
assert isinstance(val, (tuple, list)), \
@@ -237,16 +252,8 @@ class Servers(Setting[list]):
return [f"{host}:{port}" for host, port in value]
return value
def contribute_to_argparse(self, parser: ArgumentParser):
parser.add_argument(
self.cli_name,
nargs="*",
help=self.doc,
default=NOT_SET
)
class Strings(Setting[list]):
class Strings(ListSetting):
def validate(self, val):
assert isinstance(val, (tuple, list)), \
@@ -281,7 +288,7 @@ class ArgumentAccess:
def load(self, args):
for setting in self.configuration.get_settings():
value = getattr(args, setting.name, NOT_SET)
if value != NOT_SET:
if value != NOT_SET and not (isinstance(setting, ListSetting) and value is None):
self.args[setting.name] = setting.deserialize(value)
def __contains__(self, item: str):
@@ -352,6 +359,9 @@ class ConfigFileAccess:
del self.data[key]
TBC = typing.TypeVar('TBC', bound='BaseConfig')
class BaseConfig:
config = Path("Path to configuration file.", metavar='FILE')
@@ -367,12 +377,11 @@ class BaseConfig:
@contextmanager
def update_config(self):
if not isinstance(self.persisted, ConfigFileAccess):
raise TypeError("Config file cannot be updated.")
self._updating_config = True
yield self
self._updating_config = False
self.persisted.save()
if isinstance(self.persisted, ConfigFileAccess):
self.persisted.save()
@property
def modify_order(self):
@@ -408,7 +417,7 @@ class BaseConfig:
}
@classmethod
def create_from_arguments(cls, args):
def create_from_arguments(cls, args) -> TBC:
conf = cls()
conf.set_arguments(args)
conf.set_environment()
@@ -492,10 +501,15 @@ class Config(CLIConfig):
# protocol timeouts
download_timeout = Float("Cumulative timeout for a stream to begin downloading before giving up", 30.0)
blob_download_timeout = Float("Timeout to download a blob from a peer", 30.0)
peer_connect_timeout = Float("Timeout to establish a TCP connection to a peer", 2.0)
peer_connect_timeout = Float("Timeout to establish a TCP connection to a peer", 3.0)
node_rpc_timeout = Float("Timeout when making a DHT request", constants.rpc_timeout)
# blob announcement and download
save_blobs = Toggle("Save encrypted blob files for hosting, otherwise download blobs to memory only.", True)
blob_lru_cache_size = Integer(
"LRU cache size for decrypted downloaded blobs used to minimize re-downloading the same blobs when "
"replying to a range request. Set to 0 to disable.", 32
)
announce_head_and_sd_only = Toggle(
"Announce only the descriptor and first (rather than all) data blob for a stream to the DHT", True,
previous_names=['announce_head_blobs_only']
@@ -505,7 +519,7 @@ class Config(CLIConfig):
previous_names=['concurrent_announcers']
)
max_connections_per_download = Integer(
"Maximum number of peers to connect to while downloading a blob", 8,
"Maximum number of peers to connect to while downloading a blob", 4,
previous_names=['max_connections_per_stream']
)
fixed_peer_delay = Float(
@@ -527,30 +541,59 @@ class Config(CLIConfig):
# servers
reflector_servers = Servers("Reflector re-hosting servers", [
('reflector.lbry.io', 5566)
('reflector.lbry.com', 5566)
])
lbryum_servers = Servers("SPV wallet servers", [
('lbryumx1.lbry.io', 50001),
('lbryumx2.lbry.io', 50001)
('spv1.lbry.com', 50001),
('spv2.lbry.com', 50001),
('spv3.lbry.com', 50001),
('spv4.lbry.com', 50001),
('spv5.lbry.com', 50001),
('spv6.lbry.com', 50001),
('spv7.lbry.com', 50001),
('spv8.lbry.com', 50001),
('spv9.lbry.com', 50001),
])
known_dht_nodes = Servers("Known nodes for bootstrapping connection to the DHT", [
('lbrynet1.lbry.io', 4444), # US EAST
('lbrynet2.lbry.io', 4444), # US WEST
('lbrynet3.lbry.io', 4444), # EU
('lbrynet4.lbry.io', 4444) # ASIA
('lbrynet1.lbry.com', 4444), # US EAST
('lbrynet2.lbry.com', 4444), # US WEST
('lbrynet3.lbry.com', 4444), # EU
('lbrynet4.lbry.com', 4444) # ASIA
])
comment_server = String("Comment server API URL", "https://comments.lbry.com/api")
# blockchain
blockchain_name = String("Blockchain name - lbrycrd_main, lbrycrd_regtest, or lbrycrd_testnet", 'lbrycrd_main')
s3_headers_depth = Integer("download headers from s3 when the local height is more than 10 chunks behind", 96 * 10)
cache_time = Integer("Time to cache resolved claims", 150) # TODO: use this
# daemon
save_files = Toggle("Save downloaded files when calling `get` by default", True)
components_to_skip = Strings("components which will be skipped during start-up of daemon", [])
share_usage_data = Toggle(
"Whether to share usage stats and diagnostic info with LBRY.", True,
previous_names=['upload_log', 'upload_log', 'share_debug_info']
)
track_bandwidth = Toggle("Track bandwidth usage", True)
# media server
streaming_server = String('Host name and port to serve streaming media over range requests',
'localhost:5280', metavar='HOST:PORT')
streaming_get = Toggle("Enable the /get endpoint for the streaming media server. "
"Disable to prevent new streams from being added.", True)
coin_selection_strategy = StringChoice(
"Strategy to use when selecting UTXOs for a transaction",
STRATEGIES, "standard")
@property
def streaming_host(self):
return self.streaming_server.split(':')[0]
@property
def streaming_port(self):
return int(self.streaming_server.split(':')[1])
def __init__(self, **kwargs):
super().__init__(**kwargs)
@@ -577,7 +620,7 @@ class Config(CLIConfig):
def get_windows_directories() -> typing.Tuple[str, str, str]:
from lbrynet.winpaths import get_path, FOLDERID, UserHandle
from lbry.winpaths import get_path, FOLDERID, UserHandle
download_dir = get_path(FOLDERID.Downloads, UserHandle.current)
+101
View File
@@ -0,0 +1,101 @@
import asyncio
import typing
import collections
import logging
log = logging.getLogger(__name__)
CONNECTED_EVENT = "connected"
DISCONNECTED_EVENT = "disconnected"
TRANSFERRED_EVENT = "transferred"
class ConnectionManager:
def __init__(self, loop: asyncio.AbstractEventLoop):
self.loop = loop
self.incoming_connected: typing.Set[str] = set()
self.incoming: typing.DefaultDict[str, int] = collections.defaultdict(int)
self.outgoing_connected: typing.Set[str] = set()
self.outgoing: typing.DefaultDict[str, int] = collections.defaultdict(int)
self._status = {}
self._total_sent = 0
self._total_received = 0
self._running = False
self._task: typing.Optional[asyncio.Task] = None
@property
def status(self):
return self._status
def sent_data(self, host_and_port: str, size: int):
if self._running:
self.outgoing[host_and_port] += size
def received_data(self, host_and_port: str, size: int):
if self._running:
self.incoming[host_and_port] += size
def connection_made(self, host_and_port: str):
if self._running:
self.outgoing_connected.add(host_and_port)
def connection_received(self, host_and_port: str):
# self.incoming_connected.add(host_and_port)
pass
def outgoing_connection_lost(self, host_and_port: str):
if self._running and host_and_port in self.outgoing_connected:
self.outgoing_connected.remove(host_and_port)
def incoming_connection_lost(self, host_and_port: str):
if self._running and host_and_port in self.incoming_connected:
self.incoming_connected.remove(host_and_port)
async def _update(self):
self._status = {
'incoming_bps': {},
'outgoing_bps': {},
'total_incoming_mbs': 0.0,
'total_outgoing_mbs': 0.0,
'total_sent': self._total_sent,
'total_received': self._total_received,
'time': self.loop.time()
}
while True:
last = self.loop.time()
await asyncio.sleep(1, loop=self.loop)
self._status['incoming_bps'].clear()
self._status['outgoing_bps'].clear()
while self.outgoing:
k, v = self.outgoing.popitem()
self._status['outgoing_bps'][k] = v
while self.incoming:
k, v = self.incoming.popitem()
self._status['incoming_bps'][k] = v
now = self.loop.time()
self._status['total_outgoing_mbs'] = int(sum(list(self._status['outgoing_bps'].values())
) / (now - last)) / 1000000.0
self._status['total_incoming_mbs'] = int(sum(list(self._status['incoming_bps'].values())
) / (now - last)) / 1000000.0
self._total_sent += sum(list(self._status['outgoing_bps'].values()))
self._total_received += sum(list(self._status['incoming_bps'].values()))
self._status['time'] = now
def stop(self):
if self._task:
self._task.cancel()
self._task = None
self.outgoing.clear()
self.outgoing_connected.clear()
self.incoming.clear()
self.incoming_connected.clear()
self._status.clear()
self._running = False
def start(self):
self.stop()
self._running = True
self._task = self.loop.create_task(self._update())
File renamed without changes.
File renamed without changes.
+58
View File
@@ -0,0 +1,58 @@
import asyncio
import typing
import logging
if typing.TYPE_CHECKING:
from lbry.dht.node import Node
from lbry.extras.daemon.storage import SQLiteStorage
log = logging.getLogger(__name__)
class BlobAnnouncer:
def __init__(self, loop: asyncio.AbstractEventLoop, node: 'Node', storage: 'SQLiteStorage'):
self.loop = loop
self.node = node
self.storage = storage
self.announce_task: asyncio.Task = None
self.announce_queue: typing.List[str] = []
async def _submit_announcement(self, blob_hash):
try:
peers = len(await self.node.announce_blob(blob_hash))
if peers > 4:
return blob_hash
else:
log.debug("failed to announce %s, could only find %d peers, retrying soon.", blob_hash[:8], peers)
except Exception as err:
if isinstance(err, asyncio.CancelledError):
raise err
log.warning("error announcing %s: %s", blob_hash[:8], str(err))
async def _announce(self, batch_size: typing.Optional[int] = 10):
while batch_size:
if not self.node.joined.is_set():
await self.node.joined.wait()
await asyncio.sleep(60, loop=self.loop)
if not self.node.protocol.routing_table.get_peers():
log.warning("No peers in DHT, announce round skipped")
continue
self.announce_queue.extend(await self.storage.get_blobs_to_announce())
log.debug("announcer task wake up, %d blobs to announce", len(self.announce_queue))
while len(self.announce_queue):
log.info("%i blobs to announce", len(self.announce_queue))
announced = await asyncio.gather(*[
self._submit_announcement(
self.announce_queue.pop()) for _ in range(batch_size) if self.announce_queue
], loop=self.loop)
announced = list(filter(None, announced))
if announced:
await self.storage.update_last_announced_blobs(announced)
log.info("announced %i blobs", len(announced))
def start(self, batch_size: typing.Optional[int] = 10):
assert not self.announce_task or self.announce_task.done(), "already running"
self.announce_task = self.loop.create_task(self._announce(batch_size))
def stop(self):
if self.announce_task and not self.announce_task.done():
self.announce_task.cancel()
@@ -18,11 +18,10 @@ data_expiration = 86400 # 24 hours
token_secret_refresh_interval = 300 # 5 minutes
maybe_ping_delay = 300 # 5 minutes
check_refresh_interval = refresh_interval / 5
max_datagram_size = 8192 # 8 KB
rpc_id_length = 20
protocol_version = 1
bottom_out_limit = 3
msg_size_limit = max_datagram_size - 26
msg_size_limit = 1400
def digest(data: bytes) -> bytes:
File renamed without changes.
+105 -116
View File
@@ -2,24 +2,21 @@ import logging
import asyncio
import typing
import binascii
import contextlib
from lbrynet.utils import resolve_host
from lbrynet.dht import constants
from lbrynet.dht.error import RemoteException
from lbrynet.dht.protocol.async_generator_junction import AsyncGeneratorJunction
from lbrynet.dht.protocol.distance import Distance
from lbrynet.dht.protocol.iterative_find import IterativeNodeFinder, IterativeValueFinder
from lbrynet.dht.protocol.protocol import KademliaProtocol
from lbrynet.dht.peer import KademliaPeer
from lbry.utils import resolve_host
from lbry.dht import constants
from lbry.dht.protocol.distance import Distance
from lbry.dht.protocol.iterative_find import IterativeNodeFinder, IterativeValueFinder
from lbry.dht.protocol.protocol import KademliaProtocol
from lbry.dht.peer import KademliaPeer
if typing.TYPE_CHECKING:
from lbrynet.dht.peer import PeerManager
from lbry.dht.peer import PeerManager
log = logging.getLogger(__name__)
class Node:
def __init__(self, loop: asyncio.BaseEventLoop, peer_manager: 'PeerManager', node_id: bytes, udp_port: int,
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager', node_id: bytes, udp_port: int,
internal_udp_port: int, peer_port: int, external_ip: str, rpc_timeout: float = constants.rpc_timeout,
split_buckets_under_index: int = constants.split_buckets_under_index):
self.loop = loop
@@ -31,7 +28,7 @@ class Node:
self._join_task: asyncio.Task = None
self._refresh_task: asyncio.Task = None
async def refresh_node(self):
async def refresh_node(self, force_once=False):
while True:
# remove peers with expired blob announcements from the datastore
self.protocol.data_store.removed_expired_peers()
@@ -58,6 +55,8 @@ class Node:
peers = await self.peer_search(node_ids.pop())
total_peers.extend(peers)
else:
if force_once:
break
fut = asyncio.Future(loop=self.loop)
self.loop.call_later(constants.refresh_interval // 4, fut.set_result, None)
await fut
@@ -67,30 +66,33 @@ class Node:
to_ping = [peer for peer in set(total_peers) if self.protocol.peer_manager.peer_is_good(peer) is not True]
if to_ping:
self.protocol.ping_queue.enqueue_maybe_ping(*to_ping, delay=0)
if force_once:
break
fut = asyncio.Future(loop=self.loop)
self.loop.call_later(constants.refresh_interval, fut.set_result, None)
await fut
async def announce_blob(self, blob_hash: str) -> typing.List[bytes]:
announced_to_node_ids = []
while not announced_to_node_ids:
hash_value = binascii.unhexlify(blob_hash.encode())
assert len(hash_value) == constants.hash_length
peers = await self.peer_search(hash_value)
hash_value = binascii.unhexlify(blob_hash.encode())
assert len(hash_value) == constants.hash_length
peers = await self.peer_search(hash_value)
if not self.protocol.external_ip:
raise Exception("Cannot determine external IP")
log.debug("Store to %i peers", len(peers))
for peer in peers:
log.debug("store to %s %s %s", peer.address, peer.udp_port, peer.tcp_port)
stored_to_tup = await asyncio.gather(
*(self.protocol.store_to_peer(hash_value, peer) for peer in peers), loop=self.loop
)
announced_to_node_ids.extend([node_id for node_id, contacted in stored_to_tup if contacted])
log.info("Stored %s to %i of %i attempted peers", binascii.hexlify(hash_value).decode()[:8],
len(announced_to_node_ids), len(peers))
return announced_to_node_ids
if not self.protocol.external_ip:
raise Exception("Cannot determine external IP")
log.debug("Store to %i peers", len(peers))
for peer in peers:
log.debug("store to %s %s %s", peer.address, peer.udp_port, peer.tcp_port)
stored_to_tup = await asyncio.gather(
*(self.protocol.store_to_peer(hash_value, peer) for peer in peers), loop=self.loop
)
stored_to = [node_id for node_id, contacted in stored_to_tup if contacted]
if stored_to:
log.debug("Stored %s to %i of %i attempted peers", binascii.hexlify(hash_value).decode()[:8],
len(stored_to), len(peers))
else:
log.debug("Failed announcing %s, stored to 0 peers", blob_hash[:8])
return stored_to
def stop(self) -> None:
if self.joined.is_set():
@@ -101,6 +103,7 @@ class Node:
self._refresh_task.cancel()
if self.protocol and self.protocol.ping_queue.running:
self.protocol.ping_queue.stop()
self.protocol.stop()
if self.listening_port is not None:
self.listening_port.close()
self._join_task = None
@@ -113,6 +116,7 @@ class Node:
lambda: self.protocol, (interface, self.internal_udp_port)
)
log.info("DHT node listening on UDP %s:%i", interface, self.internal_udp_port)
self.protocol.start()
else:
log.warning("Already bound to port %s", self.listening_port)
@@ -130,33 +134,31 @@ class Node:
if known_node_urls:
for host, port in known_node_urls:
address = await resolve_host(host, port, proto='udp')
if (address, port) not in known_node_addresses and address != self.protocol.external_ip:
if (address, port) not in known_node_addresses and\
(address, port) != (self.protocol.external_ip, self.protocol.udp_port):
known_node_addresses.append((address, port))
url_to_addr[address] = host
if known_node_addresses:
while not self.protocol.routing_table.get_peers():
success = False
# ping the seed nodes, this will set their node ids (since we don't know them ahead of time)
for address, port in known_node_addresses:
peer = self.protocol.get_rpc_peer(KademliaPeer(self.loop, address, udp_port=port))
try:
await peer.ping()
success = True
except asyncio.TimeoutError:
log.warning("seed node (%s:%i) timed out in %s", url_to_addr.get(address, address), port,
round(self.protocol.rpc_timeout, 2))
if success:
break
# now that we have the seed nodes in routing, to an iterative lookup of our own id to populate the buckets
# in the routing table with good peers who are near us
async with self.peer_search_junction(self.protocol.node_id, max_results=16) as junction:
async for peers in junction:
for peer in peers:
try:
await self.protocol.get_rpc_peer(peer).ping()
except (asyncio.TimeoutError, RemoteException):
pass
peers = [
KademliaPeer(self.loop, address, udp_port=port)
for (address, port) in known_node_addresses
]
while True:
if not self.protocol.routing_table.get_peers():
if self.joined.is_set():
self.joined.clear()
self.protocol.peer_manager.reset()
self.protocol.ping_queue.enqueue_maybe_ping(*peers, delay=0.0)
peers.extend(await self.peer_search(self.protocol.node_id, shortlist=peers, count=32))
if self.protocol.routing_table.get_peers():
self.joined.set()
log.info(
"Joined DHT, %i peers known in %i buckets", len(self.protocol.routing_table.get_peers()),
self.protocol.routing_table.buckets_with_contacts())
else:
continue
await asyncio.sleep(1, loop=self.loop)
log.info("Joined DHT, %i peers known in %i buckets", len(self.protocol.routing_table.get_peers()),
self.protocol.routing_table.buckets_with_contacts())
@@ -169,89 +171,76 @@ class Node:
)
)
def get_iterative_node_finder(self, key: bytes, shortlist: typing.Optional[typing.List] = None,
def get_iterative_node_finder(self, key: bytes, shortlist: typing.Optional[typing.List['KademliaPeer']] = None,
bottom_out_limit: int = constants.bottom_out_limit,
max_results: int = constants.k) -> IterativeNodeFinder:
return IterativeNodeFinder(self.loop, self.protocol.peer_manager, self.protocol.routing_table, self.protocol,
key, bottom_out_limit, max_results, None, shortlist)
def get_iterative_value_finder(self, key: bytes, shortlist: typing.Optional[typing.List] = None,
def get_iterative_value_finder(self, key: bytes, shortlist: typing.Optional[typing.List['KademliaPeer']] = None,
bottom_out_limit: int = 40,
max_results: int = -1) -> IterativeValueFinder:
return IterativeValueFinder(self.loop, self.protocol.peer_manager, self.protocol.routing_table, self.protocol,
key, bottom_out_limit, max_results, None, shortlist)
@contextlib.asynccontextmanager
async def stream_peer_search_junction(self, hash_queue: asyncio.Queue, bottom_out_limit=20,
max_results=-1) -> AsyncGeneratorJunction:
peer_generator = AsyncGeneratorJunction(self.loop)
async def _add_hashes_from_queue():
while True:
try:
blob_hash = await hash_queue.get()
except asyncio.CancelledError:
break
peer_generator.add_generator(
self.get_iterative_value_finder(
binascii.unhexlify(blob_hash.encode()), bottom_out_limit=bottom_out_limit,
max_results=max_results
)
)
add_hashes_task = self.loop.create_task(_add_hashes_from_queue())
try:
async with peer_generator as junction:
yield junction
await peer_generator.finished.wait()
except asyncio.CancelledError:
if add_hashes_task and not (add_hashes_task.done() or add_hashes_task.cancelled()):
add_hashes_task.cancel()
raise
finally:
if add_hashes_task and not (add_hashes_task.done() or add_hashes_task.cancelled()):
add_hashes_task.cancel()
def peer_search_junction(self, node_id: bytes, max_results=constants.k*2,
bottom_out_limit=20) -> AsyncGeneratorJunction:
peer_generator = AsyncGeneratorJunction(self.loop)
peer_generator.add_generator(
self.get_iterative_node_finder(
node_id, bottom_out_limit=bottom_out_limit, max_results=max_results
)
)
return peer_generator
async def peer_search(self, node_id: bytes, count=constants.k, max_results=constants.k*2,
bottom_out_limit=20) -> typing.List['KademliaPeer']:
accumulated: typing.List['KademliaPeer'] = []
async with self.peer_search_junction(node_id, max_results=max_results,
bottom_out_limit=bottom_out_limit) as junction:
async for peers in junction:
accumulated.extend(peers)
bottom_out_limit=20, shortlist: typing.Optional[typing.List['KademliaPeer']] = None
) -> typing.List['KademliaPeer']:
peers = []
async for iteration_peers in self.get_iterative_node_finder(
node_id, shortlist=shortlist, bottom_out_limit=bottom_out_limit, max_results=max_results):
peers.extend(iteration_peers)
distance = Distance(node_id)
accumulated.sort(key=lambda peer: distance(peer.node_id))
return accumulated[:count]
peers.sort(key=lambda peer: distance(peer.node_id))
return peers[:count]
async def _accumulate_search_junction(self, search_queue: asyncio.Queue,
result_queue: asyncio.Queue):
tasks = []
try:
async with self.stream_peer_search_junction(search_queue) as search_junction: # pylint: disable=E1701
async for peers in search_junction:
if peers:
result_queue.put_nowait([
peer for peer in peers
if not (
peer.address == self.protocol.external_ip
and peer.tcp_port == self.protocol.peer_port
)
])
except asyncio.CancelledError:
return
while True:
blob_hash = await search_queue.get()
tasks.append(self.loop.create_task(self._value_producer(blob_hash, result_queue)))
finally:
for task in tasks:
task.cancel()
async def _value_producer(self, blob_hash: str, result_queue: asyncio.Queue):
async def ping(_peer):
try:
await self.protocol.get_rpc_peer(_peer).ping()
result_queue.put_nowait([_peer])
except asyncio.TimeoutError:
pass
async for results in self.get_iterative_value_finder(binascii.unhexlify(blob_hash.encode())):
to_put = []
for peer in results:
if peer.address == self.protocol.external_ip and self.protocol.peer_port == peer.tcp_port:
continue
is_good = self.protocol.peer_manager.peer_is_good(peer)
if is_good:
to_put.append(peer)
elif is_good is None:
udp_port_to_try = 4444
if 3400 > peer.tcp_port > 3332:
if not peer.udp_port:
udp_port_to_try = (peer.tcp_port - 3333) + 4444
elif 4500 > peer.tcp_port > 4443:
if not peer.udp_port:
udp_port_to_try = peer.tcp_port
if not peer.udp_port:
peer.update_udp_port(udp_port_to_try)
self.loop.create_task(ping(peer))
else:
log.debug("skip bad peer %s:%i for %s", peer.address, peer.tcp_port, blob_hash)
if to_put:
result_queue.put_nowait(to_put)
def accumulate_peers(self, search_queue: asyncio.Queue,
peer_queue: typing.Optional[asyncio.Queue] = None) -> typing.Tuple[
asyncio.Queue, asyncio.Task]:
q = peer_queue or asyncio.Queue()
return q, asyncio.create_task(self._accumulate_search_junction(search_queue, q))
q = peer_queue or asyncio.Queue(loop=self.loop)
return q, self.loop.create_task(self._accumulate_search_junction(search_queue, q))
+27 -11
View File
@@ -3,8 +3,10 @@ import asyncio
import logging
import ipaddress
from binascii import hexlify
from lbrynet.dht import constants
from lbrynet.dht.serialization.datagram import make_compact_address, make_compact_ip, decode_compact_address
from functools import lru_cache
from lbry.dht import constants
from lbry.dht.serialization.datagram import make_compact_address, make_compact_ip, decode_compact_address
log = logging.getLogger(__name__)
@@ -18,7 +20,7 @@ def is_valid_ipv4(address):
class PeerManager:
def __init__(self, loop: asyncio.BaseEventLoop):
def __init__(self, loop: asyncio.AbstractEventLoop):
self._loop = loop
self._rpc_failures: typing.Dict[
typing.Tuple[str, int], typing.Tuple[typing.Optional[float], typing.Optional[float]]
@@ -29,7 +31,10 @@ class PeerManager:
self._node_id_mapping: typing.Dict[typing.Tuple[str, int], bytes] = {}
self._node_id_reverse_mapping: typing.Dict[bytes, typing.Tuple[str, int]] = {}
self._node_tokens: typing.Dict[bytes, (float, bytes)] = {}
self._kademlia_peers: typing.Dict[typing.Tuple[bytes, str, int], 'KademliaPeer']
def reset(self):
for statistic in (self._rpc_failures, self._last_replied, self._last_sent, self._last_requested):
statistic.clear()
def report_failure(self, address: str, udp_port: int):
now = self._loop.time()
@@ -56,7 +61,7 @@ class PeerManager:
self._node_tokens[node_id] = (now, token)
def get_node_token(self, node_id: bytes) -> typing.Optional[bytes]:
ts, token = self._node_tokens.get(node_id, (None, None))
ts, token = self._node_tokens.get(node_id, (0, None))
if ts and ts > self._loop.time() - constants.token_secret_refresh_interval:
return token
@@ -76,6 +81,7 @@ class PeerManager:
self._node_id_mapping[(address, udp_port)] = node_id
self._node_id_reverse_mapping[node_id] = (address, udp_port)
@lru_cache(maxsize=400)
def get_kademlia_peer(self, node_id: bytes, address: str, udp_port: int) -> 'KademliaPeer':
return KademliaPeer(self._loop, address, node_id, udp_port)
@@ -101,11 +107,12 @@ class PeerManager:
delay = self._loop.time() - constants.check_refresh_interval
if node_id not in self._node_id_reverse_mapping or (address, udp_port) not in self._node_id_mapping:
return
addr_tup = (address, udp_port)
if self._node_id_reverse_mapping[node_id] != addr_tup or self._node_id_mapping[addr_tup] != node_id:
return
# fixme: find a way to re-enable that without breaking other parts
#if node_id not in self._node_id_reverse_mapping or (address, udp_port) not in self._node_id_mapping:
# return
#addr_tup = (address, udp_port)
#if self._node_id_reverse_mapping[node_id] != addr_tup or self._node_id_mapping[addr_tup] != node_id:
# return
previous_failure, most_recent_failure = self._rpc_failures.get((address, udp_port), (None, None))
last_requested = self._last_requested.get((address, udp_port))
last_replied = self._last_replied.get((address, udp_port))
@@ -132,7 +139,16 @@ class PeerManager:
class KademliaPeer:
def __init__(self, loop: asyncio.BaseEventLoop, address: str, node_id: typing.Optional[bytes] = None,
__slots__ = [
'loop',
'_node_id',
'address',
'udp_port',
'tcp_port',
'protocol_version',
]
def __init__(self, loop: asyncio.AbstractEventLoop, address: str, node_id: typing.Optional[bytes] = None,
udp_port: typing.Optional[int] = None, tcp_port: typing.Optional[int] = None):
if node_id is not None:
if not len(node_id) == constants.hash_length:
File renamed without changes.
+70
View File
@@ -0,0 +1,70 @@
import asyncio
import typing
from lbry.dht import constants
if typing.TYPE_CHECKING:
from lbry.dht.peer import KademliaPeer, PeerManager
class DictDataStore:
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager'):
# Dictionary format:
# { <key>: [(<contact>, <age>), ...] }
self._data_store: typing.Dict[bytes, typing.List[typing.Tuple['KademliaPeer', float]]] = {}
self.loop = loop
self._peer_manager = peer_manager
self.completed_blobs: typing.Set[str] = set()
def removed_expired_peers(self):
now = self.loop.time()
keys = list(self._data_store.keys())
for key in keys:
to_remove = []
for (peer, ts) in self._data_store[key]:
if ts + constants.data_expiration < now or self._peer_manager.peer_is_good(peer) is False:
to_remove.append((peer, ts))
for item in to_remove:
self._data_store[key].remove(item)
if not self._data_store[key]:
del self._data_store[key]
def filter_bad_and_expired_peers(self, key: bytes) -> typing.Iterator['KademliaPeer']:
"""
Returns only non-expired and unknown/good peers
"""
for peer in self.filter_expired_peers(key):
if self._peer_manager.peer_is_good(peer) is not False:
yield peer
def filter_expired_peers(self, key: bytes) -> typing.Iterator['KademliaPeer']:
"""
Returns only non-expired peers
"""
now = self.loop.time()
for (peer, ts) in self._data_store.get(key, []):
if ts + constants.data_expiration > now:
yield peer
def has_peers_for_blob(self, key: bytes) -> bool:
return key in self._data_store
def add_peer_to_blob(self, contact: 'KademliaPeer', key: bytes) -> None:
now = self.loop.time()
if key in self._data_store:
current = list(filter(lambda x: x[0] == contact, self._data_store[key]))
if len(current):
self._data_store[key][self._data_store[key].index(current[0])] = contact, now
else:
self._data_store[key].append((contact, now))
else:
self._data_store[key] = [(contact, now)]
def get_peers_for_blob(self, key: bytes) -> typing.List['KademliaPeer']:
return list(self.filter_bad_and_expired_peers(key))
def get_storing_contacts(self) -> typing.List['KademliaPeer']:
peers = set()
for key, stored in self._data_store.items():
peers.update(set(map(lambda tup: tup[0], stored)))
return list(peers)
@@ -1,4 +1,4 @@
from lbrynet.dht import constants
from lbry.dht import constants
class Distance:
@@ -10,11 +10,13 @@ class Distance:
def __init__(self, key: bytes):
if len(key) != constants.hash_length:
raise ValueError("invalid key length: %i" % len(key))
raise ValueError(f"invalid key length: {len(key)}")
self.key = key
self.val_key_one = int.from_bytes(key, 'big')
def __call__(self, key_two: bytes) -> int:
if len(key_two) != constants.hash_length:
raise ValueError(f"invalid length of key to compare: {len(key_two)}")
val_key_two = int.from_bytes(key_two, 'big')
return self.val_key_one ^ val_key_two
@@ -4,15 +4,15 @@ from itertools import chain
import typing
import logging
from lbrynet.dht import constants
from lbrynet.dht.error import RemoteException
from lbrynet.dht.protocol.distance import Distance
from lbry.dht import constants
from lbry.dht.error import RemoteException, TransportNotConnected
from lbry.dht.protocol.distance import Distance
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from lbrynet.dht.protocol.routing_table import TreeRoutingTable
from lbrynet.dht.protocol.protocol import KademliaProtocol
from lbrynet.dht.peer import PeerManager, KademliaPeer
from lbry.dht.protocol.routing_table import TreeRoutingTable
from lbry.dht.protocol.protocol import KademliaProtocol
from lbry.dht.peer import PeerManager, KademliaPeer
log = logging.getLogger(__name__)
@@ -66,15 +66,11 @@ def get_shortlist(routing_table: 'TreeRoutingTable', key: bytes,
"""
if len(key) != constants.hash_length:
raise ValueError("invalid key length: %i" % len(key))
if not shortlist:
shortlist = routing_table.find_close_peers(key)
distance = Distance(key)
shortlist.sort(key=lambda peer: distance(peer.node_id), reverse=True)
return shortlist
return shortlist or routing_table.find_close_peers(key)
class IterativeFinder:
def __init__(self, loop: asyncio.BaseEventLoop, peer_manager: 'PeerManager',
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
bottom_out_limit: typing.Optional[int] = 2, max_results: typing.Optional[int] = constants.k,
exclude: typing.Optional[typing.List[typing.Tuple[str, int]]] = None,
@@ -91,12 +87,11 @@ class IterativeFinder:
self.max_results = max_results
self.exclude = exclude or []
self.shortlist: typing.List['KademliaPeer'] = get_shortlist(routing_table, key, shortlist)
self.active: typing.Set['KademliaPeer'] = set()
self.contacted: typing.Set[typing.Tuple[str, int]] = set()
self.contacted: typing.Set['KademliaPeer'] = set()
self.distance = Distance(key)
self.closest_peer: typing.Optional['KademliaPeer'] = None if not self.shortlist else self.shortlist[0]
self.closest_peer: typing.Optional['KademliaPeer'] = None
self.prev_closest_peer: typing.Optional['KademliaPeer'] = None
self.iteration_queue = asyncio.Queue(loop=self.loop)
@@ -107,6 +102,12 @@ class IterativeFinder:
self.running = False
self.tasks: typing.List[asyncio.Task] = []
self.delayed_calls: typing.List[asyncio.Handle] = []
for peer in get_shortlist(routing_table, key, shortlist):
if peer.node_id:
self._add_active(peer)
else:
# seed nodes
self._schedule_probe(peer)
async def send_probe(self, peer: 'KademliaPeer') -> FindResponse:
"""
@@ -138,32 +139,23 @@ class IterativeFinder:
def _is_closer(self, peer: 'KademliaPeer') -> bool:
return not self.closest_peer or self.distance.is_closer(peer.node_id, self.closest_peer.node_id)
def _update_closest(self):
self.shortlist.sort(key=lambda peer: self.distance(peer.node_id), reverse=True)
if self.closest_peer and self.closest_peer is not self.shortlist[-1]:
if self._is_closer(self.shortlist[-1]):
def _add_active(self, peer):
if peer not in self.active and peer.node_id and peer.node_id != self.protocol.node_id:
self.active.add(peer)
if self._is_closer(peer):
self.prev_closest_peer = self.closest_peer
self.closest_peer = self.shortlist[-1]
self.closest_peer = peer
async def _handle_probe_result(self, peer: 'KademliaPeer', response: FindResponse):
if peer not in self.shortlist:
self.shortlist.append(peer)
if peer not in self.active:
self.active.add(peer)
self._add_active(peer)
for contact_triple in response.get_close_triples():
node_id, address, udp_port = contact_triple
if (address, udp_port) not in self.contacted: # and not self.peer_manager.is_ignored(addr_tuple)
found_peer = self.peer_manager.get_kademlia_peer(node_id, address, udp_port)
if found_peer not in self.shortlist and self.peer_manager.peer_is_good(peer) is not False:
self.shortlist.append(found_peer)
self._update_closest()
self._add_active(self.peer_manager.get_kademlia_peer(node_id, address, udp_port))
self.check_result_ready(response)
async def _send_probe(self, peer: 'KademliaPeer'):
try:
response = await self.send_probe(peer)
except asyncio.CancelledError:
return
except asyncio.TimeoutError:
self.active.discard(peer)
return
@@ -171,53 +163,59 @@ class IterativeFinder:
log.warning(str(err))
self.active.discard(peer)
return
except TransportNotConnected:
return self.aclose()
except RemoteException:
return
return await self._handle_probe_result(peer, response)
async def _search_round(self):
"""
Send up to constants.alpha (5) probes to the closest peers in the shortlist
Send up to constants.alpha (5) probes to closest active peers
"""
added = 0
self.shortlist.sort(key=lambda p: self.distance(p.node_id), reverse=True)
while self.running and len(self.shortlist) and added < constants.alpha:
peer = self.shortlist.pop()
to_probe = list(self.active - self.contacted)
to_probe.sort(key=lambda peer: self.distance(self.key))
for peer in to_probe:
if added >= constants.alpha:
break
origin_address = (peer.address, peer.udp_port)
if origin_address in self.exclude or self.peer_manager.peer_is_good(peer) is False:
if origin_address in self.exclude:
continue
if peer.node_id == self.protocol.node_id:
continue
if (peer.address, peer.udp_port) == (self.protocol.external_ip, self.protocol.udp_port):
if origin_address == (self.protocol.external_ip, self.protocol.udp_port):
continue
if (peer.address, peer.udp_port) not in self.contacted:
self.contacted.add((peer.address, peer.udp_port))
t = self.loop.create_task(self._send_probe(peer))
def callback(_):
self.running_probes.difference_update({
probe for probe in self.running_probes if probe.done() or probe == t
})
if not self.running_probes and self.shortlist:
self.tasks.append(self.loop.create_task(self._search_task(0.0)))
t.add_done_callback(callback)
self.running_probes.add(t)
added += 1
self._schedule_probe(peer)
added += 1
log.debug("running %d probes", len(self.running_probes))
if not added and not self.running_probes:
log.debug("search for %s exhausted", hexlify(self.key)[:8])
self.search_exhausted()
def _schedule_probe(self, peer: 'KademliaPeer'):
self.contacted.add(peer)
t = self.loop.create_task(self._send_probe(peer))
def callback(_):
self.running_probes.difference_update({
probe for probe in self.running_probes if probe.done() or probe == t
})
if not self.running_probes:
self.tasks.append(self.loop.create_task(self._search_task(0.0)))
t.add_done_callback(callback)
self.running_probes.add(t)
async def _search_task(self, delay: typing.Optional[float] = constants.iterative_lookup_delay):
try:
if self.running:
await self._search_round()
if self.running:
self.delayed_calls.append(self.loop.call_later(delay, self._search))
except (asyncio.CancelledError, StopAsyncIteration):
except (asyncio.CancelledError, StopAsyncIteration, TransportNotConnected):
if self.running:
self.loop.call_soon(self.aclose)
@@ -256,7 +254,7 @@ class IterativeFinder:
class IterativeNodeFinder(IterativeFinder):
def __init__(self, loop: asyncio.BaseEventLoop, peer_manager: 'PeerManager',
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
bottom_out_limit: typing.Optional[int] = 2, max_results: typing.Optional[int] = constants.k,
exclude: typing.Optional[typing.List[typing.Tuple[str, int]]] = None,
@@ -266,6 +264,7 @@ class IterativeNodeFinder(IterativeFinder):
self.yielded_peers: typing.Set['KademliaPeer'] = set()
async def send_probe(self, peer: 'KademliaPeer') -> FindNodeResponse:
log.debug("probing %s:%d %s", peer.address, peer.udp_port, hexlify(peer.node_id)[:8] if peer.node_id else '')
response = await self.protocol.get_rpc_peer(peer).find_node(self.key)
return FindNodeResponse(self.key, response)
@@ -273,12 +272,16 @@ class IterativeNodeFinder(IterativeFinder):
self.put_result(self.active, finish=True)
def put_result(self, from_iter: typing.Iterable['KademliaPeer'], finish=False):
not_yet_yielded = [peer for peer in from_iter if peer not in self.yielded_peers]
not_yet_yielded = [
peer for peer in from_iter
if peer not in self.yielded_peers
and peer.node_id != self.protocol.node_id
and self.peer_manager.peer_is_good(peer) is not False
]
not_yet_yielded.sort(key=lambda peer: self.distance(peer.node_id))
to_yield = not_yet_yielded[:min(constants.k, len(not_yet_yielded))]
if to_yield:
for peer in to_yield:
self.yielded_peers.add(peer)
self.yielded_peers.update(to_yield)
self.iteration_queue.put_nowait(to_yield)
if finish:
self.iteration_queue.put_nowait(None)
@@ -288,25 +291,21 @@ class IterativeNodeFinder(IterativeFinder):
if found:
log.debug("found")
return self.put_result(self.shortlist, finish=True)
return self.put_result(self.active, finish=True)
if self.prev_closest_peer and self.closest_peer and not self._is_closer(self.prev_closest_peer):
# log.info("improving, %i %i %i %i %i", len(self.shortlist), len(self.active), len(self.contacted),
# self.bottom_out_count, self.iteration_count)
self.bottom_out_count = 0
elif self.prev_closest_peer and self.closest_peer:
self.bottom_out_count += 1
log.info("bottom out %i %i %i %i", len(self.active), len(self.contacted), len(self.shortlist),
self.bottom_out_count)
log.info("bottom out %i %i %i", len(self.active), len(self.contacted), self.bottom_out_count)
if self.bottom_out_count >= self.bottom_out_limit or self.iteration_count >= self.bottom_out_limit:
log.info("limit hit")
self.put_result(self.active, True)
elif self.max_results and len(self.active) - len(self.yielded_peers) >= self.max_results:
log.debug("max results")
self.put_result(self.active, True)
class IterativeValueFinder(IterativeFinder):
def __init__(self, loop: asyncio.BaseEventLoop, peer_manager: 'PeerManager',
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
bottom_out_limit: typing.Optional[int] = 2, max_results: typing.Optional[int] = constants.k,
exclude: typing.Optional[typing.List[typing.Tuple[str, int]]] = None,
@@ -5,18 +5,19 @@ import hashlib
import asyncio
import typing
import binascii
import random
from asyncio.protocols import DatagramProtocol
from asyncio.transports import DatagramTransport
from lbrynet.dht import constants
from lbrynet.dht.serialization.datagram import decode_datagram, ErrorDatagram, ResponseDatagram, RequestDatagram
from lbrynet.dht.serialization.datagram import RESPONSE_TYPE, ERROR_TYPE
from lbrynet.dht.error import RemoteException, TransportNotConnected
from lbrynet.dht.protocol.routing_table import TreeRoutingTable
from lbrynet.dht.protocol.data_store import DictDataStore
from lbry.dht import constants
from lbry.dht.serialization.datagram import decode_datagram, ErrorDatagram, ResponseDatagram, RequestDatagram
from lbry.dht.serialization.datagram import RESPONSE_TYPE, ERROR_TYPE, PAGE_KEY
from lbry.dht.error import RemoteException, TransportNotConnected
from lbry.dht.protocol.routing_table import TreeRoutingTable
from lbry.dht.protocol.data_store import DictDataStore
if typing.TYPE_CHECKING:
from lbrynet.dht.peer import PeerManager, KademliaPeer
from lbry.dht.peer import PeerManager, KademliaPeer
log = logging.getLogger(__name__)
@@ -28,7 +29,7 @@ old_protocol_errors = {
class KademliaRPC:
def __init__(self, protocol: 'KademliaProtocol', loop: asyncio.BaseEventLoop, peer_port: int = 3333):
def __init__(self, protocol: 'KademliaProtocol', loop: asyncio.AbstractEventLoop, peer_port: int = 3333):
self.protocol = protocol
self.loop = loop
self.peer_port = peer_port
@@ -45,19 +46,19 @@ class KademliaRPC:
def ping():
return b'pong'
def store(self, rpc_contact: 'KademliaPeer', blob_hash: bytes, token: bytes, port: int,
original_publisher_id: bytes, age: int) -> bytes:
if original_publisher_id is None:
original_publisher_id = rpc_contact.node_id
def store(self, rpc_contact: 'KademliaPeer', blob_hash: bytes, token: bytes, port: int) -> bytes:
if len(blob_hash) != constants.hash_bits // 8:
raise ValueError(f"invalid length of blob hash: {len(blob_hash)}")
if not 0 < port < 65535:
raise ValueError(f"invalid tcp port: {port}")
rpc_contact.update_tcp_port(port)
if self.loop.time() - self.protocol.started_listening_time < constants.token_secret_refresh_interval:
pass
elif not self.verify_token(token, rpc_contact.compact_ip()):
raise ValueError("Invalid token")
now = int(self.loop.time())
originally_published = now - age
if not self.verify_token(token, rpc_contact.compact_ip()):
if self.loop.time() - self.protocol.started_listening_time < constants.token_secret_refresh_interval:
pass
else:
raise ValueError("Invalid token")
self.protocol.data_store.add_peer_to_blob(
rpc_contact, blob_hash, rpc_contact.compact_address_tcp(), now, originally_published, original_publisher_id
rpc_contact, blob_hash
)
return b'OK'
@@ -67,19 +68,23 @@ class KademliaRPC:
contacts = self.protocol.routing_table.find_close_peers(key, sender_node_id=rpc_contact.node_id)
contact_triples = []
for contact in contacts:
for contact in contacts[:constants.k * 2]:
contact_triples.append((contact.node_id, contact.address, contact.udp_port))
return contact_triples
def find_value(self, rpc_contact: 'KademliaPeer', key: bytes):
def find_value(self, rpc_contact: 'KademliaPeer', key: bytes, page: int = 0):
page = page if page > 0 else 0
if len(key) != constants.hash_length:
raise ValueError("invalid blob_exchange hash length: %i" % len(key))
response = {
b'token': self.make_token(rpc_contact.compact_ip()),
b'contacts': self.find_node(rpc_contact, key)
}
if not page:
response[b'contacts'] = self.find_node(rpc_contact, key)[:constants.k]
if self.protocol.protocol_version:
response[b'protocolVersion'] = self.protocol.protocol_version
@@ -92,8 +97,14 @@ class KademliaRPC:
# if we don't have k storing peers to return and we have this hash locally, include our contact information
if len(peers) < constants.k and binascii.hexlify(key).decode() in self.protocol.data_store.completed_blobs:
peers.append(self.compact_address())
if peers:
response[key] = peers
if not peers:
response[PAGE_KEY] = 0
else:
response[PAGE_KEY] = (len(peers) // (constants.k + 1)) + 1 # how many pages of peers we have for the blob
if len(peers) > constants.k:
random.Random(self.protocol.node_id).shuffle(peers)
if page * constants.k < len(peers):
response[key] = peers[page * constants.k:page * constants.k + constants.k]
return response
def refresh_token(self): # TODO: this needs to be called periodically
@@ -121,7 +132,7 @@ class RemoteKademliaRPC:
Encapsulates RPC calls to remote Peers
"""
def __init__(self, loop: asyncio.BaseEventLoop, peer_tracker: 'PeerManager', protocol: 'KademliaProtocol',
def __init__(self, loop: asyncio.AbstractEventLoop, peer_tracker: 'PeerManager', protocol: 'KademliaProtocol',
peer: 'KademliaPeer'):
self.loop = loop
self.peer_tracker = peer_tracker
@@ -166,7 +177,7 @@ class RemoteKademliaRPC:
)
return [(node_id, address.decode(), udp_port) for node_id, address, udp_port in response.response]
async def find_value(self, key: bytes) -> typing.Union[typing.Dict]:
async def find_value(self, key: bytes, page: int = 0) -> typing.Union[typing.Dict]:
"""
:return: {
b'token': <token bytes>,
@@ -177,26 +188,28 @@ class RemoteKademliaRPC:
if len(key) != constants.hash_bits // 8:
raise ValueError(f"invalid length of find value key: {len(key)}")
response = await self.protocol.send_request(
self.peer, RequestDatagram.make_find_value(self.protocol.node_id, key)
self.peer, RequestDatagram.make_find_value(self.protocol.node_id, key, page=page)
)
self.peer_tracker.update_token(self.peer.node_id, response.response[b'token'])
return response.response
class PingQueue:
def __init__(self, loop: asyncio.BaseEventLoop, protocol: 'KademliaProtocol'):
def __init__(self, loop: asyncio.AbstractEventLoop, protocol: 'KademliaProtocol'):
self._loop = loop
self._protocol = protocol
self._pending_contacts: typing.Dict['KademliaPeer', float] = {}
self._process_task: asyncio.Task = None
self._running = False
self._running_pings: typing.Set[asyncio.Task] = set()
self._default_delay = constants.maybe_ping_delay
@property
def running(self):
return self._running
def enqueue_maybe_ping(self, *peers: 'KademliaPeer', delay: float = constants.maybe_ping_delay):
def enqueue_maybe_ping(self, *peers: 'KademliaPeer', delay: typing.Optional[float] = None):
delay = delay if delay is not None else self._default_delay
now = self._loop.time()
for peer in peers:
if peer not in self._pending_contacts or now + delay < self._pending_contacts[peer]:
@@ -207,7 +220,7 @@ class PingQueue:
try:
if self._protocol.peer_manager.peer_is_good(peer):
if peer not in self._protocol.routing_table.get_peers():
await self._protocol.add_peer(peer)
self._protocol.add_peer(peer)
return
await self._protocol.get_rpc_peer(peer).ping()
except asyncio.TimeoutError:
@@ -245,7 +258,7 @@ class PingQueue:
class KademliaProtocol(DatagramProtocol):
def __init__(self, loop: asyncio.BaseEventLoop, peer_manager: 'PeerManager', node_id: bytes, external_ip: str,
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager', node_id: bytes, external_ip: str,
udp_port: int, peer_port: int, rpc_timeout: float = constants.rpc_timeout,
split_buckets_under_index: int = constants.split_buckets_under_index):
self.peer_manager = peer_manager
@@ -268,11 +281,21 @@ class KademliaProtocol(DatagramProtocol):
self.node_rpc = KademliaRPC(self, self.loop, self.peer_port)
self.rpc_timeout = rpc_timeout
self._split_lock = asyncio.Lock(loop=self.loop)
self._to_remove: typing.Set['KademliaPeer'] = set()
self._to_add: typing.Set['KademliaPeer'] = set()
self._wakeup_routing_task = asyncio.Event(loop=self.loop)
self.maintaing_routing_task: typing.Optional[asyncio.Task] = None
@functools.lru_cache(128)
def get_rpc_peer(self, peer: 'KademliaPeer') -> RemoteKademliaRPC:
return RemoteKademliaRPC(self.loop, self.peer_manager, self, peer)
def start(self):
self.maintaing_routing_task = self.loop.create_task(self.routing_table_task())
def stop(self):
if self.maintaing_routing_task:
self.maintaing_routing_task.cancel()
if self.transport:
self.disconnect()
@@ -363,13 +386,30 @@ class KademliaProtocol(DatagramProtocol):
self.routing_table.buckets[bucket_index].remove_peer(to_replace)
return await self._add_peer(peer)
async def add_peer(self, peer: 'KademliaPeer') -> bool:
def add_peer(self, peer: 'KademliaPeer'):
if peer.node_id == self.node_id:
return False
async with self._split_lock:
return await self._add_peer(peer)
self._to_add.add(peer)
self._wakeup_routing_task.set()
async def _handle_rpc(self, sender_contact: 'KademliaPeer', message: RequestDatagram):
def remove_peer(self, peer: 'KademliaPeer'):
self._to_remove.add(peer)
self._wakeup_routing_task.set()
async def routing_table_task(self):
while True:
while self._to_remove:
async with self._split_lock:
peer = self._to_remove.pop()
self.routing_table.remove_peer(peer)
self.routing_table.join_buckets()
while self._to_add:
async with self._split_lock:
await self._add_peer(self._to_add.pop())
await asyncio.gather(self._wakeup_routing_task.wait(), asyncio.sleep(.1, loop=self.loop), loop=self.loop)
self._wakeup_routing_task.clear()
def _handle_rpc(self, sender_contact: 'KademliaPeer', message: RequestDatagram):
assert sender_contact.node_id != self.node_id, (binascii.hexlify(sender_contact.node_id)[:8].decode(),
binascii.hexlify(self.node_id)[:8].decode())
method = message.method
@@ -377,30 +417,31 @@ class KademliaProtocol(DatagramProtocol):
raise AttributeError('Invalid method: %s' % message.method.decode())
if message.args and isinstance(message.args[-1], dict) and b'protocolVersion' in message.args[-1]:
# args don't need reformatting
a, kw = tuple(message.args[:-1]), message.args[-1]
args, kw = tuple(message.args[:-1]), message.args[-1]
else:
a, kw = self._migrate_incoming_rpc_args(sender_contact, message.method, *message.args)
args, kw = self._migrate_incoming_rpc_args(sender_contact, message.method, *message.args)
log.debug("%s:%i RECV CALL %s %s:%i", self.external_ip, self.udp_port, message.method.decode(),
sender_contact.address, sender_contact.udp_port)
if method == b'ping':
result = self.node_rpc.ping()
elif method == b'store':
blob_hash, token, port, original_publisher_id, age = a
result = self.node_rpc.store(sender_contact, blob_hash, token, port, original_publisher_id, age)
elif method == b'findNode':
key, = a
result = self.node_rpc.find_node(sender_contact, key)
blob_hash, token, port, original_publisher_id, age = args[:5]
result = self.node_rpc.store(sender_contact, blob_hash, token, port)
else:
assert method == b'findValue'
key, = a
result = self.node_rpc.find_value(sender_contact, key)
key = args[0]
page = kw.get(PAGE_KEY, 0)
if method == b'findNode':
result = self.node_rpc.find_node(sender_contact, key)
else:
assert method == b'findValue'
result = self.node_rpc.find_value(sender_contact, key, page)
await self.send_response(
self.send_response(
sender_contact, ResponseDatagram(RESPONSE_TYPE, message.rpc_id, self.node_id, result),
)
async def handle_request_datagram(self, address: typing.Tuple[str, int], request_datagram: RequestDatagram):
def handle_request_datagram(self, address: typing.Tuple[str, int], request_datagram: RequestDatagram):
# This is an RPC method request
self.peer_manager.report_last_requested(address[0], address[1])
try:
@@ -408,7 +449,7 @@ class KademliaProtocol(DatagramProtocol):
except IndexError:
peer = self.peer_manager.get_kademlia_peer(request_datagram.node_id, address[0], address[1])
try:
await self._handle_rpc(peer, request_datagram)
self._handle_rpc(peer, request_datagram)
# if the contact is not known to be bad (yet) and we haven't yet queried it, send it a ping so that it
# will be added to our routing table if successful
is_good = self.peer_manager.peer_is_good(peer)
@@ -416,12 +457,12 @@ class KademliaProtocol(DatagramProtocol):
self.ping_queue.enqueue_maybe_ping(peer)
# only add a requesting contact to the routing table if it has replied to one of our requests
elif is_good is True:
await self.add_peer(peer)
self.add_peer(peer)
except ValueError as err:
log.debug("error raised handling %s request from %s:%i - %s(%s)",
request_datagram.method, peer.address, peer.udp_port, str(type(err)),
str(err))
await self.send_error(
self.send_error(
peer,
ErrorDatagram(ERROR_TYPE, request_datagram.rpc_id, self.node_id, str(type(err)).encode(),
str(err).encode())
@@ -430,13 +471,13 @@ class KademliaProtocol(DatagramProtocol):
log.warning("error raised handling %s request from %s:%i - %s(%s)",
request_datagram.method, peer.address, peer.udp_port, str(type(err)),
str(err))
await self.send_error(
self.send_error(
peer,
ErrorDatagram(ERROR_TYPE, request_datagram.rpc_id, self.node_id, str(type(err)).encode(),
str(err).encode())
)
async def handle_response_datagram(self, address: typing.Tuple[str, int], response_datagram: ResponseDatagram):
def handle_response_datagram(self, address: typing.Tuple[str, int], response_datagram: ResponseDatagram):
# Find the message that triggered this response
if response_datagram.rpc_id in self.sent_messages:
peer, df, request = self.sent_messages[response_datagram.rpc_id]
@@ -459,7 +500,7 @@ class KademliaProtocol(DatagramProtocol):
self.peer_manager.update_contact_triple(peer.node_id, address[0], address[1])
if not df.cancelled():
df.set_result(response_datagram)
await self.add_peer(peer)
self.add_peer(peer)
else:
log.warning("%s:%i replied, but after we cancelled the request attempt",
peer.address, peer.udp_port)
@@ -510,43 +551,48 @@ class KademliaProtocol(DatagramProtocol):
return
if isinstance(message, RequestDatagram):
self.loop.create_task(self.handle_request_datagram(address, message))
self.handle_request_datagram(address, message)
elif isinstance(message, ErrorDatagram):
self.handle_error_datagram(address, message)
else:
assert isinstance(message, ResponseDatagram), "sanity"
self.loop.create_task(self.handle_response_datagram(address, message))
self.handle_response_datagram(address, message)
async def send_request(self, peer: 'KademliaPeer', request: RequestDatagram) -> ResponseDatagram:
await self._send(peer, request)
self._send(peer, request)
response_fut = self.sent_messages[request.rpc_id][1]
try:
response = await asyncio.wait_for(response_fut, self.rpc_timeout)
self.peer_manager.report_last_replied(peer.address, peer.udp_port)
return response
except asyncio.CancelledError:
if not response_fut.done():
response_fut.cancel()
raise
except (asyncio.TimeoutError, RemoteException):
self.peer_manager.report_failure(peer.address, peer.udp_port)
if self.peer_manager.peer_is_good(peer) is False:
async with self._split_lock:
self.routing_table.remove_peer(peer)
self.routing_table.join_buckets()
self.remove_peer(peer)
raise
async def send_response(self, peer: 'KademliaPeer', response: ResponseDatagram):
await self._send(peer, response)
def send_response(self, peer: 'KademliaPeer', response: ResponseDatagram):
self._send(peer, response)
async def send_error(self, peer: 'KademliaPeer', error: ErrorDatagram):
await self._send(peer, error)
def send_error(self, peer: 'KademliaPeer', error: ErrorDatagram):
self._send(peer, error)
async def _send(self, peer: 'KademliaPeer', message: typing.Union[RequestDatagram, ResponseDatagram,
ErrorDatagram]):
if not self.transport:
def _send(self, peer: 'KademliaPeer', message: typing.Union[RequestDatagram, ResponseDatagram, ErrorDatagram]):
if not self.transport or self.transport.is_closing():
raise TransportNotConnected()
data = message.bencode()
if len(data) > constants.msg_size_limit:
log.exception("unexpected: %i vs %i", len(data), constants.msg_size_limit)
raise ValueError()
log.warning("cannot send datagram larger than %i bytes (packet is %i bytes)",
constants.msg_size_limit, len(data))
log.debug("Packet is too large to send: %s", binascii.hexlify(data[:3500]).decode())
raise ValueError(
f"cannot send datagram larger than {constants.msg_size_limit} bytes (packet is {len(data)} bytes)"
)
if isinstance(message, (RequestDatagram, ResponseDatagram)):
assert message.node_id == self.node_id, message
if isinstance(message, RequestDatagram):
@@ -598,19 +644,26 @@ class KademliaProtocol(DatagramProtocol):
return True
async def store_to_peer(self, hash_value: bytes, peer: 'KademliaPeer') -> typing.Tuple[bytes, bool]:
try:
async def __store():
res = await self.get_rpc_peer(peer).store(hash_value)
if res != b"OK":
raise ValueError(res)
log.debug("Stored %s to %s", binascii.hexlify(hash_value).decode()[:8], peer)
return peer.node_id, True
try:
return await __store()
except asyncio.TimeoutError:
log.debug("Timeout while storing blob_hash %s at %s", binascii.hexlify(hash_value).decode()[:8], peer)
except ValueError as err:
log.error("Unexpected response: %s" % err)
except Exception as err:
except RemoteException as err:
if 'Invalid token' in str(err):
self.peer_manager.clear_token(peer.node_id)
try:
return await __store()
except (ValueError, asyncio.TimeoutError, RemoteException):
return peer.node_id, False
else:
log.exception("Unexpected error while storing blob_hash")
return peer.node_id, False
@@ -4,10 +4,10 @@ import logging
import typing
import itertools
from lbrynet.dht import constants
from lbrynet.dht.protocol.distance import Distance
from lbry.dht import constants
from lbry.dht.protocol.distance import Distance
if typing.TYPE_CHECKING:
from lbrynet.dht.peer import KademliaPeer, PeerManager
from lbry.dht.peer import KademliaPeer, PeerManager
log = logging.getLogger(__name__)
@@ -163,7 +163,7 @@ class TreeRoutingTable:
that paper.
"""
def __init__(self, loop: asyncio.BaseEventLoop, peer_manager: 'PeerManager', parent_node_id: bytes,
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager', parent_node_id: bytes,
split_buckets_under_index: int = constants.split_buckets_under_index):
self._loop = loop
self._peer_manager = peer_manager
@@ -1,5 +1,5 @@
import typing
from lbrynet.dht.error import DecodeError
from lbry.dht.error import DecodeError
def _bencode(data: typing.Union[int, bytes, bytearray, str, list, tuple, dict]) -> bytes:
@@ -63,7 +63,7 @@ def bencode(data: typing.Dict) -> bytes:
def bdecode(data: bytes, allow_non_dict_return: typing.Optional[bool] = False) -> typing.Dict:
assert type(data) == bytes, DecodeError(f"invalid data type: {str(type(data))}")
assert isinstance(data, bytes), DecodeError(f"invalid data type: {str(type(data))}")
if len(data) == 0:
raise DecodeError('Cannot decode empty string')
@@ -1,12 +1,15 @@
import typing
from functools import reduce
from lbrynet.dht import constants
from lbrynet.dht.serialization.bencoding import bencode, bdecode
from lbry.dht import constants
from lbry.dht.serialization.bencoding import bencode, bdecode
REQUEST_TYPE = 0
RESPONSE_TYPE = 1
ERROR_TYPE = 2
# bencode representation of argument keys
PAGE_KEY = b'p'
class KademliaDatagramBase:
"""
@@ -65,47 +68,39 @@ class RequestDatagram(KademliaDatagramBase):
@classmethod
def make_ping(cls, from_node_id: bytes, rpc_id: typing.Optional[bytes] = None) -> 'RequestDatagram':
if rpc_id and len(rpc_id) != constants.rpc_id_length:
raise ValueError("invalid rpc id length")
elif not rpc_id:
rpc_id = constants.generate_id()[:constants.rpc_id_length]
if len(from_node_id) != constants.hash_bits // 8:
raise ValueError("invalid node id")
rpc_id = rpc_id or constants.generate_id()[:constants.rpc_id_length]
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'ping')
@classmethod
def make_store(cls, from_node_id: bytes, blob_hash: bytes, token: bytes, port: int,
rpc_id: typing.Optional[bytes] = None) -> 'RequestDatagram':
if rpc_id and len(rpc_id) != constants.rpc_id_length:
raise ValueError("invalid rpc id length")
if not rpc_id:
rpc_id = constants.generate_id()[:constants.rpc_id_length]
if len(from_node_id) != constants.hash_bits // 8:
raise ValueError("invalid node id")
rpc_id = rpc_id or constants.generate_id()[:constants.rpc_id_length]
if len(blob_hash) != constants.hash_bits // 8:
raise ValueError(f"invalid blob hash length: {len(blob_hash)}")
if not 0 < port < 65536:
raise ValueError(f"invalid port: {port}")
if len(token) != constants.hash_bits // 8:
raise ValueError(f"invalid token length: {len(token)}")
store_args = [blob_hash, token, port, from_node_id, 0]
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'store', store_args)
@classmethod
def make_find_node(cls, from_node_id: bytes, key: bytes,
rpc_id: typing.Optional[bytes] = None) -> 'RequestDatagram':
if rpc_id and len(rpc_id) != constants.rpc_id_length:
raise ValueError("invalid rpc id length")
if not rpc_id:
rpc_id = constants.generate_id()[:constants.rpc_id_length]
if len(from_node_id) != constants.hash_bits // 8:
raise ValueError("invalid node id")
rpc_id = rpc_id or constants.generate_id()[:constants.rpc_id_length]
if len(key) != constants.hash_bits // 8:
raise ValueError(f"invalid key length: {len(key)}")
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'findNode', [key])
@classmethod
def make_find_value(cls, from_node_id: bytes, key: bytes,
rpc_id: typing.Optional[bytes] = None) -> 'RequestDatagram':
if rpc_id and len(rpc_id) != constants.rpc_id_length:
raise ValueError("invalid rpc id length")
if not rpc_id:
rpc_id = constants.generate_id()[:constants.rpc_id_length]
if len(from_node_id) != constants.hash_bits // 8:
raise ValueError("invalid node id")
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'findValue', [key])
rpc_id: typing.Optional[bytes] = None, page: int = 0) -> 'RequestDatagram':
rpc_id = rpc_id or constants.generate_id()[:constants.rpc_id_length]
if len(key) != constants.hash_bits // 8:
raise ValueError(f"invalid key length: {len(key)}")
if page < 0:
raise ValueError(f"cannot request a negative page ({page})")
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'findValue', [key, {PAGE_KEY: page}])
class ResponseDatagram(KademliaDatagramBase):
@@ -148,8 +143,6 @@ def decode_datagram(datagram: bytes) -> typing.Union[RequestDatagram, ResponseDa
}
primitive: typing.Dict = bdecode(datagram)
if not isinstance(primitive, dict):
raise ValueError("invalid datagram type")
if primitive[0] in [REQUEST_TYPE, ERROR_TYPE, RESPONSE_TYPE]: # pylint: disable=unsubscriptable-object
datagram_type = primitive[0] # pylint: disable=unsubscriptable-object
else:
@@ -163,14 +156,19 @@ def decode_datagram(datagram: bytes) -> typing.Union[RequestDatagram, ResponseDa
)
def make_compact_ip(address: str):
return reduce(lambda buff, x: buff + bytearray([int(x)]), address.split('.'), bytearray())
def make_compact_ip(address: str) -> bytearray:
compact_ip = reduce(lambda buff, x: buff + bytearray([int(x)]), address.split('.'), bytearray())
if len(compact_ip) != 4:
raise ValueError(f"invalid IPv4 length")
return compact_ip
def make_compact_address(node_id: bytes, address: str, port: int) -> bytearray:
compact_ip = make_compact_ip(address)
if not 0 <= port <= 65536:
if not 0 < port < 65536:
raise ValueError(f'Invalid port: {port}')
if len(node_id) != constants.hash_bits // 8:
raise ValueError(f"invalid node node_id length")
return compact_ip + port.to_bytes(2, 'big') + node_id
@@ -178,4 +176,8 @@ def decode_compact_address(compact_address: bytes) -> typing.Tuple[bytes, str, i
address = "{}.{}.{}.{}".format(*compact_address[:4])
port = int.from_bytes(compact_address[4:6], 'big')
node_id = compact_address[6:]
if not 0 < port < 65536:
raise ValueError(f'Invalid port: {port}')
if len(node_id) != constants.hash_bits // 8:
raise ValueError(f"invalid node node_id length")
return node_id, address, port
+7 -2
View File
@@ -28,11 +28,16 @@ class DownloadTimeoutError(Exception):
class DownloadDataTimeout(Exception):
def __init__(self, download):
super().__init__('Failed to download data blobs for sd hash '
'{} within timeout'.format(download))
super().__init__(f'Failed to download data blobs for sd hash {download} within timeout')
self.download = download
class ResolveTimeout(Exception):
def __init__(self, uri):
super().__init__(f'Failed to resolve "{uri}" within the timeout')
self.uri = uri
class RequestCanceledError(Exception):
pass
File renamed without changes.
@@ -1,5 +1,6 @@
import os
import sys
import signal
import pathlib
import json
import asyncio
@@ -11,12 +12,12 @@ from docopt import docopt
import aiohttp
from aiohttp.web import GracefulExit
from lbrynet import __name__ as lbrynet_name, __version__ as lbrynet_version
from lbrynet.extras.daemon.loggly_handler import get_loggly_handler
from lbrynet.conf import Config, CLIConfig, PIPE_NAME, IS_THE_SPECIAL_CYTHON_PR_MERGED
from lbrynet.extras.daemon.Daemon import Daemon
from lbry import __version__ as lbrynet_version
from lbry.extras.daemon.loggly_handler import get_loggly_handler
from lbry.conf import Config, CLIConfig
from lbry.extras.daemon.Daemon import Daemon
log = logging.getLogger(lbrynet_name)
log = logging.getLogger('lbry')
log.addHandler(logging.NullHandler())
@@ -24,24 +25,17 @@ def display(data):
print(json.dumps(data, indent=2))
async def execute_command(conf, method, params):
conn = None
if sys.platform == "win32" and IS_THE_SPECIAL_CYTHON_PR_MERGED:
conn = aiohttp.NamedPipeConnector(path=PIPE_NAME)
async with aiohttp.ClientSession(connector=conn) as session:
async def execute_command(conf, method, params, callback=display):
async with aiohttp.ClientSession() as session:
try:
message = {'method': method, 'params': params}
async with session.get(conf.api_connection_url, json=message) as resp:
try:
data = await resp.json()
if 'result' in data:
display(data['result'])
return callback(data['result'])
elif 'error' in data:
if 'message' in data['error']:
display(data['error']['message'])
else:
display(data['error'])
return callback(data['error'])
except Exception as e:
log.exception('Could not process response from server:', exc_info=e)
except aiohttp.ClientConnectionError:
@@ -175,7 +169,7 @@ def add_command_parser(parent, command):
def get_argument_parser():
main = ArgumentParser(
'lbrynet', description='An interface to the LBRY Network.'
'lbrynet', description='An interface to the LBRY Network.', allow_abbrev=False,
)
main.add_argument(
'-v', '--version', dest='cli_version', action="store_true",
@@ -198,8 +192,8 @@ def get_argument_parser():
help=('Enable debug output. Optionally specify loggers for which debug output '
'should selectively be applied.')
)
start.set_defaults(command='start', start_parser=start)
Config.contribute_to_argparse(start)
start.set_defaults(command='start', start_parser=start, doc=start.format_help())
api = Daemon.get_api_definitions()
groups = {}
@@ -227,10 +221,62 @@ def ensure_directory_exists(path: str):
pathlib.Path(path).mkdir(parents=True, exist_ok=True)
def main(argv=None):
if sys.platform == 'win32' and IS_THE_SPECIAL_CYTHON_PR_MERGED:
asyncio.set_event_loop(asyncio.ProactorEventLoop())
def run_daemon(args: list, conf: Config):
default_formatter = logging.Formatter("%(asctime)s %(levelname)-8s %(name)s:%(lineno)d: %(message)s")
file_handler = logging.handlers.RotatingFileHandler(
conf.log_file_path, maxBytes=2097152, backupCount=5
)
file_handler.setFormatter(default_formatter)
log.addHandler(file_handler)
logging.getLogger('torba').addHandler(file_handler)
if not args.quiet:
handler = logging.StreamHandler()
handler.setFormatter(default_formatter)
log.addHandler(handler)
logging.getLogger('torba').addHandler(handler)
logging.getLogger('torba').setLevel(logging.INFO)
logging.getLogger('aioupnp').setLevel(logging.WARNING)
logging.getLogger('aiohttp').setLevel(logging.CRITICAL)
loop = asyncio.get_event_loop()
if args.verbose:
log.setLevel(logging.DEBUG)
loop.set_debug(True)
else:
log.setLevel(logging.INFO)
if conf.share_usage_data:
loggly_handler = get_loggly_handler()
loggly_handler.setLevel(logging.ERROR)
log.addHandler(loggly_handler)
daemon = Daemon(conf)
def __exit():
raise GracefulExit()
try:
loop.add_signal_handler(signal.SIGINT, __exit)
loop.add_signal_handler(signal.SIGTERM, __exit)
except NotImplementedError:
pass # Not implemented on Windows
try:
loop.run_until_complete(daemon.start())
loop.run_forever()
except (GracefulExit, KeyboardInterrupt, asyncio.CancelledError):
pass
finally:
loop.run_until_complete(daemon.stop())
if hasattr(loop, 'shutdown_asyncgens'):
loop.run_until_complete(loop.shutdown_asyncgens())
def main(argv=None):
argv = argv or sys.argv[1:]
parser = get_argument_parser()
args, command_args = parser.parse_known_args(argv)
@@ -240,75 +286,27 @@ def main(argv=None):
ensure_directory_exists(directory)
if args.cli_version:
print(f"{lbrynet_name} {lbrynet_version}")
return 0
print(f"lbrynet {lbrynet_version}")
elif args.command == 'start':
if args.help:
args.start_parser.print_help()
return 0
default_formatter = logging.Formatter("%(asctime)s %(levelname)-8s %(name)s:%(lineno)d: %(message)s")
file_handler = logging.handlers.RotatingFileHandler(
conf.log_file_path, maxBytes=2097152, backupCount=5
)
file_handler.setFormatter(default_formatter)
log.addHandler(file_handler)
logging.getLogger('torba').addHandler(file_handler)
if not args.quiet:
handler = logging.StreamHandler()
handler.setFormatter(default_formatter)
log.addHandler(handler)
logging.getLogger('torba').addHandler(handler)
logging.getLogger('torba').setLevel(logging.INFO)
logging.getLogger('aioupnp').setLevel(logging.WARNING)
logging.getLogger('aiohttp').setLevel(logging.CRITICAL)
loop = asyncio.get_event_loop()
if args.verbose:
log.setLevel(logging.DEBUG)
loop.set_debug(True)
else:
log.setLevel(logging.INFO)
if conf.share_usage_data:
loggly_handler = get_loggly_handler()
loggly_handler.setLevel(logging.ERROR)
log.addHandler(loggly_handler)
daemon = Daemon(conf)
try:
loop.run_until_complete(daemon.start())
loop.run_until_complete(daemon.stop_event.wait())
except (GracefulExit, KeyboardInterrupt):
pass
finally:
loop.run_until_complete(daemon.stop())
if hasattr(loop, 'shutdown_asyncgens'):
loop.run_until_complete(loop.shutdown_asyncgens())
run_daemon(args, conf)
elif args.command is not None:
doc = args.doc
api_method_name = args.api_method_name
if args.replaced_by:
print(f"{args.api_method_name} is deprecated, using {args.replaced_by['api_method_name']}.")
doc = args.replaced_by['doc']
api_method_name = args.replaced_by['api_method_name']
if args.help:
print(doc)
else:
parsed = docopt(doc, command_args)
params = set_kwargs(parsed)
loop = asyncio.get_event_loop()
loop.run_until_complete(execute_command(conf, api_method_name, params))
asyncio.get_event_loop().run_until_complete(execute_command(conf, api_method_name, params))
elif args.group is not None:
args.group_parser.print_help()
else:
parser.print_help()
@@ -1,7 +1,7 @@
import asyncio
import logging
from lbrynet.conf import Config
from lbrynet.extras.daemon.ComponentManager import ComponentManager
from lbry.conf import Config
from lbry.extras.daemon.ComponentManager import ComponentManager
log = logging.getLogger(__name__)
@@ -16,7 +16,7 @@ class ComponentType(type):
class Component(metaclass=ComponentType):
"""
lbrynet-daemon component helper
lbry-daemon component helper
Inheriting classes will be automatically registered with the ComponentManager and must implement setup and stop
methods
@@ -56,7 +56,8 @@ class Component(metaclass=ComponentType):
self._running = True
return result
except asyncio.CancelledError:
pass
log.info("Cancelled setup of %s component", self.__class__.__name__)
raise
except Exception as err:
log.exception("Error setting up %s", self.component_name or self.__class__.__name__)
raise err
@@ -67,7 +68,8 @@ class Component(metaclass=ComponentType):
self._running = False
return result
except asyncio.CancelledError:
pass
log.info("Cancelled stop of %s component", self.__class__.__name__)
raise
except Exception as err:
log.exception("Error stopping %s", self.__class__.__name__)
raise err
@@ -1,8 +1,8 @@
import logging
import asyncio
from lbrynet.conf import Config
from lbrynet.error import ComponentStartConditionNotMet
from lbrynet.dht.peer import PeerManager
from lbry.conf import Config
from lbry.error import ComponentStartConditionNotMet
from lbry.dht.peer import PeerManager
log = logging.getLogger(__name__)
@@ -163,3 +163,6 @@ class ComponentManager:
if component.component_name == component_name:
return component.component
raise NameError(component_name)
def has_component(self, component_name):
return any(component for component in self.components if component_name == component.component_name)
@@ -1,32 +1,27 @@
import hashlib
import os
import asyncio
import aiohttp
import logging
import math
import binascii
import typing
from hashlib import sha256
from types import SimpleNamespace
import base58
from aioupnp import __version__ as aioupnp_version
from aioupnp.upnp import UPnP
from aioupnp.fault import UPnPError
import lbrynet.schema
from lbrynet import utils
from lbrynet.conf import HEADERS_FILE_SHA256_CHECKSUM
from lbrynet.dht.node import Node
from lbrynet.dht.blob_announcer import BlobAnnouncer
from lbrynet.blob.blob_manager import BlobFileManager
from lbrynet.blob_exchange.server import BlobServer
from lbrynet.stream.stream_manager import StreamManager
from lbrynet.extras.daemon.Component import Component
from lbrynet.extras.daemon.exchange_rate_manager import ExchangeRateManager
from lbrynet.extras.daemon.storage import SQLiteStorage
from lbrynet.extras.wallet import LbryWalletManager
from lbrynet.extras.wallet import Network
from lbry import utils
from lbry.dht.node import Node
from lbry.dht.blob_announcer import BlobAnnouncer
from lbry.blob.blob_manager import BlobManager
from lbry.blob_exchange.server import BlobServer
from lbry.stream.stream_manager import StreamManager
from lbry.extras.daemon.Component import Component
from lbry.extras.daemon.exchange_rate_manager import ExchangeRateManager
from lbry.extras.daemon.storage import SQLiteStorage
from lbry.wallet import LbryWalletManager
from lbry.wallet.header import Headers
log = logging.getLogger(__name__)
@@ -44,25 +39,6 @@ UPNP_COMPONENT = "upnp"
EXCHANGE_RATE_MANAGER_COMPONENT = "exchange_rate_manager"
async def gather_dict(tasks: dict):
async def wait_value(key, value):
return key, await value
return dict(await asyncio.gather(*(
wait_value(*kv) for kv in tasks.items()
)))
async def get_external_ip(): # used if upnp is disabled or non-functioning
try:
async with aiohttp.ClientSession() as session:
async with session.get("https://api.lbry.io/ip") as resp:
response = await resp.json()
if response['success']:
return response['data']['ip']
except Exception as e:
pass
class DatabaseComponent(Component):
component_name = DATABASE_COMPONENT
@@ -76,7 +52,7 @@ class DatabaseComponent(Component):
@staticmethod
def get_current_db_revision():
return 9
return 11
@property
def revision_filename(self):
@@ -103,7 +79,7 @@ class DatabaseComponent(Component):
'Your database is revision %i, expected %i' %
(old_revision, self.get_current_db_revision()))
if old_revision < self.get_current_db_revision():
from lbrynet.extras.daemon.migrator import dbmigrator
from lbry.extras.daemon.migrator import dbmigrator
log.info("Upgrading your databases (revision %i to %i)", old_revision, self.get_current_db_revision())
await asyncio.get_event_loop().run_in_executor(
None, dbmigrator.migrate_db, self.conf, old_revision, self.get_current_db_revision()
@@ -121,115 +97,103 @@ class DatabaseComponent(Component):
self.storage = None
HEADERS_URL = "https://headers.lbry.io/blockchain_headers_latest"
HEADER_SIZE = 112
class HeadersComponent(Component):
component_name = HEADERS_COMPONENT
HEADERS_URL = "https://headers.lbry.io/blockchain_headers_latest"
CHECKPOINT = ('100b33ca3d0b86a48f0d6d6f30458a130ecb89d5affefe4afccb134d5a40f4c2', 600_000)
def __init__(self, component_manager):
super().__init__(component_manager)
self.headers_dir = os.path.join(self.conf.wallet_dir, 'lbc_mainnet')
self.headers_file = os.path.join(self.headers_dir, 'headers')
self.old_file = os.path.join(self.conf.wallet_dir, 'blockchain_headers')
self._downloading_headers = None
self.headers = Headers(self.headers_file)
self.is_downloading_headers = False
self._headers_progress_percent = 0
@property
def component(self):
return self
async def get_status(self):
return {} if not self._downloading_headers else {
'downloading_headers': self._downloading_headers,
'download_progress': self._headers_progress_percent
}
def _round_progress(self, local_height, remote_height):
return min(max(math.ceil(float(local_height) / float(remote_height) * 100), 0), 100)
async def get_status(self) -> dict:
progress = None
if self.is_downloading_headers:
progress = self._headers_progress_percent
elif self.component_manager.has_component(WALLET_COMPONENT):
wallet_manager = self.component_manager.get_component(WALLET_COMPONENT)
if wallet_manager and wallet_manager.ledger.network.remote_height > 0:
local_height = wallet_manager.ledger.headers.height
remote_height = wallet_manager.ledger.network.remote_height
progress = self._round_progress(local_height, remote_height)
return {
'downloading_headers': True,
'download_progress': progress
} if progress is not None and progress < 100 else {}
async def fetch_headers_from_s3(self):
local_header_size = self.local_header_file_size()
resume_header = {"Range": f"bytes={local_header_size}-"}
async with aiohttp.request('get', HEADERS_URL, headers=resume_header) as response:
if response.status == 406 or response.content_length < HEADER_SIZE: # our file is bigger
async with utils.aiohttp_request('get', self.HEADERS_URL, headers=resume_header) as response:
if response.status == 406 or response.content_length < self.headers.header_size: # our file is bigger
log.warning("s3 is more out of date than we are")
return
if response.content_length % HEADER_SIZE != 0:
final_size_after_download = response.content_length + local_header_size
if final_size_after_download % self.headers.header_size != 0:
log.warning("s3 appears to have corrupted header")
return
final_size_after_download = response.content_length + local_header_size
write_mode = "wb"
if local_header_size > 0:
log.info("Resuming download of %i bytes from s3", response.content_length)
write_mode = "a+b"
with open(self.headers_file, write_mode) as fd:
while True:
chunk = await response.content.read(512)
if not chunk:
break
fd.write(chunk)
self._headers_progress_percent = math.ceil(
float(fd.tell()) / float(final_size_after_download) * 100
while not response.content.at_eof():
local_header_size += fd.write(await response.content.readany())
self._headers_progress_percent = self._round_progress(
local_header_size, final_size_after_download
)
log.info("fetched headers from s3, now verifying integrity after download.")
self._check_header_file_integrity()
def local_header_file_height(self):
return max((self.local_header_file_size() / HEADER_SIZE) - 1, 0)
def local_header_file_size(self):
def local_header_file_size(self) -> int:
if os.path.isfile(self.headers_file):
return os.stat(self.headers_file).st_size
return 0
async def get_remote_height(self):
ledger = SimpleNamespace()
ledger.config = {
'default_servers': self.conf.lbryum_servers,
'data_path': self.conf.wallet_dir
}
net = Network(ledger)
first_connection = net.on_connected.first
asyncio.ensure_future(net.start())
await first_connection
remote_height = await net.get_server_height()
await net.stop()
return remote_height
async def get_downloadable_header_height(self) -> typing.Optional[int]:
async with utils.aiohttp_request('HEAD', self.HEADERS_URL) as response:
if response.status != 200:
log.warning("Header download error, unexpected response code: %s", response.status)
return -1
return response.content_length // self.headers.header_size
async def should_download_headers_from_s3(self):
async def should_download_headers_from_s3(self) -> bool:
if self.conf.blockchain_name != "lbrycrd_main":
return False
self._check_header_file_integrity()
s3_headers_depth = self.conf.s3_headers_depth
if not s3_headers_depth:
return False
local_height = self.local_header_file_height()
remote_height = await self.get_remote_height()
log.info("remote height: %i, local height: %i", remote_height, local_height)
if remote_height > (local_height + s3_headers_depth):
return True
local_height = self.local_header_file_size() // self.headers.header_size
remote_height = await self.get_downloadable_header_height()
if remote_height is not None:
log.info("remote height: %i, local height: %i", remote_height, local_height)
if remote_height > (local_height + s3_headers_depth):
return True
return False
def _check_header_file_integrity(self):
# TODO: temporary workaround for usability. move to txlbryum and check headers instead of file integrity
if self.conf.blockchain_name != "lbrycrd_main":
return
hashsum = sha256()
checksum_height, checksum = HEADERS_FILE_SHA256_CHECKSUM
checksum_length_in_bytes = checksum_height * HEADER_SIZE
if self.local_header_file_size() < checksum_length_in_bytes:
return
with open(self.headers_file, "rb") as headers_file:
hashsum.update(headers_file.read(checksum_length_in_bytes))
current_checksum = hashsum.hexdigest()
if current_checksum != checksum:
msg = f"Expected checksum {checksum}, got {current_checksum}"
log.warning("Wallet file corrupted, checksum mismatch. " + msg)
log.warning("Deleting header file so it can be downloaded again.")
os.unlink(self.headers_file)
elif (self.local_header_file_size() % HEADER_SIZE) != 0:
log.warning("Header file is good up to checkpoint height, but incomplete. Truncating to checkpoint.")
with open(self.headers_file, "rb+") as headers_file:
headers_file.truncate(checksum_length_in_bytes)
def verify_checkpoint(self):
expected_hash, at_height = self.CHECKPOINT
if self.local_header_file_size() // self.headers.header_size < at_height:
return False
hash = hashlib.sha256()
chunk_size = self.headers.header_size * 1000
with open(self.headers_file, 'rb') as header_file:
data = header_file.read(chunk_size)
while data and header_file.tell() <= at_height * self.headers.header_size:
hash.update(data)
data = header_file.read(chunk_size)
return hash.hexdigest() == expected_hash
async def start(self):
if not os.path.exists(self.headers_dir):
@@ -237,14 +201,23 @@ class HeadersComponent(Component):
if os.path.exists(self.old_file):
log.warning("Moving old headers from %s to %s.", self.old_file, self.headers_file)
os.rename(self.old_file, self.headers_file)
self._downloading_headers = await self.should_download_headers_from_s3()
if self._downloading_headers:
try:
try:
if await self.should_download_headers_from_s3():
self.is_downloading_headers = True
await self.fetch_headers_from_s3()
except Exception as err:
log.error("failed to fetch headers from s3: %s", err)
finally:
self._downloading_headers = False
except Exception as err:
log.error("failed to fetch headers from s3: %s", err)
finally:
self.is_downloading_headers = False
# fixme: workaround, this should happen before download but happens after because headers.connect fail
if not self.verify_checkpoint():
log.info("Checkpoint failed, verifying headers using slower method.")
await self.headers.open()
await self.headers.repair()
await self.headers.close()
else:
log.info("Header checkpoint verified.")
async def stop(self):
pass
@@ -263,24 +236,29 @@ class WalletComponent(Component):
return self.wallet_manager
async def get_status(self):
if self.wallet_manager and self.running:
local_height = self.wallet_manager.network.get_local_height()
remote_height = self.wallet_manager.network.get_server_height()
if self.wallet_manager and self.wallet_manager.ledger.network.remote_height:
local_height = self.wallet_manager.ledger.headers.height
remote_height = self.wallet_manager.ledger.network.remote_height
best_hash = self.wallet_manager.get_best_blockhash()
return {
'connected_servers': [
{
'host': session.server[0],
'port': session.server[1],
'latency': round(session.connection_latency, 2),
} for session in self.wallet_manager.ledger.network.session_pool.sessions
if session and session.available
],
'blocks': max(local_height, 0),
'blocks_behind': max(remote_height - local_height, 0),
'best_blockhash': best_hash,
'is_encrypted': self.wallet_manager.wallet.use_encryption,
'is_encrypted': self.wallet_manager.use_encryption,
'is_locked': not self.wallet_manager.is_wallet_unlocked,
}
async def start(self):
log.info("Starting torba wallet")
storage = self.component_manager.get_component(DATABASE_COMPONENT)
lbrynet.schema.BLOCKCHAIN_NAME = self.conf.blockchain_name
self.wallet_manager = await LbryWalletManager.from_lbrynet_config(self.conf, storage)
self.wallet_manager.old_db = storage
self.wallet_manager = await LbryWalletManager.from_lbrynet_config(self.conf)
await self.wallet_manager.start()
async def stop(self):
@@ -294,10 +272,10 @@ class BlobComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self.blob_manager: BlobFileManager = None
self.blob_manager: typing.Optional[BlobManager] = None
@property
def component(self) -> typing.Optional[BlobFileManager]:
def component(self) -> typing.Optional[BlobManager]:
return self.blob_manager
async def start(self):
@@ -307,8 +285,10 @@ class BlobComponent(Component):
dht_node: Node = self.component_manager.get_component(DHT_COMPONENT)
if dht_node:
data_store = dht_node.protocol.data_store
self.blob_manager = BlobFileManager(asyncio.get_event_loop(), os.path.join(self.conf.data_dir, "blobfiles"),
storage, data_store)
blob_dir = os.path.join(self.conf.data_dir, 'blobfiles')
if not os.path.isdir(blob_dir):
os.mkdir(blob_dir)
self.blob_manager = BlobManager(self.component_manager.loop, blob_dir, storage, self.conf, data_store)
return await self.blob_manager.setup()
async def stop(self):
@@ -318,7 +298,10 @@ class BlobComponent(Component):
count = 0
if self.blob_manager:
count = len(self.blob_manager.completed_blob_hashes)
return {'finished_blobs': count}
return {
'finished_blobs': count,
'connections': {} if not self.blob_manager else self.blob_manager.connection_manager.status
}
class DHTComponent(Component):
@@ -327,8 +310,7 @@ class DHTComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self.dht_node: Node = None
self.upnp_component = None
self.dht_node: typing.Optional[Node] = None
self.external_udp_port = None
self.external_peer_port = None
@@ -354,18 +336,18 @@ class DHTComponent(Component):
async def start(self):
log.info("start the dht")
self.upnp_component = self.component_manager.get_component(UPNP_COMPONENT)
self.external_peer_port = self.upnp_component.upnp_redirects.get("TCP", self.conf.tcp_port)
self.external_udp_port = self.upnp_component.upnp_redirects.get("UDP", self.conf.udp_port)
external_ip = self.upnp_component.external_ip
upnp_component = self.component_manager.get_component(UPNP_COMPONENT)
self.external_peer_port = upnp_component.upnp_redirects.get("TCP", self.conf.tcp_port)
self.external_udp_port = upnp_component.upnp_redirects.get("UDP", self.conf.udp_port)
external_ip = upnp_component.external_ip
if not external_ip:
log.warning("UPnP component failed to get external ip")
external_ip = await get_external_ip()
external_ip = await utils.get_external_ip()
if not external_ip:
log.warning("failed to get external ip")
self.dht_node = Node(
asyncio.get_event_loop(),
self.component_manager.loop,
self.component_manager.peer_manager,
node_id=self.get_node_id(),
internal_udp_port=self.conf.udp_port,
@@ -390,7 +372,7 @@ class HashAnnouncerComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self.hash_announcer: BlobAnnouncer = None
self.hash_announcer: typing.Optional[BlobAnnouncer] = None
@property
def component(self) -> typing.Optional[BlobAnnouncer]:
@@ -399,7 +381,7 @@ class HashAnnouncerComponent(Component):
async def start(self):
storage = self.component_manager.get_component(DATABASE_COMPONENT)
dht_node = self.component_manager.get_component(DHT_COMPONENT)
self.hash_announcer = BlobAnnouncer(asyncio.get_event_loop(), dht_node, storage)
self.hash_announcer = BlobAnnouncer(self.component_manager.loop, dht_node, storage)
self.hash_announcer.start(self.conf.concurrent_blob_announcers)
log.info("Started blob announcer")
@@ -419,7 +401,7 @@ class StreamManagerComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self.stream_manager: StreamManager = None
self.stream_manager: typing.Optional[StreamManager] = None
@property
def component(self) -> typing.Optional[StreamManager]:
@@ -429,21 +411,19 @@ class StreamManagerComponent(Component):
if not self.stream_manager:
return
return {
'managed_files': len(self.stream_manager.streams)
'managed_files': len(self.stream_manager.streams),
}
async def start(self):
blob_manager = self.component_manager.get_component(BLOB_COMPONENT)
storage = self.component_manager.get_component(DATABASE_COMPONENT)
wallet = self.component_manager.get_component(WALLET_COMPONENT)
try:
node = self.component_manager.get_component(DHT_COMPONENT)
except NameError:
node = None
node = self.component_manager.get_component(DHT_COMPONENT)\
if self.component_manager.has_component(DHT_COMPONENT) else None
log.info('Starting the file manager')
loop = asyncio.get_event_loop()
self.stream_manager = StreamManager(
loop, self.conf, blob_manager, wallet, storage, node,
loop, self.conf, blob_manager, wallet, storage, node, self.component_manager.analytics_manager
)
await self.stream_manager.start()
log.info('Done setting up file manager')
@@ -458,7 +438,7 @@ class PeerProtocolServerComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self.blob_server: BlobServer = None
self.blob_server: typing.Optional[BlobServer] = None
@property
def component(self) -> typing.Optional[BlobServer]:
@@ -467,7 +447,7 @@ class PeerProtocolServerComponent(Component):
async def start(self):
log.info("start blob server")
upnp = self.component_manager.get_component(UPNP_COMPONENT)
blob_manager: BlobFileManager = self.component_manager.get_component(BLOB_COMPONENT)
blob_manager: BlobManager = self.component_manager.get_component(BLOB_COMPONENT)
wallet: LbryWalletManager = self.component_manager.get_component(WALLET_COMPONENT)
peer_port = self.conf.tcp_port
address = await wallet.get_unused_address()
@@ -488,9 +468,9 @@ class UPnPComponent(Component):
self._int_peer_port = self.conf.tcp_port
self._int_dht_node_port = self.conf.udp_port
self.use_upnp = self.conf.use_upnp
self.upnp = None
self.upnp: typing.Optional[UPnP] = None
self.upnp_redirects = {}
self.external_ip = None
self.external_ip: typing.Optional[str] = None
self._maintain_redirects_task = None
@property
@@ -501,15 +481,17 @@ class UPnPComponent(Component):
while True:
if now:
await self._maintain_redirects()
await asyncio.sleep(360)
await asyncio.sleep(360, loop=self.component_manager.loop)
async def _maintain_redirects(self):
# setup the gateway if necessary
if not self.upnp:
try:
self.upnp = await UPnP.discover()
self.upnp = await UPnP.discover(loop=self.component_manager.loop)
log.info("found upnp gateway: %s", self.upnp.gateway.manufacturer_string)
except Exception as err:
if isinstance(err, asyncio.CancelledError):
raise
log.warning("upnp discovery failed: %s", err)
self.upnp = None
@@ -520,52 +502,59 @@ class UPnPComponent(Component):
external_ip = await self.upnp.get_external_ip()
if external_ip != "0.0.0.0" and not self.external_ip:
log.info("got external ip from UPnP: %s", external_ip)
except (asyncio.TimeoutError, UPnPError):
except (asyncio.TimeoutError, UPnPError, NotImplementedError):
pass
if external_ip == "0.0.0.0" or not external_ip:
log.warning("unable to get external ip from UPnP, checking lbry.io fallback")
external_ip = await get_external_ip()
if external_ip == "0.0.0.0" or (external_ip and external_ip.startswith("192.")):
log.warning("unable to get external ip from UPnP, checking lbry.com fallback")
external_ip = await utils.get_external_ip()
if self.external_ip and self.external_ip != external_ip:
log.info("external ip changed from %s to %s", self.external_ip, external_ip)
self.external_ip = external_ip
assert self.external_ip is not None # TODO: handle going/starting offline
if external_ip:
self.external_ip = external_ip
# assert self.external_ip is not None # TODO: handle going/starting offline
if not self.upnp_redirects and self.upnp: # setup missing redirects
try:
log.info("add UPnP port mappings")
d = {}
if PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components:
d["TCP"] = self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port")
if DHT_COMPONENT not in self.component_manager.skip_components:
d["UDP"] = self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port")
upnp_redirects = await gather_dict(d)
log.info("add UPnP port mappings")
upnp_redirects = {}
if PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components:
try:
upnp_redirects["TCP"] = await self.upnp.get_next_mapping(
self._int_peer_port, "TCP", "LBRY peer port", self._int_peer_port
)
except (UPnPError, asyncio.TimeoutError, NotImplementedError):
pass
if DHT_COMPONENT not in self.component_manager.skip_components:
try:
upnp_redirects["UDP"] = await self.upnp.get_next_mapping(
self._int_dht_node_port, "UDP", "LBRY DHT port", self._int_dht_node_port
)
except (UPnPError, asyncio.TimeoutError, NotImplementedError):
pass
if upnp_redirects:
log.info("set up redirects: %s", upnp_redirects)
self.upnp_redirects.update(upnp_redirects)
except (asyncio.TimeoutError, UPnPError):
self.upnp = None
return self._maintain_redirects()
elif self.upnp: # check existing redirects are still active
found = set()
mappings = await self.upnp.get_redirects()
for mapping in mappings:
proto = mapping['NewProtocol']
if proto in self.upnp_redirects and mapping['NewExternalPort'] == self.upnp_redirects[proto]:
if mapping['NewInternalClient'] == self.upnp.lan_address:
proto = mapping.protocol
if proto in self.upnp_redirects and mapping.external_port == self.upnp_redirects[proto]:
if mapping.lan_address == self.upnp.lan_address:
found.add(proto)
if 'UDP' not in found and DHT_COMPONENT not in self.component_manager.skip_components:
try:
udp_port = await self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port")
self.upnp_redirects['UDP'] = udp_port
log.info("refreshed upnp redirect for dht port: %i", udp_port)
except (asyncio.TimeoutError, UPnPError):
except (asyncio.TimeoutError, UPnPError, NotImplementedError):
del self.upnp_redirects['UDP']
if 'TCP' not in found and PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components:
try:
tcp_port = await self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port")
self.upnp_redirects['TCP'] = tcp_port
log.info("refreshed upnp redirect for peer port: %i", tcp_port)
except (asyncio.TimeoutError, UPnPError):
except (asyncio.TimeoutError, UPnPError, NotImplementedError):
del self.upnp_redirects['TCP']
if ('TCP' in self.upnp_redirects
and PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components) and (
@@ -576,14 +565,14 @@ class UPnPComponent(Component):
async def start(self):
log.info("detecting external ip")
if not self.use_upnp:
self.external_ip = await get_external_ip()
self.external_ip = await utils.get_external_ip()
return
success = False
await self._maintain_redirects()
if self.upnp:
if not self.upnp_redirects and not all([x in self.component_manager.skip_components for x in
(DHT_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT)]):
log.error("failed to setup upnp, debugging infomation: %s", self.upnp.zipped_debugging_info)
log.error("failed to setup upnp")
else:
success = True
if self.upnp_redirects:
@@ -594,13 +583,16 @@ class UPnPComponent(Component):
await self.component_manager.analytics_manager.send_upnp_setup_success_fail(
success, await self.get_status()
)
self._maintain_redirects_task = asyncio.create_task(self._repeatedly_maintain_redirects(now=False))
self._maintain_redirects_task = self.component_manager.loop.create_task(
self._repeatedly_maintain_redirects(now=False)
)
async def stop(self):
if self.upnp_redirects:
log.info("Removing upnp redirects: %s", self.upnp_redirects)
await asyncio.wait([
self.upnp.delete_port_mapping(port, protocol) for protocol, port in self.upnp_redirects.items()
])
], loop=self.component_manager.loop)
if self._maintain_redirects_task and not self._maintain_redirects_task.done():
self._maintain_redirects_task.cancel()
File diff suppressed because it is too large. Load diff
+1
View File
@@ -0,0 +1 @@
from lbry.extras.daemon import Components # register Component classes
+230
View File
@@ -0,0 +1,230 @@
import asyncio
import collections
import logging
import aiohttp
import typing
from lbry import utils
from lbry.conf import Config
from lbry.extras import system_info
ANALYTICS_ENDPOINT = 'https://api.segment.io/v1'
ANALYTICS_TOKEN = 'Ax5LZzR1o3q3Z3WjATASDwR5rKyHH0qOIRIbLmMXn2H='
# Things We Track
SERVER_STARTUP = 'Server Startup'
SERVER_STARTUP_SUCCESS = 'Server Startup Success'
SERVER_STARTUP_ERROR = 'Server Startup Error'
DOWNLOAD_STARTED = 'Download Started'
DOWNLOAD_ERRORED = 'Download Errored'
DOWNLOAD_FINISHED = 'Download Finished'
HEARTBEAT = 'Heartbeat'
CLAIM_ACTION = 'Claim Action' # publish/create/update/abandon
NEW_CHANNEL = 'New Channel'
CREDITS_SENT = 'Credits Sent'
UPNP_SETUP = "UPnP Setup"
BLOB_BYTES_UPLOADED = 'Blob Bytes Uploaded'
TIME_TO_FIRST_BYTES = "Time To First Bytes"
log = logging.getLogger(__name__)
def _event_properties(installation_id: str, session_id: str,
event_properties: typing.Optional[typing.Dict]) -> typing.Dict:
properties = {
'lbry_id': installation_id,
'session_id': session_id,
}
properties.update(event_properties or {})
return properties
def _download_properties(conf: Config, external_ip: str, resolve_duration: float,
total_duration: typing.Optional[float], download_id: str, name: str,
outpoint: str, active_peer_count: typing.Optional[int],
tried_peers_count: typing.Optional[int], connection_failures_count: typing.Optional[int],
added_fixed_peers: bool, fixed_peer_delay: float, sd_hash: str,
sd_download_duration: typing.Optional[float] = None,
head_blob_hash: typing.Optional[str] = None,
head_blob_length: typing.Optional[int] = None,
head_blob_download_duration: typing.Optional[float] = None,
error: typing.Optional[str] = None, error_msg: typing.Optional[str] = None,
wallet_server: typing.Optional[str] = None) -> typing.Dict:
return {
"external_ip": external_ip,
"download_id": download_id,
"total_duration": round(total_duration, 4),
"resolve_duration": None if not resolve_duration else round(resolve_duration, 4),
"error": error,
"error_message": error_msg,
'name': name,
"outpoint": outpoint,
"node_rpc_timeout": conf.node_rpc_timeout,
"peer_connect_timeout": conf.peer_connect_timeout,
"blob_download_timeout": conf.blob_download_timeout,
"use_fixed_peers": len(conf.reflector_servers) > 0,
"fixed_peer_delay": fixed_peer_delay,
"added_fixed_peers": added_fixed_peers,
"active_peer_count": active_peer_count,
"tried_peers_count": tried_peers_count,
"sd_blob_hash": sd_hash,
"sd_blob_duration": None if not sd_download_duration else round(sd_download_duration, 4),
"head_blob_hash": head_blob_hash,
"head_blob_length": head_blob_length,
"head_blob_duration": None if not head_blob_download_duration else round(head_blob_download_duration, 4),
"connection_failures_count": connection_failures_count,
"wallet_server": wallet_server
}
def _make_context(platform):
# see https://segment.com/docs/spec/common/#context
# they say they'll ignore fields outside the spec, but evidently they don't
context = {
'app': {
'version': platform['lbrynet_version'],
'build': platform['build'],
},
# TODO: expand os info to give linux/osx specific info
'os': {
'name': platform['os_system'],
'version': platform['os_release']
},
}
if 'desktop' in platform and 'distro' in platform:
context['os']['desktop'] = platform['desktop']
context['os']['distro'] = platform['distro']
return context
class AnalyticsManager:
def __init__(self, conf: Config, installation_id: str, session_id: str):
self.conf = conf
self.cookies = {}
self.url = ANALYTICS_ENDPOINT
self._write_key = utils.deobfuscate(ANALYTICS_TOKEN)
self._enabled = conf.share_usage_data
self._tracked_data = collections.defaultdict(list)
self.context = _make_context(system_info.get_platform())
self.installation_id = installation_id
self.session_id = session_id
self.task: typing.Optional[asyncio.Task] = None
self.external_ip: typing.Optional[str] = None
@property
def is_started(self):
return self.task is not None
async def start(self):
if self._enabled and self.task is None:
self.external_ip = await utils.get_external_ip()
self.task = asyncio.create_task(self.run())
async def run(self):
while True:
await self._send_heartbeat()
await asyncio.sleep(1800)
self.external_ip = await utils.get_external_ip()
def stop(self):
if self.task is not None and not self.task.done():
self.task.cancel()
async def _post(self, data: typing.Dict):
request_kwargs = {
'method': 'POST',
'url': self.url + '/track',
'headers': {'Connection': 'Close'},
'auth': aiohttp.BasicAuth(self._write_key, ''),
'json': data,
'cookies': self.cookies
}
try:
async with utils.aiohttp_request(**request_kwargs) as response:
self.cookies.update(response.cookies)
except Exception as e:
log.debug('Encountered an exception while POSTing to %s: ', self.url + '/track', exc_info=e)
async def track(self, event: typing.Dict):
"""Send a single tracking event"""
if self._enabled:
log.debug('Sending track event: %s', event)
await self._post(event)
async def send_upnp_setup_success_fail(self, success, status):
await self.track(
self._event(UPNP_SETUP, {
'success': success,
'status': status,
})
)
async def send_server_startup(self):
await self.track(self._event(SERVER_STARTUP))
async def send_server_startup_success(self):
await self.track(self._event(SERVER_STARTUP_SUCCESS))
async def send_server_startup_error(self, message):
await self.track(self._event(SERVER_STARTUP_ERROR, {'message': message}))
async def send_time_to_first_bytes(self, resolve_duration: typing.Optional[float],
total_duration: typing.Optional[float], download_id: str,
name: str, outpoint: typing.Optional[str],
found_peers_count: typing.Optional[int],
tried_peers_count: typing.Optional[int],
connection_failures_count: typing.Optional[int],
added_fixed_peers: bool,
fixed_peers_delay: float, sd_hash: str,
sd_download_duration: typing.Optional[float] = None,
head_blob_hash: typing.Optional[str] = None,
head_blob_length: typing.Optional[int] = None,
head_blob_duration: typing.Optional[int] = None,
error: typing.Optional[str] = None,
error_msg: typing.Optional[str] = None,
wallet_server: typing.Optional[str] = None):
await self.track(self._event(TIME_TO_FIRST_BYTES, _download_properties(
self.conf, self.external_ip, resolve_duration, total_duration, download_id, name, outpoint,
found_peers_count, tried_peers_count, connection_failures_count, added_fixed_peers, fixed_peers_delay,
sd_hash, sd_download_duration, head_blob_hash, head_blob_length, head_blob_duration, error, error_msg,
wallet_server
)))
async def send_download_finished(self, download_id, name, sd_hash):
await self.track(
self._event(
DOWNLOAD_FINISHED, {
'download_id': download_id,
'name': name,
'stream_info': sd_hash
}
)
)
async def send_claim_action(self, action):
await self.track(self._event(CLAIM_ACTION, {'action': action}))
async def send_new_channel(self):
await self.track(self._event(NEW_CHANNEL))
async def send_credits_sent(self):
await self.track(self._event(CREDITS_SENT))
async def _send_heartbeat(self):
await self.track(self._event(HEARTBEAT))
def _event(self, event, properties: typing.Optional[typing.Dict] = None):
return {
'userId': 'lbry',
'event': event,
'properties': _event_properties(self.installation_id, self.session_id, properties),
'context': self.context,
'timestamp': utils.isonow()
}
+6
View File
@@ -0,0 +1,6 @@
from lbry.conf import Config
from lbry.extras.cli import execute_command
def daemon_rpc(conf: Config, method: str, **kwargs):
return execute_command(conf, method, kwargs, callback=lambda data: data)
+66
View File
@@ -0,0 +1,66 @@
import logging
import time
import hashlib
import binascii
from lbry import utils
import ecdsa
from torba.client.hash import sha256
from lbry.wallet.transaction import Output
log = logging.getLogger(__name__)
def get_encoded_signature(signature):
signature = signature.encode() if type(signature) is str else signature
r = int(signature[:int(len(signature) / 2)], 16)
s = int(signature[int(len(signature) / 2):], 16)
return ecdsa.util.sigencode_der(r, s, len(signature) * 4)
def cid2hash(claim_id: str) -> bytes:
return binascii.unhexlify(claim_id.encode())[::-1]
def is_comment_signed_by_channel(comment: dict, channel: Output, abandon=False):
if type(channel) is Output:
try:
signing_field = comment['comment_id'] if abandon else comment['comment']
pieces = [
comment['signing_ts'].encode(),
cid2hash(comment['channel_id']),
signing_field.encode()
]
return Output.is_signature_valid(
get_encoded_signature(comment['signature']),
sha256(b''.join(pieces)),
channel.claim.channel.public_key_bytes
)
except KeyError:
pass
return False
def sign_comment(comment: dict, channel: Output, abandon=False):
timestamp = str(int(time.time()))
signing_field = comment['comment_id'] if abandon else comment['comment']
pieces = [timestamp.encode(), channel.claim_hash, signing_field.encode()]
digest = sha256(b''.join(pieces))
signature = channel.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
comment.update({
'signature': binascii.hexlify(signature).decode(),
'signing_ts': timestamp
})
async def jsonrpc_post(url: str, method: str, params: dict = None, **kwargs) -> any:
params = params or {}
params.update(kwargs)
json_body = {'jsonrpc': '2.0', 'id': None, 'method': method, 'params': params}
async with utils.aiohttp_request('POST', url, json=json_body) as response:
try:
result = await response.json()
return result['result'] if 'result' in result else result
except Exception as cte:
log.exception('Unable to decode response from server: %s', cte)
return await response.text()
Loaded 100 of 471 files, more files were not shown because too many files have changed in this diff. Show more