Compare commits

..
Author SHA1 Message Date
Victor Shyba cf73e4599f Merge pull request #3585 from lbryio/fix_blob_db_queries
ci / lint (push) Waiting to run
ci / tests / unit (macos-latest) (push) Waiting to run
ci / tests / unit (ubuntu-latest) (push) Waiting to run
ci / tests / unit (windows-latest) (push) Waiting to run
ci / tests / integration (blockchain) (push) Waiting to run
ci / tests / integration (claims) (push) Waiting to run
ci / tests / integration (claims_legacy_search) (push) Waiting to run
ci / tests / integration (datanetwork) (push) Waiting to run
ci / tests / integration (other) (push) Waiting to run
ci / tests / integration (takeovers) (push) Waiting to run
ci / tests / integration (takeovers_legacy_search) (push) Waiting to run
ci / tests / integration (transactions) (push) Waiting to run
Fixes bugs on disk space management and stream recovery
2022-03-24 21:05:09 -03:00
Victor Shyba 2a698932da implement announcer as a consumer task on gather 2022-03-15 15:36:00 -03:00
Victor Shyba ee36162b16 make active an explicit ordered dict 2022-03-15 15:36:00 -03:00
Victor Shyba 97f472d4e1 remove unused search rounds 2022-03-15 15:36:00 -03:00
Victor Shyba 160e227e90 timeout is now supported on dht tests 2022-03-15 15:36:00 -03:00
Victor Shyba 475ba010e3 fix and enable test_blob_announcer 2022-03-15 15:36:00 -03:00
Victor Shyba 57ebbbcb78 simplify dht mock and restore clock after accelerating 2022-03-15 15:36:00 -03:00
Victor Shyba cbf973bf2d better representation of kademliapeer on debug logs 2022-03-15 15:36:00 -03:00
Victor Shyba 9d26ad96c6 add a way to wait announcements to finish so tests are reliable 2022-03-15 15:36:00 -03:00
Victor Shyba c2a3ec3265 make timeout handler immune to asyncio time tricks 2022-03-15 15:36:00 -03:00
Victor Shyba 521d783260 allow running some extra probes for k replacements 2022-03-15 15:36:00 -03:00
Victor Shyba 50846cd37e remove all references to bottoming out 2022-03-15 15:36:00 -03:00
Victor Shyba 40ed5f311b no stop condition, let it exhaust 2022-03-15 15:36:00 -03:00
Victor Shyba 293410ac9f bottoming out is now warning and no results for peer search 2022-03-15 15:36:00 -03:00
Victor Shyba 068f46d137 don't probe peers too far from the top closest 2022-03-15 15:36:00 -03:00
Victor Shyba 5c60c09fef use a dict for the active queue 2022-03-15 15:36:00 -03:00
Victor Shyba aca4a243d6 log bottom out of peer search in debug, show short key id for find value 2022-03-15 15:36:00 -03:00
Victor Shyba 422f292d82 bump bottom out limit of peer search so people can use 100 concurrent announcers 2022-03-15 15:36:00 -03:00
Victor Shyba c81598aab8 wait until k peers are ready. do not double add peers 2022-03-15 15:36:00 -03:00
Victor Shyba 3f8c8e01c6 only return good (contacted) peers 2022-03-15 15:36:00 -03:00
Victor Shyba 8d90653395 reset closest peer on failure 2022-03-15 15:36:00 -03:00
Victor Shyba 01841694ad add peers from shortlist regardless, but check from other nodes 2022-03-15 15:36:00 -03:00
Victor Shyba c3d0dd3073 bump split index to 2 2022-03-15 15:36:00 -03:00
Victor Shyba 4851f5c300 fix distance sorting and improve logging 2022-03-15 15:36:00 -03:00
Victor Shyba aef5f11a18 closest peer is only ready when it was contacted and isn't known to be bad 2022-03-15 15:36:00 -03:00
Victor Shyba 700f1cb8e5 dont probe and ignore bad peers 2022-03-15 15:36:00 -03:00
Victor Shyba 7e5f7c1725 simplify, genaralize to any size and fix tests 2022-03-15 15:36:00 -03:00
Victor Shyba aa7a17b112 stop after finding what to download 2022-03-15 15:36:00 -03:00
Victor Shyba 838fd71538 replace duplicated code 2022-03-15 15:36:00 -03:00
Victor Shyba 0146044b90 add get_colliding_prefix_bits, docs and tests 2022-03-15 15:36:00 -03:00
Victor Shyba f8f73ed4a6 extract method and avoid using hash builtin name 2022-03-15 15:36:00 -03:00
Victor Shyba 74bf8c551b extract min_prefix_colliding_bits to a contanst 2022-03-15 15:36:00 -03:00
Victor Shyba 4be346d0bb check that the stored blob is at least 1 prefix byte close to peer id 2022-03-15 15:36:00 -03:00
Victor Shyba 4fd1fe0046 add migrator to set head blobs should_announce=0 2022-03-15 15:36:00 -03:00
Victor Shyba 9524f03eac fix test_announces 2022-03-15 15:36:00 -03:00
Victor Shyba 90c8a8bea3 do not search for the head blob 2022-03-15 15:36:00 -03:00
Victor Shyba 154c41bd8d dont set head blob to announce on save 2022-03-15 15:36:00 -03:00
Victor Shyba f271511484 break tie by length 2022-03-15 15:36:00 -03:00
Victor Shyba dde4a5b495 fix and test case for blob_clean after disabling network storage 2022-03-15 15:36:00 -03:00
Victor Shyba 68a940f3a6 put back all the peers, get rid of re_add 2022-03-15 15:36:00 -03:00
Alex Grintsvayg 8c773151a3 let stream_update work on non-stream claims 2022-03-15 15:36:00 -03:00
Jeffrey Picard f1f5276a04 Update __init__.py
Update go hub binary to fix es sync test.
2022-03-15 15:36:00 -03:00
Lex Berezhny 51490b36e6 fix 2022-03-15 15:36:00 -03:00
Lex Berezhny e00dc78e1f prevent creation of change which is below the dust threshold of 1000 dewies 2022-03-15 15:36:00 -03:00
Victor Shyba 72beb02ec1 bump DHT peer manager cache to 16384 2022-03-15 15:36:00 -03:00
Victor Shyba b13c346418 add gauge for queue size 2022-03-15 15:36:00 -03:00
Victor Shyba 2520465fc1 count announcements and how many peers we were able to announce to 2022-03-15 15:36:00 -03:00
Eugene Dubinin dd868d5148 adds tests for guess_media_type
removes unnecessary comments
2022-03-15 15:36:00 -03:00
Eugene Dubinin e90fb9a28f adjusts code style 2022-03-15 15:36:00 -03:00
Eugene Dubinin 96d1745232 fixes KeyError on missing synonyms 2022-03-15 15:36:00 -03:00
Eugene Dubinin cab5f9188f detect media_type from the file contents 2022-03-15 15:36:00 -03:00
Victor Shyba 44e647d85d remove request_flight metric 2022-03-15 15:36:00 -03:00
Victor Shyba 5686c2ade9 disable CSV endpoints by default 2022-03-15 15:36:00 -03:00
Victor Shyba 1cc321e9cd remove estimation endpoints as that is done over prometheus metrics now 2022-03-15 15:36:00 -03:00
Victor Shyba b4d144da21 add granular metric for stored blob prefix, for network announcements calculation 2022-03-15 15:36:00 -03:00
Victor Shyba 39c2c0b922 change colliding bits metric to gauge 2022-03-15 15:36:00 -03:00
Victor Shyba dbf11fe750 count bit collisions between 8 and 16 2022-03-15 15:36:00 -03:00
Victor Shyba cc391324e4 add counter for peers with colliding bytes 2022-03-15 15:36:00 -03:00
Victor Shyba 273b4f6d3b add requests in flight and error 2022-03-15 15:36:00 -03:00
Victor Shyba d359d25935 add request received 2022-03-15 15:36:00 -03:00
Victor Shyba 645a81cec2 add request_sent and request_time metric on dht 2022-03-15 15:36:00 -03:00
Victor Shyba 06da53ef09 add storing_peers and peer_manager_keys 2022-03-15 15:36:00 -03:00
Victor Shyba 692af0cc55 prometheus: move blobs_stored and peers to SDK. add buckets_in_routing_table 2022-03-15 15:36:00 -03:00
Victor Shyba 32b95fa04a add passive estimation to prometheus 2022-03-15 15:36:00 -03:00
Victor Shyba 5ebcbc6370 fix missing async 2022-03-15 15:36:00 -03:00
Victor Shyba 3978d64056 keep same node id between runs 2022-03-15 15:35:59 -03:00
Victor Shyba d35f7a5ed5 add semaphore on active estimation to avoid abuse 2022-03-15 15:35:59 -03:00
Victor Shyba 2f84c38f6a same api across different estimation methods 2022-03-15 15:35:59 -03:00
Victor Shyba b656759e0e be explicit about ignoring params 2022-03-15 15:35:59 -03:00
Victor Shyba 15ac365463 better endpoint names, small docs 2022-03-15 15:35:59 -03:00
Victor Shyba 2ab78a92ca add prefix_neighbors_count to routing table debug api 2022-03-15 15:35:59 -03:00
Jack Robison 71b5c4c6ad improve script 2022-03-15 15:35:59 -03:00
Victor Shyba eae1a0912b first attempt at crawling 2022-03-15 15:35:59 -03:00
Lex Berezhny d5ec647f46 added integration test 2022-03-15 15:35:59 -03:00
vertbyqb e066b2a9da jsonrpc_channel_sign - Convert hexdata to a string before signing
Fixes #3533
2022-03-15 15:35:59 -03:00
Victor Shyba 8adec9d0ad extract cache values, increase peer cache to 2048 2022-03-15 15:35:59 -03:00
Victor Shyba 030e407584 DHT bugfix: failures tracking should be bound to 2048 LRU cache size 2022-03-15 15:35:59 -03:00
Victor Shyba 7ed807e41b fix missing docopt argument 2022-03-15 15:35:59 -03:00
Victor Shyba 354cc9b22d bump hub version to latest supporting sd_hash search 2022-03-15 15:35:59 -03:00
Victor Shyba ff51a5bdc6 update hub protobuf including sd_hash field 2022-03-15 15:35:59 -03:00
Victor Shyba 8618be1b5c increase indexed sd_hash prefix to 4 chars 2022-03-15 15:35:59 -03:00
Victor Shyba ebe351c67e enable and test prefix search for sd hash 2022-03-15 15:35:59 -03:00
Victor Shyba 6ad2e31976 sync and search sd_hash 2022-03-15 15:35:59 -03:00
Victor Shyba d44dc103d5 add test 2022-03-15 15:35:59 -03:00
Victor Shyba 331ce1f12c add sd hash to API 2022-03-15 15:35:59 -03:00
Lex Berezhny 37c408a6e7 changes default coin selection strategy from standard to prefer_confirmed 2022-03-15 15:34:09 -03:00
Victor Shyba a866621ccb do not limit DHT results by K, respect max_results 2022-03-15 15:34:09 -03:00
Victor Shyba b45b87e72d clarify DHT debug logging on key and operation 2022-03-15 15:34:09 -03:00
Jeffrey Picard 300ae6b134 Switch RangeField back to ints 2022-03-15 15:34:09 -03:00
Jeffrey Picard 0cb66e1f0d Try forcing tox reset 2022-03-15 15:34:09 -03:00
Jeffrey Picard 618f739049 Debugging 2022-03-15 15:34:09 -03:00
Jeffrey Picard 5c8329e951 Update es version in workflow 2022-03-15 15:34:09 -03:00
Jeffrey Picard 683f6b366d Update protobufs, go hub shim, and claim test. 2022-03-15 15:34:09 -03:00
Jack Robison ce63c2281f remove unused ES fields 2022-03-15 15:34:09 -03:00
Jack Robison 251cb4925a support lists of constraints for all range fields 2022-03-15 15:34:09 -03:00
FemtosecondLaser f8fe205066 returned conditional check in add_timeout() as it was making test_node.py tests unhappy 2022-03-15 15:34:09 -03:00
FemtosecondLaser 9908775576 removed conditional check in add_timeout() 2022-03-15 15:34:09 -03:00
FemtosecondLaser bf7745ec8d changed addTimeout to add_timeout for lint compliance 2022-03-15 15:34:09 -03:00
FemtosecondLaser 1aa296e168 added timeout of async operations to integration test setup/teardown 2022-03-15 15:34:09 -03:00
Jack Robison 171c99b12b v0.106.0 2022-03-15 15:34:09 -03:00
Jack Robison 83bb12265e update json docs 2022-03-15 15:34:09 -03:00
Jack Robison b4e99faba7 update claim_search doc
backward compatibility for `trending_mixed`, `trending_local`, `trending_global`, and `trending_group` args to `claim_search`
2022-03-15 15:34:09 -03:00
Jack Robison 78b559b274 update json docs 2022-03-15 15:34:09 -03:00
Jack Robison 974b933612 update header checkpoints 2022-03-15 15:34:09 -03:00
Victor Shyba 3aac113d45 test case for stream_type search on claims missing source + fix 2022-03-15 15:34:09 -03:00
Lex Berezhny a04d50a3bc fee per name env var 2022-03-15 15:34:09 -03:00
Victor Shyba 6ec30798ca fix typo from arg name 2022-03-15 15:34:09 -03:00
Jack Robison b4d47e423d keep touched_or_deleted records 2022-03-15 15:34:09 -03:00
Victor Shyba bf86e7658e add blob endpoint for listing announced blobs 2022-03-15 15:34:09 -03:00
Victor Shyba 98c0200c53 add /peers.csv to monitoring endpoint 2022-03-15 15:34:09 -03:00
Brendon J. Brewer a850fb64c8 Rename trending 2022-03-15 15:34:09 -03:00
Jack Robison 02b0f3e9f2 fix missing es notification for support amount changing 2022-03-15 15:34:09 -03:00
Jack Robison 948a58f628 improve tests 2022-03-15 15:34:09 -03:00
Jack Robison 21a2076f26 improve resolve tests 2022-03-15 15:34:09 -03:00
Jack Robison efd8fd81ed fix duplicate trending notification to ES 2022-03-15 15:34:09 -03:00
Jack Robison 5e73f82c23 fix effective amount for resolve/ES being off while claims/supports are unactivated 2022-03-15 15:34:09 -03:00
Jack Robison 01691dd92a fix test_colliding_short_id 2022-03-15 15:34:09 -03:00
Jack Robison 5fe339c53b update default tcp/blob port to be the same as the default udp/dht port (4444) 2022-03-15 15:34:09 -03:00
Victor Shyba c4b86454b5 log unexpected errors, rename task/loop 2022-03-15 15:34:09 -03:00
Victor Shyba 60969e4817 clear cache on test assertions 2022-03-15 15:34:09 -03:00
Victor Shyba 48bb84fc1e make sure the downloader always stops gracefully 2022-03-15 15:34:09 -03:00
Victor Shyba b055c25156 cache space stats from running components so status is instant 2022-03-15 15:34:09 -03:00
Victor Shyba 35905b99ff add index for blob table so size summaries are faster 2022-03-15 15:34:09 -03:00
Victor Shyba 806857c066 download from stored announcements and dont reannounce 2022-03-15 15:34:09 -03:00
Victor Shyba ba8a7fc351 improve disk space manager status, include more info and unify space queries 2022-03-15 15:34:09 -03:00
Victor Shyba a36fd76eb1 normal_blobs->stream_blobs, proactive->background 2022-03-15 15:34:09 -03:00
Victor Shyba 02994164be fix free space calculation, test it and give a margin of 10mb before starting so it doesnt insist when full 2022-03-15 15:34:09 -03:00
Victor Shyba c3b89a896b move more logic out of the downloader component 2022-03-15 15:34:09 -03:00
Victor Shyba b0bf6eff16 extract background downloader to its own class 2022-03-15 15:34:09 -03:00
Victor Shyba 47c4cb800d add analytics event for network disk space 2022-03-15 15:34:09 -03:00
Victor Shyba c5f6a4ca1c announce orphan blobs manually, as that was done when save stream 2022-03-15 15:34:09 -03:00
Victor Shyba 813108b9d8 cleanup background downloader blobs from conf 2022-03-15 15:34:09 -03:00
Victor Shyba 95703b3af5 separated network seeding space metrics 2022-03-15 15:34:09 -03:00
Victor Shyba 2ceec7c3d4 don't save streams for network blobs and bypass disk space manager 2022-03-15 15:34:09 -03:00
Victor Shyba 3985f1578e add conf for network seeding space limit 2022-03-15 15:34:09 -03:00
Victor Shyba 355ccb1e32 schedule the download task instead 2022-03-15 15:34:09 -03:00
Victor Shyba 04d2c6f013 fix unit tests from component dependency chain changes 2022-03-15 15:34:09 -03:00
Victor Shyba 8d6a7101f6 download from DHT 2022-03-15 15:34:09 -03:00
Victor Shyba abbd8473bb drop channel support, prepare to hook into DHT 2022-03-15 15:34:09 -03:00
Victor Shyba 02b2103d94 handle case where something that isn't a sd blob gets hit 2022-03-15 15:34:09 -03:00
Victor Shyba 85474ae381 no api yet 2022-03-15 15:34:09 -03:00
Victor Shyba 9d3c401abb download only blobs 2022-03-15 15:34:09 -03:00
Victor Shyba 32a7d1a4a0 fix tests 2022-03-15 15:34:09 -03:00
Victor Shyba 602cd5bd2e fix exception arguments 2022-03-15 15:34:09 -03:00
Victor Shyba 4b88b191e7 test add/remove/list subscriptions 2022-03-15 15:34:09 -03:00
Victor Shyba e449cd01ba fix and test main api 2022-03-15 15:34:08 -03:00
Victor Shyba b1db42acf4 download all blobs and check that on tests 2022-03-15 15:34:08 -03:00
Victor Shyba fcbe8cf00b create downloader component and initial tests 2022-03-15 15:34:08 -03:00
Victor Shyba a1866c40f5 with the fix we no longer need to restart the stream 2022-03-15 15:34:08 -03:00
Victor Shyba d78540f3cc fix tests by checking there are actual blobs being deleted 2022-03-15 15:34:08 -03:00
Victor Shyba 7c1c04674c dont lose results on duplicates, just warn 2022-03-15 15:34:08 -03:00
Victor Shyba 694aeab75c remove tried_for_this_blob so banned peers are retried for same blob 2022-03-15 15:34:08 -03:00
Cristian Vicas 90aaf64b77 Bug [#2070] where blob_get RPC timed out.
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.

Bug [#2070] where blob_get RPC timed out.

Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
2022-03-15 15:34:08 -03:00
Jack Robison 6eeabb1a1a use mempool cache in transaction_get_batch 2022-03-15 15:34:08 -03:00
FemtosecondLaser 77d58b82a0 Added an integration test covering the following scenario:
On start, if download dir is non-writable - daemon terminates with a helpful message.
2022-03-15 15:34:08 -03:00
FemtosecondLaser a7af877e9e Improved the readability of the tests. 2022-03-15 15:34:08 -03:00
FemtosecondLaser 25092f56be Changed the tests to execute against a real file system instead of a fake one. 2022-03-15 15:34:08 -03:00
FemtosecondLaser e55f9dd21e renamed the test class to be more specific about the sut 2022-03-15 15:34:08 -03:00
FemtosecondLaser 6c10509705 removed redundant tests
renamed a test to be more specific about the kind of the precondition
2022-03-15 15:34:08 -03:00
FemtosecondLaser bed2e253b2 Modified ensure_directory_exists() to check if the directory is writable by the process. 2022-03-15 15:34:08 -03:00
Jack Robison c683ae9ed6 Update docker-compose-wallet-server.yml 2022-03-15 15:34:08 -03:00
Jack Robison 8ce1cebd07 add script to setup docker volumes from snapshots 2022-03-15 15:34:08 -03:00
Jack Robison 9029b4b5c3 fix attempting to update trending on abandoned claims 2022-03-15 15:34:08 -03:00
Jack Robison db0831d55b update docs 2022-03-15 15:34:08 -03:00
Jack Robison b3705073a3 skip loading tx/claim caches in the elastic sync script when not needed 2022-03-15 15:34:08 -03:00
Jack Robison 755e8ce101 update trending with help from @eggplantbren 2022-03-15 15:34:08 -03:00
Jack Robison fd2ab47a16 update sync script to handle ES falling behind leveldb on shutdown 2022-03-15 15:34:08 -03:00
Jack Robison 9aafb7a743 logging 2022-03-15 15:34:08 -03:00
Jack Robison aa1b20cf7b update docker 2022-03-15 15:34:08 -03:00
Jack Robison 3cd2227c29 update Env to accept parameters from cli args 2022-03-15 15:34:08 -03:00
Jack Robison 27cc83c03b set default CACHE_MB to 1024mb and the default QUERY_TIMEOUT_MS to 10s 2022-03-15 15:34:08 -03:00
Jack Robison cb6db3f3d8 remove unused hub env settings 2022-03-15 15:34:08 -03:00
Jack Robison 9020e39a83 update lbry-hub-elastic-sync to support resyncing recent blocks 2022-03-15 15:34:08 -03:00
Jack Robison 64509ca95d add CACHE_ALL_CLAIM_TXOS hub setting 2022-03-15 15:34:08 -03:00
Jack Robison dafd62104b remove dead code 2022-03-15 15:34:08 -03:00
Jack Robison e09588e433 add CACHE_ALL_TX_HASHES setting to optionally use more memory to save i/o 2022-03-15 15:34:08 -03:00
Jack Robison 50e00192e8 sleeps 2022-03-15 15:34:08 -03:00
Jack Robison e6d470f110 improve resolve caching 2022-03-15 15:34:08 -03:00
Jack Robison eeaf9a72e2 threadpools for block processor and es sync reader 2022-03-15 15:34:08 -03:00
Jack Robison a4ad1bb0a9 doc strings 2022-03-15 15:34:08 -03:00
Jack Robison e0086682b9 improve claims_producer performance 2022-03-15 15:34:08 -03:00
Jack Robison 15ac2ade59 resolve lru cache 2022-03-15 15:34:08 -03:00
Jack Robison 09e0d5c55e add block_txs index 2022-03-15 15:34:08 -03:00
Jack Robison dfe855c0c9 smaller caches 2022-03-15 15:34:08 -03:00
Victor Shyba a2996768fe add optional prometheus to dht_node script 2022-03-15 15:34:08 -03:00
Jack Robison a53b454d86 Update daemon.py
docstring
2022-03-15 15:34:08 -03:00
Cristian Vicas 01557b599a Updated documentation for RPC calls: status, blob_list. 2022-03-15 15:34:08 -03:00
belikor 65323b4169 file_manager: raise new InvalidStreamURLError if the URL is invalid
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.

For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```

```
Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```

Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING  lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING  lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```

This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
2022-03-15 15:34:08 -03:00
Jack Robison cdef8b4852 clear es attributes during initial sync 2022-03-15 15:34:08 -03:00
Jack Robison 8d72142390 fix setting references on txos in extra_txos 2022-03-15 15:34:08 -03:00
Jack Robison 172a0f0ac2 fix reposted channel being missing from resolve result
-improve names of the resolve related methods in `LevelDB`
2022-03-15 15:34:08 -03:00
Jack Robison 1022080be6 fix compactify script 2022-03-15 15:34:08 -03:00
Jack Robison 8031a55dbb fix test 2022-03-15 15:34:08 -03:00
Jack Robison 1ddd29656e move test_transaction_commands, test_internal_transaction_api , and test_transactions into their own runner
-move test_resolve_command to its own runner
2022-03-15 15:34:08 -03:00
Jack Robison cb5c39a159 only save undo info for blocks within reorg limit 2022-03-15 15:34:08 -03:00
Jack Robison 9ec510c742 fix channel count 2022-03-15 15:34:08 -03:00
Jack Robison aa80cf47b9 try to fix test_sqlite_coin_chooser 2022-03-15 15:34:08 -03:00
Jack Robison e4e1d42e09 move test_claim_commands and test_resolve_command into new directory 2022-03-15 15:34:08 -03:00
Jack Robison ccc9a900ef sort touched or deleted claim hashes 2022-03-15 15:34:08 -03:00
Jack Robison c2a5ff0ae3 block processor db refactoring
-access db through HubDB class, don't use plyvel.DB directly
-add channel count and support amount prefixes
2022-03-15 15:34:08 -03:00
Jack Robison afa710dcb5 fix and add test for abandoning a controlling in the same block a new claim is made 2022-03-15 15:34:08 -03:00
Jack Robison d0ed7593de fix test_sqlite_coin_chooser 2022-03-15 15:34:08 -03:00
Jack Robison 830fc7f8cc fix ES index name so it stays the same within a test case 2022-03-15 15:34:08 -03:00
Jack Robison 88e8926a59 fix bug with early takeover by an update 2022-03-15 15:34:08 -03:00
Jack Robison 65e39b8e76 fix test 2022-03-15 15:34:08 -03:00
Jack Robison acd0c2188f fix activating non existent claim 2022-03-15 15:34:08 -03:00
Jack Robison 6155cda66f add tests for takeovers from amount changes in updates before/on/after activation 2022-03-15 15:34:08 -03:00
Jack Robison f77f2f6e80 batch address history notifications 2022-03-15 15:34:08 -03:00
Jack Robison 68d397a269 improve leveldb caching 2022-03-15 15:34:08 -03:00
Jack Robison 66ed57c834 refactor reload_blocking_filtering_streams 2022-03-15 15:34:08 -03:00
Jack Robison 8b42475d59 fix filtering error upon abandon 2022-03-15 15:34:08 -03:00
Jack Robison de9159d83c faster read_claim_txos 2022-03-15 15:34:08 -03:00
Jack Robison b66adfdf78 remove unused executor 2022-03-15 15:34:08 -03:00
Victor Shyba a4be5eb1d8 fix blocking and filtering 2022-03-15 15:34:08 -03:00
Jack Robison c8b6db4b84 fix test 2022-03-15 15:34:08 -03:00
Jack Robison 7d3eef1fd9 fix logging number of notified sessions 2022-03-15 15:34:08 -03:00
Jack Robison 6e93c5bc0d fix update that initiates takeover not being delayed 2022-03-15 15:34:08 -03:00
Jack Robison 16b21b8bda use hub binary from https://github.com/lbryio/hub/pull/13 2022-03-15 15:34:08 -03:00
Jack Robison a0e0039988 handle invalid release time 2022-03-15 15:34:08 -03:00
Jack Robison 3970e91766 name and normalized -> claim_name and normalized_name
-update generated pb files
2022-03-15 15:34:08 -03:00
Jack Robison c4126edb79 fix es sync.py 2022-03-15 15:34:08 -03:00
Jack Robison 589a6588da fix all_claims_producer 2022-03-15 15:34:08 -03:00
Jack Robison 1ec4111b9f merge conflicts 2022-03-15 15:34:08 -03:00
Jack Robison c00912015a claim search fixes 2022-03-15 15:34:08 -03:00
Jack Robison 0d19439982 fix release_time and creation_timestamp 2022-03-15 15:34:08 -03:00
Jack Robison 12f790ab01 delete unused code 2022-03-15 15:34:08 -03:00
Jack Robison 4e687c4fd8 test_spec_example 2022-03-15 15:34:08 -03:00
Jack Robison ff960fda0e non blocking claim producer 2022-03-15 15:34:08 -03:00
Jack Robison 72e45b5cb1 fix tests 2022-03-15 15:34:08 -03:00
Jack Robison b59e0490a2 renormalization 2022-03-15 15:34:08 -03:00
Jack Robison 94754f2047 add trending integration test 2022-03-15 15:34:08 -03:00
Jack Robison ca335b7a65 fix trending overflow 2022-03-15 15:34:08 -03:00
Jack Robison 8f9e6a519d refactor trending 2022-03-15 15:34:08 -03:00
Jack Robison da75968078 trending fixes 2022-03-15 15:34:08 -03:00
Jack Robison 3800fb1ab0 make app backward compatible with trending_score
-update trending decay function to zero out low trending score values faster
2022-03-15 15:34:08 -03:00
Jack Robison a51bf6a4fa log time to update and decay trending in elasticsearch 2022-03-15 15:34:08 -03:00
Jack Robison 0be141188c skip integrity errors for trending spikes 2022-03-15 15:34:08 -03:00
Jack Robison 6f2b985b73 update trending in elasticsearch
-add TrendingPrefixSpike to leveldb
-expose `TRENDING_HALF_LIFE`, `TRENDING_WHALE_HALF_LIFE` and `TRENDING_WHALE_THRESHOLD` hub settings
2022-03-15 15:34:08 -03:00
Brendon J. Brewer 8eba05308d constants 2022-03-15 15:34:08 -03:00
Brendon J. Brewer ecab123a3a Put trending score into ES 2022-03-15 15:34:08 -03:00
Brendon J. Brewer 180f6c4519 Mark claims as touched 2022-03-15 15:34:08 -03:00
Jack Robison dc4e362e10 fix non normalized canonical urls 2022-03-15 15:34:08 -03:00
Jack Robison 7e78fdec04 handle unicode error for unnormalized names 2022-03-15 15:34:08 -03:00
Jack Robison e212ce23e3 prefix db 2022-03-15 15:34:08 -03:00
Brendon J. Brewer 0c7be8975f trending 2022-03-15 15:34:08 -03:00
Jack Robison 6cba95c148 fix claims not having non-normalized names 2022-03-15 15:34:08 -03:00
Jack Robison f62d128621 update test 2022-03-15 15:34:08 -03:00
Jack Robison 91a86fd12f merge conflicts 2022-03-15 15:34:08 -03:00
Jack Robison c819d494c3 fix missing fields in reposts 2022-03-15 15:34:08 -03:00
Jack Robison b4853c5f67 fix merge conflicts and simplify extract_doc 2022-03-15 15:34:08 -03:00
Jack Robison c68334b421 fix bulk es sync 2022-03-15 15:34:08 -03:00
Jack Robison 613acc7b00 fix tests 2022-03-15 15:34:08 -03:00
Jack Robison 371fc4d68c fix tests 2022-03-15 15:34:08 -03:00
Jack Robison 7ef0ae12a1 handle invalid claim update 2022-03-15 15:34:08 -03:00
Jack Robison dc34e8884c fix keeping claim_hash_to_txo and txo_to_claim in sync 2022-03-15 15:34:08 -03:00
Victor Shyba 08d635322e implement blocking and filtering 2022-03-15 15:34:08 -03:00
Jack Robison bc0c1b9a3e fix _get_pending_claim_name 2022-03-15 15:34:08 -03:00
Jack Robison c42ee926da fix applying expiration fork 2022-03-15 15:34:08 -03:00
Jack Robison c59d08080e update iterators to use pack_partial_key 2022-03-15 15:34:08 -03:00
Jack Robison 94e0624024 delete lbry/wallet/server/storage.py
-expose leveldb lru cache size as `CACHE_MB` hub param
2022-03-15 15:34:08 -03:00
Jack Robison e94a2c7c94 remove dead code 2022-03-15 15:34:07 -03:00
Jack Robison cd1ce32377 non blocking mempool loop 2022-03-15 15:34:07 -03:00
Jack Robison d1560ef09b faster es sync 2022-03-15 15:34:07 -03:00
Jack Robison 6699d1e2f8 run advance_block in threadpool 2022-03-15 15:34:07 -03:00
Jack Robison f1fbfa1b39 use claim_to_txo cache 2022-03-15 15:34:07 -03:00
Jack Robison 0d86717a9a faster _cached_get_active_amount for claims
-remove dead code
2022-03-15 15:34:07 -03:00
Jack Robison 528af27e4a fix claim search by fee for claims without fees 2022-03-15 15:34:07 -03:00
Jack Robison fe69afaa56 clear claim_to_txo cache before reading 2022-03-15 15:34:07 -03:00
Jack Robison 31d7823498 handle claims that dont exist in ES sync 2022-03-15 15:34:07 -03:00
Jack Robison ef6ec03161 in memory claim_to_txo and txo_to_claim dictionaries 2022-03-15 15:34:07 -03:00
Jack Robison 68596be1b9 fix spend_utxo 2022-03-15 15:34:07 -03:00
Jack Robison a48fe84971 split flush from advance_block 2022-03-15 15:34:07 -03:00
Jack Robison bfbe7c1bf5 rename extend_ops 2022-03-15 15:34:07 -03:00
Jack Robison 085ce1ff16 faster spend_utxo 2022-03-15 15:34:07 -03:00
Jack Robison c81c0d9480 faster get_future_activated 2022-03-15 15:34:07 -03:00
Jack Robison 05b6bdb8f6 threadpool 2022-03-15 15:34:07 -03:00
Jack Robison 991d4f8859 cleanup 2022-03-15 15:34:07 -03:00
Jack Robison 22c75605ee fix getting block hash during reorg 2022-03-15 15:34:07 -03:00
Jack Robison f52faa8d14 fix test 2022-03-15 15:34:07 -03:00
Jack Robison 8dd88a2780 fix spends in address histories 2022-03-15 15:34:07 -03:00
Jack Robison 7ad2234983 faster es sync 2022-03-15 15:34:07 -03:00
Jack Robison fb5c008fc5 fix expiring channels 2022-03-15 15:34:07 -03:00
Jack Robison f55b6bdc71 fix abandoning signed claims in the same tx as their channel
-fix canonical/short url in es
2022-03-15 15:34:07 -03:00
Jack Robison de9edb0695 handle failure to generate a short id 2022-03-15 15:34:07 -03:00
Jack Robison 2180e24bc1 fix resolve by short id 2022-03-15 15:34:07 -03:00
Jack Robison 749e64b101 tests 2022-03-15 15:34:07 -03:00
Jack Robison b4eaa5f918 move MemPool into BlockProcessor 2022-03-15 15:34:07 -03:00
Jack Robison d4194954d3 combine MemPool and Notifications classes 2022-03-15 15:34:07 -03:00
Jack Robison 6e221fc7d9 fix touched hashXs notifications 2022-03-15 15:34:07 -03:00
Jack Robison ea1285cd9f reorg claims in the search index 2022-03-15 15:34:07 -03:00
Jack Robison 4e77fa100b small fixes 2022-03-15 15:34:07 -03:00
Jack Robison babb76d90d rename 2022-03-15 15:34:07 -03:00
Jack Robison f73153ed8d delete stale code 2022-03-15 15:34:07 -03:00
Jack Robison acfc1f56ee simplify advance and reorg 2022-03-15 15:34:07 -03:00
Jack Robison 81773a6497 update limited_history 2022-03-15 15:34:07 -03:00
Jack Robison 25cf751158 update lookup_utxos 2022-03-15 15:34:07 -03:00
Jack Robison a9c8061c0c update RevertableOpStack 2022-03-15 15:34:07 -03:00
Jack Robison f53b1ee290 add remaining db prefixes 2022-03-15 15:34:07 -03:00
Jack Robison 85b46ecff6 update plyvel to 1.3.0
https://github.com/lbryio/lbry-sdk/pull/3205#issuecomment-877564489
2022-03-15 15:34:07 -03:00
Jack Robison fd2753b95a cleanup 2022-03-15 15:34:07 -03:00
Jack Robison 768934e1cc skip es sync during initial hub sync, halt the hub upon finishing initial sync 2022-03-15 15:34:07 -03:00
Jack Robison ad7dee3e7f fix es sync 2022-03-15 15:34:07 -03:00
Jack Robison 354c69bf4f fix non localhost elasticsearch 2022-03-15 15:34:07 -03:00
Jack Robison 52aa045635 improve channel invalidation test 2022-03-15 15:34:07 -03:00
Jack Robison 1c8f92239d fix stream_update --clear_channel flag 2022-03-15 15:34:07 -03:00
Jack Robison 0c85de7839 fix signed claim invalidation corner cases 2022-03-15 15:34:07 -03:00
Jack Robison d74d06d97b extra deletes
-the channel_to_claim/claim_to_channel entries already get deleted when the claim txo is spent
2022-03-15 15:34:07 -03:00
Jack Robison 55351c5842 fix has_no_source for reposts 2022-03-15 15:34:07 -03:00
Jack Robison e67152ec14 update channel_to_claim and claim_to_channel at the same time 2022-03-15 15:34:07 -03:00
Jack Robison 287ff0a557 typing and fix error string 2022-03-15 15:34:07 -03:00
Jack Robison bf539d67ea update staged txo_to_claim after invalidating channel sig
-fixes abandon of claim with invalidated signature and an update in same block
2022-03-15 15:34:07 -03:00
Jack Robison 69651453dd missing channel_to_claim delete 2022-03-15 15:34:07 -03:00
Jack Robison 7f743ff3f1 fix RepostKey 2022-03-15 15:34:07 -03:00
Jack Robison a1b7c61b56 typing 2022-03-15 15:34:07 -03:00
Jack Robison fbd1d53542 fix 2022-03-15 15:34:07 -03:00
Jack Robison 02adc74e2c cleanup 2022-03-15 15:34:07 -03:00
Jack Robison 3a452a3b2a rename effective_amount prefix 2022-03-15 15:34:07 -03:00
Jack Robison 91f8b3b505 fix undeleted claim_to_channel record 2022-03-15 15:34:07 -03:00
Jack Robison ce09f2eb0f cleanup 2022-03-15 15:34:07 -03:00
Jack Robison 02d479b92d fix mismatch in claim_to_txo<->txo_to_claim 2022-03-15 15:34:07 -03:00
Jack Robison bce14c3d79 rename things
-fix effective amount integrity error
2022-03-15 15:34:07 -03:00
Jack Robison 90cfcaac40 fix 2022-03-15 15:34:07 -03:00
Jack Robison 02563db2c7 pretty print 2022-03-15 15:34:07 -03:00
Jack Robison cfa32a3986 fix duplicate activate 2022-03-15 15:34:07 -03:00
Jack Robison ab4a6bc5b8 fix updating resolve by effective amount after abandoning support 2022-03-15 15:34:07 -03:00
Jack Robison f01b8c849d use RevertableOpStack in _get_takeover_ops 2022-03-15 15:34:07 -03:00
Jack Robison 6a46f50a35 remove debug prints 2022-03-15 15:34:07 -03:00
Jack Robison 468ed91ee3 add RevertableOpStack to verify consistency of ops as they're staged 2022-03-15 15:34:07 -03:00
Jack Robison b856e2120a fix fee amount overflow in es 2022-03-15 15:34:07 -03:00
Jack Robison fd7bfbea78 faster claim producer
-make batches of claim txos from the iterator, and sort by tx hash before fetching to maximize cache and read ahead hits
2022-03-15 15:34:07 -03:00
Jack Robison a74434e269 imports 2022-03-15 15:34:07 -03:00
Jack Robison 92be63ff5b close db in sync script 2022-03-15 15:34:07 -03:00
Jack Robison 4ce16b6509 try default block size 2022-03-15 15:34:07 -03:00
Jack Robison 2341667d04 update elastic sync 2022-03-15 15:34:07 -03:00
Jack Robison 89e7c8582e invalidate channel signatures upon channel abandon 2022-03-15 15:34:07 -03:00
Jack Robison da4e4ecd23 _prepare_claim_for_sync generators 2022-03-15 15:34:07 -03:00
Jack Robison c85648d43b logging 2022-03-15 15:34:07 -03:00
Jack Robison c90331af98 use default sync=False during write_batch 2022-03-15 15:34:07 -03:00
Jack Robison e5461f6d4f genesis_bytes attribute 2022-03-15 15:34:07 -03:00
Jack Robison ac82d6b27b fix 2022-03-15 15:34:07 -03:00
Jack Robison 01db974040 disable es (revert) 2022-03-15 15:34:07 -03:00
Jack Robison 6ae6cf733d _cached_get_active_amount 2022-03-15 15:34:07 -03:00
Jack Robison ad1d25d945 fix removing unactivated support 2022-03-15 15:34:07 -03:00
Jack Robison 69ed47fc22 debug 2022-03-15 15:34:07 -03:00
Jack Robison 7c62654534 leveldb tuning 2022-03-15 15:34:07 -03:00
Jack Robison 531e6c1a61 debug 2022-03-15 15:34:07 -03:00
Jack Robison 066f797ad4 refactor ClaimToTXO prefix 2022-03-15 15:34:07 -03:00
Jack Robison 9f0611f3d9 filter abandoned claims from those considered for early activation 2022-03-15 15:34:07 -03:00
Jack Robison b0d2efd613 only do early takeover on a larger amount (fix case where they're equal) 2022-03-15 15:34:07 -03:00
Jack Robison 073283a433 filter supported claim hashes for claims that dont exist from early takeover/activations 2022-03-15 15:34:07 -03:00
Jack Robison 5be04448ea remove extra open functions 2022-03-15 15:34:07 -03:00
Jack Robison 5541b80179 fix flush id 2022-03-15 15:34:07 -03:00
Jack Robison 35232b3650 fix clearing pending_support caches upon abandon 2022-03-15 15:34:07 -03:00
Jack Robison f4ca3ea66b fix putting spent unactivated supports in removed_active_support 2022-03-15 15:34:07 -03:00
Jack Robison 306efa17cc fix syncing claim to es where channel is in the same block 2022-03-15 15:34:07 -03:00
Jack Robison 83107ad877 ignore activation for headless supports 2022-03-15 15:34:07 -03:00
Jack Robison b0cf25bf5c faster get_future_activated 2022-03-15 15:34:07 -03:00
Jack Robison 6330424bcd debugging 2022-03-15 15:34:07 -03:00
Jack Robison 5f3850bfa2 prints 2022-03-15 15:34:07 -03:00
Jack Robison 82e6658483 flush count 2022-03-15 15:34:07 -03:00
Jack Robison 49f4add8d1 tests 2022-03-15 15:34:07 -03:00
Jack Robison 6ea96e79bd reposts 2022-03-15 15:34:07 -03:00
Jack Robison 9ad31008a5 fix updating the ES search index
-update search index to use ResolveResult tuples
2022-03-15 15:34:07 -03:00
Jack Robison 966f47a5b1 bid ordered resolve, feed ES claim data from block processor 2022-03-15 15:34:07 -03:00
Jack Robison 8711ece274 fix duplicate update op for early activating claim 2022-03-15 15:34:07 -03:00
Jack Robison c0e2d56f55 fix early takeovers by not-yet activated claims 2022-03-15 15:34:07 -03:00
Jack Robison 73da5a35b8 test_early_takeover_abandoned_controlling_support 2022-03-15 15:34:07 -03:00
Jack Robison 2ba044ee4e require previous_winning arg for get_takeover_name_ops 2022-03-15 15:34:07 -03:00
Jack Robison d69180da71 fix takeover edge case
if a claim with a higher value than that of a claim taking over a name exists but isn't yet activated, activate it early and have it take over the name
2022-03-15 15:34:07 -03:00
Jack Robison b6e4cb9102 comments 2022-03-15 15:34:07 -03:00
Jack Robison 99d16fcb5a bid ordered resolve (WIP) 2022-03-15 15:34:07 -03:00
Jack Robison c8d0d765d1 fix udp ping test 2022-03-15 15:34:07 -03:00
Jack Robison 96c318ee88 DBError 2022-03-15 15:34:07 -03:00
Jack Robison c1ac4d8261 remove unused COIN file 2022-03-15 15:34:07 -03:00
Jack Robison e678df86e0 claim takeovers 2022-03-15 15:34:07 -03:00
Jack Robison 6aa124592d move get_expiration_height and claimtrie constants to Coin class 2022-03-15 15:34:07 -03:00
Jack Robison 86b6b860dc tests 2022-03-15 15:34:07 -03:00
Jack Robison 53ee3a5f80 claim activations and takeovers (WIP) 2022-03-15 15:34:07 -03:00
Jack Robison 8af410b184 advance_blocks -> advance_block 2022-03-15 15:34:07 -03:00
Jack Robison 8cb3fe8831 dead code 2022-03-15 15:34:07 -03:00
Jack Robison 2e92f3acad LBRYBlockProcessor -> BlockProcessor
- temporarily disable claim_search
2022-03-15 15:34:07 -03:00
Jack Robison 4e58094e4b rebase 2022-03-15 15:34:07 -03:00
Jack Robison 1445340bba tests 2022-03-15 15:34:07 -03:00
Jack Robison eb0eacd404 add wrapper for getnamesintrie
-used for verifying db state against lbrycrd
2022-03-15 15:34:07 -03:00
Jack Robison 1ff3ab4b83 db state struct
-remove dead code
2022-03-15 15:34:07 -03:00
Jack Robison 06841a4fde claim expiration 2022-03-15 15:34:07 -03:00
Jack Robison d57cd5acd7 get_claim_by_claim_id 2022-03-15 15:34:07 -03:00
Jack Robison b40cda78ee claims db
-move all leveldb prefixes to DB_PREFIXES enum
-add serializable RevertableOp interface for key/value puts and deletes
-resolve urls from leveldb
2022-03-15 15:34:07 -03:00
Jack Robison 0a833f5f83 transaction_num_mapping 2022-03-15 15:34:07 -03:00
Jack Robison ef3bab16d3 disable sqlite in block processor 2022-03-15 15:34:07 -03:00
Jack Robison 3e826d0a5d named tuples 2022-03-15 15:34:07 -03:00
Jack Robison bcd2c7d90b consolidate flush_backup 2022-03-15 15:34:07 -03:00
Jack Robison eda1b0b3fc remove lbry.wallet.server.history 2022-03-15 15:34:07 -03:00
Jack Robison bc7fe680c0 consolidate leveldb block advance/reorg
-move methods from History to LevelDB
2022-03-15 15:34:07 -03:00
Jack Robison 3ed748f2fd atomic flush_dbs 2022-03-15 15:34:07 -03:00
Jack Robison 9634753efd combine leveldb databases 2022-03-15 15:34:07 -03:00
Jack Robison 31df4f0bb5 Merkle staticmethods 2022-03-15 15:34:07 -03:00
Lex Berezhny 1673b8debc v0.105.0 2022-03-15 15:34:07 -03:00
Victor Shyba 3d7ece91eb fix file reflect and add test 2022-03-15 15:34:07 -03:00
Lex Berezhny 1004a83dae disk space metrics 2022-03-15 15:34:07 -03:00
Victor Shyba e7b5b82909 dht: use bytes hex/fromhex instead of binascii 2022-03-15 15:34:07 -03:00
Victor Shyba c17fddddcd add grin to dht known list 2022-03-15 15:34:07 -03:00
Victor Shyba 2664a34d52 add madiator to known dht nodes 2022-03-15 15:34:07 -03:00
Victor Shyba c9bf9691e3 add option to set bootstrap_node 2022-03-15 15:34:07 -03:00
Victor Shyba ad5e5fed82 add dockerfile for dht node 2022-03-15 15:34:07 -03:00
Victor Shyba 0e2fbe1c40 configure where to save peers 2022-03-15 15:34:07 -03:00
Victor Shyba 62e65d61f4 define arg types 2022-03-15 15:34:07 -03:00
Victor Shyba a237cbd963 add dht seed node script 2022-03-15 15:34:07 -03:00
Victor Shyba 78b12dc411 fix conflict with imported function 2022-03-15 15:34:07 -03:00
Victor Shyba 27457db5c3 errors for empyt and misssing file on publish 2022-03-15 15:34:07 -03:00
Victor Shyba e569fdd43c generalize stream empty to argument empty 2022-03-15 15:34:07 -03:00
Victor Shyba 00a850500d empty stream name error for user input 2022-03-15 15:34:07 -03:00
Victor Shyba 25aa2f95a5 error for missing channel private key 2022-03-15 15:34:07 -03:00
Victor Shyba dc2ccc4fe8 error for already purchased claims 2022-03-15 15:34:07 -03:00
belikor cd5f260061 api.json: correct the error in the generated documentation
From `"name": "blobs_in_stream<blobs_in_stream>"`
to `"name": "blobs_in_stream"`.
2022-03-15 15:34:06 -03:00
belikor c47ba498a4 daemon: fix documentation in the file_list docstring
This is necessary to produce the `docs/api.json`
(through `scripts/generate_json_api.py`)
with correct information, and to be able to parse this file later on
by other tools.
2022-03-15 15:34:06 -03:00
Lex Berezhny a6bf8e0eb7 v0.104.0 2022-03-15 15:34:06 -03:00
Lex Berezhny 930d4b3acf re-enable coveralls 2022-03-15 15:34:06 -03:00
Lex Berezhny c481838179 ubuntu 16.04 is deprecated on github actions, upgrading to 18.04 2022-03-15 15:34:06 -03:00
Lex Berezhny 963649998a revert release 2022-03-15 15:34:06 -03:00
Lex Berezhny dc4b950e8f v0.104.0 2022-03-15 15:34:06 -03:00
Lex Berezhny 5fea68a9b9 default is_mine to true during migration 2022-03-15 15:34:06 -03:00
Lex Berezhny 568a7ae16a coveralls still down, will have to merged with coveralls off 2022-03-15 15:34:06 -03:00
Lex Berezhny f69d47587f tests 2022-03-15 15:34:06 -03:00
Lex Berezhny 7148767b6f lint 2022-03-15 15:34:06 -03:00
Lex Berezhny c66b1646a6 during disk clean your own sd blob is now kept and file status of deleted files is set to stopped 2022-03-15 15:34:06 -03:00
Lex Berezhny bbcdc881cd db migration and other fixes 2022-03-15 15:34:06 -03:00
Lex Berezhny 07a78cf73d use databse to track blob disk space use and preserve own blobs 2022-03-15 15:34:06 -03:00
Lex Berezhny 93ac2e3bc9 v0.103.0 2022-03-15 15:34:06 -03:00
Lex Berezhny 7e9614b8d1 omit just node.py 2022-03-15 15:34:06 -03:00
Lex Berezhny 7d704f966b coverage omit fix 2022-03-15 15:34:06 -03:00
Lex Berezhny 43a2c6515d omit coverage inside tox 2022-03-15 15:34:06 -03:00
99 changed files with 12632 additions and 2714 deletions

No files matched your search

+1 -1
View File
@@ -48,7 +48,7 @@ jobs:
uses: ASzc/change-string-case-action@v1
with:
string: ${{ runner.os }}
- run: python -m pip install --user --upgrade pip wheel
- run: pip install --user --upgrade pip wheel
- if: startsWith(runner.os, 'linux')
run: pip install -e .[torrent,test]
- if: startsWith(runner.os, 'linux')
+1 -1
View File
@@ -1,6 +1,6 @@
The MIT License (MIT)
Copyright (c) 2015-2022 LBRY Inc
Copyright (c) 2015-2020 LBRY Inc
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish,
+1 -1
View File
@@ -1,2 +1,2 @@
__version__ = "0.109.0"
__version__ = "0.106.0"
version = tuple(map(int, __version__.split('.'))) # pylint: disable=invalid-name
+1 -9
View File
@@ -1,7 +1,6 @@
import asyncio
import binascii
import logging
import socket
import typing
from json.decoder import JSONDecodeError
from lbry.blob_exchange.serialization import BlobResponse, BlobRequest, blob_response_types
@@ -138,7 +137,7 @@ class BlobServerProtocol(asyncio.Protocol):
try:
request = BlobRequest.deserialize(self.buf + data)
self.buf = remainder
except (UnicodeDecodeError, JSONDecodeError):
except JSONDecodeError:
log.error("request from %s is not valid json (%i bytes): %s", self.peer_address_and_port,
len(self.buf + data), '' if not data else binascii.hexlify(self.buf + data).decode())
self.close()
@@ -168,13 +167,6 @@ class BlobServer:
raise Exception("already running")
async def _start_server():
# checking if the port is in use
# thx https://stackoverflow.com/a/52872579
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
if s.connect_ex(('localhost', port)) == 0:
# the port is already in use!
log.error("Failed to bind TCP %s:%d", interface, port)
server = await self.loop.create_server(
lambda: self.server_protocol_class(self.loop, self.blob_manager, self.lbrycrd_address,
self.idle_timeout, self.transfer_timeout),
-5
View File
@@ -681,11 +681,6 @@ class Config(CLIConfig):
('cdn.reflector.lbry.com', 5567)
])
tracker_servers = Servers("BitTorrent-compatible (BEP15) UDP trackers for helping P2P discovery", [
('tracker.lbry.com', 9252),
('tracker.lbry.grin.io', 9252),
])
lbryum_servers = Servers("SPV wallet servers", [
('spv11.lbry.com', 50001),
('spv12.lbry.com', 50001),
+33 -34
View File
@@ -5,7 +5,7 @@ import socket
from prometheus_client import Gauge
from lbry.utils import aclosing, resolve_host
from lbry.utils import resolve_host
from lbry.dht import constants
from lbry.dht.peer import make_kademlia_peer
from lbry.dht.protocol.distance import Distance
@@ -217,10 +217,9 @@ class Node:
shortlist: typing.Optional[typing.List['KademliaPeer']] = None
) -> typing.List['KademliaPeer']:
peers = []
async with aclosing(self.get_iterative_node_finder(
node_id, shortlist=shortlist, max_results=max_results)) as node_finder:
async for iteration_peers in node_finder:
peers.extend(iteration_peers)
async for iteration_peers in self.get_iterative_node_finder(
node_id, shortlist=shortlist, max_results=max_results):
peers.extend(iteration_peers)
distance = Distance(node_id)
peers.sort(key=lambda peer: distance(peer.node_id))
return peers[:count]
@@ -246,36 +245,36 @@ class Node:
# prioritize peers who reply to a dht ping first
# this minimizes attempting to make tcp connections that won't work later to dead or unreachable peers
async with aclosing(self.get_iterative_value_finder(bytes.fromhex(blob_hash))) as value_finder:
async for results in value_finder:
to_put = []
for peer in results:
if peer.address == self.protocol.external_ip and self.protocol.peer_port == peer.tcp_port:
continue
is_good = self.protocol.peer_manager.peer_is_good(peer)
if is_good:
# the peer has replied recently over UDP, it can probably be reached on the TCP port
to_put.append(peer)
elif is_good is None:
if not peer.udp_port:
# TODO: use the same port for TCP and UDP
# the udp port must be guessed
# default to the ports being the same. if the TCP port appears to be <=0.48.0 default,
# including on a network with several nodes, then assume the udp port is proportionately
# based on a starting port of 4444
udp_port_to_try = peer.tcp_port
if 3400 > peer.tcp_port > 3332:
udp_port_to_try = (peer.tcp_port - 3333) + 4444
self.loop.create_task(put_into_result_queue_after_pong(
make_kademlia_peer(peer.node_id, peer.address, udp_port_to_try, peer.tcp_port)
))
else:
self.loop.create_task(put_into_result_queue_after_pong(peer))
async for results in self.get_iterative_value_finder(bytes.fromhex(blob_hash)):
to_put = []
for peer in results:
if peer.address == self.protocol.external_ip and self.protocol.peer_port == peer.tcp_port:
continue
is_good = self.protocol.peer_manager.peer_is_good(peer)
if is_good:
# the peer has replied recently over UDP, it can probably be reached on the TCP port
to_put.append(peer)
elif is_good is None:
if not peer.udp_port:
# TODO: use the same port for TCP and UDP
# the udp port must be guessed
# default to the ports being the same. if the TCP port appears to be <=0.48.0 default,
# including on a network with several nodes, then assume the udp port is proportionately
# based on a starting port of 4444
udp_port_to_try = peer.tcp_port
if 3400 > peer.tcp_port > 3332:
udp_port_to_try = (peer.tcp_port - 3333) + 4444
self.loop.create_task(put_into_result_queue_after_pong(
make_kademlia_peer(peer.node_id, peer.address, udp_port_to_try, peer.tcp_port)
))
else:
# the peer is known to be bad/unreachable, skip trying to connect to it over TCP
log.debug("skip bad peer %s:%i for %s", peer.address, peer.tcp_port, blob_hash)
if to_put:
result_queue.put_nowait(to_put)
self.loop.create_task(put_into_result_queue_after_pong(peer))
else:
# the peer is known to be bad/unreachable, skip trying to connect to it over TCP
log.debug("skip bad peer %s:%i for %s", peer.address, peer.tcp_port, blob_hash)
if to_put:
result_queue.put_nowait(to_put)
def accumulate_peers(self, search_queue: asyncio.Queue,
peer_queue: typing.Optional[asyncio.Queue] = None
+12 -38
View File
@@ -1,7 +1,6 @@
import asyncio
from itertools import chain
from collections import defaultdict, OrderedDict
from collections.abc import AsyncIterator
import typing
import logging
from typing import TYPE_CHECKING
@@ -72,7 +71,7 @@ def get_shortlist(routing_table: 'TreeRoutingTable', key: bytes,
return shortlist or routing_table.find_close_peers(key)
class IterativeFinder(AsyncIterator):
class IterativeFinder:
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
max_results: typing.Optional[int] = constants.K,
@@ -152,7 +151,7 @@ class IterativeFinder(AsyncIterator):
log.warning("misbehaving peer %s:%i returned peer with reserved ip %s:%i", peer.address,
peer.udp_port, address, udp_port)
self.check_result_ready(response)
self._log_state(reason="check result")
self._log_state()
def _reset_closest(self, peer):
if peer in self.active:
@@ -164,17 +163,12 @@ class IterativeFinder(AsyncIterator):
except asyncio.TimeoutError:
self._reset_closest(peer)
return
except asyncio.CancelledError:
log.debug("%s[%x] cancelled probe",
type(self).__name__, id(self))
raise
except ValueError as err:
log.warning(str(err))
self._reset_closest(peer)
return
except TransportNotConnected:
await self._aclose(reason="not connected")
return
return self.aclose()
except RemoteException:
self._reset_closest(peer)
return
@@ -188,9 +182,7 @@ class IterativeFinder(AsyncIterator):
added = 0
for index, peer in enumerate(self.active.keys()):
if index == 0:
log.debug("%s[%x] closest to probe: %s",
type(self).__name__, id(self),
peer.node_id.hex()[:8])
log.debug("closest to probe: %s", peer.node_id.hex()[:8])
if peer in self.contacted:
continue
if len(self.running_probes) >= constants.ALPHA:
@@ -206,13 +198,9 @@ class IterativeFinder(AsyncIterator):
continue
self._schedule_probe(peer)
added += 1
log.debug("%s[%x] running %d probes for key %s",
type(self).__name__, id(self),
len(self.running_probes), self.key.hex()[:8])
log.debug("running %d probes for key %s", len(self.running_probes), self.key.hex()[:8])
if not added and not self.running_probes:
log.debug("%s[%x] search for %s exhausted",
type(self).__name__, id(self),
self.key.hex()[:8])
log.debug("search for %s exhausted", self.key.hex()[:8])
self.search_exhausted()
def _schedule_probe(self, peer: 'KademliaPeer'):
@@ -228,11 +216,9 @@ class IterativeFinder(AsyncIterator):
t.add_done_callback(callback)
self.running_probes[peer] = t
def _log_state(self, reason="?"):
log.debug("%s[%x] [%s] %s: %i active nodes %i contacted %i produced %i queued",
type(self).__name__, id(self), self.key.hex()[:8],
reason, len(self.active), len(self.contacted),
self.iteration_count, self.iteration_queue.qsize())
def _log_state(self):
log.debug("[%s] check result: %i active nodes %i contacted",
self.key.hex()[:8], len(self.active), len(self.contacted))
def __aiter__(self):
if self.running:
@@ -251,18 +237,11 @@ class IterativeFinder(AsyncIterator):
raise StopAsyncIteration
self.iteration_count += 1
return result
except asyncio.CancelledError:
await self._aclose(reason="cancelled")
raise
except StopAsyncIteration:
await self._aclose(reason="no more results")
except (asyncio.CancelledError, StopAsyncIteration):
self.loop.call_soon(self.aclose)
raise
async def _aclose(self, reason="?"):
log.debug("%s[%x] [%s] shutdown because %s: %i active nodes %i contacted %i produced %i queued",
type(self).__name__, id(self), self.key.hex()[:8],
reason, len(self.active), len(self.contacted),
self.iteration_count, self.iteration_queue.qsize())
def aclose(self):
self.running = False
self.iteration_queue.put_nowait(None)
for task in chain(self.tasks, self.running_probes.values()):
@@ -270,11 +249,6 @@ class IterativeFinder(AsyncIterator):
self.tasks.clear()
self.running_probes.clear()
async def aclose(self):
if self.running:
await self._aclose(reason="aclose")
log.debug("%s[%x] [%s] async close completed",
type(self).__name__, id(self), self.key.hex()[:8])
class IterativeNodeFinder(IterativeFinder):
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
-49
View File
@@ -27,8 +27,6 @@ from lbry.extras.daemon.storage import SQLiteStorage
from lbry.torrent.torrent_manager import TorrentManager
from lbry.wallet import WalletManager
from lbry.wallet.usage_payment import WalletServerPayer
from lbry.torrent.tracker import TrackerClient
try:
from lbry.torrent.session import TorrentSession
except ImportError:
@@ -50,7 +48,6 @@ BACKGROUND_DOWNLOADER_COMPONENT = "background_downloader"
PEER_PROTOCOL_SERVER_COMPONENT = "peer_protocol_server"
UPNP_COMPONENT = "upnp"
EXCHANGE_RATE_MANAGER_COMPONENT = "exchange_rate_manager"
TRACKER_ANNOUNCER_COMPONENT = "tracker_announcer_component"
LIBTORRENT_COMPONENT = "libtorrent_component"
@@ -711,49 +708,3 @@ class ExchangeRateManagerComponent(Component):
async def stop(self):
self.exchange_rate_manager.stop()
class TrackerAnnouncerComponent(Component):
component_name = TRACKER_ANNOUNCER_COMPONENT
depends_on = [FILE_MANAGER_COMPONENT]
def __init__(self, component_manager):
super().__init__(component_manager)
self.file_manager = None
self.announce_task = None
self.tracker_client: typing.Optional[TrackerClient] = None
@property
def component(self):
return self.tracker_client
@property
def running(self):
return self._running and self.announce_task and not self.announce_task.done()
async def announce_forever(self):
while True:
sleep_seconds = 60.0
announce_sd_hashes = []
for file in self.file_manager.get_filtered():
if not file.downloader:
continue
announce_sd_hashes.append(bytes.fromhex(file.sd_hash))
await self.tracker_client.announce_many(*announce_sd_hashes)
await asyncio.sleep(sleep_seconds)
async def start(self):
node = self.component_manager.get_component(DHT_COMPONENT) \
if self.component_manager.has_component(DHT_COMPONENT) else None
node_id = node.protocol.node_id if node else None
self.tracker_client = TrackerClient(node_id, self.conf.tcp_port, lambda: self.conf.tracker_servers)
await self.tracker_client.start()
self.file_manager = self.component_manager.get_component(FILE_MANAGER_COMPONENT)
self.announce_task = asyncio.create_task(self.announce_forever())
async def stop(self):
self.file_manager = None
if self.announce_task and not self.announce_task.done():
self.announce_task.cancel()
self.announce_task = None
self.tracker_client.stop()
+25 -105
View File
@@ -8,6 +8,7 @@ import time
import inspect
import typing
import random
import hashlib
import tracemalloc
from decimal import Decimal
from urllib.parse import urlencode, quote
@@ -16,6 +17,7 @@ from binascii import hexlify, unhexlify
from traceback import format_exc
from functools import wraps, partial
import ecdsa
import base58
from aiohttp import web
from prometheus_client import generate_latest as prom_generate_latest, Gauge, Histogram, Counter
@@ -27,8 +29,6 @@ from lbry.wallet import (
)
from lbry.wallet.dewies import dewies_to_lbc, lbc_to_dewies, dict_values_to_lbc
from lbry.wallet.constants import TXO_TYPES, CLAIM_TYPE_NAMES
from lbry.wallet.bip32 import PrivateKey
from lbry.crypto.base58 import Base58
from lbry import utils
from lbry.conf import Config, Setting, NOT_SET
@@ -44,7 +44,7 @@ from lbry.error import (
from lbry.extras import system_info
from lbry.extras.daemon import analytics
from lbry.extras.daemon.components import WALLET_COMPONENT, DATABASE_COMPONENT, DHT_COMPONENT, BLOB_COMPONENT
from lbry.extras.daemon.components import FILE_MANAGER_COMPONENT, DISK_SPACE_COMPONENT, TRACKER_ANNOUNCER_COMPONENT
from lbry.extras.daemon.components import FILE_MANAGER_COMPONENT, DISK_SPACE_COMPONENT
from lbry.extras.daemon.components import EXCHANGE_RATE_MANAGER_COMPONENT, UPNP_COMPONENT
from lbry.extras.daemon.componentmanager import RequiredCondition
from lbry.extras.daemon.componentmanager import ComponentManager
@@ -54,7 +54,8 @@ from lbry.extras.daemon.security import ensure_request_allowed
from lbry.file_analysis import VideoFileAnalyzer
from lbry.schema.claim import Claim
from lbry.schema.url import URL, normalize_name
from lbry.wallet.server.db.elasticsearch.constants import RANGE_FIELDS, REPLACEMENTS
MY_RANGE_FIELDS = RANGE_FIELDS - {"limit_claims_per_channel"}
if typing.TYPE_CHECKING:
from lbry.blob.blob_manager import BlobManager
@@ -67,29 +68,6 @@ if typing.TYPE_CHECKING:
log = logging.getLogger(__name__)
RANGE_FIELDS = {
'height', 'creation_height', 'activation_height', 'expiration_height',
'timestamp', 'creation_timestamp', 'duration', 'release_time', 'fee_amount',
'tx_position', 'repost_count', 'limit_claims_per_channel',
'amount', 'effective_amount', 'support_amount',
'trending_score', 'censor_type', 'tx_num'
}
MY_RANGE_FIELDS = RANGE_FIELDS - {"limit_claims_per_channel"}
REPLACEMENTS = {
'claim_name': 'normalized_name',
'name': 'normalized_name',
'txid': 'tx_id',
'nout': 'tx_nout',
'trending_group': 'trending_score',
'trending_mixed': 'trending_score',
'trending_global': 'trending_score',
'trending_local': 'trending_score',
'reposted': 'repost_count',
'stream_types': 'stream_type',
'media_types': 'media_type',
'valid_channel_signature': 'is_signature_valid'
}
def is_transactional_function(name):
for action in ('create', 'update', 'abandon', 'send', 'fund'):
@@ -1873,48 +1851,6 @@ class Daemon(metaclass=JSONRPCServerType):
outputs=outputs, broadcast=broadcast
)
@requires("wallet")
async def jsonrpc_account_deposit(
self, txid, nout, redeem_script, private_key,
to_account=None, wallet_id=None, preview=False, blocking=False
):
"""
Spend a time locked transaction into your account.
Usage:
account_deposit <txid> <nout> <redeem_script> <private_key>
[<to_account> | --to_account=<to_account>]
[--wallet_id=<wallet_id>] [--preview] [--blocking]
Options:
--txid=<txid> : (str) id of the transaction
--nout=<nout> : (int) output number in the transaction
--redeem_script=<redeem_script> : (str) redeem script for output
--private_key=<private_key> : (str) private key to sign transaction
--to_account=<to_account> : (str) deposit to this account
--wallet_id=<wallet_id> : (str) limit operation to specific wallet.
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until tx has synced
Returns: {Transaction}
"""
wallet = self.wallet_manager.get_wallet_or_default(wallet_id)
account = wallet.get_account_or_default(to_account)
other_tx = await self.wallet_manager.get_transaction(txid)
tx = await Transaction.spend_time_lock(
other_tx.outputs[nout], unhexlify(redeem_script), account
)
pk = PrivateKey.from_bytes(
account.ledger, Base58.decode_check(private_key)[1:-1]
)
await tx.sign([account], {pk.address: pk})
if not preview:
await self.broadcast_or_release(tx, blocking)
self.component_manager.loop.create_task(self.analytics_manager.send_credits_sent())
else:
await self.ledger.release_tx(tx)
return tx
@requires(WALLET_COMPONENT)
def jsonrpc_account_send(self, amount, addresses, account_id=None, wallet_id=None, preview=False, blocking=False):
"""
@@ -2771,13 +2707,12 @@ class Daemon(metaclass=JSONRPCServerType):
name, claim, amount, claim_address, funding_accounts, funding_accounts[0]
)
txo = tx.outputs[0]
txo.set_channel_private_key(
await funding_accounts[0].generate_channel_private_key()
)
await txo.generate_channel_private_key()
await tx.sign(funding_accounts)
if not preview:
account.add_channel_private_key(txo.private_key)
wallet.save()
await self.broadcast_or_release(tx, blocking)
self.component_manager.loop.create_task(self.storage.save_claims([self._old_get_temp_claim_info(
@@ -2926,9 +2861,7 @@ class Daemon(metaclass=JSONRPCServerType):
new_txo = tx.outputs[0]
if new_signing_key:
new_txo.set_channel_private_key(
await funding_accounts[0].generate_channel_private_key()
)
await new_txo.generate_channel_private_key()
else:
new_txo.private_key = old_txo.private_key
@@ -2937,6 +2870,7 @@ class Daemon(metaclass=JSONRPCServerType):
await tx.sign(funding_accounts)
if not preview:
account.add_channel_private_key(new_txo.private_key)
wallet.save()
await self.broadcast_or_release(tx, blocking)
self.component_manager.loop.create_task(self.storage.save_claims([self._old_get_temp_claim_info(
@@ -3108,7 +3042,7 @@ class Daemon(metaclass=JSONRPCServerType):
'channel_id': channel.claim_id,
'holding_address': address,
'holding_public_key': public_key.extended_key_string(),
'signing_private_key': channel.private_key.signing_key.to_pem().decode()
'signing_private_key': channel.private_key.to_pem().decode()
}
return base58.b58encode(json.dumps(export, separators=(',', ':')))
@@ -3131,14 +3065,15 @@ class Daemon(metaclass=JSONRPCServerType):
decoded = base58.b58decode(channel_data)
data = json.loads(decoded)
channel_private_key = PrivateKey.from_pem(
self.ledger, data['signing_private_key']
channel_private_key = ecdsa.SigningKey.from_pem(
data['signing_private_key'], hashfunc=hashlib.sha256
)
public_key_der = channel_private_key.get_verifying_key().to_der()
# check that the holding_address hasn't changed since the export was made
holding_address = data['holding_address']
channels, _, _, _ = await self.ledger.claim_search(
wallet.accounts, public_key_id=channel_private_key.address
wallet.accounts, public_key_id=self.ledger.public_key_to_address(public_key_der)
)
if channels and channels[0].get_address(self.ledger) != holding_address:
holding_address = channels[0].get_address(self.ledger)
@@ -3301,17 +3236,15 @@ class Daemon(metaclass=JSONRPCServerType):
)
@requires(WALLET_COMPONENT, FILE_MANAGER_COMPONENT, BLOB_COMPONENT, DATABASE_COMPONENT)
async def jsonrpc_stream_repost(
self, name, bid, claim_id, allow_duplicate_name=False, channel_id=None,
channel_name=None, channel_account_id=None, account_id=None, wallet_id=None,
claim_address=None, funding_account_ids=None, preview=False, blocking=False, **kwargs):
async def jsonrpc_stream_repost(self, name, bid, claim_id, allow_duplicate_name=False, channel_id=None,
channel_name=None, channel_account_id=None, account_id=None, wallet_id=None,
claim_address=None, funding_account_ids=None, preview=False, blocking=False):
"""
Creates a claim that references an existing stream by its claim id.
Usage:
stream_repost (<name> | --name=<name>) (<bid> | --bid=<bid>) (<claim_id> | --claim_id=<claim_id>)
[--allow_duplicate_name=<allow_duplicate_name>]
[--title=<title>] [--description=<description>] [--tags=<tags>...]
[--channel_id=<channel_id> | --channel_name=<channel_name>]
[--channel_account_id=<channel_account_id>...]
[--account_id=<account_id>] [--wallet_id=<wallet_id>]
@@ -3324,9 +3257,6 @@ class Daemon(metaclass=JSONRPCServerType):
--claim_id=<claim_id> : (str) id of the claim being reposted
--allow_duplicate_name=<allow_duplicate_name> : (bool) create new claim even if one already exists with
given name. default: false.
--title=<title> : (str) title of the repost
--description=<description> : (str) description of the repost
--tags=<tags> : (list) add repost tags
--channel_id=<channel_id> : (str) claim id of the publisher channel
--channel_name=<channel_name> : (str) name of the publisher channel
--channel_account_id=<channel_account_id>: (str) one or more account ids for accounts to look in
@@ -3361,7 +3291,6 @@ class Daemon(metaclass=JSONRPCServerType):
raise Exception('Invalid claim id. It is expected to be a 40 characters long hexadecimal string.')
claim = Claim()
claim.repost.update(**kwargs)
claim.repost.reference.claim_id = claim_id
tx = await Transaction.claim_create(
name, claim, amount, claim_address, funding_accounts, funding_accounts[0], channel
@@ -3742,8 +3671,6 @@ class Daemon(metaclass=JSONRPCServerType):
if old_txo.claim.is_stream:
claim.stream.update(file_path=file_path, **kwargs)
elif old_txo.claim.is_repost:
claim.repost.update(**kwargs)
if clear_channel:
claim.clear_signature()
@@ -4957,6 +4884,7 @@ class Daemon(metaclass=JSONRPCServerType):
DHT / Blob Exchange peer commands.
"""
@requires(DHT_COMPONENT)
async def jsonrpc_peer_list(self, blob_hash, page=None, page_size=None):
"""
Get peers for blob hash
@@ -4978,29 +4906,21 @@ class Daemon(metaclass=JSONRPCServerType):
if not is_valid_blobhash(blob_hash):
# TODO: use error from lbry.error
raise Exception("invalid blob hash")
peer_q = asyncio.Queue(loop=self.component_manager.loop)
if self.component_manager.has_component(TRACKER_ANNOUNCER_COMPONENT):
tracker = self.component_manager.get_component(TRACKER_ANNOUNCER_COMPONENT)
tracker_peers = await tracker.get_kademlia_peer_list(bytes.fromhex(blob_hash))
log.info("Found %d peers for %s from trackers.", len(tracker_peers), blob_hash[:8])
peer_q.put_nowait(tracker_peers)
elif not self.component_manager.has_component(DHT_COMPONENT):
raise Exception("Peer list needs, at least, either a DHT component or a Tracker component for discovery.")
peers = []
if self.component_manager.has_component(DHT_COMPONENT):
await self.dht_node._peers_for_value_producer(blob_hash, peer_q)
peer_q = asyncio.Queue(loop=self.component_manager.loop)
await self.dht_node._peers_for_value_producer(blob_hash, peer_q)
while not peer_q.empty():
peers.extend(peer_q.get_nowait())
results = {
(peer.address, peer.tcp_port): {
"node_id": hexlify(peer.node_id).decode() if peer.node_id else None,
results = [
{
"node_id": hexlify(peer.node_id).decode(),
"address": peer.address,
"udp_port": peer.udp_port,
"tcp_port": peer.tcp_port,
}
for peer in peers
}
return paginate_list(list(results.values()), page, page_size)
]
return paginate_list(results, page, page_size)
@requires(DATABASE_COMPONENT)
async def jsonrpc_blob_announce(self, blob_hash=None, stream_hash=None, sd_hash=None):
+2 -2
View File
@@ -10,7 +10,7 @@ from lbry.schema.claim import Claim
from lbry.schema.support import Support
from lbry.torrent.torrent_manager import TorrentSource
from lbry.wallet import Wallet, Ledger, Account, Transaction, Output
from lbry.wallet.bip32 import PublicKey
from lbry.wallet.bip32 import PubKey
from lbry.wallet.dewies import dewies_to_lbc
from lbry.stream.managed_stream import ManagedStream
@@ -138,7 +138,7 @@ class JSONResponseEncoder(JSONEncoder):
return self.encode_claim(obj)
if isinstance(obj, Support):
return obj.to_dict()
if isinstance(obj, PublicKey):
if isinstance(obj, PubKey):
return obj.extended_key_string()
if isinstance(obj, datetime):
return obj.strftime("%Y%m%dT%H:%M:%S")
+3 -5
View File
@@ -449,7 +449,7 @@ class SQLiteStorage(SQLiteMixin):
return await self.db.execute_fetchall(
"select blob.blob_hash, blob.blob_length, blob.added_on "
"from blob left join stream_blob using (blob_hash) "
"where stream_blob.stream_hash is null and blob.is_mine=? and blob.status='finished'"
"where stream_blob.stream_hash is null and blob.is_mine=? "
"order by blob.blob_length desc, blob.added_on asc",
(is_mine,)
)
@@ -463,8 +463,7 @@ class SQLiteStorage(SQLiteMixin):
content_blobs = await self.db.execute_fetchall(
"select blob.blob_hash, blob.blob_length, blob.added_on "
"from blob join stream_blob using (blob_hash) cross join stream using (stream_hash)"
"cross join file using (stream_hash)"
"where blob.is_mine=? and blob.status='finished' order by blob.added_on asc, blob.blob_length asc",
"cross join file using (stream_hash) where blob.is_mine=? order by blob.added_on asc, blob.blob_length asc",
(is_mine,)
)
return content_blobs + sd_blobs
@@ -481,8 +480,7 @@ class SQLiteStorage(SQLiteMixin):
coalesce(sum(case when
is_mine=1
then blob_length else 0 end), 0) as private_storage
from blob left join stream_blob using (blob_hash)
where blob_hash not in (select sd_hash from stream) and blob.status="finished"
from blob left join stream_blob using (blob_hash) where blob_hash not in (select sd_hash from stream)
""")
return {
'network_storage': network_size,
-1
View File
@@ -45,7 +45,6 @@ class ManagedDownloadSource:
self.purchase_receipt = None
self._added_on = added_on
self.analytics_manager = analytics_manager
self.downloader = None
self.saving = asyncio.Event(loop=self.loop)
self.finished_writing = asyncio.Event(loop=self.loop)
-1
View File
@@ -2,5 +2,4 @@ build:
rm types/v2/* -rf
touch types/v2/__init__.py
cd types/v2/ && protoc --python_out=. -I ../../../../../types/v2/proto/ ../../../../../types/v2/proto/*.proto
cd types/v2/ && cp ../../../../../types/jsonschema/* ./
sed -e 's/^import\ \(.*\)_pb2\ /from . import\ \1_pb2\ /g' -i types/v2/*.py
+2 -9
View File
@@ -2,9 +2,6 @@ import logging
from typing import List
from binascii import hexlify, unhexlify
from asn1crypto.keys import PublicKeyInfo
from coincurve import PublicKey as cPublicKey
from google.protobuf.json_format import MessageToDict
from google.protobuf.message import DecodeError
from hachoir.core.log import log as hachoir_log
@@ -349,7 +346,7 @@ class Channel(BaseClaim):
@property
def public_key(self) -> str:
return hexlify(self.public_key_bytes).decode()
return hexlify(self.message.public_key).decode()
@public_key.setter
def public_key(self, sd_public_key: str):
@@ -357,11 +354,7 @@ class Channel(BaseClaim):
@property
def public_key_bytes(self) -> bytes:
if len(self.message.public_key) == 33:
return self.message.public_key
public_key_info = PublicKeyInfo.load(self.message.public_key)
public_key = cPublicKey(public_key_info.native['public_key'])
return public_key.format(compressed=True)
return self.message.public_key
@public_key_bytes.setter
def public_key_bytes(self, public_key: bytes):
+3 -26
View File
@@ -1,11 +1,13 @@
import base64
from typing import List, Union, Optional, NamedTuple
from typing import List, TYPE_CHECKING, Union, Optional
from binascii import hexlify
from itertools import chain
from lbry.error import ResolveCensoredError
from lbry.schema.types.v2.result_pb2 import Outputs as OutputsMessage
from lbry.schema.types.v2.result_pb2 import Error as ErrorMessage
if TYPE_CHECKING:
from lbry.wallet.server.leveldb import ResolveResult
INVALID = ErrorMessage.Code.Name(ErrorMessage.INVALID)
NOT_FOUND = ErrorMessage.Code.Name(ErrorMessage.NOT_FOUND)
@@ -22,31 +24,6 @@ def set_reference(reference, claim_hash, rows):
return
class ResolveResult(NamedTuple):
name: str
normalized_name: str
claim_hash: bytes
tx_num: int
position: int
tx_hash: bytes
height: int
amount: int
short_url: str
is_controlling: bool
canonical_url: str
creation_height: int
activation_height: int
expiration_height: int
effective_amount: int
support_amount: int
reposted: int
last_takeover_height: Optional[int]
claims_in_channel: Optional[int]
channel_hash: Optional[bytes]
reposted_claim_hash: Optional[bytes]
signature_valid: Optional[bool]
class Censor:
NOT_CENSORED = 0
-139
View File
@@ -1,139 +0,0 @@
{
"title": "Wallet",
"description": "An LBC wallet",
"type": "object",
"required": ["name", "version", "accounts", "preferences"],
"additionalProperties": false,
"properties": {
"name": {
"description": "Human readable name for this wallet",
"type": "string"
},
"version": {
"description": "Wallet spec version",
"type": "integer",
"$comment": "Should this be a string? We may need some sort of decimal type if we want exact decimal versions."
},
"accounts": {
"description": "Accounts associated with this wallet",
"type": "array",
"items": {
"type": "object",
"required": ["address_generator", "certificates", "encrypted", "ledger", "modified_on", "name", "private_key", "public_key", "seed"],
"additionalProperties": false,
"properties": {
"address_generator": {
"description": "Higher level manager of either singular or deterministically generated addresses",
"type": "object",
"oneOf": [
{
"required": ["name", "change", "receiving"],
"additionalProperties": false,
"properties": {
"name": {
"description": "type of address generator: a deterministic chain of addresses",
"enum": ["deterministic-chain"],
"type": "string"
},
"change": {
"$ref": "#/$defs/address_manager",
"description": "Manager for deterministically generated change address (not used for single address)"
},
"receiving": {
"$ref": "#/$defs/address_manager",
"description": "Manager for deterministically generated receiving address (not used for single address)"
}
}
}, {
"required": ["name"],
"additionalProperties": false,
"properties": {
"name": {
"description": "type of address generator: a single address",
"enum": ["single-address"],
"type": "string"
}
}
}
]
},
"certificates": {
"type": "object",
"description": "Channel keys. Mapping from public key address to pem-formatted private key.",
"additionalProperties": {"type": "string"}
},
"encrypted": {
"type": "boolean",
"description": "Whether private key and seed are encrypted with a password"
},
"ledger": {
"description": "Which network to use",
"type": "string",
"examples": [
"lbc_mainnet",
"lbc_testnet"
]
},
"modified_on": {
"description": "last modified time in Unix Time",
"type": "integer"
},
"name": {
"description": "Name for account, possibly human readable",
"type": "string"
},
"private_key": {
"description": "Private key for address if `address_generator` is a single address. Root of chain of private keys for addresses if `address_generator` is a deterministic chain of addresses. Encrypted if `encrypted` is true.",
"type": "string"
},
"public_key": {
"description": "Public key for address if `address_generator` is a single address. Root of chain of public keys for addresses if `address_generator` is a deterministic chain of addresses.",
"type": "string"
},
"seed": {
"description": "Human readable representation of `private_key`. encrypted if `encrypted` is set to `true`",
"type": "string"
}
}
}
},
"preferences": {
"description": "Timestamped application-level preferences. Values can be objects or of a primitive type.",
"$comment": "enable-sync is seen in example wallet. encrypt-on-disk is seen in example wallet. they both have a boolean `value` field. Do we want them explicitly defined here? local and shared seem to have at least a similar structure (type, value [yes, again], version), value being the free-form part. Should we define those here? Or can there be any key under preferences, and `value` be literally be anything in any form?",
"type": "object",
"additionalProperties": {
"type": "object",
"required": ["ts", "value"],
"additionalProperties": false,
"properties": {
"ts": {
"type": "number",
"description": "When the item was set, in Unix time format.",
"$comment": "Do we want a string (decimal)?"
},
"value": {
"$comment": "Sometimes this has been an object, sometimes just a boolean. I don't want to prescribe anything."
}
}
}
}
},
"$defs": {
"address_manager": {
"description": "Manager for deterministically generated addresses",
"type": "object",
"required": ["gap", "maximum_uses_per_address"],
"additionalProperties": false,
"properties": {
"gap": {
"description": "Maximum allowed consecutive generated addresses with no transactions",
"type": "integer"
},
"maximum_uses_per_address": {
"description": "Maximum number of uses for each generated address",
"type": "integer"
}
}
}
}
}
-3
View File
@@ -8,8 +8,6 @@ from lbry.error import DownloadSDTimeoutError
from lbry.utils import lru_cache_concurrent
from lbry.stream.descriptor import StreamDescriptor
from lbry.blob_exchange.downloader import BlobDownloader
from lbry.torrent.tracker import enqueue_tracker_search
if typing.TYPE_CHECKING:
from lbry.conf import Config
from lbry.dht.node import Node
@@ -93,7 +91,6 @@ class StreamDownloader:
self.accumulate_task.cancel()
_, self.accumulate_task = self.node.accumulate_peers(self.search_queue, self.peer_queue)
await self.add_fixed_peers()
enqueue_tracker_search(bytes.fromhex(self.sd_hash), self.peer_queue)
# start searching for peers for the sd hash
self.search_queue.put_nowait(self.sd_hash)
log.info("searching for peers for stream %s", self.sd_hash)
+2
View File
@@ -16,8 +16,10 @@ from lbry.file.source import ManagedDownloadSource
if typing.TYPE_CHECKING:
from lbry.conf import Config
from lbry.schema.claim import Claim
from lbry.blob.blob_manager import BlobManager
from lbry.blob.blob_info import BlobInfo
from lbry.dht.node import Node
from lbry.extras.daemon.analytics import AnalyticsManager
from lbry.wallet.transaction import Transaction
+21 -81
View File
@@ -17,10 +17,8 @@ from functools import partial
from lbry.wallet import WalletManager, Wallet, Ledger, Account, Transaction
from lbry.conf import Config
from lbry.wallet.util import satoshis_to_coins
from lbry.wallet.dewies import lbc_to_dewies
from lbry.wallet.orchstr8 import Conductor
from lbry.wallet.orchstr8.node import LBCWalletNode, WalletNode, HubNode
from lbry.schema.claim import Claim
from lbry.wallet.orchstr8.node import BlockchainNode, WalletNode, HubNode
from lbry.extras.daemon.daemon import Daemon, jsonrpc_dumps_pretty
from lbry.extras.daemon.components import Component, WalletComponent
@@ -236,7 +234,7 @@ class IntegrationTestCase(AsyncioTestCase):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.conductor: Optional[Conductor] = None
self.blockchain: Optional[LBCWalletNode] = None
self.blockchain: Optional[BlockchainNode] = None
self.hub: Optional[HubNode] = None
self.wallet_node: Optional[WalletNode] = None
self.manager: Optional[WalletManager] = None
@@ -246,17 +244,15 @@ class IntegrationTestCase(AsyncioTestCase):
async def asyncSetUp(self):
self.conductor = Conductor(seed=self.SEED)
await self.conductor.start_lbcd()
self.addCleanup(self.conductor.stop_lbcd)
await self.conductor.start_lbcwallet()
self.addCleanup(self.conductor.stop_lbcwallet)
await self.conductor.start_blockchain()
self.addCleanup(self.conductor.stop_blockchain)
await self.conductor.start_spv()
self.addCleanup(self.conductor.stop_spv)
await self.conductor.start_wallet()
self.addCleanup(self.conductor.stop_wallet)
await self.conductor.start_hub()
self.addCleanup(self.conductor.stop_hub)
self.blockchain = self.conductor.lbcwallet_node
self.blockchain = self.conductor.blockchain_node
self.hub = self.conductor.hub_node
self.wallet_node = self.conductor.wallet_node
self.manager = self.wallet_node.manager
@@ -271,13 +267,6 @@ class IntegrationTestCase(AsyncioTestCase):
def broadcast(self, tx):
return self.ledger.broadcast(tx)
async def broadcast_and_confirm(self, tx, ledger=None):
ledger = ledger or self.ledger
notifications = asyncio.create_task(ledger.wait(tx))
await ledger.broadcast(tx)
await notifications
await self.generate_and_wait(1, [tx.id], ledger)
async def on_header(self, height):
if self.ledger.headers.height < height:
await self.ledger.on_header.where(
@@ -285,36 +274,11 @@ class IntegrationTestCase(AsyncioTestCase):
)
return True
async def send_to_address_and_wait(self, address, amount, blocks_to_generate=0, ledger=None):
tx_watch = []
txid = None
done = False
watcher = (ledger or self.ledger).on_transaction.where(
lambda e: e.tx.id == txid or done or tx_watch.append(e.tx.id)
def on_transaction_id(self, txid, ledger=None):
return (ledger or self.ledger).on_transaction.where(
lambda e: e.tx.id == txid
)
txid = await self.blockchain.send_to_address(address, amount)
done = txid in tx_watch
await watcher
await self.generate_and_wait(blocks_to_generate, [txid], ledger)
return txid
async def generate_and_wait(self, blocks_to_generate, txids, ledger=None):
if blocks_to_generate > 0:
watcher = (ledger or self.ledger).on_transaction.where(
lambda e: ((e.tx.id in txids and txids.remove(e.tx.id)), len(txids) <= 0)[-1] # multi-statement lambda
)
self.conductor.spv_node.server.synchronized.clear()
await self.blockchain.generate(blocks_to_generate)
height = self.blockchain.block_expected
await watcher
while True:
await self.conductor.spv_node.server.synchronized.wait()
self.conductor.spv_node.server.synchronized.clear()
if self.conductor.spv_node.server.db.db_height >= height:
break
def on_address_update(self, address):
return self.ledger.on_transaction.where(
lambda e: e.address == address
@@ -325,19 +289,6 @@ class IntegrationTestCase(AsyncioTestCase):
lambda e: e.tx.id == tx.id and e.address == address
)
async def generate(self, blocks):
""" Ask lbrycrd to generate some blocks and wait until ledger has them. """
prepare = self.ledger.on_header.where(self.blockchain.is_expected_block)
height = self.blockchain.block_expected
self.conductor.spv_node.server.synchronized.clear()
await self.blockchain.generate(blocks)
await prepare # no guarantee that it didn't happen already, so start waiting from before calling generate
while True:
await self.conductor.spv_node.server.synchronized.wait()
self.conductor.spv_node.server.synchronized.clear()
if self.conductor.spv_node.server.db.db_height >= height:
break
class FakeExchangeRateManager(ExchangeRateManager):
@@ -398,19 +349,20 @@ class CommandTestCase(IntegrationTestCase):
self.skip_libtorrent = True
async def asyncSetUp(self):
await super().asyncSetUp()
logging.getLogger('lbry.blob_exchange').setLevel(self.VERBOSITY)
logging.getLogger('lbry.daemon').setLevel(self.VERBOSITY)
logging.getLogger('lbry.stream').setLevel(self.VERBOSITY)
logging.getLogger('lbry.wallet').setLevel(self.VERBOSITY)
await super().asyncSetUp()
self.daemon = await self.add_daemon(self.wallet_node)
await self.account.ensure_address_gap()
address = (await self.account.receiving.get_addresses(limit=1, only_usable=True))[0]
await self.send_to_address_and_wait(address, 10, 6)
sendtxid = await self.blockchain.send_to_address(address, 10)
await self.confirm_tx(sendtxid)
await self.generate(5)
server_tmp_dir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, server_tmp_dir)
@@ -507,14 +459,9 @@ class CommandTestCase(IntegrationTestCase):
async def confirm_tx(self, txid, ledger=None):
""" Wait for tx to be in mempool, then generate a block, wait for tx to be in a block. """
# await (ledger or self.ledger).on_transaction.where(lambda e: e.tx.id == txid)
on_tx = (ledger or self.ledger).on_transaction.where(lambda e: e.tx.id == txid)
await asyncio.wait([self.generate(1), on_tx], timeout=5)
# # actually, if it's in the mempool or in the block we're fine
# await self.generate_and_wait(1, [txid], ledger=ledger)
# return txid
await self.on_transaction_id(txid, ledger)
await self.generate(1)
await self.on_transaction_id(txid, ledger)
return txid
async def on_transaction_dict(self, tx):
@@ -529,6 +476,12 @@ class CommandTestCase(IntegrationTestCase):
addresses.add(txo['address'])
return list(addresses)
async def generate(self, blocks):
""" Ask lbrycrd to generate some blocks and wait until ledger has them. """
prepare = self.ledger.on_header.where(self.blockchain.is_expected_block)
await self.blockchain.generate(blocks)
await prepare # no guarantee that it didn't happen already, so start waiting from before calling generate
async def blockchain_claim_name(self, name: str, value: str, amount: str, confirm=True):
txid = await self.blockchain._cli_cmnd('claimname', name, value, amount)
if confirm:
@@ -559,19 +512,6 @@ class CommandTestCase(IntegrationTestCase):
return self.sout(tx)
return tx
async def create_nondeterministic_channel(self, name, price, pubkey_bytes, daemon=None, blocking=False):
account = (daemon or self.daemon).wallet_manager.default_account
claim_address = await account.receiving.get_or_create_usable_address()
claim = Claim()
claim.channel.public_key_bytes = pubkey_bytes
tx = await Transaction.claim_create(
name, claim, lbc_to_dewies(price),
claim_address, [self.account], self.account
)
await tx.sign([self.account])
await (daemon or self.daemon).broadcast_or_release(tx, blocking)
return self.sout(tx)
def create_upload_file(self, data, prefix=None, suffix=None):
file_path = tempfile.mktemp(prefix=prefix or "tmp", suffix=suffix or "", dir=self.daemon.conf.upload_dir)
with open(file_path, 'w+b') as file:
-285
View File
@@ -1,285 +0,0 @@
import random
import socket
import string
import struct
import asyncio
import logging
import time
import ipaddress
from collections import namedtuple
from functools import reduce
from typing import Optional
from lbry.dht.node import get_kademlia_peers_from_hosts
from lbry.utils import resolve_host, async_timed_cache, cache_concurrent
from lbry.wallet.stream import StreamController
from lbry import version
log = logging.getLogger(__name__)
CONNECTION_EXPIRES_AFTER_SECONDS = 50
PREFIX = 'LB' # todo: PR BEP20 to add ourselves
DEFAULT_TIMEOUT_SECONDS = 10.0
DEFAULT_CONCURRENCY_LIMIT = 100
# see: http://bittorrent.org/beps/bep_0015.html and http://xbtt.sourceforge.net/udp_tracker_protocol.html
ConnectRequest = namedtuple("ConnectRequest", ["connection_id", "action", "transaction_id"])
ConnectResponse = namedtuple("ConnectResponse", ["action", "transaction_id", "connection_id"])
AnnounceRequest = namedtuple("AnnounceRequest",
["connection_id", "action", "transaction_id", "info_hash", "peer_id", "downloaded", "left",
"uploaded", "event", "ip_addr", "key", "num_want", "port"])
AnnounceResponse = namedtuple("AnnounceResponse",
["action", "transaction_id", "interval", "leechers", "seeders", "peers"])
CompactIPv4Peer = namedtuple("CompactPeer", ["address", "port"])
ScrapeRequest = namedtuple("ScrapeRequest", ["connection_id", "action", "transaction_id", "infohashes"])
ScrapeResponse = namedtuple("ScrapeResponse", ["action", "transaction_id", "items"])
ScrapeResponseItem = namedtuple("ScrapeResponseItem", ["seeders", "completed", "leechers"])
ErrorResponse = namedtuple("ErrorResponse", ["action", "transaction_id", "message"])
structs = {
ConnectRequest: struct.Struct(">QII"),
ConnectResponse: struct.Struct(">IIQ"),
AnnounceRequest: struct.Struct(">QII20s20sQQQIIIiH"),
AnnounceResponse: struct.Struct(">IIIII"),
CompactIPv4Peer: struct.Struct(">IH"),
ScrapeRequest: struct.Struct(">QII"),
ScrapeResponse: struct.Struct(">II"),
ScrapeResponseItem: struct.Struct(">III"),
ErrorResponse: struct.Struct(">II")
}
def decode(cls, data, offset=0):
decoder = structs[cls]
if cls is AnnounceResponse:
return AnnounceResponse(*decoder.unpack_from(data, offset),
peers=[decode(CompactIPv4Peer, data, index) for index in range(20, len(data), 6)])
elif cls is ScrapeResponse:
return ScrapeResponse(*decoder.unpack_from(data, offset),
items=[decode(ScrapeResponseItem, data, index) for index in range(8, len(data), 12)])
elif cls is ErrorResponse:
return ErrorResponse(*decoder.unpack_from(data, offset), data[decoder.size:])
return cls(*decoder.unpack_from(data, offset))
def encode(obj):
if isinstance(obj, ScrapeRequest):
return structs[ScrapeRequest].pack(*obj[:-1]) + b''.join(obj.infohashes)
elif isinstance(obj, ErrorResponse):
return structs[ErrorResponse].pack(*obj[:-1]) + obj.message
elif isinstance(obj, AnnounceResponse):
return structs[AnnounceResponse].pack(*obj[:-1]) + b''.join([encode(peer) for peer in obj.peers])
return structs[type(obj)].pack(*obj)
def make_peer_id(random_part: Optional[str] = None) -> bytes:
# see https://wiki.theory.org/BitTorrentSpecification#peer_id and https://www.bittorrent.org/beps/bep_0020.html
# not to confuse with node id; peer id identifies uniquely the software, version and instance
random_part = random_part or ''.join(random.choice(string.ascii_letters) for _ in range(20))
return f"{PREFIX}-{'-'.join(map(str, version))}-{random_part}"[:20].encode()
class UDPTrackerClientProtocol(asyncio.DatagramProtocol):
def __init__(self, timeout: float = DEFAULT_TIMEOUT_SECONDS):
self.transport = None
self.data_queue = {}
self.timeout = timeout
self.semaphore = asyncio.Semaphore(DEFAULT_CONCURRENCY_LIMIT)
def connection_made(self, transport: asyncio.DatagramTransport) -> None:
self.transport = transport
async def request(self, obj, tracker_ip, tracker_port):
self.data_queue[obj.transaction_id] = asyncio.get_running_loop().create_future()
try:
async with self.semaphore:
self.transport.sendto(encode(obj), (tracker_ip, tracker_port))
return await asyncio.wait_for(self.data_queue[obj.transaction_id], self.timeout)
finally:
self.data_queue.pop(obj.transaction_id, None)
async def connect(self, tracker_ip, tracker_port):
transaction_id = random.getrandbits(32)
return decode(ConnectResponse,
await self.request(ConnectRequest(0x41727101980, 0, transaction_id), tracker_ip, tracker_port))
@cache_concurrent
@async_timed_cache(CONNECTION_EXPIRES_AFTER_SECONDS)
async def ensure_connection_id(self, peer_id, tracker_ip, tracker_port):
# peer_id is just to ensure cache coherency
return (await self.connect(tracker_ip, tracker_port)).connection_id
async def announce(self, info_hash, peer_id, port, tracker_ip, tracker_port, stopped=False):
connection_id = await self.ensure_connection_id(peer_id, tracker_ip, tracker_port)
# this should make the key deterministic but unique per info hash + peer id
key = int.from_bytes(info_hash[:4], "big") ^ int.from_bytes(peer_id[:4], "big") ^ port
transaction_id = random.getrandbits(32)
req = AnnounceRequest(
connection_id, 1, transaction_id, info_hash, peer_id, 0, 0, 0, 3 if stopped else 1, 0, key, -1, port)
return decode(AnnounceResponse, await self.request(req, tracker_ip, tracker_port))
async def scrape(self, infohashes, tracker_ip, tracker_port, connection_id=None):
connection_id = await self.ensure_connection_id(None, tracker_ip, tracker_port)
transaction_id = random.getrandbits(32)
reply = await self.request(
ScrapeRequest(connection_id, 2, transaction_id, infohashes), tracker_ip, tracker_port)
return decode(ScrapeResponse, reply), connection_id
def datagram_received(self, data: bytes, addr: (str, int)) -> None:
if len(data) < 8:
return
transaction_id = int.from_bytes(data[4:8], byteorder="big", signed=False)
if transaction_id in self.data_queue:
if not self.data_queue[transaction_id].done():
if data[3] == 3:
return self.data_queue[transaction_id].set_exception(Exception(decode(ErrorResponse, data).message))
return self.data_queue[transaction_id].set_result(data)
log.debug("unexpected packet (can be a response for a previously timed out request): %s", data.hex())
def connection_lost(self, exc: Exception = None) -> None:
self.transport = None
class TrackerClient:
event_controller = StreamController()
def __init__(self, node_id, announce_port, get_servers, timeout=10.0):
self.client = UDPTrackerClientProtocol(timeout=timeout)
self.transport = None
self.peer_id = make_peer_id(node_id.hex() if node_id else None)
self.announce_port = announce_port
self._get_servers = get_servers
self.results = {} # we can't probe the server before the interval, so we keep the result here until it expires
self.tasks = {}
async def start(self):
self.transport, _ = await asyncio.get_running_loop().create_datagram_endpoint(
lambda: self.client, local_addr=("0.0.0.0", 0))
self.event_controller.stream.listen(
lambda request: self.on_hash(request[1], request[2]) if request[0] == 'search' else None)
def stop(self):
while self.tasks:
self.tasks.popitem()[1].cancel()
if self.transport is not None:
self.transport.close()
self.client = None
self.transport = None
self.event_controller.close()
def on_hash(self, info_hash, on_announcement=None):
if info_hash not in self.tasks:
task = asyncio.create_task(self.get_peer_list(info_hash, on_announcement=on_announcement))
task.add_done_callback(lambda *_: self.tasks.pop(info_hash, None))
self.tasks[info_hash] = task
async def announce_many(self, *info_hashes, stopped=False):
await asyncio.gather(
*[self._announce_many(server, info_hashes, stopped=stopped) for server in self._get_servers()],
return_exceptions=True)
async def _announce_many(self, server, info_hashes, stopped=False):
tracker_ip = await resolve_host(*server, 'udp')
still_good_info_hashes = {
info_hash for (info_hash, (next_announcement, _)) in self.results.get(tracker_ip, {}).items()
if time.time() < next_announcement
}
results = await asyncio.gather(
*[self._probe_server(info_hash, tracker_ip, server[1], stopped=stopped)
for info_hash in info_hashes if info_hash not in still_good_info_hashes],
return_exceptions=True)
if results:
errors = sum([1 for result in results if result is None or isinstance(result, Exception)])
log.info("Tracker: finished announcing %d files to %s:%d, %d errors", len(results), *server, errors)
async def get_peer_list(self, info_hash, stopped=False, on_announcement=None, no_port=False):
found = []
probes = [self._probe_server(info_hash, *server, stopped, no_port) for server in self._get_servers()]
for done in asyncio.as_completed(probes):
result = await done
if result is not None:
await asyncio.gather(*filter(asyncio.iscoroutine, [on_announcement(result)] if on_announcement else []))
found.append(result)
return found
async def get_kademlia_peer_list(self, info_hash):
responses = await self.get_peer_list(info_hash, no_port=True)
return await announcement_to_kademlia_peers(*responses)
async def _probe_server(self, info_hash, tracker_host, tracker_port, stopped=False, no_port=False):
result = None
try:
tracker_host = await resolve_host(tracker_host, tracker_port, 'udp')
except socket.error:
log.warning("DNS failure while resolving tracker host: %s, skipping.", tracker_host)
return
self.results.setdefault(tracker_host, {})
if info_hash in self.results[tracker_host]:
next_announcement, result = self.results[tracker_host][info_hash]
if time.time() < next_announcement:
return result
try:
result = await self.client.announce(
info_hash, self.peer_id, 0 if no_port else self.announce_port, tracker_host, tracker_port, stopped)
self.results[tracker_host][info_hash] = (time.time() + result.interval, result)
except asyncio.TimeoutError: # todo: this is UDP, timeout is common, we need a better metric for failures
self.results[tracker_host][info_hash] = (time.time() + 60.0, result)
log.debug("Tracker timed out: %s:%d", tracker_host, tracker_port)
return None
log.debug("Announced: %s found %d peers for %s", tracker_host, len(result.peers), info_hash.hex()[:8])
return result
def enqueue_tracker_search(info_hash: bytes, peer_q: asyncio.Queue):
async def on_announcement(announcement: AnnounceResponse):
peers = await announcement_to_kademlia_peers(announcement)
log.info("Found %d peers from tracker for %s", len(peers), info_hash.hex()[:8])
peer_q.put_nowait(peers)
TrackerClient.event_controller.add(('search', info_hash, on_announcement))
def announcement_to_kademlia_peers(*announcements: AnnounceResponse):
peers = [
(str(ipaddress.ip_address(peer.address)), peer.port)
for announcement in announcements for peer in announcement.peers if peer.port > 1024 # no privileged or 0
]
return get_kademlia_peers_from_hosts(peers)
class UDPTrackerServerProtocol(asyncio.DatagramProtocol): # for testing. Not suitable for production
def __init__(self):
self.transport = None
self.known_conns = set()
self.peers = {}
def connection_made(self, transport: asyncio.DatagramTransport) -> None:
self.transport = transport
def add_peer(self, info_hash, ip_address: str, port: int):
self.peers.setdefault(info_hash, [])
self.peers[info_hash].append(encode_peer(ip_address, port))
def datagram_received(self, data: bytes, addr: (str, int)) -> None:
if len(data) < 16:
return
action = int.from_bytes(data[8:12], "big", signed=False)
if action == 0:
req = decode(ConnectRequest, data)
connection_id = random.getrandbits(32)
self.known_conns.add(connection_id)
return self.transport.sendto(encode(ConnectResponse(0, req.transaction_id, connection_id)), addr)
elif action == 1:
req = decode(AnnounceRequest, data)
if req.connection_id not in self.known_conns:
resp = encode(ErrorResponse(3, req.transaction_id, b'Connection ID missmatch.\x00'))
else:
compact_address = encode_peer(addr[0], req.port)
if req.event != 3:
self.add_peer(req.info_hash, addr[0], req.port)
elif compact_address in self.peers.get(req.info_hash, []):
self.peers[req.info_hash].remove(compact_address)
peers = [decode(CompactIPv4Peer, peer) for peer in self.peers[req.info_hash]]
resp = encode(AnnounceResponse(1, req.transaction_id, 1700, 0, len(peers), peers))
return self.transport.sendto(resp, addr)
def encode_peer(ip_address: str, port: int):
compact_ip = reduce(lambda buff, x: buff + bytearray([int(x)]), ip_address.split('.'), bytearray())
return compact_ip + port.to_bytes(2, "big", signed=False)
+16 -11
View File
@@ -130,16 +130,21 @@ def get_sd_hash(stream_info):
def json_dumps_pretty(obj, **kwargs):
return json.dumps(obj, sort_keys=True, indent=2, separators=(',', ': '), **kwargs)
try:
# the standard contextlib.aclosing() is available in 3.10+
from contextlib import aclosing # pylint: disable=unused-import
except ImportError:
@contextlib.asynccontextmanager
async def aclosing(thing):
try:
yield thing
finally:
await thing.aclose()
def cancel_task(task: typing.Optional[asyncio.Task]):
if task and not task.done():
task.cancel()
def cancel_tasks(tasks: typing.List[typing.Optional[asyncio.Task]]):
for task in tasks:
cancel_task(task)
def drain_tasks(tasks: typing.List[typing.Optional[asyncio.Task]]):
while tasks:
cancel_task(tasks.pop())
def async_timed_cache(duration: int):
def wrapper(func):
@@ -400,7 +405,7 @@ async def fallback_get_external_ip(): # used if spv servers can't be used for i
async def _get_external_ip(default_servers) -> typing.Tuple[typing.Optional[str], typing.Optional[str]]:
# used if upnp is disabled or non-functioning
from lbry.wallet.udp import SPVStatusClientProtocol # pylint: disable=C0415
from lbry.wallet.server.udp import SPVStatusClientProtocol # pylint: disable=C0415
hostname_to_ip = {}
ip_to_hostnames = collections.defaultdict(list)
+14 -20
View File
@@ -1,23 +1,17 @@
__lbcd__ = 'lbcd'
__lbcctl__ = 'lbcctl'
__lbcwallet__ = 'lbcwallet'
__lbcd_url__ = (
'https://github.com/lbryio/lbcd/releases/download/' +
'v0.22.100-rc.0/lbcd_0.22.100-rc.0_TARGET_PLATFORM.tar.gz'
)
__lbcwallet_url__ = (
'https://github.com/lbryio/lbcwallet/releases/download/' +
'v0.13.100-alpha.0/lbcwallet_0.13.100-alpha.0_TARGET_PLATFORM.tar.gz'
__node_daemon__ = 'lbrycrdd'
__node_cli__ = 'lbrycrd-cli'
__node_bin__ = ''
__node_url__ = (
'https://github.com/lbryio/lbrycrd/releases/download/v0.17.4.6/lbrycrd-linux-1746.zip'
)
__spvserver__ = 'lbry.wallet.server.coin.LBCRegTest'
from lbry.wallet.wallet import Wallet, WalletStorage, TimestampedPreferences, ENCRYPT_ON_DISK
from lbry.wallet.manager import WalletManager
from lbry.wallet.network import Network
from lbry.wallet.ledger import Ledger, RegTestLedger, TestNetLedger, BlockHeightEvent
from lbry.wallet.account import Account, AddressManager, SingleKey, HierarchicalDeterministic, \
DeterministicChannelKeyManager
from lbry.wallet.transaction import Transaction, Output, Input
from lbry.wallet.script import OutputScript, InputScript
from lbry.wallet.database import SQLiteMixin, Database
from lbry.wallet.header import Headers
from .wallet import Wallet, WalletStorage, TimestampedPreferences, ENCRYPT_ON_DISK
from .manager import WalletManager
from .network import Network
from .ledger import Ledger, RegTestLedger, TestNetLedger, BlockHeightEvent
from .account import Account, AddressManager, SingleKey, HierarchicalDeterministic
from .transaction import Transaction, Output, Input
from .script import OutputScript, InputScript
from .database import SQLiteMixin, Database
from .header import Headers
+33 -70
View File
@@ -9,10 +9,11 @@ from hashlib import sha256
from string import hexdigits
from typing import Type, Dict, Tuple, Optional, Any, List
import ecdsa
from lbry.error import InvalidPasswordError
from lbry.crypto.crypt import aes_encrypt, aes_decrypt
from .bip32 import PrivateKey, PublicKey, KeyPath, from_extended_key_string
from .bip32 import PrivateKey, PubKey, from_extended_key_string
from .mnemonic import Mnemonic
from .constants import COIN, TXO_TYPES
from .transaction import Transaction, Input, Output
@@ -33,49 +34,6 @@ def validate_claim_id(claim_id):
raise Exception("Claim id is not hex encoded")
class DeterministicChannelKeyManager:
def __init__(self, account: 'Account'):
self.account = account
self.last_known = 0
self.cache = {}
self._private_key: Optional[PrivateKey] = None
@property
def private_key(self):
if self._private_key is None:
if self.account.private_key is not None:
self._private_key = self.account.private_key.child(KeyPath.CHANNEL)
return self._private_key
def maybe_generate_deterministic_key_for_channel(self, txo):
if self.private_key is None:
return
next_private_key = self.private_key.child(self.last_known)
public_key = next_private_key.public_key
public_key_bytes = public_key.pubkey_bytes
if txo.claim.channel.public_key_bytes == public_key_bytes:
self.cache[public_key.address] = next_private_key
self.last_known += 1
async def ensure_cache_primed(self):
if self.private_key is not None:
await self.generate_next_key()
async def generate_next_key(self) -> PrivateKey:
db = self.account.ledger.db
while True:
next_private_key = self.private_key.child(self.last_known)
public_key = next_private_key.public_key
self.cache[public_key.address] = next_private_key
if not await db.is_channel_key_used(self.account, public_key):
return next_private_key
self.last_known += 1
def get_private_key_from_pubkey_hash(self, pubkey_hash) -> PrivateKey:
return self.cache.get(pubkey_hash)
class AddressManager:
name: str
@@ -121,7 +79,7 @@ class AddressManager:
def get_private_key(self, index: int) -> PrivateKey:
raise NotImplementedError
def get_public_key(self, index: int) -> PublicKey:
def get_public_key(self, index: int) -> PubKey:
raise NotImplementedError
async def get_max_gap(self):
@@ -161,8 +119,8 @@ class HierarchicalDeterministic(AddressManager):
@classmethod
def from_dict(cls, account: 'Account', d: dict) -> Tuple[AddressManager, AddressManager]:
return (
cls(account, KeyPath.RECEIVE, **d.get('receiving', {'gap': 20, 'maximum_uses_per_address': 1})),
cls(account, KeyPath.CHANGE, **d.get('change', {'gap': 6, 'maximum_uses_per_address': 1}))
cls(account, 0, **d.get('receiving', {'gap': 20, 'maximum_uses_per_address': 1})),
cls(account, 1, **d.get('change', {'gap': 6, 'maximum_uses_per_address': 1}))
)
def merge(self, d: dict):
@@ -175,7 +133,7 @@ class HierarchicalDeterministic(AddressManager):
def get_private_key(self, index: int) -> PrivateKey:
return self.account.private_key.child(self.chain_number).child(index)
def get_public_key(self, index: int) -> PublicKey:
def get_public_key(self, index: int) -> PubKey:
return self.account.public_key.child(self.chain_number).child(index)
async def get_max_gap(self) -> int:
@@ -235,7 +193,7 @@ class SingleKey(AddressManager):
@classmethod
def from_dict(cls, account: 'Account', d: dict) \
-> Tuple[AddressManager, AddressManager]:
same_address_manager = cls(account, account.public_key, KeyPath.RECEIVE)
same_address_manager = cls(account, account.public_key, 0)
return same_address_manager, same_address_manager
def to_dict_instance(self):
@@ -244,7 +202,7 @@ class SingleKey(AddressManager):
def get_private_key(self, index: int) -> PrivateKey:
return self.account.private_key
def get_public_key(self, index: int) -> PublicKey:
def get_public_key(self, index: int) -> PubKey:
return self.account.public_key
async def get_max_gap(self) -> int:
@@ -266,6 +224,9 @@ class SingleKey(AddressManager):
class Account:
mnemonic_class = Mnemonic
private_key_class = PrivateKey
public_key_class = PubKey
address_generators: Dict[str, Type[AddressManager]] = {
SingleKey.name: SingleKey,
HierarchicalDeterministic.name: HierarchicalDeterministic,
@@ -273,7 +234,7 @@ class Account:
def __init__(self, ledger: 'Ledger', wallet: 'Wallet', name: str,
seed: str, private_key_string: str, encrypted: bool,
private_key: Optional[PrivateKey], public_key: PublicKey,
private_key: Optional[PrivateKey], public_key: PubKey,
address_generator: dict, modified_on: float, channel_keys: dict) -> None:
self.ledger = ledger
self.wallet = wallet
@@ -284,14 +245,13 @@ class Account:
self.private_key_string = private_key_string
self.init_vectors: Dict[str, bytes] = {}
self.encrypted = encrypted
self.private_key: Optional[PrivateKey] = private_key
self.public_key: PublicKey = public_key
self.private_key = private_key
self.public_key = public_key
generator_name = address_generator.get('name', HierarchicalDeterministic.name)
self.address_generator = self.address_generators[generator_name]
self.receiving, self.change = self.address_generator.from_dict(self, address_generator)
self.address_managers = {am.chain_number: am for am in (self.receiving, self.change)}
self.channel_keys = channel_keys
self.deterministic_channel_keys = DeterministicChannelKeyManager(self)
ledger.add_account(self)
wallet.add_account(self)
@@ -306,19 +266,19 @@ class Account:
name: str = None, address_generator: dict = None):
return cls.from_dict(ledger, wallet, {
'name': name,
'seed': Mnemonic().make_seed(),
'seed': cls.mnemonic_class().make_seed(),
'address_generator': address_generator or {}
})
@classmethod
def get_private_key_from_seed(cls, ledger: 'Ledger', seed: str, password: str):
return PrivateKey.from_seed(
ledger, Mnemonic.mnemonic_to_seed(seed, password or 'lbryum')
return cls.private_key_class.from_seed(
ledger, cls.mnemonic_class.mnemonic_to_seed(seed, password or 'lbryum')
)
@classmethod
def keys_from_dict(cls, ledger: 'Ledger', d: dict) \
-> Tuple[str, Optional[PrivateKey], PublicKey]:
-> Tuple[str, Optional[PrivateKey], PubKey]:
seed = d.get('seed', '')
private_key_string = d.get('private_key', '')
private_key = None
@@ -489,7 +449,7 @@ class Account:
assert not self.encrypted, "Cannot get private key on encrypted wallet account."
return self.address_managers[chain].get_private_key(index)
def get_public_key(self, chain: int, index: int) -> PublicKey:
def get_public_key(self, chain: int, index: int) -> PubKey:
return self.address_managers[chain].get_public_key(index)
def get_balance(self, confirmations=0, include_claims=False, read_only=False, **constraints):
@@ -560,30 +520,33 @@ class Account:
return tx
async def generate_channel_private_key(self):
return await self.deterministic_channel_keys.generate_next_key()
def add_channel_private_key(self, private_key):
public_key_bytes = private_key.get_verifying_key().to_der()
channel_pubkey_hash = self.ledger.public_key_to_address(public_key_bytes)
self.channel_keys[channel_pubkey_hash] = private_key.to_pem().decode()
def add_channel_private_key(self, private_key: PrivateKey):
self.channel_keys[private_key.address] = private_key.to_pem().decode()
async def get_channel_private_key(self, public_key_bytes) -> PrivateKey:
async def get_channel_private_key(self, public_key_bytes):
channel_pubkey_hash = self.ledger.public_key_to_address(public_key_bytes)
private_key_pem = self.channel_keys.get(channel_pubkey_hash)
if private_key_pem:
return PrivateKey.from_pem(self.ledger, private_key_pem)
return self.deterministic_channel_keys.get_private_key_from_pubkey_hash(channel_pubkey_hash)
return await asyncio.get_event_loop().run_in_executor(
None, ecdsa.SigningKey.from_pem, private_key_pem, sha256
)
async def maybe_migrate_certificates(self):
def to_der(private_key_pem):
return ecdsa.SigningKey.from_pem(private_key_pem, hashfunc=sha256).get_verifying_key().to_der()
if not self.channel_keys:
return
channel_keys = {}
for private_key_pem in self.channel_keys.values():
if not isinstance(private_key_pem, str):
continue
if not private_key_pem.startswith("-----BEGIN"):
if "-----BEGIN EC PRIVATE KEY-----" not in private_key_pem:
continue
private_key = PrivateKey.from_pem(self.ledger, private_key_pem)
channel_keys[private_key.address] = private_key_pem
public_key_der = await asyncio.get_event_loop().run_in_executor(None, to_der, private_key_pem)
channel_keys[self.ledger.public_key_to_address(public_key_der)] = private_key_pem
if self.channel_keys != channel_keys:
self.channel_keys = channel_keys
self.wallet.save()
+17 -104
View File
@@ -1,21 +1,10 @@
from asn1crypto.keys import PrivateKeyInfo, ECPrivateKey
from coincurve import PublicKey as cPublicKey, PrivateKey as cPrivateKey
from coincurve.utils import (
pem_to_der, lib as libsecp256k1, ffi as libsecp256k1_ffi
)
from coincurve.ecdsa import CDATA_SIG_LENGTH
from coincurve import PublicKey, PrivateKey as _PrivateKey
from lbry.crypto.hash import hmac_sha512, hash160, double_sha256
from lbry.crypto.base58 import Base58
from .util import cachedproperty
class KeyPath:
RECEIVE = 0
CHANGE = 1
CHANNEL = 2
class DerivationError(Exception):
""" Raised when an invalid derivation occurs. """
@@ -82,30 +71,26 @@ class _KeyBase:
return Base58.encode_check(self.extended_key())
class PublicKey(_KeyBase):
class PubKey(_KeyBase):
""" A BIP32 public key. """
def __init__(self, ledger, pubkey, chain_code, n, depth, parent=None):
super().__init__(ledger, chain_code, n, depth, parent)
if isinstance(pubkey, cPublicKey):
if isinstance(pubkey, PublicKey):
self.verifying_key = pubkey
else:
self.verifying_key = self._verifying_key_from_pubkey(pubkey)
@classmethod
def from_compressed(cls, public_key_bytes, ledger=None) -> 'PublicKey':
return cls(ledger, public_key_bytes, bytes((0,)*32), 0, 0)
@classmethod
def _verifying_key_from_pubkey(cls, pubkey):
""" Converts a 33-byte compressed pubkey into an coincurve.PublicKey object. """
""" Converts a 33-byte compressed pubkey into an PublicKey object. """
if not isinstance(pubkey, (bytes, bytearray)):
raise TypeError('pubkey must be raw bytes')
if len(pubkey) != 33:
raise ValueError('pubkey must be 33 bytes')
if pubkey[0] not in (2, 3):
raise ValueError('invalid pubkey prefix byte')
return cPublicKey(pubkey)
return PublicKey(pubkey)
@cachedproperty
def pubkey_bytes(self):
@@ -120,7 +105,7 @@ class PublicKey(_KeyBase):
def ec_point(self):
return self.verifying_key.point()
def child(self, n: int) -> 'PublicKey':
def child(self, n: int):
""" Return the derived child extended pubkey at index N. """
if not 0 <= n < (1 << 31):
raise ValueError('invalid BIP32 public key child number')
@@ -128,7 +113,7 @@ class PublicKey(_KeyBase):
msg = self.pubkey_bytes + n.to_bytes(4, 'big')
L_b, R_b = self._hmac_sha512(msg) # pylint: disable=invalid-name
derived_key = self.verifying_key.add(L_b)
return PublicKey(self.ledger, derived_key, R_b, n, self.depth + 1, self)
return PubKey(self.ledger, derived_key, R_b, n, self.depth + 1, self)
def identifier(self):
""" Return the key's identifier as 20 bytes. """
@@ -141,36 +126,6 @@ class PublicKey(_KeyBase):
self.pubkey_bytes
)
def verify(self, signature, digest) -> bool:
""" Verify that a signature is valid for a 32 byte digest. """
if len(signature) != 64:
raise ValueError('Signature must be 64 bytes long.')
if len(digest) != 32:
raise ValueError('Digest must be 32 bytes long.')
key = self.verifying_key
raw_signature = libsecp256k1_ffi.new('secp256k1_ecdsa_signature *')
parsed = libsecp256k1.secp256k1_ecdsa_signature_parse_compact(
key.context.ctx, raw_signature, signature
)
assert parsed == 1
normalized_signature = libsecp256k1_ffi.new('secp256k1_ecdsa_signature *')
libsecp256k1.secp256k1_ecdsa_signature_normalize(
key.context.ctx, normalized_signature, raw_signature
)
verified = libsecp256k1.secp256k1_ecdsa_verify(
key.context.ctx, normalized_signature, digest, key.public_key
)
return bool(verified)
class PrivateKey(_KeyBase):
"""A BIP32 private key."""
@@ -179,7 +134,7 @@ class PrivateKey(_KeyBase):
def __init__(self, ledger, privkey, chain_code, n, depth, parent=None):
super().__init__(ledger, chain_code, n, depth, parent)
if isinstance(privkey, cPrivateKey):
if isinstance(privkey, _PrivateKey):
self.signing_key = privkey
else:
self.signing_key = self._signing_key_from_privkey(privkey)
@@ -187,7 +142,7 @@ class PrivateKey(_KeyBase):
@classmethod
def _signing_key_from_privkey(cls, private_key):
""" Converts a 32-byte private key into an coincurve.PrivateKey object. """
return cPrivateKey.from_int(PrivateKey._private_key_secret_exponent(private_key))
return _PrivateKey.from_int(PrivateKey._private_key_secret_exponent(private_key))
@classmethod
def _private_key_secret_exponent(cls, private_key):
@@ -199,40 +154,24 @@ class PrivateKey(_KeyBase):
return int.from_bytes(private_key, 'big')
@classmethod
def from_seed(cls, ledger, seed) -> 'PrivateKey':
def from_seed(cls, ledger, seed):
# This hard-coded message string seems to be coin-independent...
hmac = hmac_sha512(b'Bitcoin seed', seed)
privkey, chain_code = hmac[:32], hmac[32:]
return cls(ledger, privkey, chain_code, 0, 0)
@classmethod
def from_pem(cls, ledger, pem) -> 'PrivateKey':
der = pem_to_der(pem.encode())
try:
key_int = ECPrivateKey.load(der).native['private_key']
except ValueError:
key_int = PrivateKeyInfo.load(der).native['private_key']['private_key']
private_key = cPrivateKey.from_int(key_int)
return cls(ledger, private_key, bytes((0,)*32), 0, 0)
@classmethod
def from_bytes(cls, ledger, key_bytes) -> 'PrivateKey':
return cls(ledger, cPrivateKey(key_bytes), bytes((0,)*32), 0, 0)
@cachedproperty
def private_key_bytes(self):
""" Return the serialized private key (no leading zero byte). """
return self.signing_key.secret
@cachedproperty
def public_key(self) -> PublicKey:
def public_key(self):
""" Return the corresponding extended public key. """
verifying_key = self.signing_key.public_key
parent_pubkey = self.parent.public_key if self.parent else None
return PublicKey(
self.ledger, verifying_key, self.chain_code,
self.n, self.depth, parent_pubkey
)
return PubKey(self.ledger, verifying_key, self.chain_code, self.n, self.depth,
parent_pubkey)
def ec_point(self):
return self.public_key.ec_point()
@@ -245,12 +184,11 @@ class PrivateKey(_KeyBase):
""" Return the private key encoded in Wallet Import Format. """
return self.ledger.private_key_to_wif(self.private_key_bytes)
@property
def address(self):
""" The public key as a P2PKH address. """
return self.public_key.address
def child(self, n) -> 'PrivateKey':
def child(self, n):
""" Return the derived child extended private key at index N."""
if not 0 <= n < (1 << 32):
raise ValueError('invalid BIP32 private key child number')
@@ -269,28 +207,6 @@ class PrivateKey(_KeyBase):
""" Produce a signature for piece of data by double hashing it and signing the hash. """
return self.signing_key.sign(data, hasher=double_sha256)
def sign_compact(self, digest):
""" Produce a compact signature. """
key = self.signing_key
signature = libsecp256k1_ffi.new('secp256k1_ecdsa_signature *')
signed = libsecp256k1.secp256k1_ecdsa_sign(
key.context.ctx, signature, digest, key.secret,
libsecp256k1_ffi.NULL, libsecp256k1_ffi.NULL
)
if not signed:
raise ValueError('The private key was invalid.')
serialized = libsecp256k1_ffi.new('unsigned char[%d]' % CDATA_SIG_LENGTH)
compacted = libsecp256k1.secp256k1_ecdsa_signature_serialize_compact(
key.context.ctx, serialized, signature
)
if compacted != 1:
raise ValueError('The signature could not be compacted.')
return bytes(libsecp256k1_ffi.buffer(serialized, CDATA_SIG_LENGTH))
def identifier(self):
"""Return the key's identifier as 20 bytes."""
return self.public_key.identifier()
@@ -302,12 +218,9 @@ class PrivateKey(_KeyBase):
b'\0' + self.private_key_bytes
)
def to_pem(self):
return self.signing_key.to_pem()
def _from_extended_key(ledger, ekey):
"""Return a PublicKey or PrivateKey from an extended key raw bytes."""
"""Return a PubKey or PrivateKey from an extended key raw bytes."""
if not isinstance(ekey, (bytes, bytearray)):
raise TypeError('extended key must be raw bytes')
if len(ekey) != 78:
@@ -319,7 +232,7 @@ def _from_extended_key(ledger, ekey):
if ekey[:4] == ledger.extended_public_key_prefix:
pubkey = ekey[45:]
key = PublicKey(ledger, pubkey, chain_code, n, depth)
key = PubKey(ledger, pubkey, chain_code, n, depth)
elif ekey[:4] == ledger.extended_private_key_prefix:
if ekey[45] != 0:
raise ValueError('invalid extended private key prefix byte')
@@ -337,6 +250,6 @@ def from_extended_key_string(ledger, ekey_str):
xpub6BsnM1W2Y7qLMiuhi7f7dbAwQZ5Cz5gYJCRzTNainXzQXYjFwtuQXHd
3qfi3t3KJtHxshXezfjft93w4UE7BGMtKwhqEHae3ZA7d823DVrL
return a PublicKey or PrivateKey.
return a PubKey or PrivateKey.
"""
return _from_extended_key(ledger, Base58.decode_check(ekey_str))
+3 -19
View File
@@ -9,11 +9,10 @@ from dataclasses import dataclass
from contextvars import ContextVar
from typing import Tuple, List, Union, Callable, Any, Awaitable, Iterable, Dict, Optional
from datetime import date
from prometheus_client import Gauge, Counter, Histogram
from lbry.utils import LockWithMetrics
from .bip32 import PublicKey
from .bip32 import PubKey
from .transaction import Transaction, Output, OutputScript, TXRefImmutable, Input
from .constants import TXO_TYPES, CLAIM_TYPES
from .util import date_to_julian_day
@@ -976,9 +975,7 @@ class Database(SQLiteMixin):
sql.append("LEFT JOIN txi ON (txi.position=0 AND txi.txid=txo.txid)")
return await self.db.execute_fetchall(*query(' '.join(sql), **constraints), read_only=read_only)
async def get_txos(
self, wallet=None, no_tx=False, no_channel_info=False, read_only=False, **constraints
) -> List[Output]:
async def get_txos(self, wallet=None, no_tx=False, no_channel_info=False, read_only=False, **constraints):
include_is_spent = constraints.get('include_is_spent', False)
include_is_my_input = constraints.get('include_is_my_input', False)
include_is_my_output = constraints.pop('include_is_my_output', False)
@@ -1204,14 +1201,13 @@ class Database(SQLiteMixin):
addresses = await self.select_addresses(', '.join(cols), read_only=read_only, **constraints)
if 'pubkey' in cols:
for address in addresses:
address['pubkey'] = PublicKey(
address['pubkey'] = PubKey(
self.ledger, address.pop('pubkey'), address.pop('chain_code'),
address.pop('n'), address.pop('depth')
)
return addresses
async def get_address_count(self, cols=None, read_only=False, **constraints):
self._clean_txo_constraints_for_aggregation(constraints)
count = await self.select_addresses('COUNT(*) as total', read_only=read_only, **constraints)
return count[0]['total'] or 0
@@ -1245,18 +1241,6 @@ class Database(SQLiteMixin):
async def set_address_history(self, address, history):
await self._set_address_history(address, history)
async def is_channel_key_used(self, account, key: PublicKey):
channels = await self.get_txos(
accounts=[account], txo_type=TXO_TYPES['channel'],
no_tx=True, no_channel_info=True
)
other_key_bytes = key.pubkey_bytes
for channel in channels:
claim = channel.can_decode_claim
if claim and claim.channel.public_key_bytes == other_key_bytes:
return True
return False
@staticmethod
def constrain_purchases(constraints):
accounts = constraints.pop('accounts', None)
+16 -28
View File
@@ -16,18 +16,18 @@ from lbry.crypto.hash import hash160, double_sha256, sha256
from lbry.crypto.base58 import Base58
from lbry.utils import LRUCacheWithMetrics
from lbry.wallet.tasks import TaskGroup
from lbry.wallet.database import Database
from lbry.wallet.stream import StreamController
from lbry.wallet.dewies import dewies_to_lbc
from lbry.wallet.account import Account, AddressManager, SingleKey
from lbry.wallet.network import Network
from lbry.wallet.transaction import Transaction, Output
from lbry.wallet.header import Headers, UnvalidatedHeaders
from lbry.wallet.checkpoints import HASHES
from lbry.wallet.constants import TXO_TYPES, CLAIM_TYPES, COIN, NULL_HASH32
from lbry.wallet.bip32 import PublicKey, PrivateKey
from lbry.wallet.coinselection import CoinSelector
from .tasks import TaskGroup
from .database import Database
from .stream import StreamController
from .dewies import dewies_to_lbc
from .account import Account, AddressManager, SingleKey
from .network import Network
from .transaction import Transaction, Output
from .header import Headers, UnvalidatedHeaders
from .checkpoints import HASHES
from .constants import TXO_TYPES, CLAIM_TYPES, COIN, NULL_HASH32
from .bip32 import PubKey, PrivateKey
from .coinselection import CoinSelector
log = logging.getLogger(__name__)
@@ -106,7 +106,7 @@ class Ledger(metaclass=LedgerRegistry):
target_timespan = 150
default_fee_per_byte = 50
default_fee_per_name_char = 0
default_fee_per_name_char = 200000
checkpoints = HASHES
@@ -226,7 +226,7 @@ class Ledger(metaclass=LedgerRegistry):
return account.get_private_key(address_info['chain'], address_info['pubkey'].n)
return None
async def get_public_key_for_address(self, wallet, address) -> Optional[PublicKey]:
async def get_public_key_for_address(self, wallet, address) -> Optional[PubKey]:
match = await self._get_account_and_address_info_for_address(wallet, address)
if match:
_, address_info = match
@@ -365,10 +365,6 @@ class Ledger(metaclass=LedgerRegistry):
await self.db.close()
await self.headers.close()
async def tasks_are_done(self):
await self._update_tasks.done.wait()
await self._other_tasks.done.wait()
@property
def local_height_including_downloaded_height(self):
return max(self.headers.height, self._download_height)
@@ -474,7 +470,6 @@ class Ledger(metaclass=LedgerRegistry):
for address_manager in account.address_managers.values():
await self.subscribe_addresses(address_manager, await address_manager.get_addresses())
await account.ensure_address_gap()
await account.deterministic_channel_keys.ensure_cache_primed()
async def unsubscribe_account(self, account: Account):
for address in await account.get_addresses():
@@ -555,7 +550,6 @@ class Ledger(metaclass=LedgerRegistry):
)
remote_history_txids = {txid for txid, _ in remote_history}
async for tx in self.request_synced_transactions(to_request, remote_history_txids, address):
self.maybe_has_channel_key(tx)
pending_synced_history[tx_indexes[tx.id]] = f"{tx.id}:{tx.height}:"
if len(pending_synced_history) % 100 == 0:
log.info("Syncing address %s: %d/%d", address, len(pending_synced_history), len(to_request))
@@ -623,12 +617,6 @@ class Ledger(metaclass=LedgerRegistry):
tx.is_verified = merkle_root == header['merkle_root']
return tx
def maybe_has_channel_key(self, tx):
for txo in tx._outputs:
if txo.can_decode_claim and txo.claim.is_channel:
for account in self.accounts:
account.deterministic_channel_keys.maybe_generate_deterministic_key_for_channel(txo)
async def request_transactions(self, to_request: Tuple[Tuple[str, int], ...], cached=False):
batches = [[]]
remote_heights = {}
@@ -743,7 +731,7 @@ class Ledger(metaclass=LedgerRegistry):
while timeout and (int(time.perf_counter()) - start) <= timeout:
if await self._wait_round(tx, height, addresses):
return
raise asyncio.TimeoutError(f'Timed out waiting for transaction. {tx.id}')
raise asyncio.TimeoutError('Timed out waiting for transaction.')
async def _wait_round(self, tx: Transaction, height: int, addresses: Iterable[str]):
records = await self.db.get_addresses(address__in=addresses)
@@ -786,7 +774,7 @@ class Ledger(metaclass=LedgerRegistry):
if hub_server:
outputs = Outputs.from_grpc(encoded_outputs)
else:
outputs = Outputs.from_base64(encoded_outputs or '') # TODO: why is the server returning None?
outputs = Outputs.from_base64(encoded_outputs or b'') # TODO: why is the server returning None?
txs: List[Transaction] = []
if len(outputs.txs) > 0:
async for tx in self.request_transactions(tuple(outputs.txs), cached=True):
+7 -7
View File
@@ -12,13 +12,13 @@ from typing import List, Type, MutableSequence, MutableMapping, Optional
from lbry.error import KeyFeeAboveMaxAllowedError, WalletNotLoadedError
from lbry.conf import Config, NOT_SET
from lbry.wallet.dewies import dewies_to_lbc
from lbry.wallet.account import Account
from lbry.wallet.ledger import Ledger, LedgerRegistry
from lbry.wallet.transaction import Transaction, Output
from lbry.wallet.database import Database
from lbry.wallet.wallet import Wallet, WalletStorage, ENCRYPT_ON_DISK
from lbry.wallet.rpc.jsonrpc import CodeMessageError
from .dewies import dewies_to_lbc
from .account import Account
from .ledger import Ledger, LedgerRegistry
from .transaction import Transaction, Output
from .database import Database
from .wallet import Wallet, WalletStorage, ENCRYPT_ON_DISK
from .rpc.jsonrpc import CodeMessageError
if typing.TYPE_CHECKING:
from lbry.extras.daemon.exchange_rate_manager import ExchangeRateManager
+6 -5
View File
@@ -16,7 +16,7 @@ from lbry.utils import resolve_host
from lbry.error import IncompatibleWalletServerError
from lbry.wallet.rpc import RPCSession as BaseClientSession, Connector, RPCError, ProtocolError
from lbry.wallet.stream import StreamController
from lbry.wallet.udp import SPVStatusClientProtocol, SPVPong
from lbry.wallet.server.udp import SPVStatusClientProtocol, SPVPong
from lbry.conf import KnownHubsList
log = logging.getLogger(__name__)
@@ -122,7 +122,7 @@ class ClientSession(BaseClientSession):
await asyncio.sleep(max(0, max_idle - (now - self.last_send)))
except Exception as err:
if isinstance(err, asyncio.CancelledError):
log.info("closing connection to %s:%i", *self.server)
log.warning("closing connection to %s:%i", *self.server)
else:
log.exception("lost connection to spv")
finally:
@@ -140,7 +140,7 @@ class ClientSession(BaseClientSession):
controller.add(request.args)
def connection_lost(self, exc):
log.debug("Connection lost: %s:%d", *self.server)
log.warning("Connection lost: %s:%d", *self.server)
super().connection_lost(exc)
self.response_time = None
self.connection_latency = None
@@ -303,7 +303,7 @@ class Network:
concurrency=self.config.get('concurrent_hub_requests', 30))
try:
await client.create_connection()
log.info("Connected to spv server %s:%i", host, port)
log.warning("Connected to spv server %s:%i", host, port)
await client.ensure_server_version()
return client
except (asyncio.TimeoutError, ConnectionError, OSError, IncompatibleWalletServerError, RPCError):
@@ -348,6 +348,7 @@ class Network:
await self._keepalive_task
if self._urgent_need_reconnect.is_set():
log.warning("urgent reconnect needed")
self._urgent_need_reconnect.clear()
if self._keepalive_task and not self._keepalive_task.done():
self._keepalive_task.cancel()
except asyncio.CancelledError:
@@ -356,7 +357,7 @@ class Network:
self._keepalive_task = None
self.client = None
self.server_features = None
log.info("connection lost to %s", server_str)
log.warning("connection lost to %s", server_str)
log.info("network loop finished")
async def stop(self):
+3 -3
View File
@@ -1,5 +1,5 @@
__hub_url__ = (
"https://github.com/lbryio/herald/releases/download/v0.2022.01.21.1/hub"
"https://github.com/lbryio/hub/releases/download/v0.2022.01.21.1/hub"
)
from lbry.wallet.orchstr8.node import Conductor
from lbry.wallet.orchstr8.service import ConductorService
from .node import Conductor
from .service import ConductorService
+7 -14
View File
@@ -5,9 +5,7 @@ import aiohttp
from lbry import wallet
from lbry.wallet.orchstr8.node import (
Conductor,
get_lbcd_node_from_ledger,
get_lbcwallet_node_from_ledger
Conductor, get_blockchain_node_from_ledger
)
from lbry.wallet.orchstr8.service import ConductorService
@@ -18,11 +16,10 @@ def get_argument_parser():
)
subparsers = parser.add_subparsers(dest='command', help='sub-command help')
subparsers.add_parser("download", help="Download lbcd and lbcwallet node binaries.")
subparsers.add_parser("download", help="Download blockchain node binary.")
start = subparsers.add_parser("start", help="Start orchstr8 service.")
start.add_argument("--lbcd", help="Hostname to start lbcd node.")
start.add_argument("--lbcwallet", help="Hostname to start lbcwallet node.")
start.add_argument("--blockchain", help="Hostname to start blockchain node.")
start.add_argument("--spv", help="Hostname to start SPV server.")
start.add_argument("--wallet", help="Hostname to start wallet daemon.")
@@ -50,8 +47,7 @@ def main():
if command == 'download':
logging.getLogger('blockchain').setLevel(logging.INFO)
get_lbcd_node_from_ledger(wallet).ensure()
get_lbcwallet_node_from_ledger(wallet).ensure()
get_blockchain_node_from_ledger(wallet).ensure()
elif command == 'generate':
loop.run_until_complete(run_remote_command(
@@ -61,12 +57,9 @@ def main():
elif command == 'start':
conductor = Conductor()
if getattr(args, 'lbcd', False):
conductor.lbcd_node.hostname = args.lbcd
loop.run_until_complete(conductor.start_lbcd())
if getattr(args, 'lbcwallet', False):
conductor.lbcwallet_node.hostname = args.lbcwallet
loop.run_until_complete(conductor.start_lbcwallet())
if getattr(args, 'blockchain', False):
conductor.blockchain_node.hostname = args.blockchain
loop.run_until_complete(conductor.start_blockchain())
if getattr(args, 'spv', False):
conductor.spv_node.hostname = args.spv
loop.run_until_complete(conductor.start_spv())
+171 -413
View File
@@ -1,6 +1,4 @@
# pylint: disable=import-error
import os
import signal
import json
import shutil
import asyncio
@@ -9,7 +7,7 @@ import tarfile
import logging
import tempfile
import subprocess
import platform
import importlib
from distutils.util import strtobool
from binascii import hexlify
@@ -17,38 +15,27 @@ from typing import Type, Optional
import urllib.request
from uuid import uuid4
import lbry
from lbry.wallet.server.server import Server
from lbry.wallet.server.env import Env
from lbry.wallet import Wallet, Ledger, RegTestLedger, WalletManager, Account, BlockHeightEvent
from lbry.conf import KnownHubsList, Config
from lbry.wallet.orchstr8 import __hub_url__
log = logging.getLogger(__name__)
try:
from hub.herald.env import ServerEnv
from hub.scribe.env import BlockchainEnv
from hub.elastic_sync.env import ElasticEnv
from hub.herald.service import HubServerService
from hub.elastic_sync.service import ElasticSyncService
from hub.scribe.service import BlockchainProcessorService
except ImportError:
pass
def get_spvserver_from_ledger(ledger_module):
spvserver_path, regtest_class_name = ledger_module.__spvserver__.rsplit('.', 1)
spvserver_module = importlib.import_module(spvserver_path)
return getattr(spvserver_module, regtest_class_name)
def get_lbcd_node_from_ledger(ledger_module):
return LBCDNode(
ledger_module.__lbcd_url__,
ledger_module.__lbcd__,
ledger_module.__lbcctl__
)
def get_lbcwallet_node_from_ledger(ledger_module):
return LBCWalletNode(
ledger_module.__lbcwallet_url__,
ledger_module.__lbcwallet__,
ledger_module.__lbcctl__
def get_blockchain_node_from_ledger(ledger_module):
return BlockchainNode(
ledger_module.__node_url__,
os.path.join(ledger_module.__node_bin__, ledger_module.__node_daemon__),
os.path.join(ledger_module.__node_bin__, ledger_module.__node_cli__)
)
@@ -56,51 +43,53 @@ class Conductor:
def __init__(self, seed=None):
self.manager_module = WalletManager
self.lbcd_node = get_lbcd_node_from_ledger(lbry.wallet)
self.lbcwallet_node = get_lbcwallet_node_from_ledger(lbry.wallet)
self.spv_node = SPVNode()
self.spv_module = get_spvserver_from_ledger(lbry.wallet)
self.blockchain_node = get_blockchain_node_from_ledger(lbry.wallet)
self.spv_node = SPVNode(self.spv_module)
self.wallet_node = WalletNode(
self.manager_module, RegTestLedger, default_seed=seed
)
self.hub_node = HubNode(__hub_url__, "hub", self.spv_node)
self.lbcd_started = False
self.lbcwallet_started = False
self.blockchain_started = False
self.spv_started = False
self.wallet_started = False
self.hub_started = False
self.log = log.getChild('conductor')
async def start_lbcd(self):
if not self.lbcd_started:
await self.lbcd_node.start()
self.lbcd_started = True
async def start_blockchain(self):
if not self.blockchain_started:
asyncio.create_task(self.blockchain_node.start())
await self.blockchain_node.running.wait()
await self.blockchain_node.generate(200)
self.blockchain_started = True
async def stop_lbcd(self, cleanup=True):
if self.lbcd_started:
await self.lbcd_node.stop(cleanup)
self.lbcd_started = False
async def stop_blockchain(self):
if self.blockchain_started:
await self.blockchain_node.stop(cleanup=True)
self.blockchain_started = False
async def start_hub(self):
if not self.hub_started:
await self.hub_node.start()
await self.lbcwallet_node.running.wait()
asyncio.create_task(self.hub_node.start())
await self.blockchain_node.running.wait()
self.hub_started = True
async def stop_hub(self, cleanup=True):
async def stop_hub(self):
if self.hub_started:
await self.hub_node.stop(cleanup)
await self.hub_node.stop(cleanup=True)
self.hub_started = False
async def start_spv(self):
if not self.spv_started:
await self.spv_node.start(self.lbcwallet_node)
await self.spv_node.start(self.blockchain_node)
self.spv_started = True
async def stop_spv(self, cleanup=True):
async def stop_spv(self):
if self.spv_started:
await self.spv_node.stop(cleanup)
await self.spv_node.stop(cleanup=True)
self.spv_started = False
async def start_wallet(self):
@@ -108,41 +97,21 @@ class Conductor:
await self.wallet_node.start(self.spv_node)
self.wallet_started = True
async def stop_wallet(self, cleanup=True):
async def stop_wallet(self):
if self.wallet_started:
await self.wallet_node.stop(cleanup)
await self.wallet_node.stop(cleanup=True)
self.wallet_started = False
async def start_lbcwallet(self, clean=True):
if not self.lbcwallet_started:
await self.lbcwallet_node.start()
if clean:
mining_addr = await self.lbcwallet_node.get_new_address()
self.lbcwallet_node.mining_addr = mining_addr
await self.lbcwallet_node.generate(200)
# unlock the wallet for the next 1 hour
await self.lbcwallet_node.wallet_passphrase("password", 3600)
self.lbcwallet_started = True
async def stop_lbcwallet(self, cleanup=True):
if self.lbcwallet_started:
await self.lbcwallet_node.stop(cleanup)
self.lbcwallet_started = False
async def start(self):
await self.start_lbcd()
await self.start_lbcwallet()
await self.start_blockchain()
await self.start_spv()
await self.start_hub()
await self.start_wallet()
async def stop(self):
all_the_stops = [
self.stop_wallet,
self.stop_hub,
self.stop_spv,
self.stop_lbcwallet,
self.stop_lbcd
self.stop_blockchain
]
for stop in all_the_stops:
try:
@@ -150,12 +119,6 @@ class Conductor:
except Exception as e:
log.exception('Exception raised while stopping services:', exc_info=e)
async def clear_mempool(self):
await self.stop_lbcwallet(cleanup=False)
await self.stop_lbcd(cleanup=False)
await self.start_lbcd()
await self.start_lbcwallet(clean=False)
class WalletNode:
@@ -176,11 +139,10 @@ class WalletNode:
async def start(self, spv_node: 'SPVNode', seed=None, connect=True, config=None):
wallets_dir = os.path.join(self.data_path, 'wallets')
os.mkdir(wallets_dir)
wallet_file_name = os.path.join(wallets_dir, 'my_wallet.json')
if not os.path.isdir(wallets_dir):
os.mkdir(wallets_dir)
with open(wallet_file_name, 'w') as wallet_file:
wallet_file.write('{"version": 1, "accounts": []}\n')
with open(wallet_file_name, 'w') as wallet_file:
wallet_file.write('{"version": 1, "accounts": []}\n')
self.manager = self.manager_class.from_config({
'ledgers': {
self.ledger_class.get_id(): {
@@ -192,7 +154,6 @@ class WalletNode:
'known_hubs': config.known_hubs if config else KnownHubsList(),
'hub_timeout': 30,
'concurrent_hub_requests': 32,
'fee_per_name_char': 200000
}
},
'wallets': [wallet_file_name]
@@ -223,83 +184,55 @@ class WalletNode:
class SPVNode:
def __init__(self, node_number=1):
self.node_number = node_number
def __init__(self, coin_class, node_number=1):
self.coin_class = coin_class
self.controller = None
self.data_path = None
self.server: Optional[HubServerService] = None
self.writer: Optional[BlockchainProcessorService] = None
self.es_writer: Optional[ElasticSyncService] = None
self.server = None
self.hostname = 'localhost'
self.port = 50001 + node_number # avoid conflict with default daemon
self.udp_port = self.port
self.elastic_notifier_port = 19080 + node_number
self.session_timeout = 600
self.stopped = True
self.rpc_port = '0' # disabled by default
self.stopped = False
self.index_name = uuid4().hex
async def start(self, lbcwallet_node: 'LBCWalletNode', extraconf=None):
if not self.stopped:
log.warning("spv node is already running")
return
self.stopped = False
try:
self.data_path = tempfile.mkdtemp()
conf = {
'description': '',
'payment_address': '',
'daily_fee': '0',
'db_dir': self.data_path,
'daemon_url': lbcwallet_node.rpc_url,
'reorg_limit': 100,
'host': self.hostname,
'tcp_port': self.port,
'udp_port': self.udp_port,
'elastic_notifier_port': self.elastic_notifier_port,
'session_timeout': self.session_timeout,
'max_query_workers': 0,
'es_index_prefix': self.index_name,
'chain': 'regtest',
'index_address_status': False
}
if extraconf:
conf.update(extraconf)
self.writer = BlockchainProcessorService(
BlockchainEnv(db_dir=self.data_path, daemon_url=lbcwallet_node.rpc_url,
reorg_limit=100, max_query_workers=0, chain='regtest', index_address_status=False)
)
self.server = HubServerService(ServerEnv(**conf))
self.es_writer = ElasticSyncService(
ElasticEnv(
db_dir=self.data_path, reorg_limit=100, max_query_workers=0, chain='regtest',
elastic_notifier_port=self.elastic_notifier_port,
es_index_prefix=self.index_name,
filtering_channel_ids=(extraconf or {}).get('filtering_channel_ids'),
blocking_channel_ids=(extraconf or {}).get('blocking_channel_ids')
)
)
await self.writer.start()
await self.es_writer.start()
await self.server.start()
except Exception as e:
self.stopped = True
if not isinstance(e, asyncio.CancelledError):
log.exception("failed to start spv node")
raise e
async def start(self, blockchain_node: 'BlockchainNode', extraconf=None):
self.data_path = tempfile.mkdtemp()
conf = {
'DESCRIPTION': '',
'PAYMENT_ADDRESS': '',
'DAILY_FEE': '0',
'DB_DIRECTORY': self.data_path,
'DAEMON_URL': blockchain_node.rpc_url,
'REORG_LIMIT': '100',
'HOST': self.hostname,
'TCP_PORT': str(self.port),
'UDP_PORT': str(self.udp_port),
'SESSION_TIMEOUT': str(self.session_timeout),
'MAX_QUERY_WORKERS': '0',
'INDIVIDUAL_TAG_INDEXES': '',
'RPC_PORT': self.rpc_port,
'ES_INDEX_PREFIX': self.index_name,
'ES_MODE': 'writer',
}
if extraconf:
conf.update(extraconf)
# TODO: don't use os.environ
os.environ.update(conf)
self.server = Server(Env(self.coin_class))
self.server.bp.mempool.refresh_secs = self.server.bp.prefetcher.polling_delay = 0.5
await self.server.start()
async def stop(self, cleanup=True):
if self.stopped:
log.warning("spv node is already stopped")
return
try:
await self.server.db.search_index.delete_index()
await self.server.db.search_index.stop()
await self.server.stop()
await self.es_writer.delete_index()
await self.es_writer.stop()
await self.writer.stop()
self.stopped = True
except Exception as e:
log.exception("failed to stop spv node")
raise e
finally:
cleanup and self.cleanup()
@@ -307,19 +240,18 @@ class SPVNode:
shutil.rmtree(self.data_path, ignore_errors=True)
class LBCDProcess(asyncio.SubprocessProtocol):
class BlockchainProcess(asyncio.SubprocessProtocol):
IGNORE_OUTPUT = [
b'keypool keep',
b'keypool reserve',
b'keypool return',
b'Block submitted',
]
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('lbcd')
self.log = log.getChild('blockchain')
def pipe_data_received(self, fd, data):
if self.log and not any(ignore in data for ignore in self.IGNORE_OUTPUT):
@@ -330,7 +262,7 @@ class LBCDProcess(asyncio.SubprocessProtocol):
if b'Error:' in data:
self.ready.set()
raise SystemError(data.decode())
if b'RPCS: RPC server listening on' in data:
if b'Done loading' in data:
self.ready.set()
def process_exited(self):
@@ -338,57 +270,39 @@ class LBCDProcess(asyncio.SubprocessProtocol):
self.ready.set()
class WalletProcess(asyncio.SubprocessProtocol):
class BlockchainNode:
IGNORE_OUTPUT = [
]
P2SH_SEGWIT_ADDRESS = "p2sh-segwit"
BECH32_ADDRESS = "bech32"
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('lbcwallet')
self.transport: Optional[asyncio.transports.SubprocessTransport] = None
def pipe_data_received(self, fd, data):
if self.log and not any(ignore in data for ignore in self.IGNORE_OUTPUT):
if b'Error:' in data:
self.log.error(data.decode())
else:
self.log.info(data.decode())
if b'Error:' in data:
self.ready.set()
raise SystemError(data.decode())
if b'WLLT: Finished rescan' in data:
self.ready.set()
def process_exited(self):
self.stopped.set()
self.ready.set()
class LBCDNode:
def __init__(self, url, daemon, cli):
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
self.daemon_bin = os.path.join(self.bin_dir, daemon)
self.cli_bin = os.path.join(self.bin_dir, cli)
self.log = log.getChild('lbcd')
self.data_path = tempfile.mkdtemp()
self.log = log.getChild('blockchain')
self.data_path = None
self.protocol = None
self.transport = None
self.block_expected = 0
self.hostname = 'localhost'
self.peerport = 29246
self.rpcport = 29245
self.peerport = 9246 + 2 # avoid conflict with default peer port
self.rpcport = 9245 + 2 # avoid conflict with default rpc port
self.rpcuser = 'rpcuser'
self.rpcpassword = 'rpcpassword'
self.stopped = True
self.stopped = False
self.restart_ready = asyncio.Event()
self.restart_ready.set()
self.running = asyncio.Event()
@property
def rpc_url(self):
return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.rpcport}/'
def is_expected_block(self, e: BlockHeightEvent):
return self.block_expected == e.height
@property
def exists(self):
return (
@@ -397,12 +311,6 @@ class LBCDNode:
)
def download(self):
uname = platform.uname()
target_os = str.lower(uname.system)
target_arch = str.replace(uname.machine, 'x86_64', 'amd64')
target_platform = target_os + '_' + target_arch
self.latest_release_url = str.replace(self.latest_release_url, 'TARGET_PLATFORM', target_platform)
downloaded_file = os.path.join(
self.bin_dir,
self.latest_release_url[self.latest_release_url.rfind('/')+1:]
@@ -435,207 +343,73 @@ class LBCDNode:
def ensure(self):
return self.exists or self.download()
async def start(self):
if not self.stopped:
return
self.stopped = False
try:
assert self.ensure()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.daemon_bin,
'--notls',
f'--datadir={self.data_path}',
'--regtest', f'--listen=127.0.0.1:{self.peerport}', f'--rpclisten=127.0.0.1:{self.rpcport}',
'--txindex', f'--rpcuser={self.rpcuser}', f'--rpcpass={self.rpcpassword}'
]
self.log.info(' '.join(command))
self.transport, self.protocol = await loop.subprocess_exec(
LBCDProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
except asyncio.CancelledError:
self.running.clear()
self.stopped = True
raise
except Exception as e:
self.running.clear()
self.stopped = True
log.exception('failed to start lbcd', exc_info=e)
raise
async def stop(self, cleanup=True):
if self.stopped:
return
try:
if self.transport:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
except Exception as e:
log.exception('failed to stop lbcd', exc_info=e)
raise
finally:
self.log.info("Done shutting down " + self.daemon_bin)
self.stopped = True
if cleanup:
self.cleanup()
self.running.clear()
def cleanup(self):
assert self.stopped
shutil.rmtree(self.data_path, ignore_errors=True)
class LBCWalletNode:
P2SH_SEGWIT_ADDRESS = "p2sh-segwit"
BECH32_ADDRESS = "bech32"
def __init__(self, url, lbcwallet, cli):
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
self.lbcwallet_bin = os.path.join(self.bin_dir, lbcwallet)
self.cli_bin = os.path.join(self.bin_dir, cli)
self.log = log.getChild('lbcwallet')
self.protocol = None
self.transport = None
self.hostname = 'localhost'
self.lbcd_rpcport = 29245
self.lbcwallet_rpcport = 29244
self.rpcuser = 'rpcuser'
self.rpcpassword = 'rpcpassword'
self.data_path = tempfile.mkdtemp()
self.stopped = True
self.running = asyncio.Event()
self.block_expected = 0
self.mining_addr = ''
@property
def rpc_url(self):
# FIXME: somehow the hub/sdk doesn't learn the blocks through the Walet RPC port, why?
# return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.lbcwallet_rpcport}/'
return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.lbcd_rpcport}/'
def is_expected_block(self, e: BlockHeightEvent):
return self.block_expected == e.height
@property
def exists(self):
return (
os.path.exists(self.lbcwallet_bin)
)
def download(self):
uname = platform.uname()
target_os = str.lower(uname.system)
target_arch = str.replace(uname.machine, 'x86_64', 'amd64')
target_platform = target_os + '_' + target_arch
self.latest_release_url = str.replace(self.latest_release_url, 'TARGET_PLATFORM', target_platform)
downloaded_file = os.path.join(
self.bin_dir,
self.latest_release_url[self.latest_release_url.rfind('/')+1:]
)
if not os.path.exists(self.bin_dir):
os.mkdir(self.bin_dir)
if not os.path.exists(downloaded_file):
self.log.info('Downloading: %s', self.latest_release_url)
with urllib.request.urlopen(self.latest_release_url) as response:
with open(downloaded_file, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
self.log.info('Extracting: %s', downloaded_file)
if downloaded_file.endswith('.zip'):
with zipfile.ZipFile(downloaded_file) as dotzip:
dotzip.extractall(self.bin_dir)
# zipfile bug https://bugs.python.org/issue15795
os.chmod(self.lbcwallet_bin, 0o755)
elif downloaded_file.endswith('.tar.gz'):
with tarfile.open(downloaded_file) as tar:
tar.extractall(self.bin_dir)
return self.exists
def ensure(self):
return self.exists or self.download()
async def start(self):
assert self.ensure()
self.data_path = tempfile.mkdtemp()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.lbcwallet_bin,
'--noservertls', '--noclienttls',
'--regtest',
f'--rpcconnect=127.0.0.1:{self.lbcd_rpcport}', f'--rpclisten=127.0.0.1:{self.lbcwallet_rpcport}',
'--createtemp', f'--appdata={self.data_path}',
f'--username={self.rpcuser}', f'--password={self.rpcpassword}'
self.daemon_bin,
f'-datadir={self.data_path}', '-printtoconsole', '-regtest', '-server', '-txindex',
f'-rpcuser={self.rpcuser}', f'-rpcpassword={self.rpcpassword}', f'-rpcport={self.rpcport}',
f'-port={self.peerport}'
]
self.log.info(' '.join(command))
try:
self.transport, self.protocol = await loop.subprocess_exec(
WalletProcess, *command
)
self.protocol.transport = self.transport
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
self.stopped = False
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start lbcwallet', exc_info=e)
def cleanup(self):
assert self.stopped
shutil.rmtree(self.data_path, ignore_errors=True)
while not self.stopped:
if self.running.is_set():
await asyncio.sleep(1)
continue
await self.restart_ready.wait()
try:
self.transport, self.protocol = await loop.subprocess_exec(
BlockchainProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start lbrycrdd', exc_info=e)
async def stop(self, cleanup=True):
if self.stopped:
return
self.stopped = True
try:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
except Exception as e:
log.exception('failed to stop lbcwallet', exc_info=e)
raise
finally:
self.log.info("Done shutting down " + self.lbcwallet_bin)
self.stopped = True
if cleanup:
self.cleanup()
self.running.clear()
async def clear_mempool(self):
self.restart_ready.clear()
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
self.running.clear()
os.remove(os.path.join(self.data_path, 'regtest', 'mempool.dat'))
self.restart_ready.set()
await self.running.wait()
def cleanup(self):
shutil.rmtree(self.data_path, ignore_errors=True)
async def _cli_cmnd(self, *args):
cmnd_args = [
self.cli_bin,
f'--rpcuser={self.rpcuser}', f'--rpcpass={self.rpcpassword}', '--notls', '--regtest', '--wallet'
self.cli_bin, f'-datadir={self.data_path}', '-regtest',
f'-rpcuser={self.rpcuser}', f'-rpcpassword={self.rpcpassword}', f'-rpcport={self.rpcport}'
] + list(args)
self.log.info(' '.join(cmnd_args))
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
process = await asyncio.create_subprocess_exec(
*cmnd_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE
*cmnd_args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
out, err = await process.communicate()
out, _ = await process.communicate()
result = out.decode().strip()
err = err.decode().strip()
if len(result) <= 0 and err.startswith('-'):
raise Exception(err)
if err and 'creating a default config file' not in err:
log.warning(err)
self.log.info(result)
if result.startswith('error code'):
raise Exception(result)
@@ -643,14 +417,7 @@ class LBCWalletNode:
def generate(self, blocks):
self.block_expected += blocks
return self._cli_cmnd('generatetoaddress', str(blocks), self.mining_addr)
def generate_to_address(self, blocks, addr):
self.block_expected += blocks
return self._cli_cmnd('generatetoaddress', str(blocks), addr)
def wallet_passphrase(self, passphrase, timeout):
return self._cli_cmnd('walletpassphrase', passphrase, str(timeout))
return self._cli_cmnd('generate', str(blocks))
def invalidate_block(self, blockhash):
return self._cli_cmnd('invalidateblock', blockhash)
@@ -667,7 +434,7 @@ class LBCWalletNode:
def get_raw_change_address(self):
return self._cli_cmnd('getrawchangeaddress')
def get_new_address(self, address_type='legacy'):
def get_new_address(self, address_type):
return self._cli_cmnd('getnewaddress', "", address_type)
async def get_balance(self):
@@ -683,10 +450,7 @@ class LBCWalletNode:
return self._cli_cmnd('createrawtransaction', json.dumps(inputs), json.dumps(outputs))
async def sign_raw_transaction_with_wallet(self, tx):
# the "withwallet" portion should only come into play if we are doing segwit.
# and "withwallet" doesn't exist on lbcd yet.
result = await self._cli_cmnd('signrawtransaction', tx)
return json.loads(result)['hex'].encode()
return json.loads(await self._cli_cmnd('signrawtransactionwithwallet', tx))['hex'].encode()
def decode_raw_transaction(self, tx):
return self._cli_cmnd('decoderawtransaction', hexlify(tx.raw).decode())
@@ -696,15 +460,12 @@ class LBCWalletNode:
class HubProcess(asyncio.SubprocessProtocol):
def __init__(self, ready, stopped):
self.ready = ready
self.stopped = stopped
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('hub')
self.transport = None
def pipe_data_received(self, fd, data):
self.stopped.clear()
self.ready.set()
if self.log:
self.log.info(data.decode())
if b'error' in data.lower():
@@ -718,26 +479,16 @@ class HubProcess(asyncio.SubprocessProtocol):
print(line)
def process_exited(self):
self.ready.clear()
self.stopped.set()
async def stop(self):
t = asyncio.create_task(self.stopped.wait())
try:
self.transport.send_signal(signal.SIGINT)
await asyncio.wait_for(t, 3)
# log.warning("stopped go hub")
except asyncio.TimeoutError:
if not t.done():
t.cancel()
self.transport.terminate()
await self.stopped.wait()
log.warning("terminated go hub")
self.ready.set()
class HubNode:
def __init__(self, url, daemon, spv_node):
self.spv_node = spv_node
self.debug = False
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
@@ -748,13 +499,11 @@ class HubNode:
self.protocol = None
self.hostname = 'localhost'
self.rpcport = 50051 # avoid conflict with default rpc port
self._stopped = asyncio.Event()
self.stopped = False
self.restart_ready = asyncio.Event()
self.restart_ready.set()
self.running = asyncio.Event()
@property
def stopped(self):
return not self.running.is_set()
@property
def exists(self):
return (
@@ -805,24 +554,33 @@ class HubNode:
self.daemon_bin, 'serve', '--esindex', self.spv_node.index_name + 'claims', '--debug'
]
self.log.info(' '.join(command))
self.protocol = HubProcess(self.running, self._stopped)
try:
self.transport, _ = await loop.subprocess_exec(
lambda: self.protocol, *command
)
self.protocol.transport = self.transport
except Exception as e:
log.exception('failed to start go hub', exc_info=e)
raise e
await self.protocol.ready.wait()
while not self.stopped:
if self.running.is_set():
await asyncio.sleep(1)
continue
await self.restart_ready.wait()
try:
if not self.debug:
self.transport, self.protocol = await loop.subprocess_exec(
HubProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start hub', exc_info=e)
async def stop(self, cleanup=True):
self.stopped = True
try:
if self.protocol:
await self.protocol.stop()
except Exception as e:
log.exception('failed to stop go hub', exc_info=e)
raise e
if not self.debug:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
finally:
if cleanup:
self.cleanup()
+8 -13
View File
@@ -61,10 +61,8 @@ class ConductorService:
#set_logging(
# self.stack.ledger_module, logging.DEBUG, WebSocketLogHandler(self.send_message)
#)
self.stack.lbcd_started or await self.stack.start_lbcd()
self.send_message({'type': 'service', 'name': 'lbcd', 'port': self.stack.lbcd_node.port})
self.stack.lbcwallet_started or await self.stack.start_lbcwallet()
self.send_message({'type': 'service', 'name': 'lbcwallet', 'port': self.stack.lbcwallet_node.port})
self.stack.blockchain_started or await self.stack.start_blockchain()
self.send_message({'type': 'service', 'name': 'blockchain', 'port': self.stack.blockchain_node.port})
self.stack.spv_started or await self.stack.start_spv()
self.send_message({'type': 'service', 'name': 'spv', 'port': self.stack.spv_node.port})
self.stack.wallet_started or await self.stack.start_wallet()
@@ -76,7 +74,7 @@ class ConductorService:
async def generate(self, request):
data = await request.post()
blocks = data.get('blocks', 1)
await self.stack.lbcwallet_node.generate(int(blocks))
await self.stack.blockchain_node.generate(int(blocks))
return json_response({'blocks': blocks})
async def transfer(self, request):
@@ -87,14 +85,11 @@ class ConductorService:
if not address:
raise ValueError("No address was provided.")
amount = data.get('amount', 1)
txid = await self.stack.blockchain_node.send_to_address(address, amount)
if self.stack.wallet_started:
watcher = self.stack.wallet_node.ledger.on_transaction.where(
lambda e: e.address == address # and e.tx.id == txid -- might stall; see send_to_address_and_wait
await self.stack.wallet_node.ledger.on_transaction.where(
lambda e: e.tx.id == txid and e.address == address
)
txid = await self.stack.lbcwallet_node.send_to_address(address, amount)
await watcher
else:
txid = await self.stack.lbcwallet_node.send_to_address(address, amount)
return json_response({
'address': address,
'amount': amount,
@@ -103,7 +98,7 @@ class ConductorService:
async def balance(self, _):
return json_response({
'balance': await self.stack.lbcwallet_node.get_balance()
'balance': await self.stack.blockchain_node.get_balance()
})
async def log(self, request):
@@ -134,7 +129,7 @@ class ConductorService:
'type': 'status',
'height': self.stack.wallet_node.ledger.headers.height,
'balance': satoshis_to_coins(await self.stack.wallet_node.account.get_balance()),
'miner': await self.stack.lbcwallet_node.get_balance()
'miner': await self.stack.blockchain_node.get_balance()
})
def send_message(self, msg):
+67 -42
View File
@@ -17,7 +17,6 @@ OP_HASH160 = 0xa9
OP_EQUALVERIFY = 0x88
OP_CHECKSIG = 0xac
OP_CHECKMULTISIG = 0xae
OP_CHECKLOCKTIMEVERIFY = 0xb1
OP_EQUAL = 0x87
OP_PUSHDATA1 = 0x4c
OP_PUSHDATA2 = 0x4d
@@ -277,7 +276,7 @@ class Template:
elif isinstance(opcode, PUSH_INTEGER):
data = values[opcode.name]
source.write_many(push_data(
data.to_bytes((data.bit_length() + 8) // 8, byteorder='little', signed=True)
data.to_bytes((data.bit_length() + 7) // 8, byteorder='little')
))
elif isinstance(opcode, PUSH_SUBSCRIPT):
data = values[opcode.name]
@@ -358,27 +357,19 @@ class InputScript(Script):
REDEEM_PUBKEY_HASH = Template('pubkey_hash', (
PUSH_SINGLE('signature'), PUSH_SINGLE('pubkey')
))
MULTI_SIG_SCRIPT = Template('multi_sig', (
REDEEM_SCRIPT = Template('script', (
SMALL_INTEGER('signatures_count'), PUSH_MANY('pubkeys'), SMALL_INTEGER('pubkeys_count'),
OP_CHECKMULTISIG
))
REDEEM_SCRIPT_HASH_MULTI_SIG = Template('script_hash+multi_sig', (
OP_0, PUSH_MANY('signatures'), PUSH_SUBSCRIPT('script', MULTI_SIG_SCRIPT)
))
TIME_LOCK_SCRIPT = Template('timelock', (
PUSH_INTEGER('height'), OP_CHECKLOCKTIMEVERIFY, OP_DROP,
# rest is identical to OutputScript.PAY_PUBKEY_HASH:
OP_DUP, OP_HASH160, PUSH_SINGLE('pubkey_hash'), OP_EQUALVERIFY, OP_CHECKSIG
))
REDEEM_SCRIPT_HASH_TIME_LOCK = Template('script_hash+timelock', (
PUSH_SINGLE('signature'), PUSH_SINGLE('pubkey'), PUSH_SUBSCRIPT('script', TIME_LOCK_SCRIPT)
REDEEM_SCRIPT_HASH = Template('script_hash', (
OP_0, PUSH_MANY('signatures'), PUSH_SUBSCRIPT('script', REDEEM_SCRIPT)
))
templates = [
REDEEM_PUBKEY,
REDEEM_PUBKEY_HASH,
REDEEM_SCRIPT_HASH_TIME_LOCK,
REDEEM_SCRIPT_HASH_MULTI_SIG,
REDEEM_SCRIPT_HASH,
REDEEM_SCRIPT
]
@classmethod
@@ -389,38 +380,20 @@ class InputScript(Script):
})
@classmethod
def redeem_multi_sig_script_hash(cls, signatures, pubkeys):
return cls(template=cls.REDEEM_SCRIPT_HASH_MULTI_SIG, values={
def redeem_script_hash(cls, signatures, pubkeys):
return cls(template=cls.REDEEM_SCRIPT_HASH, values={
'signatures': signatures,
'script': cls(template=cls.MULTI_SIG_SCRIPT, values={
'signatures_count': len(signatures),
'pubkeys': pubkeys,
'pubkeys_count': len(pubkeys)
})
'script': cls.redeem_script(signatures, pubkeys)
})
@classmethod
def redeem_time_lock_script_hash(cls, signature, pubkey, height=None, pubkey_hash=None, script_source=None):
if height and pubkey_hash:
script = cls(template=cls.TIME_LOCK_SCRIPT, values={
'height': height,
'pubkey_hash': pubkey_hash
})
elif script_source:
script = cls(source=script_source, template=cls.TIME_LOCK_SCRIPT)
script.parse(script.template)
else:
raise ValueError("script_source or both height and pubkey_hash are required.")
return cls(template=cls.REDEEM_SCRIPT_HASH_TIME_LOCK, values={
'signature': signature,
'pubkey': pubkey,
'script': script
def redeem_script(cls, signatures, pubkeys):
return cls(template=cls.REDEEM_SCRIPT, values={
'signatures_count': len(signatures),
'pubkeys': pubkeys,
'pubkeys_count': len(pubkeys)
})
@property
def is_script_hash(self):
return self.template.name.startswith('script_hash+')
class OutputScript(Script):
@@ -487,6 +460,21 @@ class OutputScript(Script):
UPDATE_CLAIM_OPCODES + PAY_SCRIPT_HASH.opcodes
))
SELL_SCRIPT = Template('sell_script', (
OP_VERIFY, OP_DROP, OP_DROP, OP_DROP, PUSH_INTEGER('price'), OP_PRICECHECK
))
SELL_CLAIM = Template('sell_claim+pay_script_hash', (
OP_SELL_CLAIM, PUSH_SINGLE('claim_id'), PUSH_SUBSCRIPT('sell_script', SELL_SCRIPT),
PUSH_SUBSCRIPT('receive_script', InputScript.REDEEM_SCRIPT), OP_2DROP, OP_2DROP
) + PAY_SCRIPT_HASH.opcodes)
BUY_CLAIM = Template('buy_claim+pay_script_hash', (
OP_BUY_CLAIM, PUSH_SINGLE('sell_id'),
PUSH_SINGLE('claim_id'), PUSH_SINGLE('claim_version'),
PUSH_SINGLE('owner_pubkey_hash'), PUSH_SINGLE('negotiation_signature'),
OP_2DROP, OP_2DROP, OP_2DROP,
) + PAY_SCRIPT_HASH.opcodes)
templates = [
PAY_PUBKEY_FULL,
PAY_PUBKEY_HASH,
@@ -501,6 +489,8 @@ class OutputScript(Script):
SUPPORT_CLAIM_DATA_SCRIPT,
UPDATE_CLAIM_PUBKEY,
UPDATE_CLAIM_SCRIPT,
SELL_CLAIM, SELL_SCRIPT,
BUY_CLAIM,
]
@classmethod
@@ -560,6 +550,30 @@ class OutputScript(Script):
'pubkey_hash': pubkey_hash
})
@classmethod
def sell_script(cls, price):
return cls(template=cls.SELL_SCRIPT, values={
'price': price,
})
@classmethod
def sell_claim(cls, claim_id, price, signatures, pubkeys):
return cls(template=cls.SELL_CLAIM, values={
'claim_id': claim_id,
'sell_script': OutputScript.sell_script(price),
'receive_script': InputScript.redeem_script(signatures, pubkeys)
})
@classmethod
def buy_claim(cls, sell_id, claim_id, claim_version, owner_pubkey_hash, negotiation_signature):
return cls(template=cls.BUY_CLAIM, values={
'sell_id': sell_id,
'claim_id': claim_id,
'claim_version': claim_version,
'owner_pubkey_hash': owner_pubkey_hash,
'negotiation_signature': negotiation_signature,
})
@property
def is_pay_pubkey_hash(self):
return self.template.name.endswith('pay_pubkey_hash')
@@ -588,6 +602,17 @@ class OutputScript(Script):
def is_support_claim_data(self):
return self.template.name.startswith('support_claim+data+')
@property
def is_sell_claim(self):
return self.template.name.startswith('sell_claim+')
@property
def is_buy_claim(self):
return self.template.name.startswith('buy_claim+')
@property
def is_claim_involved(self):
return any((self.is_claim_name, self.is_support_claim, self.is_update_claim))
return any((
self.is_claim_name, self.is_support_claim, self.is_update_claim,
self.is_sell_claim, self.is_buy_claim
))
File renamed without changes.
File diff suppressed because it is too large. Load diff
+34
View File
@@ -0,0 +1,34 @@
import logging
import traceback
import argparse
from lbry.wallet.server.env import Env
from lbry.wallet.server.server import Server
def get_argument_parser():
parser = argparse.ArgumentParser(
prog="lbry-hub"
)
Env.contribute_to_arg_parser(parser)
return parser
def main():
parser = get_argument_parser()
args = parser.parse_args()
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-4s %(name)s:%(lineno)d: %(message)s")
logging.info('lbry.server starting')
logging.getLogger('aiohttp').setLevel(logging.WARNING)
logging.getLogger('elasticsearch').setLevel(logging.WARNING)
try:
server = Server(Env.from_arg_parser(args))
server.run()
except Exception:
traceback.print_exc()
logging.critical('lbry.server terminated abnormally')
else:
logging.info('lbry.server terminated normally')
if __name__ == "__main__":
main()
+386
View File
@@ -0,0 +1,386 @@
import re
import struct
from typing import List
from hashlib import sha256
from decimal import Decimal
from collections import namedtuple
import lbry.wallet.server.tx as lib_tx
from lbry.wallet.script import OutputScript, OP_CLAIM_NAME, OP_UPDATE_CLAIM, OP_SUPPORT_CLAIM
from lbry.wallet.server.tx import DeserializerSegWit
from lbry.wallet.server.util import cachedproperty, subclasses
from lbry.wallet.server.hash import Base58, hash160, double_sha256, hash_to_hex_str, HASHX_LEN
from lbry.wallet.server.daemon import Daemon, LBCDaemon
from lbry.wallet.server.script import ScriptPubKey, OpCodes
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.session import LBRYElectrumX, LBRYSessionManager
from lbry.wallet.server.block_processor import BlockProcessor
Block = namedtuple("Block", "raw header transactions")
OP_RETURN = OpCodes.OP_RETURN
class CoinError(Exception):
"""Exception raised for coin-related errors."""
class Coin:
"""Base class of coin hierarchy."""
REORG_LIMIT = 200
# Not sure if these are coin-specific
RPC_URL_REGEX = re.compile('.+@(\\[[0-9a-fA-F:]+\\]|[^:]+)(:[0-9]+)?')
VALUE_PER_COIN = 100000000
CHUNK_SIZE = 2016
BASIC_HEADER_SIZE = 80
STATIC_BLOCK_HEADERS = True
SESSIONCLS = LBRYElectrumX
DESERIALIZER = lib_tx.Deserializer
DAEMON = Daemon
BLOCK_PROCESSOR = BlockProcessor
SESSION_MANAGER = LBRYSessionManager
DB = LevelDB
HEADER_VALUES = [
'version', 'prev_block_hash', 'merkle_root', 'timestamp', 'bits', 'nonce'
]
HEADER_UNPACK = struct.Struct('< I 32s 32s I I I').unpack_from
MEMPOOL_HISTOGRAM_REFRESH_SECS = 500
XPUB_VERBYTES = bytes('????', 'utf-8')
XPRV_VERBYTES = bytes('????', 'utf-8')
ENCODE_CHECK = Base58.encode_check
DECODE_CHECK = Base58.decode_check
# Peer discovery
PEER_DEFAULT_PORTS = {'t': '50001', 's': '50002'}
PEERS: List[str] = []
@classmethod
def lookup_coin_class(cls, name, net):
"""Return a coin class given name and network.
Raise an exception if unrecognised."""
req_attrs = ['TX_COUNT', 'TX_COUNT_HEIGHT', 'TX_PER_BLOCK']
for coin in subclasses(Coin):
if (coin.NAME.lower() == name.lower() and
coin.NET.lower() == net.lower()):
coin_req_attrs = req_attrs.copy()
missing = [attr for attr in coin_req_attrs
if not hasattr(coin, attr)]
if missing:
raise CoinError(f'coin {name} missing {missing} attributes')
return coin
raise CoinError(f'unknown coin {name} and network {net} combination')
@classmethod
def sanitize_url(cls, url):
# Remove surrounding ws and trailing /s
url = url.strip().rstrip('/')
match = cls.RPC_URL_REGEX.match(url)
if not match:
raise CoinError(f'invalid daemon URL: "{url}"')
if match.groups()[1] is None:
url += f':{cls.RPC_PORT:d}'
if not url.startswith('http://') and not url.startswith('https://'):
url = 'http://' + url
return url + '/'
@classmethod
def genesis_block(cls, block):
"""Check the Genesis block is the right one for this coin.
Return the block less its unspendable coinbase.
"""
header = cls.block_header(block, 0)
header_hex_hash = hash_to_hex_str(cls.header_hash(header))
if header_hex_hash != cls.GENESIS_HASH:
raise CoinError(f'genesis block has hash {header_hex_hash} expected {cls.GENESIS_HASH}')
return header + bytes(1)
@classmethod
def hashX_from_script(cls, script):
"""Returns a hashX from a script, or None if the script is provably
unspendable so the output can be dropped.
"""
if script and script[0] == OP_RETURN:
return None
return sha256(script).digest()[:HASHX_LEN]
@staticmethod
def lookup_xverbytes(verbytes):
"""Return a (is_xpub, coin_class) pair given xpub/xprv verbytes."""
# Order means BTC testnet will override NMC testnet
for coin in subclasses(Coin):
if verbytes == coin.XPUB_VERBYTES:
return True, coin
if verbytes == coin.XPRV_VERBYTES:
return False, coin
raise CoinError('version bytes unrecognised')
@classmethod
def address_to_hashX(cls, address):
"""Return a hashX given a coin address."""
return cls.hashX_from_script(cls.pay_to_address_script(address))
@classmethod
def P2PKH_address_from_hash160(cls, hash160):
"""Return a P2PKH address given a public key."""
assert len(hash160) == 20
return cls.ENCODE_CHECK(cls.P2PKH_VERBYTE + hash160)
@classmethod
def P2PKH_address_from_pubkey(cls, pubkey):
"""Return a coin address given a public key."""
return cls.P2PKH_address_from_hash160(hash160(pubkey))
@classmethod
def P2SH_address_from_hash160(cls, hash160):
"""Return a coin address given a hash160."""
assert len(hash160) == 20
return cls.ENCODE_CHECK(cls.P2SH_VERBYTES[0] + hash160)
@classmethod
def hash160_to_P2PKH_script(cls, hash160):
return ScriptPubKey.P2PKH_script(hash160)
@classmethod
def hash160_to_P2PKH_hashX(cls, hash160):
return cls.hashX_from_script(cls.hash160_to_P2PKH_script(hash160))
@classmethod
def pay_to_address_script(cls, address):
"""Return a pubkey script that pays to a pubkey hash.
Pass the address (either P2PKH or P2SH) in base58 form.
"""
raw = cls.DECODE_CHECK(address)
# Require version byte(s) plus hash160.
verbyte = -1
verlen = len(raw) - 20
if verlen > 0:
verbyte, hash160 = raw[:verlen], raw[verlen:]
if verbyte == cls.P2PKH_VERBYTE:
return cls.hash160_to_P2PKH_script(hash160)
if verbyte in cls.P2SH_VERBYTES:
return ScriptPubKey.P2SH_script(hash160)
raise CoinError(f'invalid address: {address}')
@classmethod
def privkey_WIF(cls, privkey_bytes, compressed):
"""Return the private key encoded in Wallet Import Format."""
payload = bytearray(cls.WIF_BYTE) + privkey_bytes
if compressed:
payload.append(0x01)
return cls.ENCODE_CHECK(payload)
@classmethod
def header_hash(cls, header):
"""Given a header return hash"""
return double_sha256(header)
@classmethod
def header_prevhash(cls, header):
"""Given a header return previous hash"""
return header[4:36]
@classmethod
def static_header_offset(cls, height):
"""Given a header height return its offset in the headers file.
If header sizes change at some point, this is the only code
that needs updating."""
assert cls.STATIC_BLOCK_HEADERS
return height * cls.BASIC_HEADER_SIZE
@classmethod
def static_header_len(cls, height):
"""Given a header height return its length."""
return (cls.static_header_offset(height + 1)
- cls.static_header_offset(height))
@classmethod
def block_header(cls, block, height):
"""Returns the block header given a block and its height."""
return block[:cls.static_header_len(height)]
@classmethod
def block(cls, raw_block, height):
"""Return a Block namedtuple given a raw block and its height."""
header = cls.block_header(raw_block, height)
txs = cls.DESERIALIZER(raw_block, start=len(header)).read_tx_block()
return Block(raw_block, header, txs)
@classmethod
def transaction(cls, raw_tx: bytes):
"""Return a Block namedtuple given a raw block and its height."""
return cls.DESERIALIZER(raw_tx).read_tx()
@classmethod
def decimal_value(cls, value):
"""Return the number of standard coin units as a Decimal given a
quantity of smallest units.
For example 1 BTC is returned for 100 million satoshis.
"""
return Decimal(value) / cls.VALUE_PER_COIN
@classmethod
def electrum_header(cls, header, height):
h = dict(zip(cls.HEADER_VALUES, cls.HEADER_UNPACK(header)))
# Add the height that is not present in the header itself
h['block_height'] = height
# Convert bytes to str
h['prev_block_hash'] = hash_to_hex_str(h['prev_block_hash'])
h['merkle_root'] = hash_to_hex_str(h['merkle_root'])
return h
class LBC(Coin):
DAEMON = LBCDaemon
SESSIONCLS = LBRYElectrumX
SESSION_MANAGER = LBRYSessionManager
DESERIALIZER = DeserializerSegWit
DB = LevelDB
NAME = "LBRY"
SHORTNAME = "LBC"
NET = "mainnet"
BASIC_HEADER_SIZE = 112
CHUNK_SIZE = 96
XPUB_VERBYTES = bytes.fromhex("0488b21e")
XPRV_VERBYTES = bytes.fromhex("0488ade4")
P2PKH_VERBYTE = bytes.fromhex("55")
P2SH_VERBYTES = bytes.fromhex("7A")
WIF_BYTE = bytes.fromhex("1C")
GENESIS_HASH = ('9c89283ba0f3227f6c03b70216b9f665'
'f0118d5e0fa729cedf4fb34d6a34f463')
TX_COUNT = 2716936
TX_COUNT_HEIGHT = 329554
TX_PER_BLOCK = 1
RPC_PORT = 9245
REORG_LIMIT = 200
nOriginalClaimExpirationTime = 262974
nExtendedClaimExpirationTime = 2102400
nExtendedClaimExpirationForkHeight = 400155
nNormalizedNameForkHeight = 539940 # targeting 21 March 2019
nMinTakeoverWorkaroundHeight = 496850
nMaxTakeoverWorkaroundHeight = 658300 # targeting 30 Oct 2019
nWitnessForkHeight = 680770 # targeting 11 Dec 2019
nAllClaimsInMerkleForkHeight = 658310 # targeting 30 Oct 2019
proportionalDelayFactor = 32
maxTakeoverDelay = 4032
PEERS = [
]
@classmethod
def genesis_block(cls, block):
'''Check the Genesis block is the right one for this coin.
Return the block less its unspendable coinbase.
'''
header = cls.block_header(block, 0)
header_hex_hash = hash_to_hex_str(cls.header_hash(header))
if header_hex_hash != cls.GENESIS_HASH:
raise CoinError(f'genesis block has hash {header_hex_hash} expected {cls.GENESIS_HASH}')
return block
@classmethod
def electrum_header(cls, header, height):
version, = struct.unpack('<I', header[:4])
timestamp, bits, nonce = struct.unpack('<III', header[100:112])
return {
'version': version,
'prev_block_hash': hash_to_hex_str(header[4:36]),
'merkle_root': hash_to_hex_str(header[36:68]),
'claim_trie_root': hash_to_hex_str(header[68:100]),
'timestamp': timestamp,
'bits': bits,
'nonce': nonce,
'block_height': height,
}
@cachedproperty
def address_handlers(self):
return ScriptPubKey.PayToHandlers(
address=self.P2PKH_address_from_hash160,
script_hash=self.P2SH_address_from_hash160,
pubkey=self.P2PKH_address_from_pubkey,
unspendable=lambda: None,
strange=self.claim_address_handler,
)
@classmethod
def address_from_script(cls, script):
'''Given a pk_script, return the address it pays to, or None.'''
return ScriptPubKey.pay_to(cls.address_handlers, script)
@classmethod
def claim_address_handler(cls, script):
'''Parse a claim script, returns the address
'''
output = OutputScript(script)
if output.is_pay_pubkey_hash:
return cls.P2PKH_address_from_hash160(output.values['pubkey_hash'])
if output.is_pay_script_hash:
return cls.P2SH_address_from_hash160(output.values['script_hash'])
if output.is_pay_pubkey:
return cls.P2PKH_address_from_pubkey(output.values['pubkey'])
if output.is_return_data:
return None
return None
@classmethod
def hashX_from_script(cls, script):
'''
Overrides electrumx hashX from script by extracting addresses from claim scripts.
'''
if script and script[0] == OpCodes.OP_RETURN or not script:
return None
if script[0] in [
OP_CLAIM_NAME,
OP_UPDATE_CLAIM,
OP_SUPPORT_CLAIM,
]:
return cls.address_to_hashX(cls.claim_address_handler(script))
else:
return sha256(script).digest()[:HASHX_LEN]
@classmethod
def get_expiration_height(cls, last_updated_height: int, extended: bool = False) -> int:
if extended:
return last_updated_height + cls.nExtendedClaimExpirationTime
if last_updated_height < cls.nExtendedClaimExpirationForkHeight:
return last_updated_height + cls.nOriginalClaimExpirationTime
return last_updated_height + cls.nExtendedClaimExpirationTime
@classmethod
def get_delay_for_name(cls, blocks_of_continuous_ownership: int) -> int:
return min(blocks_of_continuous_ownership // cls.proportionalDelayFactor, cls.maxTakeoverDelay)
class LBCRegTest(LBC):
NET = "regtest"
GENESIS_HASH = '6e3fcf1299d4ec5d79c3a4c91d624a4acf9e2e173d95a1a0504f677669687556'
XPUB_VERBYTES = bytes.fromhex('043587cf')
XPRV_VERBYTES = bytes.fromhex('04358394')
P2PKH_VERBYTE = bytes.fromhex("6f")
P2SH_VERBYTES = bytes.fromhex("c4")
nOriginalClaimExpirationTime = 500
nExtendedClaimExpirationTime = 600
nExtendedClaimExpirationForkHeight = 800
nNormalizedNameForkHeight = 250
nMinTakeoverWorkaroundHeight = -1
nMaxTakeoverWorkaroundHeight = -1
nWitnessForkHeight = 150
nAllClaimsInMerkleForkHeight = 350
class LBCTestNet(LBCRegTest):
NET = "testnet"
GENESIS_HASH = '9c89283ba0f3227f6c03b70216b9f665f0118d5e0fa729cedf4fb34d6a34f463'
+375
View File
@@ -0,0 +1,375 @@
import asyncio
import itertools
import json
import time
from functools import wraps
import aiohttp
from prometheus_client import Gauge, Histogram
from lbry.utils import LRUCacheWithMetrics
from lbry.wallet.rpc.jsonrpc import RPCError
from lbry.wallet.server.util import hex_to_bytes, class_logger
from lbry.wallet.rpc import JSONRPC
class DaemonError(Exception):
"""Raised when the daemon returns an error in its results."""
class WarmingUpError(Exception):
"""Internal - when the daemon is warming up."""
class WorkQueueFullError(Exception):
"""Internal - when the daemon's work queue is full."""
NAMESPACE = "wallet_server"
class Daemon:
"""Handles connections to a daemon at the given URL."""
WARMING_UP = -28
id_counter = itertools.count()
lbrycrd_request_time_metric = Histogram(
"lbrycrd_request", "lbrycrd requests count", namespace=NAMESPACE, labelnames=("method",)
)
lbrycrd_pending_count_metric = Gauge(
"lbrycrd_pending_count", "Number of lbrycrd rpcs that are in flight", namespace=NAMESPACE,
labelnames=("method",)
)
def __init__(self, coin, url, max_workqueue=10, init_retry=0.25,
max_retry=4.0):
self.coin = coin
self.logger = class_logger(__name__, self.__class__.__name__)
self.set_url(url)
# Limit concurrent RPC calls to this number.
# See DEFAULT_HTTP_WORKQUEUE in bitcoind, which is typically 16
self.workqueue_semaphore = asyncio.Semaphore(value=max_workqueue)
self.init_retry = init_retry
self.max_retry = max_retry
self._height = None
self.available_rpcs = {}
self.connector = aiohttp.TCPConnector()
self._block_hash_cache = LRUCacheWithMetrics(100000)
self._block_cache = LRUCacheWithMetrics(2 ** 13, metric_name='block', namespace=NAMESPACE)
async def close(self):
if self.connector:
await self.connector.close()
self.connector = None
def set_url(self, url):
"""Set the URLS to the given list, and switch to the first one."""
urls = url.split(',')
urls = [self.coin.sanitize_url(url) for url in urls]
for n, url in enumerate(urls):
status = '' if n else ' (current)'
logged_url = self.logged_url(url)
self.logger.info(f'daemon #{n + 1} at {logged_url}{status}')
self.url_index = 0
self.urls = urls
def current_url(self):
"""Returns the current daemon URL."""
return self.urls[self.url_index]
def logged_url(self, url=None):
"""The host and port part, for logging."""
url = url or self.current_url()
return url[url.rindex('@') + 1:]
def failover(self):
"""Call to fail-over to the next daemon URL.
Returns False if there is only one, otherwise True.
"""
if len(self.urls) > 1:
self.url_index = (self.url_index + 1) % len(self.urls)
self.logger.info(f'failing over to {self.logged_url()}')
return True
return False
def client_session(self):
"""An aiohttp client session."""
return aiohttp.ClientSession(connector=self.connector, connector_owner=False)
async def _send_data(self, data):
if not self.connector:
raise asyncio.CancelledError('Tried to send request during shutdown.')
async with self.workqueue_semaphore:
async with self.client_session() as session:
async with session.post(self.current_url(), data=data) as resp:
kind = resp.headers.get('Content-Type', None)
if kind == 'application/json':
return await resp.json()
# bitcoind's HTTP protocol "handling" is a bad joke
text = await resp.text()
if 'Work queue depth exceeded' in text:
raise WorkQueueFullError
text = text.strip() or resp.reason
self.logger.error(text)
raise DaemonError(text)
async def _send(self, payload, processor):
"""Send a payload to be converted to JSON.
Handles temporary connection issues. Daemon response errors
are raise through DaemonError.
"""
def log_error(error):
nonlocal last_error_log, retry
now = time.time()
if now - last_error_log > 60:
last_error_log = now
self.logger.error(f'{error} Retrying occasionally...')
if retry == self.max_retry and self.failover():
retry = 0
on_good_message = None
last_error_log = 0
data = json.dumps(payload)
retry = self.init_retry
methods = tuple(
[payload['method']] if isinstance(payload, dict) else [request['method'] for request in payload]
)
while True:
try:
for method in methods:
self.lbrycrd_pending_count_metric.labels(method=method).inc()
result = await self._send_data(data)
result = processor(result)
if on_good_message:
self.logger.info(on_good_message)
return result
except asyncio.TimeoutError:
log_error('timeout error.')
except aiohttp.ServerDisconnectedError:
log_error('disconnected.')
on_good_message = 'connection restored'
except aiohttp.ClientConnectionError:
log_error('connection problem - is your daemon running?')
on_good_message = 'connection restored'
except aiohttp.ClientError as e:
log_error(f'daemon error: {e}')
on_good_message = 'running normally'
except WarmingUpError:
log_error('starting up checking blocks.')
on_good_message = 'running normally'
except WorkQueueFullError:
log_error('work queue full.')
on_good_message = 'running normally'
finally:
for method in methods:
self.lbrycrd_pending_count_metric.labels(method=method).dec()
await asyncio.sleep(retry)
retry = max(min(self.max_retry, retry * 2), self.init_retry)
async def _send_single(self, method, params=None):
"""Send a single request to the daemon."""
start = time.perf_counter()
def processor(result):
err = result['error']
if not err:
return result['result']
if err.get('code') == self.WARMING_UP:
raise WarmingUpError
raise DaemonError(err)
payload = {'method': method, 'id': next(self.id_counter)}
if params:
payload['params'] = params
result = await self._send(payload, processor)
self.lbrycrd_request_time_metric.labels(method=method).observe(time.perf_counter() - start)
return result
async def _send_vector(self, method, params_iterable, replace_errs=False):
"""Send several requests of the same method.
The result will be an array of the same length as params_iterable.
If replace_errs is true, any item with an error is returned as None,
otherwise an exception is raised."""
start = time.perf_counter()
def processor(result):
errs = [item['error'] for item in result if item['error']]
if any(err.get('code') == self.WARMING_UP for err in errs):
raise WarmingUpError
if not errs or replace_errs:
return [item['result'] for item in result]
raise DaemonError(errs)
payload = [{'method': method, 'params': p, 'id': next(self.id_counter)}
for p in params_iterable]
result = []
if payload:
result = await self._send(payload, processor)
self.lbrycrd_request_time_metric.labels(method=method).observe(time.perf_counter() - start)
return result
async def _is_rpc_available(self, method):
"""Return whether given RPC method is available in the daemon.
Results are cached and the daemon will generally not be queried with
the same method more than once."""
available = self.available_rpcs.get(method)
if available is None:
available = True
try:
await self._send_single(method)
except DaemonError as e:
err = e.args[0]
error_code = err.get("code")
available = error_code != JSONRPC.METHOD_NOT_FOUND
self.available_rpcs[method] = available
return available
async def block_hex_hashes(self, first, count):
"""Return the hex hashes of count block starting at height first."""
if first + count < (self.cached_height() or 0) - 200:
return await self._cached_block_hex_hashes(first, count)
params_iterable = ((h, ) for h in range(first, first + count))
return await self._send_vector('getblockhash', params_iterable)
async def _cached_block_hex_hashes(self, first, count):
"""Return the hex hashes of count block starting at height first."""
cached = self._block_hash_cache.get((first, count))
if cached:
return cached
params_iterable = ((h, ) for h in range(first, first + count))
self._block_hash_cache[(first, count)] = await self._send_vector('getblockhash', params_iterable)
return self._block_hash_cache[(first, count)]
async def deserialised_block(self, hex_hash):
"""Return the deserialised block with the given hex hash."""
if hex_hash not in self._block_cache:
block = await self._send_single('getblock', (hex_hash, True))
self._block_cache[hex_hash] = block
return block
return self._block_cache[hex_hash]
async def raw_blocks(self, hex_hashes):
"""Return the raw binary blocks with the given hex hashes."""
params_iterable = ((h, False) for h in hex_hashes)
blocks = await self._send_vector('getblock', params_iterable)
# Convert hex string to bytes
return [hex_to_bytes(block) for block in blocks]
async def mempool_hashes(self):
"""Update our record of the daemon's mempool hashes."""
return await self._send_single('getrawmempool')
async def estimatefee(self, block_count):
"""Return the fee estimate for the block count. Units are whole
currency units per KB, e.g. 0.00000995, or -1 if no estimate
is available.
"""
args = (block_count, )
if await self._is_rpc_available('estimatesmartfee'):
estimate = await self._send_single('estimatesmartfee', args)
return estimate.get('feerate', -1)
return await self._send_single('estimatefee', args)
async def getnetworkinfo(self):
"""Return the result of the 'getnetworkinfo' RPC call."""
return await self._send_single('getnetworkinfo')
async def relayfee(self):
"""The minimum fee a low-priority tx must pay in order to be accepted
to the daemon's memory pool."""
network_info = await self.getnetworkinfo()
return network_info['relayfee']
async def getrawtransaction(self, hex_hash, verbose=False):
"""Return the serialized raw transaction with the given hash."""
# Cast to int because some coin daemons are old and require it
return await self._send_single('getrawtransaction',
(hex_hash, int(verbose)))
async def getrawtransactions(self, hex_hashes, replace_errs=True):
"""Return the serialized raw transactions with the given hashes.
Replaces errors with None by default."""
params_iterable = ((hex_hash, 0) for hex_hash in hex_hashes)
txs = await self._send_vector('getrawtransaction', params_iterable,
replace_errs=replace_errs)
# Convert hex strings to bytes
return [hex_to_bytes(tx) if tx else None for tx in txs]
async def broadcast_transaction(self, raw_tx):
"""Broadcast a transaction to the network."""
return await self._send_single('sendrawtransaction', (raw_tx, ))
async def height(self):
"""Query the daemon for its current height."""
self._height = await self._send_single('getblockcount')
return self._height
def cached_height(self):
"""Return the cached daemon height.
If the daemon has not been queried yet this returns None."""
return self._height
def handles_errors(decorated_function):
@wraps(decorated_function)
async def wrapper(*args, **kwargs):
try:
return await decorated_function(*args, **kwargs)
except DaemonError as daemon_error:
raise RPCError(1, daemon_error.args[0])
return wrapper
class LBCDaemon(Daemon):
@handles_errors
async def getrawtransaction(self, hex_hash, verbose=False):
return await super().getrawtransaction(hex_hash=hex_hash, verbose=verbose)
@handles_errors
async def getclaimbyid(self, claim_id):
'''Given a claim id, retrieves claim information.'''
return await self._send_single('getclaimbyid', (claim_id,))
@handles_errors
async def getclaimsbyids(self, claim_ids):
'''Given a list of claim ids, batches calls to retrieve claim information.'''
return await self._send_vector('getclaimbyid', ((claim_id,) for claim_id in claim_ids))
@handles_errors
async def getclaimsforname(self, name):
'''Given a name, retrieves all claims matching that name.'''
return await self._send_single('getclaimsforname', (name,))
@handles_errors
async def getclaimsfortx(self, txid):
'''Given a txid, returns the claims it make.'''
return await self._send_single('getclaimsfortx', (txid,)) or []
@handles_errors
async def getnameproof(self, name, block_hash=None):
'''Given a name and optional block_hash, returns a name proof and winner, if any.'''
return await self._send_single('getnameproof', (name, block_hash,) if block_hash else (name,))
@handles_errors
async def getvalueforname(self, name):
'''Given a name, returns the winning claim value.'''
return await self._send_single('getvalueforname', (name,))
@handles_errors
async def getnamesintrie(self):
'''Given a name, returns the winning claim value.'''
return await self._send_single('getnamesintrie')
@handles_errors
async def claimname(self, name, hexvalue, amount):
'''Claim a name, used for functional tests only.'''
return await self._send_single('claimname', (name, hexvalue, float(amount)))
+42
View File
@@ -0,0 +1,42 @@
import enum
@enum.unique
class DB_PREFIXES(enum.Enum):
claim_to_support = b'K'
support_to_claim = b'L'
claim_to_txo = b'E'
txo_to_claim = b'G'
claim_to_channel = b'I'
channel_to_claim = b'J'
claim_short_id_prefix = b'F'
effective_amount = b'D'
claim_expiration = b'O'
claim_takeover = b'P'
pending_activation = b'Q'
activated_claim_and_support = b'R'
active_amount = b'S'
repost = b'V'
reposted_claim = b'W'
undo = b'M'
claim_diff = b'Y'
tx = b'B'
block_hash = b'C'
header = b'H'
tx_num = b'N'
tx_count = b'T'
tx_hash = b'X'
utxo = b'u'
hashx_utxo = b'h'
hashx_history = b'x'
db_state = b's'
channel_count = b'Z'
support_amount = b'a'
block_txs = b'b'
+447
View File
@@ -0,0 +1,447 @@
import typing
CLAIM_TYPES = {
'stream': 1,
'channel': 2,
'repost': 3,
'collection': 4,
}
STREAM_TYPES = {
'video': 1,
'audio': 2,
'image': 3,
'document': 4,
'binary': 5,
'model': 6,
}
# 9/21/2020
MOST_USED_TAGS = {
"gaming",
"people & blogs",
"entertainment",
"music",
"pop culture",
"education",
"technology",
"blockchain",
"news",
"funny",
"science & technology",
"learning",
"gameplay",
"news & politics",
"comedy",
"bitcoin",
"beliefs",
"nature",
"art",
"economics",
"film & animation",
"lets play",
"games",
"sports",
"howto & style",
"game",
"cryptocurrency",
"playstation 4",
"automotive",
"crypto",
"mature",
"sony interactive entertainment",
"walkthrough",
"tutorial",
"video game",
"weapons",
"playthrough",
"pc",
"anime",
"how to",
"btc",
"fun",
"ethereum",
"food",
"travel & events",
"minecraft",
"science",
"autos & vehicles",
"play",
"politics",
"commentary",
"twitch",
"ps4live",
"love",
"ps4",
"nonprofits & activism",
"ps4share",
"fortnite",
"xbox",
"porn",
"video games",
"trump",
"español",
"money",
"music video",
"nintendo",
"movie",
"coronavirus",
"donald trump",
"steam",
"trailer",
"android",
"podcast",
"xbox one",
"survival",
"audio",
"linux",
"travel",
"funny moments",
"litecoin",
"animation",
"gamer",
"lets",
"playstation",
"bitcoin news",
"history",
"xxx",
"fox news",
"dance",
"god",
"adventure",
"liberal",
"2020",
"horror",
"government",
"freedom",
"reaction",
"meme",
"photography",
"truth",
"health",
"lbry",
"family",
"online",
"eth",
"crypto news",
"diy",
"trading",
"gold",
"memes",
"world",
"space",
"lol",
"covid-19",
"rpg",
"humor",
"democrat",
"film",
"call of duty",
"tech",
"religion",
"conspiracy",
"rap",
"cnn",
"hangoutsonair",
"unboxing",
"fiction",
"conservative",
"cars",
"hoa",
"epic",
"programming",
"progressive",
"cryptocurrency news",
"classical",
"jesus",
"movies",
"book",
"ps3",
"republican",
"fitness",
"books",
"multiplayer",
"animals",
"pokemon",
"bitcoin price",
"facebook",
"sharefactory",
"criptomonedas",
"cod",
"bible",
"business",
"stream",
"comics",
"how",
"fail",
"nsfw",
"new music",
"satire",
"pets & animals",
"computer",
"classical music",
"indie",
"musica",
"msnbc",
"fps",
"mod",
"sport",
"sony",
"ripple",
"auto",
"rock",
"marvel",
"complete",
"mining",
"political",
"mobile",
"pubg",
"hip hop",
"flat earth",
"xbox 360",
"reviews",
"vlogging",
"latest news",
"hack",
"tarot",
"iphone",
"media",
"cute",
"christian",
"free speech",
"trap",
"war",
"remix",
"ios",
"xrp",
"spirituality",
"song",
"league of legends",
"cat"
}
MATURE_TAGS = [
'nsfw', 'porn', 'xxx', 'mature', 'adult', 'sex'
]
def normalize_tag(tag):
return tag.replace(" ", "_").replace("&", "and").replace("-", "_")
COMMON_TAGS = {
tag: normalize_tag(tag) for tag in list(MOST_USED_TAGS)
}
INDEXED_LANGUAGES = [
'none',
'en',
'aa',
'ab',
'ae',
'af',
'ak',
'am',
'an',
'ar',
'as',
'av',
'ay',
'az',
'ba',
'be',
'bg',
'bh',
'bi',
'bm',
'bn',
'bo',
'br',
'bs',
'ca',
'ce',
'ch',
'co',
'cr',
'cs',
'cu',
'cv',
'cy',
'da',
'de',
'dv',
'dz',
'ee',
'el',
'eo',
'es',
'et',
'eu',
'fa',
'ff',
'fi',
'fj',
'fo',
'fr',
'fy',
'ga',
'gd',
'gl',
'gn',
'gu',
'gv',
'ha',
'he',
'hi',
'ho',
'hr',
'ht',
'hu',
'hy',
'hz',
'ia',
'id',
'ie',
'ig',
'ii',
'ik',
'io',
'is',
'it',
'iu',
'ja',
'jv',
'ka',
'kg',
'ki',
'kj',
'kk',
'kl',
'km',
'kn',
'ko',
'kr',
'ks',
'ku',
'kv',
'kw',
'ky',
'la',
'lb',
'lg',
'li',
'ln',
'lo',
'lt',
'lu',
'lv',
'mg',
'mh',
'mi',
'mk',
'ml',
'mn',
'mr',
'ms',
'mt',
'my',
'na',
'nb',
'nd',
'ne',
'ng',
'nl',
'nn',
'no',
'nr',
'nv',
'ny',
'oc',
'oj',
'om',
'or',
'os',
'pa',
'pi',
'pl',
'ps',
'pt',
'qu',
'rm',
'rn',
'ro',
'ru',
'rw',
'sa',
'sc',
'sd',
'se',
'sg',
'si',
'sk',
'sl',
'sm',
'sn',
'so',
'sq',
'sr',
'ss',
'st',
'su',
'sv',
'sw',
'ta',
'te',
'tg',
'th',
'ti',
'tk',
'tl',
'tn',
'to',
'tr',
'ts',
'tt',
'tw',
'ty',
'ug',
'uk',
'ur',
'uz',
've',
'vi',
'vo',
'wa',
'wo',
'xh',
'yi',
'yo',
'za',
'zh',
'zu'
]
class ResolveResult(typing.NamedTuple):
name: str
normalized_name: str
claim_hash: bytes
tx_num: int
position: int
tx_hash: bytes
height: int
amount: int
short_url: str
is_controlling: bool
canonical_url: str
creation_height: int
activation_height: int
expiration_height: int
effective_amount: int
support_amount: int
reposted: int
last_takeover_height: typing.Optional[int]
claims_in_channel: typing.Optional[int]
channel_hash: typing.Optional[bytes]
reposted_claim_hash: typing.Optional[bytes]
signature_valid: typing.Optional[bool]
+119
View File
@@ -0,0 +1,119 @@
import struct
from typing import Optional
from lbry.wallet.server.db import DB_PREFIXES
from lbry.wallet.server.db.revertable import RevertableOpStack, RevertablePut, RevertableDelete
class KeyValueStorage:
def get(self, key: bytes, fill_cache: bool = True) -> Optional[bytes]:
raise NotImplemented()
def iterator(self, reverse=False, start=None, stop=None, include_start=True, include_stop=False, prefix=None,
include_key=True, include_value=True, fill_cache=True):
raise NotImplemented()
def write_batch(self, transaction: bool = False):
raise NotImplemented()
def close(self):
raise NotImplemented()
@property
def closed(self) -> bool:
raise NotImplemented()
class PrefixDB:
UNDO_KEY_STRUCT = struct.Struct(b'>Q')
def __init__(self, db: KeyValueStorage, max_undo_depth: int = 200, unsafe_prefixes=None):
self._db = db
self._op_stack = RevertableOpStack(db.get, unsafe_prefixes=unsafe_prefixes)
self._max_undo_depth = max_undo_depth
def unsafe_commit(self):
"""
Write staged changes to the database without keeping undo information
Changes written cannot be undone
"""
try:
with self._db.write_batch(transaction=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
for staged_change in self._op_stack:
if staged_change.is_put:
batch_put(staged_change.key, staged_change.value)
else:
batch_delete(staged_change.key)
finally:
self._op_stack.clear()
def commit(self, height: int):
"""
Write changes for a block height to the database and keep undo information so that the changes can be reverted
"""
undo_ops = self._op_stack.get_undo_ops()
delete_undos = []
if height > self._max_undo_depth:
delete_undos.extend(self._db.iterator(
start=DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(0),
stop=DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(height - self._max_undo_depth),
include_value=False
))
try:
with self._db.write_batch(transaction=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
for staged_change in self._op_stack:
if staged_change.is_put:
batch_put(staged_change.key, staged_change.value)
else:
batch_delete(staged_change.key)
for undo_to_delete in delete_undos:
batch_delete(undo_to_delete)
batch_put(DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(height), undo_ops)
finally:
self._op_stack.clear()
def rollback(self, height: int):
"""
Revert changes for a block height
"""
undo_key = DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(height)
self._op_stack.apply_packed_undo_ops(self._db.get(undo_key))
try:
with self._db.write_batch(transaction=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
for staged_change in self._op_stack:
if staged_change.is_put:
batch_put(staged_change.key, staged_change.value)
else:
batch_delete(staged_change.key)
batch_delete(undo_key)
finally:
self._op_stack.clear()
def get(self, key: bytes, fill_cache: bool = True) -> Optional[bytes]:
return self._db.get(key, fill_cache=fill_cache)
def iterator(self, reverse=False, start=None, stop=None, include_start=True, include_stop=False, prefix=None,
include_key=True, include_value=True, fill_cache=True):
return self._db.iterator(
reverse=reverse, start=start, stop=stop, include_start=include_start, include_stop=include_stop,
prefix=prefix, include_key=include_key, include_value=include_value, fill_cache=fill_cache
)
def close(self):
if not self._db.closed:
self._db.close()
@property
def closed(self):
return self._db.closed
def stage_raw_put(self, key: bytes, value: bytes):
self._op_stack.append_op(RevertablePut(key, value))
def stage_raw_delete(self, key: bytes, value: bytes):
self._op_stack.append_op(RevertableDelete(key, value))
@@ -0,0 +1 @@
from .search import SearchIndex
@@ -0,0 +1,100 @@
INDEX_DEFAULT_SETTINGS = {
"settings":
{"analysis":
{"analyzer": {
"default": {"tokenizer": "whitespace", "filter": ["lowercase", "porter_stem"]}}},
"index":
{"refresh_interval": -1,
"number_of_shards": 1,
"number_of_replicas": 0,
"sort": {
"field": ["trending_score", "release_time"],
"order": ["desc", "desc"]
}}
},
"mappings": {
"properties": {
"claim_id": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text",
"index_prefixes": {
"min_chars": 1,
"max_chars": 10
}
},
"sd_hash": {
"fields": {
"keyword": {
"ignore_above": 96,
"type": "keyword"
}
},
"type": "text",
"index_prefixes": {
"min_chars": 1,
"max_chars": 4
}
},
"height": {"type": "integer"},
"claim_type": {"type": "byte"},
"censor_type": {"type": "byte"},
"trending_score": {"type": "double"},
"release_time": {"type": "long"}
}
}
}
FIELDS = {
'_id',
'claim_id', 'claim_type', 'claim_name', 'normalized_name',
'tx_id', 'tx_nout', 'tx_position',
'short_url', 'canonical_url',
'is_controlling', 'last_take_over_height',
'public_key_bytes', 'public_key_id', 'claims_in_channel',
'channel_id', 'signature', 'signature_digest', 'is_signature_valid',
'amount', 'effective_amount', 'support_amount',
'fee_amount', 'fee_currency',
'height', 'creation_height', 'activation_height', 'expiration_height',
'stream_type', 'media_type', 'censor_type',
'title', 'author', 'description',
'timestamp', 'creation_timestamp',
'duration', 'release_time',
'tags', 'languages', 'has_source', 'reposted_claim_type',
'reposted_claim_id', 'repost_count', 'sd_hash',
'trending_score', 'tx_num'
}
TEXT_FIELDS = {'author', 'canonical_url', 'channel_id', 'description', 'claim_id', 'censoring_channel_id',
'media_type', 'normalized_name', 'public_key_bytes', 'public_key_id', 'short_url', 'signature',
'claim_name', 'signature_digest', 'title', 'tx_id', 'fee_currency', 'reposted_claim_id',
'tags', 'sd_hash'}
RANGE_FIELDS = {
'height', 'creation_height', 'activation_height', 'expiration_height',
'timestamp', 'creation_timestamp', 'duration', 'release_time', 'fee_amount',
'tx_position', 'repost_count', 'limit_claims_per_channel',
'amount', 'effective_amount', 'support_amount',
'trending_score', 'censor_type', 'tx_num'
}
ALL_FIELDS = RANGE_FIELDS | TEXT_FIELDS | FIELDS
REPLACEMENTS = {
'claim_name': 'normalized_name',
'name': 'normalized_name',
'txid': 'tx_id',
'nout': 'tx_nout',
'trending_group': 'trending_score',
'trending_mixed': 'trending_score',
'trending_global': 'trending_score',
'trending_local': 'trending_score',
'reposted': 'repost_count',
'stream_types': 'stream_type',
'media_types': 'media_type',
'valid_channel_signature': 'is_signature_valid'
}
@@ -0,0 +1,726 @@
import time
import asyncio
import struct
from binascii import unhexlify
from collections import Counter, deque
from decimal import Decimal
from operator import itemgetter
from typing import Optional, List, Iterable, Union
from elasticsearch import AsyncElasticsearch, NotFoundError, ConnectionError
from elasticsearch.helpers import async_streaming_bulk
from lbry.error import ResolveCensoredError, TooManyClaimSearchParametersError
from lbry.schema.result import Outputs, Censor
from lbry.schema.tags import clean_tags
from lbry.schema.url import URL, normalize_name
from lbry.utils import LRUCache
from lbry.wallet.server.db.common import CLAIM_TYPES, STREAM_TYPES
from lbry.wallet.server.db.elasticsearch.constants import INDEX_DEFAULT_SETTINGS, REPLACEMENTS, FIELDS, TEXT_FIELDS, \
RANGE_FIELDS, ALL_FIELDS
from lbry.wallet.server.util import class_logger
from lbry.wallet.server.db.common import ResolveResult
class ChannelResolution(str):
@classmethod
def lookup_error(cls, url):
return LookupError(f'Could not find channel in "{url}".')
class StreamResolution(str):
@classmethod
def lookup_error(cls, url):
return LookupError(f'Could not find claim at "{url}".')
class IndexVersionMismatch(Exception):
def __init__(self, got_version, expected_version):
self.got_version = got_version
self.expected_version = expected_version
class SearchIndex:
VERSION = 1
def __init__(self, index_prefix: str, search_timeout=3.0, elastic_host='localhost', elastic_port=9200):
self.search_timeout = search_timeout
self.sync_timeout = 600 # wont hit that 99% of the time, but can hit on a fresh import
self.search_client: Optional[AsyncElasticsearch] = None
self.sync_client: Optional[AsyncElasticsearch] = None
self.index = index_prefix + 'claims'
self.logger = class_logger(__name__, self.__class__.__name__)
self.claim_cache = LRUCache(2 ** 15)
self.search_cache = LRUCache(2 ** 17)
self._elastic_host = elastic_host
self._elastic_port = elastic_port
async def get_index_version(self) -> int:
try:
template = await self.sync_client.indices.get_template(self.index)
return template[self.index]['version']
except NotFoundError:
return 0
async def set_index_version(self, version):
await self.sync_client.indices.put_template(
self.index, body={'version': version, 'index_patterns': ['ignored']}, ignore=400
)
async def start(self) -> bool:
if self.sync_client:
return False
hosts = [{'host': self._elastic_host, 'port': self._elastic_port}]
self.sync_client = AsyncElasticsearch(hosts, timeout=self.sync_timeout)
self.search_client = AsyncElasticsearch(hosts, timeout=self.search_timeout)
while True:
try:
await self.sync_client.cluster.health(wait_for_status='yellow')
break
except ConnectionError:
self.logger.warning("Failed to connect to Elasticsearch. Waiting for it!")
await asyncio.sleep(1)
res = await self.sync_client.indices.create(self.index, INDEX_DEFAULT_SETTINGS, ignore=400)
acked = res.get('acknowledged', False)
if acked:
await self.set_index_version(self.VERSION)
return acked
index_version = await self.get_index_version()
if index_version != self.VERSION:
self.logger.error("es search index has an incompatible version: %s vs %s", index_version, self.VERSION)
raise IndexVersionMismatch(index_version, self.VERSION)
await self.sync_client.indices.refresh(self.index)
return acked
def stop(self):
clients = [self.sync_client, self.search_client]
self.sync_client, self.search_client = None, None
return asyncio.ensure_future(asyncio.gather(*(client.close() for client in clients)))
def delete_index(self):
return self.sync_client.indices.delete(self.index, ignore_unavailable=True)
async def _consume_claim_producer(self, claim_producer):
count = 0
async for op, doc in claim_producer:
if op == 'delete':
yield {
'_index': self.index,
'_op_type': 'delete',
'_id': doc
}
else:
yield {
'doc': {key: value for key, value in doc.items() if key in ALL_FIELDS},
'_id': doc['claim_id'],
'_index': self.index,
'_op_type': 'update',
'doc_as_upsert': True
}
count += 1
if count % 100 == 0:
self.logger.info("Indexing in progress, %d claims.", count)
if count:
self.logger.info("Indexing done for %d claims.", count)
else:
self.logger.debug("Indexing done for %d claims.", count)
async def claim_consumer(self, claim_producer):
touched = set()
async for ok, item in async_streaming_bulk(self.sync_client, self._consume_claim_producer(claim_producer),
raise_on_error=False):
if not ok:
self.logger.warning("indexing failed for an item: %s", item)
else:
item = item.popitem()[1]
touched.add(item['_id'])
await self.sync_client.indices.refresh(self.index)
self.logger.debug("Indexing done.")
def update_filter_query(self, censor_type, blockdict, channels=False):
blockdict = {blocked.hex(): blocker.hex() for blocked, blocker in blockdict.items()}
if channels:
update = expand_query(channel_id__in=list(blockdict.keys()), censor_type=f"<{censor_type}")
else:
update = expand_query(claim_id__in=list(blockdict.keys()), censor_type=f"<{censor_type}")
key = 'channel_id' if channels else 'claim_id'
update['script'] = {
"source": f"ctx._source.censor_type={censor_type}; "
f"ctx._source.censoring_channel_id=params[ctx._source.{key}];",
"lang": "painless",
"params": blockdict
}
return update
async def update_trending_score(self, params):
update_trending_score_script = """
double softenLBC(double lbc) { return (Math.pow(lbc, 1.0 / 3.0)); }
double logsumexp(double x, double y)
{
double top;
if(x > y)
top = x;
else
top = y;
double result = top + Math.log(Math.exp(x-top) + Math.exp(y-top));
return(result);
}
double logdiffexp(double big, double small)
{
return big + Math.log(1.0 - Math.exp(small - big));
}
double squash(double x)
{
if(x < 0.0)
return -Math.log(1.0 - x);
else
return Math.log(x + 1.0);
}
double unsquash(double x)
{
if(x < 0.0)
return 1.0 - Math.exp(-x);
else
return Math.exp(x) - 1.0;
}
double log_to_squash(double x)
{
return logsumexp(x, 0.0);
}
double squash_to_log(double x)
{
//assert x > 0.0;
return logdiffexp(x, 0.0);
}
double squashed_add(double x, double y)
{
// squash(unsquash(x) + unsquash(y)) but avoiding overflow.
// Cases where the signs are the same
if (x < 0.0 && y < 0.0)
return -logsumexp(-x, logdiffexp(-y, 0.0));
if (x >= 0.0 && y >= 0.0)
return logsumexp(x, logdiffexp(y, 0.0));
// Where the signs differ
if (x >= 0.0 && y < 0.0)
if (Math.abs(x) >= Math.abs(y))
return logsumexp(0.0, logdiffexp(x, -y));
else
return -logsumexp(0.0, logdiffexp(-y, x));
if (x < 0.0 && y >= 0.0)
{
// Addition is commutative, hooray for new math
return squashed_add(y, x);
}
return 0.0;
}
double squashed_multiply(double x, double y)
{
// squash(unsquash(x)*unsquash(y)) but avoiding overflow.
int sign;
if(x*y >= 0.0)
sign = 1;
else
sign = -1;
return sign*logsumexp(squash_to_log(Math.abs(x))
+ squash_to_log(Math.abs(y)), 0.0);
}
// Squashed inflated units
double inflateUnits(int height) {
double timescale = 576.0; // Half life of 400 = e-folding time of a day
// by coincidence, so may as well go with it
return log_to_squash(height / timescale);
}
double spikePower(double newAmount) {
if (newAmount < 50.0) {
return(0.5);
} else if (newAmount < 85.0) {
return(newAmount / 100.0);
} else {
return(0.85);
}
}
double spikeMass(double oldAmount, double newAmount) {
double softenedChange = softenLBC(Math.abs(newAmount - oldAmount));
double changeInSoftened = Math.abs(softenLBC(newAmount) - softenLBC(oldAmount));
double power = spikePower(newAmount);
if (oldAmount > newAmount) {
-1.0 * Math.pow(changeInSoftened, power) * Math.pow(softenedChange, 1.0 - power)
} else {
Math.pow(changeInSoftened, power) * Math.pow(softenedChange, 1.0 - power)
}
}
for (i in params.src.changes) {
double units = inflateUnits(i.height);
if (ctx._source.trending_score == null) {
ctx._source.trending_score = 0.0;
}
double bigSpike = squashed_multiply(units, squash(spikeMass(i.prev_amount, i.new_amount)));
ctx._source.trending_score = squashed_add(ctx._source.trending_score, bigSpike);
}
"""
start = time.perf_counter()
def producer():
for claim_id, claim_updates in params.items():
yield {
'_id': claim_id,
'_index': self.index,
'_op_type': 'update',
'script': {
'lang': 'painless',
'source': update_trending_score_script,
'params': {'src': {
'changes': [
{
'height': p.height,
'prev_amount': p.prev_amount / 1E8,
'new_amount': p.new_amount / 1E8,
} for p in claim_updates
]
}}
},
}
if not params:
return
async for ok, item in async_streaming_bulk(self.sync_client, producer(), raise_on_error=False):
if not ok:
self.logger.warning("updating trending failed for an item: %s", item)
await self.sync_client.indices.refresh(self.index)
self.logger.info("updated trending scores in %ims", int((time.perf_counter() - start) * 1000))
async def apply_filters(self, blocked_streams, blocked_channels, filtered_streams, filtered_channels):
if filtered_streams:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.SEARCH, filtered_streams), slices=4)
await self.sync_client.indices.refresh(self.index)
if filtered_channels:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.SEARCH, filtered_channels), slices=4)
await self.sync_client.indices.refresh(self.index)
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.SEARCH, filtered_channels, True), slices=4)
await self.sync_client.indices.refresh(self.index)
if blocked_streams:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_streams), slices=4)
await self.sync_client.indices.refresh(self.index)
if blocked_channels:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_channels), slices=4)
await self.sync_client.indices.refresh(self.index)
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_channels, True), slices=4)
await self.sync_client.indices.refresh(self.index)
self.clear_caches()
def clear_caches(self):
self.search_cache.clear()
self.claim_cache.clear()
async def cached_search(self, kwargs):
total_referenced = []
cache_item = ResultCacheItem.from_cache(str(kwargs), self.search_cache)
if cache_item.result is not None:
return cache_item.result
async with cache_item.lock:
if cache_item.result:
return cache_item.result
censor = Censor(Censor.SEARCH)
if kwargs.get('no_totals'):
response, offset, total = await self.search(**kwargs, censor_type=Censor.NOT_CENSORED)
else:
response, offset, total = await self.search(**kwargs)
censor.apply(response)
total_referenced.extend(response)
if censor.censored:
response, _, _ = await self.search(**kwargs, censor_type=Censor.NOT_CENSORED)
total_referenced.extend(response)
response = [
ResolveResult(
name=r['claim_name'],
normalized_name=r['normalized_name'],
claim_hash=r['claim_hash'],
tx_num=r['tx_num'],
position=r['tx_nout'],
tx_hash=r['tx_hash'],
height=r['height'],
amount=r['amount'],
short_url=r['short_url'],
is_controlling=r['is_controlling'],
canonical_url=r['canonical_url'],
creation_height=r['creation_height'],
activation_height=r['activation_height'],
expiration_height=r['expiration_height'],
effective_amount=r['effective_amount'],
support_amount=r['support_amount'],
last_takeover_height=r['last_take_over_height'],
claims_in_channel=r['claims_in_channel'],
channel_hash=r['channel_hash'],
reposted_claim_hash=r['reposted_claim_hash'],
reposted=r['reposted'],
signature_valid=r['signature_valid']
) for r in response
]
extra = [
ResolveResult(
name=r['claim_name'],
normalized_name=r['normalized_name'],
claim_hash=r['claim_hash'],
tx_num=r['tx_num'],
position=r['tx_nout'],
tx_hash=r['tx_hash'],
height=r['height'],
amount=r['amount'],
short_url=r['short_url'],
is_controlling=r['is_controlling'],
canonical_url=r['canonical_url'],
creation_height=r['creation_height'],
activation_height=r['activation_height'],
expiration_height=r['expiration_height'],
effective_amount=r['effective_amount'],
support_amount=r['support_amount'],
last_takeover_height=r['last_take_over_height'],
claims_in_channel=r['claims_in_channel'],
channel_hash=r['channel_hash'],
reposted_claim_hash=r['reposted_claim_hash'],
reposted=r['reposted'],
signature_valid=r['signature_valid']
) for r in await self._get_referenced_rows(total_referenced)
]
result = Outputs.to_base64(
response, extra, offset, total, censor
)
cache_item.result = result
return result
async def get_many(self, *claim_ids):
await self.populate_claim_cache(*claim_ids)
return filter(None, map(self.claim_cache.get, claim_ids))
async def populate_claim_cache(self, *claim_ids):
missing = [claim_id for claim_id in claim_ids if self.claim_cache.get(claim_id) is None]
if missing:
results = await self.search_client.mget(
index=self.index, body={"ids": missing}
)
for result in expand_result(filter(lambda doc: doc['found'], results["docs"])):
self.claim_cache.set(result['claim_id'], result)
async def search(self, **kwargs):
try:
return await self.search_ahead(**kwargs)
except NotFoundError:
return [], 0, 0
# return expand_result(result['hits']), 0, result.get('total', {}).get('value', 0)
async def search_ahead(self, **kwargs):
# 'limit_claims_per_channel' case. Fetch 1000 results, reorder, slice, inflate and return
per_channel_per_page = kwargs.pop('limit_claims_per_channel', 0) or 0
remove_duplicates = kwargs.pop('remove_duplicates', False)
page_size = kwargs.pop('limit', 10)
offset = kwargs.pop('offset', 0)
kwargs['limit'] = 1000
cache_item = ResultCacheItem.from_cache(f"ahead{per_channel_per_page}{kwargs}", self.search_cache)
if cache_item.result is not None:
reordered_hits = cache_item.result
else:
async with cache_item.lock:
if cache_item.result:
reordered_hits = cache_item.result
else:
query = expand_query(**kwargs)
search_hits = deque((await self.search_client.search(
query, index=self.index, track_total_hits=False,
_source_includes=['_id', 'channel_id', 'reposted_claim_id', 'creation_height']
))['hits']['hits'])
if remove_duplicates:
search_hits = self.__remove_duplicates(search_hits)
if per_channel_per_page > 0:
reordered_hits = self.__search_ahead(search_hits, page_size, per_channel_per_page)
else:
reordered_hits = [(hit['_id'], hit['_source']['channel_id']) for hit in search_hits]
cache_item.result = reordered_hits
result = list(await self.get_many(*(claim_id for claim_id, _ in reordered_hits[offset:(offset + page_size)])))
return result, 0, len(reordered_hits)
def __remove_duplicates(self, search_hits: deque) -> deque:
known_ids = {} # claim_id -> (creation_height, hit_id), where hit_id is either reposted claim id or original
dropped = set()
for hit in search_hits:
hit_height, hit_id = hit['_source']['creation_height'], hit['_source']['reposted_claim_id'] or hit['_id']
if hit_id not in known_ids:
known_ids[hit_id] = (hit_height, hit['_id'])
else:
previous_height, previous_id = known_ids[hit_id]
if hit_height < previous_height:
known_ids[hit_id] = (hit_height, hit['_id'])
dropped.add(previous_id)
else:
dropped.add(hit['_id'])
return deque(hit for hit in search_hits if hit['_id'] not in dropped)
def __search_ahead(self, search_hits: list, page_size: int, per_channel_per_page: int):
reordered_hits = []
channel_counters = Counter()
next_page_hits_maybe_check_later = deque()
while search_hits or next_page_hits_maybe_check_later:
if reordered_hits and len(reordered_hits) % page_size == 0:
channel_counters.clear()
elif not reordered_hits:
pass
else:
break # means last page was incomplete and we are left with bad replacements
for _ in range(len(next_page_hits_maybe_check_later)):
claim_id, channel_id = next_page_hits_maybe_check_later.popleft()
if per_channel_per_page > 0 and channel_counters[channel_id] < per_channel_per_page:
reordered_hits.append((claim_id, channel_id))
channel_counters[channel_id] += 1
else:
next_page_hits_maybe_check_later.append((claim_id, channel_id))
while search_hits:
hit = search_hits.popleft()
hit_id, hit_channel_id = hit['_id'], hit['_source']['channel_id']
if hit_channel_id is None or per_channel_per_page <= 0:
reordered_hits.append((hit_id, hit_channel_id))
elif channel_counters[hit_channel_id] < per_channel_per_page:
reordered_hits.append((hit_id, hit_channel_id))
channel_counters[hit_channel_id] += 1
if len(reordered_hits) % page_size == 0:
break
else:
next_page_hits_maybe_check_later.append((hit_id, hit_channel_id))
return reordered_hits
async def _get_referenced_rows(self, txo_rows: List[dict]):
txo_rows = [row for row in txo_rows if isinstance(row, dict)]
referenced_ids = set(filter(None, map(itemgetter('reposted_claim_id'), txo_rows)))
referenced_ids |= set(filter(None, (row['channel_id'] for row in txo_rows)))
referenced_ids |= set(filter(None, (row['censoring_channel_id'] for row in txo_rows)))
referenced_txos = []
if referenced_ids:
referenced_txos.extend(await self.get_many(*referenced_ids))
referenced_ids = set(filter(None, (row['channel_id'] for row in referenced_txos)))
if referenced_ids:
referenced_txos.extend(await self.get_many(*referenced_ids))
return referenced_txos
def expand_query(**kwargs):
if "amount_order" in kwargs:
kwargs["limit"] = 1
kwargs["order_by"] = "effective_amount"
kwargs["offset"] = int(kwargs["amount_order"]) - 1
if 'name' in kwargs:
kwargs['name'] = normalize_name(kwargs.pop('name'))
if kwargs.get('is_controlling') is False:
kwargs.pop('is_controlling')
query = {'must': [], 'must_not': []}
collapse = None
if 'fee_currency' in kwargs and kwargs['fee_currency'] is not None:
kwargs['fee_currency'] = kwargs['fee_currency'].upper()
for key, value in kwargs.items():
key = key.replace('claim.', '')
many = key.endswith('__in') or isinstance(value, list)
if many and len(value) > 2048:
raise TooManyClaimSearchParametersError(key, 2048)
if many:
key = key.replace('__in', '')
value = list(filter(None, value))
if value is None or isinstance(value, list) and len(value) == 0:
continue
key = REPLACEMENTS.get(key, key)
if key in FIELDS:
partial_id = False
if key == 'claim_type':
if isinstance(value, str):
value = CLAIM_TYPES[value]
else:
value = [CLAIM_TYPES[claim_type] for claim_type in value]
elif key == 'stream_type':
value = [STREAM_TYPES[value]] if isinstance(value, str) else list(map(STREAM_TYPES.get, value))
if key == '_id':
if isinstance(value, Iterable):
value = [item[::-1].hex() for item in value]
else:
value = value[::-1].hex()
if not many and key in ('_id', 'claim_id', 'sd_hash') and len(value) < 20:
partial_id = True
if key in ('signature_valid', 'has_source'):
continue # handled later
if key in TEXT_FIELDS:
key += '.keyword'
ops = {'<=': 'lte', '>=': 'gte', '<': 'lt', '>': 'gt'}
if partial_id:
query['must'].append({"prefix": {key: value}})
elif key in RANGE_FIELDS and isinstance(value, str) and value[0] in ops:
operator_length = 2 if value[:2] in ops else 1
operator, value = value[:operator_length], value[operator_length:]
if key == 'fee_amount':
value = str(Decimal(value)*1000)
query['must'].append({"range": {key: {ops[operator]: value}}})
elif key in RANGE_FIELDS and isinstance(value, list) and all(v[0] in ops for v in value):
range_constraints = []
for v in value:
operator_length = 2 if v[:2] in ops else 1
operator, stripped_op_v = v[:operator_length], v[operator_length:]
if key == 'fee_amount':
stripped_op_v = str(Decimal(stripped_op_v)*1000)
range_constraints.append((operator, stripped_op_v))
query['must'].append({"range": {key: {ops[operator]: v for operator, v in range_constraints}}})
elif many:
query['must'].append({"terms": {key: value}})
else:
if key == 'fee_amount':
value = str(Decimal(value)*1000)
query['must'].append({"term": {key: {"value": value}}})
elif key == 'not_channel_ids':
for channel_id in value:
query['must_not'].append({"term": {'channel_id.keyword': channel_id}})
query['must_not'].append({"term": {'_id': channel_id}})
elif key == 'channel_ids':
query['must'].append({"terms": {'channel_id.keyword': value}})
elif key == 'claim_ids':
query['must'].append({"terms": {'claim_id.keyword': value}})
elif key == 'media_types':
query['must'].append({"terms": {'media_type.keyword': value}})
elif key == 'any_languages':
query['must'].append({"terms": {'languages': clean_tags(value)}})
elif key == 'any_languages':
query['must'].append({"terms": {'languages': value}})
elif key == 'all_languages':
query['must'].extend([{"term": {'languages': tag}} for tag in value])
elif key == 'any_tags':
query['must'].append({"terms": {'tags.keyword': clean_tags(value)}})
elif key == 'all_tags':
query['must'].extend([{"term": {'tags.keyword': tag}} for tag in clean_tags(value)])
elif key == 'not_tags':
query['must_not'].extend([{"term": {'tags.keyword': tag}} for tag in clean_tags(value)])
elif key == 'not_claim_id':
query['must_not'].extend([{"term": {'claim_id.keyword': cid}} for cid in value])
elif key == 'limit_claims_per_channel':
collapse = ('channel_id.keyword', value)
if kwargs.get('has_channel_signature'):
query['must'].append({"exists": {"field": "signature"}})
if 'signature_valid' in kwargs:
query['must'].append({"term": {"is_signature_valid": bool(kwargs["signature_valid"])}})
elif 'signature_valid' in kwargs:
query.setdefault('should', [])
query["minimum_should_match"] = 1
query['should'].append({"bool": {"must_not": {"exists": {"field": "signature"}}}})
query['should'].append({"term": {"is_signature_valid": bool(kwargs["signature_valid"])}})
if 'has_source' in kwargs:
query.setdefault('should', [])
query["minimum_should_match"] = 1
is_stream_or_repost = {"terms": {"claim_type": [CLAIM_TYPES['stream'], CLAIM_TYPES['repost']]}}
query['should'].append(
{"bool": {"must": [{"match": {"has_source": kwargs['has_source']}}, is_stream_or_repost]}})
query['should'].append({"bool": {"must_not": [is_stream_or_repost]}})
query['should'].append({"bool": {"must": [{"term": {"reposted_claim_type": CLAIM_TYPES['channel']}}]}})
if kwargs.get('text'):
query['must'].append(
{"simple_query_string":
{"query": kwargs["text"], "fields": [
"claim_name^4", "channel_name^8", "title^1", "description^.5", "author^1", "tags^.5"
]}})
query = {
"_source": {"excludes": ["description", "title"]},
'query': {'bool': query},
"sort": [],
}
if "limit" in kwargs:
query["size"] = kwargs["limit"]
if 'offset' in kwargs:
query["from"] = kwargs["offset"]
if 'order_by' in kwargs:
if isinstance(kwargs["order_by"], str):
kwargs["order_by"] = [kwargs["order_by"]]
for value in kwargs['order_by']:
if 'trending_group' in value:
# fixme: trending_mixed is 0 for all records on variable decay, making sort slow.
continue
is_asc = value.startswith('^')
value = value[1:] if is_asc else value
value = REPLACEMENTS.get(value, value)
if value in TEXT_FIELDS:
value += '.keyword'
query['sort'].append({value: "asc" if is_asc else "desc"})
if collapse:
query["collapse"] = {
"field": collapse[0],
"inner_hits": {
"name": collapse[0],
"size": collapse[1],
"sort": query["sort"]
}
}
return query
def expand_result(results):
inner_hits = []
expanded = []
for result in results:
if result.get("inner_hits"):
for _, inner_hit in result["inner_hits"].items():
inner_hits.extend(inner_hit["hits"]["hits"])
continue
result = result['_source']
result['claim_hash'] = unhexlify(result['claim_id'])[::-1]
if result['reposted_claim_id']:
result['reposted_claim_hash'] = unhexlify(result['reposted_claim_id'])[::-1]
else:
result['reposted_claim_hash'] = None
result['channel_hash'] = unhexlify(result['channel_id'])[::-1] if result['channel_id'] else None
result['txo_hash'] = unhexlify(result['tx_id'])[::-1] + struct.pack('<I', result['tx_nout'])
result['tx_hash'] = unhexlify(result['tx_id'])[::-1]
result['reposted'] = result.pop('repost_count')
result['signature_valid'] = result.pop('is_signature_valid')
# result['normalized'] = result.pop('normalized_name')
# if result['censoring_channel_hash']:
# result['censoring_channel_hash'] = unhexlify(result['censoring_channel_hash'])[::-1]
expanded.append(result)
if inner_hits:
return expand_result(inner_hits)
return expanded
class ResultCacheItem:
__slots__ = '_result', 'lock', 'has_result'
def __init__(self):
self.has_result = asyncio.Event()
self.lock = asyncio.Lock()
self._result = None
@property
def result(self) -> str:
return self._result
@result.setter
def result(self, result: str):
self._result = result
if result is not None:
self.has_result.set()
@classmethod
def from_cache(cls, cache_key, cache):
cache_item = cache.get(cache_key)
if cache_item is None:
cache_item = cache[cache_key] = ResultCacheItem()
return cache_item
+138
View File
@@ -0,0 +1,138 @@
import os
import argparse
import asyncio
import logging
from elasticsearch import AsyncElasticsearch
from elasticsearch.helpers import async_streaming_bulk
from lbry.wallet.server.env import Env
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.db.elasticsearch.search import SearchIndex, IndexVersionMismatch
from lbry.wallet.server.db.elasticsearch.constants import ALL_FIELDS
async def get_recent_claims(env, index_name='claims', db=None):
log = logging.getLogger()
need_open = db is None
db = db or LevelDB(env)
try:
if need_open:
db.open_db()
if db.es_sync_height == db.db_height or db.db_height <= 0:
return
if need_open:
await db.initialize_caches()
log.info(f"catching up ES ({db.es_sync_height}) to leveldb height: {db.db_height}")
cnt = 0
touched_claims = set()
deleted_claims = set()
for height in range(db.es_sync_height, db.db_height + 1):
touched_or_deleted = db.prefix_db.touched_or_deleted.get(height)
touched_claims.update(touched_or_deleted.touched_claims)
deleted_claims.update(touched_or_deleted.deleted_claims)
touched_claims.difference_update(deleted_claims)
for deleted in deleted_claims:
yield {
'_index': index_name,
'_op_type': 'delete',
'_id': deleted.hex()
}
for touched in touched_claims:
claim = db.claim_producer(touched)
if claim:
yield {
'doc': {key: value for key, value in claim.items() if key in ALL_FIELDS},
'_id': claim['claim_id'],
'_index': index_name,
'_op_type': 'update',
'doc_as_upsert': True
}
cnt += 1
else:
logging.warning("could not sync claim %s", touched.hex())
if cnt % 10000 == 0:
logging.info("%i claims sent to ES", cnt)
db.es_sync_height = db.db_height
db.write_db_state()
db.prefix_db.unsafe_commit()
db.assert_db_state()
logging.info("finished sending %i claims to ES, deleted %i", cnt, len(deleted_claims))
finally:
if need_open:
db.close()
async def get_all_claims(env, index_name='claims', db=None):
need_open = db is None
db = db or LevelDB(env)
if need_open:
db.open_db()
await db.initialize_caches()
logging.info("Fetching claims to send ES from leveldb")
try:
cnt = 0
async for claim in db.all_claims_producer():
yield {
'doc': {key: value for key, value in claim.items() if key in ALL_FIELDS},
'_id': claim['claim_id'],
'_index': index_name,
'_op_type': 'update',
'doc_as_upsert': True
}
cnt += 1
if cnt % 10000 == 0:
logging.info("sent %i claims to ES", cnt)
finally:
if need_open:
db.close()
async def make_es_index_and_run_sync(env: Env, clients=32, force=False, db=None, index_name='claims'):
index = SearchIndex(env.es_index_prefix, elastic_host=env.elastic_host, elastic_port=env.elastic_port)
logging.info("ES sync host: %s:%i", env.elastic_host, env.elastic_port)
try:
created = await index.start()
except IndexVersionMismatch as err:
logging.info(
"dropping ES search index (version %s) for upgrade to version %s", err.got_version, err.expected_version
)
await index.delete_index()
await index.stop()
created = await index.start()
finally:
index.stop()
es = AsyncElasticsearch([{'host': env.elastic_host, 'port': env.elastic_port}])
if force or created:
claim_generator = get_all_claims(env, index_name=index_name, db=db)
else:
claim_generator = get_recent_claims(env, index_name=index_name, db=db)
try:
async for ok, item in async_streaming_bulk(es, claim_generator, request_timeout=600, raise_on_error=False):
if not ok:
logging.warning("indexing failed for an item: %s", item)
await es.indices.refresh(index=index_name)
finally:
await es.close()
def run_elastic_sync():
logging.basicConfig(level=logging.INFO)
logging.getLogger('aiohttp').setLevel(logging.WARNING)
logging.getLogger('elasticsearch').setLevel(logging.WARNING)
logging.info('lbry.server starting')
parser = argparse.ArgumentParser(prog="lbry-hub-elastic-sync")
parser.add_argument("-c", "--clients", type=int, default=32)
parser.add_argument("-f", "--force", default=False, action='store_true')
Env.contribute_to_arg_parser(parser)
args = parser.parse_args()
env = Env.from_arg_parser(args)
if not os.path.exists(os.path.join(args.db_dir, 'lbry-leveldb')):
logging.info("DB path doesnt exist, nothing to sync to ES")
return
asyncio.run(make_es_index_and_run_sync(env, clients=args.clients, force=args.force))
File diff suppressed because it is too large. Load diff
+175
View File
@@ -0,0 +1,175 @@
import struct
import logging
from string import printable
from collections import defaultdict
from typing import Tuple, Iterable, Callable, Optional
from lbry.wallet.server.db import DB_PREFIXES
_OP_STRUCT = struct.Struct('>BLL')
log = logging.getLogger()
class RevertableOp:
__slots__ = [
'key',
'value',
]
is_put = 0
def __init__(self, key: bytes, value: bytes):
self.key = key
self.value = value
@property
def is_delete(self) -> bool:
return not self.is_put
def invert(self) -> 'RevertableOp':
raise NotImplementedError()
def pack(self) -> bytes:
"""
Serialize to bytes
"""
return struct.pack(
f'>BLL{len(self.key)}s{len(self.value)}s', int(self.is_put), len(self.key), len(self.value), self.key,
self.value
)
@classmethod
def unpack(cls, packed: bytes) -> Tuple['RevertableOp', bytes]:
"""
Deserialize from bytes
:param packed: bytes containing at least one packed revertable op
:return: tuple of the deserialized op (a put or a delete) and the remaining serialized bytes
"""
is_put, key_len, val_len = _OP_STRUCT.unpack(packed[:9])
key = packed[9:9 + key_len]
value = packed[9 + key_len:9 + key_len + val_len]
if is_put == 1:
return RevertablePut(key, value), packed[9 + key_len + val_len:]
return RevertableDelete(key, value), packed[9 + key_len + val_len:]
def __eq__(self, other: 'RevertableOp') -> bool:
return (self.is_put, self.key, self.value) == (other.is_put, other.key, other.value)
def __repr__(self) -> str:
return str(self)
def __str__(self) -> str:
from lbry.wallet.server.db.prefixes import auto_decode_item
k, v = auto_decode_item(self.key, self.value)
key = ''.join(c if c in printable else '.' for c in str(k))
val = ''.join(c if c in printable else '.' for c in str(v))
return f"{'PUT' if self.is_put else 'DELETE'} {DB_PREFIXES(self.key[:1]).name}: {key} | {val}"
class RevertableDelete(RevertableOp):
def invert(self):
return RevertablePut(self.key, self.value)
class RevertablePut(RevertableOp):
is_put = True
def invert(self):
return RevertableDelete(self.key, self.value)
class OpStackIntegrity(Exception):
pass
class RevertableOpStack:
def __init__(self, get_fn: Callable[[bytes], Optional[bytes]], unsafe_prefixes=None):
"""
This represents a sequence of revertable puts and deletes to a key-value database that checks for integrity
violations when applying the puts and deletes. The integrity checks assure that keys that do not exist
are not deleted, and that when keys are deleted the current value is correctly known so that the delete
may be undone. When putting values, the integrity checks assure that existing values are not overwritten
without first being deleted. Updates are performed by applying a delete op for the old value and a put op
for the new value.
:param get_fn: getter function from an object implementing `KeyValueStorage`
:param unsafe_prefixes: optional set of prefixes to ignore integrity errors for, violations are still logged
"""
self._get = get_fn
self._items = defaultdict(list)
self._unsafe_prefixes = unsafe_prefixes or set()
def append_op(self, op: RevertableOp):
"""
Apply a put or delete op, checking that it introduces no integrity errors
"""
inverted = op.invert()
if self._items[op.key] and inverted == self._items[op.key][-1]:
self._items[op.key].pop() # if the new op is the inverse of the last op, we can safely null both
return
elif self._items[op.key] and self._items[op.key][-1] == op: # duplicate of last op
return # raise an error?
stored_val = self._get(op.key)
has_stored_val = stored_val is not None
delete_stored_op = None if not has_stored_val else RevertableDelete(op.key, stored_val)
will_delete_existing_stored = False if delete_stored_op is None else (delete_stored_op in self._items[op.key])
try:
if op.is_put and has_stored_val and not will_delete_existing_stored:
raise OpStackIntegrity(
f"db op tries to add on top of existing key without deleting first: {op}"
)
elif op.is_delete and has_stored_val and stored_val != op.value and not will_delete_existing_stored:
# there is a value and we're not deleting it in this op
# check that a delete for the stored value is in the stack
raise OpStackIntegrity(f"db op tries to delete with incorrect existing value {op}")
elif op.is_delete and not has_stored_val:
raise OpStackIntegrity(f"db op tries to delete nonexistent key: {op}")
elif op.is_delete and stored_val != op.value:
raise OpStackIntegrity(f"db op tries to delete with incorrect value: {op}")
except OpStackIntegrity as err:
if op.key[:1] in self._unsafe_prefixes:
log.debug(f"skipping over integrity error: {err}")
else:
raise err
self._items[op.key].append(op)
def extend_ops(self, ops: Iterable[RevertableOp]):
"""
Apply a sequence of put or delete ops, checking that they introduce no integrity errors
"""
for op in ops:
self.append_op(op)
def clear(self):
self._items.clear()
def __len__(self):
return sum(map(len, self._items.values()))
def __iter__(self):
for key, ops in self._items.items():
for op in ops:
yield op
def __reversed__(self):
for key, ops in self._items.items():
for op in reversed(ops):
yield op
def get_undo_ops(self) -> bytes:
"""
Get the serialized bytes to undo all of the changes made by the pending ops
"""
return b''.join(op.invert().pack() for op in reversed(self))
def apply_packed_undo_ops(self, packed: bytes):
"""
Unpack and apply a sequence of undo ops from serialized undo bytes
"""
while packed:
op, packed = RevertableOp.unpack(packed)
self.append_op(op)
def get_last_op_for_key(self, key: bytes) -> Optional[RevertableOp]:
if key in self._items and self._items[key]:
return self._items[key][-1]
+384
View File
@@ -0,0 +1,384 @@
# Copyright (c) 2016, Neil Booth
#
# All rights reserved.
#
# See the file "LICENCE" for information about the copyright
# and warranty status of this software.
import math
import re
import resource
from os import environ
from collections import namedtuple
from ipaddress import ip_address
from lbry.wallet.server.util import class_logger
from lbry.wallet.server.coin import Coin, LBC, LBCTestNet, LBCRegTest
import lbry.wallet.server.util as lib_util
NetIdentity = namedtuple('NetIdentity', 'host tcp_port ssl_port nick_suffix')
class Env:
# Peer discovery
PD_OFF, PD_SELF, PD_ON = range(3)
class Error(Exception):
pass
def __init__(self, coin=None, db_dir=None, daemon_url=None, host=None, rpc_host=None, elastic_host=None,
elastic_port=None, loop_policy=None, max_query_workers=None, websocket_host=None, websocket_port=None,
chain=None, es_index_prefix=None, es_mode=None, cache_MB=None, reorg_limit=None, tcp_port=None,
udp_port=None, ssl_port=None, ssl_certfile=None, ssl_keyfile=None, rpc_port=None,
prometheus_port=None, max_subscriptions=None, banner_file=None, anon_logs=None, log_sessions=None,
allow_lan_udp=None, cache_all_tx_hashes=None, cache_all_claim_txos=None, country=None,
payment_address=None, donation_address=None, max_send=None, max_receive=None, max_sessions=None,
session_timeout=None, drop_client=None, description=None, daily_fee=None,
database_query_timeout=None, db_max_open_files=512):
self.logger = class_logger(__name__, self.__class__.__name__)
self.db_dir = db_dir if db_dir is not None else self.required('DB_DIRECTORY')
self.daemon_url = daemon_url if daemon_url is not None else self.required('DAEMON_URL')
self.db_max_open_files = db_max_open_files
self.host = host if host is not None else self.default('HOST', 'localhost')
self.rpc_host = rpc_host if rpc_host is not None else self.default('RPC_HOST', 'localhost')
self.elastic_host = elastic_host if elastic_host is not None else self.default('ELASTIC_HOST', 'localhost')
self.elastic_port = elastic_port if elastic_port is not None else self.integer('ELASTIC_PORT', 9200)
self.loop_policy = self.set_event_loop_policy(
loop_policy if loop_policy is not None else self.default('EVENT_LOOP_POLICY', None)
)
self.obsolete(['UTXO_MB', 'HIST_MB', 'NETWORK'])
self.max_query_workers = max_query_workers if max_query_workers is not None else self.integer('MAX_QUERY_WORKERS', 4)
self.websocket_host = websocket_host if websocket_host is not None else self.default('WEBSOCKET_HOST', self.host)
self.websocket_port = websocket_port if websocket_port is not None else self.integer('WEBSOCKET_PORT', None)
if coin is not None:
assert issubclass(coin, Coin)
self.coin = coin
else:
chain = chain if chain is not None else self.default('NET', 'mainnet').strip().lower()
if chain == 'mainnet':
self.coin = LBC
elif chain == 'testnet':
self.coin = LBCTestNet
else:
self.coin = LBCRegTest
self.es_index_prefix = es_index_prefix if es_index_prefix is not None else self.default('ES_INDEX_PREFIX', '')
self.es_mode = es_mode if es_mode is not None else self.default('ES_MODE', 'writer')
self.cache_MB = cache_MB if cache_MB is not None else self.integer('CACHE_MB', 1024)
self.reorg_limit = reorg_limit if reorg_limit is not None else self.integer('REORG_LIMIT', self.coin.REORG_LIMIT)
# Server stuff
self.tcp_port = tcp_port if tcp_port is not None else self.integer('TCP_PORT', None)
self.udp_port = udp_port if udp_port is not None else self.integer('UDP_PORT', self.tcp_port)
self.ssl_port = ssl_port if ssl_port is not None else self.integer('SSL_PORT', None)
if self.ssl_port:
self.ssl_certfile = ssl_certfile if ssl_certfile is not None else self.required('SSL_CERTFILE')
self.ssl_keyfile = ssl_keyfile if ssl_keyfile is not None else self.required('SSL_KEYFILE')
self.rpc_port = rpc_port if rpc_port is not None else self.integer('RPC_PORT', 8000)
self.prometheus_port = prometheus_port if prometheus_port is not None else self.integer('PROMETHEUS_PORT', 0)
self.max_subscriptions = max_subscriptions if max_subscriptions is not None else self.integer('MAX_SUBSCRIPTIONS', 10000)
self.banner_file = banner_file if banner_file is not None else self.default('BANNER_FILE', None)
# self.tor_banner_file = self.default('TOR_BANNER_FILE', self.banner_file)
self.anon_logs = anon_logs if anon_logs is not None else self.boolean('ANON_LOGS', False)
self.log_sessions = log_sessions if log_sessions is not None else self.integer('LOG_SESSIONS', 3600)
self.allow_lan_udp = allow_lan_udp if allow_lan_udp is not None else self.boolean('ALLOW_LAN_UDP', False)
self.cache_all_tx_hashes = cache_all_tx_hashes if cache_all_tx_hashes is not None else self.boolean('CACHE_ALL_TX_HASHES', False)
self.cache_all_claim_txos = cache_all_claim_txos if cache_all_claim_txos is not None else self.boolean('CACHE_ALL_CLAIM_TXOS', False)
self.country = country if country is not None else self.default('COUNTRY', 'US')
# Peer discovery
self.peer_discovery = self.peer_discovery_enum()
self.peer_announce = self.boolean('PEER_ANNOUNCE', True)
self.peer_hubs = self.extract_peer_hubs()
# self.tor_proxy_host = self.default('TOR_PROXY_HOST', 'localhost')
# self.tor_proxy_port = self.integer('TOR_PROXY_PORT', None)
# The electrum client takes the empty string as unspecified
self.payment_address = payment_address if payment_address is not None else self.default('PAYMENT_ADDRESS', '')
self.donation_address = donation_address if donation_address is not None else self.default('DONATION_ADDRESS', '')
# Server limits to help prevent DoS
self.max_send = max_send if max_send is not None else self.integer('MAX_SEND', 1000000)
self.max_receive = max_receive if max_receive is not None else self.integer('MAX_RECEIVE', 1000000)
# self.max_subs = self.integer('MAX_SUBS', 250000)
self.max_sessions = max_sessions if max_sessions is not None else self.sane_max_sessions()
# self.max_session_subs = self.integer('MAX_SESSION_SUBS', 50000)
self.session_timeout = session_timeout if session_timeout is not None else self.integer('SESSION_TIMEOUT', 600)
self.drop_client = drop_client if drop_client is not None else self.custom("DROP_CLIENT", None, re.compile)
self.description = description if description is not None else self.default('DESCRIPTION', '')
self.daily_fee = daily_fee if daily_fee is not None else self.string_amount('DAILY_FEE', '0')
# Identities
clearnet_identity = self.clearnet_identity()
tor_identity = self.tor_identity(clearnet_identity)
self.identities = [identity
for identity in (clearnet_identity, tor_identity)
if identity is not None]
self.database_query_timeout = database_query_timeout if database_query_timeout is not None else \
(float(self.integer('QUERY_TIMEOUT_MS', 10000)) / 1000.0)
@classmethod
def default(cls, envvar, default):
return environ.get(envvar, default)
@classmethod
def boolean(cls, envvar, default):
default = 'Yes' if default else ''
return bool(cls.default(envvar, default).strip())
@classmethod
def required(cls, envvar):
value = environ.get(envvar)
if value is None:
raise cls.Error(f'required envvar {envvar} not set')
return value
@classmethod
def string_amount(cls, envvar, default):
value = environ.get(envvar, default)
amount_pattern = re.compile("[0-9]{0,10}(\.[0-9]{1,8})?")
if len(value) > 0 and not amount_pattern.fullmatch(value):
raise cls.Error(f'{value} is not a valid amount for {envvar}')
return value
@classmethod
def integer(cls, envvar, default):
value = environ.get(envvar)
if value is None:
return default
try:
return int(value)
except Exception:
raise cls.Error(f'cannot convert envvar {envvar} value {value} to an integer')
@classmethod
def custom(cls, envvar, default, parse):
value = environ.get(envvar)
if value is None:
return default
try:
return parse(value)
except Exception as e:
raise cls.Error(f'cannot parse envvar {envvar} value {value}') from e
@classmethod
def obsolete(cls, envvars):
bad = [envvar for envvar in envvars if environ.get(envvar)]
if bad:
raise cls.Error(f'remove obsolete environment variables {bad}')
@classmethod
def set_event_loop_policy(cls, policy_name: str = None):
if not policy_name or policy_name == 'default':
import asyncio
return asyncio.get_event_loop_policy()
elif policy_name == 'uvloop':
import uvloop
import asyncio
loop_policy = uvloop.EventLoopPolicy()
asyncio.set_event_loop_policy(loop_policy)
return loop_policy
raise cls.Error(f'unknown event loop policy "{policy_name}"')
def cs_host(self, *, for_rpc):
"""Returns the 'host' argument to pass to asyncio's create_server
call. The result can be a single host name string, a list of
host name strings, or an empty string to bind to all interfaces.
If rpc is True the host to use for the RPC server is returned.
Otherwise the host to use for SSL/TCP servers is returned.
"""
host = self.rpc_host if for_rpc else self.host
result = [part.strip() for part in host.split(',')]
if len(result) == 1:
result = result[0]
# An empty result indicates all interfaces, which we do not
# permitted for an RPC server.
if for_rpc and not result:
result = 'localhost'
if result == 'localhost':
# 'localhost' resolves to ::1 (ipv6) on many systems, which fails on default setup of
# docker, using 127.0.0.1 instead forces ipv4
result = '127.0.0.1'
return result
def sane_max_sessions(self):
"""Return the maximum number of sessions to permit. Normally this
is MAX_SESSIONS. However, to prevent open file exhaustion, ajdust
downwards if running with a small open file rlimit."""
env_value = self.integer('MAX_SESSIONS', 1000)
nofile_limit = resource.getrlimit(resource.RLIMIT_NOFILE)[0]
# We give the DB 250 files; allow ElectrumX 100 for itself
value = max(0, min(env_value, nofile_limit - 350))
if value < env_value:
self.logger.warning(f'lowered maximum sessions from {env_value:,d} to {value:,d} '
f'because your open file limit is {nofile_limit:,d}')
return value
def clearnet_identity(self):
host = self.default('REPORT_HOST', None)
if host is None:
return None
try:
ip = ip_address(host)
except ValueError:
bad = (not lib_util.is_valid_hostname(host)
or host.lower() == 'localhost')
else:
bad = (ip.is_multicast or ip.is_unspecified
or (ip.is_private and self.peer_announce))
if bad:
raise self.Error(f'"{host}" is not a valid REPORT_HOST')
tcp_port = self.integer('REPORT_TCP_PORT', self.tcp_port) or None
ssl_port = self.integer('REPORT_SSL_PORT', self.ssl_port) or None
if tcp_port == ssl_port:
raise self.Error('REPORT_TCP_PORT and REPORT_SSL_PORT '
f'both resolve to {tcp_port}')
return NetIdentity(
host,
tcp_port,
ssl_port,
''
)
def tor_identity(self, clearnet):
host = self.default('REPORT_HOST_TOR', None)
if host is None:
return None
if not host.endswith('.onion'):
raise self.Error(f'tor host "{host}" must end with ".onion"')
def port(port_kind):
"""Returns the clearnet identity port, if any and not zero,
otherwise the listening port."""
result = 0
if clearnet:
result = getattr(clearnet, port_kind)
return result or getattr(self, port_kind)
tcp_port = self.integer('REPORT_TCP_PORT_TOR',
port('tcp_port')) or None
ssl_port = self.integer('REPORT_SSL_PORT_TOR',
port('ssl_port')) or None
if tcp_port == ssl_port:
raise self.Error('REPORT_TCP_PORT_TOR and REPORT_SSL_PORT_TOR '
f'both resolve to {tcp_port}')
return NetIdentity(
host,
tcp_port,
ssl_port,
'_tor',
)
def hosts_dict(self):
return {identity.host: {'tcp_port': identity.tcp_port,
'ssl_port': identity.ssl_port}
for identity in self.identities}
def peer_discovery_enum(self):
pd = self.default('PEER_DISCOVERY', 'on').strip().lower()
if pd in ('off', ''):
return self.PD_OFF
elif pd == 'self':
return self.PD_SELF
else:
return self.PD_ON
def extract_peer_hubs(self):
return [hub.strip() for hub in self.default('PEER_HUBS', '').split(',') if hub.strip()]
@classmethod
def contribute_to_arg_parser(cls, parser):
parser.add_argument('--db_dir', type=str, help='path of the directory containing lbry-leveldb',
default=cls.default('DB_DIRECTORY', None))
parser.add_argument('--daemon_url',
help='URL for rpc from lbrycrd, <rpcuser>:<rpcpassword>@<lbrycrd rpc ip><lbrycrd rpc port>',
default=cls.default('DAEMON_URL', None))
parser.add_argument('--db_max_open_files', type=int, default=512,
help='number of files leveldb can have open at a time')
parser.add_argument('--host', type=str, default=cls.default('HOST', 'localhost'),
help='Interface for hub server to listen on')
parser.add_argument('--tcp_port', type=int, default=cls.integer('TCP_PORT', 50001),
help='TCP port to listen on for hub server')
parser.add_argument('--udp_port', type=int, default=cls.integer('UDP_PORT', 50001),
help='UDP port to listen on for hub server')
parser.add_argument('--rpc_host', default=cls.default('RPC_HOST', 'localhost'), type=str,
help='Listening interface for admin rpc')
parser.add_argument('--rpc_port', default=cls.integer('RPC_PORT', 8000), type=int,
help='Listening port for admin rpc')
parser.add_argument('--websocket_host', default=cls.default('WEBSOCKET_HOST', 'localhost'), type=str,
help='Listening interface for websocket')
parser.add_argument('--websocket_port', default=cls.integer('WEBSOCKET_PORT', None), type=int,
help='Listening port for websocket')
parser.add_argument('--ssl_port', default=cls.integer('SSL_PORT', None), type=int,
help='SSL port to listen on for hub server')
parser.add_argument('--ssl_certfile', default=cls.default('SSL_CERTFILE', None), type=str,
help='Path to SSL cert file')
parser.add_argument('--ssl_keyfile', default=cls.default('SSL_KEYFILE', None), type=str,
help='Path to SSL key file')
parser.add_argument('--reorg_limit', default=cls.integer('REORG_LIMIT', 200), type=int, help='Max reorg depth')
parser.add_argument('--elastic_host', default=cls.default('ELASTIC_HOST', 'localhost'), type=str,
help='elasticsearch host')
parser.add_argument('--elastic_port', default=cls.integer('ELASTIC_PORT', 9200), type=int,
help='elasticsearch port')
parser.add_argument('--es_mode', default=cls.default('ES_MODE', 'writer'), type=str,
choices=['reader', 'writer'])
parser.add_argument('--es_index_prefix', default=cls.default('ES_INDEX_PREFIX', ''), type=str)
parser.add_argument('--loop_policy', default=cls.default('EVENT_LOOP_POLICY', 'default'), type=str,
choices=['default', 'uvloop'])
parser.add_argument('--max_query_workers', type=int, default=cls.integer('MAX_QUERY_WORKERS', 4),
help='number of threads used by the request handler to read the database')
parser.add_argument('--cache_MB', type=int, default=cls.integer('CACHE_MB', 1024),
help='size of the leveldb lru cache, in megabytes')
parser.add_argument('--cache_all_tx_hashes', type=bool,
help='Load all tx hashes into memory. This will make address subscriptions and sync, '
'resolve, transaction fetching, and block sync all faster at the expense of higher '
'memory usage')
parser.add_argument('--cache_all_claim_txos', type=bool,
help='Load all claim txos into memory. This will make address subscriptions and sync, '
'resolve, transaction fetching, and block sync all faster at the expense of higher '
'memory usage')
parser.add_argument('--prometheus_port', type=int, default=cls.integer('PROMETHEUS_PORT', 0),
help='port for hub prometheus metrics to listen on, disabled by default')
parser.add_argument('--max_subscriptions', type=int, default=cls.integer('MAX_SUBSCRIPTIONS', 10000),
help='max subscriptions per connection')
parser.add_argument('--banner_file', type=str, default=cls.default('BANNER_FILE', None),
help='path to file containing banner text')
parser.add_argument('--anon_logs', type=bool, default=cls.boolean('ANON_LOGS', False),
help="don't log ip addresses")
parser.add_argument('--allow_lan_udp', type=bool, default=cls.boolean('ALLOW_LAN_UDP', False),
help='reply to hub UDP ping messages from LAN ip addresses')
parser.add_argument('--country', type=str, default=cls.default('COUNTRY', 'US'), help='')
parser.add_argument('--max_send', type=int, default=cls.default('MAX_SEND', 1000000), help='')
parser.add_argument('--max_receive', type=int, default=cls.default('MAX_RECEIVE', 1000000), help='')
parser.add_argument('--max_sessions', type=int, default=cls.default('MAX_SESSIONS', 1000), help='')
parser.add_argument('--session_timeout', type=int, default=cls.default('SESSION_TIMEOUT', 600), help='')
parser.add_argument('--drop_client', type=str, default=cls.default('DROP_CLIENT', None), help='')
parser.add_argument('--description', type=str, default=cls.default('DESCRIPTION', ''), help='')
parser.add_argument('--daily_fee', type=float, default=cls.default('DAILY_FEE', 0.0), help='')
parser.add_argument('--payment_address', type=str, default=cls.default('PAYMENT_ADDRESS', ''), help='')
parser.add_argument('--donation_address', type=str, default=cls.default('DONATION_ADDRESS', ''), help='')
parser.add_argument('--chain', type=str, default=cls.default('NET', 'mainnet'),
help="Which chain to use, default is mainnet")
parser.add_argument('--query_timeout_ms', type=int, default=cls.integer('QUERY_TIMEOUT_MS', 10000),
help="elasticsearch query timeout")
@classmethod
def from_arg_parser(cls, args):
return cls(
db_dir=args.db_dir, daemon_url=args.daemon_url, db_max_open_files=args.db_max_open_files,
host=args.host, rpc_host=args.rpc_host, elastic_host=args.elastic_host, elastic_port=args.elastic_port,
loop_policy=args.loop_policy, max_query_workers=args.max_query_workers, websocket_host=args.websocket_host,
websocket_port=args.websocket_port, chain=args.chain, es_index_prefix=args.es_index_prefix,
es_mode=args.es_mode, cache_MB=args.cache_MB, reorg_limit=args.reorg_limit, tcp_port=args.tcp_port,
udp_port=args.udp_port, ssl_port=args.ssl_port, ssl_certfile=args.ssl_certfile,
ssl_keyfile=args.ssl_keyfile, rpc_port=args.rpc_port, prometheus_port=args.prometheus_port,
max_subscriptions=args.max_subscriptions, banner_file=args.banner_file, anon_logs=args.anon_logs,
log_sessions=None, allow_lan_udp=args.allow_lan_udp,
cache_all_tx_hashes=args.cache_all_tx_hashes, cache_all_claim_txos=args.cache_all_claim_txos,
country=args.country, payment_address=args.payment_address, donation_address=args.donation_address,
max_send=args.max_send, max_receive=args.max_receive, max_sessions=args.max_sessions,
session_timeout=args.session_timeout, drop_client=args.drop_client, description=args.description,
daily_fee=args.daily_fee, database_query_timeout=(args.query_timeout_ms / 1000)
)
+160
View File
@@ -0,0 +1,160 @@
# Copyright (c) 2016-2017, Neil Booth
#
# All rights reserved.
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""Cryptograph hash functions and related classes."""
import hashlib
import hmac
from lbry.wallet.server.util import bytes_to_int, int_to_bytes, hex_to_bytes
_sha256 = hashlib.sha256
_sha512 = hashlib.sha512
_new_hash = hashlib.new
_new_hmac = hmac.new
HASHX_LEN = 11
CLAIM_HASH_LEN = 20
def sha256(x):
"""Simple wrapper of hashlib sha256."""
return _sha256(x).digest()
def ripemd160(x):
"""Simple wrapper of hashlib ripemd160."""
h = _new_hash('ripemd160')
h.update(x)
return h.digest()
def double_sha256(x):
"""SHA-256 of SHA-256, as used extensively in bitcoin."""
return sha256(sha256(x))
def hmac_sha512(key, msg):
"""Use SHA-512 to provide an HMAC."""
return _new_hmac(key, msg, _sha512).digest()
def hash160(x):
"""RIPEMD-160 of SHA-256.
Used to make bitcoin addresses from pubkeys."""
return ripemd160(sha256(x))
def hash_to_hex_str(x: bytes) -> str:
"""Convert a big-endian binary hash to displayed hex string.
Display form of a binary hash is reversed and converted to hex.
"""
return x[::-1].hex()
def hex_str_to_hash(x: str) -> bytes:
"""Convert a displayed hex string to a binary hash."""
return hex_to_bytes(x)[::-1]
class Base58Error(Exception):
"""Exception used for Base58 errors."""
class Base58:
"""Class providing base 58 functionality."""
chars = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz'
assert len(chars) == 58
cmap = {c: n for n, c in enumerate(chars)}
@staticmethod
def char_value(c):
val = Base58.cmap.get(c)
if val is None:
raise Base58Error(f'invalid base 58 character "{c}"')
return val
@staticmethod
def decode(txt):
"""Decodes txt into a big-endian bytearray."""
if not isinstance(txt, str):
raise TypeError('a string is required')
if not txt:
raise Base58Error('string cannot be empty')
value = 0
for c in txt:
value = value * 58 + Base58.char_value(c)
result = int_to_bytes(value)
# Prepend leading zero bytes if necessary
count = 0
for c in txt:
if c != '1':
break
count += 1
if count:
result = bytes(count) + result
return result
@staticmethod
def encode(be_bytes):
"""Converts a big-endian bytearray into a base58 string."""
value = bytes_to_int(be_bytes)
txt = ''
while value:
value, mod = divmod(value, 58)
txt += Base58.chars[mod]
for byte in be_bytes:
if byte != 0:
break
txt += '1'
return txt[::-1]
@staticmethod
def decode_check(txt, *, hash_fn=double_sha256):
"""Decodes a Base58Check-encoded string to a payload. The version
prefixes it."""
be_bytes = Base58.decode(txt)
result, check = be_bytes[:-4], be_bytes[-4:]
if check != hash_fn(result)[:4]:
raise Base58Error(f'invalid base 58 checksum for {txt}')
return result
@staticmethod
def encode_check(payload, *, hash_fn=double_sha256):
"""Encodes a payload bytearray (which includes the version byte(s))
into a Base58Check string."""
be_bytes = payload + hash_fn(payload)[:4]
return Base58.encode(be_bytes)
File diff suppressed because it is too large. Load diff
+361
View File
@@ -0,0 +1,361 @@
# Copyright (c) 2016-2018, Neil Booth
#
# All rights reserved.
#
# See the file "LICENCE" for information about the copyright
# and warranty status of this software.
"""Mempool handling."""
import asyncio
import itertools
import time
import attr
import typing
from typing import Set, Optional, Callable, Awaitable
from collections import defaultdict
from prometheus_client import Histogram
from lbry.wallet.server.hash import hash_to_hex_str, hex_str_to_hash
from lbry.wallet.server.util import class_logger, chunks
from lbry.wallet.server.leveldb import UTXO
if typing.TYPE_CHECKING:
from lbry.wallet.server.session import LBRYSessionManager
@attr.s(slots=True)
class MemPoolTx:
prevouts = attr.ib()
# A pair is a (hashX, value) tuple
in_pairs = attr.ib()
out_pairs = attr.ib()
fee = attr.ib()
size = attr.ib()
raw_tx = attr.ib()
@attr.s(slots=True)
class MemPoolTxSummary:
hash = attr.ib()
fee = attr.ib()
has_unconfirmed_inputs = attr.ib()
NAMESPACE = "wallet_server"
HISTOGRAM_BUCKETS = (
.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0, 7.5, 10.0, 15.0, 20.0, 30.0, 60.0, float('inf')
)
mempool_process_time_metric = Histogram(
"processed_mempool", "Time to process mempool and notify touched addresses",
namespace=NAMESPACE, buckets=HISTOGRAM_BUCKETS
)
class MemPool:
def __init__(self, coin, daemon, db, state_lock: asyncio.Lock, refresh_secs=1.0, log_status_secs=120.0):
self.coin = coin
self._daemon = daemon
self._db = db
self._touched_mp = {}
self._touched_bp = {}
self._highest_block = -1
self.logger = class_logger(__name__, self.__class__.__name__)
self.txs = {}
self.hashXs = defaultdict(set) # None can be a key
self.cached_compact_histogram = []
self.refresh_secs = refresh_secs
self.log_status_secs = log_status_secs
# Prevents mempool refreshes during fee histogram calculation
self.lock = state_lock
self.wakeup = asyncio.Event()
self.mempool_process_time_metric = mempool_process_time_metric
self.notified_mempool_txs = set()
self.notify_sessions: Optional[Callable[[int, Set[bytes], Set[bytes]], Awaitable[None]]] = None
async def _logging(self, synchronized_event):
"""Print regular logs of mempool stats."""
self.logger.info('beginning processing of daemon mempool. '
'This can take some time...')
start = time.perf_counter()
await synchronized_event.wait()
elapsed = time.perf_counter() - start
self.logger.info(f'synced in {elapsed:.2f}s')
while True:
self.logger.info(f'{len(self.txs):,d} txs '
f'touching {len(self.hashXs):,d} addresses')
await asyncio.sleep(self.log_status_secs)
await synchronized_event.wait()
def _accept_transactions(self, tx_map, utxo_map, touched):
"""Accept transactions in tx_map to the mempool if all their inputs
can be found in the existing mempool or a utxo_map from the
DB.
Returns an (unprocessed tx_map, unspent utxo_map) pair.
"""
hashXs = self.hashXs
txs = self.txs
deferred = {}
unspent = set(utxo_map)
# Try to find all prevouts so we can accept the TX
for hash, tx in tx_map.items():
in_pairs = []
try:
for prevout in tx.prevouts:
utxo = utxo_map.get(prevout)
if not utxo:
prev_hash, prev_index = prevout
# Raises KeyError if prev_hash is not in txs
utxo = txs[prev_hash].out_pairs[prev_index]
in_pairs.append(utxo)
except KeyError:
deferred[hash] = tx
continue
# Spend the prevouts
unspent.difference_update(tx.prevouts)
# Save the in_pairs, compute the fee and accept the TX
tx.in_pairs = tuple(in_pairs)
# Avoid negative fees if dealing with generation-like transactions
# because some in_parts would be missing
tx.fee = max(0, (sum(v for _, v in tx.in_pairs) -
sum(v for _, v in tx.out_pairs)))
txs[hash] = tx
for hashX, value in itertools.chain(tx.in_pairs, tx.out_pairs):
touched.add(hashX)
hashXs[hashX].add(hash)
return deferred, {prevout: utxo_map[prevout] for prevout in unspent}
async def _mempool_loop(self, synchronized_event):
try:
return await self._refresh_hashes(synchronized_event)
except asyncio.CancelledError:
raise
except Exception as e:
self.logger.exception("MEMPOOL DIED")
raise e
async def _refresh_hashes(self, synchronized_event):
"""Refresh our view of the daemon's mempool."""
while True:
start = time.perf_counter()
height = self._daemon.cached_height()
hex_hashes = await self._daemon.mempool_hashes()
if height != await self._daemon.height():
continue
hashes = {hex_str_to_hash(hh) for hh in hex_hashes}
async with self.lock:
new_hashes = hashes.difference(self.notified_mempool_txs)
touched = await self._process_mempool(hashes)
self.notified_mempool_txs.update(new_hashes)
new_touched = {
touched_hashx for touched_hashx, txs in self.hashXs.items() if txs.intersection(new_hashes)
}
synchronized_event.set()
synchronized_event.clear()
await self.on_mempool(touched, new_touched, height)
duration = time.perf_counter() - start
self.mempool_process_time_metric.observe(duration)
try:
# we wait up to `refresh_secs` but go early if a broadcast happens (which triggers wakeup event)
await asyncio.wait_for(self.wakeup.wait(), timeout=self.refresh_secs)
except asyncio.TimeoutError:
pass
finally:
self.wakeup.clear()
async def _process_mempool(self, all_hashes):
# Re-sync with the new set of hashes
txs = self.txs
hashXs = self.hashXs # hashX: [tx_hash, ...]
touched = set()
# First handle txs that have disappeared
for tx_hash in set(txs).difference(all_hashes):
tx = txs.pop(tx_hash)
tx_hashXs = {hashX for hashX, value in tx.in_pairs}
tx_hashXs.update(hashX for hashX, value in tx.out_pairs)
for hashX in tx_hashXs:
hashXs[hashX].remove(tx_hash)
if not hashXs[hashX]:
del hashXs[hashX]
touched.update(tx_hashXs)
# Process new transactions
new_hashes = list(all_hashes.difference(txs))
if new_hashes:
fetches = []
for hashes in chunks(new_hashes, 200):
fetches.append(self._fetch_and_accept(hashes, all_hashes, touched))
tx_map = {}
utxo_map = {}
for fetch in asyncio.as_completed(fetches):
deferred, unspent = await fetch
tx_map.update(deferred)
utxo_map.update(unspent)
prior_count = 0
# FIXME: this is not particularly efficient
while tx_map and len(tx_map) != prior_count:
prior_count = len(tx_map)
tx_map, utxo_map = self._accept_transactions(tx_map, utxo_map, touched)
if tx_map:
self.logger.info(f'{len(tx_map)} txs dropped')
return touched
async def _fetch_and_accept(self, hashes, all_hashes, touched):
"""Fetch a list of mempool transactions."""
raw_txs = await self._daemon.getrawtransactions((hash_to_hex_str(hash) for hash in hashes))
to_hashX = self.coin.hashX_from_script
deserializer = self.coin.DESERIALIZER
tx_map = {}
for hash, raw_tx in zip(hashes, raw_txs):
# The daemon may have evicted the tx from its
# mempool or it may have gotten in a block
if not raw_tx:
continue
tx, tx_size = deserializer(raw_tx).read_tx_and_vsize()
# Convert the inputs and outputs into (hashX, value) pairs
# Drop generation-like inputs from MemPoolTx.prevouts
txin_pairs = tuple((txin.prev_hash, txin.prev_idx)
for txin in tx.inputs
if not txin.is_generation())
txout_pairs = tuple((to_hashX(txout.pk_script), txout.value)
for txout in tx.outputs)
tx_map[hash] = MemPoolTx(txin_pairs, None, txout_pairs,
0, tx_size, raw_tx)
# Determine all prevouts not in the mempool, and fetch the
# UTXO information from the database. Failed prevout lookups
# return None - concurrent database updates happen - which is
# relied upon by _accept_transactions. Ignore prevouts that are
# generation-like.
prevouts = tuple(prevout for tx in tx_map.values()
for prevout in tx.prevouts
if prevout[0] not in all_hashes)
utxos = await self._db.lookup_utxos(prevouts)
utxo_map = dict(zip(prevouts, utxos))
return self._accept_transactions(tx_map, utxo_map, touched)
#
# External interface
#
async def keep_synchronized(self, synchronized_event):
"""Keep the mempool synchronized with the daemon."""
await asyncio.wait([
self._mempool_loop(synchronized_event),
# self._refresh_histogram(synchronized_event),
self._logging(synchronized_event)
])
async def balance_delta(self, hashX):
"""Return the unconfirmed amount in the mempool for hashX.
Can be positive or negative.
"""
value = 0
if hashX in self.hashXs:
for hash in self.hashXs[hashX]:
tx = self.txs[hash]
value -= sum(v for h168, v in tx.in_pairs if h168 == hashX)
value += sum(v for h168, v in tx.out_pairs if h168 == hashX)
return value
def compact_fee_histogram(self):
"""Return a compact fee histogram of the current mempool."""
return self.cached_compact_histogram
async def potential_spends(self, hashX):
"""Return a set of (prev_hash, prev_idx) pairs from mempool
transactions that touch hashX.
None, some or all of these may be spends of the hashX, but all
actual spends of it (in the DB or mempool) will be included.
"""
result = set()
for tx_hash in self.hashXs.get(hashX, ()):
tx = self.txs[tx_hash]
result.update(tx.prevouts)
return result
def transaction_summaries(self, hashX):
"""Return a list of MemPoolTxSummary objects for the hashX."""
result = []
for tx_hash in self.hashXs.get(hashX, ()):
tx = self.txs[tx_hash]
has_ui = any(hash in self.txs for hash, idx in tx.prevouts)
result.append(MemPoolTxSummary(tx_hash, tx.fee, has_ui))
return result
async def unordered_UTXOs(self, hashX):
"""Return an unordered list of UTXO named tuples from mempool
transactions that pay to hashX.
This does not consider if any other mempool transactions spend
the outputs.
"""
utxos = []
for tx_hash in self.hashXs.get(hashX, ()):
tx = self.txs.get(tx_hash)
for pos, (hX, value) in enumerate(tx.out_pairs):
if hX == hashX:
utxos.append(UTXO(-1, pos, tx_hash, 0, value))
return utxos
def get_mempool_height(self, tx_hash):
# Height Progression
# -2: not broadcast
# -1: in mempool but has unconfirmed inputs
# 0: in mempool and all inputs confirmed
# +num: confirmed in a specific block (height)
if tx_hash not in self.txs:
return -2
tx = self.txs[tx_hash]
unspent_inputs = sum(1 if hash in self.txs else 0 for hash, idx in tx.prevouts)
if unspent_inputs:
return -1
return 0
async def _maybe_notify(self, new_touched):
tmp, tbp = self._touched_mp, self._touched_bp
common = set(tmp).intersection(tbp)
if common:
height = max(common)
elif tmp and max(tmp) == self._highest_block:
height = self._highest_block
else:
# Either we are processing a block and waiting for it to
# come in, or we have not yet had a mempool update for the
# new block height
return
touched = tmp.pop(height)
for old in [h for h in tmp if h <= height]:
del tmp[old]
for old in [h for h in tbp if h <= height]:
touched.update(tbp.pop(old))
# print("notify", height, len(touched), len(new_touched))
await self.notify_sessions(height, touched, new_touched)
async def start(self, height, session_manager: 'LBRYSessionManager'):
self._highest_block = height
self.notify_sessions = session_manager._notify_sessions
await self.notify_sessions(height, set(), set())
async def on_mempool(self, touched, new_touched, height):
self._touched_mp[height] = touched
await self._maybe_notify(new_touched)
async def on_block(self, touched, height):
self._touched_bp[height] = touched
self._highest_block = height
await self._maybe_notify(set())
+258
View File
@@ -0,0 +1,258 @@
# Copyright (c) 2018, Neil Booth
#
# All rights reserved.
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# and warranty status of this software.
"""Merkle trees, branches, proofs and roots."""
from asyncio import Event
from math import ceil, log
from lbry.wallet.server.hash import double_sha256
class Merkle:
"""Perform merkle tree calculations on binary hashes using a given hash
function.
If the hash count is not even, the final hash is repeated when
calculating the next merkle layer up the tree.
"""
def __init__(self, hash_func=double_sha256):
self.hash_func = hash_func
@staticmethod
def tree_depth(hash_count):
return Merkle.branch_length(hash_count) + 1
@staticmethod
def branch_length(hash_count):
"""Return the length of a merkle branch given the number of hashes."""
if not isinstance(hash_count, int):
raise TypeError('hash_count must be an integer')
if hash_count < 1:
raise ValueError('hash_count must be at least 1')
return ceil(log(hash_count, 2))
@staticmethod
def branch_and_root(hashes, index, length=None, hash_func=double_sha256):
"""Return a (merkle branch, merkle_root) pair given hashes, and the
index of one of those hashes.
"""
hashes = list(hashes)
if not isinstance(index, int):
raise TypeError('index must be an integer')
# This also asserts hashes is not empty
if not 0 <= index < len(hashes):
raise ValueError(f"index '{index}/{len(hashes)}' out of range")
natural_length = Merkle.branch_length(len(hashes))
if length is None:
length = natural_length
else:
if not isinstance(length, int):
raise TypeError('length must be an integer')
if length < natural_length:
raise ValueError('length out of range')
branch = []
for _ in range(length):
if len(hashes) & 1:
hashes.append(hashes[-1])
branch.append(hashes[index ^ 1])
index >>= 1
hashes = [hash_func(hashes[n] + hashes[n + 1])
for n in range(0, len(hashes), 2)]
return branch, hashes[0]
@staticmethod
def root(hashes, length=None):
"""Return the merkle root of a non-empty iterable of binary hashes."""
branch, root = Merkle.branch_and_root(hashes, 0, length)
return root
# @staticmethod
# def root_from_proof(hash, branch, index, hash_func=double_sha256):
# """Return the merkle root given a hash, a merkle branch to it, and
# its index in the hashes array.
#
# branch is an iterable sorted deepest to shallowest. If the
# returned root is the expected value then the merkle proof is
# verified.
#
# The caller should have confirmed the length of the branch with
# branch_length(). Unfortunately this is not easily done for
# bitcoin transactions as the number of transactions in a block
# is unknown to an SPV client.
# """
# for elt in branch:
# if index & 1:
# hash = hash_func(elt + hash)
# else:
# hash = hash_func(hash + elt)
# index >>= 1
# if index:
# raise ValueError('index out of range for branch')
# return hash
@staticmethod
def level(hashes, depth_higher):
"""Return a level of the merkle tree of hashes the given depth
higher than the bottom row of the original tree."""
size = 1 << depth_higher
root = Merkle.root
return [root(hashes[n: n + size], depth_higher)
for n in range(0, len(hashes), size)]
@staticmethod
def branch_and_root_from_level(level, leaf_hashes, index,
depth_higher):
"""Return a (merkle branch, merkle_root) pair when a merkle-tree has a
level cached.
To maximally reduce the amount of data hashed in computing a
markle branch, cache a tree of depth N at level N // 2.
level is a list of hashes in the middle of the tree (returned
by level())
leaf_hashes are the leaves needed to calculate a partial branch
up to level.
depth_higher is how much higher level is than the leaves of the tree
index is the index in the full list of hashes of the hash whose
merkle branch we want.
"""
if not isinstance(level, list):
raise TypeError("level must be a list")
if not isinstance(leaf_hashes, list):
raise TypeError("leaf_hashes must be a list")
leaf_index = (index >> depth_higher) << depth_higher
leaf_branch, leaf_root = Merkle.branch_and_root(
leaf_hashes, index - leaf_index, depth_higher)
index >>= depth_higher
level_branch, root = Merkle.branch_and_root(level, index)
# Check last so that we know index is in-range
if leaf_root != level[index]:
raise ValueError('leaf hashes inconsistent with level')
return leaf_branch + level_branch, root
class MerkleCache:
"""A cache to calculate merkle branches efficiently."""
def __init__(self, merkle, source_func):
"""Initialise a cache hashes taken from source_func:
async def source_func(index, count):
...
"""
self.merkle = merkle
self.source_func = source_func
self.length = 0
self.depth_higher = 0
self.initialized = Event()
def _segment_length(self):
return 1 << self.depth_higher
def _leaf_start(self, index):
"""Given a level's depth higher and a hash index, return the leaf
index and leaf hash count needed to calculate a merkle branch.
"""
depth_higher = self.depth_higher
return (index >> depth_higher) << depth_higher
def _level(self, hashes):
return self.merkle.level(hashes, self.depth_higher)
async def _extend_to(self, length):
"""Extend the length of the cache if necessary."""
if length <= self.length:
return
# Start from the beginning of any final partial segment.
# Retain the value of depth_higher; in practice this is fine
start = self._leaf_start(self.length)
hashes = await self.source_func(start, length - start)
self.level[start >> self.depth_higher:] = self._level(hashes)
self.length = length
async def _level_for(self, length):
"""Return a (level_length, final_hash) pair for a truncation
of the hashes to the given length."""
if length == self.length:
return self.level
level = self.level[:length >> self.depth_higher]
leaf_start = self._leaf_start(length)
count = min(self._segment_length(), length - leaf_start)
hashes = await self.source_func(leaf_start, count)
level += self._level(hashes)
return level
async def initialize(self, length):
"""Call to initialize the cache to a source of given length."""
self.length = length
self.depth_higher = self.merkle.tree_depth(length) // 2
self.level = self._level(await self.source_func(0, length))
self.initialized.set()
def truncate(self, length):
"""Truncate the cache so it covers no more than length underlying
hashes."""
if not isinstance(length, int):
raise TypeError('length must be an integer')
if length <= 0:
raise ValueError('length must be positive')
if length >= self.length:
return
length = self._leaf_start(length)
self.length = length
self.level[length >> self.depth_higher:] = []
async def branch_and_root(self, length, index):
"""Return a merkle branch and root. Length is the number of
hashes used to calculate the merkle root, index is the position
of the hash to calculate the branch of.
index must be less than length, which must be at least 1."""
if not isinstance(length, int):
raise TypeError('length must be an integer')
if not isinstance(index, int):
raise TypeError('index must be an integer')
if length <= 0:
raise ValueError('length must be positive')
if index >= length:
raise ValueError('index must be less than length')
await self.initialized.wait()
await self._extend_to(length)
leaf_start = self._leaf_start(index)
count = min(self._segment_length(), length - leaf_start)
leaf_hashes = await self.source_func(leaf_start, count)
if length < self._segment_length():
return self.merkle.branch_and_root(leaf_hashes, index)
level = await self._level_for(length)
return self.merkle.branch_and_root_from_level(
level, leaf_hashes, index, self.depth_higher)
+135
View File
@@ -0,0 +1,135 @@
import time
import math
from typing import Tuple
def calculate_elapsed(start) -> int:
return int((time.perf_counter() - start) * 1000)
def calculate_avg_percentiles(data) -> Tuple[int, int, int, int, int, int, int, int]:
if not data:
return 0, 0, 0, 0, 0, 0, 0, 0
data.sort()
size = len(data)
return (
int(sum(data) / size),
data[0],
data[math.ceil(size * .05) - 1],
data[math.ceil(size * .25) - 1],
data[math.ceil(size * .50) - 1],
data[math.ceil(size * .75) - 1],
data[math.ceil(size * .95) - 1],
data[-1]
)
def remove_select_list(sql) -> str:
return sql[sql.index('FROM'):]
class APICallMetrics:
def __init__(self, name):
self.name = name
# total requests received
self.receive_count = 0
self.cache_response_count = 0
# millisecond timings for query based responses
self.query_response_times = []
self.query_intrp_times = []
self.query_error_times = []
self.query_python_times = []
self.query_wait_times = []
self.query_sql_times = [] # aggregate total of multiple SQL calls made per request
self.individual_sql_times = [] # every SQL query run on server
# actual queries
self.errored_queries = set()
self.interrupted_queries = set()
def to_json(self):
return {
# total requests received
"receive_count": self.receive_count,
# sum of these is total responses made
"cache_response_count": self.cache_response_count,
"query_response_count": len(self.query_response_times),
"intrp_response_count": len(self.query_intrp_times),
"error_response_count": len(self.query_error_times),
# millisecond timings for non-cache responses
"response": calculate_avg_percentiles(self.query_response_times),
"interrupt": calculate_avg_percentiles(self.query_intrp_times),
"error": calculate_avg_percentiles(self.query_error_times),
# response, interrupt and error each also report the python, wait and sql stats:
"python": calculate_avg_percentiles(self.query_python_times),
"wait": calculate_avg_percentiles(self.query_wait_times),
"sql": calculate_avg_percentiles(self.query_sql_times),
# extended timings for individual sql executions
"individual_sql": calculate_avg_percentiles(self.individual_sql_times),
"individual_sql_count": len(self.individual_sql_times),
# actual queries
"errored_queries": list(self.errored_queries),
"interrupted_queries": list(self.interrupted_queries),
}
def start(self):
self.receive_count += 1
def cache_response(self):
self.cache_response_count += 1
def _add_query_timings(self, request_total_time, metrics):
if metrics and 'execute_query' in metrics:
sub_process_total = metrics[self.name][0]['total']
individual_query_times = [f['total'] for f in metrics['execute_query']]
aggregated_query_time = sum(individual_query_times)
self.individual_sql_times.extend(individual_query_times)
self.query_sql_times.append(aggregated_query_time)
self.query_python_times.append(sub_process_total - aggregated_query_time)
self.query_wait_times.append(request_total_time - sub_process_total)
@staticmethod
def _add_queries(query_set, metrics):
if metrics and 'execute_query' in metrics:
for execute_query in metrics['execute_query']:
if 'sql' in execute_query:
query_set.add(remove_select_list(execute_query['sql']))
def query_response(self, start, metrics):
self.query_response_times.append(calculate_elapsed(start))
self._add_query_timings(self.query_response_times[-1], metrics)
def query_interrupt(self, start, metrics):
self.query_intrp_times.append(calculate_elapsed(start))
self._add_queries(self.interrupted_queries, metrics)
self._add_query_timings(self.query_intrp_times[-1], metrics)
def query_error(self, start, metrics):
self.query_error_times.append(calculate_elapsed(start))
self._add_queries(self.errored_queries, metrics)
self._add_query_timings(self.query_error_times[-1], metrics)
class ServerLoadData:
def __init__(self):
self._apis = {}
def for_api(self, name) -> APICallMetrics:
if name not in self._apis:
self._apis[name] = APICallMetrics(name)
return self._apis[name]
def to_json_and_reset(self, status):
try:
return {
'api': {name: api.to_json() for name, api in self._apis.items()},
'status': status
}
finally:
self._apis = {}
+289
View File
@@ -0,0 +1,289 @@
# Copyright (c) 2016-2017, Neil Booth
#
# All rights reserved.
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# and warranty status of this software.
"""Script-related classes and functions."""
from collections import namedtuple
from lbry.wallet.server.util import unpack_le_uint16_from, unpack_le_uint32_from, \
pack_le_uint16, pack_le_uint32
class EnumError(Exception):
pass
class Enumeration:
def __init__(self, name, enumList):
self.__doc__ = name
lookup = {}
reverseLookup = {}
i = 0
uniqueNames = set()
uniqueValues = set()
for x in enumList:
if isinstance(x, tuple):
x, i = x
if not isinstance(x, str):
raise EnumError(f"enum name {x} not a string")
if not isinstance(i, int):
raise EnumError(f"enum value {i} not an integer")
if x in uniqueNames:
raise EnumError(f"enum name {x} not unique")
if i in uniqueValues:
raise EnumError(f"enum value {i} not unique")
uniqueNames.add(x)
uniqueValues.add(i)
lookup[x] = i
reverseLookup[i] = x
i = i + 1
self.lookup = lookup
self.reverseLookup = reverseLookup
def __getattr__(self, attr):
result = self.lookup.get(attr)
if result is None:
raise AttributeError(f'enumeration has no member {attr}')
return result
def whatis(self, value):
return self.reverseLookup[value]
class ScriptError(Exception):
"""Exception used for script errors."""
OpCodes = Enumeration("Opcodes", [
("OP_0", 0), ("OP_PUSHDATA1", 76),
"OP_PUSHDATA2", "OP_PUSHDATA4", "OP_1NEGATE",
"OP_RESERVED",
"OP_1", "OP_2", "OP_3", "OP_4", "OP_5", "OP_6", "OP_7", "OP_8",
"OP_9", "OP_10", "OP_11", "OP_12", "OP_13", "OP_14", "OP_15", "OP_16",
"OP_NOP", "OP_VER", "OP_IF", "OP_NOTIF", "OP_VERIF", "OP_VERNOTIF",
"OP_ELSE", "OP_ENDIF", "OP_VERIFY", "OP_RETURN",
"OP_TOALTSTACK", "OP_FROMALTSTACK", "OP_2DROP", "OP_2DUP", "OP_3DUP",
"OP_2OVER", "OP_2ROT", "OP_2SWAP", "OP_IFDUP", "OP_DEPTH", "OP_DROP",
"OP_DUP", "OP_NIP", "OP_OVER", "OP_PICK", "OP_ROLL", "OP_ROT",
"OP_SWAP", "OP_TUCK",
"OP_CAT", "OP_SUBSTR", "OP_LEFT", "OP_RIGHT", "OP_SIZE",
"OP_INVERT", "OP_AND", "OP_OR", "OP_XOR", "OP_EQUAL", "OP_EQUALVERIFY",
"OP_RESERVED1", "OP_RESERVED2",
"OP_1ADD", "OP_1SUB", "OP_2MUL", "OP_2DIV", "OP_NEGATE", "OP_ABS",
"OP_NOT", "OP_0NOTEQUAL", "OP_ADD", "OP_SUB", "OP_MUL", "OP_DIV", "OP_MOD",
"OP_LSHIFT", "OP_RSHIFT", "OP_BOOLAND", "OP_BOOLOR", "OP_NUMEQUAL",
"OP_NUMEQUALVERIFY", "OP_NUMNOTEQUAL", "OP_LESSTHAN", "OP_GREATERTHAN",
"OP_LESSTHANOREQUAL", "OP_GREATERTHANOREQUAL", "OP_MIN", "OP_MAX",
"OP_WITHIN",
"OP_RIPEMD160", "OP_SHA1", "OP_SHA256", "OP_HASH160", "OP_HASH256",
"OP_CODESEPARATOR", "OP_CHECKSIG", "OP_CHECKSIGVERIFY", "OP_CHECKMULTISIG",
"OP_CHECKMULTISIGVERIFY",
"OP_NOP1",
"OP_CHECKLOCKTIMEVERIFY", "OP_CHECKSEQUENCEVERIFY"
])
# Paranoia to make it hard to create bad scripts
assert OpCodes.OP_DUP == 0x76
assert OpCodes.OP_HASH160 == 0xa9
assert OpCodes.OP_EQUAL == 0x87
assert OpCodes.OP_EQUALVERIFY == 0x88
assert OpCodes.OP_CHECKSIG == 0xac
assert OpCodes.OP_CHECKMULTISIG == 0xae
def _match_ops(ops, pattern):
if len(ops) != len(pattern):
return False
for op, pop in zip(ops, pattern):
if pop != op:
# -1 means 'data push', whose op is an (op, data) tuple
if pop == -1 and isinstance(op, tuple):
continue
return False
return True
class ScriptPubKey:
"""A class for handling a tx output script that gives conditions
necessary for spending.
"""
TO_ADDRESS_OPS = [OpCodes.OP_DUP, OpCodes.OP_HASH160, -1,
OpCodes.OP_EQUALVERIFY, OpCodes.OP_CHECKSIG]
TO_P2SH_OPS = [OpCodes.OP_HASH160, -1, OpCodes.OP_EQUAL]
TO_PUBKEY_OPS = [-1, OpCodes.OP_CHECKSIG]
PayToHandlers = namedtuple('PayToHandlers', 'address script_hash pubkey '
'unspendable strange')
@classmethod
def pay_to(cls, handlers, script):
"""Parse a script, invoke the appropriate handler and
return the result.
One of the following handlers is invoked:
handlers.address(hash160)
handlers.script_hash(hash160)
handlers.pubkey(pubkey)
handlers.unspendable()
handlers.strange(script)
"""
try:
ops = Script.get_ops(script)
except ScriptError:
return handlers.unspendable()
match = _match_ops
if match(ops, cls.TO_ADDRESS_OPS):
return handlers.address(ops[2][-1])
if match(ops, cls.TO_P2SH_OPS):
return handlers.script_hash(ops[1][-1])
if match(ops, cls.TO_PUBKEY_OPS):
return handlers.pubkey(ops[0][-1])
if ops and ops[0] == OpCodes.OP_RETURN:
return handlers.unspendable()
return handlers.strange(script)
@classmethod
def P2SH_script(cls, hash160):
return (bytes([OpCodes.OP_HASH160])
+ Script.push_data(hash160)
+ bytes([OpCodes.OP_EQUAL]))
@classmethod
def P2PKH_script(cls, hash160):
return (bytes([OpCodes.OP_DUP, OpCodes.OP_HASH160])
+ Script.push_data(hash160)
+ bytes([OpCodes.OP_EQUALVERIFY, OpCodes.OP_CHECKSIG]))
@classmethod
def validate_pubkey(cls, pubkey, req_compressed=False):
if isinstance(pubkey, (bytes, bytearray)):
if len(pubkey) == 33 and pubkey[0] in (2, 3):
return # Compressed
if len(pubkey) == 65 and pubkey[0] == 4:
if not req_compressed:
return
raise PubKeyError('uncompressed pubkeys are invalid')
raise PubKeyError(f'invalid pubkey {pubkey}')
@classmethod
def pubkey_script(cls, pubkey):
cls.validate_pubkey(pubkey)
return Script.push_data(pubkey) + bytes([OpCodes.OP_CHECKSIG])
@classmethod
def multisig_script(cls, m, pubkeys):
"""Returns the script for a pay-to-multisig transaction."""
n = len(pubkeys)
if not 1 <= m <= n <= 15:
raise ScriptError(f'{m:d} of {n:d} multisig script not possible')
for pubkey in pubkeys:
cls.validate_pubkey(pubkey, req_compressed=True)
# See https://bitcoin.org/en/developer-guide
# 2 of 3 is: OP_2 pubkey1 pubkey2 pubkey3 OP_3 OP_CHECKMULTISIG
return (bytes([OP_1 + m - 1])
+ b''.join(cls.push_data(pubkey) for pubkey in pubkeys)
+ bytes([OP_1 + n - 1, OP_CHECK_MULTISIG]))
class Script:
@classmethod
def get_ops(cls, script):
ops = []
# The unpacks or script[n] below throw on truncated scripts
try:
n = 0
while n < len(script):
op = script[n]
n += 1
if op <= OpCodes.OP_PUSHDATA4:
# Raw bytes follow
if op < OpCodes.OP_PUSHDATA1:
dlen = op
elif op == OpCodes.OP_PUSHDATA1:
dlen = script[n]
n += 1
elif op == OpCodes.OP_PUSHDATA2:
dlen, = unpack_le_uint16_from(script[n: n + 2])
n += 2
else:
dlen, = unpack_le_uint32_from(script[n: n + 4])
n += 4
if n + dlen > len(script):
raise IndexError
op = (op, script[n:n + dlen])
n += dlen
ops.append(op)
except Exception:
# Truncated script; e.g. tx_hash
# ebc9fa1196a59e192352d76c0f6e73167046b9d37b8302b6bb6968dfd279b767
raise ScriptError('truncated script')
return ops
@classmethod
def push_data(cls, data):
"""Returns the opcodes to push the data on the stack."""
assert isinstance(data, (bytes, bytearray))
n = len(data)
if n < OpCodes.OP_PUSHDATA1:
return bytes([n]) + data
if n < 256:
return bytes([OpCodes.OP_PUSHDATA1, n]) + data
if n < 65536:
return bytes([OpCodes.OP_PUSHDATA2]) + pack_le_uint16(n) + data
return bytes([OpCodes.OP_PUSHDATA4]) + pack_le_uint32(n) + data
@classmethod
def opcode_name(cls, opcode):
if OpCodes.OP_0 < opcode < OpCodes.OP_PUSHDATA1:
return f'OP_{opcode:d}'
try:
return OpCodes.whatis(opcode)
except KeyError:
return f'OP_UNKNOWN:{opcode:d}'
@classmethod
def dump(cls, script):
opcodes, datas = cls.get_ops(script)
for opcode, data in zip(opcodes, datas):
name = cls.opcode_name(opcode)
if data is None:
print(name)
else:
print(f'{name} {data.hex()} ({len(data):d} bytes)')
+91
View File
@@ -0,0 +1,91 @@
import signal
import logging
import asyncio
from concurrent.futures.thread import ThreadPoolExecutor
import typing
import lbry
from lbry.wallet.server.mempool import MemPool
from lbry.wallet.server.block_processor import BlockProcessor
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.session import LBRYSessionManager
from lbry.prometheus import PrometheusServer
class Server:
def __init__(self, env):
self.env = env
self.log = logging.getLogger(__name__).getChild(self.__class__.__name__)
self.shutdown_event = asyncio.Event()
self.cancellable_tasks = []
self.daemon = daemon = env.coin.DAEMON(env.coin, env.daemon_url)
self.db = db = LevelDB(env)
self.bp = bp = BlockProcessor(env, db, daemon, self.shutdown_event)
self.prometheus_server: typing.Optional[PrometheusServer] = None
self.session_mgr = LBRYSessionManager(
env, db, bp, daemon, self.shutdown_event
)
self._indexer_task = None
async def start(self):
env = self.env
min_str, max_str = env.coin.SESSIONCLS.protocol_min_max_strings()
self.log.info(f'software version: {lbry.__version__}')
self.log.info(f'supported protocol versions: {min_str}-{max_str}')
self.log.info(f'event loop policy: {env.loop_policy}')
self.log.info(f'reorg limit is {env.reorg_limit:,d} blocks')
await self.daemon.height()
def _start_cancellable(run, *args):
_flag = asyncio.Event()
self.cancellable_tasks.append(asyncio.ensure_future(run(*args, _flag)))
return _flag.wait()
await self.start_prometheus()
if self.env.udp_port:
await self.bp.status_server.start(
0, bytes.fromhex(self.bp.coin.GENESIS_HASH)[::-1], self.env.country,
self.env.host, self.env.udp_port, self.env.allow_lan_udp
)
await _start_cancellable(self.bp.fetch_and_process_blocks)
await self.db.populate_header_merkle_cache()
await _start_cancellable(self.bp.mempool.keep_synchronized)
await _start_cancellable(self.session_mgr.serve, self.bp.mempool)
async def stop(self):
for task in reversed(self.cancellable_tasks):
task.cancel()
await asyncio.wait(self.cancellable_tasks)
if self.prometheus_server:
await self.prometheus_server.stop()
self.prometheus_server = None
self.shutdown_event.set()
await self.daemon.close()
def run(self):
loop = asyncio.get_event_loop()
executor = ThreadPoolExecutor(self.env.max_query_workers, thread_name_prefix='hub-worker')
loop.set_default_executor(executor)
def __exit():
raise SystemExit()
try:
loop.add_signal_handler(signal.SIGINT, __exit)
loop.add_signal_handler(signal.SIGTERM, __exit)
loop.run_until_complete(self.start())
loop.run_until_complete(self.shutdown_event.wait())
except (SystemExit, KeyboardInterrupt):
pass
finally:
loop.run_until_complete(self.stop())
executor.shutdown(True)
async def start_prometheus(self):
if not self.prometheus_server and self.env.prometheus_port:
self.prometheus_server = PrometheusServer()
await self.prometheus_server.start("0.0.0.0", self.env.prometheus_port)
File diff suppressed because it is too large. Load diff
+626
View File
@@ -0,0 +1,626 @@
# Copyright (c) 2016-2017, Neil Booth
# Copyright (c) 2017, the ElectrumX authors
#
# All rights reserved.
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# and warranty status of this software.
"""Transaction-related classes and functions."""
import typing
from collections import namedtuple
from lbry.wallet.server.hash import sha256, double_sha256, hash_to_hex_str
from lbry.wallet.server.script import OpCodes
from lbry.wallet.server.util import (
unpack_le_int32_from, unpack_le_int64_from, unpack_le_uint16_from,
unpack_le_uint32_from, unpack_le_uint64_from, pack_le_int32, pack_varint,
pack_le_uint32, pack_le_int64, pack_varbytes,
)
ZERO = bytes(32)
MINUS_1 = 4294967295
class Tx(typing.NamedTuple):
version: int
inputs: typing.List['TxInput']
outputs: typing.List['TxOutput']
locktime: int
raw: bytes
class TxInput(typing.NamedTuple):
prev_hash: bytes
prev_idx: int
script: bytes
sequence: int
"""Class representing a transaction input."""
def __str__(self):
script = self.script.hex()
prev_hash = hash_to_hex_str(self.prev_hash)
return (f"Input({prev_hash}, {self.prev_idx:d}, script={script}, sequence={self.sequence:d})")
def is_generation(self):
"""Test if an input is generation/coinbase like"""
return self.prev_idx == MINUS_1 and self.prev_hash == ZERO
def serialize(self):
return b''.join((
self.prev_hash,
pack_le_uint32(self.prev_idx),
pack_varbytes(self.script),
pack_le_uint32(self.sequence),
))
class TxOutput(typing.NamedTuple):
value: int
pk_script: bytes
def serialize(self):
return b''.join((
pack_le_int64(self.value),
pack_varbytes(self.pk_script),
))
class Deserializer:
"""Deserializes blocks into transactions.
External entry points are read_tx(), read_tx_and_hash(),
read_tx_and_vsize() and read_block().
This code is performance sensitive as it is executed 100s of
millions of times during sync.
"""
TX_HASH_FN = staticmethod(double_sha256)
def __init__(self, binary, start=0):
assert isinstance(binary, bytes)
self.binary = binary
self.binary_length = len(binary)
self.cursor = start
self.flags = 0
def read_tx(self):
"""Return a deserialized transaction."""
start = self.cursor
return Tx(
self._read_le_int32(), # version
self._read_inputs(), # inputs
self._read_outputs(), # outputs
self._read_le_uint32(), # locktime
self.binary[start:self.cursor],
)
def read_tx_and_hash(self):
"""Return a (deserialized TX, tx_hash) pair.
The hash needs to be reversed for human display; for efficiency
we process it in the natural serialized order.
"""
start = self.cursor
return self.read_tx(), self.TX_HASH_FN(self.binary[start:self.cursor])
def read_tx_and_vsize(self):
"""Return a (deserialized TX, vsize) pair."""
return self.read_tx(), self.binary_length
def read_tx_block(self):
"""Returns a list of (deserialized_tx, tx_hash) pairs."""
read = self.read_tx_and_hash
# Some coins have excess data beyond the end of the transactions
return [read() for _ in range(self._read_varint())]
def _read_inputs(self):
read_input = self._read_input
return [read_input() for i in range(self._read_varint())]
def _read_input(self):
return TxInput(
self._read_nbytes(32), # prev_hash
self._read_le_uint32(), # prev_idx
self._read_varbytes(), # script
self._read_le_uint32() # sequence
)
def _read_outputs(self):
read_output = self._read_output
return [read_output() for i in range(self._read_varint())]
def _read_output(self):
return TxOutput(
self._read_le_int64(), # value
self._read_varbytes(), # pk_script
)
def _read_byte(self):
cursor = self.cursor
self.cursor += 1
return self.binary[cursor]
def _read_nbytes(self, n):
cursor = self.cursor
self.cursor = end = cursor + n
assert self.binary_length >= end
return self.binary[cursor:end]
def _read_varbytes(self):
return self._read_nbytes(self._read_varint())
def _read_varint(self):
n = self.binary[self.cursor]
self.cursor += 1
if n < 253:
return n
if n == 253:
return self._read_le_uint16()
if n == 254:
return self._read_le_uint32()
return self._read_le_uint64()
def _read_le_int32(self):
result, = unpack_le_int32_from(self.binary, self.cursor)
self.cursor += 4
return result
def _read_le_int64(self):
result, = unpack_le_int64_from(self.binary, self.cursor)
self.cursor += 8
return result
def _read_le_uint16(self):
result, = unpack_le_uint16_from(self.binary, self.cursor)
self.cursor += 2
return result
def _read_le_uint32(self):
result, = unpack_le_uint32_from(self.binary, self.cursor)
self.cursor += 4
return result
def _read_le_uint64(self):
result, = unpack_le_uint64_from(self.binary, self.cursor)
self.cursor += 8
return result
class TxSegWit(namedtuple("Tx", "version marker flag inputs outputs "
"witness locktime raw")):
"""Class representing a SegWit transaction."""
class DeserializerSegWit(Deserializer):
# https://bitcoincore.org/en/segwit_wallet_dev/#transaction-serialization
def _read_witness(self, fields):
read_witness_field = self._read_witness_field
return [read_witness_field() for i in range(fields)]
def _read_witness_field(self):
read_varbytes = self._read_varbytes
return [read_varbytes() for i in range(self._read_varint())]
def _read_tx_parts(self):
"""Return a (deserialized TX, tx_hash, vsize) tuple."""
start = self.cursor
marker = self.binary[self.cursor + 4]
if marker:
tx = super().read_tx()
tx_hash = self.TX_HASH_FN(self.binary[start:self.cursor])
return tx, tx_hash, self.binary_length
# Ugh, this is nasty.
version = self._read_le_int32()
orig_ser = self.binary[start:self.cursor]
marker = self._read_byte()
flag = self._read_byte()
start = self.cursor
inputs = self._read_inputs()
outputs = self._read_outputs()
orig_ser += self.binary[start:self.cursor]
base_size = self.cursor - start
witness = self._read_witness(len(inputs))
start = self.cursor
locktime = self._read_le_uint32()
orig_ser += self.binary[start:self.cursor]
vsize = (3 * base_size + self.binary_length) // 4
return TxSegWit(version, marker, flag, inputs, outputs, witness,
locktime, orig_ser), self.TX_HASH_FN(orig_ser), vsize
def read_tx(self):
return self._read_tx_parts()[0]
def read_tx_and_hash(self):
tx, tx_hash, vsize = self._read_tx_parts()
return tx, tx_hash
def read_tx_and_vsize(self):
tx, tx_hash, vsize = self._read_tx_parts()
return tx, vsize
class DeserializerAuxPow(Deserializer):
VERSION_AUXPOW = (1 << 8)
def read_header(self, height, static_header_size):
"""Return the AuxPow block header bytes"""
start = self.cursor
version = self._read_le_uint32()
if version & self.VERSION_AUXPOW:
# We are going to calculate the block size then read it as bytes
self.cursor = start
self.cursor += static_header_size # Block normal header
self.read_tx() # AuxPow transaction
self.cursor += 32 # Parent block hash
merkle_size = self._read_varint()
self.cursor += 32 * merkle_size # Merkle branch
self.cursor += 4 # Index
merkle_size = self._read_varint()
self.cursor += 32 * merkle_size # Chain merkle branch
self.cursor += 4 # Chain index
self.cursor += 80 # Parent block header
header_end = self.cursor
else:
header_end = static_header_size
self.cursor = start
return self._read_nbytes(header_end)
class DeserializerAuxPowSegWit(DeserializerSegWit, DeserializerAuxPow):
pass
class DeserializerEquihash(Deserializer):
def read_header(self, height, static_header_size):
"""Return the block header bytes"""
start = self.cursor
# We are going to calculate the block size then read it as bytes
self.cursor += static_header_size
solution_size = self._read_varint()
self.cursor += solution_size
header_end = self.cursor
self.cursor = start
return self._read_nbytes(header_end)
class DeserializerEquihashSegWit(DeserializerSegWit, DeserializerEquihash):
pass
class TxJoinSplit(namedtuple("Tx", "version inputs outputs locktime")):
"""Class representing a JoinSplit transaction."""
class DeserializerZcash(DeserializerEquihash):
def read_tx(self):
header = self._read_le_uint32()
overwintered = ((header >> 31) == 1)
if overwintered:
version = header & 0x7fffffff
self.cursor += 4 # versionGroupId
else:
version = header
is_overwinter_v3 = version == 3
is_sapling_v4 = version == 4
base_tx = TxJoinSplit(
version,
self._read_inputs(), # inputs
self._read_outputs(), # outputs
self._read_le_uint32() # locktime
)
if is_overwinter_v3 or is_sapling_v4:
self.cursor += 4 # expiryHeight
has_shielded = False
if is_sapling_v4:
self.cursor += 8 # valueBalance
shielded_spend_size = self._read_varint()
self.cursor += shielded_spend_size * 384 # vShieldedSpend
shielded_output_size = self._read_varint()
self.cursor += shielded_output_size * 948 # vShieldedOutput
has_shielded = shielded_spend_size > 0 or shielded_output_size > 0
if base_tx.version >= 2:
joinsplit_size = self._read_varint()
if joinsplit_size > 0:
joinsplit_desc_len = 1506 + (192 if is_sapling_v4 else 296)
# JSDescription
self.cursor += joinsplit_size * joinsplit_desc_len
self.cursor += 32 # joinSplitPubKey
self.cursor += 64 # joinSplitSig
if is_sapling_v4 and has_shielded:
self.cursor += 64 # bindingSig
return base_tx
class TxTime(namedtuple("Tx", "version time inputs outputs locktime")):
"""Class representing transaction that has a time field."""
class DeserializerTxTime(Deserializer):
def read_tx(self):
return TxTime(
self._read_le_int32(), # version
self._read_le_uint32(), # time
self._read_inputs(), # inputs
self._read_outputs(), # outputs
self._read_le_uint32(), # locktime
)
class DeserializerReddcoin(Deserializer):
def read_tx(self):
version = self._read_le_int32()
inputs = self._read_inputs()
outputs = self._read_outputs()
locktime = self._read_le_uint32()
if version > 1:
time = self._read_le_uint32()
else:
time = 0
return TxTime(version, time, inputs, outputs, locktime)
class DeserializerTxTimeAuxPow(DeserializerTxTime):
VERSION_AUXPOW = (1 << 8)
def is_merged_block(self):
start = self.cursor
self.cursor = 0
version = self._read_le_uint32()
self.cursor = start
if version & self.VERSION_AUXPOW:
return True
return False
def read_header(self, height, static_header_size):
"""Return the AuxPow block header bytes"""
start = self.cursor
version = self._read_le_uint32()
if version & self.VERSION_AUXPOW:
# We are going to calculate the block size then read it as bytes
self.cursor = start
self.cursor += static_header_size # Block normal header
self.read_tx() # AuxPow transaction
self.cursor += 32 # Parent block hash
merkle_size = self._read_varint()
self.cursor += 32 * merkle_size # Merkle branch
self.cursor += 4 # Index
merkle_size = self._read_varint()
self.cursor += 32 * merkle_size # Chain merkle branch
self.cursor += 4 # Chain index
self.cursor += 80 # Parent block header
header_end = self.cursor
else:
header_end = static_header_size
self.cursor = start
return self._read_nbytes(header_end)
class DeserializerBitcoinAtom(DeserializerSegWit):
FORK_BLOCK_HEIGHT = 505888
def read_header(self, height, static_header_size):
"""Return the block header bytes"""
header_len = static_header_size
if height >= self.FORK_BLOCK_HEIGHT:
header_len += 4 # flags
return self._read_nbytes(header_len)
class DeserializerGroestlcoin(DeserializerSegWit):
TX_HASH_FN = staticmethod(sha256)
class TxInputTokenPay(TxInput):
"""Class representing a TokenPay transaction input."""
OP_ANON_MARKER = 0xb9
# 2byte marker (cpubkey + sigc + sigr)
MIN_ANON_IN_SIZE = 2 + (33 + 32 + 32)
def _is_anon_input(self):
return (len(self.script) >= self.MIN_ANON_IN_SIZE and
self.script[0] == OpCodes.OP_RETURN and
self.script[1] == self.OP_ANON_MARKER)
def is_generation(self):
# Transactions coming in from stealth addresses are seen by
# the blockchain as newly minted coins. The reverse, where coins
# are sent TO a stealth address, are seen by the blockchain as
# a coin burn.
if self._is_anon_input():
return True
return super().is_generation()
class TxInputTokenPayStealth(
namedtuple("TxInput", "keyimage ringsize script sequence")):
"""Class representing a TokenPay stealth transaction input."""
def __str__(self):
script = self.script.hex()
keyimage = bytes(self.keyimage).hex()
return (f"Input({keyimage}, {self.ringsize[1]:d}, script={script}, sequence={self.sequence:d})")
def is_generation(self):
return True
def serialize(self):
return b''.join((
self.keyimage,
self.ringsize,
pack_varbytes(self.script),
pack_le_uint32(self.sequence),
))
class DeserializerTokenPay(DeserializerTxTime):
def _read_input(self):
txin = TxInputTokenPay(
self._read_nbytes(32), # prev_hash
self._read_le_uint32(), # prev_idx
self._read_varbytes(), # script
self._read_le_uint32(), # sequence
)
if txin._is_anon_input():
# Not sure if this is actually needed, and seems
# extra work for no immediate benefit, but it at
# least correctly represents a stealth input
raw = txin.serialize()
deserializer = Deserializer(raw)
txin = TxInputTokenPayStealth(
deserializer._read_nbytes(33), # keyimage
deserializer._read_nbytes(3), # ringsize
deserializer._read_varbytes(), # script
deserializer._read_le_uint32() # sequence
)
return txin
# Decred
class TxInputDcr(namedtuple("TxInput", "prev_hash prev_idx tree sequence")):
"""Class representing a Decred transaction input."""
def __str__(self):
prev_hash = hash_to_hex_str(self.prev_hash)
return (f"Input({prev_hash}, {self.prev_idx:d}, tree={self.tree}, sequence={self.sequence:d})")
def is_generation(self):
"""Test if an input is generation/coinbase like"""
return self.prev_idx == MINUS_1 and self.prev_hash == ZERO
class TxOutputDcr(namedtuple("TxOutput", "value version pk_script")):
"""Class representing a Decred transaction output."""
pass
class TxDcr(namedtuple("Tx", "version inputs outputs locktime expiry "
"witness")):
"""Class representing a Decred transaction."""
class DeserializerDecred(Deserializer):
@staticmethod
def blake256(data):
from blake256.blake256 import blake_hash
return blake_hash(data)
@staticmethod
def blake256d(data):
from blake256.blake256 import blake_hash
return blake_hash(blake_hash(data))
def read_tx(self):
return self._read_tx_parts(produce_hash=False)[0]
def read_tx_and_hash(self):
tx, tx_hash, vsize = self._read_tx_parts()
return tx, tx_hash
def read_tx_and_vsize(self):
tx, tx_hash, vsize = self._read_tx_parts(produce_hash=False)
return tx, vsize
def read_tx_block(self):
"""Returns a list of (deserialized_tx, tx_hash) pairs."""
read = self.read_tx_and_hash
txs = [read() for _ in range(self._read_varint())]
stxs = [read() for _ in range(self._read_varint())]
return txs + stxs
def read_tx_tree(self):
"""Returns a list of deserialized_tx without tx hashes."""
read_tx = self.read_tx
return [read_tx() for _ in range(self._read_varint())]
def _read_input(self):
return TxInputDcr(
self._read_nbytes(32), # prev_hash
self._read_le_uint32(), # prev_idx
self._read_byte(), # tree
self._read_le_uint32(), # sequence
)
def _read_output(self):
return TxOutputDcr(
self._read_le_int64(), # value
self._read_le_uint16(), # version
self._read_varbytes(), # pk_script
)
def _read_witness(self, fields):
read_witness_field = self._read_witness_field
assert fields == self._read_varint()
return [read_witness_field() for _ in range(fields)]
def _read_witness_field(self):
value_in = self._read_le_int64()
block_height = self._read_le_uint32()
block_index = self._read_le_uint32()
script = self._read_varbytes()
return value_in, block_height, block_index, script
def _read_tx_parts(self, produce_hash=True):
start = self.cursor
version = self._read_le_int32()
inputs = self._read_inputs()
outputs = self._read_outputs()
locktime = self._read_le_uint32()
expiry = self._read_le_uint32()
end_prefix = self.cursor
witness = self._read_witness(len(inputs))
if produce_hash:
# TxSerializeNoWitness << 16 == 0x10000
no_witness_header = pack_le_uint32(0x10000 | (version & 0xffff))
prefix_tx = no_witness_header + self.binary[start+4:end_prefix]
tx_hash = self.blake256(prefix_tx)
else:
tx_hash = None
return TxDcr(
version,
inputs,
outputs,
locktime,
expiry,
witness
), tx_hash, self.cursor - start
@@ -23,7 +23,7 @@ class SPVPing(NamedTuple):
pad_bytes: bytes
def encode(self):
return struct.pack(b'!lB64s', *self) # pylint: disable=not-an-iterable
return struct.pack(b'!lB64s', *self)
@staticmethod
def make() -> bytes:
@@ -49,7 +49,7 @@ class SPVPong(NamedTuple):
country: int
def encode(self):
return struct.pack(PONG_ENCODING, *self) # pylint: disable=not-an-iterable
return struct.pack(PONG_ENCODING, *self)
@staticmethod
def encode_address(address: str):
@@ -110,7 +110,6 @@ class SPVServerStatusProtocol(asyncio.DatagramProtocol):
self._min_delay = 1 / throttle_reqs_per_sec
self._allow_localhost = allow_localhost
self._allow_lan = allow_lan
self.closed = asyncio.Event()
def update_cached_response(self):
self._left_cache, self._right_cache = SPVPong.make_sans_source_address(
@@ -161,16 +160,13 @@ class SPVServerStatusProtocol(asyncio.DatagramProtocol):
def connection_made(self, transport) -> None:
self.transport = transport
self.closed.clear()
def connection_lost(self, exc: Optional[Exception]) -> None:
self.transport = None
self.closed.set()
async def close(self):
def close(self):
if self.transport:
self.transport.close()
await self.closed.wait()
class StatusServer:
@@ -188,9 +184,9 @@ class StatusServer:
await loop.create_datagram_endpoint(lambda: self._protocol, (interface, port))
log.info("started udp status server on %s:%i", interface, port)
async def stop(self):
def stop(self):
if self.is_running:
await self._protocol.close()
self._protocol.close()
self._protocol = None
@property
+361
View File
@@ -0,0 +1,361 @@
# Copyright (c) 2016-2017, Neil Booth
#
# All rights reserved.
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# and warranty status of this software.
"""Miscellaneous utility classes and functions."""
import array
import inspect
from ipaddress import ip_address
import logging
import re
import sys
from collections import Container, Mapping
from struct import pack, Struct
# Logging utilities
class ConnectionLogger(logging.LoggerAdapter):
"""Prepends a connection identifier to a logging message."""
def process(self, msg, kwargs):
conn_id = self.extra.get('conn_id', 'unknown')
return f'[{conn_id}] {msg}', kwargs
class CompactFormatter(logging.Formatter):
"""Strips the module from the logger name to leave the class only."""
def format(self, record):
record.name = record.name.rpartition('.')[-1]
return super().format(record)
def make_logger(name, *, handler, level):
"""Return the root ElectrumX logger."""
logger = logging.getLogger(name)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.propagate = False
return logger
def class_logger(path, classname):
"""Return a hierarchical logger for a class."""
return logging.getLogger(path).getChild(classname)
# Method decorator. To be used for calculations that will always
# deliver the same result. The method cannot take any arguments
# and should be accessed as an attribute.
class cachedproperty:
def __init__(self, f):
self.f = f
def __get__(self, obj, type):
obj = obj or type
value = self.f(obj)
setattr(obj, self.f.__name__, value)
return value
def formatted_time(t, sep=' '):
"""Return a number of seconds as a string in days, hours, mins and
maybe secs."""
t = int(t)
fmts = (('{:d}d', 86400), ('{:02d}h', 3600), ('{:02d}m', 60))
parts = []
for fmt, n in fmts:
val = t // n
if parts or val:
parts.append(fmt.format(val))
t %= n
if len(parts) < 3:
parts.append(f'{t:02d}s')
return sep.join(parts)
def deep_getsizeof(obj):
"""Find the memory footprint of a Python object.
Based on code from code.tutsplus.com: http://goo.gl/fZ0DXK
This is a recursive function that drills down a Python object graph
like a dictionary holding nested dictionaries with lists of lists
and tuples and sets.
The sys.getsizeof function does a shallow size of only. It counts each
object inside a container as pointer only regardless of how big it
really is.
"""
ids = set()
def size(o):
if id(o) in ids:
return 0
r = sys.getsizeof(o)
ids.add(id(o))
if isinstance(o, (str, bytes, bytearray, array.array)):
return r
if isinstance(o, Mapping):
return r + sum(size(k) + size(v) for k, v in o.items())
if isinstance(o, Container):
return r + sum(size(x) for x in o)
return r
return size(obj)
def subclasses(base_class, strict=True):
"""Return a list of subclasses of base_class in its module."""
def select(obj):
return (inspect.isclass(obj) and issubclass(obj, base_class) and
(not strict or obj != base_class))
pairs = inspect.getmembers(sys.modules[base_class.__module__], select)
return [pair[1] for pair in pairs]
def chunks(items, size):
"""Break up items, an iterable, into chunks of length size."""
for i in range(0, len(items), size):
yield items[i: i + size]
def resolve_limit(limit):
if limit is None:
return -1
assert isinstance(limit, int) and limit >= 0
return limit
def bytes_to_int(be_bytes):
"""Interprets a big-endian sequence of bytes as an integer"""
return int.from_bytes(be_bytes, 'big')
def int_to_bytes(value):
"""Converts an integer to a big-endian sequence of bytes"""
return value.to_bytes((value.bit_length() + 7) // 8, 'big')
def increment_byte_string(bs):
"""Return the lexicographically next byte string of the same length.
Return None if there is none (when the input is all 0xff bytes)."""
for n in range(1, len(bs) + 1):
if bs[-n] != 0xff:
return bs[:-n] + bytes([bs[-n] + 1]) + bytes(n - 1)
return None
class LogicalFile:
"""A logical binary file split across several separate files on disk."""
def __init__(self, prefix, digits, file_size):
digit_fmt = f'{{:0{digits:d}d}}'
self.filename_fmt = prefix + digit_fmt
self.file_size = file_size
def read(self, start, size=-1):
"""Read up to size bytes from the virtual file, starting at offset
start, and return them.
If size is -1 all bytes are read."""
parts = []
while size != 0:
try:
with self.open_file(start, False) as f:
part = f.read(size)
if not part:
break
except FileNotFoundError:
break
parts.append(part)
start += len(part)
if size > 0:
size -= len(part)
return b''.join(parts)
def write(self, start, b):
"""Write the bytes-like object, b, to the underlying virtual file."""
while b:
size = min(len(b), self.file_size - (start % self.file_size))
with self.open_file(start, True) as f:
f.write(b if size == len(b) else b[:size])
b = b[size:]
start += size
def open_file(self, start, create):
"""Open the virtual file and seek to start. Return a file handle.
Raise FileNotFoundError if the file does not exist and create
is False.
"""
file_num, offset = divmod(start, self.file_size)
filename = self.filename_fmt.format(file_num)
f = open_file(filename, create)
f.seek(offset)
return f
def open_file(filename, create=False):
"""Open the file name. Return its handle."""
try:
return open(filename, 'rb+')
except FileNotFoundError:
if create:
return open(filename, 'wb+')
raise
def open_truncate(filename):
"""Open the file name. Return its handle."""
return open(filename, 'wb+')
def address_string(address):
"""Return an address as a correctly formatted string."""
fmt = '{}:{:d}'
host, port = address
try:
host = ip_address(host)
except ValueError:
pass
else:
if host.version == 6:
fmt = '[{}]:{:d}'
return fmt.format(host, port)
# See http://stackoverflow.com/questions/2532053/validate-a-hostname-string
# Note underscores are valid in domain names, but strictly invalid in host
# names. We ignore that distinction.
SEGMENT_REGEX = re.compile("(?!-)[A-Z_\\d-]{1,63}(?<!-)$", re.IGNORECASE)
def is_valid_hostname(hostname):
if len(hostname) > 255:
return False
# strip exactly one dot from the right, if present
if hostname and hostname[-1] == ".":
hostname = hostname[:-1]
return all(SEGMENT_REGEX.match(x) for x in hostname.split("."))
def protocol_tuple(s):
"""Converts a protocol version number, such as "1.0" to a tuple (1, 0).
If the version number is bad, (0, ) indicating version 0 is returned."""
try:
return tuple(int(part) for part in s.split('.'))
except Exception:
return (0, )
def version_string(ptuple):
"""Convert a version tuple such as (1, 2) to "1.2".
There is always at least one dot, so (1, ) becomes "1.0"."""
while len(ptuple) < 2:
ptuple += (0, )
return '.'.join(str(p) for p in ptuple)
def protocol_version(client_req, min_tuple, max_tuple):
"""Given a client's protocol version string, return a pair of
protocol tuples:
(negotiated version, client min request)
If the request is unsupported, the negotiated protocol tuple is
None.
"""
if client_req is None:
client_min = client_max = min_tuple
else:
if isinstance(client_req, list) and len(client_req) == 2:
client_min, client_max = client_req
else:
client_min = client_max = client_req
client_min = protocol_tuple(client_min)
client_max = protocol_tuple(client_max)
result = min(client_max, max_tuple)
if result < max(client_min, min_tuple) or result == (0, ):
result = None
return result, client_min
struct_le_i = Struct('<i')
struct_le_q = Struct('<q')
struct_le_H = Struct('<H')
struct_le_I = Struct('<I')
struct_le_Q = Struct('<Q')
struct_be_H = Struct('>H')
struct_be_I = Struct('>I')
structB = Struct('B')
unpack_le_int32_from = struct_le_i.unpack_from
unpack_le_int64_from = struct_le_q.unpack_from
unpack_le_uint16_from = struct_le_H.unpack_from
unpack_le_uint32_from = struct_le_I.unpack_from
unpack_le_uint64_from = struct_le_Q.unpack_from
unpack_be_uint16_from = struct_be_H.unpack_from
unpack_be_uint32_from = struct_be_I.unpack_from
unpack_be_uint64 = lambda x: int.from_bytes(x, byteorder='big')
pack_le_int32 = struct_le_i.pack
pack_le_int64 = struct_le_q.pack
pack_le_uint16 = struct_le_H.pack
pack_le_uint32 = struct_le_I.pack
pack_be_uint64 = lambda x: x.to_bytes(8, byteorder='big')
pack_be_uint16 = lambda x: x.to_bytes(2, byteorder='big')
pack_be_uint32 = struct_be_I.pack
pack_byte = structB.pack
hex_to_bytes = bytes.fromhex
def pack_varint(n):
if n < 253:
return pack_byte(n)
if n < 65536:
return pack_byte(253) + pack_le_uint16(n)
if n < 4294967296:
return pack_byte(254) + pack_le_uint32(n)
return pack_byte(255) + pack_le_uint64(n)
def pack_varbytes(data):
return pack_varint(len(data)) + data
+3
View File
@@ -0,0 +1,3 @@
# need this to avoid circular import
PROTOCOL_MIN = (0, 54, 0)
PROTOCOL_MAX = (0, 199, 0)
+55
View File
@@ -0,0 +1,55 @@
import asyncio
from weakref import WeakSet
from aiohttp.web import Application, AppRunner, WebSocketResponse, TCPSite
from aiohttp.http_websocket import WSMsgType, WSCloseCode
class AdminWebSocket:
def __init__(self, manager):
self.manager = manager
self.app = Application()
self.app['websockets'] = WeakSet()
self.app.router.add_get('/', self.on_connect)
self.app.on_shutdown.append(self.on_shutdown)
self.runner = AppRunner(self.app)
async def on_status(self, _):
if not self.app['websockets']:
return
self.send_message({
'type': 'status',
'height': self.manager.daemon.cached_height(),
})
def send_message(self, msg):
for web_socket in self.app['websockets']:
asyncio.create_task(web_socket.send_json(msg))
async def start(self):
await self.runner.setup()
await TCPSite(self.runner, self.manager.env.websocket_host, self.manager.env.websocket_port).start()
async def stop(self):
await self.runner.cleanup()
async def on_connect(self, request):
web_socket = WebSocketResponse()
await web_socket.prepare(request)
self.app['websockets'].add(web_socket)
try:
async for msg in web_socket:
if msg.type == WSMsgType.TEXT:
await self.on_status(None)
elif msg.type == WSMsgType.ERROR:
print('web socket connection closed with exception %s' %
web_socket.exception())
finally:
self.app['websockets'].discard(web_socket)
return web_socket
@staticmethod
async def on_shutdown(app):
for web_socket in set(app['websockets']):
await web_socket.close(code=WSCloseCode.GOING_AWAY, message='Server shutdown')
+43 -46
View File
@@ -1,9 +1,19 @@
import struct
import hashlib
import logging
import typing
import asyncio
from binascii import hexlify, unhexlify
from typing import List, Iterable, Optional, Tuple
import ecdsa
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.serialization import load_der_public_key
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.asymmetric.utils import Prehashed
from cryptography.exceptions import InvalidSignature
from lbry.error import InsufficientFundsError
from lbry.crypto.hash import hash160, sha256
from lbry.crypto.base58 import Base58
@@ -18,7 +28,6 @@ from .constants import COIN, DUST, NULL_HASH32
from .bcd_data_stream import BCDataStream
from .hash import TXRef, TXRefImmutable
from .util import ReadOnlyList
from .bip32 import PrivateKey, PublicKey
if typing.TYPE_CHECKING:
from lbry.wallet.account import Account
@@ -145,14 +154,6 @@ class Input(InputOutput):
script = InputScript.redeem_pubkey_hash(cls.NULL_SIGNATURE, cls.NULL_PUBLIC_KEY)
return cls(txo.ref, script)
@classmethod
def spend_time_lock(cls, txo: 'Output', script_source: bytes) -> 'Input':
""" Create an input to spend time lock script."""
script = InputScript.redeem_time_lock_script_hash(
cls.NULL_SIGNATURE, cls.NULL_PUBLIC_KEY, script_source=script_source
)
return cls(txo.ref, script)
@property
def amount(self) -> int:
""" Amount this input adds to the transaction. """
@@ -221,8 +222,7 @@ class Output(InputOutput):
is_my_output: Optional[bool] = None, is_my_input: Optional[bool] = None,
sent_supports: Optional[int] = None, sent_tips: Optional[int] = None,
received_tips: Optional[int] = None,
channel: Optional['Output'] = None,
private_key: Optional[PrivateKey] = None
channel: Optional['Output'] = None, private_key: Optional[str] = None
) -> None:
super().__init__(tx_ref, position)
self.amount = amount
@@ -235,7 +235,7 @@ class Output(InputOutput):
self.sent_tips = sent_tips
self.received_tips = received_tips
self.channel = channel
self.private_key: PrivateKey = private_key
self.private_key = private_key
self.purchase: 'Output' = None # txo containing purchase metadata
self.purchased_claim: 'Output' = None # resolved claim pointed to by purchase
self.purchase_receipt: 'Output' = None # txo representing purchase receipt for this claim
@@ -425,15 +425,25 @@ class Output(InputOutput):
]
return sha256(b''.join(pieces))
def get_encoded_signature(self):
signature = hexlify(self.signable.signature)
r = int(signature[:int(len(signature)/2)], 16)
s = int(signature[int(len(signature)/2):], 16)
return ecdsa.util.sigencode_der(r, s, len(signature)*4)
@staticmethod
def is_signature_valid(signature, digest, public_key_bytes):
return PublicKey\
.from_compressed(public_key_bytes)\
.verify(signature, digest)
def is_signature_valid(encoded_signature, signature_digest, public_key_bytes):
try:
public_key = load_der_public_key(public_key_bytes, default_backend())
public_key.verify(encoded_signature, signature_digest, ec.ECDSA(Prehashed(hashes.SHA256())))
return True
except (ValueError, InvalidSignature):
pass
return False
def is_signed_by(self, channel: 'Output', ledger=None):
return self.is_signature_valid(
self.signable.signature,
self.get_encoded_signature(),
self.get_signature_digest(ledger),
channel.claim.channel.public_key_bytes
)
@@ -446,27 +456,29 @@ class Output(InputOutput):
self.signable.signing_channel_hash,
self.signable.to_message_bytes()
]))
self.signable.signature = channel.private_key.sign_compact(digest)
self.signable.signature = channel.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
self.script.generate()
def sign_data(self, data: bytes, timestamp: str) -> str:
def sign_data(self, data:bytes, timestamp:str) -> str:
pieces = [timestamp.encode(), self.claim_hash, data]
digest = sha256(b''.join(pieces))
signature = self.private_key.sign_compact(digest)
signature = self.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
return hexlify(signature).decode()
def clear_signature(self):
self.channel = None
self.signable.clear_signature()
def set_channel_private_key(self, private_key: PrivateKey):
self.private_key = private_key
self.claim.channel.public_key_bytes = private_key.public_key.pubkey_bytes
async def generate_channel_private_key(self):
self.private_key = await asyncio.get_event_loop().run_in_executor(
None, ecdsa.SigningKey.generate, ecdsa.SECP256k1, None, hashlib.sha256
)
self.claim.channel.public_key_bytes = self.private_key.get_verifying_key().to_der()
self.script.generate()
return self.private_key
def is_channel_private_key(self, private_key: PrivateKey):
return self.claim.channel.public_key_bytes == private_key.public_key.pubkey_bytes
def is_channel_private_key(self, private_key):
return self.claim.channel.public_key_bytes == private_key.get_verifying_key().to_der()
@classmethod
def pay_claim_name_pubkey_hash(
@@ -718,11 +730,8 @@ class Transaction:
stream.write_compact_size(len(self._inputs))
for i, txin in enumerate(self._inputs):
if signing_input == i:
if txin.script.is_script_hash:
txin.serialize_to(stream, txin.script.values['script'].source)
else:
assert txin.txo_ref.txo is not None
txin.serialize_to(stream, txin.txo_ref.txo.script.source)
assert txin.txo_ref.txo is not None
txin.serialize_to(stream, txin.txo_ref.txo.script.source)
else:
txin.serialize_to(stream, b'')
self._serialize_outputs(stream)
@@ -865,19 +874,16 @@ class Transaction:
def signature_hash_type(hash_type):
return hash_type
async def sign(self, funding_accounts: Iterable['Account'], extra_keys: dict = None):
async def sign(self, funding_accounts: Iterable['Account']):
self._reset()
ledger, wallet = self.ensure_all_have_same_ledger_and_wallet(funding_accounts)
for i, txi in enumerate(self._inputs):
assert txi.script is not None
assert txi.txo_ref.txo is not None
txo_script = txi.txo_ref.txo.script
if txo_script.is_pay_pubkey_hash or txo_script.is_pay_script_hash:
if 'pubkey_hash' in txo_script.values:
address = ledger.hash160_to_address(txo_script.values.get('pubkey_hash', ''))
private_key = await ledger.get_private_key_for_address(wallet, address)
else:
private_key = next(iter(extra_keys.values()))
if txo_script.is_pay_pubkey_hash:
address = ledger.hash160_to_address(txo_script.values['pubkey_hash'])
private_key = await ledger.get_private_key_for_address(wallet, address)
assert private_key is not None, 'Cannot find private key for signing output.'
tx = self._serialize_for_signature(i)
txi.script.values['signature'] = \
@@ -951,15 +957,6 @@ class Transaction:
data = Output.add_purchase_data(Purchase(claim_id))
return cls.create([], [payment, data], funding_accounts, change_account)
@classmethod
async def spend_time_lock(cls, time_locked_txo: Output, script: bytes, account: 'Account'):
txi = Input.spend_time_lock(time_locked_txo, script)
txi.sequence = 0xFFFFFFFE
tx = await cls.create([txi], [], [account], account, sign=False)
tx.locktime = txi.script.values['script'].values['height']
tx._reset()
return tx
@property
def my_inputs(self):
for txi in self.inputs:
+1 -2
View File
@@ -203,12 +203,11 @@ class Wallet:
return True
return False
async def unlock(self, password):
def unlock(self, password):
for account in self.accounts:
if account.encrypted:
if not account.decrypt(password):
return False
await account.deterministic_channel_keys.ensure_cache_primed()
self.encryption_password = password
return True
+9 -4
View File
@@ -40,17 +40,22 @@ def checkrecord(record, expected_winner, expected_claim):
async def checkcontrolling(daemon: Daemon, db: SQLDB):
records, names, futs = [], [], []
records, claim_ids, names, futs = [], [], [], []
for record in db.get_claims('claimtrie.claim_hash as is_controlling, claim.*', is_controlling=True):
records.append(record)
claim_id = hex_reverted(record['claim_hash'])
names.append((record['normalized'], (claim_id,), "", True)) # last parameter is IncludeValues
claim_ids.append((claim_id,))
names.append((record['normalized'],))
if len(names) > 50000:
futs.append(daemon._send_vector('getclaimsfornamebyid', names))
futs.append(daemon._send_vector('getvalueforname', names[:]))
futs.append(daemon._send_vector('getclaimbyid', claim_ids[:]))
names.clear()
claim_ids.clear()
if names:
futs.append(daemon._send_vector('getclaimsfornamebyid', names))
futs.append(daemon._send_vector('getvalueforname', names[:]))
futs.append(daemon._send_vector('getclaimbyid', claim_ids[:]))
names.clear()
claim_ids.clear()
while futs:
winners, claims = futs.pop(0), futs.pop(0)
+3 -3
View File
@@ -1,12 +1,12 @@
#!/bin/bash
SNAPSHOT_HEIGHT="1072108"
SNAPSHOT_HEIGHT="1049658"
HUB_VOLUME_PATH="/var/lib/docker/volumes/${USER}_wallet_server"
ES_VOLUME_PATH="/var/lib/docker/volumes/${USER}_es01"
SNAPSHOT_TAR_NAME="wallet_server_snapshot_${SNAPSHOT_HEIGHT}.tar.gz"
ES_SNAPSHOT_TAR_NAME="es_snapshot_${SNAPSHOT_HEIGHT}.tar.gz"
SNAPSHOT_TAR_NAME="wallet_server_snapshot_${SNAPSHOT_HEIGHT}.tar"
ES_SNAPSHOT_TAR_NAME="es_snapshot_${SNAPSHOT_HEIGHT}.tar"
SNAPSHOT_URL="https://snapshots.lbry.com/hub/${SNAPSHOT_TAR_NAME}"
ES_SNAPSHOT_URL="https://snapshots.lbry.com/hub/${ES_SNAPSHOT_TAR_NAME}"
+17 -9
View File
@@ -2,7 +2,7 @@ import os
import re
import io
import sys
import yaml
import json
import argparse
import unittest
from datetime import date
@@ -25,17 +25,25 @@ AREA_RENAME = {
def get_github():
config_path = os.path.expanduser('~/.config/gh/hosts.yml')
config_path = os.path.expanduser('~/.lbry-release-tool.json')
if os.path.exists(config_path):
with open(config_path, 'r') as config_file:
config = yaml.load(config_file, Loader=yaml.FullLoader)
return github3.login(token=config['github.com']['oauth_token'])
config = json.load(config_file)
return github3.login(token=config['token'])
print('To run release tool you need to first login using the github cli:')
print('')
print(' $ gh auth login')
print('')
sys.exit(1)
token = os.environ.get("GH_TOKEN")
if not token:
print('GitHub Credentials')
username = input('username: ')
password = getpass('password: ')
gh = github3.authorize(
username, password, ['repo'], 'lbry release tool',
two_factor_callback=lambda: input('Enter 2FA: ')
)
with open(config_path, 'w') as config_file:
json.dump({'token': gh.token}, config_file)
token = gh.token
return github3.login(token=token)
def get_labels(pr, prefix):
+1 -1
View File
@@ -18,7 +18,7 @@ ignore=words,server,rpc,schema,winpaths.py,migrator,undecorated.py
max-parents=10
max-args=10
max-line-length=120
good-names=T,t,n,i,j,k,x,y,s,f,d,h,c,e,op,db,tx,io,cachedproperty,log,id,r,iv,ts,l,pk
good-names=T,t,n,i,j,k,x,y,s,f,d,h,c,e,op,db,tx,io,cachedproperty,log,id,r,iv,ts,l
valid-metaclass-classmethod-first-arg=mcs
disable=
c-extension-no-member,
+13 -21
View File
@@ -7,11 +7,9 @@ BASE = os.path.dirname(__file__)
with open(os.path.join(BASE, 'README.md'), encoding='utf-8') as fh:
long_description = fh.read()
ROCKSDB = []
if sys.platform.startswith('linux') or sys.platform.startswith('darwin'):
ROCKSDB.append('lbry-rocksdb==0.8.2')
PLYVEL = []
if sys.platform.startswith('linux'):
PLYVEL.append('plyvel==1.3.0')
setup(
name=__name__,
@@ -30,14 +28,16 @@ setup(
entry_points={
'console_scripts': [
'lbrynet=lbry.extras.cli:main',
'orchstr8=lbry.wallet.orchstr8.cli:main'
'lbry-hub=lbry.wallet.server.cli:main',
'orchstr8=lbry.wallet.orchstr8.cli:main',
'lbry-hub-elastic-sync=lbry.wallet.server.db.elasticsearch.sync:run_elastic_sync'
],
},
install_requires=[
'aiohttp==3.7.4',
'aiohttp==3.5.4',
'aioupnp==0.0.18',
'appdirs==1.4.3',
'certifi>=2021.10.08',
'certifi>=2018.11.29',
'colorama==0.3.7',
'distro==1.4.0',
'base58==1.0.0',
@@ -49,7 +49,7 @@ setup(
'ecdsa==0.13.3',
'pyyaml==5.3.1',
'docopt==0.6.2',
'hachoir==3.1.2',
'hachoir',
'multidict==4.6.1',
'coincurve==15.0.0',
'pbkdf2==1.3',
@@ -57,20 +57,12 @@ setup(
'pylru==1.1.0',
'elasticsearch==7.10.1',
'grpcio==1.38.0',
'filetype==1.0.9',
] + ROCKSDB,
'filetype==1.0.9'
] + PLYVEL,
extras_require={
'torrent': ['lbry-libtorrent'],
'lint': [
'pylint==2.10.0'
],
'test': [
'coverage',
'jsonschema==4.4.0',
],
'hub': [
'hub@git+https://github.com/lbryio/hub.git@76dd9c392b776a2823015762814f375794120076'
]
'lint': ['pylint==2.10.0'],
'test': ['coverage'],
},
classifiers=[
'Framework :: AsyncIO',
@@ -1,11 +1,5 @@
from binascii import hexlify, unhexlify
from lbry.testcase import CommandTestCase
from lbry.wallet.script import InputScript
from lbry.wallet.dewies import dewies_to_lbc
from lbry.wallet.account import DeterministicChannelKeyManager
from lbry.crypto.hash import hash160
from lbry.crypto.base58 import Base58
def extract(d, keys):
@@ -66,30 +60,15 @@ class AccountManagement(CommandTestCase):
self.assertEqual(accounts['items'][0]['name'], 'recreated account')
async def test_wallet_migration(self):
old_id, new_id, valid_key = (
'mi9E8KqFfW5ngktU22pN2jpgsdf81ZbsGY',
'mqs77XbdnuxWN4cXrjKbSoGLkvAHa4f4B8',
'-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIBZRTZ7tHnYCH3IE9mCo95'
'466L/ShYFhXGrjmSMFJw8eoAcGBSuBBAAK\noUQDQgAEmucoPz9nI+ChZrfhnh'
'0RZ/bcX0r2G0pYBmoNKovtKzXGa8y07D66MWsW\nqXptakqO/9KddIkBu5eJNS'
'UZzQCxPQ==\n-----END EC PRIVATE KEY-----\n'
)
# null certificates should get deleted
self.account.channel_keys = {
new_id: 'not valid key',
'foo': 'bar',
}
await self.channel_create('@foo1')
await self.channel_create('@foo2')
await self.channel_create('@foo3')
keys = list(self.account.channel_keys.keys())
self.account.channel_keys[keys[0]] = None
self.account.channel_keys[keys[1]] = "some invalid junk"
await self.account.maybe_migrate_certificates()
self.assertEqual(self.account.channel_keys, {})
self.account.channel_keys = {
new_id: 'not valid key',
'foo': 'bar',
'invalid address': valid_key,
}
await self.account.maybe_migrate_certificates()
self.assertEqual(self.account.channel_keys, {
new_id: valid_key
})
self.assertEqual(list(self.account.channel_keys.keys()), [keys[2]])
async def assertFindsClaims(self, claim_names, awaitable):
self.assertEqual(claim_names, [txo.claim_name for txo in (await awaitable)['items']])
@@ -106,7 +85,7 @@ class AccountManagement(CommandTestCase):
second_account = await self.daemon.jsonrpc_account_create('second account')
tx = await self.daemon.jsonrpc_account_send(
'0.05', await self.daemon.jsonrpc_address_unused(account_id=second_account.id), blocking=True
'0.05', await self.daemon.jsonrpc_address_unused(account_id=second_account.id)
)
await self.confirm_tx(tx.id)
await self.assertOutputAmount(['0.05', '9.949876'], utxo_list())
@@ -195,122 +174,3 @@ class AccountManagement(CommandTestCase):
bad_address = address[0:20] + '9999999' + address[27:]
with self.assertRaisesRegex(Exception, f"'{bad_address}' is not a valid address"):
await self.daemon.jsonrpc_account_send('0.1', addresses=[bad_address])
async def test_hybrid_channel_keys(self):
# non-deterministic channel
self.account.channel_keys = {
'mqs77XbdnuxWN4cXrjKbSoGLkvAHa4f4B8':
'-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIBZRTZ7tHnYCH3IE9mCo95'
'466L/ShYFhXGrjmSMFJw8eoAcGBSuBBAAK\noUQDQgAEmucoPz9nI+ChZrfhnh'
'0RZ/bcX0r2G0pYBmoNKovtKzXGa8y07D66MWsW\nqXptakqO/9KddIkBu5eJNS'
'UZzQCxPQ==\n-----END EC PRIVATE KEY-----\n'
}
channel1 = await self.create_nondeterministic_channel('@foo1', '1.0', unhexlify(
'3056301006072a8648ce3d020106052b8104000a034200049ae7283f3f6723e0a1'
'66b7e19e1d1167f6dc5f4af61b4a58066a0d2a8bed2b35c66bccb4ec3eba316b16'
'a97a6d6a4a8effd29d748901bb9789352519cd00b13d'
))
await self.confirm_tx(channel1['txid'])
# deterministic channel
channel2 = await self.channel_create('@foo2')
await self.stream_create('stream-in-channel1', '0.01', channel_id=self.get_claim_id(channel1))
await self.stream_create('stream-in-channel2', '0.01', channel_id=self.get_claim_id(channel2))
resolved_stream1 = await self.resolve('@foo1/stream-in-channel1')
self.assertEqual('stream-in-channel1', resolved_stream1['name'])
self.assertTrue(resolved_stream1['is_channel_signature_valid'])
resolved_stream2 = await self.resolve('@foo2/stream-in-channel2')
self.assertEqual('stream-in-channel2', resolved_stream2['name'])
self.assertTrue(resolved_stream2['is_channel_signature_valid'])
async def test_deterministic_channel_keys(self):
seed = self.account.seed
keys = self.account.deterministic_channel_keys
# create two channels and make sure they have different keys
channel1a = await self.channel_create('@foo1')
channel2a = await self.channel_create('@foo2')
self.assertNotEqual(
channel1a['outputs'][0]['value']['public_key'],
channel2a['outputs'][0]['value']['public_key'],
)
# start another daemon from the same seed
self.daemon2 = await self.add_daemon(seed=seed)
channel2b, channel1b = (await self.daemon2.jsonrpc_channel_list())['items']
# both daemons end up with the same channel signing keys automagically
self.assertTrue(channel1b.has_private_key)
self.assertEqual(
channel1a['outputs'][0]['value']['public_key_id'],
channel1b.private_key.address
)
self.assertTrue(channel2b.has_private_key)
self.assertEqual(
channel2a['outputs'][0]['value']['public_key_id'],
channel2b.private_key.address
)
# repeatedly calling next channel key returns the same key when not used
current_known = keys.last_known
next_key = await keys.generate_next_key()
self.assertEqual(current_known, keys.last_known)
self.assertEqual(next_key.address, (await keys.generate_next_key()).address)
# again, should be idempotent
next_key = await keys.generate_next_key()
self.assertEqual(current_known, keys.last_known)
self.assertEqual(next_key.address, (await keys.generate_next_key()).address)
# create third channel while both daemons running, second daemon should pick it up
channel3a = await self.channel_create('@foo3')
self.assertEqual(current_known+1, keys.last_known)
self.assertNotEqual(next_key.address, (await keys.generate_next_key()).address)
channel3b, = (await self.daemon2.jsonrpc_channel_list(name='@foo3'))['items']
self.assertTrue(channel3b.has_private_key)
self.assertEqual(
channel3a['outputs'][0]['value']['public_key_id'],
channel3b.private_key.address
)
# channel key cache re-populated after simulated restart
# reset cache
self.account.deterministic_channel_keys = DeterministicChannelKeyManager(self.account)
channel3c, channel2c, channel1c = (await self.daemon.jsonrpc_channel_list())['items']
self.assertFalse(channel1c.has_private_key)
self.assertFalse(channel2c.has_private_key)
self.assertFalse(channel3c.has_private_key)
# repopulate cache
await self.account.deterministic_channel_keys.ensure_cache_primed()
self.assertEqual(self.account.deterministic_channel_keys.last_known, keys.last_known)
channel3c, channel2c, channel1c = (await self.daemon.jsonrpc_channel_list())['items']
self.assertTrue(channel1c.has_private_key)
self.assertTrue(channel2c.has_private_key)
self.assertTrue(channel3c.has_private_key)
async def test_time_locked_transactions(self):
address = await self.account.receiving.get_or_create_usable_address()
private_key = await self.ledger.get_private_key_for_address(self.wallet, address)
script = InputScript(
template=InputScript.TIME_LOCK_SCRIPT,
values={'height': 210, 'pubkey_hash': self.ledger.address_to_hash160(address)}
)
script_address = self.ledger.hash160_to_script_address(hash160(script.source))
script_source = hexlify(script.source).decode()
await self.assertBalance(self.account, '10.0')
tx = await self.daemon.jsonrpc_account_send('4.0', script_address)
await self.confirm_tx(tx.id)
await self.generate(510)
await self.assertBalance(self.account, '5.999877')
tx = await self.daemon.jsonrpc_account_deposit(
tx.id, 0, script_source,
Base58.encode_check(self.ledger.private_key_to_wif(private_key.private_key_bytes))
)
await self.confirm_tx(tx.id)
await self.assertBalance(self.account, '9.9997545')
@@ -9,37 +9,35 @@ class BlockchainReorganizationTests(CommandTestCase):
VERBOSITY = logging.WARN
async def assertBlockHash(self, height):
bp = self.conductor.spv_node.writer
reader = self.conductor.spv_node.server
bp = self.conductor.spv_node.server.bp
def get_txids():
return [
reader.db.fs_tx_hash(tx_num)[0][::-1].hex()
bp.db.fs_tx_hash(tx_num)[0][::-1].hex()
for tx_num in range(bp.db.tx_counts[height - 1], bp.db.tx_counts[height])
]
block_hash = await self.blockchain.get_block_hash(height)
self.assertEqual(block_hash, (await self.ledger.headers.hash(height)).decode())
self.assertEqual(block_hash, (await reader.db.fs_block_hashes(height, 1))[0][::-1].hex())
self.assertEqual(block_hash, (await bp.db.fs_block_hashes(height, 1))[0][::-1].hex())
txids = await asyncio.get_event_loop().run_in_executor(None, get_txids)
txs = await reader.db.get_transactions_and_merkles(txids)
txs = await bp.db.get_transactions_and_merkles(txids)
block_txs = (await bp.daemon.deserialised_block(block_hash))['tx']
self.assertSetEqual(set(block_txs), set(txs.keys()), msg='leveldb/lbrycrd is missing transactions')
self.assertListEqual(block_txs, list(txs.keys()), msg='leveldb/lbrycrd transactions are of order')
async def test_reorg(self):
bp = self.conductor.spv_node.writer
bp = self.conductor.spv_node.server.bp
bp.reorg_count_metric.set(0)
# invalidate current block, move forward 2
height = 206
self.assertEqual(self.ledger.headers.height, height)
await self.assertBlockHash(height)
block_hash = (await self.ledger.headers.hash(206)).decode()
await self.blockchain.invalidate_block(block_hash)
await self.blockchain.invalidate_block((await self.ledger.headers.hash(206)).decode())
await self.blockchain.generate(2)
await asyncio.wait_for(self.on_header(207), 3.0)
await self.ledger.on_header.where(lambda e: e.height == 207)
self.assertEqual(self.ledger.headers.height, 207)
await self.assertBlockHash(206)
await self.assertBlockHash(207)
@@ -48,14 +46,14 @@ class BlockchainReorganizationTests(CommandTestCase):
# invalidate current block, move forward 3
await self.blockchain.invalidate_block((await self.ledger.headers.hash(206)).decode())
await self.blockchain.generate(3)
await asyncio.wait_for(self.on_header(208), 3.0)
await self.ledger.on_header.where(lambda e: e.height == 208)
self.assertEqual(self.ledger.headers.height, 208)
await self.assertBlockHash(206)
await self.assertBlockHash(207)
await self.assertBlockHash(208)
self.assertEqual(2, bp.reorg_count_metric._samples()[0][2])
await self.blockchain.generate(3)
await asyncio.wait_for(self.on_header(211), 3.0)
await self.ledger.on_header.where(lambda e: e.height == 211)
await self.assertBlockHash(209)
await self.assertBlockHash(210)
await self.assertBlockHash(211)
@@ -64,7 +62,7 @@ class BlockchainReorganizationTests(CommandTestCase):
)
await self.ledger.wait(still_valid)
await self.blockchain.generate(1)
await asyncio.wait_for(self.on_header(212), 1.0)
await self.ledger.on_header.where(lambda e: e.height == 212)
claim_id = still_valid.outputs[0].claim_id
c1 = (await self.resolve(f'still-valid#{claim_id}'))['claim_id']
c2 = (await self.resolve(f'still-valid#{claim_id[:2]}'))['claim_id']
@@ -73,7 +71,7 @@ class BlockchainReorganizationTests(CommandTestCase):
abandon_tx = await self.daemon.jsonrpc_stream_abandon(claim_id=claim_id)
await self.blockchain.generate(1)
await asyncio.wait_for(self.on_header(213), 1.0)
await self.ledger.on_header.where(lambda e: e.height == 213)
c1 = await self.resolve(f'still-valid#{still_valid.outputs[0].claim_id}')
c2 = await self.daemon.jsonrpc_resolve([f'still-valid#{claim_id[:2]}'])
c3 = await self.daemon.jsonrpc_resolve([f'still-valid'])
@@ -114,10 +112,11 @@ class BlockchainReorganizationTests(CommandTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
await asyncio.wait_for(self.on_header(209), 3.0)
# wait for the client to catch up and verify the reorg
await asyncio.wait_for(self.on_header(209), 3.0)
await self.assertBlockHash(207)
await self.assertBlockHash(208)
await self.assertBlockHash(209)
@@ -143,8 +142,9 @@ class BlockchainReorganizationTests(CommandTestCase):
# broadcast the claim in a different block
new_txid = await self.blockchain.sendrawtransaction(hexlify(broadcast_tx.raw).decode())
self.assertEqual(broadcast_tx.id, new_txid)
await self.blockchain.generate(1)
# wait for the client to catch up
await asyncio.wait_for(self.on_header(210), 1.0)
# verify the claim is in the new block and that it is returned by claim_search
@@ -191,7 +191,7 @@ class BlockchainReorganizationTests(CommandTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -222,6 +222,8 @@ class BlockchainReorganizationTests(CommandTestCase):
new_txid = await self.blockchain.sendrawtransaction(hexlify(broadcast_tx.raw).decode())
self.assertEqual(broadcast_tx.id, new_txid)
await self.blockchain.generate(1)
# wait for the client to catch up
await asyncio.wait_for(self.on_header(210), 1.0)
# verify the claim is in the new block and that it is returned by claim_search
+21 -34
View File
@@ -1,17 +1,13 @@
import asyncio
import hub
import lbry
from unittest.mock import Mock
from hub.herald import HUB_PROTOCOL_VERSION
from hub.herald.udp import StatusServer
from hub.herald.session import LBRYElectrumX
from hub.scribe.network import LBCRegTest
from lbry.wallet.network import Network
from lbry.wallet.orchstr8 import Conductor
from lbry.wallet.orchstr8.node import SPVNode
from lbry.wallet.rpc import RPCSession
from lbry.wallet.server.udp import StatusServer
from lbry.testcase import IntegrationTestCase, AsyncioTestCase
from lbry.conf import Config
@@ -26,7 +22,7 @@ class NetworkTests(IntegrationTestCase):
async def test_server_features(self):
self.assertDictEqual({
'genesis_hash': LBCRegTest.GENESIS_HASH,
'genesis_hash': self.conductor.spv_node.coin_class.GENESIS_HASH,
'hash_function': 'sha256',
'hosts': {},
'protocol_max': '0.199.0',
@@ -36,27 +32,22 @@ class NetworkTests(IntegrationTestCase):
'payment_address': '',
'donation_address': '',
'daily_fee': '0',
'server_version': HUB_PROTOCOL_VERSION,
'server_version': lbry.__version__,
'trending_algorithm': 'fast_ar',
}, await self.ledger.network.get_server_features())
# await self.conductor.spv_node.stop()
payment_address, donation_address = await self.account.get_addresses(limit=2)
original_address = self.conductor.spv_node.server.env.payment_address
original_donation_address = self.conductor.spv_node.server.env.donation_address
original_description = self.conductor.spv_node.server.env.description
original_daily_fee = self.conductor.spv_node.server.env.daily_fee
self.conductor.spv_node.server.env.payment_address = payment_address
self.conductor.spv_node.server.env.donation_address = donation_address
self.conductor.spv_node.server.env.description = 'Fastest server in the west.'
self.conductor.spv_node.server.env.daily_fee = '42'
from lbry.wallet.server.session import LBRYElectrumX
LBRYElectrumX.set_server_features(self.conductor.spv_node.server.env)
# await self.ledger.network.on_connected.first
self.assertDictEqual({
'genesis_hash': LBCRegTest.GENESIS_HASH,
'genesis_hash': self.conductor.spv_node.coin_class.GENESIS_HASH,
'hash_function': 'sha256',
'hosts': {},
'protocol_max': '0.199.0',
@@ -66,23 +57,16 @@ class NetworkTests(IntegrationTestCase):
'payment_address': payment_address,
'donation_address': donation_address,
'daily_fee': '42',
'server_version': HUB_PROTOCOL_VERSION,
'server_version': lbry.__version__,
'trending_algorithm': 'fast_ar',
}, await self.ledger.network.get_server_features())
# cleanup the changes since the attributes are set on the class
self.conductor.spv_node.server.env.payment_address = original_address
self.conductor.spv_node.server.env.donation_address = original_donation_address
self.conductor.spv_node.server.env.description = original_description
self.conductor.spv_node.server.env.daily_fee = original_daily_fee
LBRYElectrumX.set_server_features(self.conductor.spv_node.server.env)
class ReconnectTests(IntegrationTestCase):
async def test_multiple_servers(self):
# we have a secondary node that connects later, so
node2 = SPVNode(node_number=2)
node2 = SPVNode(self.conductor.spv_module, node_number=2)
await node2.start(self.blockchain)
self.ledger.network.config['explicit_servers'].append((node2.hostname, node2.port))
@@ -102,7 +86,7 @@ class ReconnectTests(IntegrationTestCase):
await self.ledger.stop()
initial_height = self.ledger.local_height_including_downloaded_height
await self.blockchain.generate(100)
while self.conductor.spv_node.server.session_manager.notified_height < initial_height + 99: # off by 1
while self.conductor.spv_node.server.session_mgr.notified_height < initial_height + 99: # off by 1
await asyncio.sleep(0.1)
self.assertEqual(initial_height, self.ledger.local_height_including_downloaded_height)
await self.ledger.headers.open()
@@ -116,8 +100,13 @@ class ReconnectTests(IntegrationTestCase):
# disconnect and send a new tx, should reconnect and get it
self.ledger.network.client.transport.close()
self.assertFalse(self.ledger.network.is_connected)
await self.ledger.resolve([], ['derp'])
sendtxid = await self.send_to_address_and_wait(address1, 1.1337, 1)
await self.ledger.resolve([], 'derp')
sendtxid = await self.blockchain.send_to_address(address1, 1.1337)
# await self.ledger.resolve([], 'derp')
# self.assertTrue(self.ledger.network.is_connected)
await asyncio.wait_for(self.on_transaction_id(sendtxid), 10.0) # mempool
await self.blockchain.generate(1)
await self.on_transaction_id(sendtxid) # confirmed
self.assertLess(self.ledger.network.client.response_time, 1) # response time properly set lower, we are fine
await self.assertBalance(self.account, '1.1337')
@@ -139,7 +128,6 @@ class ReconnectTests(IntegrationTestCase):
if not self.ledger.network.is_connected:
await asyncio.wait_for(self.ledger.network.on_connected.first, timeout=10.0)
# omg, the burned cable still works! torba is fire proof!
await self.ledger.on_header.where(self.blockchain.is_expected_block)
await self.ledger.network.get_transaction(sendtxid)
async def test_timeout_then_reconnect(self):
@@ -147,7 +135,7 @@ class ReconnectTests(IntegrationTestCase):
await self.conductor.spv_node.stop()
self.assertFalse(self.ledger.network.is_connected)
await asyncio.sleep(0.2) # let it retry and fail once
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.conductor.spv_node.start(self.conductor.blockchain_node)
await self.ledger.network.on_connected.first
self.assertTrue(self.ledger.network.is_connected)
@@ -173,16 +161,15 @@ class ReconnectTests(IntegrationTestCase):
class UDPServerFailDiscoveryTest(AsyncioTestCase):
async def test_wallet_connects_despite_lack_of_udp(self):
conductor = Conductor()
conductor.spv_node.udp_port = '0'
await conductor.start_lbcd()
self.addCleanup(conductor.stop_lbcd)
await conductor.start_lbcwallet()
self.addCleanup(conductor.stop_lbcwallet)
await conductor.start_blockchain()
self.addCleanup(conductor.stop_blockchain)
await conductor.start_spv()
self.addCleanup(conductor.stop_spv)
self.assertFalse(conductor.spv_node.server.status_server.is_running)
self.assertFalse(conductor.spv_node.server.bp.status_server.is_running)
await asyncio.wait_for(conductor.start_wallet(), timeout=5)
self.addCleanup(conductor.stop_wallet)
self.assertTrue(conductor.wallet_node.ledger.network.is_connected)
@@ -103,7 +103,7 @@ class PurchaseCommandTests(CommandTestCase):
# purchase non-existent claim fails
with self.assertRaisesRegex(Exception, "Could not find claim with claim_id"):
await self.daemon.jsonrpc_purchase_create('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
await self.daemon.jsonrpc_purchase_create('abc123')
# purchase stream with no price fails
no_price_stream = await self.priced_stream('no_price_stream', price=None)
@@ -174,7 +174,8 @@ class PurchaseCommandTests(CommandTestCase):
self.merchant_address = await self.account.receiving.get_or_create_usable_address()
daemon2 = await self.add_daemon()
address2 = await daemon2.wallet_manager.default_account.receiving.get_or_create_usable_address()
await self.send_to_address_and_wait(address2, 2, 1, ledger=daemon2.ledger)
sendtxid = await self.blockchain.send_to_address(address2, 2)
await self.confirm_tx(sendtxid, daemon2.ledger)
stream = await self.priced_stream('a', '1.0')
await self.assertBalance(self.account, '9.987893')
+3 -3
View File
@@ -63,7 +63,7 @@ class SyncTests(IntegrationTestCase):
await self.assertBalance(account1, '1.0')
await self.assertBalance(account2, '1.0')
await self.generate(1)
await self.blockchain.generate(1)
# pay 0.01 from main node to receiving node, would have increased change addresses
address0 = (await account0.receiving.get_addresses())[0]
@@ -79,7 +79,7 @@ class SyncTests(IntegrationTestCase):
account1.ledger.wait(tx),
account2.ledger.wait(tx),
])
await self.generate(1)
await self.blockchain.generate(1)
await asyncio.wait([
account0.ledger.wait(tx),
account1.ledger.wait(tx),
@@ -92,7 +92,7 @@ class SyncTests(IntegrationTestCase):
await self.assertBalance(account1, '0.989876')
await self.assertBalance(account2, '0.989876')
await self.generate(1)
await self.blockchain.generate(1)
# create a new mirror node and see if it syncs to same balance from scratch
node3 = await self.make_wallet_node(account1.seed)
@@ -1,6 +1,5 @@
import asyncio
import json
from binascii import unhexlify
from lbry.wallet import ENCRYPT_ON_DISK
from lbry.error import InvalidPasswordError
@@ -11,7 +10,7 @@ from lbry.wallet.dewies import dict_values_to_lbc
class WalletCommands(CommandTestCase):
async def test_wallet_create_and_add_subscribe(self):
session = next(iter(self.conductor.spv_node.server.session_manager.sessions.values()))
session = next(iter(self.conductor.spv_node.server.session_mgr.sessions.values()))
self.assertEqual(len(session.hashX_subs), 27)
wallet = await self.daemon.jsonrpc_wallet_create('foo', create_account=True, single_key=True)
self.assertEqual(len(session.hashX_subs), 28)
@@ -23,7 +22,7 @@ class WalletCommands(CommandTestCase):
async def test_wallet_syncing_status(self):
address = await self.daemon.jsonrpc_address_unused()
self.assertFalse(self.daemon.jsonrpc_wallet_status()['is_syncing'])
await self.send_to_address_and_wait(address, 1)
await self.blockchain.send_to_address(address, 1)
await self.ledger._update_tasks.started.wait()
self.assertTrue(self.daemon.jsonrpc_wallet_status()['is_syncing'])
await self.ledger._update_tasks.done.wait()
@@ -47,9 +46,9 @@ class WalletCommands(CommandTestCase):
status = await self.daemon.jsonrpc_status()
self.assertEqual(len(status['wallet']['servers']), 1)
self.assertEqual(status['wallet']['servers'][0]['port'], 50002)
await self.conductor.spv_node.stop()
await self.conductor.spv_node.stop(True)
self.conductor.spv_node.port = 54320
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.conductor.spv_node.start(self.conductor.blockchain_node)
status = await self.daemon.jsonrpc_status()
self.assertEqual(len(status['wallet']['servers']), 0)
self.daemon.jsonrpc_settings_set('lbryum_servers', ['localhost:54320'])
@@ -59,22 +58,23 @@ class WalletCommands(CommandTestCase):
self.assertEqual(status['wallet']['servers'][0]['port'], 54320)
async def test_sending_to_scripthash_address(self):
bal = await self.blockchain.get_balance()
self.assertEqual(await self.blockchain.get_balance(), '95.99973580')
await self.assertBalance(self.account, '10.0')
p2sh_address1 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
tx = await self.account_send('2.0', p2sh_address1)
self.assertEqual(tx['outputs'][0]['address'], p2sh_address1)
self.assertEqual(await self.blockchain.get_balance(), str(float(bal)+3)) # +1 lbc for confirm block
self.assertEqual(await self.blockchain.get_balance(), '98.99973580') # +1 lbc for confirm block
await self.assertBalance(self.account, '7.999877')
await self.wallet_send('3.0', p2sh_address1)
self.assertEqual(await self.blockchain.get_balance(), str(float(bal)+7)) # +1 lbc for confirm block
self.assertEqual(await self.blockchain.get_balance(), '102.99973580') # +1 lbc for confirm block
await self.assertBalance(self.account, '4.999754')
async def test_balance_caching(self):
account2 = await self.daemon.jsonrpc_account_create("Tip-er")
address2 = await self.daemon.jsonrpc_address_unused(account2.id)
await self.send_to_address_and_wait(address2, 10, 2)
await self.ledger.tasks_are_done() # don't mess with the query count while we need it
sendtxid = await self.blockchain.send_to_address(address2, 10)
await self.confirm_tx(sendtxid)
await self.generate(1)
wallet_balance = self.daemon.jsonrpc_wallet_balance
ledger = self.ledger
@@ -89,16 +89,14 @@ class WalletCommands(CommandTestCase):
self.assertIsNone(ledger._balance_cache.get(self.account.id))
query_count += 2
balance = await wallet_balance()
self.assertEqual(await wallet_balance(), expected)
self.assertEqual(self.ledger.db.db.query_count, query_count)
self.assertEqual(balance, expected)
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(self.account.id))['total'], '10.0')
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(account2.id))['total'], '10.0')
# calling again uses cache
balance = await wallet_balance()
self.assertEqual(await wallet_balance(), expected)
self.assertEqual(self.ledger.db.db.query_count, query_count)
self.assertEqual(balance, expected)
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(self.account.id))['total'], '10.0')
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(account2.id))['total'], '10.0')
@@ -124,7 +122,8 @@ class WalletCommands(CommandTestCase):
wallet2 = await self.daemon.jsonrpc_wallet_create('foo', create_account=True)
account3 = wallet2.default_account
address3 = await self.daemon.jsonrpc_address_unused(account3.id, wallet2.id)
await self.send_to_address_and_wait(address3, 1, 1)
await self.confirm_tx(await self.blockchain.send_to_address(address3, 1))
await self.generate(1)
account_balance = self.daemon.jsonrpc_account_balance
wallet_balance = self.daemon.jsonrpc_wallet_balance
@@ -154,7 +153,7 @@ class WalletCommands(CommandTestCase):
address2 = await self.daemon.jsonrpc_address_unused(account2.id)
# send lbc to someone else
tx = await self.daemon.jsonrpc_account_send('1.0', address2, blocking=True)
tx = await self.daemon.jsonrpc_account_send('1.0', address2)
await self.confirm_tx(tx.id)
self.assertEqual(await account_balance(), {
'total': '8.97741',
@@ -187,7 +186,7 @@ class WalletCommands(CommandTestCase):
})
# tip claimed
tx = await self.daemon.jsonrpc_support_abandon(txid=support1['txid'], nout=0, blocking=True)
tx = await self.daemon.jsonrpc_support_abandon(txid=support1['txid'], nout=0)
await self.confirm_tx(tx.id)
self.assertEqual(await account_balance(), {
'total': '9.277303',
@@ -238,7 +237,8 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
"carbon smart garage balance margin twelve"
)
address = (await self.daemon2.wallet_manager.default_account.receiving.get_addresses(limit=1, only_usable=True))[0]
await self.send_to_address_and_wait(address, 1, 1, ledger=self.daemon2.ledger)
sendtxid = await self.blockchain.send_to_address(address, 1)
await self.confirm_tx(sendtxid, self.daemon2.ledger)
def assertWalletEncrypted(self, wallet_path, encrypted):
with open(wallet_path) as opened:
@@ -282,19 +282,8 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
)
# Channel Certificate
# non-deterministic channel
self.daemon2.wallet_manager.default_account.channel_keys['mqs77XbdnuxWN4cXrjKbSoGLkvAHa4f4B8'] = (
'-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIBZRTZ7tHnYCH3IE9mCo95'
'466L/ShYFhXGrjmSMFJw8eoAcGBSuBBAAK\noUQDQgAEmucoPz9nI+ChZrfhnh'
'0RZ/bcX0r2G0pYBmoNKovtKzXGa8y07D66MWsW\nqXptakqO/9KddIkBu5eJNS'
'UZzQCxPQ==\n-----END EC PRIVATE KEY-----\n'
)
channel = await self.create_nondeterministic_channel('@foo', '0.1', unhexlify(
'3056301006072a8648ce3d020106052b8104000a034200049ae7283f3f6723e0a1'
'66b7e19e1d1167f6dc5f4af61b4a58066a0d2a8bed2b35c66bccb4ec3eba316b16'
'a97a6d6a4a8effd29d748901bb9789352519cd00b13d'
), self.daemon2, blocking=True)
await self.confirm_tx(channel['txid'], self.daemon2.ledger)
channel = await daemon2.jsonrpc_channel_create('@foo', '0.1')
await self.confirm_tx(channel.id, self.daemon2.ledger)
# both daemons will have the channel but only one has the cert so far
self.assertItemCount(await daemon.jsonrpc_channel_list(), 1)
@@ -326,7 +315,7 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
with self.assertRaisesRegex(AssertionError, "Cannot lock an unencrypted wallet, encrypt first."):
daemon.jsonrpc_wallet_lock()
# safe to call unlock and decrypt, they are no-ops at this point
await daemon.jsonrpc_wallet_unlock('password') # already unlocked
daemon.jsonrpc_wallet_unlock('password') # already unlocked
daemon.jsonrpc_wallet_decrypt() # already not encrypted
daemon.jsonrpc_wallet_encrypt('password')
@@ -342,7 +331,7 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
# can't sign transactions with locked wallet
with self.assertRaises(AssertionError):
await daemon.jsonrpc_channel_create('@foo', '1.0')
await daemon.jsonrpc_wallet_unlock('password')
daemon.jsonrpc_wallet_unlock('password')
self.assertEqual(daemon.jsonrpc_wallet_status(), {'is_locked': False, 'is_encrypted': True,
'is_syncing': False})
await daemon.jsonrpc_channel_create('@foo', '1.0')
@@ -360,17 +349,10 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
await daemon2.jsonrpc_channel_import(exported)
self.assertTrue(daemon2.jsonrpc_wallet_encrypt('password'))
self.assertTrue(daemon2.jsonrpc_wallet_lock())
self.assertTrue(await daemon2.jsonrpc_wallet_unlock("password"))
self.assertTrue(daemon2.jsonrpc_wallet_unlock("password"))
self.assertEqual(daemon2.jsonrpc_wallet_status(),
{'is_locked': False, 'is_encrypted': True, 'is_syncing': False})
async def test_locking_unlocking_does_not_break_deterministic_channels(self):
self.assertTrue(self.daemon.jsonrpc_wallet_encrypt("password"))
self.assertTrue(self.daemon.jsonrpc_wallet_lock())
self.account.deterministic_channel_keys._private_key = None
self.assertTrue(await self.daemon.jsonrpc_wallet_unlock("password"))
await self.channel_create()
async def test_sync_with_encryption_and_password_change(self):
daemon, daemon2 = self.daemon, self.daemon2
wallet, wallet2 = daemon.wallet_manager.default_wallet, daemon2.wallet_manager.default_wallet
@@ -397,8 +379,8 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
# check new password is active
daemon.jsonrpc_wallet_lock()
self.assertFalse(await daemon.jsonrpc_wallet_unlock('password'))
self.assertTrue(await daemon.jsonrpc_wallet_unlock('password2'))
self.assertFalse(daemon.jsonrpc_wallet_unlock('password'))
self.assertTrue(daemon.jsonrpc_wallet_unlock('password2'))
# propagate disk encryption to daemon2
data = await daemon.jsonrpc_sync_apply('password3')
@@ -414,4 +396,4 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
self.assertWalletEncrypted(wallet2.storage.path, True)
daemon2.jsonrpc_wallet_lock()
self.assertTrue(await daemon2.jsonrpc_wallet_unlock('password3'))
self.assertTrue(daemon2.jsonrpc_wallet_unlock('password3'))
@@ -1,11 +1,12 @@
import asyncio
from hub.herald import HUB_PROTOCOL_VERSION
from hub.herald.session import LBRYElectrumX
import lbry
import lbry.wallet
from lbry.error import ServerPaymentFeeAboveMaxAllowedError
from lbry.wallet.network import ClientSession
from lbry.wallet.rpc import RPCError
from lbry.wallet.server.db.elasticsearch.sync import make_es_index_and_run_sync
from lbry.wallet.server.session import LBRYElectrumX
from lbry.testcase import IntegrationTestCase, CommandTestCase
from lbry.wallet.orchstr8.node import SPVNode
@@ -24,17 +25,17 @@ class TestSessions(IntegrationTestCase):
)
await session.create_connection()
await session.send_request('server.banner', ())
self.assertEqual(len(self.conductor.spv_node.server.session_manager.sessions), 1)
self.assertEqual(len(self.conductor.spv_node.server.session_mgr.sessions), 1)
self.assertFalse(session.is_closing())
await asyncio.sleep(1.1)
with self.assertRaises(asyncio.TimeoutError):
await session.send_request('server.banner', ())
self.assertTrue(session.is_closing())
self.assertEqual(len(self.conductor.spv_node.server.session_manager.sessions), 0)
self.assertEqual(len(self.conductor.spv_node.server.session_mgr.sessions), 0)
async def test_proper_version(self):
info = await self.ledger.network.get_server_features()
self.assertEqual(HUB_PROTOCOL_VERSION, info['server_version'])
self.assertEqual(lbry.__version__, info['server_version'])
async def test_client_errors(self):
# Goal is ensuring thsoe are raised and not trapped accidentally
@@ -45,7 +46,7 @@ class TestSessions(IntegrationTestCase):
class TestUsagePayment(CommandTestCase):
async def test_single_server_payment(self):
async def _test_single_server_payment(self):
wallet_pay_service = self.daemon.component_manager.get_component('wallet_server_payments')
wallet_pay_service.payment_period = 1
# only starts with a positive max key fee
@@ -62,8 +63,8 @@ class TestUsagePayment(CommandTestCase):
_, history = await self.ledger.get_local_status_and_history(address)
self.assertEqual(history, [])
node = SPVNode(node_number=2)
await node.start(self.blockchain, extraconf={"payment_address": address, "daily_fee": "1.1"})
node = SPVNode(self.conductor.spv_module, node_number=2)
await node.start(self.blockchain, extraconf={"PAYMENT_ADDRESS": address, "DAILY_FEE": "1.1"})
self.addCleanup(node.stop)
self.daemon.jsonrpc_settings_set('lbryum_servers', [f"{node.hostname}:{node.port}"])
await self.daemon.jsonrpc_wallet_reconnect()
@@ -89,78 +90,56 @@ class TestUsagePayment(CommandTestCase):
class TestESSync(CommandTestCase):
async def test_es_sync_utility(self):
es_writer = self.conductor.spv_node.es_writer
server_search_client = self.conductor.spv_node.server.session_manager.search_index
for i in range(10):
await self.stream_create(f"stream{i}", bid='0.001')
await self.generate(1)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
db = self.conductor.spv_node.server.db
env = self.conductor.spv_node.server.env
await db.search_index.delete_index()
db.search_index.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
await db.search_index.stop()
async def resync():
await db.search_index.start()
db.search_index.clear_caches()
await make_es_index_and_run_sync(env, db=db, index_name=db.search_index.index, force=True)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# delete the index and verify nothing is returned by claim search
await es_writer.delete_index()
server_search_client.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
# reindex, 10 claims should be returned
await es_writer.reindex(force=True)
await resync()
# this time we will test a migration from unversioned to v1
await db.search_index.sync_client.indices.delete_template(db.search_index.index)
await db.search_index.stop()
await make_es_index_and_run_sync(env, db=db, index_name=db.search_index.index, force=True)
await db.search_index.start()
await resync()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
server_search_client.clear_caches()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# reindex again, this should not appear to do anything but will delete and reinsert the same 10 claims
await es_writer.reindex(force=True)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
server_search_client.clear_caches()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# delete the index again and stop the writer, upon starting it the writer should reindex automatically
await es_writer.delete_index()
await es_writer.stop()
server_search_client.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
await es_writer.start(reindex=True)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# stop the es writer and advance the chain by 1, adding a new claim. upon resuming the es writer, it should
# add the new claim
await es_writer.stop()
await self.stream_create(f"stream11", bid='0.001', confirm=False)
generate_block_task = asyncio.create_task(self.generate(1))
await es_writer.start()
await generate_block_task
self.assertEqual(11, len(await self.claim_search(order_by=['height'])))
# # this time we will test a migration from unversioned to v1
# await db.search_index.sync_client.indices.delete_template(db.search_index.index)
# await db.search_index.stop()
#
# await make_es_index_and_run_sync(env, db=db, index_name=db.search_index.index, force=True)
# await db.search_index.start()
#
# await es_writer.reindex()
# self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
class TestHubDiscovery(CommandTestCase):
async def test_hub_discovery(self):
us_final_node = SPVNode(node_number=2)
await us_final_node.start(self.blockchain, extraconf={"country": "US"})
us_final_node = SPVNode(self.conductor.spv_module, node_number=2)
await us_final_node.start(self.blockchain, extraconf={"COUNTRY": "US"})
self.addCleanup(us_final_node.stop)
final_node_host = f"{us_final_node.hostname}:{us_final_node.port}"
kp_final_node = SPVNode(node_number=3)
await kp_final_node.start(self.blockchain, extraconf={"country": "KP"})
kp_final_node = SPVNode(self.conductor.spv_module, node_number=3)
await kp_final_node.start(self.blockchain, extraconf={"COUNTRY": "KP"})
self.addCleanup(kp_final_node.stop)
kp_final_node_host = f"{kp_final_node.hostname}:{kp_final_node.port}"
relay_node = SPVNode(node_number=4)
relay_node = SPVNode(self.conductor.spv_module, node_number=4)
await relay_node.start(self.blockchain, extraconf={
"country": "FR",
"peer_hubs": ",".join([kp_final_node_host, final_node_host])
"COUNTRY": "FR",
"PEER_HUBS": ",".join([kp_final_node_host, final_node_host])
})
relay_node_host = f"{relay_node.hostname}:{relay_node.port}"
self.addCleanup(relay_node.stop)
@@ -207,7 +186,7 @@ class TestHubDiscovery(CommandTestCase):
self.daemon.ledger.network.client.server_address_and_port, ('127.0.0.1', kp_final_node.port)
)
kp_final_node.server.session_manager._notify_peer('127.0.0.1:9988')
kp_final_node.server.session_mgr._notify_peer('127.0.0.1:9988')
await self.daemon.ledger.network.on_hub.first
await asyncio.sleep(0.5) # wait for above event to be processed by other listeners
self.assertEqual(
+50 -89
View File
@@ -12,21 +12,18 @@ from lbry.error import InsufficientFundsError
from lbry.extras.daemon.daemon import DEFAULT_PAGE_SIZE
from lbry.testcase import CommandTestCase
from lbry.wallet.orchstr8.node import SPVNode
from lbry.wallet.server.db.common import STREAM_TYPES
from lbry.wallet.transaction import Transaction, Output
from lbry.wallet.util import satoshis_to_coins as lbc
from lbry.crypto.hash import sha256
log = logging.getLogger(__name__)
STREAM_TYPES = {
'video': 1,
'audio': 2,
'image': 3,
'document': 4,
'binary': 5,
'model': 6,
}
def get_encoded_signature(signature):
signature = signature.encode() if isinstance(signature, str) else signature
r = int(signature[:int(len(signature) / 2)], 16)
s = int(signature[int(len(signature) / 2):], 16)
return ecdsa.util.sigencode_der(r, s, len(signature) * 4)
def verify(channel, data, signature, channel_hash=None):
@@ -36,7 +33,7 @@ def verify(channel, data, signature, channel_hash=None):
data
]
return Output.is_signature_valid(
unhexlify(signature['signature']),
get_encoded_signature(signature['signature']),
sha256(b''.join(pieces)),
channel.claim.channel.public_key_bytes
)
@@ -134,6 +131,18 @@ class ClaimSearchCommand(ClaimTestCase):
with self.assertRaises(ConnectionResetError):
await self.claim_search(claim_ids=claim_ids)
async def test_claim_search_as_reader_server(self):
node2 = SPVNode(self.conductor.spv_module, node_number=2)
current_prefix = self.conductor.spv_node.server.bp.env.es_index_prefix
await node2.start(self.blockchain, extraconf={'ES_MODE': 'reader', 'ES_INDEX_PREFIX': current_prefix})
self.addCleanup(node2.stop)
self.ledger.network.config['default_servers'] = [(node2.hostname, node2.port)]
await self.ledger.stop()
await self.ledger.start()
channel2 = await self.channel_create('@abc', '0.1', allow_duplicate_name=True)
await asyncio.sleep(1) # fixme: find a way to block on the writer
await self.assertFindsClaims([channel2], name='@abc')
async def test_basic_claim_search(self):
await self.create_channel()
channel_txo = self.channel['outputs'][0]
@@ -402,18 +411,6 @@ class ClaimSearchCommand(ClaimTestCase):
not_channel_ids=[chan2_id], has_channel_signature=True, valid_channel_signature=True)
await match([], not_channel_ids=[chan1_id, chan2_id], has_channel_signature=True, valid_channel_signature=True)
@skip
async def test_no_source_and_valid_channel_signature_and_media_type(self):
await self.channel_create('@spam2', '1.0')
await self.stream_create('barrrrrr', '1.0', channel_name='@spam2', file_path=self.video_file_name)
paradox_no_source_claims = await self.claim_search(has_no_source=True, valid_channel_signature=True,
media_type="video/mp4")
mp4_claims = await self.claim_search(media_type="video/mp4")
no_source_claims = await self.claim_search(has_no_source=True, valid_channel_signature=True)
self.assertEqual(0, len(paradox_no_source_claims))
self.assertEqual(1, len(no_source_claims))
self.assertEqual(1, len(mp4_claims))
async def test_limit_claims_per_channel(self):
match = self.assertFindsClaims
chan1_id = self.get_claim_id(await self.channel_create('@chan1'))
@@ -503,7 +500,8 @@ class ClaimSearchCommand(ClaimTestCase):
tx = await Transaction.claim_create(
'unknown', b'{"sources":{"lbry_sd_hash":""}}', 1, address, [self.account], self.account)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
octet = await self.stream_create()
video = await self.stream_create('chrome', file_path=self.video_file_name)
@@ -1129,17 +1127,17 @@ class ChannelCommands(CommandTestCase):
tx = await self.channel_update(claim_id, bid='4.0')
self.assertEqual(tx['outputs'][0]['amount'], '4.0')
await self.assertBalance(self.account, '5.991503')
await self.assertBalance(self.account, '5.991447')
# not enough funds
with self.assertRaisesRegex(
InsufficientFundsError, "Not enough funds to cover this transaction."):
await self.channel_create('@foo2', '9.0')
self.assertItemCount(await self.daemon.jsonrpc_channel_list(), 1)
await self.assertBalance(self.account, '5.991503')
await self.assertBalance(self.account, '5.991447')
# spend exactly amount available, no change
tx = await self.channel_create('@foo3', '5.981322')
tx = await self.channel_create('@foo3', '5.981266')
await self.assertBalance(self.account, '0.0')
self.assertEqual(len(tx['outputs']), 1) # no change
self.assertItemCount(await self.daemon.jsonrpc_channel_list(), 2)
@@ -1234,7 +1232,7 @@ class ChannelCommands(CommandTestCase):
data_to_sign = "CAFEBABE"
# claim new name
await self.channel_create('@someotherchan')
channel_tx = await self.daemon.jsonrpc_channel_create('@signer', '0.1', blocking=True)
channel_tx = await self.daemon.jsonrpc_channel_create('@signer', '0.1')
await self.confirm_tx(channel_tx.id)
channel = channel_tx.outputs[0]
signature1 = await self.out(self.daemon.jsonrpc_channel_sign(channel_name='@signer', hexdata=data_to_sign))
@@ -1257,7 +1255,7 @@ class ChannelCommands(CommandTestCase):
await daemon2.jsonrpc_channel_import(exported_data)
channels = (await daemon2.jsonrpc_channel_list())['items']
self.assertEqual(1, len(channels))
self.assertEqual(channel_private_key.private_key_bytes, channels[0].private_key.private_key_bytes)
self.assertEqual(channel_private_key.to_string(), channels[0].private_key.to_string())
# second wallet can't update until channel is sent to it
with self.assertRaisesRegex(AssertionError, 'Cannot find private key for signing output.'):
@@ -1381,7 +1379,7 @@ class StreamCommands(ClaimTestCase):
self.assertEqual('8.989893', (await self.daemon.jsonrpc_account_balance())['available'])
result = await self.out(self.daemon.jsonrpc_account_send(
'5.0', await self.daemon.jsonrpc_address_unused(account2_id), blocking=True
'5.0', await self.daemon.jsonrpc_address_unused(account2_id)
))
await self.confirm_tx(result['txid'])
@@ -1455,31 +1453,17 @@ class StreamCommands(ClaimTestCase):
self.assertItemCount(await self.daemon.jsonrpc_txo_list(reposted_claim_id=claim_id), 0)
self.assertItemCount(await self.daemon.jsonrpc_txo_list(type='repost'), 0)
tx = await self.stream_repost(
claim_id, 'newstuff-again', '1.1', channel_name='@spam',
title="repost title", description="repost desc", tags=["tag1", "tag2"]
)
tx = await self.stream_repost(claim_id, 'newstuff-again', '1.1', channel_name='@spam')
repost_id = self.get_claim_id(tx)
# test inflating reposted channels works
repost_url = f'newstuff-again:{repost_id}'
self.ledger._tx_cache.clear()
repost_resolve = await self.out(self.daemon.jsonrpc_resolve(repost_url))
repost = repost_resolve[repost_url]
self.assertEqual(goodies_claim_id, repost['reposted_claim']['signing_channel']['claim_id'])
self.assertEqual("repost title", repost["value"]["title"])
self.assertEqual("repost desc", repost["value"]["description"])
self.assertEqual(["tag1", "tag2"], repost["value"]["tags"])
await self.stream_update(
repost_id, title="title 2", description="desc 2", tags=["tag3"]
self.assertEqual(
goodies_claim_id,
(await self.out(self.daemon.jsonrpc_resolve(repost_url))
)[repost_url]['reposted_claim']['signing_channel']['claim_id']
)
repost_resolve = await self.out(self.daemon.jsonrpc_resolve(repost_url))
repost = repost_resolve[repost_url]
self.assertEqual(goodies_claim_id, repost['reposted_claim']['signing_channel']['claim_id'])
self.assertEqual("title 2", repost["value"]["title"])
self.assertEqual("desc 2", repost["value"]["description"])
self.assertEqual(["tag1", "tag2", "tag3"], repost["value"]["tags"])
self.assertItemCount(await self.daemon.jsonrpc_claim_list(claim_type='repost'), 1)
self.assertEqual((await self.claim_search(name='newstuff'))[0]['meta']['reposted'], 1)
@@ -1536,13 +1520,10 @@ class StreamCommands(ClaimTestCase):
await self.channel_create('@filtering', '0.1')
)
self.conductor.spv_node.server.db.filtering_channel_hashes.add(bytes.fromhex(filtering_channel_id))
self.conductor.spv_node.es_writer.db.filtering_channel_hashes.add(bytes.fromhex(filtering_channel_id))
self.assertEqual(0, len(self.conductor.spv_node.es_writer.db.filtered_streams))
self.assertEqual(0, len(self.conductor.spv_node.server.db.filtered_streams))
await self.stream_repost(bad_content_id, 'filter1', '0.1', channel_name='@filtering')
self.assertEqual(1, len(self.conductor.spv_node.es_writer.db.filtered_streams))
self.assertEqual(1, len(self.conductor.spv_node.server.db.filtered_streams))
self.assertEqual('0.1', (await self.out(self.daemon.jsonrpc_resolve('bad_content')))['bad_content']['amount'])
# search for filtered content directly
result = await self.out(self.daemon.jsonrpc_claim_search(name='bad_content'))
blocked = result['blocked']
@@ -1585,14 +1566,14 @@ class StreamCommands(ClaimTestCase):
)
# test setting from env vars and starting from scratch
await self.conductor.spv_node.stop(False)
await self.conductor.spv_node.start(self.conductor.lbcwallet_node,
extraconf={'blocking_channel_ids': [blocking_channel_id],
'filtering_channel_ids': [filtering_channel_id]})
await self.conductor.spv_node.start(self.conductor.blockchain_node,
extraconf={'BLOCKING_CHANNEL_IDS': blocking_channel_id,
'FILTERING_CHANNEL_IDS': filtering_channel_id})
await self.daemon.wallet_manager.reset()
self.assertEqual(0, len(self.conductor.spv_node.es_writer.db.blocked_streams))
self.assertEqual(0, len(self.conductor.spv_node.server.db.blocked_streams))
await self.stream_repost(bad_content_id, 'block1', '0.1', channel_name='@blocking')
self.assertEqual(1, len(self.conductor.spv_node.es_writer.db.blocked_streams))
self.assertEqual(1, len(self.conductor.spv_node.server.db.blocked_streams))
# blocked content is not resolveable
error = (await self.resolve('lbry://@some_channel/bad_content'))['error']
@@ -1651,11 +1632,6 @@ class StreamCommands(ClaimTestCase):
self.assertEqual((await self.resolve('lbry://worse_content'))['error']['name'], 'BLOCKED')
self.assertEqual((await self.resolve('lbry://@bad_channel/worse_content'))['error']['name'], 'BLOCKED')
await self.stream_update(worse_content_id, channel_name='@bad_channel', tags=['bad-stuff'])
self.assertEqual((await self.resolve('lbry://@bad_channel'))['error']['name'], 'BLOCKED')
self.assertEqual((await self.resolve('lbry://worse_content'))['error']['name'], 'BLOCKED')
self.assertEqual((await self.resolve('lbry://@bad_channel/worse_content'))['error']['name'], 'BLOCKED')
async def test_publish_updates_file_list(self):
tx = await self.stream_create(title='created')
txo = tx['outputs'][0]
@@ -1821,37 +1797,22 @@ class StreamCommands(ClaimTestCase):
self.assertItemCount(await self.daemon.jsonrpc_claim_list(account_id=self.account.id), 3)
self.assertItemCount(await self.daemon.jsonrpc_claim_list(account_id=account2_id), 1)
@skip
async def test_release_time(self):
self.assertEqual(4, len(await self.claim_search(release_time='>0', order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(release_time='>0', order_by=['release_time'], claim_type='stream')))
self.assertEqual(4, len(await self.claim_search(release_time='>=0', order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(release_time='>0', order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(release_time='>=0', order_by=['release_time'])))
self.assertEqual(4, len(await self.claim_search(order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(claim_type='stream', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(claim_type='channel', order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time='>=123456', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(release_time='>=123456', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(release_time='>123456', order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time='<123457', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(release_time='>=123456', order_by=['release_time'], claim_type='stream')))
self.assertEqual(1, len(await self.claim_search(release_time='>123456', order_by=['release_time'], claim_type='stream')))
self.assertEqual(2, len(await self.claim_search(release_time='>123456', order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(release_time='<123457', order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time='<123457', order_by=['release_time'], claim_type='stream')))
self.assertEqual(2, len(await self.claim_search(release_time=['<123457'], order_by=['release_time'], claim_type='stream')))
self.assertEqual(3, len(await self.claim_search(release_time=['<123457'], order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(release_time=['>0', '<123457'], order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time=['>0', '<123457'], order_by=['release_time'], claim_type='stream')))
self.assertEqual(3, len(await self.claim_search(release_time=['<123457'], order_by=['release_time'], height=['>0'])))
self.assertEqual(4, len(await self.claim_search(order_by=['release_time'], height=['>0'])))
self.assertEqual(4, len(await self.claim_search(order_by=['release_time'], height=['>0'], claim_type=['stream', 'channel'])))
self.assertEqual(2, len(await self.claim_search(release_time=['<123457'], order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time=['>0', '<123457'], order_by=['release_time'])))
self.assertEqual(
3, len(await self.claim_search(release_time=['>=123097', '<123457'], order_by=['release_time']))
2, len(await self.claim_search(release_time=['>=123097', '<123457'], order_by=['release_time']))
)
self.assertEqual(
3, len(await self.claim_search(release_time=['<123457', '>0'], order_by=['release_time']))
2, len(await self.claim_search(release_time=['<123457', '>0'], order_by=['release_time']))
)
async def test_setting_fee_fields(self):
@@ -2222,7 +2183,7 @@ class SupportCommands(CommandTestCase):
tip = await self.out(
self.daemon.jsonrpc_support_create(
claim_id, '1.0', True, account_id=account2.id, wallet_id='wallet2',
funding_account_ids=[account2.id], blocking=True)
funding_account_ids=[account2.id])
)
await self.confirm_tx(tip['txid'])
@@ -2254,7 +2215,7 @@ class SupportCommands(CommandTestCase):
support = await self.out(
self.daemon.jsonrpc_support_create(
claim_id, '2.0', False, account_id=account2.id, wallet_id='wallet2',
funding_account_ids=[account2.id], blocking=True)
funding_account_ids=[account2.id])
)
await self.confirm_tx(support['txid'])
@@ -1,4 +1,3 @@
import unittest
from unittest import skipIf
import asyncio
import os
@@ -10,7 +9,6 @@ from lbry.stream.descriptor import StreamDescriptor
from lbry.testcase import CommandTestCase
from lbry.extras.daemon.components import TorrentSession, BACKGROUND_DOWNLOADER_COMPONENT
from lbry.wallet import Transaction
from lbry.torrent.tracker import UDPTrackerServerProtocol
class FileCommands(CommandTestCase):
@@ -38,7 +36,8 @@ class FileCommands(CommandTestCase):
tx_to_update.outputs[0], claim, 1, address, [self.account], self.account
)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
self.client_session = self.daemon.file_manager.source_managers['torrent'].torrent_session
self.client_session._session.add_dht_node(('localhost', 4040))
self.client_session.wait_start = False # fixme: this is super slow on tests
@@ -103,32 +102,6 @@ class FileCommands(CommandTestCase):
await self.daemon.jsonrpc_get('lbry://foo')
self.assertItemCount(await self.daemon.jsonrpc_file_list(), 1)
async def test_tracker_discovery(self):
port = 50990
server = UDPTrackerServerProtocol()
transport, _ = await self.loop.create_datagram_endpoint(lambda: server, local_addr=("127.0.0.1", port))
self.addCleanup(transport.close)
self.daemon.conf.fixed_peers = []
self.daemon.conf.tracker_servers = [("127.0.0.1", port)]
tx = await self.stream_create('foo', '0.01')
sd_hash = tx['outputs'][0]['value']['source']['sd_hash']
self.assertNotIn(bytes.fromhex(sd_hash)[:20], server.peers)
server.add_peer(bytes.fromhex(sd_hash)[:20], "127.0.0.1", 5567)
self.assertEqual(1, len(server.peers[bytes.fromhex(sd_hash)[:20]]))
self.assertTrue(await self.daemon.jsonrpc_file_delete(delete_all=True))
stream = await self.daemon.jsonrpc_get('foo', save_file=True)
await self.wait_files_to_complete()
self.assertEqual(0, stream.blobs_remaining)
self.assertEqual(2, len(server.peers[bytes.fromhex(sd_hash)[:20]]))
self.assertEqual([{'address': '127.0.0.1',
'node_id': None,
'tcp_port': 5567,
'udp_port': None},
{'address': '127.0.0.1',
'node_id': None,
'tcp_port': 4444,
'udp_port': None}], (await self.daemon.jsonrpc_peer_list(sd_hash))['items'])
async def test_announces(self):
# announces on publish
self.assertEqual(await self.daemon.storage.get_blobs_to_announce(), [])
@@ -243,7 +216,6 @@ class FileCommands(CommandTestCase):
await self.wait_files_to_complete()
self.assertNotEqual(first_path, second_path)
@unittest.SkipTest # FIXME: claimname/updateclaim is gone. #3480 wip, unblock #3479"
async def test_file_list_updated_metadata_on_resolve(self):
await self.stream_create('foo', '0.01')
txo = (await self.daemon.resolve(self.wallet.accounts, ['lbry://foo']))['lbry://foo']
@@ -532,7 +504,8 @@ class FileCommands(CommandTestCase):
tx.outputs[0].claim.stream.fee.address_bytes = b''
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
async def __raw_value_update_no_fee_amount(self, tx, claim_address):
tx = await self.daemon.jsonrpc_stream_update(
@@ -542,7 +515,8 @@ class FileCommands(CommandTestCase):
tx.outputs[0].claim.stream.fee.message.ClearField('amount')
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
class DiskSpaceManagement(CommandTestCase):
@@ -599,18 +573,13 @@ class DiskSpaceManagement(CommandTestCase):
self.assertTrue(blobs2.issubset(blobs))
self.assertFalse(blobs3.issubset(blobs))
self.assertTrue(blobs4.issubset(blobs))
# check that pending blobs are not accounted (#3617)
await self.daemon.storage.db.execute_fetchall("update blob set status='pending'")
await self.blob_clean() # just to refresh caches, has no effect
self.assertEqual(0, (await self.status())['disk_space']['total_used_mb'])
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(0, (await self.status())['disk_space']['published_blobs_storage_used_mb'])
# check that added_on gets set on downloads (was a bug)
self.assertLess(0, await self.daemon.storage.run_and_return_one_or_none("select min(added_on) from blob"))
await self.daemon.jsonrpc_file_delete(delete_all=True)
await self.daemon.jsonrpc_get("foo4", save_file=False)
self.assertLess(0, await self.daemon.storage.run_and_return_one_or_none("select min(added_on) from blob"))
class TestBackgroundDownloaderComponent(CommandTestCase):
async def get_blobs_from_sd_blob(self, sd_blob):
descriptor = await StreamDescriptor.from_stream_descriptor_blob(
+6 -6
View File
@@ -80,7 +80,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# After some soul searching Chris decides that his story needs more
# heart and a better ending. He takes down the story and begins the rewrite.
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(claim_id, blocking=True))
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(claim_id, blocking=False))
self.assertEqual(abandon['inputs'][0]['claim_id'], claim_id)
await self.confirm_tx(abandon['txid'])
@@ -103,7 +103,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# 1 LBC to which Chris readily obliges
ramsey_account_id = (await self.out(self.daemon.jsonrpc_account_create("Ramsey")))['id']
ramsey_address = await self.daemon.jsonrpc_address_unused(ramsey_account_id)
result = await self.out(self.daemon.jsonrpc_account_send('1.0', ramsey_address, blocking=True))
result = await self.out(self.daemon.jsonrpc_account_send('1.0', ramsey_address))
self.assertIn("txid", result)
await self.confirm_tx(result['txid'])
@@ -133,7 +133,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# And voila, and bravo and encore! His Best Friend Ramsey read the story and immediately knew this was a hit
# Now to keep this claim winning on the lbry blockchain he immediately supports the claim
tx = await self.out(self.daemon.jsonrpc_support_create(
claim_id2, '0.2', account_id=ramsey_account_id, blocking=True
claim_id2, '0.2', account_id=ramsey_account_id
))
await self.confirm_tx(tx['txid'])
@@ -147,7 +147,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# Now he also wanted to support the original creator of the Award Winning Novel
# So he quickly decides to send a tip to him
tx = await self.out(
self.daemon.jsonrpc_support_create(claim_id2, '0.3', tip=True, account_id=ramsey_account_id, blocking=True)
self.daemon.jsonrpc_support_create(claim_id2, '0.3', tip=True, account_id=ramsey_account_id)
)
await self.confirm_tx(tx['txid'])
@@ -158,7 +158,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
await self.generate(5)
# Seeing the ravishing success of his novel Chris adds support to his claim too
tx = await self.out(self.daemon.jsonrpc_support_create(claim_id2, '0.4', blocking=True))
tx = await self.out(self.daemon.jsonrpc_support_create(claim_id2, '0.4'))
await self.confirm_tx(tx['txid'])
# And check if his support showed up
@@ -183,7 +183,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# But sadly Ramsey wasn't so pleased. It was hard for him to tell Chris...
# Chris, though a bit heartbroken, abandoned the claim for now, but instantly started working on new hit lyrics
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(txid=tx['txid'], nout=0, blocking=True))
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(txid=tx['txid'], nout=0, blocking=False))
self.assertTrue(abandon['inputs'][0]['txid'], tx['txid'])
await self.confirm_tx(abandon['txid'])
+2 -2
View File
@@ -10,7 +10,7 @@ from lbry.extras.daemon.components import (
DATABASE_COMPONENT, DISK_SPACE_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT,
HASH_ANNOUNCER_COMPONENT, FILE_MANAGER_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT,
UPNP_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, WALLET_SERVER_PAYMENTS_COMPONENT,
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT, TRACKER_ANNOUNCER_COMPONENT
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT
)
from lbry.extras.daemon.daemon import Daemon
@@ -26,7 +26,7 @@ class CLIIntegrationTest(AsyncioTestCase):
DATABASE_COMPONENT, DISK_SPACE_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT,
HASH_ANNOUNCER_COMPONENT, FILE_MANAGER_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT,
UPNP_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, WALLET_SERVER_PAYMENTS_COMPONENT,
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT, TRACKER_ANNOUNCER_COMPONENT
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT
)
Daemon.component_attributes = {}
self.daemon = Daemon(conf)
@@ -24,10 +24,10 @@ class TestExchangeRateManager(AsyncioTestCase):
self.assertLessEqual(len(failures), 1, f"feed failures: {failures}. Please check exchange rate feeds!")
lbc = manager.convert_currency('USD', 'LBC', Decimal('1.0'))
self.assertGreaterEqual(lbc, 2.0)
self.assertLessEqual(lbc, 80.0)
self.assertLessEqual(lbc, 60.0)
lbc = manager.convert_currency('BTC', 'LBC', Decimal('0.01'))
self.assertGreaterEqual(lbc, 1_000)
self.assertLessEqual(lbc, 30_000)
self.assertLessEqual(lbc, 20_000)
async def test_it_handles_feed_being_offline(self):
class FakeFeed(MarketFeed):
@@ -1,7 +1,6 @@
import asyncio
import json
import hashlib
import sys
from bisect import bisect_right
from binascii import hexlify, unhexlify
from collections import defaultdict
@@ -24,7 +23,7 @@ class BaseResolveTestCase(CommandTestCase):
def assertMatchESClaim(self, claim_from_es, claim_from_db):
self.assertEqual(claim_from_es['claim_hash'][::-1].hex(), claim_from_db.claim_hash.hex())
self.assertEqual(claim_from_es['claim_id'], claim_from_db.claim_hash.hex())
self.assertEqual(claim_from_es['activation_height'], claim_from_db.activation_height, f"es height: {claim_from_es['activation_height']}, rocksdb height: {claim_from_db.activation_height}")
self.assertEqual(claim_from_es['activation_height'], claim_from_db.activation_height)
self.assertEqual(claim_from_es['last_take_over_height'], claim_from_db.last_takeover_height)
self.assertEqual(claim_from_es['tx_id'], claim_from_db.tx_hash[::-1].hex())
self.assertEqual(claim_from_es['tx_nout'], claim_from_db.position)
@@ -32,151 +31,125 @@ class BaseResolveTestCase(CommandTestCase):
self.assertEqual(claim_from_es['effective_amount'], claim_from_db.effective_amount)
def assertMatchDBClaim(self, expected, claim):
self.assertEqual(expected['claimid'], claim.claim_hash.hex())
self.assertEqual(expected['validatheight'], claim.activation_height)
self.assertEqual(expected['lasttakeoverheight'], claim.last_takeover_height)
self.assertEqual(expected['txid'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['claimId'], claim.claim_hash.hex())
self.assertEqual(expected['validAtHeight'], claim.activation_height)
self.assertEqual(expected['lastTakeoverHeight'], claim.last_takeover_height)
self.assertEqual(expected['txId'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['n'], claim.position)
self.assertEqual(expected['amount'], claim.amount)
self.assertEqual(expected['effectiveamount'], claim.effective_amount)
self.assertEqual(expected['effectiveAmount'], claim.effective_amount)
async def assertResolvesToClaimId(self, name, claim_id):
other = await self.resolve(name)
if claim_id is None:
self.assertIn('error', other)
self.assertEqual(other['error']['name'], 'NOT_FOUND')
claims_from_es = (await self.conductor.spv_node.server.session_manager.search_index.search(name=name))[0]
claims_from_es = (await self.conductor.spv_node.server.bp.db.search_index.search(name=name))[0]
claims_from_es = [c['claim_hash'][::-1].hex() for c in claims_from_es]
self.assertNotIn(claim_id, claims_from_es)
else:
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(claim_id=claim_id)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(claim_id=claim_id)
self.assertEqual(claim_id, other['claim_id'])
self.assertEqual(claim_id, claim_from_es[0][0]['claim_hash'][::-1].hex())
async def assertNoClaimForName(self, name: str):
lbrycrd_winning = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.db.resolve(name)
if 'claims' in lbrycrd_winning and lbrycrd_winning['claims'] is not None:
self.assertEqual(len(lbrycrd_winning['claims']), 0)
lbrycrd_winning = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.bp.db.resolve(name)
self.assertNotIn('claimId', lbrycrd_winning)
if stream is not None:
self.assertIsInstance(stream, LookupError)
else:
self.assertIsInstance(channel, LookupError)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(name=name)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(name=name)
self.assertListEqual([], claim_from_es[0])
async def assertNoClaim(self, name: str, claim_id: str):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebyid', name, '["' + claim_id + '"]'))
if 'claims' in expected and expected['claims'] is not None:
# ensure that if we do have the matching claim that it is not active
self.assertEqual(expected['claims'][0]['effectiveamount'], 0)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(claim_id=claim_id)
async def assertNoClaim(self, claim_id: str):
self.assertDictEqual(
{}, json.loads(await self.blockchain._cli_cmnd('getclaimbyid', claim_id))
)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(claim_id=claim_id)
self.assertListEqual([], claim_from_es[0])
claim = await self.conductor.spv_node.server.db.fs_getclaimbyid(claim_id)
claim = await self.conductor.spv_node.server.bp.db.fs_getclaimbyid(claim_id)
self.assertIsNone(claim)
async def assertMatchWinningClaim(self, name):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebybid', name, "[0]"))
stream, channel, _, _ = await self.conductor.spv_node.server.db.resolve(name)
expected = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.bp.db.resolve(name)
claim = stream if stream else channel
expected['claims'][0]['lasttakeoverheight'] = expected['lasttakeoverheight']
await self._assertMatchClaim(expected['claims'][0], claim)
await self._assertMatchClaim(expected, claim)
return claim
async def _assertMatchClaim(self, expected, claim):
self.assertMatchDBClaim(expected, claim)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=claim.claim_hash.hex()
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertMatchESClaim(claim_from_es[0][0], claim)
self._check_supports(claim.claim_hash.hex(), expected.get('supports', []),
claim_from_es[0][0]['support_amount'])
self._check_supports(claim.claim_hash.hex(), expected['supports'], claim_from_es[0][0]['support_amount'])
async def assertMatchClaim(self, name, claim_id, is_active_in_lbrycrd=True):
claim = await self.conductor.spv_node.server.db.fs_getclaimbyid(claim_id)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(
async def assertMatchClaim(self, claim_id, is_active_in_lbrycrd=True):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimbyid', claim_id))
claim = await self.conductor.spv_node.server.bp.db.fs_getclaimbyid(claim_id)
if is_active_in_lbrycrd:
if not expected:
self.assertIsNone(claim)
return
self.assertMatchDBClaim(expected, claim)
else:
self.assertDictEqual({}, expected)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=claim.claim_hash.hex()
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim.claim_hash.hex())
self.assertMatchESClaim(claim_from_es[0][0], claim)
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebyid', name, '["' + claim_id + '"]'))
if is_active_in_lbrycrd:
if not expected:
self.assertIsNone(claim)
return
expected['claims'][0]['lasttakeoverheight'] = expected['lasttakeoverheight']
self.assertMatchDBClaim(expected['claims'][0], claim)
self._check_supports(claim.claim_hash.hex(), expected['claims'][0].get('supports', []),
claim_from_es[0][0]['support_amount'])
else:
if 'claims' in expected and expected['claims'] is not None:
# ensure that if we do have the matching claim that it is not active
self.assertEqual(expected['claims'][0]['effectiveamount'], 0)
self._check_supports(
claim.claim_hash.hex(), expected.get('supports', []), claim_from_es[0][0]['support_amount'],
is_active_in_lbrycrd
)
return claim
async def assertMatchClaimIsWinning(self, name, claim_id):
self.assertEqual(claim_id, (await self.assertMatchWinningClaim(name)).claim_hash.hex())
await self.assertMatchClaimsForName(name)
def _check_supports(self, claim_id, lbrycrd_supports, es_support_amount):
total_lbrycrd_amount = 0.0
total_es_amount = 0.0
active_es_amount = 0.0
db = self.conductor.spv_node.server.db
es_supports = db.get_supports(bytes.fromhex(claim_id))
def _check_supports(self, claim_id, lbrycrd_supports, es_support_amount, is_active_in_lbrycrd=True):
total_amount = 0
db = self.conductor.spv_node.server.bp.db
# we're only concerned about active supports here, and they should match
self.assertTrue(len(es_supports) >= len(lbrycrd_supports))
for i, (tx_num, position, amount) in enumerate(es_supports):
total_es_amount += amount
valid_height = db.get_activation(tx_num, position, is_support=True)
if valid_height > db.db_height:
continue
active_es_amount += amount
txid = db.prefix_db.tx_hash.get(tx_num, deserialize_value=False)[::-1].hex()
support = next(filter(lambda s: s['txid'] == txid and s['n'] == position, lbrycrd_supports))
total_lbrycrd_amount += support['amount']
self.assertEqual(support['height'], bisect_right(db.tx_counts, tx_num))
self.assertEqual(support['validatheight'], valid_height)
self.assertEqual(total_es_amount, es_support_amount)
self.assertEqual(active_es_amount, total_lbrycrd_amount)
for i, (tx_num, position, amount) in enumerate(db.get_supports(bytes.fromhex(claim_id))):
total_amount += amount
if is_active_in_lbrycrd:
support = lbrycrd_supports[i]
self.assertEqual(support['txId'], db.prefix_db.tx_hash.get(tx_num, deserialize_value=False)[::-1].hex())
self.assertEqual(support['n'], position)
self.assertEqual(support['height'], bisect_right(db.tx_counts, tx_num))
self.assertEqual(support['validAtHeight'], db.get_activation(tx_num, position, is_support=True))
self.assertEqual(total_amount, es_support_amount, f"lbrycrd support amount: {total_amount} vs es: {es_support_amount}")
async def assertMatchClaimsForName(self, name):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name, "", "true"))
db = self.conductor.spv_node.server.db
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name))
db = self.conductor.spv_node.server.bp.db
# self.assertEqual(len(expected['claims']), len(db_claims.claims))
# self.assertEqual(expected['lastTakeoverHeight'], db_claims.lastTakeoverHeight)
last_takeover = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))['lastTakeoverHeight']
for c in expected['claims']:
c['lasttakeoverheight'] = expected['lasttakeoverheight']
claim_id = c['claimid']
c['lastTakeoverHeight'] = last_takeover
claim_id = c['claimId']
claim_hash = bytes.fromhex(claim_id)
claim = db._fs_get_claim_by_hash(claim_hash)
self.assertMatchDBClaim(c, claim)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(
claim_id=claim_id
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=c['claimId']
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim_id)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), c['claimId'])
self.assertMatchESClaim(claim_from_es[0][0], claim)
self._check_supports(claim_id, c.get('supports', []),
claim_from_es[0][0]['support_amount'])
async def assertNameState(self, height: int, name: str, winning_claim_id: str, last_takeover_height: int,
non_winning_claims: List[ClaimStateValue]):
self.assertEqual(height, self.conductor.spv_node.server.db.db_height)
await self.assertMatchClaimIsWinning(name, winning_claim_id)
for non_winning in non_winning_claims:
claim = await self.assertMatchClaim(
name, non_winning.claim_id, is_active_in_lbrycrd=non_winning.active_in_lbrycrd
)
self.assertEqual(non_winning.activation_height, claim.activation_height)
self.assertEqual(last_takeover_height, claim.last_takeover_height)
self._check_supports(c['claimId'], c['supports'], claim_from_es[0][0]['support_amount'])
class ResolveCommand(BaseResolveTestCase):
@@ -288,20 +261,19 @@ class ResolveCommand(BaseResolveTestCase):
tx_details = await self.blockchain.get_raw_transaction(claim['txid'])
self.assertEqual(claim['confirmations'], json.loads(tx_details)['confirmations'])
# FIXME : claimname/updateclaim is gone. #3480 wip, unblock #3479"
# resolve handles invalid data
# await self.blockchain_claim_name("gibberish", hexlify(b"{'invalid':'json'}").decode(), "0.1")
# await self.generate(1)
# response = await self.out(self.daemon.jsonrpc_resolve("lbry://gibberish"))
# self.assertSetEqual({'lbry://gibberish'}, set(response))
# claim = response['lbry://gibberish']
# self.assertEqual(claim['name'], 'gibberish')
# self.assertNotIn('value', claim)
await self.blockchain_claim_name("gibberish", hexlify(b"{'invalid':'json'}").decode(), "0.1")
await self.generate(1)
response = await self.out(self.daemon.jsonrpc_resolve("lbry://gibberish"))
self.assertSetEqual({'lbry://gibberish'}, set(response))
claim = response['lbry://gibberish']
self.assertEqual(claim['name'], 'gibberish')
self.assertNotIn('value', claim)
# resolve retries
await self.conductor.spv_node.stop()
resolve_task = asyncio.create_task(self.resolve('foo'))
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.conductor.spv_node.start(self.conductor.blockchain_node)
self.assertIsNotNone((await resolve_task)['claim_id'])
async def test_winning_by_effective_amount(self):
@@ -326,30 +298,6 @@ class ResolveCommand(BaseResolveTestCase):
await self.support_abandon(claim_id1)
await self.assertResolvesToClaimId('@foo', claim_id2)
async def test_resolve_duplicate_name_in_channel(self):
db_resolve = self.conductor.spv_node.server.db.resolve
# first one remains winner unless something else changes
channel_id = self.get_claim_id(await self.channel_create('@foo'))
file_path = self.create_upload_file(data=b'hi!')
tx = await self.daemon.jsonrpc_stream_create('duplicate', '0.1', file_path=file_path, allow_duplicate_name=True, channel_id=channel_id)
await self.ledger.wait(tx)
first_claim = tx.outputs[0].claim_id
file_path = self.create_upload_file(data=b'hi!')
tx = await self.daemon.jsonrpc_stream_create('duplicate', '0.1', file_path=file_path, allow_duplicate_name=True, channel_id=channel_id)
await self.ledger.wait(tx)
duplicate_claim = tx.outputs[0].claim_id
await self.generate(1)
stream, channel, _, _ = await db_resolve(f"@foo:{channel_id}/duplicate:{first_claim}")
self.assertEqual(stream.claim_hash.hex(), first_claim)
self.assertEqual(channel.claim_hash.hex(), channel_id)
stream, channel, _, _ = await db_resolve(f"@foo:{channel_id}/duplicate:{duplicate_claim}")
self.assertEqual(stream.claim_hash.hex(), duplicate_claim)
self.assertEqual(channel.claim_hash.hex(), channel_id)
async def test_advanced_resolve(self):
claim_id1 = self.get_claim_id(
await self.stream_create('foo', '0.7', allow_duplicate_name=True))
@@ -495,16 +443,16 @@ class ResolveCommand(BaseResolveTestCase):
self.assertEqual(one, claim6['name'])
async def test_resolve_old_claim(self):
channel = await self.daemon.jsonrpc_channel_create('@olds', '1.0', blocking=True)
channel = await self.daemon.jsonrpc_channel_create('@olds', '1.0')
await self.confirm_tx(channel.id)
address = channel.outputs[0].get_address(self.account.ledger)
claim = generate_signed_legacy(address, channel.outputs[0])
tx = await Transaction.claim_create('example', claim.SerializeToString(), 1, address, [self.account], self.account)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
response = await self.resolve('@olds/example')
self.assertTrue('is_channel_signature_valid' in response, str(response))
self.assertTrue(response['is_channel_signature_valid'])
claim.publisherSignature.signature = bytes(reversed(claim.publisherSignature.signature))
@@ -512,7 +460,8 @@ class ResolveCommand(BaseResolveTestCase):
'bad_example', claim.SerializeToString(), 1, address, [self.account], self.account
)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
response = await self.resolve('bad_example')
self.assertFalse(response['is_channel_signature_valid'])
@@ -657,12 +606,6 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertDictEqual(await self.resolve('@other/signed4'),
await self.resolve('signed4'))
self.assertEqual(2, len(await self.claim_search(channel_ids=[channel_id2])))
await self.channel_update(channel_id2)
await make_claim('third_signed', '0.01', channel_id=channel_id2)
self.assertEqual(3, len(await self.claim_search(channel_ids=[channel_id2])))
async def _test_activation_delay(self):
name = 'derp'
# initially claim the name
@@ -700,10 +643,10 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
async def assertNameState(self, height: int, name: str, winning_claim_id: str, last_takeover_height: int,
non_winning_claims: List[ClaimStateValue]):
self.assertEqual(height, self.conductor.spv_node.server.db.db_height)
self.assertEqual(height, self.conductor.spv_node.server.bp.db.db_height)
await self.assertMatchClaimIsWinning(name, winning_claim_id)
for non_winning in non_winning_claims:
claim = await self.assertMatchClaim(name,
claim = await self.assertMatchClaim(
non_winning.claim_id, is_active_in_lbrycrd=non_winning.active_in_lbrycrd
)
self.assertEqual(non_winning.activation_height, claim.activation_height)
@@ -1018,7 +961,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
)
greater_than_or_equal_to_zero = [
claim['claim_id'] for claim in (
await self.conductor.spv_node.server.session_manager.search_index.search(
await self.conductor.spv_node.server.bp.db.search_index.search(
channel_id=channel_id, fee_amount=">=0"
))[0]
]
@@ -1026,7 +969,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertSetEqual(set(greater_than_or_equal_to_zero), {stream_with_no_fee, stream_with_fee})
greater_than_zero = [
claim['claim_id'] for claim in (
await self.conductor.spv_node.server.session_manager.search_index.search(
await self.conductor.spv_node.server.bp.db.search_index.search(
channel_id=channel_id, fee_amount=">0"
))[0]
]
@@ -1034,7 +977,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertSetEqual(set(greater_than_zero), {stream_with_fee})
equal_to_zero = [
claim['claim_id'] for claim in (
await self.conductor.spv_node.server.session_manager.search_index.search(
await self.conductor.spv_node.server.bp.db.search_index.search(
channel_id=channel_id, fee_amount="<=0"
))[0]
]
@@ -1049,10 +992,10 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
name = 'test'
await self.generate(494)
address = (await self.account.receiving.get_addresses(True))[0]
await self.send_to_address_and_wait(address, 400.0)
await self.blockchain.send_to_address(address, 400.0)
await self.account.ledger.on_address.first
await self.generate(100)
self.assertEqual(800, self.conductor.spv_node.server.db.db_height)
self.assertEqual(800, self.conductor.spv_node.server.bp.db.db_height)
# Block 801: Claim A for 10 LBC is accepted.
# It is the first claim, so it immediately becomes active and controlling.
@@ -1064,10 +1007,10 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
# Its activation height is 1121 + min(4032, floor((1121-801) / 32)) = 1121 + 10 = 1131.
# State: A(10) is controlling, B(20) is accepted.
await self.generate(32 * 10 - 1)
self.assertEqual(1120, self.conductor.spv_node.server.db.db_height)
self.assertEqual(1120, self.conductor.spv_node.server.bp.db.db_height)
claim_id_B = (await self.stream_create(name, '20.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
claim_B, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_B}")
self.assertEqual(1121, self.conductor.spv_node.server.db.db_height)
claim_B, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_B}")
self.assertEqual(1121, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(1131, claim_B.activation_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
@@ -1075,33 +1018,33 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
# Since it is a support for the controlling claim, it activates immediately.
# State: A(10+14) is controlling, B(20) is accepted.
await self.support_create(claim_id_A, bid='14.0')
self.assertEqual(1122, self.conductor.spv_node.server.db.db_height)
self.assertEqual(1122, self.conductor.spv_node.server.bp.db.db_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
# Block 1123: Claim C for 50 LBC is accepted.
# The activation height is 1123 + min(4032, floor((1123-801) / 32)) = 1123 + 10 = 1133.
# State: A(10+14) is controlling, B(20) is accepted, C(50) is accepted.
claim_id_C = (await self.stream_create(name, '50.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
self.assertEqual(1123, self.conductor.spv_node.server.db.db_height)
claim_C, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_C}")
self.assertEqual(1123, self.conductor.spv_node.server.bp.db.db_height)
claim_C, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_C}")
self.assertEqual(1133, claim_C.activation_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
await self.generate(7)
self.assertEqual(1130, self.conductor.spv_node.server.db.db_height)
self.assertEqual(1130, self.conductor.spv_node.server.bp.db.db_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
await self.generate(1)
# Block 1131: Claim B activates. It has 20 LBC, while claim A has 24 LBC (10 original + 14 from support X). There is no takeover, and claim A remains controlling.
# State: A(10+14) is controlling, B(20) is active, C(50) is accepted.
self.assertEqual(1131, self.conductor.spv_node.server.db.db_height)
self.assertEqual(1131, self.conductor.spv_node.server.bp.db.db_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
# Block 1132: Claim D for 300 LBC is accepted. The activation height is 1132 + min(4032, floor((1132-801) / 32)) = 1132 + 10 = 1142.
# State: A(10+14) is controlling, B(20) is active, C(50) is accepted, D(300) is accepted.
claim_id_D = (await self.stream_create(name, '300.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
self.assertEqual(1132, self.conductor.spv_node.server.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(1132, self.conductor.spv_node.server.bp.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(False, claim_D.is_controlling)
self.assertEqual(801, claim_D.last_takeover_height)
self.assertEqual(1142, claim_D.activation_height)
@@ -1110,8 +1053,8 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
# Block 1133: Claim C activates. It has 50 LBC, while claim A has 24 LBC, so a takeover is initiated. The takeover height for this name is set to 1133, and therefore the activation delay for all the claims becomes min(4032, floor((1133-1133) / 32)) = 0. All the claims become active. The totals for each claim are recalculated, and claim D becomes controlling because it has the highest total.
# State: A(10+14) is active, B(20) is active, C(50) is active, D(300) is controlling
await self.generate(1)
self.assertEqual(1133, self.conductor.spv_node.server.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(1133, self.conductor.spv_node.server.bp.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(True, claim_D.is_controlling)
self.assertEqual(1133, claim_D.last_takeover_height)
self.assertEqual(1133, claim_D.activation_height)
@@ -1384,15 +1327,15 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
await self.generate(8)
await self.assertMatchClaimIsWinning(name, first_claim_id)
# abandon the support that causes the winning claim to have the highest staked
tx = await self.daemon.jsonrpc_txo_spend(type='support', txid=controlling_support_tx.id, blocking=True)
tx = await self.daemon.jsonrpc_txo_spend(type='support', txid=controlling_support_tx.id)
await self.generate(1)
await self.assertNameState(538, name, first_claim_id, last_takeover_height=207, non_winning_claims=[
ClaimStateValue(second_claim_id, activation_height=539, active_in_lbrycrd=False)
])
await self.assertMatchClaimIsWinning(name, first_claim_id)
# await self.assertMatchClaim(second_claim_id)
await self.generate(1)
await self.assertNameState(539, name, second_claim_id, last_takeover_height=539, non_winning_claims=[
ClaimStateValue(first_claim_id, activation_height=207, active_in_lbrycrd=True)
])
await self.assertMatchClaim(first_claim_id)
await self.assertMatchClaimIsWinning(name, second_claim_id)
async def test_remove_controlling_support(self):
name = 'derp'
@@ -1462,14 +1405,14 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
await self.generate(32)
second_claim_id = (await self.stream_create(name, '0.01', allow_duplicate_name=True))['outputs'][0]['claim_id']
await self.assertNoClaim(name, second_claim_id)
await self.assertNoClaim(second_claim_id)
self.assertEqual(
len((await self.conductor.spv_node.server.session_manager.search_index.search(claim_name=name))[0]), 1
len((await self.conductor.spv_node.server.bp.db.search_index.search(claim_name=name))[0]), 1
)
await self.generate(1)
await self.assertMatchClaim(name, second_claim_id)
await self.assertMatchClaim(second_claim_id)
self.assertEqual(
len((await self.conductor.spv_node.server.session_manager.search_index.search(claim_name=name))[0]), 2
len((await self.conductor.spv_node.server.bp.db.search_index.search(claim_name=name))[0]), 2
)
async def test_abandon_controlling_same_block_as_new_claim(self):
@@ -1485,47 +1428,35 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
async def test_trending(self):
async def get_trending_score(claim_id):
return (await self.conductor.spv_node.server.session_manager.search_index.search(
return (await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=claim_id
))[0][0]['trending_score']
claim_id1 = (await self.stream_create('derp', '1.0'))['outputs'][0]['claim_id']
COIN = int(1E8)
COIN = 1E8
self.assertEqual(self.conductor.spv_node.writer.height, 207)
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(208, bytes.fromhex(claim_id1)), (0, 10 * COIN)
height = 99000
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id1, height, 0, 10 * COIN
)
await self.generate(1)
self.assertEqual(self.conductor.spv_node.writer.height, 208)
self.assertEqual(1.7090807854206793, await get_trending_score(claim_id1))
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(209, bytes.fromhex(claim_id1)), (10 * COIN, 100 * COIN)
self.assertEqual(172.64252836433135, await get_trending_score(claim_id1))
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id1, height + 1, 10 * COIN, 100 * COIN
)
await self.generate(1)
self.assertEqual(self.conductor.spv_node.writer.height, 209)
self.assertEqual(2.2437974397778886, await get_trending_score(claim_id1))
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(309, bytes.fromhex(claim_id1)), (100 * COIN, 1000000 * COIN)
self.assertEqual(173.45931832928875, await get_trending_score(claim_id1))
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id1, height + 100, 100 * COIN, 1000000 * COIN
)
await self.generate(100)
self.assertEqual(self.conductor.spv_node.writer.height, 309)
self.assertEqual(5.157053472135866, await get_trending_score(claim_id1))
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(409, bytes.fromhex(claim_id1)), (1000000 * COIN, 1 * COIN)
)
await self.generate(99)
self.assertEqual(self.conductor.spv_node.writer.height, 408)
self.assertEqual(5.157053472135866, await get_trending_score(claim_id1))
await self.generate(1)
self.assertEqual(self.conductor.spv_node.writer.height, 409)
self.assertEqual(-3.4256156592205627, await get_trending_score(claim_id1))
search_results = (await self.conductor.spv_node.server.session_manager.search_index.search(claim_name="derp"))[0]
self.assertEqual(176.65517070393514, await get_trending_score(claim_id1))
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id1, height + 200, 1000000 * COIN, 1 * COIN
)
await self.generate(1)
self.assertEqual(-174.951347102643, await get_trending_score(claim_id1))
search_results = (await self.conductor.spv_node.server.bp.db.search_index.search(claim_name="derp"))[0]
self.assertEqual(1, len(search_results))
self.assertListEqual([claim_id1], [c['claim_id'] for c in search_results])
@@ -1534,31 +1465,22 @@ class ResolveAfterReorg(BaseResolveTestCase):
async def reorg(self, start):
blocks = self.ledger.headers.height - start
self.blockchain.block_expected = start - 1
prepare = self.ledger.on_header.where(self.blockchain.is_expected_block)
self.conductor.spv_node.server.synchronized.clear()
# go back to start
await self.blockchain.invalidate_block((await self.ledger.headers.hash(start)).decode())
# go to previous + 1
await self.blockchain.generate(blocks + 2)
await prepare # no guarantee that it didn't happen already, so start waiting from before calling generate
await self.conductor.spv_node.server.synchronized.wait()
# await asyncio.wait_for(self.on_header(self.blockchain.block_expected), 30.0)
await self.generate(blocks + 2)
async def assertBlockHash(self, height):
reader_db = self.conductor.spv_node.server.db
bp = self.conductor.spv_node.server.bp
block_hash = await self.blockchain.get_block_hash(height)
self.assertEqual(block_hash, (await self.ledger.headers.hash(height)).decode())
self.assertEqual(block_hash, (await reader_db.fs_block_hashes(height, 1))[0][::-1].hex())
self.assertEqual(block_hash, (await bp.db.fs_block_hashes(height, 1))[0][::-1].hex())
txids = [
tx_hash[::-1].hex() for tx_hash in reader_db.get_block_txs(height)
tx_hash[::-1].hex() for tx_hash in bp.db.get_block_txs(height)
]
txs = await reader_db.get_transactions_and_merkles(txids)
block_txs = (await self.conductor.spv_node.server.daemon.deserialised_block(block_hash))['tx']
txs = await bp.db.get_transactions_and_merkles(txids)
block_txs = (await bp.daemon.deserialised_block(block_hash))['tx']
self.assertSetEqual(set(block_txs), set(txs.keys()), msg='leveldb/lbrycrd is missing transactions')
self.assertListEqual(block_txs, list(txs.keys()), msg='leveldb/lbrycrd transactions are of order')
@@ -1569,18 +1491,9 @@ class ResolveAfterReorg(BaseResolveTestCase):
channel_id = self.get_claim_id(
await self.channel_create(channel_name, '0.01')
)
await self.assertNameState(
height=207, name='@abc', winning_claim_id=channel_id, last_takeover_height=207,
non_winning_claims=[]
)
self.assertEqual(channel_id, (await self.assertMatchWinningClaim(channel_name)).claim_hash.hex())
await self.reorg(206)
await self.assertNameState(
height=208, name='@abc', winning_claim_id=channel_id, last_takeover_height=207,
non_winning_claims=[]
)
self.assertEqual(channel_id, (await self.assertMatchWinningClaim(channel_name)).claim_hash.hex())
# await self.assertNoClaimForName(channel_name)
# self.assertNotIn('error', await self.resolve(channel_name))
@@ -1589,29 +1502,16 @@ class ResolveAfterReorg(BaseResolveTestCase):
stream_id = self.get_claim_id(
await self.stream_create(stream_name, '0.01', channel_id=channel_id)
)
await self.assertNameState(
height=209, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.reorg(206)
await self.assertNameState(
height=210, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.support_create(stream_id, '0.01')
await self.assertNameState(
height=211, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
self.assertNotIn('error', await self.resolve(stream_name))
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.reorg(206)
# self.assertNotIn('error', await self.resolve(stream_name))
await self.assertNameState(
height=212, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.stream_abandon(stream_id)
self.assertNotIn('error', await self.resolve(channel_name))
@@ -1653,6 +1553,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
await self.ledger.wait(broadcast_tx)
await self.support_create(still_valid.outputs[0].claim_id, '0.01')
# await self.generate(1)
await self.ledger.wait(broadcast_tx, self.blockchain.block_expected)
self.assertEqual(self.ledger.headers.height, 208)
await self.assertBlockHash(208)
@@ -1669,7 +1570,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -1702,7 +1603,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
await self.blockchain.generate(1)
# wait for the client to catch up
await asyncio.wait_for(self.on_header(210), 3.0)
await asyncio.wait_for(self.on_header(210), 1.0)
# verify the claim is in the new block and that it is returned by claim_search
republished = await self.resolve('hovercraft')
@@ -1748,11 +1649,11 @@ class ResolveAfterReorg(BaseResolveTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
await asyncio.wait_for(self.on_header(209), 30.0)
await asyncio.wait_for(self.on_header(209), 3.0)
await self.assertBlockHash(207)
await self.assertBlockHash(208)
await self.assertBlockHash(209)
@@ -1820,7 +1721,7 @@ def generate_signed_legacy(address: bytes, output: Output):
claim.SerializeToString(),
output.claim_hash[::-1]
]))
signature = output.private_key.sign_compact(digest)
signature = output.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
claim.publisherSignature.version = 1
claim.publisherSignature.signatureType = 1
claim.publisherSignature.signature = signature
@@ -21,8 +21,9 @@ class BasicTransactionTest(IntegrationTestCase):
[asyncio.ensure_future(self.on_address_update(address1)),
asyncio.ensure_future(self.on_address_update(address2))]
))
await self.send_to_address_and_wait(address1, 5)
await self.send_to_address_and_wait(address2, 5, 1)
sendtxid1 = await self.blockchain.send_to_address(address1, 5)
sendtxid2 = await self.blockchain.send_to_address(address2, 5)
await self.blockchain.generate(1)
await notifications
self.assertEqual(d2l(await self.account.get_balance()), '10.0')
@@ -31,9 +32,7 @@ class BasicTransactionTest(IntegrationTestCase):
channel_txo = Output.pay_claim_name_pubkey_hash(
l2d('1.0'), '@bar', channel, self.account.ledger.address_to_hash160(address1)
)
channel_txo.set_channel_private_key(
await self.account.generate_channel_private_key()
)
await channel_txo.generate_channel_private_key()
channel_txo.script.generate()
channel_tx = await Transaction.create([], [channel_txo], [self.account], self.account)
@@ -56,7 +55,7 @@ class BasicTransactionTest(IntegrationTestCase):
notifications = asyncio.create_task(asyncio.wait(
[asyncio.ensure_future(self.ledger.wait(channel_tx)), asyncio.ensure_future(self.ledger.wait(stream_tx))]
))
await self.generate(1)
await self.blockchain.generate(1)
await notifications
self.assertEqual(d2l(await self.account.get_balance()), '7.985786')
self.assertEqual(d2l(await self.account.get_balance(include_claims=True)), '9.985786')
@@ -69,7 +68,7 @@ class BasicTransactionTest(IntegrationTestCase):
await self.broadcast(abandon_tx)
await notify
notify = asyncio.create_task(self.ledger.wait(abandon_tx))
await self.generate(1)
await self.blockchain.generate(1)
await notify
response = await self.ledger.resolve([], ['lbry://@bar/foo'])
@@ -1,6 +1,3 @@
import asyncio
import unittest
from lbry.testcase import CommandTestCase
@@ -20,7 +17,7 @@ class TransactionCommandsTestCase(CommandTestCase):
async def test_transaction_show(self):
# local tx
result = await self.out(self.daemon.jsonrpc_account_send(
'5.0', await self.daemon.jsonrpc_address_unused(self.account.id), blocking=True
'5.0', await self.daemon.jsonrpc_address_unused(self.account.id)
))
await self.confirm_tx(result['txid'])
tx = await self.daemon.jsonrpc_transaction_show(result['txid'])
@@ -29,7 +26,6 @@ class TransactionCommandsTestCase(CommandTestCase):
# someone's tx
change_address = await self.blockchain.get_raw_change_address()
sendtxid = await self.blockchain.send_to_address(change_address, 10)
await asyncio.sleep(0.2)
tx = await self.daemon.jsonrpc_transaction_show(sendtxid)
self.assertEqual(tx.id, sendtxid)
self.assertEqual(tx.height, -1)
@@ -42,9 +38,10 @@ class TransactionCommandsTestCase(CommandTestCase):
self.assertFalse(result['success'])
async def test_utxo_release(self):
await self.send_to_address_and_wait(
await self.account.receiving.get_or_create_usable_address(), 1, 1
sendtxid = await self.blockchain.send_to_address(
await self.account.receiving.get_or_create_usable_address(), 1
)
await self.confirm_tx(sendtxid)
await self.assertBalance(self.account, '11.0')
await self.ledger.reserve_outputs(await self.account.get_utxos())
await self.assertBalance(self.account, '0.0')
@@ -54,7 +51,6 @@ class TransactionCommandsTestCase(CommandTestCase):
class TestSegwit(CommandTestCase):
@unittest.SkipTest
async def test_segwit(self):
p2sh_address1 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
p2sh_address2 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
@@ -68,13 +64,14 @@ class TestSegwit(CommandTestCase):
p2sh_txid2 = await self.blockchain.send_to_address(p2sh_address2, '1.0')
bech32_txid1 = await self.blockchain.send_to_address(bech32_address1, '1.0')
bech32_txid2 = await self.blockchain.send_to_address(bech32_address2, '1.0')
await self.generate(1)
# P2SH & BECH32 can pay to P2SH address
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid1, "vout": 0},
{"txid": bech32_txid1, "vout": 0},
], {p2sh_address3: 1.9}
], [{p2sh_address3: '1.9'}]
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
p2sh_txid3 = await self.blockchain.send_raw_transaction(tx)
@@ -85,7 +82,7 @@ class TestSegwit(CommandTestCase):
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid2, "vout": 0},
{"txid": bech32_txid2, "vout": 0},
], {bech32_address3: 1.9}
], [{bech32_address3: '1.9'}]
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
bech32_txid3 = await self.blockchain.send_raw_transaction(tx)
@@ -97,9 +94,12 @@ class TestSegwit(CommandTestCase):
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid3, "vout": 0},
{"txid": bech32_txid3, "vout": 0},
], {address: 3.5}
], [{address: '3.5'}]
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
txid = await self.blockchain.send_raw_transaction(tx)
await self.generate_and_wait(1, [txid])
await self.on_transaction_id(txid)
await self.generate(1)
await self.on_transaction_id(txid)
await self.assertBalance(self.account, '13.5')
@@ -1,7 +1,7 @@
import asyncio
import random
from itertools import chain
import lbry.wallet.rpc.jsonrpc
from lbry.wallet.transaction import Transaction, Output, Input
from lbry.testcase import IntegrationTestCase
from lbry.wallet.util import satoshis_to_coins, coins_to_satoshis
@@ -9,8 +9,9 @@ from lbry.wallet.manager import WalletManager
class BasicTransactionTests(IntegrationTestCase):
async def test_variety_of_transactions_and_longish_history(self):
await self.generate(300)
await self.blockchain.generate(300)
await self.assertBalance(self.account, '0.0')
addresses = await self.account.receiving.get_addresses()
@@ -18,10 +19,10 @@ class BasicTransactionTests(IntegrationTestCase):
# to the 10th receiving address for a total of 30 UTXOs on the entire account
for i in range(10):
notification = asyncio.ensure_future(self.on_address_update(addresses[i]))
_ = await self.send_to_address_and_wait(addresses[i], 10)
txid = await self.blockchain.send_to_address(addresses[i], 10)
await notification
notification = asyncio.ensure_future(self.on_address_update(addresses[9]))
_ = await self.send_to_address_and_wait(addresses[9], 10)
txid = await self.blockchain.send_to_address(addresses[9], 10)
await notification
# use batching to reduce issues with send_to_address on cli
@@ -56,7 +57,7 @@ class BasicTransactionTests(IntegrationTestCase):
for tx in await self.ledger.db.get_transactions(txid__in=[tx.id for tx in txs])
]))
await self.generate(1)
await self.blockchain.generate(1)
await asyncio.wait([self.ledger.wait(tx) for tx in txs])
await self.assertBalance(self.account, '199.99876')
@@ -73,7 +74,7 @@ class BasicTransactionTests(IntegrationTestCase):
)
await self.broadcast(tx)
await self.ledger.wait(tx)
await self.generate(1)
await self.blockchain.generate(1)
await self.ledger.wait(tx)
self.assertEqual(2, await self.account.get_utxo_count()) # 199 + change
@@ -87,10 +88,12 @@ class BasicTransactionTests(IntegrationTestCase):
await self.assertBalance(account2, '0.0')
addresses = await account1.receiving.get_addresses()
txids = []
for address in addresses[:5]:
txids.append(await self.send_to_address_and_wait(address, 1.1))
await self.generate_and_wait(1, txids)
txids = await asyncio.gather(*(
self.blockchain.send_to_address(address, 1.1) for address in addresses[:5]
))
await asyncio.wait([self.on_transaction_id(txid) for txid in txids]) # mempool
await self.blockchain.generate(1)
await asyncio.wait([self.on_transaction_id(txid) for txid in txids]) # confirmed
await self.assertBalance(account1, '5.5')
await self.assertBalance(account2, '0.0')
@@ -104,7 +107,7 @@ class BasicTransactionTests(IntegrationTestCase):
)
await self.broadcast(tx)
await self.ledger.wait(tx) # mempool
await self.generate(1)
await self.blockchain.generate(1)
await self.ledger.wait(tx) # confirmed
await self.assertBalance(account1, '3.499802')
@@ -118,7 +121,7 @@ class BasicTransactionTests(IntegrationTestCase):
)
await self.broadcast(tx)
await self.ledger.wait(tx) # mempool
await self.generate(1)
await self.blockchain.generate(1)
await self.ledger.wait(tx) # confirmed
tx = (await account1.get_transactions(include_is_my_input=True, include_is_my_output=True))[1]
@@ -130,11 +133,11 @@ class BasicTransactionTests(IntegrationTestCase):
self.assertTrue(tx.outputs[1].is_internal_transfer)
async def test_history_edge_cases(self):
await self.generate(300)
await self.blockchain.generate(300)
await self.assertBalance(self.account, '0.0')
address = await self.account.receiving.get_or_create_usable_address()
# evil trick: mempool is unsorted on real life, but same order between python instances. reproduce it
original_summary = self.conductor.spv_node.server.mempool.transaction_summaries
original_summary = self.conductor.spv_node.server.bp.mempool.transaction_summaries
def random_summary(*args, **kwargs):
summary = original_summary(*args, **kwargs)
@@ -143,10 +146,13 @@ class BasicTransactionTests(IntegrationTestCase):
while summary == ordered:
random.shuffle(summary)
return summary
self.conductor.spv_node.server.mempool.transaction_summaries = random_summary
self.conductor.spv_node.server.bp.mempool.transaction_summaries = random_summary
# 10 unconfirmed txs, all from blockchain wallet
for i in range(10):
await self.send_to_address_and_wait(address, 10)
sends = [self.blockchain.send_to_address(address, 10) for _ in range(10)]
# use batching to reduce issues with send_to_address on cli
for batch in range(0, len(sends), 10):
txids = await asyncio.gather(*sends[batch:batch + 10])
await asyncio.wait([self.on_transaction_id(txid) for txid in txids])
remote_status = await self.ledger.network.subscribe_address(address)
self.assertTrue(await self.ledger.update_history(address, remote_status))
# 20 unconfirmed txs, 10 from blockchain, 10 from local to local
@@ -164,7 +170,8 @@ class BasicTransactionTests(IntegrationTestCase):
remote_status = await self.ledger.network.subscribe_address(address)
self.assertTrue(await self.ledger.update_history(address, remote_status))
# server history grows unordered
await self.send_to_address_and_wait(address, 1)
txid = await self.blockchain.send_to_address(address, 1)
await self.on_transaction_id(txid)
self.assertTrue(await self.ledger.update_history(address, remote_status))
self.assertEqual(21, len((await self.ledger.get_local_status_and_history(address))[1]))
self.assertEqual(0, len(self.ledger._known_addresses_out_of_sync))
@@ -188,37 +195,37 @@ class BasicTransactionTests(IntegrationTestCase):
self.ledger, 2000000000000, [self.account], set_reserved=False, return_insufficient_funds=True
)
got_amounts = [estimator.effective_amount for estimator in spendable]
self.assertListEqual(sorted(amounts), sorted(got_amounts))
self.assertListEqual(amounts, got_amounts)
async def test_sqlite_coin_chooser(self):
wallet_manager = WalletManager([self.wallet], {self.ledger.get_id(): self.ledger})
await self.generate(300)
await self.blockchain.generate(300)
await self.assertBalance(self.account, '0.0')
address = await self.account.receiving.get_or_create_usable_address()
other_account = self.wallet.generate_account(self.ledger)
other_address = await other_account.receiving.get_or_create_usable_address()
self.ledger.coin_selection_strategy = 'sqlite'
await self.ledger.subscribe_account(other_account)
await self.ledger.subscribe_account(self.account)
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 1.0)
txid = await self.blockchain.send_to_address(address, 1.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 1.0)
txid = await self.blockchain.send_to_address(address, 1.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 3.0)
txid = await self.blockchain.send_to_address(address, 3.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 5.0)
txid = await self.blockchain.send_to_address(address, 5.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 10.0)
txid = await self.blockchain.send_to_address(address, 10.0)
await accepted
await self.assertBalance(self.account, '20.0')
@@ -259,12 +266,6 @@ class BasicTransactionTests(IntegrationTestCase):
async def broadcast(tx):
try:
return await real_broadcast(tx)
except lbry.wallet.rpc.jsonrpc.RPCError as err:
# this is expected in tests where we try to double spend.
if 'the transaction was rejected by network rules.' in str(err):
pass
else:
raise err
finally:
e.set()
@@ -54,8 +54,7 @@ class BlobExchangeTestBase(AsyncioTestCase):
download_dir=self.client_dir,
wallet=self.client_wallet_dir,
save_files=True,
fixed_peers=[],
tracker_servers=[]
fixed_peers=[]
)
self.client_config.transaction_cache_size = 10000
self.client_storage = SQLiteStorage(self.client_config, os.path.join(self.client_dir, "lbrynet.sqlite"))
@@ -34,7 +34,6 @@ class TestComponentManager(AsyncioTestCase):
],
[
components.BackgroundDownloaderComponent,
components.TrackerAnnouncerComponent
]
]
self.component_manager = ComponentManager(Config())
@@ -151,9 +150,6 @@ class FakeDelayedFileManager(FakeComponent):
async def start(self):
await asyncio.sleep(1)
def get_filtered(self):
return []
class TestComponentManagerProperStart(AdvanceTimeTestCase):
@@ -11,7 +11,7 @@ from lbry.extras.daemon.components import (
DATABASE_COMPONENT, DISK_SPACE_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT,
HASH_ANNOUNCER_COMPONENT, FILE_MANAGER_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT,
UPNP_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, WALLET_SERVER_PAYMENTS_COMPONENT,
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT, TRACKER_ANNOUNCER_COMPONENT
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT
)
from lbry.extras.daemon.daemon import Daemon
@@ -72,7 +72,7 @@ class TestAccessHeaders(AsyncioTestCase):
DATABASE_COMPONENT, DISK_SPACE_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT,
HASH_ANNOUNCER_COMPONENT, FILE_MANAGER_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT,
UPNP_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, WALLET_SERVER_PAYMENTS_COMPONENT,
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT, TRACKER_ANNOUNCER_COMPONENT
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT
)
Daemon.component_attributes = {}
self.daemon = Daemon(conf)
+3 -1
View File
@@ -138,7 +138,9 @@ class TestTypesV1Compatibility(TestCase):
channel = cert.channel
self.assertEqual(
channel.public_key,
'033878b1edd4a1373149909ef03f4339f6da9c2bd2214c040fd2e530463ffe6609'
'3056301006072a8648ce3d020106052b8104000a034200043878b1edd4a1373149909ef03f4339f6da9c2b'
'd2214c040fd2e530463ffe66098eca14fc70b50ff3aefd106049a815f595ed5a13eda7419ad78d9ed7ae47'
'3f17'
)
def test_unsigned_with_fee(self):
+7
View File
@@ -13,6 +13,13 @@ from lbry.stream.descriptor import StreamDescriptor
from tests.unit.blob_exchange.test_transfer_blob import BlobExchangeTestBase
def get_mock_node(loop):
mock_node = mock.Mock(spec=Node)
mock_node.joined = asyncio.Event(loop=loop)
mock_node.joined.set()
return mock_node
class TestManagedStream(BlobExchangeTestBase):
async def create_stream(self, blob_count: int = 10, file_name='test_file'):
self.stream_bytes = b''
-92
View File
@@ -1,92 +0,0 @@
import asyncio
import random
from lbry.testcase import AsyncioTestCase
from lbry.dht.peer import KademliaPeer
from lbry.torrent.tracker import CompactIPv4Peer, TrackerClient, enqueue_tracker_search, UDPTrackerServerProtocol, encode_peer
class UDPTrackerClientTestCase(AsyncioTestCase):
async def asyncSetUp(self):
self.client_servers_list = []
self.servers = {}
self.client = TrackerClient(b"\x00" * 48, 4444, lambda: self.client_servers_list, timeout=1)
await self.client.start()
self.addCleanup(self.client.stop)
await self.add_server()
async def add_server(self, port=None, add_to_client=True):
port = port or len(self.servers) + 59990
assert port not in self.servers
server = UDPTrackerServerProtocol()
self.servers[port] = server
transport, _ = await self.loop.create_datagram_endpoint(lambda: server, local_addr=("127.0.0.1", port))
self.addCleanup(transport.close)
if add_to_client:
self.client_servers_list.append(("127.0.0.1", port))
async def test_announce(self):
info_hash = random.getrandbits(160).to_bytes(20, "big", signed=False)
announcement = (await self.client.get_peer_list(info_hash))[0]
self.assertEqual(announcement.seeders, 1)
self.assertEqual(announcement.peers,
[CompactIPv4Peer(int.from_bytes(bytes([127, 0, 0, 1]), "big", signed=False), 4444)])
async def test_announce_many_info_hashes_to_many_servers_with_bad_one_and_dns_error(self):
await asyncio.gather(*[self.add_server() for _ in range(3)])
self.client_servers_list.append(("no.it.does.not.exist", 7070))
self.client_servers_list.append(("127.0.0.2", 7070))
info_hashes = [random.getrandbits(160).to_bytes(20, "big", signed=False) for _ in range(5)]
await self.client.announce_many(*info_hashes)
for server in self.servers.values():
self.assertDictEqual(
server.peers, {
info_hash: [encode_peer("127.0.0.1", self.client.announce_port)] for info_hash in info_hashes
})
async def test_announce_using_helper_function(self):
info_hash = random.getrandbits(160).to_bytes(20, "big", signed=False)
queue = asyncio.Queue()
enqueue_tracker_search(info_hash, queue)
peers = await queue.get()
self.assertEqual(peers, [KademliaPeer('127.0.0.1', None, None, 4444, allow_localhost=True)])
async def test_error(self):
info_hash = random.getrandbits(160).to_bytes(20, "big", signed=False)
await self.client.get_peer_list(info_hash)
list(self.servers.values())[0].known_conns.clear()
self.client.results.clear()
with self.assertRaises(Exception) as err:
await self.client.get_peer_list(info_hash)
self.assertEqual(err.exception.args[0], b'Connection ID missmatch.\x00')
async def test_multiple_servers(self):
await asyncio.gather(*[self.add_server() for _ in range(10)])
info_hash = random.getrandbits(160).to_bytes(20, "big", signed=False)
await self.client.get_peer_list(info_hash)
for server in self.servers.values():
self.assertEqual(server.peers, {info_hash: [encode_peer("127.0.0.1", self.client.announce_port)]})
async def test_multiple_servers_with_bad_one(self):
await asyncio.gather(*[self.add_server() for _ in range(10)])
self.client_servers_list.append(("127.0.0.2", 7070))
info_hash = random.getrandbits(160).to_bytes(20, "big", signed=False)
await self.client.get_peer_list(info_hash)
for server in self.servers.values():
self.assertEqual(server.peers, {info_hash: [encode_peer("127.0.0.1", self.client.announce_port)]})
async def test_multiple_servers_with_different_peers_across_helper_function(self):
# this is how the downloader uses it
await asyncio.gather(*[self.add_server() for _ in range(10)])
info_hash = random.getrandbits(160).to_bytes(20, "big", signed=False)
fake_peers = []
for server in self.servers.values():
for _ in range(10):
peer = (f"127.0.0.{random.randint(1, 255)}", random.randint(2000, 65500))
fake_peers.append(peer)
server.add_peer(info_hash, *peer)
peer_q = asyncio.Queue()
enqueue_tracker_search(info_hash, peer_q)
await asyncio.sleep(0)
await asyncio.gather(*self.client.tasks.values())
self.assertEqual(11, peer_q.qsize())
+60
View File
@@ -0,0 +1,60 @@
import time
import unittest
from lbry.wallet.server.metrics import ServerLoadData, calculate_avg_percentiles
class TestPercentileCalculation(unittest.TestCase):
def test_calculate_percentiles(self):
self.assertEqual(calculate_avg_percentiles([]), (0, 0, 0, 0, 0, 0, 0, 0))
self.assertEqual(calculate_avg_percentiles([1]), (1, 1, 1, 1, 1, 1, 1, 1))
self.assertEqual(calculate_avg_percentiles([1, 2]), (1, 1, 1, 1, 1, 2, 2, 2))
self.assertEqual(calculate_avg_percentiles([1, 2, 3]), (2, 1, 1, 1, 2, 3, 3, 3))
self.assertEqual(calculate_avg_percentiles([4, 1, 2, 3]), (2, 1, 1, 1, 2, 3, 4, 4))
self.assertEqual(calculate_avg_percentiles([1, 2, 3, 4, 5, 6]), (3, 1, 1, 2, 3, 5, 6, 6))
self.assertEqual(calculate_avg_percentiles(
list(range(1, 101))), (50, 1, 5, 25, 50, 75, 95, 100))
class TestCollectingMetrics(unittest.TestCase):
def test_happy_path(self):
self.maxDiff = None
load = ServerLoadData()
search = load.for_api('search')
self.assertEqual(search.name, 'search')
search.start()
search.cache_response()
search.cache_response()
metrics = {
'search': [{'total': 40}],
'execute_query': [
{'total': 20},
{'total': 10}
]
}
for x in range(5):
search.query_response(time.perf_counter() - 0.055 + 0.001*x, metrics)
metrics['execute_query'][0]['total'] = 10
metrics['execute_query'][0]['sql'] = "select lots, of, stuff FROM claim where something=1"
search.query_interrupt(time.perf_counter() - 0.050, metrics)
search.query_error(time.perf_counter() - 0.050, metrics)
search.query_error(time.perf_counter() - 0.052, {})
self.assertEqual(load.to_json_and_reset({}), {'status': {}, 'api': {'search': {
"receive_count": 1,
"cache_response_count": 2,
"query_response_count": 5,
"intrp_response_count": 1,
"error_response_count": 2,
"response": (53, 51, 51, 52, 53, 54, 55, 55),
"interrupt": (50, 50, 50, 50, 50, 50, 50, 50),
"error": (51, 50, 50, 50, 50, 52, 52, 52),
"python": (12, 10, 10, 10, 10, 20, 20, 20),
"wait": (12, 10, 10, 10, 12, 14, 15, 15),
"sql": (27, 20, 20, 20, 30, 30, 30, 30),
"individual_sql": (13, 10, 10, 10, 10, 20, 20, 20),
"individual_sql_count": 14,
"errored_queries": ['FROM claim where something=1'],
"interrupted_queries": ['FROM claim where something=1'],
}}})
self.assertEqual(load.to_json_and_reset({}), {'status': {}, 'api': {}})
+153
View File
@@ -0,0 +1,153 @@
import unittest
import tempfile
import shutil
from lbry.wallet.server.db.revertable import RevertableOpStack, RevertableDelete, RevertablePut, OpStackIntegrity
from lbry.wallet.server.db.prefixes import ClaimToTXOPrefixRow, HubDB
class TestRevertableOpStack(unittest.TestCase):
def setUp(self):
self.fake_db = {}
self.stack = RevertableOpStack(self.fake_db.get)
def tearDown(self) -> None:
self.stack.clear()
self.fake_db.clear()
def process_stack(self):
for op in self.stack:
if op.is_put:
self.fake_db[op.key] = op.value
else:
self.fake_db.pop(op.key)
self.stack.clear()
def update(self, key1: bytes, value1: bytes, key2: bytes, value2: bytes):
self.stack.append_op(RevertableDelete(key1, value1))
self.stack.append_op(RevertablePut(key2, value2))
def test_simplify(self):
key1 = ClaimToTXOPrefixRow.pack_key(b'\x01' * 20)
key2 = ClaimToTXOPrefixRow.pack_key(b'\x02' * 20)
key3 = ClaimToTXOPrefixRow.pack_key(b'\x03' * 20)
key4 = ClaimToTXOPrefixRow.pack_key(b'\x04' * 20)
val1 = ClaimToTXOPrefixRow.pack_value(1, 0, 1, 0, 1, False, 'derp')
val2 = ClaimToTXOPrefixRow.pack_value(1, 0, 1, 0, 1, False, 'oops')
val3 = ClaimToTXOPrefixRow.pack_value(1, 0, 1, 0, 1, False, 'other')
# check that we can't delete a non existent value
with self.assertRaises(OpStackIntegrity):
self.stack.append_op(RevertableDelete(key1, val1))
self.stack.append_op(RevertablePut(key1, val1))
self.assertEqual(1, len(self.stack))
self.stack.append_op(RevertableDelete(key1, val1))
self.assertEqual(0, len(self.stack))
self.stack.append_op(RevertablePut(key1, val1))
self.assertEqual(1, len(self.stack))
# try to delete the wrong value
with self.assertRaises(OpStackIntegrity):
self.stack.append_op(RevertableDelete(key2, val2))
self.stack.append_op(RevertableDelete(key1, val1))
self.assertEqual(0, len(self.stack))
self.stack.append_op(RevertablePut(key2, val3))
self.assertEqual(1, len(self.stack))
self.process_stack()
self.assertDictEqual({key2: val3}, self.fake_db)
# check that we can't put on top of the existing stored value
with self.assertRaises(OpStackIntegrity):
self.stack.append_op(RevertablePut(key2, val1))
self.assertEqual(0, len(self.stack))
self.stack.append_op(RevertableDelete(key2, val3))
self.assertEqual(1, len(self.stack))
self.stack.append_op(RevertablePut(key2, val3))
self.assertEqual(0, len(self.stack))
self.update(key2, val3, key2, val1)
self.assertEqual(2, len(self.stack))
self.process_stack()
self.assertDictEqual({key2: val1}, self.fake_db)
self.update(key2, val1, key2, val2)
self.assertEqual(2, len(self.stack))
self.update(key2, val2, key2, val3)
self.update(key2, val3, key2, val2)
self.update(key2, val2, key2, val3)
self.update(key2, val3, key2, val2)
with self.assertRaises(OpStackIntegrity):
self.update(key2, val3, key2, val2)
self.update(key2, val2, key2, val3)
self.assertEqual(2, len(self.stack))
self.stack.append_op(RevertableDelete(key2, val3))
self.process_stack()
self.assertDictEqual({}, self.fake_db)
self.stack.append_op(RevertablePut(key2, val3))
self.process_stack()
with self.assertRaises(OpStackIntegrity):
self.update(key2, val2, key2, val2)
self.update(key2, val3, key2, val2)
self.assertDictEqual({key2: val3}, self.fake_db)
undo = self.stack.get_undo_ops()
self.process_stack()
self.assertDictEqual({key2: val2}, self.fake_db)
self.stack.apply_packed_undo_ops(undo)
self.process_stack()
self.assertDictEqual({key2: val3}, self.fake_db)
class TestRevertablePrefixDB(unittest.TestCase):
def setUp(self):
self.tmp_dir = tempfile.mkdtemp()
self.db = HubDB(self.tmp_dir, cache_mb=1, max_open_files=32)
def tearDown(self) -> None:
self.db.close()
shutil.rmtree(self.tmp_dir)
def test_rollback(self):
name = 'derp'
claim_hash1 = 20 * b'\x00'
claim_hash2 = 20 * b'\x01'
claim_hash3 = 20 * b'\x02'
takeover_height = 10000000
self.assertIsNone(self.db.claim_takeover.get(name))
self.db.claim_takeover.stage_put((name,), (claim_hash1, takeover_height))
self.assertIsNone(self.db.claim_takeover.get(name))
self.assertEqual(10000000, self.db.claim_takeover.get_pending(name).height)
self.db.commit(10000000)
self.assertEqual(10000000, self.db.claim_takeover.get(name).height)
self.db.claim_takeover.stage_delete((name,), (claim_hash1, takeover_height))
self.db.claim_takeover.stage_put((name,), (claim_hash2, takeover_height + 1))
self.db.claim_takeover.stage_delete((name,), (claim_hash2, takeover_height + 1))
self.db.commit(10000001)
self.assertIsNone(self.db.claim_takeover.get(name))
self.db.claim_takeover.stage_put((name,), (claim_hash3, takeover_height + 2))
self.db.commit(10000002)
self.assertEqual(10000002, self.db.claim_takeover.get(name).height)
self.db.claim_takeover.stage_delete((name,), (claim_hash3, takeover_height + 2))
self.db.claim_takeover.stage_put((name,), (claim_hash2, takeover_height + 3))
self.db.commit(10000003)
self.assertEqual(10000003, self.db.claim_takeover.get(name).height)
self.db.rollback(10000003)
self.assertEqual(10000002, self.db.claim_takeover.get(name).height)
self.db.rollback(10000002)
self.assertIsNone(self.db.claim_takeover.get(name))
self.db.rollback(10000001)
self.assertEqual(10000000, self.db.claim_takeover.get(name).height)
self.db.rollback(10000000)
self.assertIsNone(self.db.claim_takeover.get(name))
+1 -5
View File
@@ -1,11 +1,7 @@
import asyncio
from binascii import hexlify
from lbry.testcase import AsyncioTestCase
from lbry.wallet import (
Wallet, Ledger, Database, Headers,
Account, SingleKey, HierarchicalDeterministic,
DeterministicChannelKeyManager
)
from lbry.wallet import Wallet, Ledger, Database, Headers, Account, SingleKey, HierarchicalDeterministic
class TestAccount(AsyncioTestCase):
+12 -12
View File
@@ -1,7 +1,7 @@
from binascii import unhexlify, hexlify
from lbry.testcase import AsyncioTestCase
from lbry.wallet.bip32 import PublicKey, PrivateKey, from_extended_key_string
from lbry.wallet.bip32 import PubKey, PrivateKey, from_extended_key_string
from lbry.wallet import Ledger, Database, Headers
from tests.unit.wallet.key_fixtures import expected_ids, expected_privkeys, expected_hardened_privkeys
@@ -11,24 +11,24 @@ class BIP32Tests(AsyncioTestCase):
def test_pubkey_validation(self):
with self.assertRaisesRegex(TypeError, 'chain code must be raw bytes'):
PublicKey(None, None, 1, None, None, None)
PubKey(None, None, 1, None, None, None)
with self.assertRaisesRegex(ValueError, 'invalid chain code'):
PublicKey(None, None, b'abcd', None, None, None)
PubKey(None, None, b'abcd', None, None, None)
with self.assertRaisesRegex(ValueError, 'invalid child number'):
PublicKey(None, None, b'abcd'*8, -1, None, None)
PubKey(None, None, b'abcd'*8, -1, None, None)
with self.assertRaisesRegex(ValueError, 'invalid depth'):
PublicKey(None, None, b'abcd'*8, 0, 256, None)
PubKey(None, None, b'abcd'*8, 0, 256, None)
with self.assertRaisesRegex(TypeError, 'pubkey must be raw bytes'):
PublicKey(None, None, b'abcd'*8, 0, 255, None)
PubKey(None, None, b'abcd'*8, 0, 255, None)
with self.assertRaisesRegex(ValueError, 'pubkey must be 33 bytes'):
PublicKey(None, b'abcd', b'abcd'*8, 0, 255, None)
PubKey(None, b'abcd', b'abcd'*8, 0, 255, None)
with self.assertRaisesRegex(ValueError, 'invalid pubkey prefix byte'):
PublicKey(
PubKey(
None,
unhexlify('33d1a3dc8155673bc1e2214fa493ccc82d57961b66054af9b6b653ac28eeef3ffe'),
b'abcd'*8, 0, 255, None
)
pubkey = PublicKey( # success
pubkey = PubKey( # success
None,
unhexlify('03d1a3dc8155673bc1e2214fa493ccc82d57961b66054af9b6b653ac28eeef3ffe'),
b'abcd'*8, 0, 1, None
@@ -37,7 +37,7 @@ class BIP32Tests(AsyncioTestCase):
pubkey.child(-1)
for i in range(20):
new_key = pubkey.child(i)
self.assertIsInstance(new_key, PublicKey)
self.assertIsInstance(new_key, PubKey)
self.assertEqual(hexlify(new_key.identifier()), expected_ids[i])
async def test_private_key_validation(self):
@@ -60,7 +60,7 @@ class BIP32Tests(AsyncioTestCase):
self.assertEqual(
ec_point[1], 86198965946979720220333266272536217633917099472454294641561154971209433250106
)
self.assertEqual('bUDcmraBp2zCV3QWmVVeQaEgepbs1b2gC9', private_key.address)
self.assertEqual('bUDcmraBp2zCV3QWmVVeQaEgepbs1b2gC9', private_key.address())
with self.assertRaisesRegex(ValueError, 'invalid BIP32 private key child number'):
private_key.child(-1)
self.assertIsInstance(private_key.child(PrivateKey.HARDENED), PrivateKey)
@@ -100,5 +100,5 @@ class BIP32Tests(AsyncioTestCase):
ledger,
'xpub661MyMwAqRbcF84AR8yfHoMzf4S2ct6mPJtvBtvNeyN9hBHuZ6uGJszkTSn5fQUCdz3XU17eBzFeAUwV6f'
'iW44g14WF52fYC5J483wqQ5ZP',
), PublicKey
), PubKey
)
+5 -111
View File
@@ -2,13 +2,10 @@ from binascii import unhexlify
from lbry.testcase import AsyncioTestCase
from lbry.wallet.constants import CENT, NULL_HASH32
from lbry.wallet.bip32 import PrivateKey, KeyPath
from lbry.wallet.mnemonic import Mnemonic
from lbry.wallet import Ledger, Database, Headers, Transaction, Input, Output
from lbry.schema.claim import Claim
from lbry.crypto.hash import sha256
def get_output(amount=CENT, pubkey_hash=NULL_HASH32):
return Transaction() \
.add_outputs([Output.pay_pubkey_hash(amount, pubkey_hash)]) \
@@ -24,11 +21,8 @@ def get_tx():
async def get_channel(claim_name='@foo'):
seed = Mnemonic.mnemonic_to_seed(Mnemonic().make_seed(), '')
key = PrivateKey.from_seed(Ledger, seed)
channel_key = key.child(KeyPath.CHANNEL).child(0)
channel_txo = Output.pay_claim_name_pubkey_hash(CENT, claim_name, Claim(), b'abc')
channel_txo.set_channel_private_key(channel_key)
await channel_txo.generate_channel_private_key()
get_tx().add_outputs([channel_txo])
return channel_txo
@@ -120,109 +114,6 @@ class TestValidatingOldSignatures(AsyncioTestCase):
self.assertTrue(stream.is_signed_by(channel, ledger))
def test_another_signed_claim_made_by_ytsync(self):
stream_tx = Transaction(unhexlify(
b'010000000185870fabdd6bd2d57749afebc0b239e8d0ebeb6f3647d6cfcabd5ea2200ac632010000006b4'
b'83045022100877c86de154e39f21959bc2157865071924adb7930a7a8910714f27398cd2689022074270f'
b'074ae260fff319d5e0c030691821bc75b82ff0179898ac3eaeda4123eb01210200328f7f001f22ea25d72'
b'ba37379e3065020c4d8371d9199dc4e3770084e26b9ffffffff0240420f0000000000fdcc05b527746865'
b'2d637269746963616c2d6e6565642d666f722d696e646570656e64656e742d6d656469614d85050191bba'
b'd064bdc455b9ebddeeb559686b13f027615384ec7c9d981c3c21a6e3d723a654e86bd707d21174c4f697f'
b'5080cf367a3b2dfc059e6cc14a962631df69b9886f4d8b97cb339b14633966fd5ac7d75edacdf30ac5010'
b'a90010a304af34d1c1467ebfc8785e2a49c7d5bec3cc6db94db858f1dcf95e4256564fba586d6e01f496d'
b'f2a34344e021d2725ffd12197468652d637269746963616c2d6e6565642d666f722e6d703418ee97eac10'
b'22209766964656f2f6d70343230ba13e6b667a9acef7e1b1caa88b9eb1d4680dea84b1d3e838266595805'
b'ab3343855c20af35012f942ce0d5111ce080331a1f436f7079726967687465642028636f6e74616374207'
b'075626c69736865722928e2e3c98d065a0908800f10b80818f314423954686520437269746963616c204e'
b'65656420666f7220496e646570656e64656e74204d65646961207c20476c656e6e20477265656e77616c6'
b'44af006496e636c7564657320616e20696e74726f64756374696f6e20627920546f6d20576f6f64732e20'
b'5265636f7264656420696e204c616b65204a61636b736f6e2c2054657861732c206f6e20446563656d626'
b'57220342c20323032312e0a0a526f6e205061756c27732074776f2063616d706169676e7320666f722070'
b'7265736964656e7420283230303820616e64203230313229207765726520776174657273686564206d6f6'
b'd656e747320666f72206c6962657274792d6d696e6465642070656f706c652061726f756e642074686520'
b'776f726c642e205468652022526f6e205061756c205265766f6c7574696f6e22e2809463656e746572656'
b'42061726f756e642068697320756e64696c75746564206d657373616765206f662070656163652c207072'
b'6f70657274792c20616e64206d61726b657473e280946368616e6765642074686520776179206d696c6c6'
b'96f6e732074686f756768742061626f75742074686520416d65726963616e20656d7069726520616e6420'
b'74686520416d65726963616e2066696e616e6369616c2073797374656d2e2044722e205061756c2773206'
b'66f637573206f6e2063656e7472616c2062616e6b696e6720616e6420666f726569676e20706f6c696379'
b'2063617567687420706f6c6974696369616e7320616e642070756e64697473206f66662067756172642c2'
b'0666f7263696e67207468656d20746f20736372616d626c6520666f72206578706c616e6174696f6e7320'
b'6f66206f7572204d6964646c65204561737420706f6c69637920616e6420536f766965742d7374796c652'
b'063656e7472616c20706c616e6e696e6720617420746865204665642e20506f6c697469637320696e2041'
b'6d657269636120686173206e6f74206265656e207468652073616d652073696e636520746865202247697'
b'56c69616e69206d6f6d656e742220616e642022456e6420746865204665642e222054686520526f6e2050'
b'61756c205265766f6c7574696f6e2077617320626f7468206120706f6c69746963616c20616e642063756'
b'c747572616c207068656e6f6d656e6f6e2e0a0a303a303020496e74726f64756374696f6e20627920546f'
b'6d20576f6f64730a343a323720476c656e6e20477265656e77616c640a2e2e2e0a68747470733a2f2f777'
b'7772e796f75747562652e636f6d2f77617463683f763d4e4b70706d52467673453052292a276874747073'
b'3a2f2f7468756d626e61696c732e6c6272792e636f6d2f4e4b70706d5246767345305a046e6577735a096'
b'3617468656472616c5a0f636f72706f72617465206d656469615a08637269746963616c5a0f676c656e6e'
b'20677265656e77616c645a0b696e646570656e64656e745a0a6a6f75726e616c69736d5a056d656469615'
b'a056d697365735a08706f6c69746963735a0a70726f706167616e64615a08726f6e207061756c5a057472'
b'757468620208016d7576a9140969964db5b5744e2d2d0de797f5904efc80d02188acc8814200000000001'
b'976a91439086597f9cfc066f4749b8bb245bf561714fda888ac00000000'
))
stream = stream_tx.outputs[0]
channel_tx = Transaction(unhexlify(
b'01000000011d47b91b409b317e427adb87ec4b0bfc9fad2abf6ec3296f41918e4b3cb9d4e7010000006a4'
b'7304402205e53ef7fc643ed00f0240dd1c3302b82141f481ed071cbcdd6b6ec6166ffd4e002203eb28ce6'
b'39f80253f66ff3bf45288a60133d7f5625217d1ecf3b57da440b559f012103b852d61074eb995b702a800'
b'f284e937ece4fea7f023beb70e6b0d1bff36d64b9ffffffff0240420f0000000000fdde01b506406d6973'
b'65734db801001299010a583056301006072a8648ce3d020106052b8104000a034200047ddb1d639d7bdd0'
b'953d9ab0bf9e971a632f85f9823c1d85780aa3e0a702b503c2962d00f67360e803514bf5864710925aacb'
b'effd9597532c7e60eb21b4e3fd03223d2a3b68747470733a2f2f7468756d626e61696c732e6c6272792e6'
b'36f6d2f62616e6e65722d55436d54362d43684b7061694956753266684549734e7451420a6d697365736d'
b'656469614ad401466561747572656420766964656f732066726f6d20746865204d6973657320496e73746'
b'9747574652e20546865204d6973657320496e737469747574652070726f6d6f7465732041757374726961'
b'6e2065636f6e6f6d6963732c2066726565646f6d2c20616e6420706561636520696e20746865206c69626'
b'572616c20696e74656c6c65637475616c20747261646974696f6e206f66204c756477696720766f6e204d'
b'69736573207468726f7567682072657365617263682c207075626c697368696e672c20616e64206564756'
b'36174696f6e2e52362a3468747470733a2f2f7468756d626e61696c732e6c6272792e636f6d2f55436d54'
b'362d43684b7061694956753266684549734e74516d7576a914cd77ded2400e6569f03a2580244bb395f95'
b'f91fc88ac344ab701000000001976a914cabdbfce726d2fda92ffe0041a4303f6c6c34cda88ac00000000'
))
channel = channel_tx.outputs[0]
ledger = Ledger({
'db': Database(':memory:'),
'headers': Headers(':memory:')
})
self.assertTrue(stream.is_signed_by(channel, ledger))
def test_claim_signed_using_ecdsa_validates_with_coincurve(self):
channel_tx = Transaction(unhexlify(
"0100000001b91d829283c0d80cb8113d5f36b6da3dfe9df3e783f158bfb3fd1b2b178d7fc9010000006b48"
"3045022100f4e2b4ee38388c3d3a62f4b12fdd413f6f140168e85884bbeb33a3f2d3159ef502201721200f"
"4a4f3b87484d4f47c9054e31cd3ba451dd3886a7f9f854893e7c8cf90121023f9e906e0c120f3bf74feb40"
"f01ddeafbeb1856d91938c3bef25bed06767247cffffffff0200e1f5050000000081b505406368616e4c5d"
"00125a0a583056301006072a8648ce3d020106052b8104000a03420004d7fa13fd8e57f3a0b878eaaf3d17"
"9144d25ddbe4a3e4440a661f51b4134c6a13c9c98678ff8411932e60fd97d7baf03ea67ebcc21097230cfb"
"2241348aadb55e6d7576a9149c6d700f89c77f0e8c650ba05656f8f2392782d388acf47c95350000000019"
"76a914d9502233e0e1fc76e13e36c546f704c3124d5eaa88ac00000000"
))
channel = channel_tx.outputs[0]
stream_tx = Transaction(unhexlify(
"010000000116a1d90763f2e3a2348c7fb438a23f232b15e3ffe3f058c3b2ab52c8bed8dcb5010000006b48"
"30450221008f38561b3a16944c63b4f4f1562f1efe1b2060f31d249e234003ee5e3461756f02205773c99e"
"83c968728e4f2433a13871c6ad23f6c10368ac52fa62a09f3f7ef5fd012102597f39845b98e2415b777aa0"
"3849d346d287af7970deb05f11214b3418ae9d82ffffffff0200e1f50500000000fd0c01b505636c61696d"
"4ce8012e6e40fa5fee1b915af3b55131dcbcebee34ab9148292b084ce3741f2e0db49783f3d854ac885f2b"
"6304a76ef7048046e338dd414ba4c64e8468651768ffaaf550c8560637ac8c477ea481ac2a9264097240f4"
"ab0a90010a8d010a3056bf5dbae43f77a63d075b0f2ae9c7c3e3098db93779c7f9840da0f4db9c2f8c8454"
"f4edd1373e2b64ee2e68350d916e120b746d706c69647879363171180322186170706c69636174696f6e2f"
"6f637465742d73747265616d3230f293f5acf4310562d4a41f6620167fe6d83761a98d36738908ce5c8776"
"1642710e55352a396276a42eda92ff5856f46f6d7576a91434bd3dc4c45cc0635eb2ad5da658727e5442ca"
"0f88ace82f902f000000001976a91427b27c89eaebf68d063c107241584c07e5a6ccc688ac00000000"
))
stream = stream_tx.outputs[0]
ledger = Ledger({'db': Database(':memory:'), 'headers': Headers(':memory:')})
self.assertTrue(stream.is_signed_by(channel, ledger))
class TestValidateSignContent(AsyncioTestCase):
@@ -230,10 +121,13 @@ class TestValidateSignContent(AsyncioTestCase):
some_content = "MEANINGLESS CONTENT AEE3353320".encode()
timestamp_str = "1630564175"
channel = await get_channel()
stream = get_stream()
signature = channel.sign_data(some_content, timestamp_str)
stream.signable.signature = unhexlify(signature.encode())
encoded_signature = stream.get_encoded_signature()
pieces = [timestamp_str.encode(), channel.claim_hash, some_content]
self.assertTrue(Output.is_signature_valid(
unhexlify(signature.encode()),
encoded_signature,
sha256(b''.join(pieces)),
channel.claim.channel.public_key_bytes
))
+3 -3
View File
@@ -130,12 +130,12 @@ class TestRedeemScriptHash(unittest.TestCase):
def redeem_script_hash(self, sigs, pubkeys):
# this checks that factory function correctly sets up the script
src1 = InputScript.redeem_multi_sig_script_hash(
src1 = InputScript.redeem_script_hash(
[unhexlify(sig) for sig in sigs],
[unhexlify(pubkey) for pubkey in pubkeys]
)
subscript1 = src1.values['script']
self.assertEqual(src1.template.name, 'script_hash+multi_sig')
self.assertEqual(src1.template.name, 'script_hash')
self.assertListEqual([hexlify(v) for v in src1.values['signatures']], sigs)
self.assertListEqual([hexlify(p) for p in subscript1.values['pubkeys']], pubkeys)
self.assertEqual(subscript1.values['signatures_count'], len(sigs))
@@ -143,7 +143,7 @@ class TestRedeemScriptHash(unittest.TestCase):
# now we test that it will round trip
src2 = InputScript(src1.source)
subscript2 = src2.values['script']
self.assertEqual(src2.template.name, 'script_hash+multi_sig')
self.assertEqual(src2.template.name, 'script_hash')
self.assertListEqual([hexlify(v) for v in src2.values['signatures']], sigs)
self.assertListEqual([hexlify(p) for p in subscript2.values['pubkeys']], pubkeys)
self.assertEqual(subscript2.values['signatures_count'], len(sigs))
+1 -42
View File
@@ -42,8 +42,7 @@ class TestSizeAndFeeEstimation(AsyncioTestCase):
async def asyncSetUp(self):
self.ledger = Ledger({
'db': Database(':memory:'),
'headers': Headers(':memory:'),
'fee_per_name_char': 200_000
'headers': Headers(':memory:')
})
await self.ledger.db.open()
@@ -263,46 +262,6 @@ class TestTransactionSerialization(unittest.TestCase):
tx._reset()
self.assertEqual(tx.raw, raw)
def test_redeem_scripthash_transaction(self):
raw = unhexlify(
"0200000001409223c2405238fdc516d4f2e8aa57637ce52d3b1ac42b26f1accdcda9697e79010000008a4"
"730440220033d5286f161da717d9d1bc3c2bc28da7636b38fc0c6aefb1e0864212f05282c02205df3ce13"
"5e79c76d44489212f77ad4e3a838562e601e6377704fa6206a6ae44f012102261773e7eebe9da80a5653d"
"865cc600362f8e7b2b598661139dd902b5b01ea101f03aaf30ab17576a914a3328f18ac1892a6667f713d"
"7020ff3437d973c888acfeffffff0180ed3e17000000001976a914353352b7ce1e3c9c05ffcd6ae97609d"
"e2999744488accdf50a00"
)
tx = Transaction(raw)
self.assertEqual(tx.id, 'e466881128889d1cc4110627753051c22e72a81d11229a1a1337da06940bebcf')
self.assertEqual(tx.version, 2)
self.assertEqual(tx.locktime, 718285,)
self.assertEqual(len(tx.inputs), 1)
self.assertEqual(len(tx.outputs), 1)
txin = tx.inputs[0]
self.assertEqual(
txin.txo_ref.id,
'797e69a9cdcdacf1262bc41a3b2de57c6357aae8f2d416c5fd385240c2239240:1'
)
self.assertEqual(txin.txo_ref.position, 1)
self.assertEqual(txin.sequence, 4294967294)
self.assertIsNone(txin.coinbase)
self.assertEqual(txin.script.template.name, 'script_hash+timelock')
self.assertEqual(
hexlify(txin.script.values['signature']),
b'30440220033d5286f161da717d9d1bc3c2bc28da7636b38fc0c6aefb1e0864212f'
b'05282c02205df3ce135e79c76d44489212f77ad4e3a838562e601e6377704fa620'
b'6a6ae44f01'
)
self.assertEqual(
hexlify(txin.script.values['pubkey']),
b'02261773e7eebe9da80a5653d865cc600362f8e7b2b598661139dd902b5b01ea10'
)
script = txin.script.values['script']
self.assertEqual(script.template.name, 'timelock')
self.assertEqual(script.values['height'], 717738)
self.assertEqual(hexlify(script.values['pubkey_hash']), b'a3328f18ac1892a6667f713d7020ff3437d973c8')
class TestTransactionSigning(AsyncioTestCase):
-60
View File
@@ -1,10 +1,6 @@
import json
import jsonschema
import os
import tempfile
from binascii import hexlify
import lbry.schema.types.v2 as schema_v2
from unittest import TestCase, mock
from lbry.testcase import AsyncioTestCase
from lbry.wallet import (
@@ -78,62 +74,6 @@ class TestWalletCreation(AsyncioTestCase):
decrypted = Wallet.unpack('password', encrypted)
self.assertEqual(decrypted['accounts'][0]['name'], 'An Account')
def test_wallet_file_schema(self):
wallet_dict = {
'version': 1,
'name': 'Main Wallet',
'preferences': {},
'accounts': [
{
'certificates': {'x': 'y'},
'name': 'Account 1',
'ledger': 'lbc_mainnet',
'modified_on': 123,
'seed':
"carbon smart garage balance margin twelve chest sword toast envelope bottom stomac"
"h absent",
'encrypted': False,
'private_key':
'xprv9s21ZrQH143K42ovpZygnjfHdAqSd9jo7zceDfPRogM7bkkoNVv7'
'DRNLEoB8HoirMgH969NrgL8jNzLEegqFzPRWM37GXd4uE8uuRkx4LAe',
'public_key':
'xpub661MyMwAqRbcGWtPvbWh9sc2BCfw2cTeVDYF23o3N1t6UZ5wv3EMm'
'Dgp66FxHuDtWdft3B5eL5xQtyzAtkdmhhC95gjRjLzSTdkho95asu9',
'address_generator': {
'name': 'deterministic-chain',
'receiving': {'gap': 17, 'maximum_uses_per_address': 3},
'change': {'gap': 10, 'maximum_uses_per_address': 3}
}
},
{
'certificates': {'a': 'b'},
'name': 'Account 2',
'ledger': 'lbc_mainnet',
'modified_on': 123,
'seed':
"carbon smart garage balance margin twelve chest sword toast envelope bottom stomac"
"h absent",
'encrypted': True,
'private_key':
'xprv9s21ZrQH143K42ovpZygnjfHdAqSd9jo7zceDfPRogM7bkkoNVv7'
'DRNLEoB8HoirMgH969NrgL8jNzLEegqFzPRWM37GXd4uE8uuRkx4LAe',
'public_key':
'xpub661MyMwAqRbcGWtPvbWh9sc2BCfw2cTeVDYF23o3N1t6UZ5wv3EMm'
'Dgp66FxHuDtWdft3B5eL5xQtyzAtkdmhhC95gjRjLzSTdkho95asu9',
'address_generator': {
'name': 'single-address',
}
},
]
}
storage = WalletStorage(default=wallet_dict)
wallet = Wallet.from_storage(storage, self.manager)
self.assertDictEqual(wallet_dict, wallet.to_dict())
with open(os.path.join(*schema_v2.__path__, 'wallet.json')) as f:
wallet_schema = json.load(f)
jsonschema.validate(schema=wallet_schema, instance=wallet.to_dict())
def test_no_password_but_encryption_preferred(self):
wallet_dict = {
'version': 1,
-2
View File
@@ -1,10 +1,8 @@
[testenv]
usedevelop = true
deps =
coverage
extras =
test
hub
torrent
changedir = {toxinidir}/tests
setenv =