Compare commits

...
Author SHA1 Message Date
Jack Robison 973ee4f08c delete unused code
ci / lint (push) Waiting to run
ci / tests / unit (macos-latest) (push) Waiting to run
ci / tests / unit (ubuntu-latest) (push) Waiting to run
ci / tests / unit (windows-latest) (push) Waiting to run
ci / tests / integration (blockchain) (push) Waiting to run
ci / tests / integration (claims) (push) Waiting to run
ci / tests / integration (claims_legacy_search) (push) Waiting to run
ci / tests / integration (datanetwork) (push) Waiting to run
ci / tests / integration (other) (push) Waiting to run
ci / tests / integration (takeovers) (push) Waiting to run
ci / tests / integration (takeovers_legacy_search) (push) Waiting to run
ci / tests / integration (transactions) (push) Waiting to run
2022-02-22 13:10:08 -05:00
Jack Robison b83360f3e2 executors 2022-02-22 13:10:08 -05:00
Jack Robison dac1b82ea7 fix tests 2022-02-22 13:10:08 -05:00
Jack Robison 56f80cbcda remove obsolete test_claim_search_as_reader_server 2022-02-22 13:10:08 -05:00
Jack Robison 28abd9c449 fix test teardown and setup 2022-02-22 13:10:08 -05:00
Jack Robison 8d86b0754c set_default_executor 2022-02-22 13:10:08 -05:00
Jack Robison bed3255b89 less noisy test log 2022-02-22 13:10:08 -05:00
Jack Robison a17a31acf5 remove unused arg 2022-02-22 13:10:08 -05:00
Jack Robison 1f815cf2d2 cleanup mempool.py 2022-02-22 13:10:08 -05:00
Jack Robison 07ee73b653 update transaction_get_height 2022-02-22 13:10:08 -05:00
Jack Robison 7a9e8c6769 fix mempool notification 2022-02-22 13:10:08 -05:00
Jack Robison a2901e4331 fix blocking_channel_ids and filtering_channel_ids cli args 2022-02-22 13:10:08 -05:00
Jack Robison 7f8268703c update prometheus 2022-02-22 13:10:08 -05:00
Jack Robison 32d2208fd9 logging 2022-02-22 13:10:08 -05:00
Jack Robison 704ec9e553 add --reindex option to lbry-hub-elastic-sync 2022-02-22 13:10:08 -05:00
Jack Robison e0f7066163 clean up claim producer 2022-02-22 13:10:08 -05:00
Jack Robison 32b26c9fa5 convert full scan iterators to range scans 2022-02-22 13:10:08 -05:00
Jack Robison 0d9d576436 add cache_size attribute to prefix classes to set the rocksdb lru cache size
-updates rocksdb column families to use custom sized `block_cache` (an lru cache) in a `BlockBasedTableFactory`
-lowers the default max open files to 64
2022-02-22 13:10:08 -05:00
Jack Robison e6c275f86e remove unused closed attribute 2022-02-22 13:10:08 -05:00
Jack Robison 937adbf439 add estimate_num_keys to prefix interface 2022-02-22 13:10:08 -05:00
Jack Robison 6a5ff0636c debug 2022-02-22 13:10:08 -05:00
Jack Robison 888d47f88b fix test 2022-02-22 13:10:08 -05:00
Jack Robison d7e50b269f fix test 2022-02-22 13:10:08 -05:00
Jack Robison 46ce175481 fix tests 2022-02-22 13:10:08 -05:00
Jack Robison 6b2d4175be fix reorg notifications 2022-02-22 13:10:08 -05:00
Jack Robison 16bfb8589b merge conflicts 2022-02-22 13:10:08 -05:00
Jack Robison a4bb4db8dd rename PrefixDB class 2022-02-22 13:10:08 -05:00
Jack Robison de1e2d0e3b fix column family init 2022-02-22 13:10:08 -05:00
Jack Robison a4880c1cf0 flush in advance/rollback methods 2022-02-22 13:10:08 -05:00
Jack Robison 28f25538a3 remove unused attributes 2022-02-22 13:10:08 -05:00
Jack Robison 83c8576b3f fix tests 2022-02-22 13:10:08 -05:00
Jack Robison 6fc909ea41 handle mempool race errors from lbcd 2022-02-22 13:10:08 -05:00
Jack Robison c17544d8ef fix mempool race condition in hub db writer 2022-02-22 13:10:08 -05:00
Jack Robison 7c46cc0805 fix lower bound 2022-02-22 13:10:08 -05:00
Jack Robison fb4dc8342a fix tests 2022-02-22 13:10:08 -05:00
Jack Robison df91f4754a comment out part of a resolve test until lbcwallet issue is fixed 2022-02-22 13:10:08 -05:00
Jack Robison 7d8bc38cb9 fix iterator lower bound 2022-02-22 13:10:08 -05:00
Jeffrey Picard a319595f37 Fix a test? 2022-02-22 13:10:08 -05:00
Jeffrey Picard 31312af517 Small changes and fixes for hub refactor 2022-02-22 13:10:08 -05:00
Jack Robison 95ec1f3af4 fix test 2022-02-22 13:10:08 -05:00
Jack Robison b093aa3911 fix another merge conflict 2022-02-22 13:10:08 -05:00
Jack Robison 7bd157ef17 fix more merge conflicts 2022-02-22 13:10:08 -05:00
Jack Robison 7f67cbfb40 fix merge conflict 2022-02-22 13:10:08 -05:00
Jeffrey Picard 2ea48bc8c2 skip segwit test 2022-02-22 13:09:57 -05:00
Jeffrey Picard 8d42b375a0 fix ssl connections and add new docker file 2022-02-22 13:09:57 -05:00
Lex Berezhny c2acceaed5 change VERBOSITY to INFO 2022-02-22 13:09:57 -05:00
Lex Berezhny a2db18010b rebasing fixes 2022-02-22 13:09:57 -05:00
Victor Shyba 545b7c33b1 bump lbcd to valid latest version 2022-02-22 13:09:57 -05:00
Brannon King 07d584133e get better error on RPC failure 2022-02-22 13:09:57 -05:00
Brannon King a10eb30771 fixing stalls in tests 2022-02-22 13:09:57 -05:00
Brannon King 98e264f4cd fix for the send-to-address timeout in asyncSetup 2022-02-22 13:09:57 -05:00
Roy Lee 083d6a3bc3 Update lbcd and lbcwallet versions 2022-02-22 13:09:57 -05:00
Brannon King 94e87f99d8 fixed some tests; made them not timeout waiting for tx confirmation 2022-02-22 13:09:57 -05:00
Roy Lee fc4114621c update lbcd to v0.22.100-beta-rc5 2022-02-22 13:09:57 -05:00
Victor Shyba 63bd6f5792 skip test_segwit while we take a look why lbcwallet rejects the tx 2022-02-22 13:09:57 -05:00
Victor Shyba 0c86ed924b ignore the config line warning 2022-02-22 13:09:57 -05:00
Victor Shyba 4a93b906d7 add comment for temporary lbcd edge case 2022-02-22 13:09:57 -05:00
Victor Shyba b727d2815f skip the test that uses claimname/updateclaim for now 2022-02-22 13:09:57 -05:00
Victor Shyba bad8ae7832 separate stdout/stderr from lbcctl on tests 2022-02-22 13:09:57 -05:00
Roy Lee cffe895d22 update lbcd and lbcwallet veresion 2022-02-22 13:09:57 -05:00
Brannon King ca4fec272a fixing a variety of broken tests, updated getclaimbyid usage 2022-02-22 13:09:57 -05:00
Roy Lee 66419f1aa6 test: lbcd don't have a dedicated mempool.dat 2022-02-22 13:09:57 -05:00
Roy Lee a13735769b test: update lbcd to avoid txn being rejected due to munimum fee 2022-02-22 13:09:57 -05:00
Roy Lee 749f72a8c5 HACK: temporary hack to move things along 2022-02-22 13:09:57 -05:00
Roy Lee e5f124fe68 add a rocksdb setup sanity check 2022-02-22 13:09:57 -05:00
Roy Lee 88ed67a5b3 test: migrate from lbrycrd to lbcd/lbcwallet 2022-02-22 13:09:57 -05:00
Roy Lee d5598462b6 test: support generatetoaddress RPC 2022-02-22 13:09:57 -05:00
Roy Lee 31c60e167a test: support walletpassphrase RPC 2022-02-22 13:09:57 -05:00
Roy Lee fe04bfa10a test: getnewaddress RPC takes account name 2022-02-22 13:09:57 -05:00
Roy Lee c15bedfb6d test: update some RPC arguments to int type 2022-02-22 13:09:57 -05:00
Roy Lee 0ff62495c7 test: setup loggers first 2022-02-22 13:09:57 -05:00
Roy Lee b4be712a50 Update log level from WARN to INFO 2022-02-22 13:09:57 -05:00
Jack Robison faa43fc20e use rocksdb instead of leveldb
-dont use block processor directly from session manager
2022-02-22 13:09:57 -05:00
Jack Robison f7deaa3303 remove print from test, uncomment a test assert 2022-02-22 13:09:57 -05:00
Jack Robison d7ecde7040 drop ES_MODE hub setting, rename hub entry points
`lbry-hub-writer` - keeps an up-to-date database
`lbry-hub-server` - replies to clients with data from a read only snapshot of the db
`lbry-hub-elastic-sync` - watches for changes to the db and updates elasticsearch accordingly
2022-02-22 13:09:57 -05:00
Jack Robison e2a75758f8 delete unused code 2022-02-22 13:09:57 -05:00
Jack Robison f449cf61ab update mempool in thread 2022-02-22 13:09:57 -05:00
Jack Robison 04db81e954 fix tests 2022-02-22 13:09:57 -05:00
Jack Robison b49c9fd050 fix filtering/blocking 2022-02-22 13:09:57 -05:00
Jack Robison 3ff2bcf913 pylint 2022-02-22 13:09:57 -05:00
Jack Robison 556056c60d tests 2022-02-22 13:09:57 -05:00
Jack Robison 3c03fff380 logging, update lbry-rocksdb requirement 2022-02-22 13:09:57 -05:00
Jack Robison b1441d4247 wait for writing to finish before closing the db 2022-02-22 13:09:57 -05:00
Jack Robison 81458b75e4 fix unwinding readers during reorg 2022-02-22 13:09:57 -05:00
Jack Robison f0f8ef044b reduce hub writer polling delay 2022-02-22 13:09:57 -05:00
Jack Robison 47305e7446 executors 2022-02-22 13:09:57 -05:00
Jack Robison ba6b985d71 cleanup debug logging 2022-02-22 13:09:57 -05:00
Jack Robison 49802b39cb rocksdb column families 2022-02-22 13:09:57 -05:00
Jack Robison 46bcc5d725 executors for each reader, fix shutdown 2022-02-22 13:09:57 -05:00
Jack Robison 98f8fd0556 tests 2022-02-22 13:09:57 -05:00
Jack Robison c0ce27ccf3 es sync notifier 2022-02-22 13:09:57 -05:00
Jack Robison cfae30a364 reader/writer move from cli 2022-02-22 13:09:57 -05:00
Jack Robison 53e3828965 remove redundant undo cleanup, don't delete historic touched_or_deleted 2022-02-22 13:09:57 -05:00
Jack Robison 4f16f1c829 imports, impove errors 2022-02-22 13:09:57 -05:00
Jack Robison 478bd0510b es writer 2022-02-22 13:09:57 -05:00
Jack Robison 499ee74dfc add chain reader and reader server, new mempool, update block processor 2022-02-22 13:09:57 -05:00
Jack Robison d3da442727 update undo key to include the block hash 2022-02-22 13:09:57 -05:00
Jack Robison 358fa21eaf move Prefetcher to own file 2022-02-22 13:09:57 -05:00
Jack Robison 20f35d02fa move HubDB, delete leveldb.py 2022-02-22 13:09:57 -05:00
Jack Robison 77e64ef028 add mempool, trending, and touched address indexes to the hub db 2022-02-22 13:09:57 -05:00
Jack Robison 0a71e2ff91 use rocksdb instead of leveldb
-dont use block processor directly from session manager
2022-02-22 13:09:57 -05:00
Victor Shyba dd5b9ca81b add migrator to set head blobs should_announce=0 2022-02-20 22:33:57 -03:00
Victor Shyba 89ed04f8a7 fix test_announces 2022-02-20 22:33:57 -03:00
Victor Shyba ec0d9f06c5 do not search for the head blob 2022-02-20 22:33:57 -03:00
Victor Shyba 03b59ac6fc dont set head blob to announce on save 2022-02-20 22:33:57 -03:00
Victor Shyba 43ac3336d7 break tie by length 2022-02-20 22:24:04 -03:00
Victor Shyba d12c78db74 fix and test case for blob_clean after disabling network storage 2022-02-20 22:24:04 -03:00
Jack Robison bfaf1b0957 Merge pull request #3564 from lbryio/fix_downloader_losing_peers
fix handling re-adding lost peers during download
2022-02-16 11:55:22 -05:00
Victor Shyba bb60c385d5 put back all the peers, get rid of re_add 2022-02-08 21:41:52 -03:00
Alex Grin c96d1d9c32 Merge pull request #3537 from lbryio/repost_update 2022-02-08 12:20:20 -05:00
Alex Grintsvayg 7c7a0d4bdf let stream_update work on non-stream claims 2022-02-08 09:28:17 -05:00
Lex Berezhny cc829a7bf4 Merge pull request #3558 from lbryio/jeffreypicard-patch-1
Update __init__.py
2022-02-04 12:36:01 -05:00
Jeffrey Picard e0ea6383e2 Update __init__.py
Update go hub binary to fix es sync test.
2022-02-04 12:17:19 -05:00
Lex Berezhny bcec5dc2ae Merge pull request #3556 from lbryio/txo_dust_prevention
prevent creation of change which is below the dust threshold of 1000 dewies
2022-02-04 12:08:16 -05:00
Lex Berezhny cba9c16a06 fix 2022-02-04 12:07:41 -05:00
Lex Berezhny dd68fb077b prevent creation of change which is below the dust threshold of 1000 dewies 2022-02-04 12:07:41 -05:00
Jack Robison c2294e97db Merge pull request #3552 from lbryio/bump_dht_cache
Increase DHT peer manager cache size to 16384
2022-02-04 11:59:19 -05:00
Victor Shyba c0f512ace7 bump DHT peer manager cache to 16384 2022-02-02 16:54:42 -03:00
Lex Berezhny 3305eb67c6 Merge pull request #3548 from lbryio/announce_metrics
Add optional Prometheus metrics for DHT announcements
2022-02-02 11:06:48 -05:00
Victor Shyba c9d637b4da add gauge for queue size 2022-02-02 11:56:42 -03:00
Victor Shyba ae3e8fadf5 count announcements and how many peers we were able to announce to 2022-02-02 11:56:42 -03:00
Lex Berezhny a1abd94387 Merge pull request #3542 from eug3nix/gh_3481_file_type_detection
file type detection now looks inside the file to determine the type, in addition to using the file extension
2022-01-31 10:29:47 -05:00
Eugene Dubinin 9b463a8cab adds tests for guess_media_type
removes unnecessary comments
2022-01-29 20:49:42 +02:00
Eugene Dubinin babc54a240 adjusts code style 2022-01-29 15:25:17 +02:00
Eugene Dubinin 5836a93b21 fixes KeyError on missing synonyms 2022-01-29 15:25:17 +02:00
Eugene Dubinin 557348e345 detect media_type from the file contents 2022-01-29 15:25:17 +02:00
Lex Berezhny 9adfec6b00 Merge pull request #3549 from lbryio/wallet_lock_w_deterministic_channels
wallet locking/unlocking no longer breaks deterministic channel keys
2022-01-26 11:17:55 -05:00
Lex Berezhny 3a496902f8 wallet locking/unlocking no longer breaks deterministic channel keys 2022-01-24 09:45:08 -05:00
Lex Berezhny b5ead91746 Merge pull request #3534 from lbryio/normalize_signatures
drop dependency on cryptography library in wallet module
2022-01-17 13:38:20 -05:00
Lex Berezhny 302461b446 updated based on code review 2022-01-17 11:08:28 -05:00
Lex Berezhny ac201c718e drop dependency on cryptography library in wallet module 2022-01-17 10:43:59 -05:00
Jack Robison f78e3825ca Merge pull request #3500 from lbryio/fix_script
Add Prometheus metrics for DHT internals
2022-01-14 12:46:28 -05:00
Victor Shyba 0618053bd4 remove request_flight metric 2022-01-12 12:41:04 -03:00
Victor Shyba 8e6fa3490c disable CSV endpoints by default 2022-01-12 12:39:23 -03:00
Victor Shyba 8a1a1a4000 remove estimation endpoints as that is done over prometheus metrics now 2022-01-12 12:39:23 -03:00
Victor Shyba fd9dcbf9a8 add granular metric for stored blob prefix, for network announcements calculation 2022-01-12 12:39:23 -03:00
Victor Shyba beb8583436 change colliding bits metric to gauge 2022-01-12 12:39:23 -03:00
Victor Shyba b44e2c0b38 count bit collisions between 8 and 16 2022-01-12 12:39:23 -03:00
Victor Shyba 06e94640b5 add counter for peers with colliding bytes 2022-01-12 12:39:23 -03:00
Victor Shyba ff36bdc802 add requests in flight and error 2022-01-12 12:39:23 -03:00
Victor Shyba 46f576de46 add request received 2022-01-12 12:39:23 -03:00
Victor Shyba 7b09c34fce add request_sent and request_time metric on dht 2022-01-12 12:39:23 -03:00
Victor Shyba a22f50aa84 add storing_peers and peer_manager_keys 2022-01-12 12:39:23 -03:00
Victor Shyba 2d9130b4e0 prometheus: move blobs_stored and peers to SDK. add buckets_in_routing_table 2022-01-12 12:39:23 -03:00
Victor Shyba 470ee72462 add passive estimation to prometheus 2022-01-12 12:39:23 -03:00
Victor Shyba add147b409 fix missing async 2022-01-12 12:39:23 -03:00
Victor Shyba 371df6e6c2 keep same node id between runs 2022-01-12 12:39:23 -03:00
Victor Shyba 7ed5fe8f66 add semaphore on active estimation to avoid abuse 2022-01-12 12:39:23 -03:00
Victor Shyba a6ca7a6f38 same api across different estimation methods 2022-01-12 12:39:23 -03:00
Victor Shyba 1c857b8dd8 be explicit about ignoring params 2022-01-12 12:39:23 -03:00
Victor Shyba 87ff3f95ff better endpoint names, small docs 2022-01-12 12:39:23 -03:00
Victor Shyba 5cb4c06d0c add prefix_neighbors_count to routing table debug api 2022-01-12 12:39:23 -03:00
Jack Robison e7d9079389 improve script 2022-01-12 12:39:23 -03:00
Victor Shyba 9cdcff0e1e first attempt at crawling 2022-01-12 12:39:23 -03:00
Lex Berezhny a4dce8cf9f Merge pull request #3535 from vertbyqb/hexdata-string
convert hexdata argument to a string before signing in `channel_sign` command
2022-01-10 09:48:41 -05:00
Lex Berezhny aaa11c02bf added integration test 2022-01-10 08:46:10 -05:00
vertbyqb d2ebbf5db6 jsonrpc_channel_sign - Convert hexdata to a string before signing
Fixes #3533
2022-01-10 08:46:10 -05:00
Jack Robison e6efc1ad4a Merge pull request #3538 from lbryio/dht_memory
Unify and fix DHT memory caches for peer manager
2022-01-07 11:29:44 -05:00
Victor Shyba a8523996a9 extract cache values, increase peer cache to 2048 2022-01-07 12:58:52 -03:00
Victor Shyba f586de2bbe DHT bugfix: failures tracking should be bound to 2048 LRU cache size 2022-01-07 12:46:00 -03:00
Victor Shyba 7df02303b2 fix missing docopt argument 2022-01-05 17:10:31 -03:00
Victor Shyba f89c75e642 bump hub version to latest supporting sd_hash search 2022-01-05 17:10:31 -03:00
Victor Shyba d2c1961101 update hub protobuf including sd_hash field 2022-01-05 17:10:31 -03:00
Victor Shyba 2a4c5a48bf increase indexed sd_hash prefix to 4 chars 2022-01-05 17:10:31 -03:00
Victor Shyba 5f5f39a4aa enable and test prefix search for sd hash 2022-01-05 17:10:31 -03:00
Victor Shyba df54cc04af sync and search sd_hash 2022-01-05 17:10:31 -03:00
Victor Shyba 0439616480 add test 2022-01-05 17:10:31 -03:00
Victor Shyba 19fa274227 add sd hash to API 2022-01-05 17:10:31 -03:00
Lex Berezhny 8076000c27 Merge pull request #3450 from lbryio/deterministic_channel_keys
deterministic channel keys (requires wallet server re-sync)
2021-12-23 15:38:15 -05:00
Lex Berezhny c80b30f070 test another signed claim by ytsync 2021-12-22 18:29:46 -05:00
Lex Berezhny 486d5c48b0 takeover tests fix 2021-12-22 18:29:46 -05:00
Lex Berezhny 4822792ee2 create nondetermnistic channel in test to replicate old test behavior 2021-12-22 18:29:46 -05:00
Lex Berezhny 569f1d42b1 fix tests 2021-12-22 18:29:46 -05:00
Lex Berezhny 23c10faff5 lint 2021-12-22 18:29:46 -05:00
Lex Berezhny 1eaa195363 reduced crypto dependency in wallet to coincurve 2021-12-22 18:29:46 -05:00
Lex Berezhny fb57cfa5d8 moved imports for lint 2021-12-22 18:29:46 -05:00
Lex Berezhny d33086c8f7 deleted extraneous test 2021-12-22 18:29:46 -05:00
Lex Berezhny d815a6f02c use ecdsa for signing/veryfing instead of coincurve due to compatibility issues 2021-12-22 18:29:46 -05:00
Lex Berezhny 8216f4a873 work in progress 2021-12-22 18:29:46 -05:00
Lex Berezhny e4cc4521d9 channel key generation no longer arbitrarily bounded 2021-12-22 18:29:46 -05:00
Lex Berezhny 6bd9b3744d progress, channel keys generate deterministically now 2021-12-22 18:29:46 -05:00
Lex Berezhny f741b00768 progress on deterministic channel keys 2021-12-22 18:29:46 -05:00
Lex Berezhny 5eb95d7dd4 Merge pull request #3529 from lbryio/change_default_coin_selection_strategy
changes default coin selection strategy from standard to prefer_confirmed
2021-12-22 11:28:22 -05:00
Lex Berezhny e5268f43e7 changes default coin selection strategy from standard to prefer_confirmed 2021-12-21 10:22:09 -05:00
Victor Shyba 54d6fb9da4 do not limit DHT results by K, respect max_results 2021-12-09 14:34:55 -03:00
Victor Shyba 3d5c9cc1c2 clarify DHT debug logging on key and operation 2021-12-09 14:32:30 -03:00
Alex Grin 442326f1d8 Merge pull request #3499 from lbryio/multiple-release-time-constraints
@jeffreypicard assures me these timeouts are safe to ignore
2021-12-06 11:43:46 -05:00
Jeffrey Picard d66f46e07b Switch RangeField back to ints 2021-12-03 18:12:38 -05:00
Jeffrey Picard 757b53443d Try forcing tox reset 2021-12-03 17:42:56 -05:00
Jeffrey Picard 3436965b33 Debugging 2021-12-03 17:22:52 -05:00
Jeffrey Picard df71132957 Update es version in workflow 2021-12-03 13:03:00 -05:00
Jeffrey Picard 1b322dc404 Update protobufs, go hub shim, and claim test. 2021-12-03 13:03:00 -05:00
Jack Robison 58341f4ff1 remove unused ES fields 2021-12-03 13:03:00 -05:00
Jack Robison 0d3ca80008 support lists of constraints for all range fields 2021-12-03 13:03:00 -05:00
Lex Berezhny 63437712cd Merge pull request #3490 from ghost/integration_test_setup_cleanup_timeouts
added timeout of async operations to integration test setup/teardown
2021-12-02 19:52:44 -05:00
Jack Robison 26d0e87f46 v0.106.0
ci / lint (push) Waiting to run
ci / tests / unit (macos-latest) (push) Waiting to run
ci / tests / unit (ubuntu-latest) (push) Waiting to run
ci / tests / unit (windows-latest) (push) Waiting to run
ci / tests / integration (blockchain) (push) Waiting to run
ci / tests / integration (claims) (push) Waiting to run
ci / tests / integration (claims_legacy_search) (push) Waiting to run
ci / tests / integration (datanetwork) (push) Waiting to run
ci / tests / integration (other) (push) Waiting to run
ci / tests / integration (takeovers) (push) Waiting to run
ci / tests / integration (takeovers_legacy_search) (push) Waiting to run
ci / tests / integration (transactions) (push) Waiting to run
2021-12-02 17:17:00 -05:00
Jack Robison 2cad4fa1ce update json docs 2021-12-02 14:51:52 -05:00
Jack Robison 7bb293e5d6 update claim_search doc
backward compatibility for `trending_mixed`, `trending_local`, `trending_global`, and `trending_group` args to `claim_search`
2021-12-02 14:51:52 -05:00
Lex Berezhny e4777f9314 Merge branch 'master' into integration_test_setup_cleanup_timeouts 2021-12-01 22:08:18 -05:00
Jack Robison 3508f562a7 update json docs 2021-12-01 18:47:03 -05:00
Jack Robison 1aa66c6038 update header checkpoints 2021-12-01 18:46:24 -05:00
Victor Shyba e7458edb72 test case for stream_type search on claims missing source + fix 2021-12-01 18:42:47 -05:00
Lex Berezhny 7f97013703 Merge pull request #3497 from lbryio/fee_per_name_env_var
fee per name env var
2021-12-01 11:26:00 -05:00
Lex Berezhny 9e43060d41 fee per name env var 2021-12-01 10:22:34 -05:00
FemtosecondLaser d69486fb6e returned conditional check in add_timeout() as it was making test_node.py tests unhappy 2021-11-30 01:01:35 +00:00
FemtosecondLaser d4ebfdbc3c removed conditional check in add_timeout() 2021-11-29 22:56:50 +00:00
FemtosecondLaser e00c3db71a Merge branch 'master' into integration_test_setup_cleanup_timeouts 2021-11-29 21:50:05 +00:00
Victor Shyba 11c3ea0b87 fix typo from arg name 2021-11-24 13:05:43 -03:00
Jack Robison 7531401623 keep touched_or_deleted records 2021-11-21 13:52:03 -05:00
FemtosecondLaser e6c1dc251e changed addTimeout to add_timeout for lint compliance 2021-11-20 00:47:46 +00:00
FemtosecondLaser dca7977051 added timeout of async operations to integration test setup/teardown 2021-11-20 00:22:25 +00:00
Victor Shyba d19e07d661 add blob endpoint for listing announced blobs 2021-11-17 13:27:19 -03:00
Victor Shyba 751ff6e21f add /peers.csv to monitoring endpoint 2021-11-17 13:27:19 -03:00
Brendon J. Brewer 3f6fe995b8 Rename trending 2021-11-16 10:59:10 -05:00
Jack Robison 1e00fb369d fix missing es notification for support amount changing 2021-11-15 00:58:18 -05:00
Jack Robison 54b522383a improve tests 2021-11-15 00:58:18 -05:00
Jack Robison 90a7de3b5c improve resolve tests 2021-11-15 00:58:18 -05:00
Jack Robison 3fe1582432 fix duplicate trending notification to ES 2021-11-15 00:58:18 -05:00
Jack Robison 85eddd2100 fix effective amount for resolve/ES being off while claims/supports are unactivated 2021-11-15 00:58:18 -05:00
Jack Robison f5f8775c59 fix test_colliding_short_id 2021-11-10 13:02:28 -03:00
Jack Robison 0ca98678f7 update default tcp/blob port to be the same as the default udp/dht port (4444) 2021-11-10 13:02:28 -03:00
Victor Shyba a19060c08d log unexpected errors, rename task/loop 2021-11-09 14:27:06 -05:00
Victor Shyba fa2ad88cc4 clear cache on test assertions 2021-11-09 14:27:06 -05:00
Victor Shyba 63cbcd0956 make sure the downloader always stops gracefully 2021-11-09 14:27:06 -05:00
Victor Shyba d6d0ebf8f4 cache space stats from running components so status is instant 2021-11-09 14:27:06 -05:00
Victor Shyba 0d810d92ca add index for blob table so size summaries are faster 2021-11-09 14:27:06 -05:00
Victor Shyba 1ff914a6f4 download from stored announcements and dont reannounce 2021-11-09 14:27:06 -05:00
Victor Shyba 5959b1be72 improve disk space manager status, include more info and unify space queries 2021-11-09 14:27:06 -05:00
Victor Shyba d12a214c05 normal_blobs->stream_blobs, proactive->background 2021-11-09 14:27:06 -05:00
Victor Shyba 3a83052f2e fix free space calculation, test it and give a margin of 10mb before starting so it doesnt insist when full 2021-11-09 14:27:06 -05:00
Victor Shyba 510b44ca92 move more logic out of the downloader component 2021-11-09 14:27:06 -05:00
Victor Shyba 15edb6756d extract background downloader to its own class 2021-11-09 14:27:06 -05:00
Victor Shyba fbfd02b08b add analytics event for network disk space 2021-11-09 14:27:06 -05:00
Victor Shyba b39c26fc86 announce orphan blobs manually, as that was done when save stream 2021-11-09 14:27:06 -05:00
Victor Shyba 95b2c8d175 cleanup background downloader blobs from conf 2021-11-09 14:27:06 -05:00
Victor Shyba d52748b09f separated network seeding space metrics 2021-11-09 14:27:06 -05:00
Victor Shyba 34d18a3a9a don't save streams for network blobs and bypass disk space manager 2021-11-09 14:27:06 -05:00
Victor Shyba 3b27d6a9b5 add conf for network seeding space limit 2021-11-09 14:27:06 -05:00
Victor Shyba 703c391f99 schedule the download task instead 2021-11-09 14:27:06 -05:00
Victor Shyba 4f1dc29df1 fix unit tests from component dependency chain changes 2021-11-09 14:27:06 -05:00
Victor Shyba 13667df374 download from DHT 2021-11-09 14:27:06 -05:00
Victor Shyba 8800d6985f drop channel support, prepare to hook into DHT 2021-11-09 14:27:06 -05:00
Victor Shyba 364b8f2605 handle case where something that isn't a sd blob gets hit 2021-11-09 14:27:06 -05:00
Victor Shyba 67b9ea9deb no api yet 2021-11-09 14:27:06 -05:00
Victor Shyba b78f2336a7 download only blobs 2021-11-09 14:27:06 -05:00
Victor Shyba c7ba637c7d fix tests 2021-11-09 14:27:06 -05:00
Victor Shyba 23a5ce3df7 fix exception arguments 2021-11-09 14:27:06 -05:00
Victor Shyba 8f88e28e50 test add/remove/list subscriptions 2021-11-09 14:27:06 -05:00
Victor Shyba 9cf6139557 fix and test main api 2021-11-09 14:27:06 -05:00
Victor Shyba d556065a8b download all blobs and check that on tests 2021-11-09 14:27:06 -05:00
Victor Shyba 951716f7dc create downloader component and initial tests 2021-11-09 14:27:06 -05:00
Victor Shyba 1ddc7ddda3 with the fix we no longer need to restart the stream 2021-11-08 10:50:47 -05:00
Victor Shyba 903ed9f3dc fix tests by checking there are actual blobs being deleted 2021-11-08 10:50:47 -05:00
Victor Shyba c42b76dcb8 dont lose results on duplicates, just warn 2021-11-08 10:50:47 -05:00
Victor Shyba a73582d9ae remove tried_for_this_blob so banned peers are retried for same blob 2021-11-08 10:50:47 -05:00
Cristian Vicas 42c4fc7557 Bug [#2070] where blob_get RPC timed out.
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.

Bug [#2070] where blob_get RPC timed out.

Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
2021-11-08 10:49:48 -05:00
Jack Robison ddbbb6f1dd use mempool cache in transaction_get_batch 2021-10-27 20:19:08 -04:00
Lex Berezhny ff21a92330 Merge pull request #3457 from FemtosecondLaser/feature/3270-check-default-download-dir-writable
Modified ensure_directory_exists() to check if the directory is writable by the process.
2021-10-27 11:00:13 -04:00
FemtosecondLaser 07f76f7ad1 Added an integration test covering the following scenario:
On start, if download dir is non-writable - daemon terminates with a helpful message.
2021-10-26 11:17:52 +01:00
Jack Robison c90ccffd7b Update docker-compose-wallet-server.yml 2021-10-25 14:20:39 -04:00
Jack Robison a00d5f18af add script to setup docker volumes from snapshots 2021-10-24 16:25:34 -04:00
Jack Robison 1e391d211b fix attempting to update trending on abandoned claims 2021-10-23 18:39:04 -04:00
FemtosecondLaser d87f9672fa Improved the readability of the tests. 2021-10-23 13:12:49 +01:00
FemtosecondLaser 2b5838aa01 Changed the tests to execute against a real file system instead of a fake one. 2021-10-23 02:52:58 +01:00
Jack Robison e10486d6ec update docs 2021-10-22 16:51:59 -04:00
Jack Robison 1a74d6604d skip loading tx/claim caches in the elastic sync script when not needed 2021-10-22 15:10:35 -04:00
Alex Grin 6d118536b6 Merge pull request #3460 from lbryio/dht_seed_script_metrics 2021-10-22 12:44:16 -04:00
Alex Grin ca4d758db9 Merge branch 'master' into dht_seed_script_metrics 2021-10-22 11:54:19 -04:00
Victor Shyba dc18c26aa4 add optional prometheus to dht_node script 2021-10-22 03:39:46 -03:00
Jack Robison 48505c2968 update trending with help from @eggplantbren 2021-10-21 00:17:12 -04:00
Jack Robison a98ea1e66a update sync script to handle ES falling behind leveldb on shutdown 2021-10-20 23:41:11 -04:00
Jack Robison 3dec697816 logging 2021-10-20 23:41:11 -04:00
Jack Robison 88fd41e597 update docker 2021-10-20 23:41:11 -04:00
Jack Robison b05d071a1c update Env to accept parameters from cli args 2021-10-20 23:41:11 -04:00
Jack Robison a27d3b9689 set default CACHE_MB to 1024mb and the default QUERY_TIMEOUT_MS to 10s 2021-10-20 23:41:11 -04:00
Jack Robison 1facc0cd01 remove unused hub env settings 2021-10-20 23:41:11 -04:00
FemtosecondLaser 837f91d830 renamed the test class to be more specific about the sut 2021-10-21 00:31:02 +01:00
FemtosecondLaser 9c5f5aefb0 removed redundant tests
renamed a test to be more specific about the kind of the precondition
2021-10-21 00:27:31 +01:00
FemtosecondLaser 6b8d4a444b Modified ensure_directory_exists() to check if the directory is writable by the process. 2021-10-20 15:26:16 +01:00
Jack Robison 6bef09a3b1 update lbry-hub-elastic-sync to support resyncing recent blocks 2021-10-19 15:53:20 -04:00
Jack Robison e35319e5a2 add CACHE_ALL_CLAIM_TXOS hub setting 2021-10-19 15:53:20 -04:00
Jack Robison 0e548b3812 remove dead code 2021-10-19 15:53:20 -04:00
Jack Robison bfac02ccab add CACHE_ALL_TX_HASHES setting to optionally use more memory to save i/o 2021-10-19 15:53:20 -04:00
Jack Robison 7ea1a2b361 sleeps 2021-10-19 15:53:20 -04:00
Jack Robison 99df418f1d improve resolve caching 2021-10-19 15:53:20 -04:00
Jack Robison 6416d8ce9c threadpools for block processor and es sync reader 2021-10-19 15:53:20 -04:00
Jack Robison 22b43a2b01 doc strings 2021-10-19 15:53:20 -04:00
Jack Robison 05e5d24c5e improve claims_producer performance 2021-10-19 15:53:20 -04:00
Jack Robison eabcc30367 resolve lru cache 2021-10-19 15:53:20 -04:00
Jack Robison f5e0ef5223 add block_txs index 2021-10-19 15:53:20 -04:00
Jack Robison f46d9330b0 smaller caches 2021-10-19 15:53:20 -04:00
Jack Robison b62a0b4607 Update daemon.py
docstring
2021-10-15 09:40:15 -04:00
Cristian Vicas 1f044321fb Updated documentation for RPC calls: status, blob_list. 2021-10-15 09:40:15 -04:00
Jack Robison a841d49483 Merge branch 'belikor-fix-wrong-url' 2021-10-15 09:00:59 -04:00
belikor 9509acc490 file_manager: raise new InvalidStreamURLError if the URL is invalid
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.

For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```

```
Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```

Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING  lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING  lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```

This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
2021-10-15 08:59:37 -04:00
111 changed files with 7177 additions and 3874 deletions

No files matched your search

+3 -1
View File
@@ -94,7 +94,7 @@ jobs:
- name: Runs Elasticsearch
uses: elastic/elastic-github-actions/elasticsearch@master
with:
stack-version: 7.6.0
stack-version: 7.12.1
- uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
@@ -110,6 +110,8 @@ jobs:
key: tox-integration-${{ matrix.test }}-${{ hashFiles('setup.py') }}
restore-keys: txo-integration-${{ matrix.test }}-
- run: pip install tox coverage coveralls
- if: matrix.test == 'claims'
run: rm -rf .tox
- run: tox -e ${{ matrix.test }}
- name: submit coverage report
env:
+1 -1
View File
@@ -15,11 +15,11 @@ RUN apt-get update && \
build-essential \
automake libtool \
pkg-config \
libleveldb-dev \
python3.7 \
python3-dev \
python3-pip \
python3-wheel \
python3-cffi \
python3-setuptools && \
update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1 && \
rm -rf /var/lib/apt/lists/*
+48
View File
@@ -0,0 +1,48 @@
# FROM debian:10-slim
FROM python:3.7.12-slim-buster
ARG user=lbry
ARG db_dir=/database
ARG projects_dir=/home/$user
ARG DOCKER_TAG
ARG DOCKER_COMMIT=docker
ENV DOCKER_TAG=$DOCKER_TAG DOCKER_COMMIT=$DOCKER_COMMIT
RUN apt-get update && \
apt-get -y --no-install-recommends install \
wget \
tar unzip \
build-essential \
automake libtool \
pkg-config
RUN pip install uvloop
RUN groupadd -g 999 $user && useradd -m -u 999 -g $user $user
RUN mkdir -p $db_dir
RUN chown -R $user:$user $db_dir
COPY . $projects_dir
RUN chown -R $user:$user $projects_dir
USER $user
WORKDIR $projects_dir
RUN make install
RUN python3 docker/set_build.py
RUN rm ~/.cache -rf
# entry point
ARG host=0.0.0.0
ARG tcp_port=50001
ARG daemon_url=http://lbry:lbry@192.99.151.178:9245/
VOLUME $db_dir
ENV TCP_PORT=$tcp_port
ENV HOST=$host
ENV DAEMON_URL=$daemon_url
ENV DB_DIRECTORY=$db_dir
ENV MAX_SESSIONS=1000000000
ENV MAX_SEND=1000000000000000000
ENV EVENT_LOOP_POLICY=uvloop
COPY ./docker/wallet_server_entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
+11 -7
View File
@@ -18,23 +18,27 @@ services:
- "wallet_server:/database"
environment:
- DAEMON_URL=http://lbry:lbry@127.0.0.1:9245
- MAX_QUERY_WORKERS=4
- CACHE_MB=1024
- CACHE_ALL_TX_HASHES=
- CACHE_ALL_CLAIM_TXOS=
- MAX_SEND=1000000000000000000
- MAX_RECEIVE=1000000000000000000
- MAX_SESSIONS=100000
- HOST=0.0.0.0
- TCP_PORT=50001
- PROMETHEUS_PORT=2112
- QUERY_TIMEOUT_MS=3000 # how long search queries allowed to run before cancelling, in milliseconds
- TRENDING_ALGORITHMS=variable_decay
- MAX_SEND=10000000000000 # deprecated. leave it high until its removed
- MAX_SUBS=1000000000000 # deprecated. leave it high until its removed
- FILTERING_CHANNEL_IDS=770bd7ecba84fd2f7607fb15aedd2b172c2e153f 95e5db68a3101df19763f3a5182e4b12ba393ee8
- BLOCKING_CHANNEL_IDS=dd687b357950f6f271999971f43c785e8067c3a9 06871aa438032244202840ec59a469b303257cad b4a2528f436eca1bf3bf3e10ff3f98c57bd6c4c6 e4e230b131082f6b10c8f7994bbb83f29e8e6fb9
- BLOCKING_CHANNEL_IDS=dd687b357950f6f271999971f43c785e8067c3a9 06871aa438032244202840ec59a469b303257cad b4a2528f436eca1bf3bf3e10ff3f98c57bd6c4c6
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.11.0
container_name: es01
environment:
- node.name=es01
- discovery.type=single-node
- indices.query.bool.max_clause_count=4096
- indices.query.bool.max_clause_count=8192
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms8g -Xmx8g" # no more than 32, remember to disable swap
- "ES_JAVA_OPTS=-Xms4g -Xmx4g" # no more than 32, remember to disable swap
ulimits:
memlock:
soft: -1
+12 -4
View File
@@ -6,7 +6,7 @@ set -euo pipefail
SNAPSHOT_URL="${SNAPSHOT_URL:-}" #off by default. latest snapshot at https://lbry.com/snapshot/wallet
if [[ -n "$SNAPSHOT_URL" ]] && [[ ! -f /database/claims.db ]]; then
if [[ -n "$SNAPSHOT_URL" ]] && [[ ! -f /database/lbry-leveldb ]]; then
files="$(ls)"
echo "Downloading wallet snapshot from $SNAPSHOT_URL"
wget --no-verbose --trust-server-names --content-disposition "$SNAPSHOT_URL"
@@ -20,6 +20,14 @@ if [[ -n "$SNAPSHOT_URL" ]] && [[ ! -f /database/claims.db ]]; then
rm "$filename"
fi
/home/lbry/.local/bin/lbry-hub-elastic-sync /database/claims.db
echo 'starting server'
/home/lbry/.local/bin/lbry-hub "$@"
if [ -z "$HUB_COMMAND" ]; then
echo "HUB_COMMAND env variable must be writer, reader, or es_sync"
exit 1
fi
case "$HUB_COMMAND" in
writer ) /home/lbry/.local/bin/lbry-hub-writer "$@" ;;
reader ) /home/lbry/.local/bin/lbry-hub-server "$@" ;;
es_sync ) /home/lbry/.local/bin/lbry-hub-elastic-sync ;;
* ) "HUB_COMMAND env variable must be writer, reader, or es_sync" && exit 1 ;;
esac
+123 -117
View File
@@ -54,10 +54,10 @@
"examples": [
{
"title": "Get a file",
"curl": "curl -d'{\"method\": \"get\", \"params\": {\"uri\": \"astream#138446450b727414a2b9e9529341dc621a9e5470\"}}' http://localhost:5279/",
"lbrynet": "lbrynet get astream#138446450b727414a2b9e9529341dc621a9e5470",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"get\", \"params\": {\"uri\": \"astream#138446450b727414a2b9e9529341dc621a9e5470\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"added_on\": 1630567653,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": 4,\n \"content_fee\": null,\n \"download_directory\": \"/tmp/tmpbzob8k7r\",\n \"download_path\": \"/tmp/tmpbzob8k7r/tmpb146pf1q\",\n \"file_name\": \"tmpb146pf1q\",\n \"height\": 214,\n \"is_fully_reflected\": false,\n \"key\": \"5ff5a0e1491f8a7b1ccba093b2963f9e\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0170f30befb340325fd001b041c188e57c598d5f81dd671a785932239cd26885251126637352f3825ab44b6e2c2cef378b182f870b1f9050035fac0bb1956acf02bf696de46bc2526f2881f5b98c371975290eb8b00a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706231343670663171180b22186170706c69636174696f6e2f6f637465742d73747265616d32305b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"77bf955798d4e67da2f877931216008c79f97a5788167e8bbb239b5f94ff13508c40c18604f29220efa7c79756f1a2ff\",\n \"stream_name\": \"tmpb146pf1q\",\n \"streaming_url\": \"http://localhost:5280/stream/5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"suggested_file_name\": \"tmpb146pf1q\",\n \"timestamp\": 1630567677,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 11\n }\n}"
"curl": "curl -d'{\"method\": \"get\", \"params\": {\"uri\": \"astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\"}}' http://localhost:5279/",
"lbrynet": "lbrynet get astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"get\", \"params\": {\"uri\": \"astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"added_on\": 1638473043,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": 4,\n \"content_fee\": null,\n \"download_directory\": \"/tmp/tmpd_du30j6\",\n \"download_path\": \"/tmp/tmpd_du30j6/tmpj5wn5xs8\",\n \"file_name\": \"tmpj5wn5xs8\",\n \"height\": 214,\n \"is_fully_reflected\": false,\n \"key\": \"6f57dcfb5e64cc5172c07929ac8720b6\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0194e79e0212eb6cfabd4ce2da08e899eb3567d7ad7d593e2e84647ec1122144433db0a4321c77eb6e1276c12b6934064b07547cbfadd7a9aecd5fc53460eb27274cfd8914aad2a302c2c483511e46fc0473c886050a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706a35776e35787338180b22186170706c69636174696f6e2f6f637465742d73747265616d3230945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"3c18d82c0c262331785e3e6c45dd05f6fe4f6918bb08fe552772e230fefe2cb079cf171c9ba8255be068fda898119d23\",\n \"stream_name\": \"tmpj5wn5xs8\",\n \"streaming_url\": \"http://localhost:5280/stream/945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"suggested_file_name\": \"tmpj5wn5xs8\",\n \"timestamp\": 1638473068,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 11\n }\n}"
}
]
},
@@ -268,10 +268,10 @@
"examples": [
{
"title": "Publish a file",
"curl": "curl -d'{\"method\": \"publish\", \"params\": {\"name\": \"a-new-stream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmp03r7_9ca\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet publish a-new-stream --bid=1.0 --file_path=/tmp/tmp03r7_9ca",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"publish\", \"params\": {\"name\": \"a-new-stream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmp03r7_9ca\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001b765c49b00e0a553b516a41e94bcf6c38abfc705ed63736cf0c0e561698f2454010000006b483045022100c4f1270fc5a84e8a6f0bffad6440aa1b23b90e83d970b6cc60cf2398b2c762ae0220120a1226520a2d66843caa3b7c0a1d2779a8e5a98d069274b1e5c0fd8cbf3b9101210260bd481afc7f3eb2632d36de0ce39e50886e0682a7acb658c5a091a8cffb6270ffffffff0200e1f50500000000bfb50c612d6e65772d73747265616d4c94000a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d70303372375f396361180b22186170706c69636174696f6e2f6f637465742d73747265616d3230f9b1769c0e89beda641ec305c1065103a5b9d4dbf8262813431628e9703a2fe7542bc00d4edae17cb51c902b63b019e16d7576a914cd674ff84ce88a3671d5d9a946f883497d50c3aa88ac54826311000000001976a91427cf3fd6f52023fbf8776551641cf0ca8c449f4788ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mtm2gHnthSQ2n61dnJSeZCd873eHByQQ8r\",\n \"amount\": \"3.941448\",\n \"confirmations\": 3,\n \"height\": 217,\n \"nout\": 1,\n \"timestamp\": 1630567677,\n \"txid\": \"54248f6961e5c0f06c7363ed05c7bf8ac3f6bc941ea416b553a5e0009bc465b7\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mzF2aecxjjueNYMHUCipNAzZ7NNAtLv4Nc\",\n \"amount\": \"1.0\",\n \"claim_id\": \"75d6cf604b30fa292603bf2b3686d1bfd3f149cf\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"a-new-stream\",\n \"normalized_name\": \"a-new-stream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://a-new-stream#75d6cf604b30fa292603bf2b3686d1bfd3f149cf\",\n \"timestamp\": null,\n \"txid\": \"06c1465e2e68fda3d6efb5d6e209e07d3ee1dd2e3ad18d49bc926c3e241ff934\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmp03r7_9ca\",\n \"sd_hash\": \"f9b1769c0e89beda641ec305c1065103a5b9d4dbf8262813431628e9703a2fe7542bc00d4edae17cb51c902b63b019e1\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mj9SvUTwe82GnSov4HupzJSK12zx8qVZcM\",\n \"amount\": \"2.917341\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"06c1465e2e68fda3d6efb5d6e209e07d3ee1dd2e3ad18d49bc926c3e241ff934\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.024107\",\n \"total_input\": \"3.941448\",\n \"total_output\": \"3.917341\",\n \"txid\": \"06c1465e2e68fda3d6efb5d6e209e07d3ee1dd2e3ad18d49bc926c3e241ff934\"\n }\n}"
"curl": "curl -d'{\"method\": \"publish\", \"params\": {\"name\": \"a-new-stream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpm_ue2xqj\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet publish a-new-stream --bid=1.0 --file_path=/tmp/tmpm_ue2xqj",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"publish\", \"params\": {\"name\": \"a-new-stream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpm_ue2xqj\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"010000000141c20275b977773f8dcfa1dfd119b5ac7c7f505cd46105176089fcea5e35396a010000006b483045022100ac5957cdb6c737dd710f772f7b067661ef71f6c2c06f53612c45355be690459e022055a277e73c432b971c784ab0160cd07c0cea7f0d1e68fa550203351aee924c35012102f6383203e8b47a2fb7770538b3ff9f1c09311a63550d5e6d439f64d056cd2132ffffffff0200e1f50500000000bfb50c612d6e65772d73747265616d4c94000a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706d5f75653278716a180b22186170706c69636174696f6e2f6f637465742d73747265616d32305215cc76e174cf861023113e476a5d30003dca680b41a38ac5cc5479740ac271c2ed44d692270c4a56c07f9c240d48456d7576a914e0e67df181a2b2995c4d12044a81de25f95da67688ac54826311000000001976a9148ad256c1d7946dd3070879dcbb4817ab49bb86e288ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n2GSzoTb1HBSkeDzaF1g8TGZtvSqsFCxKF\",\n \"amount\": \"3.941448\",\n \"confirmations\": 3,\n \"height\": 217,\n \"nout\": 1,\n \"timestamp\": 1638473068,\n \"txid\": \"6a39355eeafc8960170561d45c507f7cacb519d1dfa1cf8d3f7777b97502c241\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n227mtNq6cMKGJEE2ojp8XqYzhp3X9fNbE\",\n \"amount\": \"1.0\",\n \"claim_id\": \"4a7102a7df73e74c0c662b79fb28a811341be3d4\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"a-new-stream\",\n \"normalized_name\": \"a-new-stream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://a-new-stream#4a7102a7df73e74c0c662b79fb28a811341be3d4\",\n \"timestamp\": null,\n \"txid\": \"89dc960e1af78d16a446410fdc79f93b812cfad73ac91d2f4f1c2c3876015af3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpm_ue2xqj\",\n \"sd_hash\": \"5215cc76e174cf861023113e476a5d30003dca680b41a38ac5cc5479740ac271c2ed44d692270c4a56c07f9c240d4845\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mtAyX8T8pZa87mxGLbyPt2Q8YoWcW27zKh\",\n \"amount\": \"2.917341\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"89dc960e1af78d16a446410fdc79f93b812cfad73ac91d2f4f1c2c3876015af3\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.024107\",\n \"total_input\": \"3.941448\",\n \"total_output\": \"3.917341\",\n \"txid\": \"89dc960e1af78d16a446410fdc79f93b812cfad73ac91d2f4f1c2c3876015af3\"\n }\n}"
}
]
},
@@ -332,10 +332,10 @@
"examples": [
{
"title": "Resolve a claim",
"curl": "curl -d'{\"method\": \"resolve\", \"params\": {\"urls\": [\"astream#138446450b727414a2b9e9529341dc621a9e5470\"], \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"include_sent_supports\": false, \"include_sent_tips\": false, \"include_received_tips\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet resolve astream#138446450b727414a2b9e9529341dc621a9e5470",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"resolve\", \"params\": {\"urls\": [\"astream#138446450b727414a2b9e9529341dc621a9e5470\"], \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"include_sent_supports\": false, \"include_sent_tips\": false, \"include_received_tips\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"astream#138446450b727414a2b9e9529341dc621a9e5470\": {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8/astream#1\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 213,\n \"creation_height\": 213,\n \"creation_timestamp\": 1630567677,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263187,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"short_url\": \"lbry://astream#1\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 8,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 209,\n \"claims_in_channel\": 2,\n \"creation_height\": 209,\n \"creation_timestamp\": 1630567676,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263183,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"short_url\": \"lbry://@channel#8\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n }\n}"
"curl": "curl -d'{\"method\": \"resolve\", \"params\": {\"urls\": [\"astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\"], \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"include_sent_supports\": false, \"include_sent_tips\": false, \"include_received_tips\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet resolve astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"resolve\", \"params\": {\"urls\": [\"astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\"], \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"include_sent_supports\": false, \"include_sent_tips\": false, \"include_received_tips\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\": {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a/astream#f\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 214,\n \"creation_height\": 213,\n \"creation_timestamp\": 1638473068,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 714,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"short_url\": \"lbry://astream#f\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 8,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 210,\n \"claims_in_channel\": 2,\n \"creation_height\": 209,\n \"creation_timestamp\": 1638473067,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 710,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"short_url\": \"lbry://@channel#a\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n }\n}"
}
]
},
@@ -350,14 +350,14 @@
"name": "status",
"description": "Get daemon status",
"arguments": [],
"returns": "(dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'database': (bool),\n 'dht': (bool),\n 'exchange_rate_manager': (bool),\n 'hash_announcer': (bool),\n 'peer_protocol_server': (bool),\n 'file_manager': (bool),\n 'libtorrent_component': (bool),\n 'upnp': (bool),\n 'wallet': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'connected': (str) host and port of the connected spv server,\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n 'connected_servers': (list) [\n {\n 'host': (str) server hostname,\n 'port': (int) server port,\n 'latency': (int) milliseconds\n }\n ],\n },\n 'libtorrent_component': {\n 'running': (bool) libtorrent was detected and started successfully,\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n 'connections': {\n 'incoming_bps': {\n <source ip and tcp port>: (int) bytes per second received,\n },\n 'outgoing_bps': {\n <destination ip and tcp port>: (int) bytes per second sent,\n },\n 'total_outgoing_mps': (float) megabytes per second sent,\n 'total_incoming_mps': (float) megabytes per second received,\n 'time': (float) timestamp\n }\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the stream manager,\n },\n 'upnp': {\n 'aioupnp_version': (str),\n 'redirects': {\n <TCP | UDP>: (int) external_port,\n },\n 'gateway': (str) manufacturer and model,\n 'dht_redirect_set': (bool),\n 'peer_redirect_set': (bool),\n 'external_ip': (str) external ip address,\n }\n }",
"returns": "(dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'database': (bool),\n 'dht': (bool),\n 'exchange_rate_manager': (bool),\n 'hash_announcer': (bool),\n 'peer_protocol_server': (bool),\n 'file_manager': (bool),\n 'libtorrent_component': (bool),\n 'upnp': (bool),\n 'wallet': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'connected': (str) host and port of the connected spv server,\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n 'connected_servers': (list) [\n {\n 'host': (str) server hostname,\n 'port': (int) server port,\n 'latency': (int) milliseconds\n }\n ],\n },\n 'libtorrent_component': {\n 'running': (bool) libtorrent was detected and started successfully,\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n 'connections': {\n 'incoming_bps': {\n <source ip and tcp port>: (int) bytes per second received,\n },\n 'outgoing_bps': {\n <destination ip and tcp port>: (int) bytes per second sent,\n },\n 'total_outgoing_mps': (float) megabytes per second sent,\n 'total_incoming_mps': (float) megabytes per second received,\n 'max_outgoing_mbs': (float) maximum bandwidth (megabytes per second) sent, since the\n daemon was started\n 'max_incoming_mbs': (float) maximum bandwidth (megabytes per second) received, since the\n daemon was started\n 'total_sent' : (int) total number of bytes sent since the daemon was started\n 'total_received' : (int) total number of bytes received since the daemon was started\n }\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the stream manager,\n },\n 'upnp': {\n 'aioupnp_version': (str),\n 'redirects': {\n <TCP | UDP>: (int) external_port,\n },\n 'gateway': (str) manufacturer and model,\n 'dht_redirect_set': (bool),\n 'peer_redirect_set': (bool),\n 'external_ip': (str) external ip address,\n }\n }",
"examples": [
{
"title": "Get status",
"curl": "curl -d'{\"method\": \"status\", \"params\": {}}' http://localhost:5279/",
"lbrynet": "lbrynet status",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"status\", \"params\": {}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"blob_manager\": {\n \"connections\": {\n \"incoming_bps\": {},\n \"max_incoming_mbs\": 0.0,\n \"max_outgoing_mbs\": 0.0,\n \"outgoing_bps\": {},\n \"total_incoming_mbs\": 0.0,\n \"total_outgoing_mbs\": 0.0,\n \"total_received\": 0,\n \"total_sent\": 0\n },\n \"finished_blobs\": 0\n },\n \"disk_space\": {\n \"running\": true,\n \"space_used\": \"0\"\n },\n \"ffmpeg_status\": {\n \"analyze_audio_volume\": true,\n \"available\": true,\n \"which\": \"/usr/bin/ffmpeg\"\n },\n \"file_manager\": {\n \"managed_files\": 0\n },\n \"installation_id\": \"sefddTD9i1NVkZqNcjEFbL6PRRGmBXUfdGe3he54UKrFjrLUepzqGwtVbY5rXhw2j\",\n \"is_running\": true,\n \"skipped_components\": [\n \"dht\",\n \"upnp\",\n \"hash_announcer\",\n \"peer_protocol_server\",\n \"libtorrent_component\"\n ],\n \"startup_status\": {\n \"blob_manager\": true,\n \"database\": true,\n \"disk_space\": true,\n \"exchange_rate_manager\": true,\n \"file_manager\": true,\n \"wallet\": true,\n \"wallet_server_payments\": true\n },\n \"wallet\": {\n \"available_servers\": 1,\n \"best_blockhash\": \"1af1c53a3386a4b30141b631129c83c7a6ae052e587fb3b8c717d603810600eb\",\n \"blocks\": 206,\n \"blocks_behind\": 0,\n \"connected\": \"localhost:50002\",\n \"connected_features\": {\n \"daily_fee\": \"0\",\n \"description\": \"\",\n \"donation_address\": \"\",\n \"genesis_hash\": \"6e3fcf1299d4ec5d79c3a4c91d624a4acf9e2e173d95a1a0504f677669687556\",\n \"hash_function\": \"sha256\",\n \"hosts\": {},\n \"payment_address\": \"\",\n \"protocol_max\": \"0.199.0\",\n \"protocol_min\": \"0.54.0\",\n \"pruning\": null,\n \"server_version\": \"0.102.0\",\n \"trending_algorithm\": \"zscore\"\n },\n \"headers_synchronization_progress\": 100,\n \"known_servers\": 0,\n \"servers\": [\n {\n \"availability\": true,\n \"host\": \"localhost\",\n \"latency\": 0.004395185000248603,\n \"port\": 50002\n }\n ]\n },\n \"wallet_server_payments\": {\n \"max_fee\": \"0.0\",\n \"running\": false\n }\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"background_downloader\": {\n \"available_free_space_mb\": null,\n \"ongoing_download\": false,\n \"running\": false\n },\n \"blob_manager\": {\n \"connections\": {\n \"incoming_bps\": {},\n \"max_incoming_mbs\": 0.0,\n \"max_outgoing_mbs\": 0.0,\n \"outgoing_bps\": {},\n \"total_incoming_mbs\": 0.0,\n \"total_outgoing_mbs\": 0.0,\n \"total_received\": 0,\n \"total_sent\": 0\n },\n \"finished_blobs\": 0\n },\n \"disk_space\": {\n \"content_blobs_storage_used_mb\": 0,\n \"published_blobs_storage_used_mb\": 0,\n \"running\": true,\n \"seed_blobs_storage_used_mb\": 0,\n \"total_used_mb\": 0\n },\n \"ffmpeg_status\": {\n \"analyze_audio_volume\": true,\n \"available\": true,\n \"which\": \"/usr/bin/ffmpeg\"\n },\n \"file_manager\": {\n \"managed_files\": 0\n },\n \"installation_id\": \"8NS7BSLdxv6fx5wVHLt3r6y7ncyWmQBakPrQjJPXWKADjNnU19Vu3719zFUQUfdbd2\",\n \"is_running\": true,\n \"skipped_components\": [\n \"dht\",\n \"upnp\",\n \"hash_announcer\",\n \"peer_protocol_server\",\n \"libtorrent_component\"\n ],\n \"startup_status\": {\n \"background_downloader\": true,\n \"blob_manager\": true,\n \"database\": true,\n \"disk_space\": true,\n \"exchange_rate_manager\": true,\n \"file_manager\": true,\n \"wallet\": true,\n \"wallet_server_payments\": true\n },\n \"wallet\": {\n \"available_servers\": 1,\n \"best_blockhash\": \"1d169da3f63ef171c5fda2deef043b7b0bbaee9f1dd50eb2e78642053de32a52\",\n \"blocks\": 206,\n \"blocks_behind\": 0,\n \"connected\": \"localhost:50002\",\n \"connected_features\": {\n \"daily_fee\": \"0\",\n \"description\": \"\",\n \"donation_address\": \"\",\n \"genesis_hash\": \"6e3fcf1299d4ec5d79c3a4c91d624a4acf9e2e173d95a1a0504f677669687556\",\n \"hash_function\": \"sha256\",\n \"hosts\": {},\n \"payment_address\": \"\",\n \"protocol_max\": \"0.199.0\",\n \"protocol_min\": \"0.54.0\",\n \"pruning\": null,\n \"server_version\": \"0.105.0\",\n \"trending_algorithm\": \"fast_ar\"\n },\n \"headers_synchronization_progress\": 100,\n \"known_servers\": 0,\n \"servers\": [\n {\n \"availability\": true,\n \"host\": \"localhost\",\n \"latency\": 0.002393823000602424,\n \"port\": 50002\n }\n ]\n },\n \"wallet_server_payments\": {\n \"max_fee\": \"0.0\",\n \"running\": false\n }\n }\n}"
}
]
},
@@ -379,7 +379,7 @@
"curl": "curl -d'{\"method\": \"version\", \"params\": {}}' http://localhost:5279/",
"lbrynet": "lbrynet version",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"version\", \"params\": {}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"build\": \"dev\",\n \"desktop\": \"ubuntu:GNOME\",\n \"distro\": {\n \"codename\": \"focal\",\n \"id\": \"ubuntu\",\n \"like\": \"debian\",\n \"version\": \"20.04\",\n \"version_parts\": {\n \"build_number\": \"\",\n \"major\": \"20\",\n \"minor\": \"04\"\n }\n },\n \"lbrynet_version\": \"0.102.0\",\n \"os_release\": \"5.11.0-27-generic\",\n \"os_system\": \"Linux\",\n \"platform\": \"Linux-5.11.0-27-generic-x86_64-with-debian-bullseye-sid\",\n \"processor\": \"x86_64\",\n \"python_version\": \"3.7.10\",\n \"version\": \"0.102.0\"\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"build\": \"dev\",\n \"desktop\": \"ubuntu:GNOME\",\n \"distro\": {\n \"codename\": \"bionic\",\n \"id\": \"ubuntu\",\n \"like\": \"debian\",\n \"version\": \"18.04\",\n \"version_parts\": {\n \"build_number\": \"\",\n \"major\": \"18\",\n \"minor\": \"04\"\n }\n },\n \"lbrynet_version\": \"0.105.0\",\n \"os_release\": \"4.15.15-041515-generic\",\n \"os_system\": \"Linux\",\n \"platform\": \"Linux-4.15.15-041515-generic-x86_64-with-debian-buster-sid\",\n \"processor\": \"x86_64\",\n \"python_version\": \"3.7.5\",\n \"version\": \"0.105.0\"\n }\n}"
}
]
}
@@ -433,10 +433,10 @@
"examples": [
{
"title": "Add an account from seed",
"curl": "curl -d'{\"method\": \"account_add\", \"params\": {\"account_name\": \"new account\", \"seed\": \"boost property since welcome armed arch goddess correct stick traffic pyramid glory\", \"single_key\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet account add \"new account\" --seed=\"boost property since welcome armed arch goddess correct stick traffic pyramid glory\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_add\", \"params\": {\"account_name\": \"new account\", \"seed\": \"boost property since welcome armed arch goddess correct stick traffic pyramid glory\", \"single_key\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1630567643,\n \"name\": \"new account\",\n \"private_key\": \"tprv8ZgxMBicQKsPduGkPH4FUHQh6vCVc5PuD2t5148Qf3ds9oJ2b6MpGzKK6BD8QVWeYfkFPAqrEvtFXckYM7p83XaQGyaCiHFAi3hdUnfScwC\",\n \"public_key\": \"tpubD6NzVbkrYhZ4XNJYGviqsh4ofwiRmQaonLUrHaAi5KSFzHYoDVBQTUwBGMQsNMBh3cvEAmp6uotNWBXa2kUJ3GiFHLyWEc7k1YtDkF7jGQW\",\n \"seed\": \"boost property since welcome armed arch goddess correct stick traffic pyramid glory\"\n }\n}"
"curl": "curl -d'{\"method\": \"account_add\", \"params\": {\"account_name\": \"new account\", \"seed\": \"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\", \"single_key\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet account add \"new account\" --seed=\"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_add\", \"params\": {\"account_name\": \"new account\", \"seed\": \"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\", \"single_key\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1638473034,\n \"name\": \"new account\",\n \"private_key\": \"tprv8ZgxMBicQKsPf3EwVBH3vG4NWPJQk4xBTC1qi1uU45s7PREzdMBdyytxxgNA5kYXmgZiTDL3TgWTU3mEgGnzncSTqJGiMZCH7J9wGo8HftD\",\n \"public_key\": \"tpubD6NzVbkrYhZ4YWGjNpweKfiV5QpLuQ962VcczXwmUMfWDuVmFk1EAUWq8pywR7wShpYF5eX3uQx1XmwZYVrmRVHDRS4yTuCHe76BCgvMSCo\",\n \"seed\": \"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\"\n }\n}"
}
]
},
@@ -474,9 +474,9 @@
},
{
"title": "Get balance for specific account by id",
"curl": "curl -d'{\"method\": \"account_balance\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"}}' http://localhost:5279/",
"lbrynet": "lbrynet account balance \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_balance\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"}}).json()",
"curl": "curl -d'{\"method\": \"account_balance\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"}}' http://localhost:5279/",
"lbrynet": "lbrynet account balance \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_balance\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"available\": \"2.0\",\n \"reserved\": \"0.0\",\n \"reserved_subtotals\": {\n \"claims\": \"0.0\",\n \"supports\": \"0.0\",\n \"tips\": \"0.0\"\n },\n \"total\": \"2.0\"\n }\n}"
}
]
@@ -511,7 +511,7 @@
"curl": "curl -d'{\"method\": \"account_create\", \"params\": {\"account_name\": \"generated account\", \"single_key\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet account create \"generated account\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_create\", \"params\": {\"account_name\": \"generated account\", \"single_key\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1630567643,\n \"name\": \"generated account\",\n \"private_key\": \"tprv8ZgxMBicQKsPduGkPH4FUHQh6vCVc5PuD2t5148Qf3ds9oJ2b6MpGzKK6BD8QVWeYfkFPAqrEvtFXckYM7p83XaQGyaCiHFAi3hdUnfScwC\",\n \"public_key\": \"tpubD6NzVbkrYhZ4XNJYGviqsh4ofwiRmQaonLUrHaAi5KSFzHYoDVBQTUwBGMQsNMBh3cvEAmp6uotNWBXa2kUJ3GiFHLyWEc7k1YtDkF7jGQW\",\n \"seed\": \"boost property since welcome armed arch goddess correct stick traffic pyramid glory\"\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1638473033,\n \"name\": \"generated account\",\n \"private_key\": \"tprv8ZgxMBicQKsPf3EwVBH3vG4NWPJQk4xBTC1qi1uU45s7PREzdMBdyytxxgNA5kYXmgZiTDL3TgWTU3mEgGnzncSTqJGiMZCH7J9wGo8HftD\",\n \"public_key\": \"tpubD6NzVbkrYhZ4YWGjNpweKfiV5QpLuQ962VcczXwmUMfWDuVmFk1EAUWq8pywR7wShpYF5eX3uQx1XmwZYVrmRVHDRS4yTuCHe76BCgvMSCo\",\n \"seed\": \"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\"\n }\n}"
}
]
},
@@ -566,24 +566,24 @@
"examples": [
{
"title": "Transfer 2 LBC from default account to specific account",
"curl": "curl -d'{\"method\": \"account_fund\", \"params\": {\"to_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"amount\": \"2.0\", \"everything\": false, \"broadcast\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet account fund --to_account=\"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\" --amount=2.0 --broadcast",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_fund\", \"params\": {\"to_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"amount\": \"2.0\", \"everything\": false, \"broadcast\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001e9f00a343869c4b4481e652594b6f8eae760887301e1567780c4270bd30019e2010000006a47304402202d1e959145e5dc3ec42ee86fb60722c03a001e8ec11d8394752bd1695ca5279f02202c714f3d40917898a862e9fae707e80666463e393c248ada114abe1f892f86e201210267f2c1b36491d9597d967005cf254782bb5887988776e7c7e21ab4377a2d7d87ffffffff0200c2eb0b000000001976a914a858150692010ad17ca54a100925effc55d0bee888ac90d7ae2f000000001976a914948a164bd4049fac04830906efab835cbd41ec5888ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mhJhtMKzFMs7iFKnGUHUQZryjXad4v3Fi5\",\n \"amount\": \"10.0\",\n \"confirmations\": 6,\n \"height\": 201,\n \"nout\": 1,\n \"timestamp\": 1630567675,\n \"txid\": \"e21900d30b27c4807756e101738860e7eaf8b69425651e48b4c46938340af0e9\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mvs5LLQGfgY49FptN2H1dVWDxXAetAXDY7\",\n \"amount\": \"2.0\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"f8283696d48f8affcf1b5d3b3166b8f30274069427c95507be9f735e24ce67ac\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mu4MiuQqZpiH9UyFtjjRrTsE8EQMxUfzqR\",\n \"amount\": \"7.999876\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"f8283696d48f8affcf1b5d3b3166b8f30274069427c95507be9f735e24ce67ac\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000124\",\n \"total_input\": \"10.0\",\n \"total_output\": \"9.999876\",\n \"txid\": \"f8283696d48f8affcf1b5d3b3166b8f30274069427c95507be9f735e24ce67ac\"\n }\n}"
"curl": "curl -d'{\"method\": \"account_fund\", \"params\": {\"to_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"amount\": \"2.0\", \"everything\": false, \"broadcast\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet account fund --to_account=\"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\" --amount=2.0 --broadcast",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_fund\", \"params\": {\"to_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"amount\": \"2.0\", \"everything\": false, \"broadcast\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001e2409fc033c8e4eef31ddac623a6fb6c8ac8f4db7a39f79a7f5c07faf56a82f2000000006a47304402203ec7fb9df09ad9e575d310faab27de22a7a5ea0307634788d93110c49201638102207edf41c6d08d4d32d5d7a3e44593a14356011df3d8960df16fdd97d5cae9ae7e0121027d1ee838ec7dcc202b034b1bcd42fc9ed0efec566738deafb821129d040a8555ffffffff0200c2eb0b000000001976a9144f133b74f3816a25a45a60f09d0c74668e001f7c88ac90d7ae2f000000001976a914eab374eb4afab8ccbcf995fad162a24311041ad688ac00000000\",\n \"inputs\": [\n {\n \"address\": \"miwFf1Qr89v15UsiXTxtrt8VGMxBDNd499\",\n \"amount\": \"10.0\",\n \"confirmations\": 6,\n \"height\": 201,\n \"nout\": 0,\n \"timestamp\": 1638473066,\n \"txid\": \"f2826af5fa075c7f9af7397adbf4c88a6cfba623c6da1df3eee4c833c09f40e2\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mnj4iVKHASwtxy6qUAmyRiqwxSKxynUGJQ\",\n \"amount\": \"2.0\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"0cd8dc8647ec9feaff476d8dea3458fcc98bfb9d2a13f65ec6edeb983246a136\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"n2uwPp8uTPdEmHw43mp8Tq6AG7dNFKo7cW\",\n \"amount\": \"7.999876\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"0cd8dc8647ec9feaff476d8dea3458fcc98bfb9d2a13f65ec6edeb983246a136\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000124\",\n \"total_input\": \"10.0\",\n \"total_output\": \"9.999876\",\n \"txid\": \"0cd8dc8647ec9feaff476d8dea3458fcc98bfb9d2a13f65ec6edeb983246a136\"\n }\n}"
},
{
"title": "Spread LBC between multiple addresses",
"curl": "curl -d'{\"method\": \"account_fund\", \"params\": {\"to_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"from_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"amount\": \"1.5\", \"everything\": false, \"outputs\": 2, \"broadcast\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet account fund --to_account=\"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\" --from_account=\"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\" --amount=1.5 --outputs=2 --broadcast",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_fund\", \"params\": {\"to_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"from_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"amount\": \"1.5\", \"everything\": false, \"outputs\": 2, \"broadcast\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001ac67ce245e739fbe0755c92794067402f3b866313b5d1bcfff8a8fd4963628f8000000006a473044022022f87e8de1c4cd7209976d90ec1de48c67558402144446f16b4de58992fd4180022013380abab11bfb497de0a304150ead3924403287f6c984e7c0350f7d303179ef012103f867e609a6ec6841b477b00656003f603c6789517b921c7723ffaa0ebcf586d5ffffffff03c0687804000000001976a914a858150692010ad17ca54a100925effc55d0bee888acc0687804000000001976a914a858150692010ad17ca54a100925effc55d0bee888ac6cb9fa02000000001976a9148d6dce2e91024e46798fac5a083b39d89ce36a5388ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mvs5LLQGfgY49FptN2H1dVWDxXAetAXDY7\",\n \"amount\": \"2.0\",\n \"confirmations\": 1,\n \"height\": 207,\n \"nout\": 0,\n \"timestamp\": 1630567676,\n \"txid\": \"f8283696d48f8affcf1b5d3b3166b8f30274069427c95507be9f735e24ce67ac\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mvs5LLQGfgY49FptN2H1dVWDxXAetAXDY7\",\n \"amount\": \"0.75\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mvs5LLQGfgY49FptN2H1dVWDxXAetAXDY7\",\n \"amount\": \"0.75\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mtQm7WpSZYAC5Wmt31xWM6uvYd8QBtea2B\",\n \"amount\": \"0.499859\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 2,\n \"timestamp\": null,\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000141\",\n \"total_input\": \"2.0\",\n \"total_output\": \"1.999859\",\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\"\n }\n}"
"curl": "curl -d'{\"method\": \"account_fund\", \"params\": {\"to_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"from_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"amount\": \"1.5\", \"everything\": false, \"outputs\": 2, \"broadcast\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet account fund --to_account=\"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\" --from_account=\"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\" --amount=1.5 --outputs=2 --broadcast",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_fund\", \"params\": {\"to_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"from_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"amount\": \"1.5\", \"everything\": false, \"outputs\": 2, \"broadcast\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"010000000136a1463298ebedc65ef6132a9dfb8bc9fc5834ea8d6d47ffea9fec4786dcd80c000000006a4730440220581b33fa9c9d67c0cd5604ea8ff32565b15899d7da49bd88b457716d3b6c85a502204be916e91f179c9f996c553da58e642c4058b499cdb9f5ee1a25fa5d0542f00e0121021cf780f862be746f90b57d02736e5ee4dbaf68fb4c29148ba75b15f835e382aaffffffff03c0687804000000001976a914b7e494ed2996f0a66616a6bc93d4fdd6a140f9bb88acc0687804000000001976a914b7e494ed2996f0a66616a6bc93d4fdd6a140f9bb88ac6cb9fa02000000001976a914abb7390236f51358e2dc1a9c783d597ae1df3db688ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mnj4iVKHASwtxy6qUAmyRiqwxSKxynUGJQ\",\n \"amount\": \"2.0\",\n \"confirmations\": 1,\n \"height\": 207,\n \"nout\": 0,\n \"timestamp\": 1638473067,\n \"txid\": \"0cd8dc8647ec9feaff476d8dea3458fcc98bfb9d2a13f65ec6edeb983246a136\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mxHHnJBJ7SJUbobnUYJdzqCCtPkwpmL3ZN\",\n \"amount\": \"0.75\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mxHHnJBJ7SJUbobnUYJdzqCCtPkwpmL3ZN\",\n \"amount\": \"0.75\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mwAuLPjKkuuQdb5sg68mfXDuZJbTTAXR2r\",\n \"amount\": \"0.499859\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 2,\n \"timestamp\": null,\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000141\",\n \"total_input\": \"2.0\",\n \"total_output\": \"1.999859\",\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\"\n }\n}"
},
{
"title": "Transfer all LBC to a specified account",
"curl": "curl -d'{\"method\": \"account_fund\", \"params\": {\"from_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"everything\": true, \"broadcast\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet account fund --from_account=\"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\" --everything --broadcast",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_fund\", \"params\": {\"from_account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"everything\": true, \"broadcast\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"01000000035e249c2d84895720c358bc33ee00a64b0ab9889732adb09a80a49134fc875ad2000000006a473044022074ac46e3d6b794b649bb229c1918f2124bea89da9b823e397e61f344e27e894b02206378546a40e1459b23c8a9f3cd07c3e18e5240a3a18cc9fd4f22ac5a965c1fb6012103f867e609a6ec6841b477b00656003f603c6789517b921c7723ffaa0ebcf586d5ffffffff5e249c2d84895720c358bc33ee00a64b0ab9889732adb09a80a49134fc875ad2010000006b483045022100f5e83245f943e62123e88cf38f2941158a20f37d4f02885d22a837b4bff66ee00220346a06c53326699cf3be01567537670e443485bb022fcb567631cf48f27e6e1d012103f867e609a6ec6841b477b00656003f603c6789517b921c7723ffaa0ebcf586d5ffffffff5e249c2d84895720c358bc33ee00a64b0ab9889732adb09a80a49134fc875ad2020000006b483045022100a1fb8e9e23ecfd29bed86e12b1a6ab977b930915f25b96e0e4b847ae6253d619022046206d189e59aa9806b810066ef26c9dd83dead212212bc35fc4101e03e6d39401210318857bbd3a2f61c04512dff77d4aac1db91eb9753d37fb7a8ddbdfbdd5c4beabffffffff015027eb0b000000001976a9145e10d78836b02121866dc204ea3e8617092e5bdd88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mvs5LLQGfgY49FptN2H1dVWDxXAetAXDY7\",\n \"amount\": \"0.75\",\n \"confirmations\": 1,\n \"height\": 208,\n \"nout\": 0,\n \"timestamp\": 1630567676,\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mvs5LLQGfgY49FptN2H1dVWDxXAetAXDY7\",\n \"amount\": \"0.75\",\n \"confirmations\": 1,\n \"height\": 208,\n \"nout\": 1,\n \"timestamp\": 1630567676,\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mtQm7WpSZYAC5Wmt31xWM6uvYd8QBtea2B\",\n \"amount\": \"0.499859\",\n \"confirmations\": 1,\n \"height\": 208,\n \"nout\": 2,\n \"timestamp\": 1630567676,\n \"txid\": \"d25a87fc3491a4809ab0ad329788b90a4ba600ee33bc58c3205789842d9c245e\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mp6KzHPSgp4DW57qGcKAWR1S48EvDxnot3\",\n \"amount\": \"1.999604\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"ec420a34acc98a91e82aa19c25c1396aacc10be80262de8177813be8f5285ec8\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000255\",\n \"total_input\": \"1.999859\",\n \"total_output\": \"1.999604\",\n \"txid\": \"ec420a34acc98a91e82aa19c25c1396aacc10be80262de8177813be8f5285ec8\"\n }\n}"
"curl": "curl -d'{\"method\": \"account_fund\", \"params\": {\"from_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"everything\": true, \"broadcast\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet account fund --from_account=\"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\" --everything --broadcast",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_fund\", \"params\": {\"from_account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"everything\": true, \"broadcast\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"01000000031a4879d551d07dc602c4e87c192543229d3c39d9b15a255258950bd5acd350d6000000006b483045022100875a47f6c221070c7107920daf1402cd791b77d485c51451b5f1a7d69c8814a702203328d78760a89ea0fe0dba4e6a700c2137e509141df6dbde2bca5b58ad8ad4690121038f2d1a26b70adc8afc0c28d07b35047a51510bd184ae5a94112aad1ac6a1ecaeffffffff1a4879d551d07dc602c4e87c192543229d3c39d9b15a255258950bd5acd350d6010000006a47304402204c488a265278e751fb1cb7d83c7126c097353164cc26b23c05db9663aebf51640220455e397266ac194e426b9b26eb99de4f882a35b09a8bff8c221ba0951f3c94480121038f2d1a26b70adc8afc0c28d07b35047a51510bd184ae5a94112aad1ac6a1ecaeffffffff1a4879d551d07dc602c4e87c192543229d3c39d9b15a255258950bd5acd350d6020000006b483045022100d40c26c6741b1df67a5495298149fb8429bff099c54821edce7737f454ecad0c02202d6ee6ce2f9ee9246e2fac87d4bb48d453326e4eff0df66369bb1ff22a87351a0121025ed0607c4daabfa78b94bfe67a951c888a272a42c91e7b35b68af84dafcb87b6ffffffff015027eb0b000000001976a914b48e2be7f09002ccf6177f7bf559a3f237d368e088ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mxHHnJBJ7SJUbobnUYJdzqCCtPkwpmL3ZN\",\n \"amount\": \"0.75\",\n \"confirmations\": 1,\n \"height\": 208,\n \"nout\": 0,\n \"timestamp\": 1638473067,\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mxHHnJBJ7SJUbobnUYJdzqCCtPkwpmL3ZN\",\n \"amount\": \"0.75\",\n \"confirmations\": 1,\n \"height\": 208,\n \"nout\": 1,\n \"timestamp\": 1638473067,\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\",\n \"type\": \"payment\"\n },\n {\n \"address\": \"mwAuLPjKkuuQdb5sg68mfXDuZJbTTAXR2r\",\n \"amount\": \"0.499859\",\n \"confirmations\": 1,\n \"height\": 208,\n \"nout\": 2,\n \"timestamp\": 1638473067,\n \"txid\": \"d650d3acd50b955852255ab1d9393c9d224325197ce8c402c67dd051d579481a\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mwyeEqGbtNjYDsoibvztKzDjkLRtAvLkFQ\",\n \"amount\": \"1.999604\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"eaa29eb1921b0c9055aae9cf445a5181a182a7168eb76c6db3395640f76e2e55\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000255\",\n \"total_input\": \"1.999859\",\n \"total_output\": \"1.999604\",\n \"txid\": \"eaa29eb1921b0c9055aae9cf445a5181a182a7168eb76c6db3395640f76e2e55\"\n }\n}"
}
]
},
@@ -641,7 +641,7 @@
"curl": "curl -d'{\"method\": \"account_list\", \"params\": {\"include_claims\": false, \"show_seed\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet account list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_list\", \"params\": {\"include_claims\": false, \"show_seed\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"certificates\": 0,\n \"coins\": 10.0,\n \"encrypted\": false,\n \"id\": \"mm5kqqENANLHxj2dhxJhs8zWiwdndbf2jM\",\n \"is_default\": true,\n \"ledger\": \"lbc_regtest\",\n \"name\": \"Account #mm5kqqENANLHxj2dhxJhs8zWiwdndbf2jM\",\n \"public_key\": \"tpubD6NzVbkrYhZ4Y27iMLaV6qbMCkraNnmt8HMmC3iSxSS61oTYHCJRz1YfzYtV2DUWvJDkvAzxnGye138VBr7FvDpUqdPK7Znf8TdhnfpAL1b\",\n \"satoshis\": 1000000000\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"certificates\": 0,\n \"coins\": 10.0,\n \"encrypted\": false,\n \"id\": \"mnSNxHg6EKYaDap8u5yrAaWXQem4Qnu1ib\",\n \"is_default\": true,\n \"ledger\": \"lbc_regtest\",\n \"name\": \"Account #mnSNxHg6EKYaDap8u5yrAaWXQem4Qnu1ib\",\n \"public_key\": \"tpubD6NzVbkrYhZ4WXNvaRftN5mgapBrqY2EQocEqoPrkBQMyzfCirS7hmZsRr5mAReK8hdNBc8SJBjRbbCBvjNnNzCKT3SbpCMjjmvT28c5ZEe\",\n \"satoshis\": 1000000000\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -686,10 +686,10 @@
"examples": [
{
"title": "Remove an account",
"curl": "curl -d'{\"method\": \"account_remove\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"}}' http://localhost:5279/",
"lbrynet": "lbrynet account remove mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_remove\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1630567643,\n \"name\": \"generated account\",\n \"private_key\": \"tprv8ZgxMBicQKsPduGkPH4FUHQh6vCVc5PuD2t5148Qf3ds9oJ2b6MpGzKK6BD8QVWeYfkFPAqrEvtFXckYM7p83XaQGyaCiHFAi3hdUnfScwC\",\n \"public_key\": \"tpubD6NzVbkrYhZ4XNJYGviqsh4ofwiRmQaonLUrHaAi5KSFzHYoDVBQTUwBGMQsNMBh3cvEAmp6uotNWBXa2kUJ3GiFHLyWEc7k1YtDkF7jGQW\",\n \"seed\": \"boost property since welcome armed arch goddess correct stick traffic pyramid glory\"\n }\n}"
"curl": "curl -d'{\"method\": \"account_remove\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"}}' http://localhost:5279/",
"lbrynet": "lbrynet account remove mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_remove\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 1\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1638473033,\n \"name\": \"generated account\",\n \"private_key\": \"tprv8ZgxMBicQKsPf3EwVBH3vG4NWPJQk4xBTC1qi1uU45s7PREzdMBdyytxxgNA5kYXmgZiTDL3TgWTU3mEgGnzncSTqJGiMZCH7J9wGo8HftD\",\n \"public_key\": \"tpubD6NzVbkrYhZ4YWGjNpweKfiV5QpLuQ962VcczXwmUMfWDuVmFk1EAUWq8pywR7wShpYF5eX3uQx1XmwZYVrmRVHDRS4yTuCHe76BCgvMSCo\",\n \"seed\": \"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\"\n }\n}"
}
]
},
@@ -782,10 +782,10 @@
"examples": [
{
"title": "Modify maximum number of times a change address can be reused",
"curl": "curl -d'{\"method\": \"account_set\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"default\": false, \"change_max_uses\": 10}}' http://localhost:5279/",
"lbrynet": "lbrynet account set mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr --change_max_uses=10",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_set\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\", \"default\": false, \"change_max_uses\": 10}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 10\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1630567643,\n \"name\": \"new account\",\n \"private_key\": \"tprv8ZgxMBicQKsPduGkPH4FUHQh6vCVc5PuD2t5148Qf3ds9oJ2b6MpGzKK6BD8QVWeYfkFPAqrEvtFXckYM7p83XaQGyaCiHFAi3hdUnfScwC\",\n \"public_key\": \"tpubD6NzVbkrYhZ4XNJYGviqsh4ofwiRmQaonLUrHaAi5KSFzHYoDVBQTUwBGMQsNMBh3cvEAmp6uotNWBXa2kUJ3GiFHLyWEc7k1YtDkF7jGQW\",\n \"seed\": \"boost property since welcome armed arch goddess correct stick traffic pyramid glory\"\n }\n}"
"curl": "curl -d'{\"method\": \"account_set\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"default\": false, \"change_max_uses\": 10}}' http://localhost:5279/",
"lbrynet": "lbrynet account set mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8 --change_max_uses=10",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"account_set\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\", \"default\": false, \"change_max_uses\": 10}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"address_generator\": {\n \"change\": {\n \"gap\": 6,\n \"maximum_uses_per_address\": 10\n },\n \"name\": \"deterministic-chain\",\n \"receiving\": {\n \"gap\": 20,\n \"maximum_uses_per_address\": 1\n }\n },\n \"encrypted\": false,\n \"id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"is_default\": false,\n \"ledger\": \"lbc_regtest\",\n \"modified_on\": 1638473034,\n \"name\": \"new account\",\n \"private_key\": \"tprv8ZgxMBicQKsPf3EwVBH3vG4NWPJQk4xBTC1qi1uU45s7PREzdMBdyytxxgNA5kYXmgZiTDL3TgWTU3mEgGnzncSTqJGiMZCH7J9wGo8HftD\",\n \"public_key\": \"tpubD6NzVbkrYhZ4YWGjNpweKfiV5QpLuQ962VcczXwmUMfWDuVmFk1EAUWq8pywR7wShpYF5eX3uQx1XmwZYVrmRVHDRS4yTuCHe76BCgvMSCo\",\n \"seed\": \"diary laptop impose focus electric expose regret thrive liquid polar cable bunker\"\n }\n}"
}
]
}
@@ -821,9 +821,9 @@
"examples": [
{
"title": "Check if address is mine",
"curl": "curl -d'{\"method\": \"address_is_mine\", \"params\": {\"address\": \"muFMoJK6g1gNktsXYWyDUzb4W8kc7bXKy3\"}}' http://localhost:5279/",
"lbrynet": "lbrynet address is_mine muFMoJK6g1gNktsXYWyDUzb4W8kc7bXKy3",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"address_is_mine\", \"params\": {\"address\": \"muFMoJK6g1gNktsXYWyDUzb4W8kc7bXKy3\"}}).json()",
"curl": "curl -d'{\"method\": \"address_is_mine\", \"params\": {\"address\": \"mxCb2mjmyDRXYdPGkC6QRqjMwpTL2wVoVe\"}}' http://localhost:5279/",
"lbrynet": "lbrynet address is_mine mxCb2mjmyDRXYdPGkC6QRqjMwpTL2wVoVe",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"address_is_mine\", \"params\": {\"address\": \"mxCb2mjmyDRXYdPGkC6QRqjMwpTL2wVoVe\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": true\n}"
}
]
@@ -870,14 +870,14 @@
"curl": "curl -d'{\"method\": \"address_list\", \"params\": {}}' http://localhost:5279/",
"lbrynet": "lbrynet address list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"address_list\", \"params\": {}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mgFHBxv5E4sFvwUGGo38kEAy17ShnxH8wR\",\n \"pubkey\": \"tpubDA9GDAntyJu4JZQ83iwTiweQZQeacjzqAggyK6fcC5Twtv1Tx9m2JQCze6gC8ygn1vmpkmVXsS1kJx9YVk63goLrL5hpZKR3yVnzaka4Ave\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mh7M3qE7uYM5xsbVPCsoy8GovkPjRnpdZD\",\n \"pubkey\": \"tpubDA9GDAntyJu4a7hndHFRPprpVAfeBCQ6HCQgcBk5Z3KjHevV68WzRRbThhTbZin8eAnEUa7XSd3Es4HNNCzAoehZLQBFnmF5SwQY9seRaJn\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mhTcH8cE3MVkWzyhnZGbEK7KgiHb73M4Dn\",\n \"pubkey\": \"tpubDA9GDAntyJu4wUjVRhu4DDBViocAiNu5pc6bHqv9JWhGBmEe4cGge36ZaHQbh2WjpAhL8sQcq9ch8zc4wsTA2V5GhdADPw1Y7x27k7FZKhD\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mhxn7CVxZke1XW5Ds9EziUR6UGn8PYD5h4\",\n \"pubkey\": \"tpubDA9GDAntyJu4NWHpn3XDifdgHCFZ2izY5J8pKLm9KW6d6j6cc79HSUPWmZ87FcCswWVzkPvHgLQEja2WgafR4TRSNqG9YWjLb4Mk4AdKf7B\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"miDhUnNDNNe32VG2g88dswPivRF28eSXiK\",\n \"pubkey\": \"tpubDA9GDAntyJu4NAJGcTKReU8YPLLKZKrVZBw69LuQErmkmoG34zHyDjU8gaQnm41pbxcrLoLavCtQXwghw6sez4heCjoaHUGCEbm78YQRvu1\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"migoW8Ks6whuwHL7bwJsaekyL1bBAxFuft\",\n \"pubkey\": \"tpubDA9GDAntyJu54TvpUgmSCUQib5DVPt32BrxppkbpJ8y2Rym7ami1g5XkxXQ7gXQXakBDStqLnM2gvRp8cxrWQ9UPGSLK2y9LeWkrwKpTqbb\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mj9Uy7F3RJDRHU11UZ1x14xDgrGA32ZcFB\",\n \"pubkey\": \"tpubDA9GDAntyJu4dLxTbBWeRXJKt3MLExpnUZtpwW9nnGacECe7V6W5XF9XD2q4jH6BqSR3TtVsAD7AxY6HxNDMCtuXCqm78jcuF8mWLbT4AS1\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mkMAEVoNHAfctodz75skDcyPAWWemQU7Ny\",\n \"pubkey\": \"tpubDA9GDAntyJu4mWKuSePxpKruFPeiQB7W2H2FjiCt8ECe5uq6cyL1NsZbDtZT5eMA5TE1UVEHVTewtPWXYRvdNJfGyEJm6Yrq8T7ZtKxZ6MN\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mkQjVtF9vv7Xaqj1wnMrJ9pN9VyeTJgDxY\",\n \"pubkey\": \"tpubDA9GDAntyJu4L65TpHRzD8x7gVNQb4cpeCX6jQ33rVFdBXE82zumk36wfAakiSYxGKmaUHgUezVsme6kvN4AwXHkKSjpGmWZLQDxFQvqFfA\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mm2nUb52jM6VYxw88YAN1yGRVwokrLkDuM\",\n \"pubkey\": \"tpubDA9GDAntyJu4QEMzBwaScsHNH4oVtx4bpFb8VhU2CKK8QJMzErjk63H4c1ZzLx47uAmVLWSRcA1tzvfYx2Rr2hUiaJJ77qm3uVQjvEDB7PX\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mmQjfvGjmD9hr2tjMbuRkiLKXkmsAAe5yG\",\n \"pubkey\": \"tpubDA9GDAntyJu4EqV2qxMJdESJ7Uf9chzp9bgY4DaWMthFVyeNBDXmYDVAbTVVJTUvykx5jkwVN7wWSGMA5aZAZETqcGUK4GQEEz5vJYdQF8u\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mmpdJrBWLKXbfg9A8jNobUNXe2QeTYBVAr\",\n \"pubkey\": \"tpubDA9GDAntyJu53ArAJ76HFHVYSWjVHLeGVw3c15oqBZP33v2sgidBwmk11kPsgx6G3MV8WvSvoR2GQznkYXSc6MoHWoCp29f7Yvwszgc9V3A\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mpXcMmM2zExJhH5NiYMP4SvRYJMCxYJUwZ\",\n \"pubkey\": \"tpubDA9GDAntyJu4zNu4NsmqpHbpHAnxQafkEbQNssaNz4Gf7EtXcNVn8BKNaADGLumF6LJXRCV35BdcUhib325aGZoLnF62bLSJDTzkR2DY5B9\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mrqWceHpMpGcbu7BCqpQ8p9ob3qRWmjoXB\",\n \"pubkey\": \"tpubDA9GDAntyJu4XfXXN8zReHfXHTNae3HwzE99dJHDVMUE2fdZb8YJ2ALh8iz7tnf1XUohVJJxZ98v6HZhipfHu4w5Gv2i63VUcaomno6dSGh\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"msDzY2VuBeD51bNUTXsQKuiyAPb9wZQSFQ\",\n \"pubkey\": \"tpubDA9GDAntyJu4sTcTdGkVL4oeqWz8JYvVAjc7stVeKLxWMU7eHEym6rvDC5ZKxJZcHGu9bNAM6yayV5VhEGsFfVP3NwHjJh1Kj9b4fQY86s3\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mtGkypAUsHYWxJ3vjJtCvNhx5hdgpjhbMZ\",\n \"pubkey\": \"tpubDA9GDAntyJu4gtxXJSkgYg6YfwkmqBbWyYuxNMJACw5g8BSiMt5gY8oXbGt5NuBZJNkAni3kZDJxpWynpvBTtx4apqxVMQCTL6D572PxQW5\",\n \"Line truncated
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mjxKPXWKSqWQTQv4QbgvQPo2m9wPhU5G6Y\",\n \"pubkey\": \"tpubDA9GDAntyJu4rZckAEC4cdKx1eh3gDTibq6fopttFzKL5aKCSJDtpy57Reorz3hPpwo82SAKqe5MsCAYEcNHauYYkVu2GgsSPMf1zhwXSAq\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mmB886XDoMKhDP3pJjHSxDrLRPBLt6mV2z\",\n \"pubkey\": \"tpubDA9GDAntyJu51RvEnZAbK4MEtNWwaZkZd2Pv2p6mUKMxNe2VML4VAKXaDn9iraqVq5jqifRCUchR8Pm67Ed9s4gTkPqx7qQkBW7KkYuNYD6\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mnj4iVKHASwtxy6qUAmyRiqwxSKxynUGJQ\",\n \"pubkey\": \"tpubDA9GDAntyJu4V5EGwwiJBBox3x6kb8KAaNNTssGLoz44FZa9uGv4FMWTDTwtCeyZ39oMroYcjezLNHpgFYVBZJzmFWCKkFDBNuaVHDRtPFK\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mp11uhp7Vz8Bu7f1aKEpqgbCDbC2E2CnUb\",\n \"pubkey\": \"tpubDA9GDAntyJu4QKLhMF3B95svxszCGDxDgTs8bqarKuCi3Nc4kHtyPPNnPNMGmdChcxrsgysVEDoRjaaqtrM6rEjcymuBVbNbaWuV5MdjJg8\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mpxiTmDnNyS8c2NMSoGQPYAV58xfVfpg5d\",\n \"pubkey\": \"tpubDA9GDAntyJu4F2Vq72YjhKMwF6eX8Wj1ooGrWpppB9o4bocmt1EeJnqEaXn6dkSyK4htWdpg3qMaGUwuVXZQnvV7saqsNvyHRFU5WT3cZ6V\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mr9cjjxLw5UWVw2jwMb9EgeuFz7FZNCbf7\",\n \"pubkey\": \"tpubDA9GDAntyJu4TXVJxR6X2RQHjMP9DHdcXSphSm2EUyWya7YHvnW1ngxekGpxjbUtziutLLkaU5dzqQwu2ageBeUFY98JdBHu4m7gB733kD3\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mrWNQ76XWtvLpLbzGQVRvG8FnkvpDtuBrA\",\n \"pubkey\": \"tpubDA9GDAntyJu4SDpyPC9h7En973t1quyHW1VHvs5pGWoJcfU7ptBVNGiGKzMgKsVciPZzJRup3uEV6CudYz1q66RCHM65GdJ4SsC8U46YJ2k\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mripohwfVX5zGnCbKsp33UGA6jXZs22pFH\",\n \"pubkey\": \"tpubDA9GDAntyJu4FACueo7iyccfqVeSv9j9AA8YqPamTtdRxLehktt54AoUnoJCUJpaDbseRxeWJCHuEG3QzYUzRzKW3uGq8B6PfsdsoTNyGrv\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"msN8mF4SZg8RLMbCi7Cv7pCrwbCf4UzHWu\",\n \"pubkey\": \"tpubDA9GDAntyJu4Yy1qHhSkxApF2hjAmciCFBf2xnLjRVU7RQi5n52ooLUBsb57KzVNHsL3kkCbprSYMEfYeaY3vFpPyiT2mQDcQR87yhx9Bki\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mu2uaYhYVgJd4MAnrJYNGaordd4jS8Xi75\",\n \"pubkey\": \"tpubDA9GDAntyJu4wQxzSnZW4hpPBDSHwhw5sHWmb2VkJikneQSe5uCE8pt9MVEeobhDjXrhYD8PnK4zsrCB261MrCMAjjhLUdbSpH3hKZ3T968\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"muCFBDAvsiwtU7pexhyEWQ9i5X3YYBw23B\",\n \"pubkey\": \"tpubDA9GDAntyJu4P5Wgtcb8Y6rhV9vzQAGz53xVnhiNTTbHaYtz2cwhCg22imZRseSi142zJDPRwNSgGFRoX9zNGbj4aF6grTHXMS9dKCzFPpD\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mumBiAeEpD5DCQLK5d7DVDfk4e1WKubqCn\",\n \"pubkey\": \"tpubDA9GDAntyJu4gqdai4msczjecaVjDAeb29rrjNC8MhxTQJi3BNbpfADCYTDScTrUG9ueK1BpdLcxNmzkjmNhEKCqmEHm4kJfhaWo5tsrH9n\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mvznsgPmwuASzqyvYwh5oHrjr4t9oreYHL\",\n \"pubkey\": \"tpubDA9GDAntyJu4NWr1nKP6BUzBT9k18xVgLU4FuJdzggfwpDVGqMvmVHg7cxA1BA77eDJHr7EM6x7bi4yQsFt2ooa2xNTz7AhZyGzWr8wi7dj\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mw4SSRZZmd1Kr1MsVU2Xd9oKHD6Qr8kmNA\",\n \"pubkey\": \"tpubDA9GDAntyJu53BcFJxfpooZLsTVE6LqdmAjNBBxfFxR1w7DoWnpoRBLc8GwyYvHyxqnbMu4Pex4QjhFp1nb5DJ6YtDEUXgDqkG2c3iZz7aU\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mwAuLPjKkuuQdb5sg68mfXDuZJbTTAXR2r\",\n \"pubkey\": \"tpubDA9GDAntyJu4L2fZrseWCbANymwo1Qy5FqsJc1FHExd1jWfL53vZM8HX7gtay7dEV5kXCtzUi2bKU1i1WZJGoByzKequDuo5FdcPoxbxVH3\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mwBmCkK9zT4RZ9K5Uz1VJVRTrAS4dgQ1DE\",\n \"pubkey\": \"tpubDA9GDAntyJu4miZVyhuCGTi5VzywDRZ336pKQTK8ZbZuN3kPB3hdrFbBm4bQBsobgBUE3bXenWJisbYF8RSeuKD1uT2YCcubbqrKVJKqPKp\",\n \"Line truncated
},
{
"title": "List addresses in specified account",
"curl": "curl -d'{\"method\": \"address_list\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"}}' http://localhost:5279/",
"lbrynet": "lbrynet address list --account_id=\"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"address_list\", \"params\": {\"account_id\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mgFHBxv5E4sFvwUGGo38kEAy17ShnxH8wR\",\n \"pubkey\": \"tpubDA9GDAntyJu4JZQ83iwTiweQZQeacjzqAggyK6fcC5Twtv1Tx9m2JQCze6gC8ygn1vmpkmVXsS1kJx9YVk63goLrL5hpZKR3yVnzaka4Ave\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mh7M3qE7uYM5xsbVPCsoy8GovkPjRnpdZD\",\n \"pubkey\": \"tpubDA9GDAntyJu4a7hndHFRPprpVAfeBCQ6HCQgcBk5Z3KjHevV68WzRRbThhTbZin8eAnEUa7XSd3Es4HNNCzAoehZLQBFnmF5SwQY9seRaJn\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mhTcH8cE3MVkWzyhnZGbEK7KgiHb73M4Dn\",\n \"pubkey\": \"tpubDA9GDAntyJu4wUjVRhu4DDBViocAiNu5pc6bHqv9JWhGBmEe4cGge36ZaHQbh2WjpAhL8sQcq9ch8zc4wsTA2V5GhdADPw1Y7x27k7FZKhD\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mhxn7CVxZke1XW5Ds9EziUR6UGn8PYD5h4\",\n \"pubkey\": \"tpubDA9GDAntyJu4NWHpn3XDifdgHCFZ2izY5J8pKLm9KW6d6j6cc79HSUPWmZ87FcCswWVzkPvHgLQEja2WgafR4TRSNqG9YWjLb4Mk4AdKf7B\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"miDhUnNDNNe32VG2g88dswPivRF28eSXiK\",\n \"pubkey\": \"tpubDA9GDAntyJu4NAJGcTKReU8YPLLKZKrVZBw69LuQErmkmoG34zHyDjU8gaQnm41pbxcrLoLavCtQXwghw6sez4heCjoaHUGCEbm78YQRvu1\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"migoW8Ks6whuwHL7bwJsaekyL1bBAxFuft\",\n \"pubkey\": \"tpubDA9GDAntyJu54TvpUgmSCUQib5DVPt32BrxppkbpJ8y2Rym7ami1g5XkxXQ7gXQXakBDStqLnM2gvRp8cxrWQ9UPGSLK2y9LeWkrwKpTqbb\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mj9Uy7F3RJDRHU11UZ1x14xDgrGA32ZcFB\",\n \"pubkey\": \"tpubDA9GDAntyJu4dLxTbBWeRXJKt3MLExpnUZtpwW9nnGacECe7V6W5XF9XD2q4jH6BqSR3TtVsAD7AxY6HxNDMCtuXCqm78jcuF8mWLbT4AS1\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mkMAEVoNHAfctodz75skDcyPAWWemQU7Ny\",\n \"pubkey\": \"tpubDA9GDAntyJu4mWKuSePxpKruFPeiQB7W2H2FjiCt8ECe5uq6cyL1NsZbDtZT5eMA5TE1UVEHVTewtPWXYRvdNJfGyEJm6Yrq8T7ZtKxZ6MN\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mkQjVtF9vv7Xaqj1wnMrJ9pN9VyeTJgDxY\",\n \"pubkey\": \"tpubDA9GDAntyJu4L65TpHRzD8x7gVNQb4cpeCX6jQ33rVFdBXE82zumk36wfAakiSYxGKmaUHgUezVsme6kvN4AwXHkKSjpGmWZLQDxFQvqFfA\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mm2nUb52jM6VYxw88YAN1yGRVwokrLkDuM\",\n \"pubkey\": \"tpubDA9GDAntyJu4QEMzBwaScsHNH4oVtx4bpFb8VhU2CKK8QJMzErjk63H4c1ZzLx47uAmVLWSRcA1tzvfYx2Rr2hUiaJJ77qm3uVQjvEDB7PX\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mmQjfvGjmD9hr2tjMbuRkiLKXkmsAAe5yG\",\n \"pubkey\": \"tpubDA9GDAntyJu4EqV2qxMJdESJ7Uf9chzp9bgY4DaWMthFVyeNBDXmYDVAbTVVJTUvykx5jkwVN7wWSGMA5aZAZETqcGUK4GQEEz5vJYdQF8u\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mmpdJrBWLKXbfg9A8jNobUNXe2QeTYBVAr\",\n \"pubkey\": \"tpubDA9GDAntyJu53ArAJ76HFHVYSWjVHLeGVw3c15oqBZP33v2sgidBwmk11kPsgx6G3MV8WvSvoR2GQznkYXSc6MoHWoCp29f7Yvwszgc9V3A\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mpXcMmM2zExJhH5NiYMP4SvRYJMCxYJUwZ\",\n \"pubkey\": \"tpubDA9GDAntyJu4zNu4NsmqpHbpHAnxQafkEbQNssaNz4Gf7EtXcNVn8BKNaADGLumF6LJXRCV35BdcUhib325aGZoLnF62bLSJDTzkR2DY5B9\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mrqWceHpMpGcbu7BCqpQ8p9ob3qRWmjoXB\",\n \"pubkey\": \"tpubDA9GDAntyJu4XfXXN8zReHfXHTNae3HwzE99dJHDVMUE2fdZb8YJ2ALh8iz7tnf1XUohVJJxZ98v6HZhipfHu4w5Gv2i63VUcaomno6dSGh\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"msDzY2VuBeD51bNUTXsQKuiyAPb9wZQSFQ\",\n \"pubkey\": \"tpubDA9GDAntyJu4sTcTdGkVL4oeqWz8JYvVAjc7stVeKLxWMU7eHEym6rvDC5ZKxJZcHGu9bNAM6yayV5VhEGsFfVP3NwHjJh1Kj9b4fQY86s3\",\n \"used_times\": 0\n },\n {\n \"account\": \"mgUF5KXtPdtbB7AKMhcDvHFzkcztoeaFjr\",\n \"address\": \"mtGkypAUsHYWxJ3vjJtCvNhx5hdgpjhbMZ\",\n \"pubkey\": \"tpubDA9GDAntyJu4gtxXJSkgYg6YfwkmqBbWyYuxNMJACw5g8BSiMt5gY8oXbGt5NuBZJNkAni3kZDJxpWynpvBTtx4apqxVMQCTL6D572PxQW5\",\n \"Line truncated
"curl": "curl -d'{\"method\": \"address_list\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"}}' http://localhost:5279/",
"lbrynet": "lbrynet address list --account_id=\"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"address_list\", \"params\": {\"account_id\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mjxKPXWKSqWQTQv4QbgvQPo2m9wPhU5G6Y\",\n \"pubkey\": \"tpubDA9GDAntyJu4rZckAEC4cdKx1eh3gDTibq6fopttFzKL5aKCSJDtpy57Reorz3hPpwo82SAKqe5MsCAYEcNHauYYkVu2GgsSPMf1zhwXSAq\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mmB886XDoMKhDP3pJjHSxDrLRPBLt6mV2z\",\n \"pubkey\": \"tpubDA9GDAntyJu51RvEnZAbK4MEtNWwaZkZd2Pv2p6mUKMxNe2VML4VAKXaDn9iraqVq5jqifRCUchR8Pm67Ed9s4gTkPqx7qQkBW7KkYuNYD6\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mnj4iVKHASwtxy6qUAmyRiqwxSKxynUGJQ\",\n \"pubkey\": \"tpubDA9GDAntyJu4V5EGwwiJBBox3x6kb8KAaNNTssGLoz44FZa9uGv4FMWTDTwtCeyZ39oMroYcjezLNHpgFYVBZJzmFWCKkFDBNuaVHDRtPFK\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mp11uhp7Vz8Bu7f1aKEpqgbCDbC2E2CnUb\",\n \"pubkey\": \"tpubDA9GDAntyJu4QKLhMF3B95svxszCGDxDgTs8bqarKuCi3Nc4kHtyPPNnPNMGmdChcxrsgysVEDoRjaaqtrM6rEjcymuBVbNbaWuV5MdjJg8\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mpxiTmDnNyS8c2NMSoGQPYAV58xfVfpg5d\",\n \"pubkey\": \"tpubDA9GDAntyJu4F2Vq72YjhKMwF6eX8Wj1ooGrWpppB9o4bocmt1EeJnqEaXn6dkSyK4htWdpg3qMaGUwuVXZQnvV7saqsNvyHRFU5WT3cZ6V\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mr9cjjxLw5UWVw2jwMb9EgeuFz7FZNCbf7\",\n \"pubkey\": \"tpubDA9GDAntyJu4TXVJxR6X2RQHjMP9DHdcXSphSm2EUyWya7YHvnW1ngxekGpxjbUtziutLLkaU5dzqQwu2ageBeUFY98JdBHu4m7gB733kD3\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mrWNQ76XWtvLpLbzGQVRvG8FnkvpDtuBrA\",\n \"pubkey\": \"tpubDA9GDAntyJu4SDpyPC9h7En973t1quyHW1VHvs5pGWoJcfU7ptBVNGiGKzMgKsVciPZzJRup3uEV6CudYz1q66RCHM65GdJ4SsC8U46YJ2k\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mripohwfVX5zGnCbKsp33UGA6jXZs22pFH\",\n \"pubkey\": \"tpubDA9GDAntyJu4FACueo7iyccfqVeSv9j9AA8YqPamTtdRxLehktt54AoUnoJCUJpaDbseRxeWJCHuEG3QzYUzRzKW3uGq8B6PfsdsoTNyGrv\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"msN8mF4SZg8RLMbCi7Cv7pCrwbCf4UzHWu\",\n \"pubkey\": \"tpubDA9GDAntyJu4Yy1qHhSkxApF2hjAmciCFBf2xnLjRVU7RQi5n52ooLUBsb57KzVNHsL3kkCbprSYMEfYeaY3vFpPyiT2mQDcQR87yhx9Bki\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mu2uaYhYVgJd4MAnrJYNGaordd4jS8Xi75\",\n \"pubkey\": \"tpubDA9GDAntyJu4wQxzSnZW4hpPBDSHwhw5sHWmb2VkJikneQSe5uCE8pt9MVEeobhDjXrhYD8PnK4zsrCB261MrCMAjjhLUdbSpH3hKZ3T968\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"muCFBDAvsiwtU7pexhyEWQ9i5X3YYBw23B\",\n \"pubkey\": \"tpubDA9GDAntyJu4P5Wgtcb8Y6rhV9vzQAGz53xVnhiNTTbHaYtz2cwhCg22imZRseSi142zJDPRwNSgGFRoX9zNGbj4aF6grTHXMS9dKCzFPpD\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mumBiAeEpD5DCQLK5d7DVDfk4e1WKubqCn\",\n \"pubkey\": \"tpubDA9GDAntyJu4gqdai4msczjecaVjDAeb29rrjNC8MhxTQJi3BNbpfADCYTDScTrUG9ueK1BpdLcxNmzkjmNhEKCqmEHm4kJfhaWo5tsrH9n\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mvznsgPmwuASzqyvYwh5oHrjr4t9oreYHL\",\n \"pubkey\": \"tpubDA9GDAntyJu4NWr1nKP6BUzBT9k18xVgLU4FuJdzggfwpDVGqMvmVHg7cxA1BA77eDJHr7EM6x7bi4yQsFt2ooa2xNTz7AhZyGzWr8wi7dj\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mw4SSRZZmd1Kr1MsVU2Xd9oKHD6Qr8kmNA\",\n \"pubkey\": \"tpubDA9GDAntyJu53BcFJxfpooZLsTVE6LqdmAjNBBxfFxR1w7DoWnpoRBLc8GwyYvHyxqnbMu4Pex4QjhFp1nb5DJ6YtDEUXgDqkG2c3iZz7aU\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mwAuLPjKkuuQdb5sg68mfXDuZJbTTAXR2r\",\n \"pubkey\": \"tpubDA9GDAntyJu4L2fZrseWCbANymwo1Qy5FqsJc1FHExd1jWfL53vZM8HX7gtay7dEV5kXCtzUi2bKU1i1WZJGoByzKequDuo5FdcPoxbxVH3\",\n \"used_times\": 0\n },\n {\n \"account\": \"mfaeim8b36T4j9wsYdyyui3ZpvfRaBvUo8\",\n \"address\": \"mwBmCkK9zT4RZ9K5Uz1VJVRTrAS4dgQ1DE\",\n \"pubkey\": \"tpubDA9GDAntyJu4miZVyhuCGTi5VzywDRZ336pKQTK8ZbZuN3kPB3hdrFbBm4bQBsobgBUE3bXenWJisbYF8RSeuKD1uT2YCcubbqrKVJKqPKp\",\n \"Line truncated
}
]
},
@@ -905,7 +905,7 @@
"curl": "curl -d'{\"method\": \"address_unused\", \"params\": {}}' http://localhost:5279/",
"lbrynet": "lbrynet address unused",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"address_unused\", \"params\": {}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": \"muFMoJK6g1gNktsXYWyDUzb4W8kc7bXKy3\"\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": \"mxCb2mjmyDRXYdPGkC6QRqjMwpTL2wVoVe\"\n}"
}
]
}
@@ -962,10 +962,10 @@
"examples": [
{
"title": "Delete a blob",
"curl": "curl -d'{\"method\": \"blob_delete\", \"params\": {\"blob_hash\": \"fcfdd5d1524e7310850316125f129d12cd5f2a3fc08e41c82d6a4dc3eec554cd383a71fc714957cb2a60ee9d62b5dd4c\"}}' http://localhost:5279/",
"lbrynet": "lbrynet blob delete fcfdd5d1524e7310850316125f129d12cd5f2a3fc08e41c82d6a4dc3eec554cd383a71fc714957cb2a60ee9d62b5dd4c",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"blob_delete\", \"params\": {\"blob_hash\": \"fcfdd5d1524e7310850316125f129d12cd5f2a3fc08e41c82d6a4dc3eec554cd383a71fc714957cb2a60ee9d62b5dd4c\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": \"Deleted fcfdd5d1524e7310850316125f129d12cd5f2a3fc08e41c82d6a4dc3eec554cd383a71fc714957cb2a60ee9d62b5dd4c\"\n}"
"curl": "curl -d'{\"method\": \"blob_delete\", \"params\": {\"blob_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\"}}' http://localhost:5279/",
"lbrynet": "lbrynet blob delete 945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"blob_delete\", \"params\": {\"blob_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": \"Deleted 945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\"\n}"
}
]
},
@@ -1020,7 +1020,7 @@
{
"name": "sd_hash",
"type": "str",
"description": "filter blobs by sd hash",
"description": "filter blobs in a stream by sd hash, ie the hash of the stream descriptor blob for a stream that has been downloaded",
"is_required": false
},
{
@@ -1043,7 +1043,7 @@
"curl": "curl -d'{\"method\": \"blob_list\", \"params\": {\"needed\": false, \"finished\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet blob list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"blob_list\", \"params\": {\"needed\": false, \"finished\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n \"fcfdd5d1524e7310850316125f129d12cd5f2a3fc08e41c82d6a4dc3eec554cd383a71fc714957cb2a60ee9d62b5dd4c\",\n \"f4a9c99dcc7c9c7d75b449b89719c614afcb0f2071a5beb6ebaeb737cb2c70b753d12556f7acfa4526dd8a0ae2c1f80d\",\n \"3654f14df3da475a8ef3dcf81ca855ca719102cfe3d905d48d77ad5a264d067e723e3cf76c41f8715e4ca9a6fe96d0d2\",\n \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\"\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 4,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"b4ee14ed36a23aaf499a5ea0b06c8fd8876be8613e19323fc109c7f06768c7cc6b979ecebf8fae6a145ac1e5583409dd\",\n \"53cbd9f603b635030a1149ef44563eb98b9c0b271c936fea628c170865b0528a57dcbada4679f6172d52a40900300ed5\",\n \"9f2c47c94a956cbe0eab3bb615c5c24ed2286144c492df6da2da800452fdb4f907a4b47fce59c56a78cce79795178314\"\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 4,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -1124,10 +1124,10 @@
"examples": [
{
"title": "Abandon a channel claim",
"curl": "curl -d'{\"method\": \"channel_abandon\", \"params\": {\"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel abandon 815f8d597ce588c141b001d05f3240b3ef0bf370",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_abandon\", \"params\": {\"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001903415ca80c2891a49dc94a4b6c24745942bef51dc55f698b5351654b763414e000000006a47304402207df0055a0ca72ca10b957db3f0b6d1638153f0a9c9f9999c6e115e7e1738d1eb02205a7f36deaee300b4690bd1becd89c6e2c18b3ffe60009a7f47ee255468f41d4801210344a78a03b26f6ce6bb8132537a36caf8826058e936f85ab6f916e844077d9c7effffffff0134b7f505000000001976a914203463813df637bd9ef85c62ecfaeee877a0288088ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 9,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"miTEgDsEndZW8oT7ssH9JTbVU7a8w3vGDD\",\n \"amount\": \"0.999893\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"79fba7e644be40ceeffb70eb4c902f847dd37114841495062424d1f0f9e3efe5\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000107\",\n \"total_input\": \"1.0\",\n \"total_output\": \"0.999893\",\n \"txid\": \"79fba7e644be40ceeffb70eb4c902f847dd37114841495062424d1f0f9e3efe5\"\n }\n}"
"curl": "curl -d'{\"method\": \"channel_abandon\", \"params\": {\"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel abandon add76735eb99e808dae24cbdfa6ceb12029ee794",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_abandon\", \"params\": {\"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001637fb47c4b1a1704e8f2e0681677096a3bdf3d5aeb15914621196ab7f3145005000000006b483045022100ffaf0dcf5d5149bf5df0d97df996ca5795db41597f07c239159abae03e6c4150022036aa8c21df3aa04c27376003f1cb1448e640f709c11cdc28e273d666f9ccde1b0121029f23db06243144e274a1a8ec56416f061822d83bb8585a513e267930965d93ddffffffff0134b7f505000000001976a91494dbecc78e652d751522b3dca65177b288b2d00a88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 9,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mu63m6Y3WBLHWzfAwwRVftiWrHXWbGVcD8\",\n \"amount\": \"0.999893\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"feb1fe447f95e1a74cde3fa2b7e3518c75dfd9549d097052767da8da1685080c\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000107\",\n \"total_input\": \"1.0\",\n \"total_output\": \"0.999893\",\n \"txid\": \"feb1fe447f95e1a74cde3fa2b7e3518c75dfd9549d097052767da8da1685080c\"\n }\n}"
}
]
},
@@ -1257,14 +1257,14 @@
"curl": "curl -d'{\"method\": \"channel_create\", \"params\": {\"name\": \"@channel\", \"bid\": \"1.0\", \"featured\": [], \"tags\": [], \"languages\": [], \"locations\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel create @channel 1.0",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_create\", \"params\": {\"name\": \"@channel\", \"bid\": \"1.0\", \"featured\": [], \"tags\": [], \"languages\": [], \"locations\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001ac67ce245e739fbe0755c92794067402f3b866313b5d1bcfff8a8fd4963628f8010000006b483045022100f4812b3ceadd83e2117b24b0fe04b95ed58b9c97925807a4d51182afac46299302200e6721670278dd7dfd35a5e79dd75e4fa451cbdc4c8d9339c08208143bacd5690121036d38ef731f352de7624d22ddc8ed881785313476fde65c094f72253825c8339dffffffff0200e1f5050000000084b508406368616e6e656c4c5d00125a0a583056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a66d7576a914d79bf84451de34c9e51af35b99a6f28f8c7a0a7e88acc462a029000000001976a914674e5620e316224972a18519f69513c70e0ec28a88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mu4MiuQqZpiH9UyFtjjRrTsE8EQMxUfzqR\",\n \"amount\": \"7.999876\",\n \"confirmations\": 2,\n \"height\": 207,\n \"nout\": 1,\n \"timestamp\": 1630567676,\n \"txid\": \"f8283696d48f8affcf1b5d3b3166b8f30274069427c95507be9f735e24ce67ac\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"has_signing_key\": true,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": null,\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mpwBjiw3Z5qyN5S8xFYPMvUuWZktaGakgu\",\n \"amount\": \"6.983769\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.016107\",\n \"total_input\": \"7.999876\",\n \"total_output\": \"7.983769\",\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\"\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"010000000136a1463298ebedc65ef6132a9dfb8bc9fc5834ea8d6d47ffea9fec4786dcd80c010000006b483045022100878a45a883405cccc4f24eed6738a0685337c4297d3edadd85e48157523b51a10220668f0910a62d6b9f11c14a285acfde714031b1a8d4ee58b2f2d05d844636de9b012103572708b3d775cd7b6ef4a95f590df566844a5a740b0805f02426716ddb5a7b2affffffff0200e1f5050000000084b508406368616e6e656c4c5d00125a0a583056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb6d7576a91426fa5a194fef6f82c18a1cc935c7561360ae015088acc462a029000000001976a9144cd145e27e059f57eec811b883a53955fc0cb14388ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n2uwPp8uTPdEmHw43mp8Tq6AG7dNFKo7cW\",\n \"amount\": \"7.999876\",\n \"confirmations\": 2,\n \"height\": 207,\n \"nout\": 1,\n \"timestamp\": 1638473067,\n \"txid\": \"0cd8dc8647ec9feaff476d8dea3458fcc98bfb9d2a13f65ec6edeb983246a136\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"has_signing_key\": true,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": null,\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mnX8MB7dEGJLLf4Jh84v3bsTDnjEqPSdwV\",\n \"amount\": \"6.983769\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.016107\",\n \"total_input\": \"7.999876\",\n \"total_output\": \"7.983769\",\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\"\n }\n}"
},
{
"title": "Create a channel claim with all metadata",
"curl": "curl -d'{\"method\": \"channel_create\", \"params\": {\"name\": \"@bigchannel\", \"bid\": \"1.0\", \"title\": \"Big Channel\", \"description\": \"A channel with lots of videos.\", \"email\": \"creator@smallmedia.com\", \"website_url\": \"http://smallmedia.com\", \"featured\": [], \"tags\": [\"music\", \"art\"], \"languages\": [\"pt-BR\", \"uk\"], \"locations\": [\"BR\", \"UA::Kiyv\"], \"thumbnail_url\": \"http://smallmedia.com/logo.jpg\", \"cover_url\": \"http://smallmedia.com/logo.jpg\", \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel create @bigchannel 1.0 --title=\"Big Channel\" --description=\"A channel with lots of videos.\" --email=\"creator@smallmedia.com\" --tags=music --tags=art --languages=pt-BR --languages=uk --locations=BR --locations=UA::Kiyv --website_url=\"http://smallmedia.com\" --thumbnail_url=\"http://smallmedia.com/logo.jpg\" --cover_url=\"http://smallmedia.com/logo.jpg\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_create\", \"params\": {\"name\": \"@bigchannel\", \"bid\": \"1.0\", \"title\": \"Big Channel\", \"description\": \"A channel with lots of videos.\", \"email\": \"creator@smallmedia.com\", \"website_url\": \"http://smallmedia.com\", \"featured\": [], \"tags\": [\"music\", \"art\"], \"languages\": [\"pt-BR\", \"uk\"], \"locations\": [\"BR\", \"UA::Kiyv\"], \"thumbnail_url\": \"http://smallmedia.com/logo.jpg\", \"cover_url\": \"http://smallmedia.com/logo.jpg\", \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001903415ca80c2891a49dc94a4b6c24745942bef51dc55f698b5351654b763414e010000006a47304402201617c0a2a7c048f691946825cdd66801f69799499706b452fa47b1d2d5aa3391022020bd9a6b460ad76abc5bfe4a6f5a3043550ba8c1770ef66841208ee789c4dbca012103b4a9ce906f0715a69b834409f3a8d6870f54d9615ed6faf5e1257378515fa3daffffffff0200e1f50500000000fd5001b50b406269676368616e6e656c4d25010012ab010a583056301006072a8648ce3d020106052b8104000a0342000486e10e17cdf3265e77b7e95ab833c185c96b81c8ade8557ee9065312e8b131a4d0f507b2a9859daf610db169e106bda6625a208689d9c17e14c818a117ac5d59121663726561746f7240736d616c6c6d656469612e636f6d1a15687474703a2f2f736d616c6c6d656469612e636f6d22202a1e687474703a2f2f736d616c6c6d656469612e636f6d2f6c6f676f2e6a7067420b426967204368616e6e656c4a1e41206368616e6e656c2077697468206c6f7473206f6620766964656f732e52202a1e687474703a2f2f736d616c6c6d656469612e636f6d2f6c6f676f2e6a70675a056d757369635a0361727462050883011820620308ab016a0208206a0908e9011a044b6979766d7576a914e9ee76a5d08f9a103004075aed2633e3e318bd7888ace221d305000000001976a914cbc79ab1ff83ffdbfe069d693519515bcf10accd88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mvi6wBni9ExLJpdeCsbThprKWtLjtMXKsG\",\n \"amount\": \"1.9993355\",\n \"confirmations\": 1,\n \"height\": 210,\n \"nout\": 1,\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n2qsQUx62EbBBiPGWEuZMuATNqC9M5SqeX\",\n \"amount\": \"1.0\",\n \"claim_id\": \"b2e9928d92a2f073161ac304fb02d04b03bbbabc\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"has_signing_key\": true,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"@bigchannel\",\n \"normalized_name\": \"@bigchannel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@bigchannel#b2e9928d92a2f073161ac304fb02d04b03bbbabc\",\n \"timestamp\": null,\n \"txid\": \"b47233c19d326cef94b48baa25a27aaef6d861f74c576758866415dd4e07ce82\",\n \"type\": \"claim\",\n \"value\": {\n \"cover\": {\n \"url\": \"http://smallmedia.com/logo.jpg\"\n },\n \"description\": \"A channel with lots of videos.\",\n \"email\": \"creator@smallmedia.com\",\n \"languages\": [\n \"pt-BR\",\n \"uk\"\n ],\n \"locations\": [\n {\n \"country\": \"BR\"\n },\n {\n \"city\": \"Kiyv\",\n \"country\": \"UA\"\n }\n ],\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000486e10e17cdf3265e77b7e95ab833c185c96b81c8ade8557ee9065312e8b131a4d0f507b2a9859daf610db169e106bda6625a208689d9c17e14c818a117ac5d59\",\n \"public_key_id\": \"n3AMnZt5A4PLyi9aSi4NsmbixkRKMMuoVD\",\n \"tags\": [\n \"music\",\n \"art\"\n ],\n \"thumbnail\": {\n \"url\": \"http://smallmedia.com/logo.jpg\"\n },\n \"title\": \"Big Channel\",\n \"website_url\": \"http://smallmedia.com\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mz6Sak75qd3EYBNPNEsNdJsXbS9ebPx3j6\",\n \"amount\": \"0.9772285\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"b47233c19d326cef94b48baa25a27aaef6d861f74c576758866415dd4e07ce82\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.022107\",\n \"total_input\": \"1.9993355\",\n \"total_output\": \"1.9772285\",\n \"txid\": \"b47233c19d326cef94b48baa25a27aaef6d861f74c576758866415dd4e07ce82\"\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001637fb47c4b1a1704e8f2e0681677096a3bdf3d5aeb15914621196ab7f3145005010000006a473044022021b23a2cf8030e39ef2d8847a874f50d53906bffa59438eb88acfaa43f7e1ba10220395c3972127ab70df1812c52be956fbd0cacef4fb1cf53b96301c49719561801012103b3fab7b529fd5adecf1209f972a98583fa58b86684f8483922747117922eb0cfffffffff0200e1f50500000000fd5001b50b406269676368616e6e656c4d25010012ab010a583056301006072a8648ce3d020106052b8104000a0342000437854772feff0a7bb63d16ca28faec3885cc4a5c476e98866b8379459e87d314e9b3de6102331d379cfe4f6fd6d6582d78f18d4a910a5d0159012e0dbf03bea8121663726561746f7240736d616c6c6d656469612e636f6d1a15687474703a2f2f736d616c6c6d656469612e636f6d22202a1e687474703a2f2f736d616c6c6d656469612e636f6d2f6c6f676f2e6a7067420b426967204368616e6e656c4a1e41206368616e6e656c2077697468206c6f7473206f6620766964656f732e52202a1e687474703a2f2f736d616c6c6d656469612e636f6d2f6c6f676f2e6a70675a056d757369635a0361727462050883011820620308ab016a0208206a0908e9011a044b6979766d7576a914d78b04afb55c235821b4ff94cae1aa934e4ff32f88ace221d305000000001976a9149383f722802f5012db922ac0b9b8df1f07a3fd8d88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mgZ9kB7PNhMPYkypE2wUbyCJtMPQVSNeRJ\",\n \"amount\": \"1.9993355\",\n \"confirmations\": 1,\n \"height\": 210,\n \"nout\": 1,\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n1Ae7TZg6rQ6xqTdfQKLZuuvxvMjKdyUPV\",\n \"amount\": \"1.0\",\n \"claim_id\": \"0fa5da8d6fc28fc5a02b41f3512fe17ad100417d\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"has_signing_key\": true,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"@bigchannel\",\n \"normalized_name\": \"@bigchannel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@bigchannel#0fa5da8d6fc28fc5a02b41f3512fe17ad100417d\",\n \"timestamp\": null,\n \"txid\": \"d99b3109b955d271bb5c813399fe0ab1657e74db938b0f1b78db115283af0f2a\",\n \"type\": \"claim\",\n \"value\": {\n \"cover\": {\n \"url\": \"http://smallmedia.com/logo.jpg\"\n },\n \"description\": \"A channel with lots of videos.\",\n \"email\": \"creator@smallmedia.com\",\n \"languages\": [\n \"pt-BR\",\n \"uk\"\n ],\n \"locations\": [\n {\n \"country\": \"BR\"\n },\n {\n \"city\": \"Kiyv\",\n \"country\": \"UA\"\n }\n ],\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000437854772feff0a7bb63d16ca28faec3885cc4a5c476e98866b8379459e87d314e9b3de6102331d379cfe4f6fd6d6582d78f18d4a910a5d0159012e0dbf03bea8\",\n \"public_key_id\": \"mpg5H8VZcxkLN8bCMxkJnnhjA5NWxVZ8Ev\",\n \"tags\": [\n \"music\",\n \"art\"\n ],\n \"thumbnail\": {\n \"url\": \"http://smallmedia.com/logo.jpg\"\n },\n \"title\": \"Big Channel\",\n \"website_url\": \"http://smallmedia.com\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mtxwiNHPpjCLsCRvXgQGbhAz4zf4tC8CNk\",\n \"amount\": \"0.9772285\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"d99b3109b955d271bb5c813399fe0ab1657e74db938b0f1b78db115283af0f2a\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.022107\",\n \"total_input\": \"1.9993355\",\n \"total_output\": \"1.9772285\",\n \"txid\": \"d99b3109b955d271bb5c813399fe0ab1657e74db938b0f1b78db115283af0f2a\"\n }\n}"
}
]
},
@@ -1386,14 +1386,14 @@
"curl": "curl -d'{\"method\": \"channel_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"resolve\": false, \"no_totals\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"resolve\": false, \"no_totals\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"has_signing_key\": true,\n \"height\": 209,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"has_signing_key\": true,\n \"height\": 209,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
},
{
"title": "Paginate your channel claims",
"curl": "curl -d'{\"method\": \"channel_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"page\": 1, \"page_size\": 20, \"resolve\": false, \"no_totals\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel list --page=1 --page_size=20",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"page\": 1, \"page_size\": 20, \"resolve\": false, \"no_totals\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"has_signing_key\": true,\n \"height\": 209,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"has_signing_key\": true,\n \"height\": 209,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -1588,10 +1588,10 @@
"examples": [
{
"title": "Update a channel claim",
"curl": "curl -d'{\"method\": \"channel_update\", \"params\": {\"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"title\": \"New Channel\", \"featured\": [], \"clear_featured\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"new_signing_key\": false, \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel update 815f8d597ce588c141b001d05f3240b3ef0bf370 --title=\"New Channel\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_update\", \"params\": {\"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"title\": \"New Channel\", \"featured\": [], \"clear_featured\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"new_signing_key\": false, \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000002eb18a9ee3369501c7e6f51c266c5fef90443a568282b62fe91a65689d510a1d3000000006b48304502210083c788a849d27647c1c54417273f824705b1dfbdf7fd8e827942da0473bc2204022077f84b5e33f1f1b32fa77a8d7524613c252dbdfa7871e00f1b895699ffe3ed8501210344a78a03b26f6ce6bb8132537a36caf8826058e936f85ab6f916e844077d9c7effffffffc85e28f5e83b817781de6202e80bc1ac6a39c1259ca12ae8918ac9ac340a42ec000000006a47304402204743e3360b3f4e438d3031ea9543ec6befacbed9a49f52abe5cabbe623d5c2c3022051d91fa11efdbbf789319ffcf68330203d961aee7c26e2eb1da7cb283028e09b012103a58d6ba353d8115d738ea8e2d13cba0600074cb718d568b98f9362a4325ab3e0ffffffff0200e1f50500000000a6b708406368616e6e656c1470f30befb340325fd001b041c188e57c598d5f814c6a00125a0a583056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6420b4e6577204368616e6e656c6d6d76a914d79bf84451de34c9e51af35b99a6f28f8c7a0a7e88ac6ebeea0b000000001976a914a6a5acc19d4111ad37c90d023207650c75fa13d488ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"has_signing_key\": true,\n \"height\": 209,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mp6KzHPSgp4DW57qGcKAWR1S48EvDxnot3\",\n \"amount\": \"1.999604\",\n \"confirmations\": 1,\n \"height\": 209,\n \"nout\": 0,\n \"timestamp\": 1630567676,\n \"txid\": \"ec420a34acc98a91e82aa19c25c1396aacc10be80262de8177813be8f5285ec8\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": -2,\n \"has_signing_key\": true,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": null,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mvi6wBni9ExLJpdeCsbThprKWtLjtMXKsG\",\n \"amount\": \"1.9993355\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.0002685\",\n \"total_input\": \"2.999604\",\n \"total_output\": \"2.9993355\",\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\"\n }\n}"
"curl": "curl -d'{\"method\": \"channel_update\", \"params\": {\"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"title\": \"New Channel\", \"featured\": [], \"clear_featured\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"new_signing_key\": false, \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet channel update add76735eb99e808dae24cbdfa6ceb12029ee794 --title=\"New Channel\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"channel_update\", \"params\": {\"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"title\": \"New Channel\", \"featured\": [], \"clear_featured\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"new_signing_key\": false, \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000002a40b84b1e68fb5c0217f4cd4a176aa508e251c794f04958d34f39387f9773dc0000000006a47304402205a720860e3b4435dfb34dda65fec36b6a6a4f9851e701f3b67ddd8711483f0a0022057a5acb8ce95a58c454a1e03aa6980b37c51fcecf36f4c679185d8fab421a9000121029f23db06243144e274a1a8ec56416f061822d83bb8585a513e267930965d93ddffffffff552e6ef7405639b36d6cb78e16a782a181515a44cfe9aa55900c1b92b19ea2ea000000006a47304402207ffbe1053cb966cd1f23c9bb5cf6152d6a58705633ac7251678b96a6b087bd3c0220361c675caff27ad4aa0533ceb1f53fdf9709ae7c4e64e166dedc4e613a9e27f40121029aa3a7eee76ca1ae759e573f693c5d0b10157484d7b81f064c8e34e9ea4c258cffffffff0200e1f50500000000a6b708406368616e6e656c1494e79e0212eb6cfabd4ce2da08e899eb3567d7ad4c6a00125a0a583056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb420b4e6577204368616e6e656c6d6d76a91426fa5a194fef6f82c18a1cc935c7561360ae015088ac6ebeea0b000000001976a9140b628417e87d2e9cca5c7f8651c5da466d1c35a388ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"has_signing_key\": true,\n \"height\": 209,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mwyeEqGbtNjYDsoibvztKzDjkLRtAvLkFQ\",\n \"amount\": \"1.999604\",\n \"confirmations\": 1,\n \"height\": 209,\n \"nout\": 0,\n \"timestamp\": 1638473067,\n \"txid\": \"eaa29eb1921b0c9055aae9cf445a5181a182a7168eb76c6db3395640f76e2e55\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": -2,\n \"has_signing_key\": true,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": null,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n {\n \"address\": \"mgZ9kB7PNhMPYkypE2wUbyCJtMPQVSNeRJ\",\n \"amount\": \"1.9993355\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.0002685\",\n \"total_input\": \"2.999604\",\n \"total_output\": \"2.9993355\",\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\"\n }\n}"
}
]
}
@@ -1702,14 +1702,14 @@
"curl": "curl -d'{\"method\": \"claim_list\", \"params\": {\"claim_type\": [], \"claim_id\": [], \"name\": [], \"is_spent\": false, \"channel_id\": [], \"has_source\": false, \"has_no_source\": false, \"resolve\": false, \"no_totals\": false, \"include_received_tips\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet claim list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"claim_list\", \"params\": {\"claim_type\": [], \"claim_id\": [], \"name\": [], \"is_spent\": false, \"channel_id\": [], \"has_source\": false, \"has_no_source\": false, \"resolve\": false, \"no_totals\": false, \"include_received_tips\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
},
{
"title": "Paginate your claims",
"curl": "curl -d'{\"method\": \"claim_list\", \"params\": {\"claim_type\": [], \"claim_id\": [], \"name\": [], \"is_spent\": false, \"channel_id\": [], \"has_source\": false, \"has_no_source\": false, \"page\": 1, \"page_size\": 20, \"resolve\": false, \"no_totals\": false, \"include_received_tips\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet claim list --page=1 --page_size=20",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"claim_list\", \"params\": {\"claim_type\": [], \"claim_id\": [], \"name\": [], \"is_spent\": false, \"channel_id\": [], \"has_source\": false, \"has_no_source\": false, \"page\": 1, \"page_size\": 20, \"resolve\": false, \"no_totals\": false, \"include_received_tips\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -1867,28 +1867,34 @@
"description": "limit by total value (initial claim value plus all tips and supports received), this amount is blank until claim has reached activation height (supports equality constraints)",
"is_required": false
},
{
"name": "trending_score",
"type": "int",
"description": "limit by trending score (supports equality constraints)",
"is_required": false
},
{
"name": "trending_group",
"type": "int",
"description": "group numbers 1 through 4 representing the trending groups of the content: 4 means content is trending globally and independently, 3 means content is not trending globally but is trending independently (locally), 2 means it is trending globally but not independently and 1 means it's not trending globally or locally (supports equality constraints)",
"description": "DEPRECATED - instead please use trending_score",
"is_required": false
},
{
"name": "trending_mixed",
"type": "int",
"description": "trending amount taken from the global or local value depending on the trending group: 4 - global value, 3 - local value, 2 - global value, 1 - local value (supports equality constraints)",
"description": "DEPRECATED - instead please use trending_score",
"is_required": false
},
{
"name": "trending_local",
"type": "int",
"description": "trending value calculated relative only to the individual contents past history (supports equality constraints)",
"description": "DEPRECATED - instead please use trending_score",
"is_required": false
},
{
"name": "trending_global",
"type": "int",
"description": "trending value calculated relative to all trending content globally (supports equality constraints)",
"description": "DEPRECATED - instead please use trending_score",
"is_required": false
},
{
@@ -2067,14 +2073,14 @@
"curl": "curl -d'{\"method\": \"claim_search\", \"params\": {\"claim_ids\": [], \"channel\": \"@channel\", \"channel_ids\": [], \"not_channel_ids\": [], \"has_channel_signature\": false, \"valid_channel_signature\": false, \"invalid_channel_signature\": false, \"is_controlling\": false, \"stream_types\": [], \"media_types\": [], \"any_tags\": [], \"all_tags\": [], \"not_tags\": [], \"any_languages\": [], \"all_languages\": [], \"not_languages\": [], \"any_locations\": [], \"all_locations\": [], \"not_locations\": [], \"order_by\": [], \"no_totals\": false, \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"remove_duplicates\": false, \"has_source\": false, \"has_no_source\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet claim search --channel=@channel",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"claim_search\", \"params\": {\"claim_ids\": [], \"channel\": \"@channel\", \"channel_ids\": [], \"not_channel_ids\": [], \"has_channel_signature\": false, \"valid_channel_signature\": false, \"invalid_channel_signature\": false, \"is_controlling\": false, \"stream_types\": [], \"media_types\": [], \"any_tags\": [], \"all_tags\": [], \"not_tags\": [], \"any_languages\": [], \"all_languages\": [], \"not_languages\": [], \"any_locations\": [], \"all_locations\": [], \"not_locations\": [], \"order_by\": [], \"no_totals\": false, \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"remove_duplicates\": false, \"has_source\": false, \"has_no_source\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"blocked\": {\n \"channels\": [],\n \"total\": 0\n },\n \"items\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8/astream#1\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 213,\n \"creation_height\": 213,\n \"creation_timestamp\": 1630567677,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263187,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"short_url\": \"lbry://astream#1\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 209,\n \"claims_in_channel\": 1,\n \"creation_height\": 209,\n \"creation_timestamp\": 1630567676,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263183,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"short_url\": \"lbry://@channel#8\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"blocked\": {\n \"channels\": [],\n \"total\": 0\n },\n \"items\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a/astream#f\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 214,\n \"creation_height\": 213,\n \"creation_timestamp\": 1638473068,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 714,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"short_url\": \"lbry://astream#f\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 210,\n \"claims_in_channel\": 1,\n \"creation_height\": 209,\n \"creation_timestamp\": 1638473067,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 710,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"short_url\": \"lbry://@channel#a\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
},
{
"title": "Search for claims matching a name",
"curl": "curl -d'{\"method\": \"claim_search\", \"params\": {\"name\": \"astream\", \"claim_ids\": [], \"channel_ids\": [], \"not_channel_ids\": [], \"has_channel_signature\": false, \"valid_channel_signature\": false, \"invalid_channel_signature\": false, \"is_controlling\": false, \"stream_types\": [], \"media_types\": [], \"any_tags\": [], \"all_tags\": [], \"not_tags\": [], \"any_languages\": [], \"all_languages\": [], \"not_languages\": [], \"any_locations\": [], \"all_locations\": [], \"not_locations\": [], \"order_by\": [], \"no_totals\": false, \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"remove_duplicates\": false, \"has_source\": false, \"has_no_source\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet claim search --name=\"astream\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"claim_search\", \"params\": {\"name\": \"astream\", \"claim_ids\": [], \"channel_ids\": [], \"not_channel_ids\": [], \"has_channel_signature\": false, \"valid_channel_signature\": false, \"invalid_channel_signature\": false, \"is_controlling\": false, \"stream_types\": [], \"media_types\": [], \"any_tags\": [], \"all_tags\": [], \"not_tags\": [], \"any_languages\": [], \"all_languages\": [], \"not_languages\": [], \"any_locations\": [], \"all_locations\": [], \"not_locations\": [], \"order_by\": [], \"no_totals\": false, \"include_purchase_receipt\": false, \"include_is_my_output\": false, \"remove_duplicates\": false, \"has_source\": false, \"has_no_source\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"blocked\": {\n \"channels\": [],\n \"total\": 0\n },\n \"items\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8/astream#1\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 213,\n \"creation_height\": 213,\n \"creation_timestamp\": 1630567677,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263187,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"short_url\": \"lbry://astream#1\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 209,\n \"claims_in_channel\": 1,\n \"creation_height\": 209,\n \"creation_timestamp\": 1630567676,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263183,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"short_url\": \"lbry://@channel#8\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"blocked\": {\n \"channels\": [],\n \"total\": 0\n },\n \"items\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a/astream#f\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 214,\n \"creation_height\": 213,\n \"creation_timestamp\": 1638473068,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 714,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"short_url\": \"lbry://astream#f\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 210,\n \"claims_in_channel\": 1,\n \"creation_height\": 209,\n \"creation_timestamp\": 1638473067,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 710,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"short_url\": \"lbry://@channel#a\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
}
]
}
@@ -2262,10 +2268,10 @@
"examples": [
{
"title": "Create a collection of one stream",
"curl": "curl -d'{\"method\": \"collection_create\", \"params\": {\"name\": \"tom\", \"bid\": \"1.0\", \"claims\": [\"138446450b727414a2b9e9529341dc621a9e5470\"], \"allow_duplicate_name\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet collection create --name=tom --bid=1.0 --channel_id=815f8d597ce588c141b001d05f3240b3ef0bf370 --claims=138446450b727414a2b9e9529341dc621a9e5470",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"collection_create\", \"params\": {\"name\": \"tom\", \"bid\": \"1.0\", \"claims\": [\"138446450b727414a2b9e9529341dc621a9e5470\"], \"allow_duplicate_name\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001699afb39155979142df871283129396c6601768b1c01b5a9dce8792b5d36e9ca010000006a47304402206915c5b6a023c9c379090ea6c0ce7b17f1947306b4e0e210ad480d1d9404c69e02205c5890d6c4de4e7dd4ec2eff7bbbd5fe4278191db787c9b69e2be1226293cc4f012103224e580ffe93840ae8b96b235e9f4c58570f57fa980774df8f8a93f42fc19f4fffffffff0200e1f5050000000091b503746f6d4c6f0170f30befb340325fd001b041c188e57c598d5f81a84a2000d064fd6747420b1753734b13fea27b2bdaeb2004bb58f647ea91386f1c1e5ccd40d5e3c9301a178c886d971fa2fca582be131d23de6e7aef4cf693c61a1812160a1470549e1a62dc419352e9b9a21474720b454684136d7576a914636031eb074819012fb131c5d785ab4a002cad0a88ac202c7e17000000001976a91491431e22d948adcac9e1e2f32a4aef46fc24744c88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mpszEfmsqHT4nQGbY4wwh3BtBuAhRAf3fV\",\n \"amount\": \"4.947555\",\n \"confirmations\": 2,\n \"height\": 215,\n \"nout\": 1,\n \"timestamp\": 1630567677,\n \"txid\": \"cae9365d2b79e8dca9b5011c8b7601666c3929312871f82d1479591539fb9a69\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mpaQRnoSqgSRrTu2MBLDvDDUM4rnh73NVu\",\n \"amount\": \"1.0\",\n \"claim_id\": \"47ead22e98caee988a0709ab559c21575e4e7556\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"is_channel_signature_valid\": true,\n \"meta\": {},\n \"name\": \"tom\",\n \"normalized_name\": \"tom\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://tom#47ead22e98caee988a0709ab559c21575e4e7556\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 7,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": null,\n \"txid\": \"54248f6961e5c0f06c7363ed05c7bf8ac3f6bc941ea416b553a5e0009bc465b7\",\n \"type\": \"claim\",\n \"value\": {\n \"claims\": [\n \"138446450b727414a2b9e9529341dc621a9e5470\"\n ]\n },\n \"value_type\": \"collection\"\n },\n {\n \"address\": \"mtm2gHnthSQ2n61dnJSeZCd873eHByQQ8r\",\n \"amount\": \"3.941448\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"54248f6961e5c0f06c7363ed05c7bf8ac3f6bc941ea416b553a5e0009bc465b7\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.006107\",\n \"total_input\": \"4.947555\",\n \"total_output\": \"4.941448\",\n \"txid\": \"54248f6961e5c0f06c7363ed05c7bf8ac3f6bc941ea416b553a5e0009bc465b7\"\n }\n}"
"curl": "curl -d'{\"method\": \"collection_create\", \"params\": {\"name\": \"tom\", \"bid\": \"1.0\", \"claims\": [\"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"], \"allow_duplicate_name\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet collection create --name=tom --bid=1.0 --channel_id=add76735eb99e808dae24cbdfa6ceb12029ee794 --claims=f6d2070225511eeb8a1c33f1d4bdb76e22716547",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"collection_create\", \"params\": {\"name\": \"tom\", \"bid\": \"1.0\", \"claims\": [\"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"], \"allow_duplicate_name\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001d691c54b384f56d01fa9432be257bc153bb9e303f31695ac0953834483cce310010000006b483045022100a48cf3fdf47f6ae1588826d44468b860cfebc8a85b7c12da7a3771ed10c5672b0220246345b3f22fbb6fdcb3fd1bfdfc97ce98f9147ecebf28163404cb3296c2ec48012102bc70b5c31d2dcefac29faf37d185bc4edca649625110e2cb6b64845eb64ff7a9ffffffff0200e1f5050000000091b503746f6d4c6f0194e79e0212eb6cfabd4ce2da08e899eb3567d7ad110f2d85972d77b9502f4fd44a7d54041245238f7f6db7a46bf03b7407b4c1906cf9db06596d6799387fe97e36e7690422ddd603abf5656cf71735023933d9d61a1812160a14476571226eb7bdd4f1331c8aeb1e51250207d2f66d7576a9146ac84980e3ce8595bb0d48441af6b93e58c6339c88ac202c7e17000000001976a914e39c5c993cdfaa76059dd7181aa7f5b0e006cf4b88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n1TjmPqMddpvhVYt2zwL47GeVkdVdsnX61\",\n \"amount\": \"4.947555\",\n \"confirmations\": 2,\n \"height\": 215,\n \"nout\": 1,\n \"timestamp\": 1638473068,\n \"txid\": \"10e3cc8344835309ac9516f303e3b93b15bc57e22b43a91fd0564f384bc591d6\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mqFZrafoJXcwUnUpNVzRxQZZszBxo798ot\",\n \"amount\": \"1.0\",\n \"claim_id\": \"ec48c448d010dc817a8b6ac9d85ff3e0f97e55a3\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"is_channel_signature_valid\": true,\n \"meta\": {},\n \"name\": \"tom\",\n \"normalized_name\": \"tom\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://tom#ec48c448d010dc817a8b6ac9d85ff3e0f97e55a3\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 7,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": null,\n \"txid\": \"6a39355eeafc8960170561d45c507f7cacb519d1dfa1cf8d3f7777b97502c241\",\n \"type\": \"claim\",\n \"value\": {\n \"claims\": [\n \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"\n ]\n },\n \"value_type\": \"collection\"\n },\n {\n \"address\": \"n2GSzoTb1HBSkeDzaF1g8TGZtvSqsFCxKF\",\n \"amount\": \"3.941448\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"6a39355eeafc8960170561d45c507f7cacb519d1dfa1cf8d3f7777b97502c241\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.006107\",\n \"total_input\": \"4.947555\",\n \"total_output\": \"4.941448\",\n \"txid\": \"6a39355eeafc8960170561d45c507f7cacb519d1dfa1cf8d3f7777b97502c241\"\n }\n}"
}
]
},
@@ -2317,7 +2323,7 @@
"curl": "curl -d'{\"method\": \"collection_list\", \"params\": {\"resolve_claims\": 1, \"resolve\": true}}' http://localhost:5279/",
"lbrynet": "lbrynet collection list --resolve --resolve_claims=1",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"collection_list\", \"params\": {\"resolve_claims\": 1, \"resolve\": true}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mpaQRnoSqgSRrTu2MBLDvDDUM4rnh73NVu\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8/tom#4\",\n \"claim_id\": \"47ead22e98caee988a0709ab559c21575e4e7556\",\n \"claim_op\": \"create\",\n \"claims\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8/astream#1\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 213,\n \"creation_height\": 213,\n \"creation_timestamp\": 1630567677,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263187,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"short_url\": \"lbry://astream#1\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#8\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 8,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 209,\n \"claims_in_channel\": 2,\n \"creation_height\": 209,\n \"creation_timestamp\": 1630567676,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263183,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"short_url\": \"lbry://@channel#8\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"confirmations\": 1,\n \"height\": 217,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 217,\n \"creation_height\": 217,\n \"creation_timestamp\": 1630567677,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 263191,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 217,\n \"trending_global\": 0.0,\n \"trending_group\": 0,\n \"trending_local\": 0.0,\n \"trending_mixed\": 0.0\n },\n \"name\"Line truncated
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mqFZrafoJXcwUnUpNVzRxQZZszBxo798ot\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a/tom#e\",\n \"claim_id\": \"ec48c448d010dc817a8b6ac9d85ff3e0f97e55a3\",\n \"claim_op\": \"create\",\n \"claims\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a/astream#f\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 214,\n \"creation_height\": 213,\n \"creation_timestamp\": 1638473068,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 714,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 213\n },\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"short_url\": \"lbry://astream#f\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"canonical_url\": \"lbry://@channel#a\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 8,\n \"has_signing_key\": false,\n \"height\": 210,\n \"meta\": {\n \"activation_height\": 210,\n \"claims_in_channel\": 2,\n \"creation_height\": 209,\n \"creation_timestamp\": 1638473067,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 710,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 209\n },\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"short_url\": \"lbry://@channel#a\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"confirmations\": 1,\n \"height\": 217,\n \"is_channel_signature_valid\": true,\n \"meta\": {\n \"activation_height\": 217,\n \"creation_height\": 217,\n \"creation_timestamp\": 1638473068,\n \"effective_amount\": \"1.0\",\n \"expiration_height\": 717,\n \"is_controlling\": true,\n \"reposted\": 0,\n \"support_amount\": \"0.0\",\n \"take_over_height\": 217\n },\n \"name\": \"tom\",\n \"normalized_name\": \"tom\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://tom#ec48c448d010dc817a8b6ac9d85ff3e0f97e55a3\",\n \"short_url\": \"lbry://tom#e\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 8,\n Line truncated
}
]
},
@@ -2573,9 +2579,9 @@
"examples": [
{
"title": "Delete a file",
"curl": "curl -d'{\"method\": \"file_delete\", \"params\": {\"delete_from_download_dir\": false, \"delete_all\": false, \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\"}}' http://localhost:5279/",
"lbrynet": "lbrynet file delete --claim_id=\"138446450b727414a2b9e9529341dc621a9e5470\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_delete\", \"params\": {\"delete_from_download_dir\": false, \"delete_all\": false, \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\"}}).json()",
"curl": "curl -d'{\"method\": \"file_delete\", \"params\": {\"delete_from_download_dir\": false, \"delete_all\": false, \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"}}' http://localhost:5279/",
"lbrynet": "lbrynet file delete --claim_id=\"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_delete\", \"params\": {\"delete_from_download_dir\": false, \"delete_all\": false, \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": true\n}"
}
]
@@ -2736,14 +2742,14 @@
"curl": "curl -d'{\"method\": \"file_list\", \"params\": {\"reverse\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet file list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_list\", \"params\": {\"reverse\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"added_on\": null,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": -1,\n \"content_fee\": null,\n \"download_directory\": null,\n \"download_path\": null,\n \"file_name\": null,\n \"height\": -1,\n \"is_fully_reflected\": true,\n \"key\": \"5ff5a0e1491f8a7b1ccba093b2963f9e\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0170f30befb340325fd001b041c188e57c598d5f81dd671a785932239cd26885251126637352f3825ab44b6e2c2cef378b182f870b1f9050035fac0bb1956acf02bf696de46bc2526f2881f5b98c371975290eb8b00a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706231343670663171180b22186170706c69636174696f6e2f6f637465742d73747265616d32305b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"77bf955798d4e67da2f877931216008c79f97a5788167e8bbb239b5f94ff13508c40c18604f29220efa7c79756f1a2ff\",\n \"stream_name\": \"tmpb146pf1q\",\n \"streaming_url\": \"http://localhost:5280/stream/5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"suggested_file_name\": \"tmpb146pf1q\",\n \"timestamp\": null,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 0\n },\n {\n \"added_on\": null,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"7e64a9dc5a51f064999b5faca3275de1e8875882\",\n \"claim_name\": \"blank-image\",\n \"completed\": false,\n \"confirmations\": -1,\n \"content_fee\": null,\n \"download_directory\": null,\n \"download_path\": null,\n \"file_name\": null,\n \"height\": -1,\n \"is_fully_reflected\": true,\n \"key\": \"12b8744a005681b025c6062f2ef64430\",\n \"metadata\": {\n \"author\": \"Picaso\",\n \"description\": \"A blank PNG that is 5x7.\",\n \"fee\": {\n \"address\": \"mnxa6w92YbS3NCccPANEzYHJY6bJGM7iUu\",\n \"amount\": \"0.3\",\n \"currency\": \"LBC\"\n },\n \"image\": {\n \"height\": 7,\n \"width\": 5\n },\n \"languages\": [\n \"en\"\n ],\n \"license\": \"Public Domain\",\n \"license_url\": \"http://public-domain.org\",\n \"locations\": [\n {\n \"city\": \"Manchester\",\n \"country\": \"US\",\n \"state\": \"NH\"\n }\n ],\n \"release_time\": \"1630567651\",\n \"source\": {\n \"hash\": \"6c7df435d412c603390f593ef658c199817c7830ba3f16b7eadd8f99fa50e85dbd0d2b3dc61eadc33fe096e3872d1545\",\n \"media_type\": \"image/png\",\n \"name\": \"tmpev348s69.png\",\n \"sd_hash\": \"3654f14df3da475a8ef3dcf81ca855ca719102cfe3d905d48d77ad5a264d067e723e3cf76c41f8715e4ca9a6fe96d0d2\",\n \"size\": \"99\"\n },\n \"stream_type\": \"image\",\n \"tags\": [\n \"blank\",\n \"art\"\n ],\n \"thumbnail\": {\n \"url\": \"http://smallmedia.com/thumbnail.jpg\"\n },\n \"title\": \"BlLine truncated
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"added_on\": null,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": -1,\n \"content_fee\": null,\n \"download_directory\": null,\n \"download_path\": null,\n \"file_name\": null,\n \"height\": -1,\n \"is_fully_reflected\": true,\n \"key\": \"6f57dcfb5e64cc5172c07929ac8720b6\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0194e79e0212eb6cfabd4ce2da08e899eb3567d7ad7d593e2e84647ec1122144433db0a4321c77eb6e1276c12b6934064b07547cbfadd7a9aecd5fc53460eb27274cfd8914aad2a302c2c483511e46fc0473c886050a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706a35776e35787338180b22186170706c69636174696f6e2f6f637465742d73747265616d3230945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"3c18d82c0c262331785e3e6c45dd05f6fe4f6918bb08fe552772e230fefe2cb079cf171c9ba8255be068fda898119d23\",\n \"stream_name\": \"tmpj5wn5xs8\",\n \"streaming_url\": \"http://localhost:5280/stream/945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"suggested_file_name\": \"tmpj5wn5xs8\",\n \"timestamp\": null,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 0\n },\n {\n \"added_on\": null,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"9fc7dfa2c958d662a1f5599440701dc1a2651581\",\n \"claim_name\": \"blank-image\",\n \"completed\": false,\n \"confirmations\": -1,\n \"content_fee\": null,\n \"download_directory\": null,\n \"download_path\": null,\n \"file_name\": null,\n \"height\": -1,\n \"is_fully_reflected\": true,\n \"key\": \"011089e17ff836aa6434e3926f5d0f67\",\n \"metadata\": {\n \"author\": \"Picaso\",\n \"description\": \"A blank PNG that is 5x7.\",\n \"fee\": {\n \"address\": \"mxGqo1Ps9zX3v2RwFJQWw9qtGDVHsTZUL3\",\n \"amount\": \"0.3\",\n \"currency\": \"LBC\"\n },\n \"image\": {\n \"height\": 7,\n \"width\": 5\n },\n \"languages\": [\n \"en\"\n ],\n \"license\": \"Public Domain\",\n \"license_url\": \"http://public-domain.org\",\n \"locations\": [\n {\n \"city\": \"Manchester\",\n \"country\": \"US\",\n \"state\": \"NH\"\n }\n ],\n \"release_time\": \"1638473040\",\n \"source\": {\n \"hash\": \"6c7df435d412c603390f593ef658c199817c7830ba3f16b7eadd8f99fa50e85dbd0d2b3dc61eadc33fe096e3872d1545\",\n \"media_type\": \"image/png\",\n \"name\": \"tmpy4uy2zdj.png\",\n \"sd_hash\": \"b4ee14ed36a23aaf499a5ea0b06c8fd8876be8613e19323fc109c7f06768c7cc6b979ecebf8fae6a145ac1e5583409dd\",\n \"size\": \"99\"\n },\n \"stream_type\": \"image\",\n \"tags\": [\n \"blank\",\n \"art\"\n ],\n \"thumbnail\": {\n \"url\": \"http://smallmedia.com/thumbnail.jpg\"\n },\n \"title\": \"BlLine truncated
},
{
"title": "List files matching a parameter",
"curl": "curl -d'{\"method\": \"file_list\", \"params\": {\"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\", \"reverse\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet file list --claim_id=\"138446450b727414a2b9e9529341dc621a9e5470\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_list\", \"params\": {\"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\", \"reverse\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"added_on\": null,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": 4,\n \"content_fee\": null,\n \"download_directory\": null,\n \"download_path\": null,\n \"file_name\": null,\n \"height\": 214,\n \"is_fully_reflected\": true,\n \"key\": \"5ff5a0e1491f8a7b1ccba093b2963f9e\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0170f30befb340325fd001b041c188e57c598d5f81dd671a785932239cd26885251126637352f3825ab44b6e2c2cef378b182f870b1f9050035fac0bb1956acf02bf696de46bc2526f2881f5b98c371975290eb8b00a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706231343670663171180b22186170706c69636174696f6e2f6f637465742d73747265616d32305b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"77bf955798d4e67da2f877931216008c79f97a5788167e8bbb239b5f94ff13508c40c18604f29220efa7c79756f1a2ff\",\n \"stream_name\": \"tmpb146pf1q\",\n \"streaming_url\": \"http://localhost:5280/stream/5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"suggested_file_name\": \"tmpb146pf1q\",\n \"timestamp\": 1630567677,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 0\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
"curl": "curl -d'{\"method\": \"file_list\", \"params\": {\"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\", \"reverse\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet file list --claim_id=\"f6d2070225511eeb8a1c33f1d4bdb76e22716547\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_list\", \"params\": {\"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\", \"reverse\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"added_on\": null,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": 4,\n \"content_fee\": null,\n \"download_directory\": null,\n \"download_path\": null,\n \"file_name\": null,\n \"height\": 214,\n \"is_fully_reflected\": true,\n \"key\": \"6f57dcfb5e64cc5172c07929ac8720b6\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0194e79e0212eb6cfabd4ce2da08e899eb3567d7ad7d593e2e84647ec1122144433db0a4321c77eb6e1276c12b6934064b07547cbfadd7a9aecd5fc53460eb27274cfd8914aad2a302c2c483511e46fc0473c886050a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706a35776e35787338180b22186170706c69636174696f6e2f6f637465742d73747265616d3230945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"3c18d82c0c262331785e3e6c45dd05f6fe4f6918bb08fe552772e230fefe2cb079cf171c9ba8255be068fda898119d23\",\n \"stream_name\": \"tmpj5wn5xs8\",\n \"streaming_url\": \"http://localhost:5280/stream/945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"suggested_file_name\": \"tmpj5wn5xs8\",\n \"timestamp\": 1638473068,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 0\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 1,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -2860,10 +2866,10 @@
"examples": [
{
"title": "Save a file to the downloads directory",
"curl": "curl -d'{\"method\": \"file_save\", \"params\": {\"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\"}}' http://localhost:5279/",
"lbrynet": "lbrynet file save --sd_hash=\"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_save\", \"params\": {\"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"added_on\": 1630567653,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": 4,\n \"content_fee\": null,\n \"download_directory\": \"/tmp/tmpbzob8k7r\",\n \"download_path\": \"/tmp/tmpbzob8k7r/tmpb146pf1q_1\",\n \"file_name\": \"tmpb146pf1q_1\",\n \"height\": 214,\n \"is_fully_reflected\": false,\n \"key\": \"5ff5a0e1491f8a7b1ccba093b2963f9e\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0170f30befb340325fd001b041c188e57c598d5f81dd671a785932239cd26885251126637352f3825ab44b6e2c2cef378b182f870b1f9050035fac0bb1956acf02bf696de46bc2526f2881f5b98c371975290eb8b00a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706231343670663171180b22186170706c69636174696f6e2f6f637465742d73747265616d32305b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"77bf955798d4e67da2f877931216008c79f97a5788167e8bbb239b5f94ff13508c40c18604f29220efa7c79756f1a2ff\",\n \"stream_name\": \"tmpb146pf1q\",\n \"streaming_url\": \"http://localhost:5280/stream/5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"suggested_file_name\": \"tmpb146pf1q\",\n \"timestamp\": 1630567677,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 11\n }\n}"
"curl": "curl -d'{\"method\": \"file_save\", \"params\": {\"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\"}}' http://localhost:5279/",
"lbrynet": "lbrynet file save --sd_hash=\"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"file_save\", \"params\": {\"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\"}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"added_on\": 1638473043,\n \"blobs_completed\": 1,\n \"blobs_in_stream\": 1,\n \"blobs_remaining\": 0,\n \"channel_claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"channel_name\": \"@channel\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_name\": \"astream\",\n \"completed\": true,\n \"confirmations\": 4,\n \"content_fee\": null,\n \"download_directory\": \"/tmp/tmpd_du30j6\",\n \"download_path\": \"/tmp/tmpd_du30j6/tmpj5wn5xs8_1\",\n \"file_name\": \"tmpj5wn5xs8_1\",\n \"height\": 214,\n \"is_fully_reflected\": false,\n \"key\": \"6f57dcfb5e64cc5172c07929ac8720b6\",\n \"metadata\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"mime_type\": \"application/octet-stream\",\n \"nout\": 0,\n \"outpoint\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a:0\",\n \"points_paid\": 0.0,\n \"protobuf\": \"0194e79e0212eb6cfabd4ce2da08e899eb3567d7ad7d593e2e84647ec1122144433db0a4321c77eb6e1276c12b6934064b07547cbfadd7a9aecd5fc53460eb27274cfd8914aad2a302c2c483511e46fc0473c886050a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706a35776e35787338180b22186170706c69636174696f6e2f6f637465742d73747265616d3230945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"purchase_receipt\": null,\n \"reflector_progress\": 0,\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"status\": \"finished\",\n \"stopped\": true,\n \"stream_hash\": \"3c18d82c0c262331785e3e6c45dd05f6fe4f6918bb08fe552772e230fefe2cb079cf171c9ba8255be068fda898119d23\",\n \"stream_name\": \"tmpj5wn5xs8\",\n \"streaming_url\": \"http://localhost:5280/stream/945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"suggested_file_name\": \"tmpj5wn5xs8\",\n \"timestamp\": 1638473068,\n \"total_bytes\": 16,\n \"total_bytes_lower_bound\": 0,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"uploading_to_reflector\": false,\n \"written_bytes\": 11\n }\n}"
}
]
},
@@ -3144,7 +3150,7 @@
"curl": "curl -d'{\"method\": \"settings_get\", \"params\": {}}' http://localhost:5279/",
"lbrynet": "lbrynet settings get",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"settings_get\", \"params\": {}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"allowed_origin\": \"\",\n \"announce_head_and_sd_only\": true,\n \"api\": \"localhost:5279\",\n \"audio_encoder\": \"aac -b:a 160k\",\n \"blob_download_timeout\": 30.0,\n \"blob_lru_cache_size\": 0,\n \"blob_storage_limit\": 0,\n \"blockchain_name\": \"lbrycrd_regtest\",\n \"coin_selection_strategy\": \"standard\",\n \"components_to_skip\": [\n \"dht\",\n \"upnp\",\n \"hash_announcer\",\n \"peer_protocol_server\",\n \"libtorrent_component\"\n ],\n \"concurrent_blob_announcers\": 10,\n \"concurrent_hub_requests\": 32,\n \"concurrent_reflector_uploads\": 10,\n \"config\": \"/tmp/tmp8_tji32x/daemon_settings.yml\",\n \"data_dir\": \"/tmp/tmpbzob8k7r\",\n \"download_dir\": \"/tmp/tmpbzob8k7r\",\n \"download_timeout\": 30.0,\n \"ffmpeg_path\": \"\",\n \"fixed_peer_delay\": 2.0,\n \"fixed_peers\": [\n [\n \"127.0.0.1\",\n 5567\n ]\n ],\n \"hub_timeout\": 30.0,\n \"jurisdiction\": null,\n \"known_dht_nodes\": [],\n \"lbryum_servers\": [\n [\n \"localhost\",\n 50002\n ]\n ],\n \"max_connections_per_download\": 4,\n \"max_key_fee\": {\n \"amount\": 50.0,\n \"currency\": \"USD\"\n },\n \"max_wallet_server_fee\": \"0.0\",\n \"network_interface\": \"0.0.0.0\",\n \"node_rpc_timeout\": 5.0,\n \"peer_connect_timeout\": 3.0,\n \"prometheus_port\": 0,\n \"reflect_streams\": true,\n \"reflector_servers\": [\n [\n \"127.0.0.1\",\n 5566\n ]\n ],\n \"save_blobs\": true,\n \"save_files\": true,\n \"save_resolved_claims\": true,\n \"share_usage_data\": false,\n \"split_buckets_under_index\": 1,\n \"streaming_get\": true,\n \"streaming_server\": \"localhost:5280\",\n \"tcp_port\": 3333,\n \"track_bandwidth\": true,\n \"transaction_cache_size\": 10000,\n \"udp_port\": 4444,\n \"use_upnp\": false,\n \"video_bitrate_maximum\": 5000000,\n \"video_encoder\": \"libx264 -crf 24 -preset faster -pix_fmt yuv420p\",\n \"video_scaler\": \"-vf \\\"scale=if(gte(iw\\\\,ih)\\\\,min(1920\\\\,iw)\\\\,-2):if(lt(iw\\\\,ih)\\\\,min(1920\\\\,ih)\\\\,-2)\\\" -maxrate 5500K -bufsize 5000K\",\n \"volume_analysis_time\": 240,\n \"volume_filter\": \"\",\n \"wallet_dir\": \"/tmp/tmpbzob8k7r\",\n \"wallets\": [\n \"default_wallet\"\n ]\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"allowed_origin\": \"\",\n \"announce_head_and_sd_only\": true,\n \"api\": \"localhost:5279\",\n \"audio_encoder\": \"aac -b:a 160k\",\n \"blob_download_timeout\": 30.0,\n \"blob_lru_cache_size\": 0,\n \"blob_storage_limit\": 0,\n \"blockchain_name\": \"lbrycrd_regtest\",\n \"coin_selection_strategy\": \"standard\",\n \"components_to_skip\": [\n \"dht\",\n \"upnp\",\n \"hash_announcer\",\n \"peer_protocol_server\",\n \"libtorrent_component\"\n ],\n \"concurrent_blob_announcers\": 10,\n \"concurrent_hub_requests\": 32,\n \"concurrent_reflector_uploads\": 10,\n \"config\": \"/tmp/tmpbb_x1g3t/daemon_settings.yml\",\n \"data_dir\": \"/tmp/tmpd_du30j6\",\n \"download_dir\": \"/tmp/tmpd_du30j6\",\n \"download_timeout\": 30.0,\n \"ffmpeg_path\": \"\",\n \"fixed_peer_delay\": 2.0,\n \"fixed_peers\": [\n [\n \"127.0.0.1\",\n 5567\n ]\n ],\n \"hub_timeout\": 30.0,\n \"jurisdiction\": null,\n \"known_dht_nodes\": [],\n \"lbryum_servers\": [\n [\n \"localhost\",\n 50002\n ]\n ],\n \"max_connections_per_download\": 4,\n \"max_key_fee\": {\n \"amount\": 50.0,\n \"currency\": \"USD\"\n },\n \"max_wallet_server_fee\": \"0.0\",\n \"network_interface\": \"0.0.0.0\",\n \"network_storage_limit\": 0,\n \"node_rpc_timeout\": 5.0,\n \"peer_connect_timeout\": 3.0,\n \"prometheus_port\": 0,\n \"reflect_streams\": true,\n \"reflector_servers\": [\n [\n \"127.0.0.1\",\n 5566\n ]\n ],\n \"save_blobs\": true,\n \"save_files\": true,\n \"save_resolved_claims\": true,\n \"share_usage_data\": false,\n \"split_buckets_under_index\": 1,\n \"streaming_get\": true,\n \"streaming_server\": \"localhost:5280\",\n \"tcp_port\": 4444,\n \"track_bandwidth\": true,\n \"transaction_cache_size\": 10000,\n \"udp_port\": 4444,\n \"use_upnp\": false,\n \"video_bitrate_maximum\": 5000000,\n \"video_encoder\": \"libx264 -crf 24 -preset faster -pix_fmt yuv420p\",\n \"video_scaler\": \"-vf \\\"scale=if(gte(iw\\\\,ih)\\\\,min(1920\\\\,iw)\\\\,-2):if(lt(iw\\\\,ih)\\\\,min(1920\\\\,ih)\\\\,-2)\\\" -maxrate 5500K -bufsize 5000K\",\n \"volume_analysis_time\": 240,\n \"volume_filter\": \"\",\n \"wallet_dir\": \"/tmp/tmpd_du30j6\",\n \"wallets\": [\n \"default_wallet\"\n ]\n }\n}"
}
]
},
@@ -3219,10 +3225,10 @@
"examples": [
{
"title": "Abandon a stream claim",
"curl": "curl -d'{\"method\": \"stream_abandon\", \"params\": {\"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\", \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream abandon 138446450b727414a2b9e9529341dc621a9e5470",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_abandon\", \"params\": {\"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\", \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001c334ca5d1b1370052f2b60fc6a653b7cad13d9f4bf1b0a3bed9a3782922deb7e000000006b483045022100c1b03c480b02034ae6529403c51de347ccb713d9ab073efc31a7e535a54eec9c0220453935ecc35b3e6e7754e74999a73581cd243780c85e31d26f18a49a6f072e1d012102b63aea1d12a0766f1eb0c7b235875334c766ee60fb856900e5b69300463350edffffffff0134b7f505000000001976a914c1195a1c4c9dc0c3700019ed92f2c5a0919527d788ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"height\": 214,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"my7y5Zi2nYZGbHcMmGukzz3dzBKeLYnJej\",\n \"amount\": \"0.999893\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"9523336623012e46f92be3b1e41773f39afe1be21a96791ecd5a3ccbcef6c59f\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000107\",\n \"total_input\": \"1.0\",\n \"total_output\": \"0.999893\",\n \"txid\": \"9523336623012e46f92be3b1e41773f39afe1be21a96791ecd5a3ccbcef6c59f\"\n }\n}"
"curl": "curl -d'{\"method\": \"stream_abandon\", \"params\": {\"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\", \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream abandon f6d2070225511eeb8a1c33f1d4bdb76e22716547",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_abandon\", \"params\": {\"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\", \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"01000000019aa49a19233261481023a4a919d6138ed9394377f2559f57c63e115fb6708606000000006b483045022100c287a54d5267662668ad5e8d386883cd776d8701533eea97e7b192f1fb1360eb0220257199938cce2d9e614dafd4ab658da781b613bbfd76c49ef690b8ac19f76b05012102b7ca42cfacdebcea518ff9d5025aef29b0710d47615d06bdbdda793c2599fdc6ffffffff0134b7f505000000001976a914010161196aef2c978630c17d49bf4ec2e6236a1088ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"height\": 214,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mfcGd5eMKvrbyQy4BQZnfGD9VZe5G7CoG9\",\n \"amount\": \"0.999893\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 0,\n \"timestamp\": null,\n \"txid\": \"c76720b1968703759af2c3def26449c7e922d6513a874faff720bcb5b79b5e26\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.000107\",\n \"total_input\": \"1.0\",\n \"total_output\": \"0.999893\",\n \"txid\": \"c76720b1968703759af2c3def26449c7e922d6513a874faff720bcb5b79b5e26\"\n }\n}"
}
]
},
@@ -3453,17 +3459,17 @@
"examples": [
{
"title": "Create a stream claim without metadata",
"curl": "curl -d'{\"method\": \"stream_create\", \"params\": {\"name\": \"astream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpb146pf1q\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream create astream 1.0 /tmp/tmpb146pf1q",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_create\", \"params\": {\"name\": \"astream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpb146pf1q\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001eb18a9ee3369501c7e6f51c266c5fef90443a568282b62fe91a65689d510a1d3010000006b483045022100a67bc9c7f59e9020652fe90a1f9011e56838997d8d7f8bfa81a48e4bd9f9aa2002205fbfc12942355ff689831bb0f1b5c5546dabf157868c4cf47e0ebf10835ac626012102a12507ff1f9dd615bdcbfe995067515ab164c0afca82c811f6512c71e73f003effffffff0200e1f50500000000bab5076173747265616d4c94000a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706231343670663171180b22186170706c69636174696f6e2f6f637465742d73747265616d32305b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad6d7576a91461f99b5d5a8b835507022131e7614ca5aed5c69f88ac38fb9423000000001976a9149bef62d82af2a9804e5bf3ecd2c0f3fd7d8f25b388ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mpwBjiw3Z5qyN5S8xFYPMvUuWZktaGakgu\",\n \"amount\": \"6.983769\",\n \"confirmations\": 4,\n \"height\": 209,\n \"nout\": 1,\n \"timestamp\": 1630567676,\n \"txid\": \"d3a110d58956a691fe622b2868a54304f9fec566c2516f7e1c506933eea918eb\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"timestamp\": null,\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mujTodHBxPXwMP5evjeH151ZsGDA8CU1jc\",\n \"amount\": \"5.969662\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.014107\",\n \"total_input\": \"6.983769\",\n \"total_output\": \"6.969662\",\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\"\n }\n}"
"curl": "curl -d'{\"method\": \"stream_create\", \"params\": {\"name\": \"astream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpj5wn5xs8\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream create astream 1.0 /tmp/tmpj5wn5xs8",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_create\", \"params\": {\"name\": \"astream\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpj5wn5xs8\", \"validate_file\": false, \"optimize_file\": false, \"tags\": [], \"languages\": [], \"locations\": [], \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001a40b84b1e68fb5c0217f4cd4a176aa508e251c794f04958d34f39387f9773dc0010000006a473044022038bdaf4255d0ec50d9fb97f192a43a3ea7a8565493b0dcc4453607b888f63776022048ec2ffc0443d46d4500b4da735a96a513e9e66187c6e5431bb7db234fb7cf340121027d111c377cd319149d0131739916c850f5b185302fe22b3fc359526d312beec5ffffffff0200e1f50500000000bab5076173747265616d4c94000a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706a35776e35787338180b22186170706c69636174696f6e2f6f637465742d73747265616d3230945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb6d7576a914f8d4a3682f66393ade07c62f149410d454125e8288ac38fb9423000000001976a91456c1787d450f02cfc09fe27d266a2ebe4bfd176388ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mnX8MB7dEGJLLf4Jh84v3bsTDnjEqPSdwV\",\n \"amount\": \"6.983769\",\n \"confirmations\": 4,\n \"height\": 209,\n \"nout\": 1,\n \"timestamp\": 1638473067,\n \"txid\": \"c03d77f98793f3348d95044f791c258e50aa76a1d44c7f21c0b58fe6b1840ba4\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"timestamp\": null,\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"moRgB8R4kZYFM3ewT4jKXoZhX3mkAPqY3a\",\n \"amount\": \"5.969662\",\n \"confirmations\": -2,\n \"height\": -2,\n \"nout\": 1,\n \"timestamp\": null,\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\",\n \"type\": \"payment\"\n }\n ],\n \"total_fee\": \"0.014107\",\n \"total_input\": \"6.983769\",\n \"total_output\": \"6.969662\",\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\"\n }\n}"
},
{
"title": "Create an image stream claim with all metadata and fee",
"curl": "curl -d'{\"method\": \"stream_create\", \"params\": {\"name\": \"blank-image\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpev348s69.png\", \"validate_file\": false, \"optimize_file\": false, \"fee_currency\": \"LBC\", \"fee_amount\": \"0.3\", \"title\": \"Blank Image\", \"description\": \"A blank PNG that is 5x7.\", \"author\": \"Picaso\", \"tags\": [\"blank\", \"art\"], \"languages\": [\"en\"], \"locations\": [\"US:NH:Manchester\"], \"license\": \"Public Domain\", \"license_url\": \"http://public-domain.org\", \"thumbnail_url\": \"http://smallmedia.com/thumbnail.jpg\", \"release_time\": 1630567651, \"channel_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream create blank-image 1.0 /tmp/tmpev348s69.png --tags=blank --tags=art --languages=en --locations=US:NH:Manchester --fee_currency=LBC --fee_amount=0.3 --title=\"Blank Image\" --description=\"A blank PNG that is 5x7.\" --author=Picaso --license=\"Public Domain\" --license_url=http://public-domain.org --thumbnail_url=\"http://smallmedia.com/thumbnail.jpg\" --release_time=1630567651 --channel_id=\"815f8d597ce588c141b001d05f3240b3ef0bf370\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_create\", \"params\": {\"name\": \"blank-image\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpev348s69.png\", \"validate_file\": false, \"optimize_file\": false, \"fee_currency\": \"LBC\", \"fee_amount\": \"0.3\", \"title\": \"Blank Image\", \"description\": \"A blank PNG that is 5x7.\", \"author\": \"Picaso\", \"tags\": [\"blank\", \"art\"], \"languages\": [\"en\"], \"locations\": [\"US:NH:Manchester\"], \"license\": \"Public Domain\", \"license_url\": \"http://public-domain.org\", \"thumbnail_url\": \"http://smallmedia.com/thumbnail.jpg\", \"release_time\": 1630567651, \"channel_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000001b50e2ed0154414ccfb821ffa6164cfa7a7dd74988d0ff9a4769b857eb1c9fe10010000006b483045022100e6ce7b778a40884b08b9e1ec4826bebc31f4fa9a52c99c50b0162368f058126c02207c47e0867bf387b6239f3db0faa44b2345f488e000310cf8c93c6066ee5cd828012103c17d084dc82340293e6fcfbc805d82fad7453851b2f1b51843a2eb2d78fbe096ffffffff0200e1f50500000000fddc01b50b626c616e6b2d696d6167654db1010170f30befb340325fd001b041c188e57c598d5f81cef45e51ab648cff8ace11e8922e3a974c83b444443442c7d1672cefa340cd9a06d0354ce6aee791c91d76fe9259499ded55e773cd91b19d6515aca52a0a2a220ae6010a82010a306c7df435d412c603390f593ef658c199817c7830ba3f16b7eadd8f99fa50e85dbd0d2b3dc61eadc33fe096e3872d1545120f746d7065763334387336392e706e6718632209696d6167652f706e6732303654f14df3da475a8ef3dcf81ca855ca719102cfe3d905d48d77ad5a264d067e723e3cf76c41f8715e4ca9a6fe96d0d2120650696361736f1a0d5075626c696320446f6d61696e2218687474703a2f2f7075626c69632d646f6d61696e2e6f726728e3f9c189063222080112196f51a12bb815be629bb9d43fdee188d93eb15513de27e775e6188087a70e520408051007420b426c616e6b20496d6167654a184120626c616e6b20504e472074686174206973203578372e52252a23687474703a2f2f736d616c6c6d656469612e636f6d2f7468756d626e61696c2e6a70675a05626c616e6b5a03617274620208016a1308ec0112024e481a0a4d616e636865737465726d7576a91451a12bb815be629bb9d43fdee188d93eb15513de88acac5e7d1d000000001976a91466b37cf116dfdbdc949634387565a22df7e4c7fd88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mujTodHBxPXwMP5evjeH151ZsGDA8CU1jc\",\n \"amount\": \"5.969662\",\n \"confirmations\": 2,\n \"height\": 213,\n \"nout\": 1,\n \"timestamp\": 1630567677,\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mnxa6w92YbS3NCccPANEzYHJY6bJGM7iUu\",\n \"amount\": \"1.0\",\n \"claim_id\": \"7e64a9dc5a51f064999b5faca3275de1e8875882\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"is_channel_signature_valid\": true,\n \"meta\": {},\n \"name\": \"blank-image\",\n \"normalized_name\": \"blank-image\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://blank-image#7e64a9dc5a51f064999b5faca3275de1e8875882\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": null,\n \"txid\": \"cae9365d2b79e8dca9b5011c8b7601666c3929312871f82d1479591539fb9a69\",\n \"type\": \"claim\",\n \"value\": {\n \"author\": \"Picaso\",\n \"description\": \"A blank PNG that is 5x7.\",\n \"fee\": {\n \"address\": \"mnxa6w92YbS3NCccPANEzYHJY6bJGM7iUu\",\n \"amount\": \"0.3\",\n \"currency\": \"LBC\"\n },\n \"image\": {\n \"height\": 7,\n \"width\": 5\n },\n \"languages\": [\n \"en\"\n ],\n \"license\": \"Public Domain\",\n \"license_url\": \"http://public-domain.org\",\n \"locations\": [\n {\n \"city\": \"Manchester\",\n \"country\": \"US\",\n \"state\": \"NH\"\n }\n ],\n \"release_time\": \"1630567651\",\n \"source\": {\n \"hash\": \"6c7df435d412c603390f593ef658c199817c7830ba3f16b7eadd8f99fa50e85dbd0d2b3dc61eadc33fe096e3872d1545\",\n \"media_type\": \"image/png\",\n \"name\": \"tmpev348s69.png\",\n \"sd_hash\": \"3654f14df3da475a8ef3dcf81ca855ca719102cfe3d905d48d77ad5a264d067e723e3cf76c41f8715e4ca9a6fe96d0d2\",\n \"size\": \"99\"\n },\n \"stream_type\": \"image\",\n \"tags\": [\n \"blank\",\n \"art\"\n ],\n \"thuLine truncated
"curl": "curl -d'{\"method\": \"stream_create\", \"params\": {\"name\": \"blank-image\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpy4uy2zdj.png\", \"validate_file\": false, \"optimize_file\": false, \"fee_currency\": \"LBC\", \"fee_amount\": \"0.3\", \"title\": \"Blank Image\", \"description\": \"A blank PNG that is 5x7.\", \"author\": \"Picaso\", \"tags\": [\"blank\", \"art\"], \"languages\": [\"en\"], \"locations\": [\"US:NH:Manchester\"], \"license\": \"Public Domain\", \"license_url\": \"http://public-domain.org\", \"thumbnail_url\": \"http://smallmedia.com/thumbnail.jpg\", \"release_time\": 1638473040, \"channel_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream create blank-image 1.0 /tmp/tmpy4uy2zdj.png --tags=blank --tags=art --languages=en --locations=US:NH:Manchester --fee_currency=LBC --fee_amount=0.3 --title=\"Blank Image\" --description=\"A blank PNG that is 5x7.\" --author=Picaso --license=\"Public Domain\" --license_url=http://public-domain.org --thumbnail_url=\"http://smallmedia.com/thumbnail.jpg\" --release_time=1638473040 --channel_id=\"add76735eb99e808dae24cbdfa6ceb12029ee794\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_create\", \"params\": {\"name\": \"blank-image\", \"bid\": \"1.0\", \"file_path\": \"/tmp/tmpy4uy2zdj.png\", \"validate_file\": false, \"optimize_file\": false, \"fee_currency\": \"LBC\", \"fee_amount\": \"0.3\", \"title\": \"Blank Image\", \"description\": \"A blank PNG that is 5x7.\", \"author\": \"Picaso\", \"tags\": [\"blank\", \"art\"], \"languages\": [\"en\"], \"locations\": [\"US:NH:Manchester\"], \"license\": \"Public Domain\", \"license_url\": \"http://public-domain.org\", \"thumbnail_url\": \"http://smallmedia.com/thumbnail.jpg\", \"release_time\": 1638473040, \"channel_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"01000000013e0c691705e692b7b6882158b63f021e3643a6d3393ee6efaabd53fa628ba77a010000006b4830450221009729e174e46570b3c035fb94e90b772b2018ac8e48821f6750ffd8fa0c5154050220778ca1e135f3590fd71282011ab0a6c8037b5b5d465ed06fd3c833a19408644f012102dbac7323952c2bd76538e94dac55dcf4b8d68febac90c0c0d00082718a0c46d2ffffffff0200e1f50500000000fddc01b50b626c616e6b2d696d6167654db1010194e79e0212eb6cfabd4ce2da08e899eb3567d7ad3efd7f997a6102834b4c7faf79435f7f8c33b760fd0854503a8ff71fcc088d28d76165b649cd284326e4103a35e5f7b8a87766c08406cf881855268d37d0f4390ae6010a82010a306c7df435d412c603390f593ef658c199817c7830ba3f16b7eadd8f99fa50e85dbd0d2b3dc61eadc33fe096e3872d1545120f746d7079347579327a646a2e706e6718632209696d6167652f706e673230b4ee14ed36a23aaf499a5ea0b06c8fd8876be8613e19323fc109c7f06768c7cc6b979ecebf8fae6a145ac1e5583409dd120650696361736f1a0d5075626c696320446f6d61696e2218687474703a2f2f7075626c69632d646f6d61696e2e6f726728d0baa48d063222080112196fb7cee3654d70cef1c8eb5aeb77c708509a9a2b94ae8ba0fc188087a70e520408051007420b426c616e6b20496d6167654a184120626c616e6b20504e472074686174206973203578372e52252a23687474703a2f2f736d616c6c6d656469612e636f6d2f7468756d626e61696c2e6a70675a05626c616e6b5a03617274620208016a1308ec0112024e481a0a4d616e636865737465726d7576a914b7cee3654d70cef1c8eb5aeb77c708509a9a2b9488acac5e7d1d000000001976a914dac6cf36e2894a1158283c2323709d5ad3a51dab88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"moRgB8R4kZYFM3ewT4jKXoZhX3mkAPqY3a\",\n \"amount\": \"5.969662\",\n \"confirmations\": 2,\n \"height\": 213,\n \"nout\": 1,\n \"timestamp\": 1638473068,\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mxGqo1Ps9zX3v2RwFJQWw9qtGDVHsTZUL3\",\n \"amount\": \"1.0\",\n \"claim_id\": \"9fc7dfa2c958d662a1f5599440701dc1a2651581\",\n \"claim_op\": \"create\",\n \"confirmations\": -2,\n \"height\": -2,\n \"is_channel_signature_valid\": true,\n \"meta\": {},\n \"name\": \"blank-image\",\n \"normalized_name\": \"blank-image\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://blank-image#9fc7dfa2c958d662a1f5599440701dc1a2651581\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": null,\n \"txid\": \"10e3cc8344835309ac9516f303e3b93b15bc57e22b43a91fd0564f384bc591d6\",\n \"type\": \"claim\",\n \"value\": {\n \"author\": \"Picaso\",\n \"description\": \"A blank PNG that is 5x7.\",\n \"fee\": {\n \"address\": \"mxGqo1Ps9zX3v2RwFJQWw9qtGDVHsTZUL3\",\n \"amount\": \"0.3\",\n \"currency\": \"LBC\"\n },\n \"image\": {\n \"height\": 7,\n \"width\": 5\n },\n \"languages\": [\n \"en\"\n ],\n \"license\": \"Public Domain\",\n \"license_url\": \"http://public-domain.org\",\n \"locations\": [\n {\n \"city\": \"Manchester\",\n \"country\": \"US\",\n \"state\": \"NH\"\n }\n ],\n \"release_time\": \"1638473040\",\n \"source\": {\n \"hash\": \"6c7df435d412c603390f593ef658c199817c7830ba3f16b7eadd8f99fa50e85dbd0d2b3dc61eadc33fe096e3872d1545\",\n \"media_type\": \"image/png\",\n \"name\": \"tmpy4uy2zdj.png\",\n \"sd_hash\": \"b4ee14ed36a23aaf499a5ea0b06c8fd8876be8613e19323fc109c7f06768c7cc6b979ecebf8fae6a145ac1e5583409dd\",\n \"size\": \"99\"\n },\n \"stream_type\": \"image\",\n \"tags\": [\n \"blank\",\n \"art\"\n ],\n \"thuLine truncated
}
]
},
@@ -3533,14 +3539,14 @@
"curl": "curl -d'{\"method\": \"stream_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"resolve\": false, \"no_totals\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"resolve\": false, \"no_totals\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"create\",\n \"confirmations\": 2,\n \"height\": 213,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": true,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"timestamp\": 1630567677,\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"create\",\n \"confirmations\": 2,\n \"height\": 213,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": true,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"timestamp\": 1638473068,\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
},
{
"title": "Paginate your stream claims",
"curl": "curl -d'{\"method\": \"stream_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"page\": 1, \"page_size\": 20, \"resolve\": false, \"no_totals\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream list --page=1 --page_size=20",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_list\", \"params\": {\"name\": [], \"claim_id\": [], \"is_spent\": false, \"page\": 1, \"page_size\": 20, \"resolve\": false, \"no_totals\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1630567677,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"create\",\n \"confirmations\": 2,\n \"height\": 213,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": true,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"timestamp\": 1630567677,\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": 1,\n \"height\": 214,\n \"is_channel_signature_valid\": true,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": false,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 5,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": 1638473068,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"create\",\n \"confirmations\": 2,\n \"height\": 213,\n \"is_internal_transfer\": false,\n \"is_my_input\": true,\n \"is_my_output\": true,\n \"is_spent\": true,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"timestamp\": 1638473068,\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -3879,10 +3885,10 @@
"examples": [
{
"title": "Update a stream claim to add channel",
"curl": "curl -d'{\"method\": \"stream_update\", \"params\": {\"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\", \"validate_file\": false, \"optimize_file\": false, \"clear_fee\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"channel_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"clear_channel\": false, \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream update 138446450b727414a2b9e9529341dc621a9e5470 --channel_id=\"815f8d597ce588c141b001d05f3240b3ef0bf370\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_update\", \"params\": {\"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\", \"validate_file\": false, \"optimize_file\": false, \"clear_fee\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"channel_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\", \"clear_channel\": false, \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"0100000002b50e2ed0154414ccfb821ffa6164cfa7a7dd74988d0ff9a4769b857eb1c9fe10000000006a47304402206fddf588d2a39d5a5061eca78bba303758a16545209097796f9182cf090c2c7302203def11a7418c51bfea86f07ad5ae19a039b556e249d055e0b16fe5b8f5dc3eea012102b63aea1d12a0766f1eb0c7b235875334c766ee60fb856900e5b69300463350edffffffff82ce074edd1564865867574cf761d8f6ae7aa225aa8bb494ef6c329dc13372b4010000006b483045022100a5eda62fbaa798da5ba005598edca5bffccad96fbdd528af51e0ef2e14cd096002205f85f2bc192dfd486fa6df8a4a3f0b4849fd48dd434ed77f4cc7d572558852bb012103dc87ecad1cd1e0d2879c20be930f913307e7feb91b5b3a8f49fc3252baf473f4ffffffff0200e1f50500000000fd2301b7076173747265616d1470549e1a62dc419352e9b9a21474720b454684134ce80170f30befb340325fd001b041c188e57c598d5f81dd671a785932239cd26885251126637352f3825ab44b6e2c2cef378b182f870b1f9050035fac0bb1956acf02bf696de46bc2526f2881f5b98c371975290eb8b00a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706231343670663171180b22186170706c69636174696f6e2f6f637465742d73747265616d32305b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad6d6d76a91461f99b5d5a8b835507022131e7614ca5aed5c69f88ac32a0d205000000001976a914d62a8c4306aa05143093d535373f2aca556c965088ac00000000\",\n \"inputs\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"height\": 213,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"timestamp\": 1630567677,\n \"txid\": \"10fec9b17e859b76a4f90f8d9874dda7a7cf6461fa1f82fbcc144415d02e0eb5\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpb146pf1q\",\n \"sd_hash\": \"5b99f91996841d06c64c7f0aa35aa3e3e9dfb79e88b74138dd6cf919551dc661f7e571b78069b61bed9f5c12cc77c6ad\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mz6Sak75qd3EYBNPNEsNdJsXbS9ebPx3j6\",\n \"amount\": \"0.9772285\",\n \"confirmations\": 3,\n \"height\": 211,\n \"nout\": 1,\n \"timestamp\": 1630567676,\n \"txid\": \"b47233c19d326cef94b48baa25a27aaef6d861f74c576758866415dd4e07ce82\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"mpSzreoT2ukyzd3c5h2Sf9vEedb6MvHX59\",\n \"amount\": \"1.0\",\n \"claim_id\": \"138446450b727414a2b9e9529341dc621a9e5470\",\n \"claim_op\": \"update\",\n \"confirmations\": -2,\n \"height\": -2,\n \"is_channel_signature_valid\": true,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#138446450b727414a2b9e9529341dc621a9e5470\",\n \"signing_channel\": {\n \"address\": \"n1AzRGYFcKbdXVAD8WnM2x8kT3HTr7L9f7\",\n \"amount\": \"1.0\",\n \"claim_id\": \"815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#815f8d597ce588c141b001d05f3240b3ef0bf370\",\n \"timestamp\": 1630567676,\n \"txid\": \"4e4163b7541635b598f655dc51ef2b944547c2b6a494dc491a89c280ca153490\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a03420004d423b3b41984ef96038ca03278bdcfd3147f88fa3054d0706a2e8c8e8064dee8dcef501042c1ec2ed062565f3119613ca475fd85f958d5ccc9e3c39beb4fc7a6\",\n \"public_key_id\": \"mhz4rWgbySXKByVvkfkhRGGGbmQgy59v2Z\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": null,\n \"txid\": \"7eeb2d9282379aed3b0a1bbff4d913ad7c3b656afc602b2f0570131b5dca34c3\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n Line truncated
"curl": "curl -d'{\"method\": \"stream_update\", \"params\": {\"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\", \"validate_file\": false, \"optimize_file\": false, \"clear_fee\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"channel_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"clear_channel\": false, \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}' http://localhost:5279/",
"lbrynet": "lbrynet stream update f6d2070225511eeb8a1c33f1d4bdb76e22716547 --channel_id=\"add76735eb99e808dae24cbdfa6ceb12029ee794\"",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"stream_update\", \"params\": {\"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\", \"validate_file\": false, \"optimize_file\": false, \"clear_fee\": false, \"tags\": [], \"clear_tags\": false, \"languages\": [], \"clear_languages\": false, \"locations\": [], \"clear_locations\": false, \"channel_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\", \"clear_channel\": false, \"channel_account_id\": [], \"funding_account_ids\": [], \"preview\": false, \"blocking\": false, \"replace\": false}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"height\": -2,\n \"hex\": \"01000000023e0c691705e692b7b6882158b63f021e3643a6d3393ee6efaabd53fa628ba77a000000006b483045022100fe93a500d582fbb93ac2b4b0c9a19c03f785c0b6be0e3b3074a663959aeaa39702207ecbaf8bdd81b0a95033c91028fe0255cf036b2f95c1ada51fb69e36ad70f931012102b7ca42cfacdebcea518ff9d5025aef29b0710d47615d06bdbdda793c2599fdc6ffffffff2a0faf835211db781b0f8b93db747e65b10afe9933815cbb71d255b909319bd9010000006a47304402200aee2e4aef0083a74d0881d9df424d2e00959a10c6701a8f4d8382bf670dc0ea022072c9e84a2912ad2b6b76d272113bfefdbf6849f87b3eb84f65b1e43d91c0fbe80121038fcc29f05e2d2c72efbf9845bac77c11f60e0a1dde067538b8b9a75c4e1073a2ffffffff0200e1f50500000000fd2301b7076173747265616d14476571226eb7bdd4f1331c8aeb1e51250207d2f64ce80194e79e0212eb6cfabd4ce2da08e899eb3567d7ad7d593e2e84647ec1122144433db0a4321c77eb6e1276c12b6934064b07547cbfadd7a9aecd5fc53460eb27274cfd8914aad2a302c2c483511e46fc0473c886050a90010a8d010a30fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd120b746d706a35776e35787338180b22186170706c69636174696f6e2f6f637465742d73747265616d3230945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb6d6d76a914f8d4a3682f66393ade07c62f149410d454125e8288ac32a0d205000000001976a914c265c0c4da4081e880cd5e151f17fdc61e45807b88ac00000000\",\n \"inputs\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"create\",\n \"confirmations\": 1,\n \"height\": 213,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"timestamp\": 1638473068,\n \"txid\": \"7aa78b62fa53bdaaefe63e39d3a643361e023fb6582188b6b792e60517690c3e\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n \"name\": \"tmpj5wn5xs8\",\n \"sd_hash\": \"945bfb89aee0053cd5ab346afa7a3ea673c4bd68b440e91727a2212887eaeca6e7d52f29533217edaa3991b073392fcb\",\n \"size\": \"11\"\n },\n \"stream_type\": \"binary\"\n },\n \"value_type\": \"stream\"\n },\n {\n \"address\": \"mtxwiNHPpjCLsCRvXgQGbhAz4zf4tC8CNk\",\n \"amount\": \"0.9772285\",\n \"confirmations\": 3,\n \"height\": 211,\n \"nout\": 1,\n \"timestamp\": 1638473067,\n \"txid\": \"d99b3109b955d271bb5c813399fe0ab1657e74db938b0f1b78db115283af0f2a\",\n \"type\": \"payment\"\n }\n ],\n \"outputs\": [\n {\n \"address\": \"n4CebzkzEW2CKUMKZhMhzFuEEDRKcesPLh\",\n \"amount\": \"1.0\",\n \"claim_id\": \"f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"claim_op\": \"update\",\n \"confirmations\": -2,\n \"height\": -2,\n \"is_channel_signature_valid\": true,\n \"meta\": {},\n \"name\": \"astream\",\n \"normalized_name\": \"astream\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://astream#f6d2070225511eeb8a1c33f1d4bdb76e22716547\",\n \"signing_channel\": {\n \"address\": \"mj53t6s8aEjq9zSoRe8XGxkVoVf4qBEYwC\",\n \"amount\": \"1.0\",\n \"claim_id\": \"add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"claim_op\": \"update\",\n \"confirmations\": 4,\n \"has_signing_key\": true,\n \"height\": 210,\n \"meta\": {},\n \"name\": \"@channel\",\n \"normalized_name\": \"@channel\",\n \"nout\": 0,\n \"permanent_url\": \"lbry://@channel#add76735eb99e808dae24cbdfa6ceb12029ee794\",\n \"timestamp\": 1638473067,\n \"txid\": \"055014f3b76a1921469115eb5a3ddf3b6a09771668e0f2e804171a4b7cb47f63\",\n \"type\": \"claim\",\n \"value\": {\n \"public_key\": \"3056301006072a8648ce3d020106052b8104000a0342000460ce4a80d624b65a5790576572ff76fdada28f7a414af45e2e10215c457f838e6b80088d0cb029cedb0b6872235a5497d0cb330445a779b7bb7e67fc153f2acb\",\n \"public_key_id\": \"mqdnuka4BhqJKtdnNbVnwyoWCGabAw4Fs1\",\n \"title\": \"New Channel\"\n },\n \"value_type\": \"channel\"\n },\n \"timestamp\": null,\n \"txid\": \"068670b65f113ec6579f55f2774339d98e13d619a9a4231048613223199aa49a\",\n \"type\": \"claim\",\n \"value\": {\n \"source\": {\n \"hash\": \"fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd\",\n \"media_type\": \"application/octet-stream\",\n Line truncated
}
]
}
@@ -4264,7 +4270,7 @@
"curl": "curl -d'{\"method\": \"transaction_list\", \"params\": {}}' http://localhost:5279/",
"lbrynet": "lbrynet transaction list",
"python": "requests.post(\"http://localhost:5279\", json={\"method\": \"transaction_list\", \"params\": {}}).json()",
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"abandon_info\": [],\n \"claim_info\": [],\n \"confirmations\": 1,\n \"date\": \"2021-09-02 10:27\",\n \"fee\": \"-0.000124\",\n \"purchase_info\": [],\n \"support_info\": [],\n \"timestamp\": 1630567676,\n \"txid\": \"f8283696d48f8affcf1b5d3b3166b8f30274069427c95507be9f735e24ce67ac\",\n \"update_info\": [],\n \"value\": \"0.0\"\n },\n {\n \"abandon_info\": [],\n \"claim_info\": [],\n \"confirmations\": 7,\n \"date\": \"2021-09-02 10:27\",\n \"fee\": \"0.0\",\n \"purchase_info\": [],\n \"support_info\": [],\n \"timestamp\": 1630567675,\n \"txid\": \"e21900d30b27c4807756e101738860e7eaf8b69425651e48b4c46938340af0e9\",\n \"update_info\": [],\n \"value\": \"10.0\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
"output": "{\n \"jsonrpc\": \"2.0\",\n \"result\": {\n \"items\": [\n {\n \"abandon_info\": [],\n \"claim_info\": [],\n \"confirmations\": 1,\n \"date\": \"2021-12-02 14:24\",\n \"fee\": \"-0.000124\",\n \"purchase_info\": [],\n \"support_info\": [],\n \"timestamp\": 1638473067,\n \"txid\": \"0cd8dc8647ec9feaff476d8dea3458fcc98bfb9d2a13f65ec6edeb983246a136\",\n \"update_info\": [],\n \"value\": \"0.0\"\n },\n {\n \"abandon_info\": [],\n \"claim_info\": [],\n \"confirmations\": 7,\n \"date\": \"2021-12-02 14:24\",\n \"fee\": \"0.0\",\n \"purchase_info\": [],\n \"support_info\": [],\n \"timestamp\": 1638473066,\n \"txid\": \"f2826af5fa075c7f9af7397adbf4c88a6cfba623c6da1df3eee4c833c09f40e2\",\n \"update_info\": [],\n \"value\": \"10.0\"\n }\n ],\n \"page\": 1,\n \"page_size\": 20,\n \"total_items\": 2,\n \"total_pages\": 1\n }\n}"
}
]
},
@@ -5062,4 +5068,4 @@
}
]
}
}
}
+1 -1
View File
@@ -1,2 +1,2 @@
__version__ = "0.105.0"
__version__ = "0.106.0"
version = tuple(map(int, __version__.split('.'))) # pylint: disable=invalid-name
+32 -13
View File
@@ -14,32 +14,51 @@ class DiskSpaceManager:
self.running = False
self.task = None
self.analytics = analytics
self._used_space_bytes = None
async def get_free_space_mb(self, is_network_blob=False):
limit_mb = self.config.network_storage_limit if is_network_blob else self.config.blob_storage_limit
space_used_mb = await self.get_space_used_mb()
space_used_mb = space_used_mb['network_storage'] if is_network_blob else space_used_mb['content_storage']
return max(0, limit_mb - space_used_mb)
async def get_space_used_bytes(self):
return await self.db.get_stored_blob_disk_usage()
self._used_space_bytes = await self.db.get_stored_blob_disk_usage()
return self._used_space_bytes
async def get_space_used_mb(self):
return int(await self.get_space_used_bytes()/1024.0/1024.0)
async def get_space_used_mb(self, cached=True):
cached = cached and self._used_space_bytes is not None
space_used_bytes = self._used_space_bytes if cached else await self.get_space_used_bytes()
return {key: int(value/1024.0/1024.0) for key, value in space_used_bytes.items()}
async def clean(self):
space_used_bytes = await self.get_space_used_bytes()
storage_limit = self.config.blob_storage_limit*1024*1024 if self.config.blob_storage_limit else None
await self._clean(False)
await self._clean(True)
async def _clean(self, is_network_blob=False):
space_used_mb = await self.get_space_used_mb(cached=False)
if is_network_blob:
space_used_mb = space_used_mb['network_storage']
else:
space_used_mb = space_used_mb['content_storage'] + space_used_mb['private_storage']
storage_limit_mb = self.config.network_storage_limit if is_network_blob else self.config.blob_storage_limit
if self.analytics:
asyncio.create_task(self.analytics.send_disk_space_used(space_used_bytes, storage_limit))
if not storage_limit:
return 0
asyncio.create_task(
self.analytics.send_disk_space_used(space_used_mb, storage_limit_mb, is_network_blob)
)
delete = []
available = storage_limit - space_used_bytes
if available > 0:
available = storage_limit_mb - space_used_mb
if storage_limit_mb == 0 if not is_network_blob else available >= 0:
return 0
for blob_hash, file_size, _ in await self.db.get_stored_blobs(is_mine=False):
for blob_hash, file_size, _ in await self.db.get_stored_blobs(is_mine=False, is_network_blob=is_network_blob):
delete.append(blob_hash)
available += file_size
if available > 0:
available += int(file_size/1024.0/1024.0)
if available >= 0:
break
if delete:
await self.db.stop_all_files()
await self.blob_manager.delete_blobs(delete, delete_from_db=True)
self._used_space_bytes = None
return len(delete)
async def cleaning_loop(self):
+8 -14
View File
@@ -3,6 +3,7 @@ import typing
import logging
from lbry.utils import cache_concurrent
from lbry.blob_exchange.client import request_blob
from lbry.dht.node import get_kademlia_peers_from_hosts
if typing.TYPE_CHECKING:
from lbry.conf import Config
from lbry.dht.node import Node
@@ -87,7 +88,6 @@ class BlobDownloader:
if blob.get_is_verified():
return blob
self.is_running.set()
tried_for_this_blob: typing.Set['KademliaPeer'] = set()
try:
while not blob.get_is_verified() and self.is_running.is_set():
batch: typing.Set['KademliaPeer'] = set(self.connections.keys())
@@ -97,24 +97,15 @@ class BlobDownloader:
"%s running, %d peers, %d ignored, %d active, %s connections", blob_hash[:6],
len(batch), len(self.ignored), len(self.active_connections), len(self.connections)
)
re_add: typing.Set['KademliaPeer'] = set()
for peer in sorted(batch, key=lambda peer: self.scores.get(peer, 0), reverse=True):
if peer in self.ignored:
continue
if peer in tried_for_this_blob:
if peer in self.active_connections or not self.should_race_continue(blob):
continue
if peer in self.active_connections:
if peer not in re_add:
re_add.add(peer)
continue
if not self.should_race_continue(blob):
break
log.debug("request %s from %s:%i", blob_hash[:8], peer.address, peer.tcp_port)
t = self.loop.create_task(self.request_blob_from_peer(blob, peer, connection_id))
self.active_connections[peer] = t
tried_for_this_blob.add(peer)
if not re_add:
self.peer_queue.put_nowait(list(batch))
self.peer_queue.put_nowait(list(batch))
await self.new_peer_or_finished()
self.cleanup_active()
log.debug("downloaded %s", blob_hash[:8])
@@ -133,11 +124,14 @@ class BlobDownloader:
protocol.close()
async def download_blob(loop, config: 'Config', blob_manager: 'BlobManager', node: 'Node',
async def download_blob(loop, config: 'Config', blob_manager: 'BlobManager', dht_node: 'Node',
blob_hash: str) -> 'AbstractBlob':
search_queue = asyncio.Queue(loop=loop, maxsize=config.max_connections_per_download)
search_queue.put_nowait(blob_hash)
peer_queue, accumulate_task = node.accumulate_peers(search_queue)
peer_queue, accumulate_task = dht_node.accumulate_peers(search_queue)
fixed_peers = None if not config.fixed_peers else await get_kademlia_peers_from_hosts(config.fixed_peers)
if fixed_peers:
loop.call_later(config.fixed_peer_delay, peer_queue.put_nowait, fixed_peers)
downloader = BlobDownloader(loop, config, blob_manager, peer_queue)
try:
return await downloader.download_blob(blob_hash)
+4 -2
View File
@@ -613,7 +613,7 @@ class Config(CLIConfig):
"ports or have firewall rules you likely want to disable this.", True
)
udp_port = Integer("UDP port for communicating on the LBRY DHT", 4444, previous_names=['dht_node_port'])
tcp_port = Integer("TCP port to listen for incoming blob requests", 3333, previous_names=['peer_port'])
tcp_port = Integer("TCP port to listen for incoming blob requests", 4444, previous_names=['peer_port'])
prometheus_port = Integer("Port to expose prometheus metrics (off by default)", 0)
network_interface = String("Interface to use for the DHT and blob exchange", '0.0.0.0')
@@ -634,6 +634,7 @@ class Config(CLIConfig):
# blob announcement and download
save_blobs = Toggle("Save encrypted blob files for hosting, otherwise download blobs to memory only.", True)
network_storage_limit = Integer("Disk space in MB to be allocated for helping the P2P network. 0 = disable", 0)
blob_storage_limit = Integer("Disk space in MB to be allocated for blob storage. 0 = no limit", 0)
blob_lru_cache_size = Integer(
"LRU cache size for decrypted downloaded blobs used to minimize re-downloading the same blobs when "
@@ -723,7 +724,8 @@ class Config(CLIConfig):
coin_selection_strategy = StringChoice(
"Strategy to use when selecting UTXOs for a transaction",
STRATEGIES, "standard")
STRATEGIES, "prefer_confirmed"
)
transaction_cache_size = Integer("Transaction cache size", 2 ** 17)
save_resolved_claims = Toggle(
+16
View File
@@ -1,6 +1,9 @@
import asyncio
import typing
import logging
from prometheus_client import Counter, Gauge
if typing.TYPE_CHECKING:
from lbry.dht.node import Node
from lbry.extras.daemon.storage import SQLiteStorage
@@ -9,6 +12,15 @@ log = logging.getLogger(__name__)
class BlobAnnouncer:
announcements_sent_metric = Counter(
"announcements_sent", "Number of announcements sent and their respective status.", namespace="dht_node",
labelnames=("peers", "error"),
)
announcement_queue_size_metric = Gauge(
"announcement_queue_size", "Number of hashes waiting to be announced.", namespace="dht_node",
labelnames=("scope",)
)
def __init__(self, loop: asyncio.AbstractEventLoop, node: 'Node', storage: 'SQLiteStorage'):
self.loop = loop
self.node = node
@@ -18,12 +30,15 @@ class BlobAnnouncer:
async def _submit_announcement(self, blob_hash):
try:
peers = len(await self.node.announce_blob(blob_hash))
self.announcements_sent_metric.labels(peers=peers, error=False).inc()
if peers > 4:
return blob_hash
else:
log.debug("failed to announce %s, could only find %d peers, retrying soon.", blob_hash[:8], peers)
except Exception as err:
self.announcements_sent_metric.labels(peers=0, error=True).inc()
if isinstance(err, asyncio.CancelledError): # TODO: remove when updated to 3.8
raise err
log.warning("error announcing %s: %s", blob_hash[:8], str(err))
@@ -37,6 +52,7 @@ class BlobAnnouncer:
log.warning("No peers in DHT, announce round skipped")
continue
self.announce_queue.extend(await self.storage.get_blobs_to_announce())
self.announcement_queue_size_metric.labels(scope="global").set(len(self.announce_queue))
log.debug("announcer task wake up, %d blobs to announce", len(self.announce_queue))
while len(self.announce_queue) > 0:
log.info("%i blobs to announce", len(self.announce_queue))
+34 -1
View File
@@ -2,6 +2,9 @@ import logging
import asyncio
import typing
import socket
from prometheus_client import Gauge
from lbry.utils import resolve_host
from lbry.dht import constants
from lbry.dht.peer import make_kademlia_peer
@@ -17,6 +20,14 @@ log = logging.getLogger(__name__)
class Node:
storing_peers_metric = Gauge(
"storing_peers", "Number of peers storing blobs announced to this node", namespace="dht_node",
labelnames=("scope",),
)
stored_blob_with_x_bytes_colliding = Gauge(
"stored_blobs_x_bytes_colliding", "Number of blobs with at least X bytes colliding with this node id prefix",
namespace="dht_node", labelnames=("amount",)
)
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager', node_id: bytes, udp_port: int,
internal_udp_port: int, peer_port: int, external_ip: str, rpc_timeout: float = constants.RPC_TIMEOUT,
split_buckets_under_index: int = constants.SPLIT_BUCKETS_UNDER_INDEX,
@@ -31,6 +42,10 @@ class Node:
self._refresh_task: asyncio.Task = None
self._storage = storage
@property
def stored_blob_hashes(self):
return self.protocol.data_store.keys()
async def refresh_node(self, force_once=False):
while True:
# remove peers with expired blob announcements from the datastore
@@ -40,7 +55,18 @@ class Node:
# add all peers in the routing table
total_peers.extend(self.protocol.routing_table.get_peers())
# add all the peers who have announced blobs to us
total_peers.extend(self.protocol.data_store.get_storing_contacts())
storing_peers = self.protocol.data_store.get_storing_contacts()
self.storing_peers_metric.labels("global").set(len(storing_peers))
total_peers.extend(storing_peers)
counts = {0: 0, 1: 0, 2: 0}
node_id = self.protocol.node_id
for blob_hash in self.protocol.data_store.keys():
bytes_colliding = 0 if blob_hash[0] != node_id[0] else 2 if blob_hash[1] == node_id[1] else 1
counts[bytes_colliding] += 1
self.stored_blob_with_x_bytes_colliding.labels(amount=0).set(counts[0])
self.stored_blob_with_x_bytes_colliding.labels(amount=1).set(counts[1])
self.stored_blob_with_x_bytes_colliding.labels(amount=2).set(counts[2])
# get ids falling in the midpoint of each bucket that hasn't been recently updated
node_ids = self.protocol.routing_table.get_refresh_list(0, True)
@@ -257,3 +283,10 @@ class Node:
) -> typing.Tuple[asyncio.Queue, asyncio.Task]:
queue = peer_queue or asyncio.Queue(loop=self.loop)
return queue, self.loop.create_task(self._accumulate_peers_for_value(search_queue, queue))
async def get_kademlia_peers_from_hosts(peer_list: typing.List[typing.Tuple[str, int]]) -> typing.List['KademliaPeer']:
peer_address_list = [(await resolve_host(url, port, proto='tcp'), port) for url, port in peer_list]
kademlia_peer_list = [make_kademlia_peer(None, address, None, tcp_port=port, allow_localhost=True)
for address, port in peer_address_list]
return kademlia_peer_list
+22 -8
View File
@@ -3,15 +3,19 @@ import asyncio
import logging
from dataclasses import dataclass, field
from functools import lru_cache
from prometheus_client import Gauge
from lbry.utils import is_valid_public_ipv4 as _is_valid_public_ipv4, LRUCache
from lbry.dht import constants
from lbry.dht.serialization.datagram import make_compact_address, make_compact_ip, decode_compact_address
ALLOW_LOCALHOST = False
CACHE_SIZE = 16384
log = logging.getLogger(__name__)
@lru_cache(1024)
@lru_cache(CACHE_SIZE)
def make_kademlia_peer(node_id: typing.Optional[bytes], address: typing.Optional[str],
udp_port: typing.Optional[int] = None,
tcp_port: typing.Optional[int] = None,
@@ -25,17 +29,26 @@ def is_valid_public_ipv4(address, allow_localhost: bool = False):
class PeerManager:
peer_manager_keys_metric = Gauge(
"peer_manager_keys", "Number of keys tracked by PeerManager dicts (sum)", namespace="dht_node",
labelnames=("scope",)
)
def __init__(self, loop: asyncio.AbstractEventLoop):
self._loop = loop
self._rpc_failures: typing.Dict[
typing.Tuple[str, int], typing.Tuple[typing.Optional[float], typing.Optional[float]]
] = {}
self._last_replied: typing.Dict[typing.Tuple[str, int], float] = LRUCache(2048)
self._last_sent: typing.Dict[typing.Tuple[str, int], float] = LRUCache(2048)
self._last_requested: typing.Dict[typing.Tuple[str, int], float] = LRUCache(2048)
self._node_id_mapping: typing.Dict[typing.Tuple[str, int], bytes] = LRUCache(2048)
self._node_id_reverse_mapping: typing.Dict[bytes, typing.Tuple[str, int]] = LRUCache(2048)
self._node_tokens: typing.Dict[bytes, (float, bytes)] = LRUCache(2048)
] = LRUCache(CACHE_SIZE)
self._last_replied: typing.Dict[typing.Tuple[str, int], float] = LRUCache(CACHE_SIZE)
self._last_sent: typing.Dict[typing.Tuple[str, int], float] = LRUCache(CACHE_SIZE)
self._last_requested: typing.Dict[typing.Tuple[str, int], float] = LRUCache(CACHE_SIZE)
self._node_id_mapping: typing.Dict[typing.Tuple[str, int], bytes] = LRUCache(CACHE_SIZE)
self._node_id_reverse_mapping: typing.Dict[bytes, typing.Tuple[str, int]] = LRUCache(CACHE_SIZE)
self._node_tokens: typing.Dict[bytes, (float, bytes)] = LRUCache(CACHE_SIZE)
def count_cache_keys(self):
return len(self._rpc_failures) + len(self._last_replied) + len(self._last_sent) + len(
self._last_requested) + len(self._node_id_mapping) + len(self._node_id_reverse_mapping) + len(
self._node_tokens)
def reset(self):
for statistic in (self._rpc_failures, self._last_replied, self._last_sent, self._last_requested):
@@ -85,6 +98,7 @@ class PeerManager:
self._node_id_mapping.pop(self._node_id_reverse_mapping.pop(node_id))
self._node_id_mapping[(address, udp_port)] = node_id
self._node_id_reverse_mapping[node_id] = (address, udp_port)
self.peer_manager_keys_metric.labels("global").set(self.count_cache_keys())
def prune(self): # TODO: periodically call this
now = self._loop.time()
+3
View File
@@ -16,6 +16,9 @@ class DictDataStore:
self._peer_manager = peer_manager
self.completed_blobs: typing.Set[str] = set()
def keys(self):
return self._data_store.keys()
def __len__(self):
return self._data_store.__len__()
+6 -4
View File
@@ -195,7 +195,7 @@ class IterativeFinder:
continue
self._schedule_probe(peer)
added += 1
log.debug("running %d probes", len(self.running_probes))
log.debug("running %d probes for key %s", len(self.running_probes), self.key.hex()[:8])
if not added and not self.running_probes:
log.debug("search for %s exhausted", self.key.hex()[:8])
self.search_exhausted()
@@ -270,7 +270,8 @@ class IterativeNodeFinder(IterativeFinder):
self.yielded_peers: typing.Set['KademliaPeer'] = set()
async def send_probe(self, peer: 'KademliaPeer') -> FindNodeResponse:
log.debug("probing %s:%d %s", peer.address, peer.udp_port, peer.node_id.hex()[:8] if peer.node_id else '')
log.debug("probe %s:%d (%s) for NODE %s",
peer.address, peer.udp_port, peer.node_id.hex()[:8] if peer.node_id else '', self.key.hex()[:8])
response = await self.protocol.get_rpc_peer(peer).find_node(self.key)
return FindNodeResponse(self.key, response)
@@ -285,7 +286,7 @@ class IterativeNodeFinder(IterativeFinder):
and self.peer_manager.peer_is_good(peer) is not False
]
not_yet_yielded.sort(key=lambda peer: self.distance(peer.node_id))
to_yield = not_yet_yielded[:min(constants.K, len(not_yet_yielded))]
to_yield = not_yet_yielded[:max(constants.K, self.max_results)]
if to_yield:
self.yielded_peers.update(to_yield)
self.iteration_queue.put_nowait(to_yield)
@@ -325,6 +326,8 @@ class IterativeValueFinder(IterativeFinder):
self.discovered_peers: typing.Dict['KademliaPeer', typing.Set['KademliaPeer']] = defaultdict(set)
async def send_probe(self, peer: 'KademliaPeer') -> FindValueResponse:
log.debug("probe %s:%d (%s) for VALUE %s",
peer.address, peer.udp_port, peer.node_id.hex()[:8], self.key.hex()[:8])
page = self.peer_pages[peer]
response = await self.protocol.get_rpc_peer(peer).find_value(self.key, page=page)
parsed = FindValueResponse(self.key, response)
@@ -346,7 +349,6 @@ class IterativeValueFinder(IterativeFinder):
already_known + len(parsed.found_compact_addresses))
if len(self.discovered_peers[peer]) != already_known + len(parsed.found_compact_addresses):
log.warning("misbehaving peer %s:%i returned duplicate peers for blob", peer.address, peer.udp_port)
parsed.found_compact_addresses.clear()
elif len(parsed.found_compact_addresses) >= constants.K and self.peer_pages[peer] < parsed.pages:
# the peer returned a full page and indicates it has more
self.peer_pages[peer] += 1
+39
View File
@@ -3,11 +3,14 @@ import socket
import functools
import hashlib
import asyncio
import time
import typing
import random
from asyncio.protocols import DatagramProtocol
from asyncio.transports import DatagramTransport
from prometheus_client import Gauge, Counter, Histogram
from lbry.dht import constants
from lbry.dht.serialization.bencoding import DecodeError
from lbry.dht.serialization.datagram import decode_datagram, ErrorDatagram, ResponseDatagram, RequestDatagram
@@ -30,6 +33,11 @@ OLD_PROTOCOL_ERRORS = {
class KademliaRPC:
stored_blob_metric = Gauge(
"stored_blobs", "Number of blobs announced by other peers", namespace="dht_node",
labelnames=("scope",),
)
def __init__(self, protocol: 'KademliaProtocol', loop: asyncio.AbstractEventLoop, peer_port: int = 3333):
self.protocol = protocol
self.loop = loop
@@ -61,6 +69,7 @@ class KademliaRPC:
self.protocol.data_store.add_peer_to_blob(
rpc_contact, blob_hash
)
self.stored_blob_metric.labels("global").set(len(self.protocol.data_store))
return b'OK'
def find_node(self, rpc_contact: 'KademliaPeer', key: bytes) -> typing.List[typing.Tuple[bytes, str, int]]:
@@ -259,6 +268,30 @@ class PingQueue:
class KademliaProtocol(DatagramProtocol):
request_sent_metric = Counter(
"request_sent", "Number of requests send from DHT RPC protocol", namespace="dht_node",
labelnames=("method",),
)
request_success_metric = Counter(
"request_success", "Number of successful requests", namespace="dht_node",
labelnames=("method",),
)
request_error_metric = Counter(
"request_error", "Number of errors returned from request to other peers", namespace="dht_node",
labelnames=("method",),
)
HISTOGRAM_BUCKETS = (
.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 3.0, 3.5, 4.0, 4.50, 5.0, 5.50, 6.0, float('inf')
)
response_time_metric = Histogram(
"response_time", "Response times of DHT RPC requests", namespace="dht_node", buckets=HISTOGRAM_BUCKETS,
labelnames=("method",)
)
received_request_metric = Counter(
"received_request", "Number of received DHT RPC requests", namespace="dht_node",
labelnames=("method",),
)
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager', node_id: bytes, external_ip: str,
udp_port: int, peer_port: int, rpc_timeout: float = constants.RPC_TIMEOUT,
split_buckets_under_index: int = constants.SPLIT_BUCKETS_UNDER_INDEX):
@@ -447,6 +480,7 @@ class KademliaProtocol(DatagramProtocol):
def handle_request_datagram(self, address: typing.Tuple[str, int], request_datagram: RequestDatagram):
# This is an RPC method request
self.received_request_metric.labels(method=request_datagram.method).inc()
self.peer_manager.report_last_requested(address[0], address[1])
try:
peer = self.routing_table.get_peer(request_datagram.node_id)
@@ -575,14 +609,19 @@ class KademliaProtocol(DatagramProtocol):
self._send(peer, request)
response_fut = self.sent_messages[request.rpc_id][1]
try:
self.request_sent_metric.labels(method=request.method).inc()
start = time.perf_counter()
response = await asyncio.wait_for(response_fut, self.rpc_timeout)
self.response_time_metric.labels(method=request.method).observe(time.perf_counter() - start)
self.peer_manager.report_last_replied(peer.address, peer.udp_port)
self.request_success_metric.labels(method=request.method).inc()
return response
except asyncio.CancelledError:
if not response_fut.done():
response_fut.cancel()
raise
except (asyncio.TimeoutError, RemoteException):
self.request_error_metric.labels(method=request.method).inc()
self.peer_manager.report_failure(peer.address, peer.udp_port)
if self.peer_manager.peer_is_good(peer) is False:
self.remove_peer(peer)
+26 -1
View File
@@ -4,6 +4,8 @@ import logging
import typing
import itertools
from prometheus_client import Gauge
from lbry.dht import constants
from lbry.dht.protocol.distance import Distance
if typing.TYPE_CHECKING:
@@ -13,8 +15,17 @@ log = logging.getLogger(__name__)
class KBucket:
""" Description - later
"""
Kademlia K-bucket implementation.
"""
peer_in_routing_table_metric = Gauge(
"peers_in_routing_table", "Number of peers on routing table", namespace="dht_node",
labelnames=("scope",)
)
peer_with_x_bit_colliding_metric = Gauge(
"peer_x_bit_colliding", "Number of peers with at least X bits colliding with this node id",
namespace="dht_node", labelnames=("amount",)
)
def __init__(self, peer_manager: 'PeerManager', range_min: int, range_max: int, node_id: bytes):
"""
@@ -58,6 +69,10 @@ class KBucket:
return True
if len(self.peers) < constants.K:
self.peers.append(peer)
self.peer_in_routing_table_metric.labels("global").inc()
if peer.node_id[0] == self._node_id[0]:
bits_colliding = 8 - (peer.node_id[1] ^ self._node_id[1]).bit_length()
self.peer_with_x_bit_colliding_metric.labels(amount=(bits_colliding + 8)).inc()
return True
else:
return False
@@ -124,6 +139,10 @@ class KBucket:
def remove_peer(self, peer: 'KademliaPeer') -> None:
self.peers.remove(peer)
self.peer_in_routing_table_metric.labels("global").dec()
if peer.node_id[0] == self._node_id[0]:
bits_colliding = 8 - (peer.node_id[1] ^ self._node_id[1]).bit_length()
self.peer_with_x_bit_colliding_metric.labels(amount=(bits_colliding + 8)).dec()
def key_in_range(self, key: bytes) -> bool:
""" Tests whether the specified key (i.e. node ID) is in the range
@@ -162,6 +181,10 @@ class TreeRoutingTable:
ping RPC-based k-bucket eviction algorithm described in section 2.2 of
that paper.
"""
bucket_in_routing_table_metric = Gauge(
"buckets_in_routing_table", "Number of buckets on routing table", namespace="dht_node",
labelnames=("scope",)
)
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager', parent_node_id: bytes,
split_buckets_under_index: int = constants.SPLIT_BUCKETS_UNDER_INDEX):
@@ -279,6 +302,7 @@ class TreeRoutingTable:
# ...and remove them from the old bucket
for contact in new_bucket.peers:
old_bucket.remove_peer(contact)
self.bucket_in_routing_table_metric.labels("global").set(len(self.buckets))
def join_buckets(self):
if len(self.buckets) == 1:
@@ -302,6 +326,7 @@ class TreeRoutingTable:
elif can_go_higher:
self.buckets[bucket_index_to_pop + 1].range_min = bucket.range_min
self.buckets.remove(bucket)
self.bucket_in_routing_table_metric.labels("global").set(len(self.buckets))
return self.join_buckets()
def contact_in_routing_table(self, address_tuple: typing.Tuple[str, int]) -> bool:
+1
View File
@@ -38,6 +38,7 @@ Code | Name | Message
114 | InputStringIsBlank | {argument} cannot be blank.
115 | EmptyPublishedFile | Cannot publish empty file: {file_path}
116 | MissingPublishedFile | File does not exist: {file_path}
117 | InvalidStreamURL | Invalid LBRY stream URL: '{url}' -- When an URL cannot be downloaded, such as '@Channel/' or a collection
**2xx** | Configuration | Configuration errors.
201 | ConfigWrite | Cannot write configuration file '{path}'. -- When writing the default config fails on startup, such as due to permission issues.
202 | ConfigRead | Cannot find provided configuration file '{path}'. -- Can't open the config file user provided via command line args.
+10
View File
@@ -105,6 +105,16 @@ class MissingPublishedFileError(InputValueError):
super().__init__(f"File does not exist: {file_path}")
class InvalidStreamURLError(InputValueError):
"""
When an URL cannot be downloaded, such as '@Channel/' or a collection
"""
def __init__(self, url):
self.url = url
super().__init__(f"Invalid LBRY stream URL: '{url}'")
class ConfigurationError(BaseError):
"""
Configuration errors.
+3
View File
@@ -226,6 +226,9 @@ def get_argument_parser():
def ensure_directory_exists(path: str):
if not os.path.isdir(path):
pathlib.Path(path).mkdir(parents=True, exist_ok=True)
use_effective_ids = os.access in os.supports_effective_ids
if not os.access(path, os.W_OK, effective_ids=use_effective_ids):
raise PermissionError(f"The following directory is not writable: {path}")
LOG_MODULES = 'lbry', 'aioupnp'
+2 -1
View File
@@ -170,11 +170,12 @@ class AnalyticsManager:
})
)
async def send_disk_space_used(self, storage_used, storage_limit):
async def send_disk_space_used(self, storage_used, storage_limit, is_from_network_quota):
await self.track(
self._event(DISK_SPACE, {
'used': storage_used,
'limit': storage_limit,
'from_network_quota': is_from_network_quota
})
)
+64 -2
View File
@@ -4,6 +4,7 @@ import asyncio
import logging
import binascii
import typing
import base58
from aioupnp import __version__ as aioupnp_version
@@ -17,6 +18,7 @@ from lbry.dht.blob_announcer import BlobAnnouncer
from lbry.blob.blob_manager import BlobManager
from lbry.blob.disk_space_manager import DiskSpaceManager
from lbry.blob_exchange.server import BlobServer
from lbry.stream.background_downloader import BackgroundDownloader
from lbry.stream.stream_manager import StreamManager
from lbry.file.file_manager import FileManager
from lbry.extras.daemon.component import Component
@@ -42,6 +44,7 @@ DHT_COMPONENT = "dht"
HASH_ANNOUNCER_COMPONENT = "hash_announcer"
FILE_MANAGER_COMPONENT = "file_manager"
DISK_SPACE_COMPONENT = "disk_space"
BACKGROUND_DOWNLOADER_COMPONENT = "background_downloader"
PEER_PROTOCOL_SERVER_COMPONENT = "peer_protocol_server"
UPNP_COMPONENT = "upnp"
EXCHANGE_RATE_MANAGER_COMPONENT = "exchange_rate_manager"
@@ -377,6 +380,61 @@ class FileManagerComponent(Component):
self.file_manager.stop()
class BackgroundDownloaderComponent(Component):
component_name = BACKGROUND_DOWNLOADER_COMPONENT
depends_on = [DATABASE_COMPONENT, BLOB_COMPONENT, DISK_SPACE_COMPONENT]
def __init__(self, component_manager):
super().__init__(component_manager)
self.background_task: typing.Optional[asyncio.Task] = None
self.download_loop_delay_seconds = 60
self.ongoing_download: typing.Optional[asyncio.Task] = None
self.space_manager: typing.Optional[DiskSpaceManager] = None
self.blob_manager: typing.Optional[BlobManager] = None
self.background_downloader: typing.Optional[BackgroundDownloader] = None
self.dht_node: typing.Optional[Node] = None
self.space_available: typing.Optional[int] = None
@property
def is_busy(self):
return bool(self.ongoing_download and not self.ongoing_download.done())
@property
def component(self) -> 'BackgroundDownloaderComponent':
return self
async def get_status(self):
return {'running': self.background_task is not None and not self.background_task.done(),
'available_free_space_mb': self.space_available,
'ongoing_download': self.is_busy}
async def download_blobs_in_background(self):
while True:
self.space_available = await self.space_manager.get_free_space_mb(True)
if not self.is_busy and self.space_available > 10:
blob_hash = next((key.hex() for key in self.dht_node.stored_blob_hashes if
key.hex() not in self.blob_manager.completed_blob_hashes), None)
if blob_hash:
self.ongoing_download = asyncio.create_task(self.background_downloader.download_blobs(blob_hash))
await asyncio.sleep(self.download_loop_delay_seconds)
async def start(self):
self.space_manager: DiskSpaceManager = self.component_manager.get_component(DISK_SPACE_COMPONENT)
if not self.component_manager.has_component(DHT_COMPONENT):
return
self.dht_node = self.component_manager.get_component(DHT_COMPONENT)
self.blob_manager = self.component_manager.get_component(BLOB_COMPONENT)
storage = self.component_manager.get_component(DATABASE_COMPONENT)
self.background_downloader = BackgroundDownloader(self.conf, storage, self.blob_manager, self.dht_node)
self.background_task = asyncio.create_task(self.download_blobs_in_background())
async def stop(self):
if self.ongoing_download and not self.ongoing_download.done():
self.ongoing_download.cancel()
if self.background_task:
self.background_task.cancel()
class DiskSpaceComponent(Component):
component_name = DISK_SPACE_COMPONENT
depends_on = [DATABASE_COMPONENT, BLOB_COMPONENT]
@@ -391,11 +449,15 @@ class DiskSpaceComponent(Component):
async def get_status(self):
if self.disk_space_manager:
space_used = await self.disk_space_manager.get_space_used_mb(cached=True)
return {
'space_used': str(await self.disk_space_manager.get_space_used_mb()),
'total_used_mb': space_used['total'],
'published_blobs_storage_used_mb': space_used['private_storage'],
'content_blobs_storage_used_mb': space_used['content_storage'],
'seed_blobs_storage_used_mb': space_used['network_storage'],
'running': self.disk_space_manager.running,
}
return {'space_used': '0', 'running': False}
return {'space_used': '0', 'network_seeding_space_used': '0', 'running': False}
async def start(self):
db = self.component_manager.get_component(DATABASE_COMPONENT)
+91 -60
View File
@@ -8,7 +8,6 @@ import time
import inspect
import typing
import random
import hashlib
import tracemalloc
from decimal import Decimal
from urllib.parse import urlencode, quote
@@ -17,7 +16,6 @@ from binascii import hexlify, unhexlify
from traceback import format_exc
from functools import wraps, partial
import ecdsa
import base58
from aiohttp import web
from prometheus_client import generate_latest as prom_generate_latest, Gauge, Histogram, Counter
@@ -29,6 +27,7 @@ from lbry.wallet import (
)
from lbry.wallet.dewies import dewies_to_lbc, lbc_to_dewies, dict_values_to_lbc
from lbry.wallet.constants import TXO_TYPES, CLAIM_TYPE_NAMES
from lbry.wallet.bip32 import PrivateKey
from lbry import utils
from lbry.conf import Config, Setting, NOT_SET
@@ -38,7 +37,8 @@ from lbry.dht.peer import make_kademlia_peer
from lbry.error import (
DownloadSDTimeoutError, ComponentsNotStartedError, ComponentStartConditionNotMetError,
CommandDoesNotExistError, BaseError, WalletNotFoundError, WalletAlreadyLoadedError, WalletAlreadyExistsError,
ConflictingInputValueError, AlreadyPurchasedError, PrivateKeyNotFoundError, InputStringIsBlankError
ConflictingInputValueError, AlreadyPurchasedError, PrivateKeyNotFoundError, InputStringIsBlankError,
InputValueError
)
from lbry.extras import system_info
from lbry.extras.daemon import analytics
@@ -205,12 +205,25 @@ def fix_kwargs_for_hub(**kwargs):
elif key == "not_channel_ids":
kwargs["channel_id"] = {"invert": True, "value": kwargs.pop("not_channel_ids")}
elif key in MY_RANGE_FIELDS:
operator = '='
if isinstance(value, str) and value[0] in opcodes:
operator_length = 2 if value[:2] in opcodes else 1
operator, value = value[:operator_length], value[operator_length:]
value = [str(value if key != 'fee_amount' else Decimal(value)*1000)]
kwargs[key] = {"op": opcodes[operator], "value": value}
constraints = []
for val in value if isinstance(value, list) else [value]:
operator = '='
if isinstance(val, str) and val[0] in opcodes:
operator_length = 2 if val[:2] in opcodes else 1
operator, val = val[:operator_length], val[operator_length:]
val = [int(val if key != 'fee_amount' else Decimal(val)*1000)]
constraints.append({"op": opcodes[operator], "value": val})
kwargs[key] = constraints
elif key == 'order_by': # TODO: remove this after removing support for old trending args from the api
value = value if isinstance(value, list) else [value]
new_value = []
for new_v in value:
migrated = new_v if new_v not in (
'trending_mixed', 'trending_local', 'trending_global', 'trending_group'
) else 'trending_score'
if migrated not in new_value:
new_value.append(migrated)
kwargs[key] = new_value
return kwargs
@@ -951,7 +964,12 @@ class Daemon(metaclass=JSONRPCServerType):
},
'total_outgoing_mps': (float) megabytes per second sent,
'total_incoming_mps': (float) megabytes per second received,
'time': (float) timestamp
'max_outgoing_mbs': (float) maximum bandwidth (megabytes per second) sent, since the
daemon was started
'max_incoming_mbs': (float) maximum bandwidth (megabytes per second) received, since the
daemon was started
'total_sent' : (int) total number of bytes sent since the daemon was started
'total_received' : (int) total number of bytes received since the daemon was started
}
},
'hash_announcer': {
@@ -2393,6 +2411,9 @@ class Daemon(metaclass=JSONRPCServerType):
value with an equality constraint such as '>', '>=', '<' and '<='
eg. --height=">400000" would limit results to only claims above 400k block height.
They also support multiple constraints passed as a list of the args described above.
eg. --release_time=[">1000000", "<2000000"]
Usage:
claim_search [<name> | --name=<name>] [--text=<text>] [--txid=<txid>] [--nout=<nout>]
[--claim_id=<claim_id> | --claim_ids=<claim_ids>...]
@@ -2407,7 +2428,7 @@ class Daemon(metaclass=JSONRPCServerType):
[--amount=<amount>] [--effective_amount=<effective_amount>]
[--support_amount=<support_amount>] [--trending_group=<trending_group>]
[--trending_mixed=<trending_mixed>] [--trending_local=<trending_local>]
[--trending_global=<trending_global]
[--trending_global=<trending_global] [--trending_score=<trending_score]
[--reposted_claim_id=<reposted_claim_id>] [--reposted=<reposted>]
[--claim_type=<claim_type>] [--stream_types=<stream_types>...] [--media_types=<media_types>...]
[--fee_currency=<fee_currency>] [--fee_amount=<fee_amount>]
@@ -2419,7 +2440,7 @@ class Daemon(metaclass=JSONRPCServerType):
[--not_locations=<not_locations>...]
[--order_by=<order_by>...] [--no_totals] [--page=<page>] [--page_size=<page_size>]
[--wallet_id=<wallet_id>] [--include_purchase_receipt] [--include_is_my_output]
[--remove_duplicates] [--has_source | --has_no_source]
[--remove_duplicates] [--has_source | --has_no_source] [--sd_hash=<sd_hash>]
[--new_sdk_server=<new_sdk_server>]
Options:
@@ -2475,25 +2496,11 @@ class Daemon(metaclass=JSONRPCServerType):
all tips and supports received), this amount is
blank until claim has reached activation height
(supports equality constraints)
--trending_group=<trending_group>: (int) group numbers 1 through 4 representing the
trending groups of the content: 4 means
content is trending globally and independently,
3 means content is not trending globally but is
trending independently (locally), 2 means it is
trending globally but not independently and 1
means it's not trending globally or locally
(supports equality constraints)
--trending_mixed=<trending_mixed>: (int) trending amount taken from the global or local
value depending on the trending group:
4 - global value, 3 - local value, 2 - global
value, 1 - local value (supports equality
constraints)
--trending_local=<trending_local>: (int) trending value calculated relative only to
the individual contents past history (supports
equality constraints)
--trending_global=<trending_global>: (int) trending value calculated relative to all
trending content globally (supports
equality constraints)
--trending_score=<trending_score>: (int) limit by trending score (supports equality constraints)
--trending_group=<trending_group>: (int) DEPRECATED - instead please use trending_score
--trending_mixed=<trending_mixed>: (int) DEPRECATED - instead please use trending_score
--trending_local=<trending_local>: (int) DEPRECATED - instead please use trending_score
--trending_global=<trending_global>: (int) DEPRECATED - instead please use trending_score
--reposted_claim_id=<reposted_claim_id>: (str) all reposts of the specified original claim id
--reposted=<reposted> : (int) claims reposted this many times (supports
equality constraints)
@@ -2531,6 +2538,8 @@ class Daemon(metaclass=JSONRPCServerType):
--remove_duplicates : (bool) removes duplicated content from search by picking either the
original claim or the oldest matching repost
--has_source : (bool) find claims containing a source field
--sd_hash=<sd_hash> : (str) find claims where the source stream descriptor hash matches
(partially or completely) the given hexadecimal string
--has_no_source : (bool) find claims not containing a source field
--new_sdk_server=<new_sdk_server> : (str) URL of the new SDK server (EXPERIMENTAL)
@@ -2561,6 +2570,17 @@ class Daemon(metaclass=JSONRPCServerType):
kwargs['signature_valid'] = 0
if 'has_no_source' in kwargs:
kwargs['has_source'] = not kwargs.pop('has_no_source')
if 'order_by' in kwargs: # TODO: remove this after removing support for old trending args from the api
value = kwargs.pop('order_by')
value = value if isinstance(value, list) else [value]
new_value = []
for new_v in value:
migrated = new_v if new_v not in (
'trending_mixed', 'trending_local', 'trending_global', 'trending_group'
) else 'trending_score'
if migrated not in new_value:
new_value.append(migrated)
kwargs['order_by'] = new_value
page_num, page_size = abs(kwargs.pop('page', 1)), min(abs(kwargs.pop('page_size', DEFAULT_PAGE_SIZE)), 50)
wallet = self.wallet_manager.get_wallet_or_default(kwargs.pop('wallet_id', None))
kwargs.update({'offset': page_size * (page_num - 1), 'limit': page_size})
@@ -2686,12 +2706,13 @@ class Daemon(metaclass=JSONRPCServerType):
name, claim, amount, claim_address, funding_accounts, funding_accounts[0]
)
txo = tx.outputs[0]
await txo.generate_channel_private_key()
txo.set_channel_private_key(
await funding_accounts[0].generate_channel_private_key()
)
await tx.sign(funding_accounts)
if not preview:
account.add_channel_private_key(txo.private_key)
wallet.save()
await self.broadcast_or_release(tx, blocking)
self.component_manager.loop.create_task(self.storage.save_claims([self._old_get_temp_claim_info(
@@ -2840,7 +2861,9 @@ class Daemon(metaclass=JSONRPCServerType):
new_txo = tx.outputs[0]
if new_signing_key:
await new_txo.generate_channel_private_key()
new_txo.set_channel_private_key(
await funding_accounts[0].generate_channel_private_key()
)
else:
new_txo.private_key = old_txo.private_key
@@ -2849,7 +2872,6 @@ class Daemon(metaclass=JSONRPCServerType):
await tx.sign(funding_accounts)
if not preview:
account.add_channel_private_key(new_txo.private_key)
wallet.save()
await self.broadcast_or_release(tx, blocking)
self.component_manager.loop.create_task(self.storage.save_claims([self._old_get_temp_claim_info(
@@ -2893,7 +2915,7 @@ class Daemon(metaclass=JSONRPCServerType):
wallet, channel_account_id, channel_id, channel_name, for_signing=True
)
timestamp = str(int(time.time()))
signature = signing_channel.sign_data(unhexlify(hexdata), timestamp)
signature = signing_channel.sign_data(unhexlify(str(hexdata)), timestamp)
return {
'signature': signature,
'signing_ts': timestamp
@@ -3021,7 +3043,7 @@ class Daemon(metaclass=JSONRPCServerType):
'channel_id': channel.claim_id,
'holding_address': address,
'holding_public_key': public_key.extended_key_string(),
'signing_private_key': channel.private_key.to_pem().decode()
'signing_private_key': channel.private_key.signing_key.to_pem().decode()
}
return base58.b58encode(json.dumps(export, separators=(',', ':')))
@@ -3044,15 +3066,14 @@ class Daemon(metaclass=JSONRPCServerType):
decoded = base58.b58decode(channel_data)
data = json.loads(decoded)
channel_private_key = ecdsa.SigningKey.from_pem(
data['signing_private_key'], hashfunc=hashlib.sha256
channel_private_key = PrivateKey.from_pem(
self.ledger, data['signing_private_key']
)
public_key_der = channel_private_key.get_verifying_key().to_der()
# check that the holding_address hasn't changed since the export was made
holding_address = data['holding_address']
channels, _, _, _ = await self.ledger.claim_search(
wallet.accounts, public_key_id=self.ledger.public_key_to_address(public_key_der)
wallet.accounts, public_key_id=channel_private_key.address
)
if channels and channels[0].get_address(self.ledger) != holding_address:
holding_address = channels[0].get_address(self.ledger)
@@ -3594,15 +3615,17 @@ class Daemon(metaclass=JSONRPCServerType):
)
if len(existing_claims) != 1:
account_ids = ', '.join(f"'{account.id}'" for account in accounts)
# TODO: use error from lbry.error
raise Exception(
raise InputValueError(
f"Can't find the stream '{claim_id}' in account(s) {account_ids}."
)
old_txo = existing_claims[0]
if not old_txo.claim.is_stream:
# TODO: use error from lbry.error
raise Exception(
f"A claim with id '{claim_id}' was found but it is not a stream claim."
if not old_txo.claim.is_stream and not old_txo.claim.is_repost:
# in principle it should work with any type of claim, but its safer to
# limit it to ones we know won't be broken. in the future we can expand
# this if we have a test case for e.g. channel or support claims
raise InputValueError(
f"A claim with id '{claim_id}' was found but it is not a stream or repost claim."
)
if bid is not None:
@@ -3633,28 +3656,32 @@ class Daemon(metaclass=JSONRPCServerType):
if replace:
claim = Claim()
if old_txo.claim.stream.has_source:
claim.stream.message.source.CopyFrom(
old_txo.claim.stream.message.source
)
stream_type = old_txo.claim.stream.stream_type
if stream_type:
old_stream_type = getattr(old_txo.claim.stream.message, stream_type)
new_stream_type = getattr(claim.stream.message, stream_type)
new_stream_type.CopyFrom(old_stream_type)
claim.stream.update(file_path=file_path, **kwargs)
if old_txo.claim.is_stream:
if old_txo.claim.stream.has_source:
claim.stream.message.source.CopyFrom(
old_txo.claim.stream.message.source
)
stream_type = old_txo.claim.stream.stream_type
if stream_type:
old_stream_type = getattr(old_txo.claim.stream.message, stream_type)
new_stream_type = getattr(claim.stream.message, stream_type)
new_stream_type.CopyFrom(old_stream_type)
else:
claim = Claim.from_bytes(old_txo.claim.to_bytes())
if old_txo.claim.is_stream:
claim.stream.update(file_path=file_path, **kwargs)
if clear_channel:
claim.clear_signature()
tx = await Transaction.claim_update(
old_txo, claim, amount, claim_address, funding_accounts, funding_accounts[0],
channel if not clear_channel else None
)
new_txo = tx.outputs[0]
stream_hash = None
if not preview:
if not preview and old_txo.claim.is_stream:
old_stream = self.file_manager.get_filtered(sd_hash=old_txo.claim.stream.source.sd_hash)
old_stream = old_stream[0] if old_stream else None
if file_path is not None:
@@ -4959,7 +4986,8 @@ class Daemon(metaclass=JSONRPCServerType):
--finished : (bool) only return finished blobs
--uri=<uri> : (str) filter blobs by stream in a uri
--stream_hash=<stream_hash> : (str) filter blobs by stream hash
--sd_hash=<sd_hash> : (str) filter blobs by sd hash
--sd_hash=<sd_hash> : (str) filter blobs in a stream by sd hash, ie the hash of the stream
descriptor blob for a stream that has been downloaded
--page=<page> : (int) page to return during paginating
--page_size=<page_size> : (int) number of items on page during pagination
@@ -5128,10 +5156,12 @@ class Daemon(metaclass=JSONRPCServerType):
]
},
"node_id": (str) the local dht node id
"prefix_neighbors_count": (int) the amount of peers sharing the same byte prefix of the local node id
}
"""
result = {
'buckets': {}
'buckets': {},
'prefix_neighbors_count': 0
}
for i, _ in enumerate(self.dht_node.protocol.routing_table.buckets):
@@ -5144,6 +5174,7 @@ class Daemon(metaclass=JSONRPCServerType):
"node_id": hexlify(peer.node_id).decode(),
}
result['buckets'][i].append(host)
result['prefix_neighbors_count'] += 1 if peer.node_id[0] == self.dht_node.protocol.node_id[0] else 0
result['node_id'] = hexlify(self.dht_node.protocol.node_id).decode()
return result
+2 -2
View File
@@ -10,7 +10,7 @@ from lbry.schema.claim import Claim
from lbry.schema.support import Support
from lbry.torrent.torrent_manager import TorrentSource
from lbry.wallet import Wallet, Ledger, Account, Transaction, Output
from lbry.wallet.bip32 import PubKey
from lbry.wallet.bip32 import PublicKey
from lbry.wallet.dewies import dewies_to_lbc
from lbry.stream.managed_stream import ManagedStream
@@ -138,7 +138,7 @@ class JSONResponseEncoder(JSONEncoder):
return self.encode_claim(obj)
if isinstance(obj, Support):
return obj.to_dict()
if isinstance(obj, PubKey):
if isinstance(obj, PublicKey):
return obj.extended_key_string()
if isinstance(obj, datetime):
return obj.strftime("%Y%m%dT%H:%M:%S")
@@ -37,6 +37,8 @@ def migrate_db(conf, start, end):
from .migrate13to14 import do_migration
elif current == 14:
from .migrate14to15 import do_migration
elif current == 15:
from .migrate15to16 import do_migration
else:
raise Exception(f"DB migration of version {current} to {current+1} is not available")
try:
@@ -0,0 +1,17 @@
import os
import sqlite3
def do_migration(conf):
db_path = os.path.join(conf.data_dir, "lbrynet.sqlite")
connection = sqlite3.connect(db_path)
cursor = connection.cursor()
cursor.executescript("""
update blob set should_announce=0
where should_announce=1 and
blob.blob_hash in (select stream_blob.blob_hash from stream_blob where position=0);
""")
connection.commit()
connection.close()
+48 -12
View File
@@ -187,8 +187,8 @@ def store_stream(transaction: sqlite3.Connection, sd_blob: 'BlobFile', descripto
).fetchall()
# ensure should_announce is set regardless if insert was ignored
transaction.execute(
"update blob set should_announce=1 where blob_hash in (?, ?)",
(sd_blob.blob_hash, descriptor.blobs[0].blob_hash,)
"update blob set should_announce=1 where blob_hash in (?)",
(sd_blob.blob_hash,)
).fetchall()
@@ -337,6 +337,7 @@ class SQLiteStorage(SQLiteMixin):
tcp_port integer,
unique (address, udp_port)
);
create index if not exists blob_data on blob(blob_hash, blob_length, is_mine);
"""
def __init__(self, conf: Config, path, loop=None, time_getter: typing.Optional[typing.Callable[[], float]] = None):
@@ -380,6 +381,11 @@ class SQLiteStorage(SQLiteMixin):
"select status from blob where blob_hash=?", blob_hash
)
def set_announce(self, *blob_hashes):
return self.db.execute_fetchall(
"update blob set should_announce=1 where blob_hash in (?, ?)", blob_hashes
)
def update_last_announced_blobs(self, blob_hashes: typing.List[str]):
def _update_last_announced_blobs(transaction: sqlite3.Connection):
last_announced = self.time_getter()
@@ -437,20 +443,50 @@ class SQLiteStorage(SQLiteMixin):
def get_all_blob_hashes(self):
return self.run_and_return_list("select blob_hash from blob")
async def get_stored_blobs(self, is_mine: bool):
async def get_stored_blobs(self, is_mine: bool, is_network_blob=False):
is_mine = 1 if is_mine else 0
return await self.db.execute_fetchall(
"select blob_hash, blob_length, added_on from blob where is_mine=? order by added_on asc",
if is_network_blob:
return await self.db.execute_fetchall(
"select blob.blob_hash, blob.blob_length, blob.added_on "
"from blob left join stream_blob using (blob_hash) "
"where stream_blob.stream_hash is null and blob.is_mine=? order by blob.added_on asc",
(is_mine,)
)
sd_blobs = await self.db.execute_fetchall(
"select blob.blob_hash, blob.blob_length, blob.added_on "
"from blob join stream on blob.blob_hash=stream.sd_hash join file using (stream_hash) "
"where blob.is_mine=? order by blob.added_on asc",
(is_mine,)
)
content_blobs = await self.db.execute_fetchall(
"select blob.blob_hash, blob.blob_length, blob.added_on "
"from blob join stream_blob using (blob_hash) cross join stream using (stream_hash)"
"cross join file using (stream_hash) where blob.is_mine=? order by blob.added_on asc, blob.blob_length asc",
(is_mine,)
)
return content_blobs + sd_blobs
async def get_stored_blob_disk_usage(self, is_mine: Optional[bool] = None):
if is_mine is None:
sql, args = "select coalesce(sum(blob_length), 0) from blob", ()
else:
is_mine = 1 if is_mine else 0
sql, args = "select coalesce(sum(blob_length), 0) from blob where is_mine=?", (is_mine,)
return (await self.db.execute_fetchone(sql, args))[0]
async def get_stored_blob_disk_usage(self):
total, network_size, content_size, private_size = await self.db.execute_fetchone("""
select coalesce(sum(blob_length), 0) as total,
coalesce(sum(case when
stream_blob.stream_hash is null
then blob_length else 0 end), 0) as network_storage,
coalesce(sum(case when
stream_blob.blob_hash is not null and is_mine=0
then blob_length else 0 end), 0) as content_storage,
coalesce(sum(case when
is_mine=1
then blob_length else 0 end), 0) as private_storage
from blob left join stream_blob using (blob_hash)
""")
return {
'network_storage': network_size,
'content_storage': content_size,
'private_storage': private_size,
'total': total
}
async def update_blob_ownership(self, sd_hash, is_mine: bool):
is_mine = 1 if is_mine else 0
+7 -3
View File
@@ -5,6 +5,7 @@ from typing import Optional
from aiohttp.web import Request
from lbry.error import ResolveError, DownloadSDTimeoutError, InsufficientFundsError
from lbry.error import ResolveTimeoutError, DownloadDataTimeoutError, KeyFeeAboveMaxAllowedError
from lbry.error import InvalidStreamURLError
from lbry.stream.managed_stream import ManagedStream
from lbry.torrent.torrent_manager import TorrentSource
from lbry.utils import cache_concurrent
@@ -81,8 +82,11 @@ class FileManager:
payment = None
try:
# resolve the claim
if not URL.parse(uri).has_stream:
raise ResolveError("cannot download a channel claim, specify a /path")
try:
if not URL.parse(uri).has_stream:
raise InvalidStreamURLError(uri)
except ValueError:
raise InvalidStreamURLError(uri)
try:
resolved_result = await asyncio.wait_for(
self.wallet_manager.ledger.resolve(
@@ -244,7 +248,7 @@ class FileManager:
raise error
except Exception as err: # forgive data timeout, don't delete stream
expected = (DownloadSDTimeoutError, DownloadDataTimeoutError, InsufficientFundsError,
KeyFeeAboveMaxAllowedError)
KeyFeeAboveMaxAllowedError, ResolveError, InvalidStreamURLError)
if isinstance(err, expected):
log.warning("Failed to download %s: %s", uri, str(err))
elif isinstance(err, asyncio.CancelledError):
+9 -2
View File
@@ -2,6 +2,9 @@ import logging
from typing import List
from binascii import hexlify, unhexlify
from asn1crypto.keys import PublicKeyInfo
from coincurve import PublicKey as cPublicKey
from google.protobuf.json_format import MessageToDict
from google.protobuf.message import DecodeError
from hachoir.core.log import log as hachoir_log
@@ -346,7 +349,7 @@ class Channel(BaseClaim):
@property
def public_key(self) -> str:
return hexlify(self.message.public_key).decode()
return hexlify(self.public_key_bytes).decode()
@public_key.setter
def public_key(self, sd_public_key: str):
@@ -354,7 +357,11 @@ class Channel(BaseClaim):
@property
def public_key_bytes(self) -> bytes:
return self.message.public_key
if len(self.message.public_key) == 33:
return self.message.public_key
public_key_info = PublicKeyInfo.load(self.message.public_key)
public_key = cPublicKey(public_key_info.native['public_key'])
return public_key.format(compressed=True)
@public_key_bytes.setter
def public_key_bytes(self, public_key: bytes):
+30
View File
@@ -1,4 +1,6 @@
import os
import filetype
import logging
types_map = {
# http://www.iana.org/assignments/media-types
@@ -166,10 +168,38 @@ types_map = {
'.wmv': ('video/x-ms-wmv', 'video')
}
# maps detected extensions to the possible analogs
# i.e. .cbz file is actually a .zip
synonyms_map = {
'.zip': ['.cbz'],
'.rar': ['.cbr'],
'.ar': ['.a']
}
log = logging.getLogger(__name__)
def guess_media_type(path):
_, ext = os.path.splitext(path)
extension = ext.strip().lower()
try:
kind = filetype.guess(path)
if kind:
real_extension = f".{kind.extension}"
if extension != real_extension:
if extension:
log.warning(f"file extension does not match it's contents: {path}, identified as {real_extension}")
else:
log.debug(f"file {path} does not have extension, identified by it's contents as {real_extension}")
if extension not in synonyms_map.get(real_extension, []):
extension = real_extension
except OSError as error:
pass
if extension[1:]:
if extension in types_map:
return types_map[extension]
+1 -1
View File
@@ -7,7 +7,7 @@ from lbry.error import ResolveCensoredError
from lbry.schema.types.v2.result_pb2 import Outputs as OutputsMessage
from lbry.schema.types.v2.result_pb2 import Error as ErrorMessage
if TYPE_CHECKING:
from lbry.wallet.server.leveldb import ResolveResult
from lbry.wallet.server.db.common import ResolveResult
INVALID = ErrorMessage.Code.Name(ErrorMessage.INVALID)
NOT_FOUND = ErrorMessage.Code.Name(ErrorMessage.NOT_FOUND)
+322 -78
View File
@@ -20,7 +20,7 @@ DESCRIPTOR = _descriptor.FileDescriptor(
syntax='proto3',
serialized_options=b'Z$github.com/lbryio/hub/protobuf/go/pb',
create_key=_descriptor._internal_create_key,
serialized_pb=b'\n\thub.proto\x12\x02pb\x1a\x0cresult.proto\"0\n\x0fInvertibleField\x12\x0e\n\x06invert\x18\x01 \x01(\x08\x12\r\n\x05value\x18\x02 \x03(\t\"\x1a\n\tBoolValue\x12\r\n\x05value\x18\x01 \x01(\x08\"\x1c\n\x0bUInt32Value\x12\r\n\x05value\x18\x01 \x01(\r\"j\n\nRangeField\x12\x1d\n\x02op\x18\x01 \x01(\x0e\x32\x11.pb.RangeField.Op\x12\r\n\x05value\x18\x02 \x03(\t\".\n\x02Op\x12\x06\n\x02\x45Q\x10\x00\x12\x07\n\x03LTE\x10\x01\x12\x07\n\x03GTE\x10\x02\x12\x06\n\x02LT\x10\x03\x12\x06\n\x02GT\x10\x04\"\xa3\x0c\n\rSearchRequest\x12%\n\x08\x63laim_id\x18\x01 \x01(\x0b\x32\x13.pb.InvertibleField\x12\'\n\nchannel_id\x18\x02 \x01(\x0b\x32\x13.pb.InvertibleField\x12\x0c\n\x04text\x18\x03 \x01(\t\x12\r\n\x05limit\x18\x04 \x01(\r\x12\x10\n\x08order_by\x18\x05 \x03(\t\x12\x0e\n\x06offset\x18\x06 \x01(\r\x12\x16\n\x0eis_controlling\x18\x07 \x01(\x08\x12\x1d\n\x15last_take_over_height\x18\x08 \x01(\t\x12\x12\n\nclaim_name\x18\t \x01(\t\x12\x17\n\x0fnormalized_name\x18\n \x01(\t\x12#\n\x0btx_position\x18\x0b \x01(\x0b\x32\x0e.pb.RangeField\x12\x1e\n\x06\x61mount\x18\x0c \x01(\x0b\x32\x0e.pb.RangeField\x12!\n\ttimestamp\x18\r \x01(\x0b\x32\x0e.pb.RangeField\x12*\n\x12\x63reation_timestamp\x18\x0e \x01(\x0b\x32\x0e.pb.RangeField\x12\x1e\n\x06height\x18\x0f \x01(\x0b\x32\x0e.pb.RangeField\x12\'\n\x0f\x63reation_height\x18\x10 \x01(\x0b\x32\x0e.pb.RangeField\x12)\n\x11\x61\x63tivation_height\x18\x11 \x01(\x0b\x32\x0e.pb.RangeField\x12)\n\x11\x65xpiration_height\x18\x12 \x01(\x0b\x32\x0e.pb.RangeField\x12$\n\x0crelease_time\x18\x13 \x01(\x0b\x32\x0e.pb.RangeField\x12\x11\n\tshort_url\x18\x14 \x01(\t\x12\x15\n\rcanonical_url\x18\x15 \x01(\t\x12\r\n\x05title\x18\x16 \x01(\t\x12\x0e\n\x06\x61uthor\x18\x17 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x18 \x01(\t\x12\x12\n\nclaim_type\x18\x19 \x03(\t\x12$\n\x0crepost_count\x18\x1a \x01(\x0b\x32\x0e.pb.RangeField\x12\x13\n\x0bstream_type\x18\x1b \x03(\t\x12\x12\n\nmedia_type\x18\x1c \x03(\t\x12\"\n\nfee_amount\x18\x1d \x01(\x0b\x32\x0e.pb.RangeField\x12\x14\n\x0c\x66\x65\x65_currency\x18\x1e \x01(\t\x12 \n\x08\x64uration\x18\x1f \x01(\x0b\x32\x0e.pb.RangeField\x12\x19\n\x11reposted_claim_id\x18 \x01(\t\x12#\n\x0b\x63\x65nsor_type\x18! \x01(\x0b\x32\x0e.pb.RangeField\x12\x19\n\x11\x63laims_in_channel\x18\" \x01(\t\x12$\n\x0c\x63hannel_join\x18# \x01(\x0b\x32\x0e.pb.RangeField\x12)\n\x12is_signature_valid\x18$ \x01(\x0b\x32\r.pb.BoolValue\x12(\n\x10\x65\x66\x66\x65\x63tive_amount\x18% \x01(\x0b\x32\x0e.pb.RangeField\x12&\n\x0esupport_amount\x18& \x01(\x0b\x32\x0e.pb.RangeField\x12&\n\x0etrending_score\x18\' \x01(\x0b\x32\x0e.pb.RangeField\x12\r\n\x05tx_id\x18+ \x01(\t\x12 \n\x07tx_nout\x18, \x01(\x0b\x32\x0f.pb.UInt32Value\x12\x11\n\tsignature\x18- \x01(\t\x12\x18\n\x10signature_digest\x18. \x01(\t\x12\x18\n\x10public_key_bytes\x18/ \x01(\t\x12\x15\n\rpublic_key_id\x18\x30 \x01(\t\x12\x10\n\x08\x61ny_tags\x18\x31 \x03(\t\x12\x10\n\x08\x61ll_tags\x18\x32 \x03(\t\x12\x10\n\x08not_tags\x18\x33 \x03(\t\x12\x1d\n\x15has_channel_signature\x18\x34 \x01(\x08\x12!\n\nhas_source\x18\x35 \x01(\x0b\x32\r.pb.BoolValue\x12 \n\x18limit_claims_per_channel\x18\x36 \x01(\r\x12\x15\n\rany_languages\x18\x37 \x03(\t\x12\x15\n\rall_languages\x18\x38 \x03(\t\x12\x19\n\x11remove_duplicates\x18\x39 \x01(\x08\x12\x11\n\tno_totals\x18: \x01(\x08\x32\x31\n\x03Hub\x12*\n\x06Search\x12\x11.pb.SearchRequest\x1a\x0b.pb.Outputs\"\x00\x42&Z$github.com/lbryio/hub/protobuf/go/pbb\x06proto3'
serialized_pb=b'\n\thub.proto\x12\x02pb\x1a\x0cresult.proto\"\x0e\n\x0c\x45mptyMessage\".\n\rServerMessage\x12\x0f\n\x07\x61\x64\x64ress\x18\x01 \x01(\t\x12\x0c\n\x04port\x18\x02 \x01(\t\"N\n\x0cHelloMessage\x12\x0c\n\x04port\x18\x01 \x01(\t\x12\x0c\n\x04host\x18\x02 \x01(\t\x12\"\n\x07servers\x18\x03 \x03(\x0b\x32\x11.pb.ServerMessage\"0\n\x0fInvertibleField\x12\x0e\n\x06invert\x18\x01 \x01(\x08\x12\r\n\x05value\x18\x02 \x03(\t\"\x1c\n\x0bStringValue\x12\r\n\x05value\x18\x01 \x01(\t\"\x1a\n\tBoolValue\x12\r\n\x05value\x18\x01 \x01(\x08\"\x1c\n\x0bUInt32Value\x12\r\n\x05value\x18\x01 \x01(\r\"j\n\nRangeField\x12\x1d\n\x02op\x18\x01 \x01(\x0e\x32\x11.pb.RangeField.Op\x12\r\n\x05value\x18\x02 \x03(\x05\".\n\x02Op\x12\x06\n\x02\x45Q\x10\x00\x12\x07\n\x03LTE\x10\x01\x12\x07\n\x03GTE\x10\x02\x12\x06\n\x02LT\x10\x03\x12\x06\n\x02GT\x10\x04\"\x8e\x0c\n\rSearchRequest\x12%\n\x08\x63laim_id\x18\x01 \x01(\x0b\x32\x13.pb.InvertibleField\x12\'\n\nchannel_id\x18\x02 \x01(\x0b\x32\x13.pb.InvertibleField\x12\x0c\n\x04text\x18\x03 \x01(\t\x12\r\n\x05limit\x18\x04 \x01(\x05\x12\x10\n\x08order_by\x18\x05 \x03(\t\x12\x0e\n\x06offset\x18\x06 \x01(\r\x12\x16\n\x0eis_controlling\x18\x07 \x01(\x08\x12\x1d\n\x15last_take_over_height\x18\x08 \x01(\t\x12\x12\n\nclaim_name\x18\t \x01(\t\x12\x17\n\x0fnormalized_name\x18\n \x01(\t\x12#\n\x0btx_position\x18\x0b \x03(\x0b\x32\x0e.pb.RangeField\x12\x1e\n\x06\x61mount\x18\x0c \x03(\x0b\x32\x0e.pb.RangeField\x12!\n\ttimestamp\x18\r \x03(\x0b\x32\x0e.pb.RangeField\x12*\n\x12\x63reation_timestamp\x18\x0e \x03(\x0b\x32\x0e.pb.RangeField\x12\x1e\n\x06height\x18\x0f \x03(\x0b\x32\x0e.pb.RangeField\x12\'\n\x0f\x63reation_height\x18\x10 \x03(\x0b\x32\x0e.pb.RangeField\x12)\n\x11\x61\x63tivation_height\x18\x11 \x03(\x0b\x32\x0e.pb.RangeField\x12)\n\x11\x65xpiration_height\x18\x12 \x03(\x0b\x32\x0e.pb.RangeField\x12$\n\x0crelease_time\x18\x13 \x03(\x0b\x32\x0e.pb.RangeField\x12\x11\n\tshort_url\x18\x14 \x01(\t\x12\x15\n\rcanonical_url\x18\x15 \x01(\t\x12\r\n\x05title\x18\x16 \x01(\t\x12\x0e\n\x06\x61uthor\x18\x17 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x18 \x01(\t\x12\x12\n\nclaim_type\x18\x19 \x03(\t\x12$\n\x0crepost_count\x18\x1a \x03(\x0b\x32\x0e.pb.RangeField\x12\x13\n\x0bstream_type\x18\x1b \x03(\t\x12\x12\n\nmedia_type\x18\x1c \x03(\t\x12\"\n\nfee_amount\x18\x1d \x03(\x0b\x32\x0e.pb.RangeField\x12\x14\n\x0c\x66\x65\x65_currency\x18\x1e \x01(\t\x12 \n\x08\x64uration\x18\x1f \x03(\x0b\x32\x0e.pb.RangeField\x12\x19\n\x11reposted_claim_id\x18 \x01(\t\x12#\n\x0b\x63\x65nsor_type\x18! \x03(\x0b\x32\x0e.pb.RangeField\x12\x19\n\x11\x63laims_in_channel\x18\" \x01(\t\x12)\n\x12is_signature_valid\x18$ \x01(\x0b\x32\r.pb.BoolValue\x12(\n\x10\x65\x66\x66\x65\x63tive_amount\x18% \x03(\x0b\x32\x0e.pb.RangeField\x12&\n\x0esupport_amount\x18& \x03(\x0b\x32\x0e.pb.RangeField\x12&\n\x0etrending_score\x18\' \x03(\x0b\x32\x0e.pb.RangeField\x12\r\n\x05tx_id\x18+ \x01(\t\x12 \n\x07tx_nout\x18, \x01(\x0b\x32\x0f.pb.UInt32Value\x12\x11\n\tsignature\x18- \x01(\t\x12\x18\n\x10signature_digest\x18. \x01(\t\x12\x18\n\x10public_key_bytes\x18/ \x01(\t\x12\x15\n\rpublic_key_id\x18\x30 \x01(\t\x12\x10\n\x08\x61ny_tags\x18\x31 \x03(\t\x12\x10\n\x08\x61ll_tags\x18\x32 \x03(\t\x12\x10\n\x08not_tags\x18\x33 \x03(\t\x12\x1d\n\x15has_channel_signature\x18\x34 \x01(\x08\x12!\n\nhas_source\x18\x35 \x01(\x0b\x32\r.pb.BoolValue\x12 \n\x18limit_claims_per_channel\x18\x36 \x01(\x05\x12\x15\n\rany_languages\x18\x37 \x03(\t\x12\x15\n\rall_languages\x18\x38 \x03(\t\x12\x19\n\x11remove_duplicates\x18\x39 \x01(\x08\x12\x11\n\tno_totals\x18: \x01(\x08\x12\x0f\n\x07sd_hash\x18; \x01(\t2\x88\x03\n\x03Hub\x12*\n\x06Search\x12\x11.pb.SearchRequest\x1a\x0b.pb.Outputs\"\x00\x12+\n\x04Ping\x12\x10.pb.EmptyMessage\x1a\x0f.pb.StringValue\"\x00\x12-\n\x05Hello\x12\x10.pb.HelloMessage\x1a\x10.pb.HelloMessage\"\x00\x12/\n\x07\x41\x64\x64Peer\x12\x11.pb.ServerMessage\x1a\x0f.pb.StringValue\"\x00\x12\x35\n\rPeerSubscribe\x12\x11.pb.ServerMessage\x1a\x0f.pb.StringValue\"\x00\x12.\n\x07Version\x12\x10.pb.EmptyMessage\x1a\x0f.pb.StringValue\"\x00\x12/\n\x08\x46\x65\x61tures\x12\x10.pb.EmptyMessage\x1a\x0f.pb.StringValue\"\x00\x12\x30\n\tBroadcast\x12\x10.pb.EmptyMessage\x1a\x0f.pb.UInt32Value\"\x00\x42&Z$github.com/lbryio/hub/protobuf/go/pbb\x06proto3'
,
dependencies=[result__pb2.DESCRIPTOR,])
@@ -61,12 +61,122 @@ _RANGEFIELD_OP = _descriptor.EnumDescriptor(
],
containing_type=None,
serialized_options=None,
serialized_start=199,
serialized_end=245,
serialized_start=373,
serialized_end=419,
)
_sym_db.RegisterEnumDescriptor(_RANGEFIELD_OP)
_EMPTYMESSAGE = _descriptor.Descriptor(
name='EmptyMessage',
full_name='pb.EmptyMessage',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=31,
serialized_end=45,
)
_SERVERMESSAGE = _descriptor.Descriptor(
name='ServerMessage',
full_name='pb.ServerMessage',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
_descriptor.FieldDescriptor(
name='address', full_name='pb.ServerMessage.address', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='port', full_name='pb.ServerMessage.port', index=1,
number=2, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=47,
serialized_end=93,
)
_HELLOMESSAGE = _descriptor.Descriptor(
name='HelloMessage',
full_name='pb.HelloMessage',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
_descriptor.FieldDescriptor(
name='port', full_name='pb.HelloMessage.port', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='host', full_name='pb.HelloMessage.host', index=1,
number=2, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='servers', full_name='pb.HelloMessage.servers', index=2,
number=3, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=95,
serialized_end=173,
)
_INVERTIBLEFIELD = _descriptor.Descriptor(
name='InvertibleField',
full_name='pb.InvertibleField',
@@ -101,8 +211,40 @@ _INVERTIBLEFIELD = _descriptor.Descriptor(
extension_ranges=[],
oneofs=[
],
serialized_start=31,
serialized_end=79,
serialized_start=175,
serialized_end=223,
)
_STRINGVALUE = _descriptor.Descriptor(
name='StringValue',
full_name='pb.StringValue',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
_descriptor.FieldDescriptor(
name='value', full_name='pb.StringValue.value', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=225,
serialized_end=253,
)
@@ -133,8 +275,8 @@ _BOOLVALUE = _descriptor.Descriptor(
extension_ranges=[],
oneofs=[
],
serialized_start=81,
serialized_end=107,
serialized_start=255,
serialized_end=281,
)
@@ -165,8 +307,8 @@ _UINT32VALUE = _descriptor.Descriptor(
extension_ranges=[],
oneofs=[
],
serialized_start=109,
serialized_end=137,
serialized_start=283,
serialized_end=311,
)
@@ -187,7 +329,7 @@ _RANGEFIELD = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='value', full_name='pb.RangeField.value', index=1,
number=2, type=9, cpp_type=9, label=3,
number=2, type=5, cpp_type=1, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
@@ -205,8 +347,8 @@ _RANGEFIELD = _descriptor.Descriptor(
extension_ranges=[],
oneofs=[
],
serialized_start=139,
serialized_end=245,
serialized_start=313,
serialized_end=419,
)
@@ -241,7 +383,7 @@ _SEARCHREQUEST = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='limit', full_name='pb.SearchRequest.limit', index=3,
number=4, type=13, cpp_type=3, label=1,
number=4, type=5, cpp_type=1, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
@@ -290,64 +432,64 @@ _SEARCHREQUEST = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='tx_position', full_name='pb.SearchRequest.tx_position', index=10,
number=11, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=11, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='amount', full_name='pb.SearchRequest.amount', index=11,
number=12, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=12, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='timestamp', full_name='pb.SearchRequest.timestamp', index=12,
number=13, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=13, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='creation_timestamp', full_name='pb.SearchRequest.creation_timestamp', index=13,
number=14, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=14, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='height', full_name='pb.SearchRequest.height', index=14,
number=15, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=15, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='creation_height', full_name='pb.SearchRequest.creation_height', index=15,
number=16, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=16, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='activation_height', full_name='pb.SearchRequest.activation_height', index=16,
number=17, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=17, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='expiration_height', full_name='pb.SearchRequest.expiration_height', index=17,
number=18, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=18, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='release_time', full_name='pb.SearchRequest.release_time', index=18,
number=19, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=19, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
@@ -395,8 +537,8 @@ _SEARCHREQUEST = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='repost_count', full_name='pb.SearchRequest.repost_count', index=25,
number=26, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=26, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
@@ -416,8 +558,8 @@ _SEARCHREQUEST = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='fee_amount', full_name='pb.SearchRequest.fee_amount', index=28,
number=29, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=29, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
@@ -430,8 +572,8 @@ _SEARCHREQUEST = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='duration', full_name='pb.SearchRequest.duration', index=30,
number=31, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=31, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
@@ -444,8 +586,8 @@ _SEARCHREQUEST = _descriptor.Descriptor(
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='censor_type', full_name='pb.SearchRequest.censor_type', index=32,
number=33, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
number=33, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
@@ -457,152 +599,152 @@ _SEARCHREQUEST = _descriptor.Descriptor(
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='channel_join', full_name='pb.SearchRequest.channel_join', index=34,
number=35, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='is_signature_valid', full_name='pb.SearchRequest.is_signature_valid', index=35,
name='is_signature_valid', full_name='pb.SearchRequest.is_signature_valid', index=34,
number=36, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='effective_amount', full_name='pb.SearchRequest.effective_amount', index=36,
number=37, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
name='effective_amount', full_name='pb.SearchRequest.effective_amount', index=35,
number=37, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='support_amount', full_name='pb.SearchRequest.support_amount', index=37,
number=38, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
name='support_amount', full_name='pb.SearchRequest.support_amount', index=36,
number=38, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='trending_score', full_name='pb.SearchRequest.trending_score', index=38,
number=39, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
name='trending_score', full_name='pb.SearchRequest.trending_score', index=37,
number=39, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='tx_id', full_name='pb.SearchRequest.tx_id', index=39,
name='tx_id', full_name='pb.SearchRequest.tx_id', index=38,
number=43, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='tx_nout', full_name='pb.SearchRequest.tx_nout', index=40,
name='tx_nout', full_name='pb.SearchRequest.tx_nout', index=39,
number=44, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='signature', full_name='pb.SearchRequest.signature', index=41,
name='signature', full_name='pb.SearchRequest.signature', index=40,
number=45, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='signature_digest', full_name='pb.SearchRequest.signature_digest', index=42,
name='signature_digest', full_name='pb.SearchRequest.signature_digest', index=41,
number=46, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='public_key_bytes', full_name='pb.SearchRequest.public_key_bytes', index=43,
name='public_key_bytes', full_name='pb.SearchRequest.public_key_bytes', index=42,
number=47, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='public_key_id', full_name='pb.SearchRequest.public_key_id', index=44,
name='public_key_id', full_name='pb.SearchRequest.public_key_id', index=43,
number=48, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='any_tags', full_name='pb.SearchRequest.any_tags', index=45,
name='any_tags', full_name='pb.SearchRequest.any_tags', index=44,
number=49, type=9, cpp_type=9, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='all_tags', full_name='pb.SearchRequest.all_tags', index=46,
name='all_tags', full_name='pb.SearchRequest.all_tags', index=45,
number=50, type=9, cpp_type=9, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='not_tags', full_name='pb.SearchRequest.not_tags', index=47,
name='not_tags', full_name='pb.SearchRequest.not_tags', index=46,
number=51, type=9, cpp_type=9, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='has_channel_signature', full_name='pb.SearchRequest.has_channel_signature', index=48,
name='has_channel_signature', full_name='pb.SearchRequest.has_channel_signature', index=47,
number=52, type=8, cpp_type=7, label=1,
has_default_value=False, default_value=False,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='has_source', full_name='pb.SearchRequest.has_source', index=49,
name='has_source', full_name='pb.SearchRequest.has_source', index=48,
number=53, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='limit_claims_per_channel', full_name='pb.SearchRequest.limit_claims_per_channel', index=50,
number=54, type=13, cpp_type=3, label=1,
name='limit_claims_per_channel', full_name='pb.SearchRequest.limit_claims_per_channel', index=49,
number=54, type=5, cpp_type=1, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='any_languages', full_name='pb.SearchRequest.any_languages', index=51,
name='any_languages', full_name='pb.SearchRequest.any_languages', index=50,
number=55, type=9, cpp_type=9, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='all_languages', full_name='pb.SearchRequest.all_languages', index=52,
name='all_languages', full_name='pb.SearchRequest.all_languages', index=51,
number=56, type=9, cpp_type=9, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='remove_duplicates', full_name='pb.SearchRequest.remove_duplicates', index=53,
name='remove_duplicates', full_name='pb.SearchRequest.remove_duplicates', index=52,
number=57, type=8, cpp_type=7, label=1,
has_default_value=False, default_value=False,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='no_totals', full_name='pb.SearchRequest.no_totals', index=54,
name='no_totals', full_name='pb.SearchRequest.no_totals', index=53,
number=58, type=8, cpp_type=7, label=1,
has_default_value=False, default_value=False,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='sd_hash', full_name='pb.SearchRequest.sd_hash', index=54,
number=59, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
],
extensions=[
],
@@ -615,10 +757,11 @@ _SEARCHREQUEST = _descriptor.Descriptor(
extension_ranges=[],
oneofs=[
],
serialized_start=248,
serialized_end=1819,
serialized_start=422,
serialized_end=1972,
)
_HELLOMESSAGE.fields_by_name['servers'].message_type = _SERVERMESSAGE
_RANGEFIELD.fields_by_name['op'].enum_type = _RANGEFIELD_OP
_RANGEFIELD_OP.containing_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['claim_id'].message_type = _INVERTIBLEFIELD
@@ -636,20 +779,44 @@ _SEARCHREQUEST.fields_by_name['repost_count'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['fee_amount'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['duration'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['censor_type'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['channel_join'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['is_signature_valid'].message_type = _BOOLVALUE
_SEARCHREQUEST.fields_by_name['effective_amount'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['support_amount'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['trending_score'].message_type = _RANGEFIELD
_SEARCHREQUEST.fields_by_name['tx_nout'].message_type = _UINT32VALUE
_SEARCHREQUEST.fields_by_name['has_source'].message_type = _BOOLVALUE
DESCRIPTOR.message_types_by_name['EmptyMessage'] = _EMPTYMESSAGE
DESCRIPTOR.message_types_by_name['ServerMessage'] = _SERVERMESSAGE
DESCRIPTOR.message_types_by_name['HelloMessage'] = _HELLOMESSAGE
DESCRIPTOR.message_types_by_name['InvertibleField'] = _INVERTIBLEFIELD
DESCRIPTOR.message_types_by_name['StringValue'] = _STRINGVALUE
DESCRIPTOR.message_types_by_name['BoolValue'] = _BOOLVALUE
DESCRIPTOR.message_types_by_name['UInt32Value'] = _UINT32VALUE
DESCRIPTOR.message_types_by_name['RangeField'] = _RANGEFIELD
DESCRIPTOR.message_types_by_name['SearchRequest'] = _SEARCHREQUEST
_sym_db.RegisterFileDescriptor(DESCRIPTOR)
EmptyMessage = _reflection.GeneratedProtocolMessageType('EmptyMessage', (_message.Message,), {
'DESCRIPTOR' : _EMPTYMESSAGE,
'__module__' : 'hub_pb2'
# @@protoc_insertion_point(class_scope:pb.EmptyMessage)
})
_sym_db.RegisterMessage(EmptyMessage)
ServerMessage = _reflection.GeneratedProtocolMessageType('ServerMessage', (_message.Message,), {
'DESCRIPTOR' : _SERVERMESSAGE,
'__module__' : 'hub_pb2'
# @@protoc_insertion_point(class_scope:pb.ServerMessage)
})
_sym_db.RegisterMessage(ServerMessage)
HelloMessage = _reflection.GeneratedProtocolMessageType('HelloMessage', (_message.Message,), {
'DESCRIPTOR' : _HELLOMESSAGE,
'__module__' : 'hub_pb2'
# @@protoc_insertion_point(class_scope:pb.HelloMessage)
})
_sym_db.RegisterMessage(HelloMessage)
InvertibleField = _reflection.GeneratedProtocolMessageType('InvertibleField', (_message.Message,), {
'DESCRIPTOR' : _INVERTIBLEFIELD,
'__module__' : 'hub_pb2'
@@ -657,6 +824,13 @@ InvertibleField = _reflection.GeneratedProtocolMessageType('InvertibleField', (_
})
_sym_db.RegisterMessage(InvertibleField)
StringValue = _reflection.GeneratedProtocolMessageType('StringValue', (_message.Message,), {
'DESCRIPTOR' : _STRINGVALUE,
'__module__' : 'hub_pb2'
# @@protoc_insertion_point(class_scope:pb.StringValue)
})
_sym_db.RegisterMessage(StringValue)
BoolValue = _reflection.GeneratedProtocolMessageType('BoolValue', (_message.Message,), {
'DESCRIPTOR' : _BOOLVALUE,
'__module__' : 'hub_pb2'
@@ -695,8 +869,8 @@ _HUB = _descriptor.ServiceDescriptor(
index=0,
serialized_options=None,
create_key=_descriptor._internal_create_key,
serialized_start=1821,
serialized_end=1870,
serialized_start=1975,
serialized_end=2367,
methods=[
_descriptor.MethodDescriptor(
name='Search',
@@ -708,6 +882,76 @@ _HUB = _descriptor.ServiceDescriptor(
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='Ping',
full_name='pb.Hub.Ping',
index=1,
containing_service=None,
input_type=_EMPTYMESSAGE,
output_type=_STRINGVALUE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='Hello',
full_name='pb.Hub.Hello',
index=2,
containing_service=None,
input_type=_HELLOMESSAGE,
output_type=_HELLOMESSAGE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='AddPeer',
full_name='pb.Hub.AddPeer',
index=3,
containing_service=None,
input_type=_SERVERMESSAGE,
output_type=_STRINGVALUE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='PeerSubscribe',
full_name='pb.Hub.PeerSubscribe',
index=4,
containing_service=None,
input_type=_SERVERMESSAGE,
output_type=_STRINGVALUE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='Version',
full_name='pb.Hub.Version',
index=5,
containing_service=None,
input_type=_EMPTYMESSAGE,
output_type=_STRINGVALUE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='Features',
full_name='pb.Hub.Features',
index=6,
containing_service=None,
input_type=_EMPTYMESSAGE,
output_type=_STRINGVALUE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
_descriptor.MethodDescriptor(
name='Broadcast',
full_name='pb.Hub.Broadcast',
index=7,
containing_service=None,
input_type=_EMPTYMESSAGE,
output_type=_UINT32VALUE,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
])
_sym_db.RegisterServiceDescriptor(_HUB)
+231
View File
@@ -20,6 +20,41 @@ class HubStub(object):
request_serializer=hub__pb2.SearchRequest.SerializeToString,
response_deserializer=result__pb2.Outputs.FromString,
)
self.Ping = channel.unary_unary(
'/pb.Hub/Ping',
request_serializer=hub__pb2.EmptyMessage.SerializeToString,
response_deserializer=hub__pb2.StringValue.FromString,
)
self.Hello = channel.unary_unary(
'/pb.Hub/Hello',
request_serializer=hub__pb2.HelloMessage.SerializeToString,
response_deserializer=hub__pb2.HelloMessage.FromString,
)
self.AddPeer = channel.unary_unary(
'/pb.Hub/AddPeer',
request_serializer=hub__pb2.ServerMessage.SerializeToString,
response_deserializer=hub__pb2.StringValue.FromString,
)
self.PeerSubscribe = channel.unary_unary(
'/pb.Hub/PeerSubscribe',
request_serializer=hub__pb2.ServerMessage.SerializeToString,
response_deserializer=hub__pb2.StringValue.FromString,
)
self.Version = channel.unary_unary(
'/pb.Hub/Version',
request_serializer=hub__pb2.EmptyMessage.SerializeToString,
response_deserializer=hub__pb2.StringValue.FromString,
)
self.Features = channel.unary_unary(
'/pb.Hub/Features',
request_serializer=hub__pb2.EmptyMessage.SerializeToString,
response_deserializer=hub__pb2.StringValue.FromString,
)
self.Broadcast = channel.unary_unary(
'/pb.Hub/Broadcast',
request_serializer=hub__pb2.EmptyMessage.SerializeToString,
response_deserializer=hub__pb2.UInt32Value.FromString,
)
class HubServicer(object):
@@ -31,6 +66,48 @@ class HubServicer(object):
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Ping(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Hello(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def AddPeer(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def PeerSubscribe(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Version(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Features(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Broadcast(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_HubServicer_to_server(servicer, server):
rpc_method_handlers = {
@@ -39,6 +116,41 @@ def add_HubServicer_to_server(servicer, server):
request_deserializer=hub__pb2.SearchRequest.FromString,
response_serializer=result__pb2.Outputs.SerializeToString,
),
'Ping': grpc.unary_unary_rpc_method_handler(
servicer.Ping,
request_deserializer=hub__pb2.EmptyMessage.FromString,
response_serializer=hub__pb2.StringValue.SerializeToString,
),
'Hello': grpc.unary_unary_rpc_method_handler(
servicer.Hello,
request_deserializer=hub__pb2.HelloMessage.FromString,
response_serializer=hub__pb2.HelloMessage.SerializeToString,
),
'AddPeer': grpc.unary_unary_rpc_method_handler(
servicer.AddPeer,
request_deserializer=hub__pb2.ServerMessage.FromString,
response_serializer=hub__pb2.StringValue.SerializeToString,
),
'PeerSubscribe': grpc.unary_unary_rpc_method_handler(
servicer.PeerSubscribe,
request_deserializer=hub__pb2.ServerMessage.FromString,
response_serializer=hub__pb2.StringValue.SerializeToString,
),
'Version': grpc.unary_unary_rpc_method_handler(
servicer.Version,
request_deserializer=hub__pb2.EmptyMessage.FromString,
response_serializer=hub__pb2.StringValue.SerializeToString,
),
'Features': grpc.unary_unary_rpc_method_handler(
servicer.Features,
request_deserializer=hub__pb2.EmptyMessage.FromString,
response_serializer=hub__pb2.StringValue.SerializeToString,
),
'Broadcast': grpc.unary_unary_rpc_method_handler(
servicer.Broadcast,
request_deserializer=hub__pb2.EmptyMessage.FromString,
response_serializer=hub__pb2.UInt32Value.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'pb.Hub', rpc_method_handlers)
@@ -65,3 +177,122 @@ class Hub(object):
result__pb2.Outputs.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Ping(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/Ping',
hub__pb2.EmptyMessage.SerializeToString,
hub__pb2.StringValue.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Hello(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/Hello',
hub__pb2.HelloMessage.SerializeToString,
hub__pb2.HelloMessage.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def AddPeer(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/AddPeer',
hub__pb2.ServerMessage.SerializeToString,
hub__pb2.StringValue.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def PeerSubscribe(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/PeerSubscribe',
hub__pb2.ServerMessage.SerializeToString,
hub__pb2.StringValue.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Version(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/Version',
hub__pb2.EmptyMessage.SerializeToString,
hub__pb2.StringValue.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Features(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/Features',
hub__pb2.EmptyMessage.SerializeToString,
hub__pb2.StringValue.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Broadcast(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/pb.Hub/Broadcast',
hub__pb2.EmptyMessage.SerializeToString,
hub__pb2.UInt32Value.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+30
View File
@@ -0,0 +1,30 @@
import asyncio
import logging
from lbry.stream.downloader import StreamDownloader
log = logging.getLogger(__name__)
class BackgroundDownloader:
def __init__(self, conf, storage, blob_manager, dht_node=None):
self.storage = storage
self.blob_manager = blob_manager
self.node = dht_node
self.conf = conf
async def download_blobs(self, sd_hash):
downloader = StreamDownloader(asyncio.get_running_loop(), self.conf, self.blob_manager, sd_hash)
try:
await downloader.start(self.node, save_stream=False)
for blob_info in downloader.descriptor.blobs[:-1]:
await downloader.download_stream_blob(blob_info)
except ValueError:
return
except asyncio.CancelledError:
raise
except Exception:
log.error("Unexpected download error on background downloader")
finally:
downloader.stop()
+8 -19
View File
@@ -3,9 +3,9 @@ import typing
import logging
import binascii
from lbry.dht.peer import make_kademlia_peer
from lbry.dht.node import get_kademlia_peers_from_hosts
from lbry.error import DownloadSDTimeoutError
from lbry.utils import resolve_host, lru_cache_concurrent
from lbry.utils import lru_cache_concurrent
from lbry.stream.descriptor import StreamDescriptor
from lbry.blob_exchange.downloader import BlobDownloader
if typing.TYPE_CHECKING:
@@ -48,26 +48,19 @@ class StreamDownloader:
self.cached_read_blob = cached_read_blob
async def add_fixed_peers(self):
def _delayed_add_fixed_peers():
def _add_fixed_peers(fixed_peers):
self.peer_queue.put_nowait(fixed_peers)
self.added_fixed_peers = True
self.peer_queue.put_nowait([
make_kademlia_peer(None, address, None, tcp_port=port, allow_localhost=True)
for address, port in addresses
])
if not self.config.fixed_peers:
return
addresses = [
(await resolve_host(url, port, proto='tcp'), port)
for url, port in self.config.fixed_peers
]
if 'dht' in self.config.components_to_skip or not self.node or not \
len(self.node.protocol.routing_table.get_peers()) > 0:
self.fixed_peers_delay = 0.0
else:
self.fixed_peers_delay = self.config.fixed_peer_delay
self.fixed_peers_handle = self.loop.call_later(self.fixed_peers_delay, _delayed_add_fixed_peers)
fixed_peers = await get_kademlia_peers_from_hosts(self.config.fixed_peers)
self.fixed_peers_handle = self.loop.call_later(self.fixed_peers_delay, _add_fixed_peers, fixed_peers)
async def load_descriptor(self, connection_id: int = 0):
# download or get the sd blob
@@ -90,7 +83,7 @@ class StreamDownloader:
)
log.info("loaded stream manifest %s", self.sd_hash)
async def start(self, node: typing.Optional['Node'] = None, connection_id: int = 0):
async def start(self, node: typing.Optional['Node'] = None, connection_id: int = 0, save_stream=True):
# set up peer accumulation
self.node = node or self.node # fixme: this shouldnt be set here!
if self.node:
@@ -105,11 +98,7 @@ class StreamDownloader:
if not self.descriptor:
await self.load_descriptor(connection_id)
# add the head blob to the peer search
self.search_queue.put_nowait(self.descriptor.blobs[0].blob_hash)
log.info("added head blob to peer search for stream %s", self.sd_hash)
if not await self.blob_manager.storage.stream_exists(self.sd_hash):
if not await self.blob_manager.storage.stream_exists(self.sd_hash) and save_stream:
await self.blob_manager.storage.store_stream(
self.blob_manager.get_blob(self.sd_hash, length=self.descriptor.length), self.descriptor
)
+89 -23
View File
@@ -17,8 +17,10 @@ from functools import partial
from lbry.wallet import WalletManager, Wallet, Ledger, Account, Transaction
from lbry.conf import Config
from lbry.wallet.util import satoshis_to_coins
from lbry.wallet.dewies import lbc_to_dewies
from lbry.wallet.orchstr8 import Conductor
from lbry.wallet.orchstr8.node import BlockchainNode, WalletNode, HubNode
from lbry.wallet.orchstr8.node import LBCWalletNode, WalletNode, HubNode
from lbry.schema.claim import Claim
from lbry.extras.daemon.daemon import Daemon, jsonrpc_dumps_pretty
from lbry.extras.daemon.components import Component, WalletComponent
@@ -132,17 +134,18 @@ class AsyncioTestCase(unittest.TestCase):
with outcome.testPartExecutor(self):
self.setUp()
self.add_timeout()
self.loop.run_until_complete(self.asyncSetUp())
if outcome.success:
outcome.expecting_failure = expecting_failure
with outcome.testPartExecutor(self, isTest=True):
maybe_coroutine = testMethod()
if asyncio.iscoroutine(maybe_coroutine):
if self.TIMEOUT:
self.loop.call_later(self.TIMEOUT, self.cancel)
self.add_timeout()
self.loop.run_until_complete(maybe_coroutine)
outcome.expecting_failure = False
with outcome.testPartExecutor(self):
self.add_timeout()
self.loop.run_until_complete(self.asyncTearDown())
self.tearDown()
@@ -190,6 +193,7 @@ class AsyncioTestCase(unittest.TestCase):
with outcome.testPartExecutor(self):
maybe_coroutine = function(*args, **kwargs)
if asyncio.iscoroutine(maybe_coroutine):
self.add_timeout()
self.loop.run_until_complete(maybe_coroutine)
def cancel(self):
@@ -198,6 +202,10 @@ class AsyncioTestCase(unittest.TestCase):
task.print_stack()
task.cancel()
def add_timeout(self):
if self.TIMEOUT:
self.loop.call_later(self.TIMEOUT, self.cancel)
class AdvanceTimeTestCase(AsyncioTestCase):
@@ -222,7 +230,7 @@ class IntegrationTestCase(AsyncioTestCase):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.conductor: Optional[Conductor] = None
self.blockchain: Optional[BlockchainNode] = None
self.blockchain: Optional[LBCWalletNode] = None
self.hub: Optional[HubNode] = None
self.wallet_node: Optional[WalletNode] = None
self.manager: Optional[WalletManager] = None
@@ -232,15 +240,17 @@ class IntegrationTestCase(AsyncioTestCase):
async def asyncSetUp(self):
self.conductor = Conductor(seed=self.SEED)
await self.conductor.start_blockchain()
self.addCleanup(self.conductor.stop_blockchain)
await self.conductor.start_lbcd()
self.addCleanup(self.conductor.stop_lbcd)
await self.conductor.start_lbcwallet()
self.addCleanup(self.conductor.stop_lbcwallet)
await self.conductor.start_spv()
self.addCleanup(self.conductor.stop_spv)
await self.conductor.start_wallet()
self.addCleanup(self.conductor.stop_wallet)
await self.conductor.start_hub()
self.addCleanup(self.conductor.stop_hub)
self.blockchain = self.conductor.blockchain_node
self.blockchain = self.conductor.lbcwallet_node
self.hub = self.conductor.hub_node
self.wallet_node = self.conductor.wallet_node
self.manager = self.wallet_node.manager
@@ -255,6 +265,13 @@ class IntegrationTestCase(AsyncioTestCase):
def broadcast(self, tx):
return self.ledger.broadcast(tx)
async def broadcast_and_confirm(self, tx, ledger=None):
ledger = ledger or self.ledger
notifications = asyncio.create_task(ledger.wait(tx))
await ledger.broadcast(tx)
await notifications
await self.generate_and_wait(1, [tx.id], ledger)
async def on_header(self, height):
if self.ledger.headers.height < height:
await self.ledger.on_header.where(
@@ -262,11 +279,36 @@ class IntegrationTestCase(AsyncioTestCase):
)
return True
def on_transaction_id(self, txid, ledger=None):
return (ledger or self.ledger).on_transaction.where(
lambda e: e.tx.id == txid
async def send_to_address_and_wait(self, address, amount, blocks_to_generate=0, ledger=None):
tx_watch = []
txid = None
done = False
watcher = (ledger or self.ledger).on_transaction.where(
lambda e: e.tx.id == txid or done or tx_watch.append(e.tx.id)
)
txid = await self.blockchain.send_to_address(address, amount)
done = txid in tx_watch
await watcher
await self.generate_and_wait(blocks_to_generate, [txid], ledger)
return txid
async def generate_and_wait(self, blocks_to_generate, txids, ledger=None):
if blocks_to_generate > 0:
watcher = (ledger or self.ledger).on_transaction.where(
lambda e: ((e.tx.id in txids and txids.remove(e.tx.id)), len(txids) <= 0)[-1] # multi-statement lambda
)
self.conductor.spv_node.server.synchronized.clear()
await self.blockchain.generate(blocks_to_generate)
height = self.blockchain.block_expected
await watcher
while True:
await self.conductor.spv_node.server.synchronized.wait()
self.conductor.spv_node.server.synchronized.clear()
if self.conductor.spv_node.server.db.db_height >= height:
break
def on_address_update(self, address):
return self.ledger.on_transaction.where(
lambda e: e.address == address
@@ -277,6 +319,19 @@ class IntegrationTestCase(AsyncioTestCase):
lambda e: e.tx.id == tx.id and e.address == address
)
async def generate(self, blocks):
""" Ask lbrycrd to generate some blocks and wait until ledger has them. """
prepare = self.ledger.on_header.where(self.blockchain.is_expected_block)
height = self.blockchain.block_expected
self.conductor.spv_node.server.synchronized.clear()
await self.blockchain.generate(blocks)
await prepare # no guarantee that it didn't happen already, so start waiting from before calling generate
while True:
await self.conductor.spv_node.server.synchronized.wait()
self.conductor.spv_node.server.synchronized.clear()
if self.conductor.spv_node.server.db.db_height >= height:
break
class FakeExchangeRateManager(ExchangeRateManager):
@@ -337,20 +392,19 @@ class CommandTestCase(IntegrationTestCase):
self.skip_libtorrent = True
async def asyncSetUp(self):
await super().asyncSetUp()
logging.getLogger('lbry.blob_exchange').setLevel(self.VERBOSITY)
logging.getLogger('lbry.daemon').setLevel(self.VERBOSITY)
logging.getLogger('lbry.stream').setLevel(self.VERBOSITY)
logging.getLogger('lbry.wallet').setLevel(self.VERBOSITY)
await super().asyncSetUp()
self.daemon = await self.add_daemon(self.wallet_node)
await self.account.ensure_address_gap()
address = (await self.account.receiving.get_addresses(limit=1, only_usable=True))[0]
sendtxid = await self.blockchain.send_to_address(address, 10)
await self.confirm_tx(sendtxid)
await self.generate(5)
await self.send_to_address_and_wait(address, 10, 6)
server_tmp_dir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, server_tmp_dir)
@@ -447,9 +501,14 @@ class CommandTestCase(IntegrationTestCase):
async def confirm_tx(self, txid, ledger=None):
""" Wait for tx to be in mempool, then generate a block, wait for tx to be in a block. """
await self.on_transaction_id(txid, ledger)
await self.generate(1)
await self.on_transaction_id(txid, ledger)
# await (ledger or self.ledger).on_transaction.where(lambda e: e.tx.id == txid)
on_tx = (ledger or self.ledger).on_transaction.where(lambda e: e.tx.id == txid)
await asyncio.wait([self.generate(1), on_tx], timeout=5)
# # actually, if it's in the mempool or in the block we're fine
# await self.generate_and_wait(1, [txid], ledger=ledger)
# return txid
return txid
async def on_transaction_dict(self, tx):
@@ -464,12 +523,6 @@ class CommandTestCase(IntegrationTestCase):
addresses.add(txo['address'])
return list(addresses)
async def generate(self, blocks):
""" Ask lbrycrd to generate some blocks and wait until ledger has them. """
prepare = self.ledger.on_header.where(self.blockchain.is_expected_block)
await self.blockchain.generate(blocks)
await prepare # no guarantee that it didn't happen already, so start waiting from before calling generate
async def blockchain_claim_name(self, name: str, value: str, amount: str, confirm=True):
txid = await self.blockchain._cli_cmnd('claimname', name, value, amount)
if confirm:
@@ -500,6 +553,19 @@ class CommandTestCase(IntegrationTestCase):
return self.sout(tx)
return tx
async def create_nondeterministic_channel(self, name, price, pubkey_bytes, daemon=None, blocking=False):
account = (daemon or self.daemon).wallet_manager.default_account
claim_address = await account.receiving.get_or_create_usable_address()
claim = Claim()
claim.channel.public_key_bytes = pubkey_bytes
tx = await Transaction.claim_create(
name, claim, lbc_to_dewies(price),
claim_address, [self.account], self.account
)
await tx.sign([self.account])
await (daemon or self.daemon).broadcast_or_release(tx, blocking)
return self.sout(tx)
def create_upload_file(self, data, prefix=None, suffix=None):
file_path = tempfile.mktemp(prefix=prefix or "tmp", suffix=suffix or "", dir=self.daemon.conf.upload_dir)
with open(file_path, 'w+b') as file:
+20 -14
View File
@@ -1,17 +1,23 @@
__node_daemon__ = 'lbrycrdd'
__node_cli__ = 'lbrycrd-cli'
__node_bin__ = ''
__node_url__ = (
'https://github.com/lbryio/lbrycrd/releases/download/v0.17.4.6/lbrycrd-linux-1746.zip'
__lbcd__ = 'lbcd'
__lbcctl__ = 'lbcctl'
__lbcwallet__ = 'lbcwallet'
__lbcd_url__ = (
'https://github.com/lbryio/lbcd/releases/download/' +
'v0.22.200-beta/lbcd_0.22.200-beta_TARGET_PLATFORM.tar.gz'
)
__lbcwallet_url__ = (
'https://github.com/lbryio/lbcwallet/releases/download/' +
'v0.13.100-alpha-rc2/lbcwallet_0.13.100-alpha-rc2_TARGET_PLATFORM.tar.gz'
)
__spvserver__ = 'lbry.wallet.server.coin.LBCRegTest'
from .wallet import Wallet, WalletStorage, TimestampedPreferences, ENCRYPT_ON_DISK
from .manager import WalletManager
from .network import Network
from .ledger import Ledger, RegTestLedger, TestNetLedger, BlockHeightEvent
from .account import Account, AddressManager, SingleKey, HierarchicalDeterministic
from .transaction import Transaction, Output, Input
from .script import OutputScript, InputScript
from .database import SQLiteMixin, Database
from .header import Headers
from lbry.wallet.wallet import Wallet, WalletStorage, TimestampedPreferences, ENCRYPT_ON_DISK
from lbry.wallet.manager import WalletManager
from lbry.wallet.network import Network
from lbry.wallet.ledger import Ledger, RegTestLedger, TestNetLedger, BlockHeightEvent
from lbry.wallet.account import Account, AddressManager, SingleKey, HierarchicalDeterministic, \
DeterministicChannelKeyManager
from lbry.wallet.transaction import Transaction, Output, Input
from lbry.wallet.script import OutputScript, InputScript
from lbry.wallet.database import SQLiteMixin, Database
from lbry.wallet.header import Headers
Binary file not shown.
+70 -33
View File
@@ -9,11 +9,10 @@ from hashlib import sha256
from string import hexdigits
from typing import Type, Dict, Tuple, Optional, Any, List
import ecdsa
from lbry.error import InvalidPasswordError
from lbry.crypto.crypt import aes_encrypt, aes_decrypt
from .bip32 import PrivateKey, PubKey, from_extended_key_string
from .bip32 import PrivateKey, PublicKey, KeyPath, from_extended_key_string
from .mnemonic import Mnemonic
from .constants import COIN, TXO_TYPES
from .transaction import Transaction, Input, Output
@@ -34,6 +33,49 @@ def validate_claim_id(claim_id):
raise Exception("Claim id is not hex encoded")
class DeterministicChannelKeyManager:
def __init__(self, account: 'Account'):
self.account = account
self.last_known = 0
self.cache = {}
self._private_key: Optional[PrivateKey] = None
@property
def private_key(self):
if self._private_key is None:
if self.account.private_key is not None:
self._private_key = self.account.private_key.child(KeyPath.CHANNEL)
return self._private_key
def maybe_generate_deterministic_key_for_channel(self, txo):
if self.private_key is None:
return
next_private_key = self.private_key.child(self.last_known)
public_key = next_private_key.public_key
public_key_bytes = public_key.pubkey_bytes
if txo.claim.channel.public_key_bytes == public_key_bytes:
self.cache[public_key.address] = next_private_key
self.last_known += 1
async def ensure_cache_primed(self):
if self.private_key is not None:
await self.generate_next_key()
async def generate_next_key(self) -> PrivateKey:
db = self.account.ledger.db
while True:
next_private_key = self.private_key.child(self.last_known)
public_key = next_private_key.public_key
self.cache[public_key.address] = next_private_key
if not await db.is_channel_key_used(self.account, public_key):
return next_private_key
self.last_known += 1
def get_private_key_from_pubkey_hash(self, pubkey_hash) -> PrivateKey:
return self.cache.get(pubkey_hash)
class AddressManager:
name: str
@@ -79,7 +121,7 @@ class AddressManager:
def get_private_key(self, index: int) -> PrivateKey:
raise NotImplementedError
def get_public_key(self, index: int) -> PubKey:
def get_public_key(self, index: int) -> PublicKey:
raise NotImplementedError
async def get_max_gap(self):
@@ -119,8 +161,8 @@ class HierarchicalDeterministic(AddressManager):
@classmethod
def from_dict(cls, account: 'Account', d: dict) -> Tuple[AddressManager, AddressManager]:
return (
cls(account, 0, **d.get('receiving', {'gap': 20, 'maximum_uses_per_address': 1})),
cls(account, 1, **d.get('change', {'gap': 6, 'maximum_uses_per_address': 1}))
cls(account, KeyPath.RECEIVE, **d.get('receiving', {'gap': 20, 'maximum_uses_per_address': 1})),
cls(account, KeyPath.CHANGE, **d.get('change', {'gap': 6, 'maximum_uses_per_address': 1}))
)
def merge(self, d: dict):
@@ -133,7 +175,7 @@ class HierarchicalDeterministic(AddressManager):
def get_private_key(self, index: int) -> PrivateKey:
return self.account.private_key.child(self.chain_number).child(index)
def get_public_key(self, index: int) -> PubKey:
def get_public_key(self, index: int) -> PublicKey:
return self.account.public_key.child(self.chain_number).child(index)
async def get_max_gap(self) -> int:
@@ -193,7 +235,7 @@ class SingleKey(AddressManager):
@classmethod
def from_dict(cls, account: 'Account', d: dict) \
-> Tuple[AddressManager, AddressManager]:
same_address_manager = cls(account, account.public_key, 0)
same_address_manager = cls(account, account.public_key, KeyPath.RECEIVE)
return same_address_manager, same_address_manager
def to_dict_instance(self):
@@ -202,7 +244,7 @@ class SingleKey(AddressManager):
def get_private_key(self, index: int) -> PrivateKey:
return self.account.private_key
def get_public_key(self, index: int) -> PubKey:
def get_public_key(self, index: int) -> PublicKey:
return self.account.public_key
async def get_max_gap(self) -> int:
@@ -224,9 +266,6 @@ class SingleKey(AddressManager):
class Account:
mnemonic_class = Mnemonic
private_key_class = PrivateKey
public_key_class = PubKey
address_generators: Dict[str, Type[AddressManager]] = {
SingleKey.name: SingleKey,
HierarchicalDeterministic.name: HierarchicalDeterministic,
@@ -234,7 +273,7 @@ class Account:
def __init__(self, ledger: 'Ledger', wallet: 'Wallet', name: str,
seed: str, private_key_string: str, encrypted: bool,
private_key: Optional[PrivateKey], public_key: PubKey,
private_key: Optional[PrivateKey], public_key: PublicKey,
address_generator: dict, modified_on: float, channel_keys: dict) -> None:
self.ledger = ledger
self.wallet = wallet
@@ -245,13 +284,14 @@ class Account:
self.private_key_string = private_key_string
self.init_vectors: Dict[str, bytes] = {}
self.encrypted = encrypted
self.private_key = private_key
self.public_key = public_key
self.private_key: Optional[PrivateKey] = private_key
self.public_key: PublicKey = public_key
generator_name = address_generator.get('name', HierarchicalDeterministic.name)
self.address_generator = self.address_generators[generator_name]
self.receiving, self.change = self.address_generator.from_dict(self, address_generator)
self.address_managers = {am.chain_number: am for am in (self.receiving, self.change)}
self.channel_keys = channel_keys
self.deterministic_channel_keys = DeterministicChannelKeyManager(self)
ledger.add_account(self)
wallet.add_account(self)
@@ -266,19 +306,19 @@ class Account:
name: str = None, address_generator: dict = None):
return cls.from_dict(ledger, wallet, {
'name': name,
'seed': cls.mnemonic_class().make_seed(),
'seed': Mnemonic().make_seed(),
'address_generator': address_generator or {}
})
@classmethod
def get_private_key_from_seed(cls, ledger: 'Ledger', seed: str, password: str):
return cls.private_key_class.from_seed(
ledger, cls.mnemonic_class.mnemonic_to_seed(seed, password or 'lbryum')
return PrivateKey.from_seed(
ledger, Mnemonic.mnemonic_to_seed(seed, password or 'lbryum')
)
@classmethod
def keys_from_dict(cls, ledger: 'Ledger', d: dict) \
-> Tuple[str, Optional[PrivateKey], PubKey]:
-> Tuple[str, Optional[PrivateKey], PublicKey]:
seed = d.get('seed', '')
private_key_string = d.get('private_key', '')
private_key = None
@@ -449,7 +489,7 @@ class Account:
assert not self.encrypted, "Cannot get private key on encrypted wallet account."
return self.address_managers[chain].get_private_key(index)
def get_public_key(self, chain: int, index: int) -> PubKey:
def get_public_key(self, chain: int, index: int) -> PublicKey:
return self.address_managers[chain].get_public_key(index)
def get_balance(self, confirmations=0, include_claims=False, read_only=False, **constraints):
@@ -520,33 +560,30 @@ class Account:
return tx
def add_channel_private_key(self, private_key):
public_key_bytes = private_key.get_verifying_key().to_der()
channel_pubkey_hash = self.ledger.public_key_to_address(public_key_bytes)
self.channel_keys[channel_pubkey_hash] = private_key.to_pem().decode()
async def generate_channel_private_key(self):
return await self.deterministic_channel_keys.generate_next_key()
async def get_channel_private_key(self, public_key_bytes):
def add_channel_private_key(self, private_key: PrivateKey):
self.channel_keys[private_key.address] = private_key.to_pem().decode()
async def get_channel_private_key(self, public_key_bytes) -> PrivateKey:
channel_pubkey_hash = self.ledger.public_key_to_address(public_key_bytes)
private_key_pem = self.channel_keys.get(channel_pubkey_hash)
if private_key_pem:
return await asyncio.get_event_loop().run_in_executor(
None, ecdsa.SigningKey.from_pem, private_key_pem, sha256
)
return PrivateKey.from_pem(self.ledger, private_key_pem)
return self.deterministic_channel_keys.get_private_key_from_pubkey_hash(channel_pubkey_hash)
async def maybe_migrate_certificates(self):
def to_der(private_key_pem):
return ecdsa.SigningKey.from_pem(private_key_pem, hashfunc=sha256).get_verifying_key().to_der()
if not self.channel_keys:
return
channel_keys = {}
for private_key_pem in self.channel_keys.values():
if not isinstance(private_key_pem, str):
continue
if "-----BEGIN EC PRIVATE KEY-----" not in private_key_pem:
if not private_key_pem.startswith("-----BEGIN"):
continue
public_key_der = await asyncio.get_event_loop().run_in_executor(None, to_der, private_key_pem)
channel_keys[self.ledger.public_key_to_address(public_key_der)] = private_key_pem
private_key = PrivateKey.from_pem(self.ledger, private_key_pem)
channel_keys[private_key.address] = private_key_pem
if self.channel_keys != channel_keys:
self.channel_keys = channel_keys
self.wallet.save()
+100 -17
View File
@@ -1,10 +1,21 @@
from coincurve import PublicKey, PrivateKey as _PrivateKey
from asn1crypto.keys import PrivateKeyInfo, ECPrivateKey
from coincurve import PublicKey as cPublicKey, PrivateKey as cPrivateKey
from coincurve.utils import (
pem_to_der, lib as libsecp256k1, ffi as libsecp256k1_ffi
)
from coincurve.ecdsa import CDATA_SIG_LENGTH
from lbry.crypto.hash import hmac_sha512, hash160, double_sha256
from lbry.crypto.base58 import Base58
from .util import cachedproperty
class KeyPath:
RECEIVE = 0
CHANGE = 1
CHANNEL = 2
class DerivationError(Exception):
""" Raised when an invalid derivation occurs. """
@@ -71,26 +82,30 @@ class _KeyBase:
return Base58.encode_check(self.extended_key())
class PubKey(_KeyBase):
class PublicKey(_KeyBase):
""" A BIP32 public key. """
def __init__(self, ledger, pubkey, chain_code, n, depth, parent=None):
super().__init__(ledger, chain_code, n, depth, parent)
if isinstance(pubkey, PublicKey):
if isinstance(pubkey, cPublicKey):
self.verifying_key = pubkey
else:
self.verifying_key = self._verifying_key_from_pubkey(pubkey)
@classmethod
def from_compressed(cls, public_key_bytes, ledger=None) -> 'PublicKey':
return cls(ledger, public_key_bytes, bytes((0,)*32), 0, 0)
@classmethod
def _verifying_key_from_pubkey(cls, pubkey):
""" Converts a 33-byte compressed pubkey into an PublicKey object. """
""" Converts a 33-byte compressed pubkey into an coincurve.PublicKey object. """
if not isinstance(pubkey, (bytes, bytearray)):
raise TypeError('pubkey must be raw bytes')
if len(pubkey) != 33:
raise ValueError('pubkey must be 33 bytes')
if pubkey[0] not in (2, 3):
raise ValueError('invalid pubkey prefix byte')
return PublicKey(pubkey)
return cPublicKey(pubkey)
@cachedproperty
def pubkey_bytes(self):
@@ -105,7 +120,7 @@ class PubKey(_KeyBase):
def ec_point(self):
return self.verifying_key.point()
def child(self, n: int):
def child(self, n: int) -> 'PublicKey':
""" Return the derived child extended pubkey at index N. """
if not 0 <= n < (1 << 31):
raise ValueError('invalid BIP32 public key child number')
@@ -113,7 +128,7 @@ class PubKey(_KeyBase):
msg = self.pubkey_bytes + n.to_bytes(4, 'big')
L_b, R_b = self._hmac_sha512(msg) # pylint: disable=invalid-name
derived_key = self.verifying_key.add(L_b)
return PubKey(self.ledger, derived_key, R_b, n, self.depth + 1, self)
return PublicKey(self.ledger, derived_key, R_b, n, self.depth + 1, self)
def identifier(self):
""" Return the key's identifier as 20 bytes. """
@@ -126,6 +141,36 @@ class PubKey(_KeyBase):
self.pubkey_bytes
)
def verify(self, signature, digest) -> bool:
""" Verify that a signature is valid for a 32 byte digest. """
if len(signature) != 64:
raise ValueError('Signature must be 64 bytes long.')
if len(digest) != 32:
raise ValueError('Digest must be 32 bytes long.')
key = self.verifying_key
raw_signature = libsecp256k1_ffi.new('secp256k1_ecdsa_signature *')
parsed = libsecp256k1.secp256k1_ecdsa_signature_parse_compact(
key.context.ctx, raw_signature, signature
)
assert parsed == 1
normalized_signature = libsecp256k1_ffi.new('secp256k1_ecdsa_signature *')
libsecp256k1.secp256k1_ecdsa_signature_normalize(
key.context.ctx, normalized_signature, raw_signature
)
verified = libsecp256k1.secp256k1_ecdsa_verify(
key.context.ctx, normalized_signature, digest, key.public_key
)
return bool(verified)
class PrivateKey(_KeyBase):
"""A BIP32 private key."""
@@ -134,7 +179,7 @@ class PrivateKey(_KeyBase):
def __init__(self, ledger, privkey, chain_code, n, depth, parent=None):
super().__init__(ledger, chain_code, n, depth, parent)
if isinstance(privkey, _PrivateKey):
if isinstance(privkey, cPrivateKey):
self.signing_key = privkey
else:
self.signing_key = self._signing_key_from_privkey(privkey)
@@ -142,7 +187,7 @@ class PrivateKey(_KeyBase):
@classmethod
def _signing_key_from_privkey(cls, private_key):
""" Converts a 32-byte private key into an coincurve.PrivateKey object. """
return _PrivateKey.from_int(PrivateKey._private_key_secret_exponent(private_key))
return cPrivateKey.from_int(PrivateKey._private_key_secret_exponent(private_key))
@classmethod
def _private_key_secret_exponent(cls, private_key):
@@ -154,24 +199,36 @@ class PrivateKey(_KeyBase):
return int.from_bytes(private_key, 'big')
@classmethod
def from_seed(cls, ledger, seed):
def from_seed(cls, ledger, seed) -> 'PrivateKey':
# This hard-coded message string seems to be coin-independent...
hmac = hmac_sha512(b'Bitcoin seed', seed)
privkey, chain_code = hmac[:32], hmac[32:]
return cls(ledger, privkey, chain_code, 0, 0)
@classmethod
def from_pem(cls, ledger, pem) -> 'PrivateKey':
der = pem_to_der(pem.encode())
try:
key_int = ECPrivateKey.load(der).native['private_key']
except ValueError:
key_int = PrivateKeyInfo.load(der).native['private_key']['private_key']
private_key = cPrivateKey.from_int(key_int)
return cls(ledger, private_key, bytes((0,)*32), 0, 0)
@cachedproperty
def private_key_bytes(self):
""" Return the serialized private key (no leading zero byte). """
return self.signing_key.secret
@cachedproperty
def public_key(self):
def public_key(self) -> PublicKey:
""" Return the corresponding extended public key. """
verifying_key = self.signing_key.public_key
parent_pubkey = self.parent.public_key if self.parent else None
return PubKey(self.ledger, verifying_key, self.chain_code, self.n, self.depth,
parent_pubkey)
return PublicKey(
self.ledger, verifying_key, self.chain_code,
self.n, self.depth, parent_pubkey
)
def ec_point(self):
return self.public_key.ec_point()
@@ -184,11 +241,12 @@ class PrivateKey(_KeyBase):
""" Return the private key encoded in Wallet Import Format. """
return self.ledger.private_key_to_wif(self.private_key_bytes)
@property
def address(self):
""" The public key as a P2PKH address. """
return self.public_key.address
def child(self, n):
def child(self, n) -> 'PrivateKey':
""" Return the derived child extended private key at index N."""
if not 0 <= n < (1 << 32):
raise ValueError('invalid BIP32 private key child number')
@@ -207,6 +265,28 @@ class PrivateKey(_KeyBase):
""" Produce a signature for piece of data by double hashing it and signing the hash. """
return self.signing_key.sign(data, hasher=double_sha256)
def sign_compact(self, digest):
""" Produce a compact signature. """
key = self.signing_key
signature = libsecp256k1_ffi.new('secp256k1_ecdsa_signature *')
signed = libsecp256k1.secp256k1_ecdsa_sign(
key.context.ctx, signature, digest, key.secret,
libsecp256k1_ffi.NULL, libsecp256k1_ffi.NULL
)
if not signed:
raise ValueError('The private key was invalid.')
serialized = libsecp256k1_ffi.new('unsigned char[%d]' % CDATA_SIG_LENGTH)
compacted = libsecp256k1.secp256k1_ecdsa_signature_serialize_compact(
key.context.ctx, serialized, signature
)
if compacted != 1:
raise ValueError('The signature could not be compacted.')
return bytes(libsecp256k1_ffi.buffer(serialized, CDATA_SIG_LENGTH))
def identifier(self):
"""Return the key's identifier as 20 bytes."""
return self.public_key.identifier()
@@ -218,9 +298,12 @@ class PrivateKey(_KeyBase):
b'\0' + self.private_key_bytes
)
def to_pem(self):
return self.signing_key.to_pem()
def _from_extended_key(ledger, ekey):
"""Return a PubKey or PrivateKey from an extended key raw bytes."""
"""Return a PublicKey or PrivateKey from an extended key raw bytes."""
if not isinstance(ekey, (bytes, bytearray)):
raise TypeError('extended key must be raw bytes')
if len(ekey) != 78:
@@ -232,7 +315,7 @@ def _from_extended_key(ledger, ekey):
if ekey[:4] == ledger.extended_public_key_prefix:
pubkey = ekey[45:]
key = PubKey(ledger, pubkey, chain_code, n, depth)
key = PublicKey(ledger, pubkey, chain_code, n, depth)
elif ekey[:4] == ledger.extended_private_key_prefix:
if ekey[45] != 0:
raise ValueError('invalid extended private key prefix byte')
@@ -250,6 +333,6 @@ def from_extended_key_string(ledger, ekey_str):
xpub6BsnM1W2Y7qLMiuhi7f7dbAwQZ5Cz5gYJCRzTNainXzQXYjFwtuQXHd
3qfi3t3KJtHxshXezfjft93w4UE7BGMtKwhqEHae3ZA7d823DVrL
return a PubKey or PrivateKey.
return a PublicKey or PrivateKey.
"""
return _from_extended_key(ledger, Base58.decode_check(ekey_str))
+183
View File
@@ -881,4 +881,187 @@ HASHES = {
879000: '0eb0810f4b81d1845b0a88f05449408df2e45715c9210a656f45278c5fdf7956',
880000: 'e7d613027e3b4ca38d09bbef07998b57db237c6d67f1e8ea50024d2e0d9a1a72',
881000: '21af4d355d8756b8bf0369b2d79b5c824148ae069026ba5c14f9dd6b7555e1db',
882000: 'bc26f028e547ec44fc3864925bd1493211773b5cb9a9583ba4c1909b89fe0d33',
883000: '170a624f4be04cd2fd435cfb6ba1f31b9ef5d7b084a25dfa23cd118c2752029e',
884000: '46cccb7a12b4d01d07c211b7b8db41321cd73f30069df27bcdb3bb600c0272b0',
885000: '7c27f79d5a99baf0f81f2b09eb5c1bf905976a0f872e02bd4ca9e82f0ed50cb0',
886000: '256e3e00cecc72dbbfef5cea627ecf1d43b56edd5fd1642a2bc4e97c17056f34',
887000: '658ebac7dfa62bc7a22b1a9ba4e5b425a866f7550a6b40fd07de47119fd1f7e8',
888000: '497a9d02868605b9ff6e7f15948a83a7e07606829107e63c2e091c90c7a7b4d4',
889000: '561daaa7ebc87e586d37a96ecfbc72484d7eb602824f38f484ed333e78208e9e',
890000: 'ab5a8cb625b28343f8fac858eab6576c856dab88bde8cda02b80b3edfd307d71',
891000: '2e81d9fc885ddc09222b298ac9efbb73638a5721802b9256de6505ecf122dbaa',
892000: '73be08881b8832e986c0bb9a06c70fff346edb2afaf69630e47e4a4a90c5fece',
893000: 'd39079dcaa4d8af1c26f0edf7e16df43cd857a31e0aa4c4123226793f1ab497f',
894000: '0a3b677d72c590d4b1ff7a9b4098d6b52d0dc10d64c30c2766d18e6eb02872cd',
895000: 'a3bbba831f48c5b68e494ee63015b487782c64c5c24bb29436283360c28fd1e0',
896000: '20af178a192ca43975ab6c838fe97ca42ba6c682682eddbc6481efd153ecb0a2',
897000: '8d0ee14b9fdb853a09ab2951d26b8f7cb8bc8038b09513bd330ee4b0bdcc4780',
898000: 'c97fbb70f804408b131a98f9fb4c04cdf2df1655d3e8ff2e0d58ed8537349f4e',
899000: 'eba2be80478e8dec2d66ca40b853580c5dad040351c64c177e3d8c25aff6c1b6',
900000: 'c4dc344a993558418b93b3f60aaef0030e2a4116086577fbf1e2f544bdbddae1',
901000: '36d84229afa63045875fc8fea0c55de8eb90694b3a37cceb825c87abf1fea998',
902000: '8ca4890ecfc5e3f9d767e4fcdf318a1e3e3597675bbcfe534d64e76bc4e8fbf4',
903000: '8b9f6a7514033c57668ca94fb3758cc6d1ef37ac982c2ff5a9f0f206fcd8d0a8',
904000: 'e9ae813991f35ca89af2fe1f1b6adf9e93c6b1dd6a74f003ebbe699a30b252ea',
905000: 'd426489d01d4f4c829f2eb68a67721d2c0e1c71e8c33ef9253593447e8603462',
906000: '63000bbed97451e68d64485c02c1c3d90b4156237dac315f4e012ffb538e375b',
907000: '96759653a4e514541effa7ef86d9f22a272ddde7b069149d17e9d9203a1edafb',
908000: 'eec6477d2f3b71bde76dc2380d6e06aa8aa306ca56ba1dd15a31c22ae0db501b',
909000: 'd5c2984cf130335aa29296ba5b17672d00360fe0ec73977326180014908c0b55',
910000: '7b99cb1c94144f606937903e173bd9ef63bfffd3db8110693fa4c2caa0abc21f',
911000: '95eed0d9dd9869ac6f83fa67863e77f24df69bcb90fef70918f30b2400e24ea8',
912000: '34c3c8780c54ecced50f0a6b394309d09ee6ce37cd98794699c63771d1d91144',
913000: '536052ddcd445702160288ef3f669ce56868c085315556c9f5ca081ef0c0b9e1',
914000: '1bcd1fe9632f93a0a1fe7d8a1891a4fc6ef1be40ccf887524a9095ed7aa9fa44',
915000: '139bad9fa12ec72a37b62ad8511300ebfda89330fa5d5a83861f864b6adeae67',
916000: '81d15282214ff83e2a034212eb58abeafcb5664d3734bff13b22b4c093b20fea',
917000: 'f31081031cebe450e4450ef397d91790fc0068e98e6746cd0aab86d17e4448f5',
918000: '4af8eb28616ef0e859b5471650c7f8e910cd692a6b4ff3a7171a709db2f18e4e',
919000: '78a197b5f9733e9e4dc9820e1c79bd335beb19f6b87056e48e8e21fbe27d83d6',
920000: '33d20f86d1367f07d6731e1e2cc9305252b281b1b092403133924cc1052f501d',
921000: '6926f1e31e7fe9b8f7a81efa73d5635f8f28c1db1708e4d57f6e7ead951a4beb',
922000: '811e2335798eb54696a4b11ca3a44b9d79486262119383d542491afa9ae80204',
923000: '8f47ac365bc380885db809f2818ffc7dd2076aaa0f9bf6c180df1b4358dc842e',
924000: '535e79802c10630c17fb8fddec3ba2bf85eedbc0c076f3575f8189fe887ba993',
925000: 'ca43bd24d17d75d55e72e45549384b395c62e1daf0d3f58f296e18168b918fbf',
926000: '9a03be89e0725877d42296e6c995d9c48bb5f4bbd971f5a9add191af2d1c144b',
927000: 'a14e0ef6bd1bc221dbba99031c16ddbbd76394186677c29bdf07b89fa2a6efac',
928000: 'b16931bd7392e9db26be975b072024210fb5fe6ee22fc0809d51980aa8068a98',
929000: '4da56a2e66fcd98a70039d9061ea5eb0fb6d9460b437d2191e47441182419a04',
930000: '87e820e2237a54c4ea100bdd0145598f05add92185cd3d0929aa2d5099f4d5e0',
931000: '515b22c91172157c443a47cf213014aff144181a77e276e291535ab3762bb1ae',
932000: 'e130c6a9eb416f96256d1f90256a148957daa32f56af228d2d9ce6ff27ce2011',
933000: '30c992ec7a9a320fb4db260373121efc7b5e7fc744f4b31defbe6a7608e0749e',
934000: 'ec490fa0de6b1d78a4121a5044f501bbb3bd9e448c18121cea87eb8e3cadba41',
935000: '603e4ae6a6d936c79b3f1c9f9e88305930953b9b390dac442976a6e8395fc520',
936000: '2b756fe2de4328e598ed511b8828e5c2c6b5cdda1b5e7c1c26f8e0424c81afa9',
937000: '1ae0f15f14a0d4819e34a6c18de9428a9e43e17d75383bffa9ffb18358e93b63',
938000: 'cbd7001825ec87b8c6917d6e9e7dc5c8d7767788b6ffd61a61d0c612dbe5de66',
939000: 'd770d0395aa79076044783fb37a1bb173cb95c93ff1ba82c34a72c4d8e425a03',
940000: '3341d0a0349d091d88d233cd6ea6e0ad553d52039b4d47af51b8a8e7573a7916',
941000: '16123b8758e99344ebe6670cd95826881b274c31d4da2a051052955a32bade3a',
942000: 'ac7430961e77f902918fe79a52cbf6b523e3f2804ec83d0b17908e131ea9ea68',
943000: '2ad08a6877e4687dcb7a623adeddc88403e8082efd6de28328b351282dc141e2',
944000: '81382e8c1f47fa7c03fa1726f9b09ed1cd38140fe50683896eaa1b403d7e5fe3',
945000: '152bfbb166da04dab16030af28ae65b3275819eed1d0bbfc11eba65616ebefd6',
946000: '25b3da0962f87a0d3e4aec8b16483efbcab9514893a42fd31f4cb544ddc45a1f',
947000: '2cb738ba342436628ff292797e3d36c4752d71bdc1af87fe758d469d06e36e0e',
948000: 'b3683e18570fcc8b986720514539181ec43fb5dbc20fe314c56ab6bd31ab766a',
949000: '94ced5bfba55ccffc909bf098d537e047d8d4cbb79f5e2a74146073f39804865',
950000: 'b11543cd2aedae27f6ddc3d2b431c897fdcfe59ed3c926b0777bc1e99de4d12a',
951000: '21508881a7f80fcd0b9b27bbcfba634b39c6525f5313968c4605cd55b4fec446',
952000: 'f9b3ed919c9ca20cd2927d899ee7a86c93c2dd919dafb6fdb792f2d9f1895cb0',
953000: 'cf578d8e80eec4102dc1b5321f10b36020b3b32f4b5d4664c90c412ca2ef6b42',
954000: 'ed17c919ae5c4be835966b47f667d6082c75917b95584b2d2aff0e32f5c8aa98',
955000: '948ea467fa01a20122e2146669214fdd3bb025038554609f7299ece5bca63e39',
956000: 'b50ff4c02957ed8764215d25f206f6f1fe6d0eb712a378b937ff952dd479afd2',
957000: '169922a3e51517ba6104a883d29aac03a9d20b4d448bd2773137b0d790e3db6b',
958000: '92258ac2e8b53167dc30436d93f385d432bd549711ab9790ba4e8263c5c54382',
959000: '7ca824697459eb302bcd7fba9d255fb269555abe7cf9d2dd5e54e196d751e682',
960000: '89f9ec925d23698076d84f9e852ab04fc956ac4465827303de0c3bb0b685eb32',
961000: '41cf75cd71bc12b93674c416e8b01b7410eb9e09eb8727ad93ff0b833c9966c9',
962000: '7db1f1dbff3e389713067879bfedf9513ec74bb1e128b13fc2fe23ad55fd0306',
963000: 'a35e71c611b2227adeac824d151d2f09bdbecd5765a4e62c6e74a3e4290abc66',
964000: 'dc1811130e249d2208d6f85838512b4e5482efb0bd2f619164a68a0c60d7f248',
965000: '92f5e25dd1c03102720dd0c3136b1a0769901bf89fcc0262a5e24405f349ca07',
966000: '08243d780d8ba96a940f409b87d9c6b8a95c92804173b9156ada0dad35b628dc',
967000: 'cb769a8935bb6faeb981da74f4079babbbb89476f825cc897f43e79790295260',
968000: 'ff3fc27d2998f4dc4ac1ff378afe14c7d0f43cc328deb9c978ec0e067d1dfaf9',
969000: 'e41a3452f45d5f025627d08c9c41017679e9c4804371dd1cc02f3ed49f85dbb2',
970000: 'f5eaaf7ba6b47245a4a8096a7785c7b25dc6db342ac2ccbba0c321e97ab58284',
971000: '75414062f1d4ed675dadc8f04ba10147a484aaca1ae316dc0b896a92809b3db6',
972000: '5bcf2ee00133774c7d060a1a1863dfccc20d5127ecb542470f607dec2504fe6f',
973000: '07d15b9656ecde2cd86a9d22c3de8b6505d6bab2aa5a94560b0db9119f1f6f6c',
974000: '2059e7924d7a210a88f5a65abc61152506a82edccd27416e796c81b9b8003f13',
975000: '7fcf5d8b2c0e51cfbdaa2502a9da0bdb323646899dad37dacc39af9f9e16fc5c',
976000: '02acb8cf87a0900436eccfca50371948531041d7b8b410a902205f84dd7fb88e',
977000: '2636dfd5a47016c893265473e78ecbf2000769d886f0d01ee7a91e9397210d15',
978000: 'ce92f52a35096b94bea73a7d4e113bc4564a4a589b66f1ab86f61c822cf9ee76',
979000: '21b8102f5b76be0c8e20d537ebc78ebe46bfcea6b6d2dda950ce5b48e85f72d7',
980000: 'f4df0bd63b36105705de62266d654612d9804bad7069d41344de269657e6f084',
981000: 'f006cd2718d98d774a5cd18394db7744c812fa149c8a63e76bab934aee89f571',
982000: 'da5d6609265d9153022d823b0260aa07e7511ceff7a3fd2ca7ce83cb3900a661',
983000: '3a26f3f02aa145fa8c5268fbe10dd9c3546d7dda57489ca5d4b161beb0d5a6e2',
984000: '968e8cd37a1137797d40f39f106cae62d1e252b46c7473b9434ad5f870ee88fb',
985000: '3129c3bf20deace1a9c92646a9d769da7a07f18dcd5b7a7b1e8cf5fd5390f8e1',
986000: '6ce830ca5da322ddbb97fc572ea03218913d070e5910516b33c6113b02b23c21',
987000: '7fb1a8635623847132ab766a99b792953379f782d1115b9649f5f9c5a742ca04',
988000: '5e8e6c6da7f271129c20c4dd891dcb1df4f9d690ee7cf391c6b7fbd028a0da4c',
989000: '12919e34bb9a9ac1d2a01e221eb8c511117fc4e1b3ae15355d95caf4673bdb08',
990000: '016f8b18227a0c09da55594a98638ad5b0fbb4896e2ab6163ac40b6015b2811e',
991000: 'ddf8cd6e2f4ee07530ae7567cef4fa2c2fd4a655cb20e20422e66fd49bde6489',
992000: 'dca77707c0caa3a9605f3dadf593402339c29448869907fb31f6c624e942dcbd',
993000: 'de9acc4c7c482ecac741fd6acbbc3a333afab52f3fe5eea4130c0770299a56dd',
994000: '54420631f8a801a1b8f391088f599ee22cedc06f24bf67f18272feb8fe70c682',
995000: '4b44b26e3e2495716dfd86fc42594cd4b1e4b70bdab4f0905cce4cb9556e008a',
996000: 'd6e41fd301fc5f519c343ceb39c9ff845656a4482e4e182abdcd3963fd5fde1c',
997000: 'd68b6a509d742b182ffb5a98b0e585a2320a5d3fe6977ad3e6cd06835ef2ea55',
998000: '1efcdcbadbec54ce3a93a1857253614536c34f05a0b1924f24bff194dc3392e1',
999000: '10a7713e46f47527f3819b4a9257a03f3e207d18e4917d6bcb43fdea3ba82b9a',
1000000: '1b4ddb1436df05f07807d6337b93ee1aa8b600fd6a910a8fd5313a39e0440eec',
1001000: 'cde0df1abdae26d2c2bdc111be15fb33231c5e167bb8b8f8eec667d71379fee4',
1002000: 'd7ce7a96a3ca73a4dfd6a1780e23f834f339142519ea7f45d256c113e27e4857',
1003000: 'b1a9b1c562ec62b9dd746d336b4211afc37482d0274ff692a44fa17ac9fe9a28',
1004000: '7afd6d0fb0014fbe16a31c84d3f1731736eaeef35e40bb1a1f232fb00345deae',
1005000: '4af61ce4cda5de58277f7a67cadea5d3f6ce56e54785b188e32306e00b0414df',
1006000: '08e1fb7295efd4a48cb999d899a3d481b682ddbce738fecd88a6d32cbe8234f0',
1007000: '14a367a41603dd690541daee8aa4a2882260059e3f85bd8978b7431e8f7db844',
1008000: 'e673230e62aaefad0678611f94ff35ee8a6e18eb96438bdfb4b614f54f54dba7',
1009000: 'e191af8fb71d0d91419abd19443af3d3f23ee4fe359bb8c390429cc838132bde',
1010000: 'ffdba58f184cf60838b75b7899b6633e7cfd34cf36eded572c0133d07387bc49',
1011000: '40801af3a5546cb9d53e05e21b74be09de9a421b762ca1d52d2266f5c2055ce8',
1012000: '552519acebed0e38102f5270dc60b1da7a123600b6b94169ae74462ae454693f',
1013000: '1eee96f48418929927eaa9642777bc806d326cfffaf077bc8695a7ecd438d631',
1014000: 'a471093e1de2a8db586412d7351c8d88e44ea890f46e9b43251af427a0a4a879',
1015000: '57532f5a522295cc139f008bdcb7a1e6d02e6035d5221b2687c7c216f06297a2',
1016000: 'ec46dba07addcb6e62f58456a53c513d876f1c49ae7d76d230adb8debd26027d',
1017000: '33ea8d25f342a7465ed71e4bab2b91007991e0994c61d321e3625301a1390322',
1018000: '4871c03cc95d4ce0a39bd2cebbb001b2ea1cce1b3561bb841d88f43bb9d12ffd',
1019000: 'f5248257576eb2ff4139d6374cc7ce34121cc942598cf9e04d2bd572e09189bb',
1020000: 'e7785286897c85cfb0276957bff216039eeb11bc1ebca89d0bb586022caa5750',
1021000: 'a30220f17d060634c5f6a1ddc5ea34b01c18fb5eb7e0e8267b66bf5a49525627',
1022000: '6083ea49e64ac0d4507c674237cf87d30b90b285ec63d082e626df0223eb7c9c',
1023000: '1dc5596d716bc33ee0f56fc40c1f073155a58a7692935c9e5854ef3b65b76828',
1024000: '065adfee40dc33abff07fb55339571712b959bc1830dc60b6691e36eab1508ae',
1025000: 'bb6903752d31278570e774b80a80782179c78f099e58c3dc4cba7afea7a471c4',
1026000: 'f3050f3c2f3a76f5084856b0f089383517caa3f51530fbc29335308f5f170625',
1027000: '746ed3701510d07958d11a06f22dbb839d9858373dc5a33249dd69e91bab01fd',
1028000: '43f7a96ea6a45b78c29ad4a2f8680ef184438c2bd3686172b0564e0ae6dd7ba1',
1029000: 'cbb9916099c59e14fe61d284374f4feaa3d43afec59e4698ed92143576f24b34',
1030000: '2e805fc2331e32e586ea692bc3d4e6b11e1ec3f1cab6e331b459f9f1ac9a1f1e',
1031000: '04f324f8f6d4f9901cf65f78dc91d6010ea6cf125f5ac0253b57b5f1f79e81e0',
1032000: '60ca62f52fdfd858b0ee0fdb380648bde85ca14e2a73565205ed4ee0bc861c77',
1033000: 'eb60aac23d599d3099cf98ed8fc3213f1bc06bc1c677429b303e9c81f79f1340',
1034000: 'f0328df2daf119ce673ddfa7a39a84576985f701f7a7dec3f56f58c2019ebd4d',
1035000: 'f9d3cbce3854de168d8835c96917c01be6244c8f82641e8d9398dfffec4e7107',
1036000: '7dca97e6e1d6ed70aa7805f74b768009a270e7ebe1dd951e8727d1d2f2d271f2',
1037000: '5329504126b2845b3044f423b521e77ff58d7d242f24bf87c87f4d8d4e03a947',
1038000: '5bad3ad55e3daa415f3182a1f2a099fe1767e8fae34e9bb95d47e242b8971434',
1039000: 'c29729b8ba49ac0043fe4aa6fc971f8ac3eda68ff92970957ada39a2989b2491',
1040000: 'f303aebfc9267600c081d0c021065743f93790df6f5c924a86b773788e0c45be',
1041000: 'a1cbe5059fa2275707785b77970c36d79b12c1ba93121bc9064ab9b64abacf7b',
1042000: '004b0dd4e438abc54ae832d733df32a6ba35b75e6d3e0c9c1dee5a7950507295',
1043000: '31893a3fe7bb4f6dd546c7a8de4a65990e94046aab442d18c68b6bf6acd54518',
1044000: '2c4dd479948acc42946f94050810000b0539864ad24a67a7251bff1c4971b035',
1045000: '1cea782d60df35a88b30ae205ce37e30abc7cad2b22181722be150bd92c53814',
1046000: 'ee808f0efb0f2ef93e8599d8b7f0e2e7c3cdc42353e4ea5165028b961f43d548',
1047000: '75f057e2a8cb1d46e5c943d63cc56936a6bac8b1cb89300593845a20baf39765',
1048000: '2abcd227f5314baed85e3c5b49d3888a60085c1845c955a8bf96aa3dd6394798',
1049000: '5d0ec24b9acd5ab21b42f68e1f3142b7bf83433b98f2fa9794586c8eff45893e',
1050000: '1d364b13a4c17bd67a6d1e5f77c26d02faa014d7cd152b4da70380f168b8e0ff',
1051000: 'b9a20cec21de84433be9b85817dd4803e875d9275dbc02907b29888431859bae',
1052000: '424cb56b00407d73b309b2081dd0bf89213cf024e3aafb3090506aa0ba10f835',
1053000: '6df3041a32fafd6a4e08778546d077cf591e1a2a16e77fe7a610efc2b542a9ff',
1054000: '78f8dee794f3d4366019339d7ba74ad2b543ecd25dc575620f66e1d535411971',
1055000: '43b8e9dae5addd58a7cccf62ba57ab46ffdaa2dcd113cc8ca537e9101b54c096',
1056000: '86b7f3741343f85d93410b78cc3fbf03d49b60a664e908703016aa56a206ae7e',
1057000: 'b033cf6ec622be6a99dff536a2cf73b36d3c3f8c3835ee17e0dd357403e85c41',
1058000: 'a65a6db692a8358e399a5ac3c818902fdb60595262ae05531084848febead249',
1059000: 'f6d781d2e2fdb4b7b074d1d8123875d899cdbd6be375cb4288e86f1d14a929f6',
1060000: 'cd9019bb1de4926cca16a7bef1a46786f10a3260d467cda0775f73361795abc9',
1061000: 'ed4f5dc6f475f95b40595632fafd9e7e5eef388b6cc15772204c0b0e9ee4e542',
1062000: 'c44d02a890aa66979b10d1cfa597c877f498841b4e12dd9a7bdf8d4a5fccab80',
1063000: '1c093734f5f241b36c1b9971e2759983f88f4033405a2588b4ebfd6998ac7465',
1064000: '9e354a83b71bbb9704053bfeea038a9c3d5daad080c6406c698b047c634706a6',
}
+1
View File
@@ -2,6 +2,7 @@ NULL_HASH32 = b'\x00'*32
CENT = 1000000
COIN = 100*CENT
DUST = 1000
TIMEOUT = 30.0
+18 -3
View File
@@ -9,10 +9,11 @@ from dataclasses import dataclass
from contextvars import ContextVar
from typing import Tuple, List, Union, Callable, Any, Awaitable, Iterable, Dict, Optional
from datetime import date
from prometheus_client import Gauge, Counter, Histogram
from lbry.utils import LockWithMetrics
from .bip32 import PubKey
from .bip32 import PublicKey
from .transaction import Transaction, Output, OutputScript, TXRefImmutable, Input
from .constants import TXO_TYPES, CLAIM_TYPES
from .util import date_to_julian_day
@@ -975,7 +976,9 @@ class Database(SQLiteMixin):
sql.append("LEFT JOIN txi ON (txi.position=0 AND txi.txid=txo.txid)")
return await self.db.execute_fetchall(*query(' '.join(sql), **constraints), read_only=read_only)
async def get_txos(self, wallet=None, no_tx=False, no_channel_info=False, read_only=False, **constraints):
async def get_txos(
self, wallet=None, no_tx=False, no_channel_info=False, read_only=False, **constraints
) -> List[Output]:
include_is_spent = constraints.get('include_is_spent', False)
include_is_my_input = constraints.get('include_is_my_input', False)
include_is_my_output = constraints.pop('include_is_my_output', False)
@@ -1201,7 +1204,7 @@ class Database(SQLiteMixin):
addresses = await self.select_addresses(', '.join(cols), read_only=read_only, **constraints)
if 'pubkey' in cols:
for address in addresses:
address['pubkey'] = PubKey(
address['pubkey'] = PublicKey(
self.ledger, address.pop('pubkey'), address.pop('chain_code'),
address.pop('n'), address.pop('depth')
)
@@ -1241,6 +1244,18 @@ class Database(SQLiteMixin):
async def set_address_history(self, address, history):
await self._set_address_history(address, history)
async def is_channel_key_used(self, account, key: PublicKey):
channels = await self.get_txos(
accounts=[account], txo_type=TXO_TYPES['channel'],
no_tx=True, no_channel_info=True
)
other_key_bytes = key.pubkey_bytes
for channel in channels:
claim = channel.can_decode_claim
if claim and claim.channel.public_key_bytes == other_key_bytes:
return True
return False
@staticmethod
def constrain_purchases(constraints):
accounts = constraints.pop('accounts', None)
+26 -14
View File
@@ -16,18 +16,18 @@ from lbry.crypto.hash import hash160, double_sha256, sha256
from lbry.crypto.base58 import Base58
from lbry.utils import LRUCacheWithMetrics
from .tasks import TaskGroup
from .database import Database
from .stream import StreamController
from .dewies import dewies_to_lbc
from .account import Account, AddressManager, SingleKey
from .network import Network
from .transaction import Transaction, Output
from .header import Headers, UnvalidatedHeaders
from .checkpoints import HASHES
from .constants import TXO_TYPES, CLAIM_TYPES, COIN, NULL_HASH32
from .bip32 import PubKey, PrivateKey
from .coinselection import CoinSelector
from lbry.wallet.tasks import TaskGroup
from lbry.wallet.database import Database
from lbry.wallet.stream import StreamController
from lbry.wallet.dewies import dewies_to_lbc
from lbry.wallet.account import Account, AddressManager, SingleKey
from lbry.wallet.network import Network
from lbry.wallet.transaction import Transaction, Output
from lbry.wallet.header import Headers, UnvalidatedHeaders
from lbry.wallet.checkpoints import HASHES
from lbry.wallet.constants import TXO_TYPES, CLAIM_TYPES, COIN, NULL_HASH32
from lbry.wallet.bip32 import PublicKey, PrivateKey
from lbry.wallet.coinselection import CoinSelector
log = logging.getLogger(__name__)
@@ -226,7 +226,7 @@ class Ledger(metaclass=LedgerRegistry):
return account.get_private_key(address_info['chain'], address_info['pubkey'].n)
return None
async def get_public_key_for_address(self, wallet, address) -> Optional[PubKey]:
async def get_public_key_for_address(self, wallet, address) -> Optional[PublicKey]:
match = await self._get_account_and_address_info_for_address(wallet, address)
if match:
_, address_info = match
@@ -365,6 +365,10 @@ class Ledger(metaclass=LedgerRegistry):
await self.db.close()
await self.headers.close()
async def tasks_are_done(self):
await self._update_tasks.done.wait()
await self._other_tasks.done.wait()
@property
def local_height_including_downloaded_height(self):
return max(self.headers.height, self._download_height)
@@ -470,6 +474,7 @@ class Ledger(metaclass=LedgerRegistry):
for address_manager in account.address_managers.values():
await self.subscribe_addresses(address_manager, await address_manager.get_addresses())
await account.ensure_address_gap()
await account.deterministic_channel_keys.ensure_cache_primed()
async def unsubscribe_account(self, account: Account):
for address in await account.get_addresses():
@@ -550,6 +555,7 @@ class Ledger(metaclass=LedgerRegistry):
)
remote_history_txids = {txid for txid, _ in remote_history}
async for tx in self.request_synced_transactions(to_request, remote_history_txids, address):
self.maybe_has_channel_key(tx)
pending_synced_history[tx_indexes[tx.id]] = f"{tx.id}:{tx.height}:"
if len(pending_synced_history) % 100 == 0:
log.info("Syncing address %s: %d/%d", address, len(pending_synced_history), len(to_request))
@@ -617,6 +623,12 @@ class Ledger(metaclass=LedgerRegistry):
tx.is_verified = merkle_root == header['merkle_root']
return tx
def maybe_has_channel_key(self, tx):
for txo in tx._outputs:
if txo.can_decode_claim and txo.claim.is_channel:
for account in self.accounts:
account.deterministic_channel_keys.maybe_generate_deterministic_key_for_channel(txo)
async def request_transactions(self, to_request: Tuple[Tuple[str, int], ...], cached=False):
batches = [[]]
remote_heights = {}
@@ -731,7 +743,7 @@ class Ledger(metaclass=LedgerRegistry):
while timeout and (int(time.perf_counter()) - start) <= timeout:
if await self._wait_round(tx, height, addresses):
return
raise asyncio.TimeoutError('Timed out waiting for transaction.')
raise asyncio.TimeoutError(f'Timed out waiting for transaction. {tx.id}')
async def _wait_round(self, tx: Transaction, height: int, addresses: Iterable[str]):
records = await self.db.get_addresses(address__in=addresses)
+9 -7
View File
@@ -12,13 +12,13 @@ from typing import List, Type, MutableSequence, MutableMapping, Optional
from lbry.error import KeyFeeAboveMaxAllowedError, WalletNotLoadedError
from lbry.conf import Config, NOT_SET
from .dewies import dewies_to_lbc
from .account import Account
from .ledger import Ledger, LedgerRegistry
from .transaction import Transaction, Output
from .database import Database
from .wallet import Wallet, WalletStorage, ENCRYPT_ON_DISK
from .rpc.jsonrpc import CodeMessageError
from lbry.wallet.dewies import dewies_to_lbc
from lbry.wallet.account import Account
from lbry.wallet.ledger import Ledger, LedgerRegistry
from lbry.wallet.transaction import Transaction, Output
from lbry.wallet.database import Database
from lbry.wallet.wallet import Wallet, WalletStorage, ENCRYPT_ON_DISK
from lbry.wallet.rpc.jsonrpc import CodeMessageError
if typing.TYPE_CHECKING:
from lbry.extras.daemon.exchange_rate_manager import ExchangeRateManager
@@ -194,6 +194,8 @@ class WalletManager:
'data_path': config.wallet_dir,
'tx_cache_size': config.transaction_cache_size
}
if 'LBRY_FEE_PER_NAME_CHAR' in os.environ:
ledger_config['fee_per_name_char'] = int(os.environ.get('LBRY_FEE_PER_NAME_CHAR'))
wallets_directory = os.path.join(config.wallet_dir, 'wallets')
if not os.path.exists(wallets_directory):
+4 -4
View File
@@ -122,7 +122,7 @@ class ClientSession(BaseClientSession):
await asyncio.sleep(max(0, max_idle - (now - self.last_send)))
except Exception as err:
if isinstance(err, asyncio.CancelledError):
log.warning("closing connection to %s:%i", *self.server)
log.info("closing connection to %s:%i", *self.server)
else:
log.exception("lost connection to spv")
finally:
@@ -140,7 +140,7 @@ class ClientSession(BaseClientSession):
controller.add(request.args)
def connection_lost(self, exc):
log.warning("Connection lost: %s:%d", *self.server)
log.debug("Connection lost: %s:%d", *self.server)
super().connection_lost(exc)
self.response_time = None
self.connection_latency = None
@@ -303,7 +303,7 @@ class Network:
concurrency=self.config.get('concurrent_hub_requests', 30))
try:
await client.create_connection()
log.warning("Connected to spv server %s:%i", host, port)
log.info("Connected to spv server %s:%i", host, port)
await client.ensure_server_version()
return client
except (asyncio.TimeoutError, ConnectionError, OSError, IncompatibleWalletServerError, RPCError):
@@ -357,7 +357,7 @@ class Network:
self._keepalive_task = None
self.client = None
self.server_features = None
log.warning("connection lost to %s", server_str)
log.info("connection lost to %s", server_str)
log.info("network loop finished")
async def stop(self):
+3 -3
View File
@@ -1,5 +1,5 @@
__hub_url__ = (
"https://github.com/lbryio/hub/releases/download/leveldb-hub/hub"
"https://github.com/lbryio/hub/releases/download/v0.2022.01.21.1/hub"
)
from .node import Conductor
from .service import ConductorService
from lbry.wallet.orchstr8.node import Conductor
from lbry.wallet.orchstr8.service import ConductorService
+14 -7
View File
@@ -5,7 +5,9 @@ import aiohttp
from lbry import wallet
from lbry.wallet.orchstr8.node import (
Conductor, get_blockchain_node_from_ledger
Conductor,
get_lbcd_node_from_ledger,
get_lbcwallet_node_from_ledger
)
from lbry.wallet.orchstr8.service import ConductorService
@@ -16,10 +18,11 @@ def get_argument_parser():
)
subparsers = parser.add_subparsers(dest='command', help='sub-command help')
subparsers.add_parser("download", help="Download blockchain node binary.")
subparsers.add_parser("download", help="Download lbcd and lbcwallet node binaries.")
start = subparsers.add_parser("start", help="Start orchstr8 service.")
start.add_argument("--blockchain", help="Hostname to start blockchain node.")
start.add_argument("--lbcd", help="Hostname to start lbcd node.")
start.add_argument("--lbcwallet", help="Hostname to start lbcwallet node.")
start.add_argument("--spv", help="Hostname to start SPV server.")
start.add_argument("--wallet", help="Hostname to start wallet daemon.")
@@ -47,7 +50,8 @@ def main():
if command == 'download':
logging.getLogger('blockchain').setLevel(logging.INFO)
get_blockchain_node_from_ledger(wallet).ensure()
get_lbcd_node_from_ledger(wallet).ensure()
get_lbcwallet_node_from_ledger(wallet).ensure()
elif command == 'generate':
loop.run_until_complete(run_remote_command(
@@ -57,9 +61,12 @@ def main():
elif command == 'start':
conductor = Conductor()
if getattr(args, 'blockchain', False):
conductor.blockchain_node.hostname = args.blockchain
loop.run_until_complete(conductor.start_blockchain())
if getattr(args, 'lbcd', False):
conductor.lbcd_node.hostname = args.lbcd
loop.run_until_complete(conductor.start_lbcd())
if getattr(args, 'lbcwallet', False):
conductor.lbcwallet_node.hostname = args.lbcwallet
loop.run_until_complete(conductor.start_lbcwallet())
if getattr(args, 'spv', False):
conductor.spv_node.hostname = args.spv
loop.run_until_complete(conductor.start_spv())
+400 -163
View File
@@ -1,4 +1,5 @@
import os
import signal
import json
import shutil
import asyncio
@@ -8,6 +9,7 @@ import logging
import tempfile
import subprocess
import importlib
import platform
from distutils.util import strtobool
from binascii import hexlify
@@ -16,11 +18,13 @@ import urllib.request
from uuid import uuid4
import lbry
from lbry.wallet.server.server import Server
from lbry.wallet.server.env import Env
from lbry.wallet import Wallet, Ledger, RegTestLedger, WalletManager, Account, BlockHeightEvent
from lbry.conf import KnownHubsList, Config
from lbry.wallet.orchstr8 import __hub_url__
from lbry.wallet.server.block_processor import BlockProcessor
from lbry.wallet.server.chain_reader import BlockchainReaderServer
from lbry.wallet.server.db.elasticsearch.sync import ElasticWriter
log = logging.getLogger(__name__)
@@ -31,11 +35,18 @@ def get_spvserver_from_ledger(ledger_module):
return getattr(spvserver_module, regtest_class_name)
def get_blockchain_node_from_ledger(ledger_module):
return BlockchainNode(
ledger_module.__node_url__,
os.path.join(ledger_module.__node_bin__, ledger_module.__node_daemon__),
os.path.join(ledger_module.__node_bin__, ledger_module.__node_cli__)
def get_lbcd_node_from_ledger(ledger_module):
return LBCDNode(
ledger_module.__lbcd_url__,
ledger_module.__lbcd__,
ledger_module.__lbcctl__
)
def get_lbcwallet_node_from_ledger(ledger_module):
return LBCWalletNode(
ledger_module.__lbcwallet_url__,
ledger_module.__lbcwallet__,
ledger_module.__lbcctl__
)
@@ -45,51 +56,51 @@ class Conductor:
self.manager_module = WalletManager
self.spv_module = get_spvserver_from_ledger(lbry.wallet)
self.blockchain_node = get_blockchain_node_from_ledger(lbry.wallet)
self.lbcd_node = get_lbcd_node_from_ledger(lbry.wallet)
self.lbcwallet_node = get_lbcwallet_node_from_ledger(lbry.wallet)
self.spv_node = SPVNode(self.spv_module)
self.wallet_node = WalletNode(
self.manager_module, RegTestLedger, default_seed=seed
)
self.hub_node = HubNode(__hub_url__, "hub", self.spv_node)
self.blockchain_started = False
self.lbcd_started = False
self.lbcwallet_started = False
self.spv_started = False
self.wallet_started = False
self.hub_started = False
self.log = log.getChild('conductor')
async def start_blockchain(self):
if not self.blockchain_started:
asyncio.create_task(self.blockchain_node.start())
await self.blockchain_node.running.wait()
await self.blockchain_node.generate(200)
self.blockchain_started = True
async def start_lbcd(self):
if not self.lbcd_started:
await self.lbcd_node.start()
self.lbcd_started = True
async def stop_blockchain(self):
if self.blockchain_started:
await self.blockchain_node.stop(cleanup=True)
self.blockchain_started = False
async def stop_lbcd(self, cleanup=True):
if self.lbcd_started:
await self.lbcd_node.stop(cleanup)
self.lbcd_started = False
async def start_hub(self):
if not self.hub_started:
asyncio.create_task(self.hub_node.start())
await self.blockchain_node.running.wait()
await self.hub_node.start()
await self.lbcwallet_node.running.wait()
self.hub_started = True
async def stop_hub(self):
async def stop_hub(self, cleanup=True):
if self.hub_started:
await self.hub_node.stop(cleanup=True)
await self.hub_node.stop(cleanup)
self.hub_started = False
async def start_spv(self):
if not self.spv_started:
await self.spv_node.start(self.blockchain_node)
await self.spv_node.start(self.lbcwallet_node)
self.spv_started = True
async def stop_spv(self):
async def stop_spv(self, cleanup=True):
if self.spv_started:
await self.spv_node.stop(cleanup=True)
await self.spv_node.stop(cleanup)
self.spv_started = False
async def start_wallet(self):
@@ -97,21 +108,41 @@ class Conductor:
await self.wallet_node.start(self.spv_node)
self.wallet_started = True
async def stop_wallet(self):
async def stop_wallet(self, cleanup=True):
if self.wallet_started:
await self.wallet_node.stop(cleanup=True)
await self.wallet_node.stop(cleanup)
self.wallet_started = False
async def start_lbcwallet(self, clean=True):
if not self.lbcwallet_started:
await self.lbcwallet_node.start()
if clean:
mining_addr = await self.lbcwallet_node.get_new_address()
self.lbcwallet_node.mining_addr = mining_addr
await self.lbcwallet_node.generate(200)
# unlock the wallet for the next 1 hour
await self.lbcwallet_node.wallet_passphrase("password", 3600)
self.lbcwallet_started = True
async def stop_lbcwallet(self, cleanup=True):
if self.lbcwallet_started:
await self.lbcwallet_node.stop(cleanup)
self.lbcwallet_started = False
async def start(self):
await self.start_blockchain()
await self.start_lbcd()
await self.start_lbcwallet()
await self.start_spv()
await self.start_hub()
await self.start_wallet()
async def stop(self):
all_the_stops = [
self.stop_wallet,
self.stop_hub,
self.stop_spv,
self.stop_blockchain
self.stop_lbcwallet,
self.stop_lbcd
]
for stop in all_the_stops:
try:
@@ -119,6 +150,12 @@ class Conductor:
except Exception as e:
log.exception('Exception raised while stopping services:', exc_info=e)
async def clear_mempool(self):
await self.stop_lbcwallet(cleanup=False)
await self.stop_lbcd(cleanup=False)
await self.start_lbcd()
await self.start_lbcwallet(clean=False)
class WalletNode:
@@ -139,10 +176,11 @@ class WalletNode:
async def start(self, spv_node: 'SPVNode', seed=None, connect=True, config=None):
wallets_dir = os.path.join(self.data_path, 'wallets')
os.mkdir(wallets_dir)
wallet_file_name = os.path.join(wallets_dir, 'my_wallet.json')
with open(wallet_file_name, 'w') as wallet_file:
wallet_file.write('{"version": 1, "accounts": []}\n')
if not os.path.isdir(wallets_dir):
os.mkdir(wallets_dir)
with open(wallet_file_name, 'w') as wallet_file:
wallet_file.write('{"version": 1, "accounts": []}\n')
self.manager = self.manager_class.from_config({
'ledgers': {
self.ledger_class.get_id(): {
@@ -186,53 +224,73 @@ class WalletNode:
class SPVNode:
def __init__(self, coin_class, node_number=1):
self.node_number = node_number
self.coin_class = coin_class
self.controller = None
self.data_path = None
self.server = None
self.server: Optional[BlockchainReaderServer] = None
self.writer: Optional[BlockProcessor] = None
self.es_writer: Optional[ElasticWriter] = None
self.hostname = 'localhost'
self.port = 50001 + node_number # avoid conflict with default daemon
self.udp_port = self.port
self.elastic_notifier_port = 19080 + node_number
self.rpc_port = 8000 + node_number
self.session_timeout = 600
self.rpc_port = '0' # disabled by default
self.stopped = False
self.stopped = True
self.index_name = uuid4().hex
async def start(self, blockchain_node: 'BlockchainNode', extraconf=None):
self.data_path = tempfile.mkdtemp()
conf = {
'DESCRIPTION': '',
'PAYMENT_ADDRESS': '',
'DAILY_FEE': '0',
'DB_DIRECTORY': self.data_path,
'DAEMON_URL': blockchain_node.rpc_url,
'REORG_LIMIT': '100',
'HOST': self.hostname,
'TCP_PORT': str(self.port),
'UDP_PORT': str(self.udp_port),
'SESSION_TIMEOUT': str(self.session_timeout),
'MAX_QUERY_WORKERS': '0',
'INDIVIDUAL_TAG_INDEXES': '',
'RPC_PORT': self.rpc_port,
'ES_INDEX_PREFIX': self.index_name,
'ES_MODE': 'writer',
}
if extraconf:
conf.update(extraconf)
# TODO: don't use os.environ
os.environ.update(conf)
self.server = Server(Env(self.coin_class))
self.server.bp.mempool.refresh_secs = self.server.bp.prefetcher.polling_delay = 0.5
await self.server.start()
async def start(self, lbcwallet_node: 'LBCWalletNode', extraconf=None):
if not self.stopped:
log.warning("spv node is already running")
return
self.stopped = False
try:
self.data_path = tempfile.mkdtemp()
conf = {
'description': '',
'payment_address': '',
'daily_fee': '0',
'db_dir': self.data_path,
'daemon_url': lbcwallet_node.rpc_url,
'reorg_limit': 100,
'host': self.hostname,
'tcp_port': self.port,
'udp_port': self.udp_port,
'rpc_port': self.rpc_port,
'elastic_notifier_port': self.elastic_notifier_port,
'session_timeout': self.session_timeout,
'max_query_workers': 0,
'es_index_prefix': self.index_name,
}
if extraconf:
conf.update(extraconf)
env = Env(self.coin_class, **conf)
self.writer = BlockProcessor(env)
self.server = BlockchainReaderServer(env)
self.es_writer = ElasticWriter(env)
await self.writer.open()
await self.writer.start()
await self.es_writer.start()
await self.server.start()
except Exception as e:
self.stopped = True
if not isinstance(e, asyncio.CancelledError):
log.exception("failed to start spv node")
raise e
async def stop(self, cleanup=True):
if self.stopped:
log.warning("spv node is already stopped")
return
try:
await self.server.db.search_index.delete_index()
await self.server.db.search_index.stop()
await self.server.stop()
await self.es_writer.stop(delete_index=True)
await self.writer.stop()
self.stopped = True
except Exception as e:
log.exception("failed to stop spv node")
raise e
finally:
cleanup and self.cleanup()
@@ -240,18 +298,19 @@ class SPVNode:
shutil.rmtree(self.data_path, ignore_errors=True)
class BlockchainProcess(asyncio.SubprocessProtocol):
class LBCDProcess(asyncio.SubprocessProtocol):
IGNORE_OUTPUT = [
b'keypool keep',
b'keypool reserve',
b'keypool return',
b'Block submitted',
]
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('blockchain')
self.log = log.getChild('lbcd')
def pipe_data_received(self, fd, data):
if self.log and not any(ignore in data for ignore in self.IGNORE_OUTPUT):
@@ -262,7 +321,7 @@ class BlockchainProcess(asyncio.SubprocessProtocol):
if b'Error:' in data:
self.ready.set()
raise SystemError(data.decode())
if b'Done loading' in data:
if b'RPCS: RPC server listening on' in data:
self.ready.set()
def process_exited(self):
@@ -270,39 +329,57 @@ class BlockchainProcess(asyncio.SubprocessProtocol):
self.ready.set()
class BlockchainNode:
class WalletProcess(asyncio.SubprocessProtocol):
P2SH_SEGWIT_ADDRESS = "p2sh-segwit"
BECH32_ADDRESS = "bech32"
IGNORE_OUTPUT = [
]
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('lbcwallet')
self.transport: Optional[asyncio.transports.SubprocessTransport] = None
def pipe_data_received(self, fd, data):
if self.log and not any(ignore in data for ignore in self.IGNORE_OUTPUT):
if b'Error:' in data:
self.log.error(data.decode())
else:
self.log.info(data.decode())
if b'Error:' in data:
self.ready.set()
raise SystemError(data.decode())
if b'WLLT: Finished rescan' in data:
self.ready.set()
def process_exited(self):
self.stopped.set()
self.ready.set()
class LBCDNode:
def __init__(self, url, daemon, cli):
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
self.daemon_bin = os.path.join(self.bin_dir, daemon)
self.cli_bin = os.path.join(self.bin_dir, cli)
self.log = log.getChild('blockchain')
self.data_path = None
self.log = log.getChild('lbcd')
self.data_path = tempfile.mkdtemp()
self.protocol = None
self.transport = None
self.block_expected = 0
self.hostname = 'localhost'
self.peerport = 9246 + 2 # avoid conflict with default peer port
self.rpcport = 9245 + 2 # avoid conflict with default rpc port
self.peerport = 29246
self.rpcport = 29245
self.rpcuser = 'rpcuser'
self.rpcpassword = 'rpcpassword'
self.stopped = False
self.restart_ready = asyncio.Event()
self.restart_ready.set()
self.stopped = True
self.running = asyncio.Event()
@property
def rpc_url(self):
return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.rpcport}/'
def is_expected_block(self, e: BlockHeightEvent):
return self.block_expected == e.height
@property
def exists(self):
return (
@@ -311,6 +388,12 @@ class BlockchainNode:
)
def download(self):
uname = platform.uname()
target_os = str.lower(uname.system)
target_arch = str.replace(uname.machine, 'x86_64', 'amd64')
target_platform = target_os + '_' + target_arch
self.latest_release_url = str.replace(self.latest_release_url, 'TARGET_PLATFORM', target_platform)
downloaded_file = os.path.join(
self.bin_dir,
self.latest_release_url[self.latest_release_url.rfind('/')+1:]
@@ -344,72 +427,206 @@ class BlockchainNode:
return self.exists or self.download()
async def start(self):
assert self.ensure()
self.data_path = tempfile.mkdtemp()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.daemon_bin,
f'-datadir={self.data_path}', '-printtoconsole', '-regtest', '-server', '-txindex',
f'-rpcuser={self.rpcuser}', f'-rpcpassword={self.rpcpassword}', f'-rpcport={self.rpcport}',
f'-port={self.peerport}'
]
self.log.info(' '.join(command))
while not self.stopped:
if self.running.is_set():
await asyncio.sleep(1)
continue
await self.restart_ready.wait()
try:
self.transport, self.protocol = await loop.subprocess_exec(
BlockchainProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start lbrycrdd', exc_info=e)
if not self.stopped:
return
self.stopped = False
try:
assert self.ensure()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.daemon_bin,
'--notls',
f'--datadir={self.data_path}',
'--regtest', f'--listen=127.0.0.1:{self.peerport}', f'--rpclisten=127.0.0.1:{self.rpcport}',
'--txindex', f'--rpcuser={self.rpcuser}', f'--rpcpass={self.rpcpassword}'
]
self.log.info(' '.join(command))
self.transport, self.protocol = await loop.subprocess_exec(
LBCDProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
except asyncio.CancelledError:
self.running.clear()
self.stopped = True
raise
except Exception as e:
self.running.clear()
self.stopped = True
log.exception('failed to start lbcd', exc_info=e)
raise
async def stop(self, cleanup=True):
if self.stopped:
return
try:
if self.transport:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
except Exception as e:
log.exception('failed to stop lbcd', exc_info=e)
raise
finally:
self.log.info("Done shutting down " + self.daemon_bin)
self.stopped = True
if cleanup:
self.cleanup()
self.running.clear()
def cleanup(self):
assert self.stopped
shutil.rmtree(self.data_path, ignore_errors=True)
class LBCWalletNode:
P2SH_SEGWIT_ADDRESS = "p2sh-segwit"
BECH32_ADDRESS = "bech32"
def __init__(self, url, lbcwallet, cli):
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
self.lbcwallet_bin = os.path.join(self.bin_dir, lbcwallet)
self.cli_bin = os.path.join(self.bin_dir, cli)
self.log = log.getChild('lbcwallet')
self.protocol = None
self.transport = None
self.hostname = 'localhost'
self.lbcd_rpcport = 29245
self.lbcwallet_rpcport = 29244
self.rpcuser = 'rpcuser'
self.rpcpassword = 'rpcpassword'
self.data_path = tempfile.mkdtemp()
self.stopped = True
self.running = asyncio.Event()
self.block_expected = 0
self.mining_addr = ''
@property
def rpc_url(self):
# FIXME: somehow the hub/sdk doesn't learn the blocks through the Walet RPC port, why?
# return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.lbcwallet_rpcport}/'
return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.lbcd_rpcport}/'
def is_expected_block(self, e: BlockHeightEvent):
return self.block_expected == e.height
@property
def exists(self):
return (
os.path.exists(self.lbcwallet_bin)
)
def download(self):
uname = platform.uname()
target_os = str.lower(uname.system)
target_arch = str.replace(uname.machine, 'x86_64', 'amd64')
target_platform = target_os + '_' + target_arch
self.latest_release_url = str.replace(self.latest_release_url, 'TARGET_PLATFORM', target_platform)
downloaded_file = os.path.join(
self.bin_dir,
self.latest_release_url[self.latest_release_url.rfind('/')+1:]
)
if not os.path.exists(self.bin_dir):
os.mkdir(self.bin_dir)
if not os.path.exists(downloaded_file):
self.log.info('Downloading: %s', self.latest_release_url)
with urllib.request.urlopen(self.latest_release_url) as response:
with open(downloaded_file, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
self.log.info('Extracting: %s', downloaded_file)
if downloaded_file.endswith('.zip'):
with zipfile.ZipFile(downloaded_file) as dotzip:
dotzip.extractall(self.bin_dir)
# zipfile bug https://bugs.python.org/issue15795
os.chmod(self.lbcwallet_bin, 0o755)
elif downloaded_file.endswith('.tar.gz'):
with tarfile.open(downloaded_file) as tar:
tar.extractall(self.bin_dir)
return self.exists
def ensure(self):
return self.exists or self.download()
async def start(self):
assert self.ensure()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.lbcwallet_bin,
'--noservertls', '--noclienttls',
'--regtest',
f'--rpcconnect=127.0.0.1:{self.lbcd_rpcport}', f'--rpclisten=127.0.0.1:{self.lbcwallet_rpcport}',
'--createtemp', f'--appdata={self.data_path}',
f'--username={self.rpcuser}', f'--password={self.rpcpassword}'
]
self.log.info(' '.join(command))
try:
self.transport, self.protocol = await loop.subprocess_exec(
WalletProcess, *command
)
self.protocol.transport = self.transport
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
self.stopped = False
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start lbcwallet', exc_info=e)
def cleanup(self):
assert self.stopped
shutil.rmtree(self.data_path, ignore_errors=True)
async def stop(self, cleanup=True):
if self.stopped:
return
try:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
except Exception as e:
log.exception('failed to stop lbcwallet', exc_info=e)
raise
finally:
self.log.info("Done shutting down " + self.lbcwallet_bin)
self.stopped = True
if cleanup:
self.cleanup()
async def clear_mempool(self):
self.restart_ready.clear()
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
self.running.clear()
os.remove(os.path.join(self.data_path, 'regtest', 'mempool.dat'))
self.restart_ready.set()
await self.running.wait()
def cleanup(self):
shutil.rmtree(self.data_path, ignore_errors=True)
self.running.clear()
async def _cli_cmnd(self, *args):
cmnd_args = [
self.cli_bin, f'-datadir={self.data_path}', '-regtest',
f'-rpcuser={self.rpcuser}', f'-rpcpassword={self.rpcpassword}', f'-rpcport={self.rpcport}'
self.cli_bin,
f'--rpcuser={self.rpcuser}', f'--rpcpass={self.rpcpassword}', '--notls', '--regtest', '--wallet'
] + list(args)
self.log.info(' '.join(cmnd_args))
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
process = await asyncio.create_subprocess_exec(
*cmnd_args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
*cmnd_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
out, _ = await process.communicate()
out, err = await process.communicate()
result = out.decode().strip()
err = err.decode().strip()
if len(result) <= 0 and err.startswith('-'):
raise Exception(err)
if err and 'creating a default config file' not in err:
log.warning(err)
self.log.info(result)
if result.startswith('error code'):
raise Exception(result)
@@ -417,7 +634,14 @@ class BlockchainNode:
def generate(self, blocks):
self.block_expected += blocks
return self._cli_cmnd('generate', str(blocks))
return self._cli_cmnd('generatetoaddress', str(blocks), self.mining_addr)
def generate_to_address(self, blocks, addr):
self.block_expected += blocks
return self._cli_cmnd('generatetoaddress', str(blocks), addr)
def wallet_passphrase(self, passphrase, timeout):
return self._cli_cmnd('walletpassphrase', passphrase, str(timeout))
def invalidate_block(self, blockhash):
return self._cli_cmnd('invalidateblock', blockhash)
@@ -434,7 +658,7 @@ class BlockchainNode:
def get_raw_change_address(self):
return self._cli_cmnd('getrawchangeaddress')
def get_new_address(self, address_type):
def get_new_address(self, address_type='legacy'):
return self._cli_cmnd('getnewaddress', "", address_type)
async def get_balance(self):
@@ -450,7 +674,10 @@ class BlockchainNode:
return self._cli_cmnd('createrawtransaction', json.dumps(inputs), json.dumps(outputs))
async def sign_raw_transaction_with_wallet(self, tx):
return json.loads(await self._cli_cmnd('signrawtransactionwithwallet', tx))['hex'].encode()
# the "withwallet" portion should only come into play if we are doing segwit.
# and "withwallet" doesn't exist on lbcd yet.
result = await self._cli_cmnd('signrawtransaction', tx)
return json.loads(result)['hex'].encode()
def decode_raw_transaction(self, tx):
return self._cli_cmnd('decoderawtransaction', hexlify(tx.raw).decode())
@@ -460,12 +687,15 @@ class BlockchainNode:
class HubProcess(asyncio.SubprocessProtocol):
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
def __init__(self, ready, stopped):
self.ready = ready
self.stopped = stopped
self.log = log.getChild('hub')
self.transport = None
def pipe_data_received(self, fd, data):
self.stopped.clear()
self.ready.set()
if self.log:
self.log.info(data.decode())
if b'error' in data.lower():
@@ -473,18 +703,32 @@ class HubProcess(asyncio.SubprocessProtocol):
raise SystemError(data.decode())
if b'listening on' in data:
self.ready.set()
str_lines = str(data.decode()).split("\n")
for line in str_lines:
if 'releaseTime' in line:
print(line)
def process_exited(self):
self.ready.clear()
self.stopped.set()
self.ready.set()
async def stop(self):
t = asyncio.create_task(self.stopped.wait())
try:
self.transport.send_signal(signal.SIGINT)
await asyncio.wait_for(t, 3)
# log.warning("stopped go hub")
except asyncio.TimeoutError:
if not t.done():
t.cancel()
self.transport.terminate()
await self.stopped.wait()
log.warning("terminated go hub")
class HubNode:
def __init__(self, url, daemon, spv_node):
self.spv_node = spv_node
self.debug = False
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
@@ -495,11 +739,13 @@ class HubNode:
self.protocol = None
self.hostname = 'localhost'
self.rpcport = 50051 # avoid conflict with default rpc port
self.stopped = False
self.restart_ready = asyncio.Event()
self.restart_ready.set()
self._stopped = asyncio.Event()
self.running = asyncio.Event()
@property
def stopped(self):
return not self.running.is_set()
@property
def exists(self):
return (
@@ -550,33 +796,24 @@ class HubNode:
self.daemon_bin, 'serve', '--esindex', self.spv_node.index_name + 'claims', '--debug'
]
self.log.info(' '.join(command))
while not self.stopped:
if self.running.is_set():
await asyncio.sleep(1)
continue
await self.restart_ready.wait()
try:
if not self.debug:
self.transport, self.protocol = await loop.subprocess_exec(
HubProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start hub', exc_info=e)
self.protocol = HubProcess(self.running, self._stopped)
try:
self.transport, _ = await loop.subprocess_exec(
lambda: self.protocol, *command
)
self.protocol.transport = self.transport
except Exception as e:
log.exception('failed to start go hub', exc_info=e)
raise e
await self.protocol.ready.wait()
async def stop(self, cleanup=True):
self.stopped = True
try:
if not self.debug:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
if self.protocol:
await self.protocol.stop()
except Exception as e:
log.exception('failed to stop go hub', exc_info=e)
raise e
finally:
if cleanup:
self.cleanup()
+13 -8
View File
@@ -61,8 +61,10 @@ class ConductorService:
#set_logging(
# self.stack.ledger_module, logging.DEBUG, WebSocketLogHandler(self.send_message)
#)
self.stack.blockchain_started or await self.stack.start_blockchain()
self.send_message({'type': 'service', 'name': 'blockchain', 'port': self.stack.blockchain_node.port})
self.stack.lbcd_started or await self.stack.start_lbcd()
self.send_message({'type': 'service', 'name': 'lbcd', 'port': self.stack.lbcd_node.port})
self.stack.lbcwallet_started or await self.stack.start_lbcwallet()
self.send_message({'type': 'service', 'name': 'lbcwallet', 'port': self.stack.lbcwallet_node.port})
self.stack.spv_started or await self.stack.start_spv()
self.send_message({'type': 'service', 'name': 'spv', 'port': self.stack.spv_node.port})
self.stack.wallet_started or await self.stack.start_wallet()
@@ -74,7 +76,7 @@ class ConductorService:
async def generate(self, request):
data = await request.post()
blocks = data.get('blocks', 1)
await self.stack.blockchain_node.generate(int(blocks))
await self.stack.lbcwallet_node.generate(int(blocks))
return json_response({'blocks': blocks})
async def transfer(self, request):
@@ -85,11 +87,14 @@ class ConductorService:
if not address:
raise ValueError("No address was provided.")
amount = data.get('amount', 1)
txid = await self.stack.blockchain_node.send_to_address(address, amount)
if self.stack.wallet_started:
await self.stack.wallet_node.ledger.on_transaction.where(
lambda e: e.tx.id == txid and e.address == address
watcher = self.stack.wallet_node.ledger.on_transaction.where(
lambda e: e.address == address # and e.tx.id == txid -- might stall; see send_to_address_and_wait
)
txid = await self.stack.lbcwallet_node.send_to_address(address, amount)
await watcher
else:
txid = await self.stack.lbcwallet_node.send_to_address(address, amount)
return json_response({
'address': address,
'amount': amount,
@@ -98,7 +103,7 @@ class ConductorService:
async def balance(self, _):
return json_response({
'balance': await self.stack.blockchain_node.get_balance()
'balance': await self.stack.lbcwallet_node.get_balance()
})
async def log(self, request):
@@ -129,7 +134,7 @@ class ConductorService:
'type': 'status',
'height': self.stack.wallet_node.ledger.headers.height,
'balance': satoshis_to_coins(await self.stack.wallet_node.account.get_balance()),
'miner': await self.stack.blockchain_node.get_balance()
'miner': await self.stack.lbcwallet_node.get_balance()
})
def send_message(self, msg):
+240 -289
View File
@@ -1,148 +1,32 @@
import time
import asyncio
import typing
import signal
from bisect import bisect_right
from struct import pack, unpack
from typing import Optional, List, Tuple, Set, DefaultDict, Dict, NamedTuple
from struct import pack
from concurrent.futures.thread import ThreadPoolExecutor
from typing import Optional, List, Tuple, Set, DefaultDict, Dict
from prometheus_client import Gauge, Histogram
from collections import defaultdict
import lbry
from lbry.schema.claim import Claim
from lbry.wallet.ledger import Ledger, TestNetLedger, RegTestLedger
from lbry.wallet.transaction import OutputScript, Output, Transaction
from lbry.wallet.rpc.jsonrpc import RPCError
from lbry.wallet.server.tx import Tx, TxOutput, TxInput
from lbry.wallet.server.daemon import DaemonError
from lbry.wallet.server.hash import hash_to_hex_str, HASHX_LEN
from lbry.wallet.server.util import chunks, class_logger
from lbry.wallet.server.hash import hash_to_hex_str
from lbry.wallet.server.util import class_logger
from lbry.crypto.hash import hash160
from lbry.wallet.server.mempool import MemPool
from lbry.wallet.server.db.prefixes import ACTIVATED_SUPPORT_TXO_TYPE, ACTIVATED_CLAIM_TXO_TYPE
from lbry.wallet.server.db.prefixes import PendingActivationKey, PendingActivationValue, ClaimToTXOValue
from lbry.wallet.server.udp import StatusServer
from lbry.wallet.server.db.revertable import RevertableOpStack
from lbry.wallet.server.prefetcher import Prefetcher
from lbry.wallet.server.db.db import HubDB
from lbry.wallet.transaction import OutputScript, Output, Transaction
if typing.TYPE_CHECKING:
from lbry.wallet.server.leveldb import LevelDB
class TrendingNotification(NamedTuple):
height: int
added: bool
prev_amount: int
new_amount: int
class Prefetcher:
"""Prefetches blocks (in the forward direction only)."""
def __init__(self, daemon, coin, blocks_event):
self.logger = class_logger(__name__, self.__class__.__name__)
self.daemon = daemon
self.coin = coin
self.blocks_event = blocks_event
self.blocks = []
self.caught_up = False
# Access to fetched_height should be protected by the semaphore
self.fetched_height = None
self.semaphore = asyncio.Semaphore()
self.refill_event = asyncio.Event()
# The prefetched block cache size. The min cache size has
# little effect on sync time.
self.cache_size = 0
self.min_cache_size = 10 * 1024 * 1024
# This makes the first fetch be 10 blocks
self.ave_size = self.min_cache_size // 10
self.polling_delay = 5
async def main_loop(self, bp_height):
"""Loop forever polling for more blocks."""
await self.reset_height(bp_height)
while True:
try:
# Sleep a while if there is nothing to prefetch
await self.refill_event.wait()
if not await self._prefetch_blocks():
await asyncio.sleep(self.polling_delay)
except DaemonError as e:
self.logger.info(f'ignoring daemon error: {e}')
def get_prefetched_blocks(self):
"""Called by block processor when it is processing queued blocks."""
blocks = self.blocks
self.blocks = []
self.cache_size = 0
self.refill_event.set()
return blocks
async def reset_height(self, height):
"""Reset to prefetch blocks from the block processor's height.
Used in blockchain reorganisations. This coroutine can be
called asynchronously to the _prefetch_blocks coroutine so we
must synchronize with a semaphore.
"""
async with self.semaphore:
self.blocks.clear()
self.cache_size = 0
self.fetched_height = height
self.refill_event.set()
daemon_height = await self.daemon.height()
behind = daemon_height - height
if behind > 0:
self.logger.info(f'catching up to daemon height {daemon_height:,d} '
f'({behind:,d} blocks behind)')
else:
self.logger.info(f'caught up to daemon height {daemon_height:,d}')
async def _prefetch_blocks(self):
"""Prefetch some blocks and put them on the queue.
Repeats until the queue is full or caught up.
"""
daemon = self.daemon
daemon_height = await daemon.height()
async with self.semaphore:
while self.cache_size < self.min_cache_size:
# Try and catch up all blocks but limit to room in cache.
# Constrain fetch count to between 0 and 500 regardless;
# testnet can be lumpy.
cache_room = self.min_cache_size // self.ave_size
count = min(daemon_height - self.fetched_height, cache_room)
count = min(500, max(count, 0))
if not count:
self.caught_up = True
return False
first = self.fetched_height + 1
hex_hashes = await daemon.block_hex_hashes(first, count)
if self.caught_up:
self.logger.info('new block height {:,d} hash {}'
.format(first + count-1, hex_hashes[-1]))
blocks = await daemon.raw_blocks(hex_hashes)
assert count == len(blocks)
# Special handling for genesis block
if first == 0:
blocks[0] = self.coin.genesis_block(blocks[0])
self.logger.info(f'verified genesis block with hash {hex_hashes[0]}')
# Update our recent average block size estimate
size = sum(len(block) for block in blocks)
if count >= 10:
self.ave_size = size // count
else:
self.ave_size = (size + (10 - count) * self.ave_size) // 10
self.blocks.extend(blocks)
self.cache_size += size
self.fetched_height += count
self.blocks_event.set()
self.refill_event.clear()
return True
from lbry.wallet.server.env import Env
from lbry.wallet.server.db.revertable import RevertableOpStack
class ChainError(Exception):
@@ -174,7 +58,6 @@ class StagedClaimtrieItem(typing.NamedTuple):
)
NAMESPACE = "wallet_server"
HISTOGRAM_BUCKETS = (
.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0, 7.5, 10.0, 15.0, 20.0, 30.0, 60.0, float('inf')
)
@@ -188,22 +71,28 @@ class BlockProcessor:
"""
block_count_metric = Gauge(
"block_count", "Number of processed blocks", namespace=NAMESPACE
"block_count", "Number of processed blocks", namespace="block_processor"
)
block_update_time_metric = Histogram(
"block_time", "Block update times", namespace=NAMESPACE, buckets=HISTOGRAM_BUCKETS
"block_time", "Block update times", namespace="block_processor", buckets=HISTOGRAM_BUCKETS
)
reorg_count_metric = Gauge(
"reorg_count", "Number of reorgs", namespace=NAMESPACE
"reorg_count", "Number of reorgs", namespace="block_processor"
)
def __init__(self, env, db: 'LevelDB', daemon, shutdown_event: asyncio.Event):
self.state_lock = asyncio.Lock()
def __init__(self, env: 'Env'):
self.cancellable_tasks = []
self.env = env
self.db = db
self.daemon = daemon
self.mempool = MemPool(env.coin, daemon, db, self.state_lock)
self.shutdown_event = shutdown_event
self.state_lock = asyncio.Lock()
self.daemon = env.coin.DAEMON(env.coin, env.daemon_url)
self._chain_executor = ThreadPoolExecutor(1, thread_name_prefix='block-processor')
self.db = HubDB(
env.coin, env.db_dir, env.cache_MB, env.reorg_limit, env.cache_all_claim_txos, env.cache_all_tx_hashes,
max_open_files=env.db_max_open_files, blocking_channel_ids=env.blocking_channel_ids,
filtering_channel_ids=env.filtering_channel_ids, executor=self._chain_executor
)
self.shutdown_event = asyncio.Event()
self.coin = env.coin
if env.coin.NET == 'mainnet':
self.ledger = Ledger
@@ -211,6 +100,7 @@ class BlockProcessor:
self.ledger = TestNetLedger
else:
self.ledger = RegTestLedger
self.wait_for_blocks_duration = 0.1
self._caught_up_event: Optional[asyncio.Event] = None
self.height = 0
@@ -218,7 +108,7 @@ class BlockProcessor:
self.tx_count = 0
self.blocks_event = asyncio.Event()
self.prefetcher = Prefetcher(daemon, env.coin, self.blocks_event)
self.prefetcher = Prefetcher(self.daemon, env.coin, self.blocks_event)
self.logger = class_logger(__name__, self.__class__.__name__)
# Meta
@@ -228,11 +118,7 @@ class BlockProcessor:
self.utxo_cache: Dict[Tuple[bytes, int], Tuple[bytes, int]] = {}
# Claimtrie cache
self.db_op_stack: Optional[RevertableOpStack] = None
# self.search_cache = {}
self.history_cache = {}
self.status_server = StatusServer()
self.db_op_stack: Optional['RevertableOpStack'] = None
#################################
# attributes used for calculating stake activations and takeovers per block
@@ -269,7 +155,6 @@ class BlockProcessor:
self.removed_claims_to_send_es = set() # cumulative changes across blocks to send ES
self.touched_claims_to_send_es = set()
self.activation_info_to_send_es: DefaultDict[str, List[TrendingNotification]] = defaultdict(list)
self.removed_claim_hashes: Set[bytes] = set() # per block changes
self.touched_claim_hashes: Set[bytes] = set()
@@ -291,14 +176,8 @@ class BlockProcessor:
self.pending_transaction_num_mapping: Dict[bytes, int] = {}
self.pending_transactions: Dict[int, bytes] = {}
async def claim_producer(self):
if self.db.db_height <= 1:
return
for claim_hash in self.removed_claims_to_send_es:
yield 'delete', claim_hash.hex()
async for claim in self.db.claims_producer(self.touched_claims_to_send_es):
yield 'update', claim
self._stopping = False
self._ready_to_stop = asyncio.Event()
async def run_in_thread_with_lock(self, func, *args):
# Run in a thread to prevent blocking. Shielded so that
@@ -308,15 +187,52 @@ class BlockProcessor:
# consistent and not being updated elsewhere.
async def run_in_thread_locked():
async with self.state_lock:
return await asyncio.get_event_loop().run_in_executor(None, func, *args)
return await asyncio.get_event_loop().run_in_executor(self._chain_executor, func, *args)
return await asyncio.shield(run_in_thread_locked())
@staticmethod
async def run_in_thread(func, *args):
async def run_in_thread(self, func, *args):
async def run_in_thread():
return await asyncio.get_event_loop().run_in_executor(None, func, *args)
return await asyncio.get_event_loop().run_in_executor(self._chain_executor, func, *args)
return await asyncio.shield(run_in_thread())
async def refresh_mempool(self):
def fetch_mempool(mempool_prefix):
return {
k.tx_hash: v.raw_tx for (k, v) in mempool_prefix.iterate()
}
def update_mempool(unsafe_commit, mempool_prefix, to_put, to_delete):
for tx_hash, raw_tx in to_put:
mempool_prefix.stage_put((tx_hash,), (raw_tx,))
for tx_hash, raw_tx in to_delete.items():
mempool_prefix.stage_delete((tx_hash,), (raw_tx,))
unsafe_commit()
async with self.state_lock:
current_mempool = await self.run_in_thread(fetch_mempool, self.db.prefix_db.mempool_tx)
_to_put = []
try:
mempool_hashes = await self.daemon.mempool_hashes()
except (TypeError, RPCError):
self.logger.warning("failed to get mempool tx hashes, reorg underway?")
return
for hh in mempool_hashes:
tx_hash = bytes.fromhex(hh)[::-1]
if tx_hash in current_mempool:
current_mempool.pop(tx_hash)
else:
try:
_to_put.append((tx_hash, bytes.fromhex(await self.daemon.getrawtransaction(hh))))
except (TypeError, RPCError):
self.logger.warning("failed to get a mempool tx, reorg underway?")
return
if current_mempool:
if bytes.fromhex(await self.daemon.getbestblockhash())[::-1] != self.coin.header_hash(self.db.headers[-1]):
return
await self.run_in_thread(
update_mempool, self.db.prefix_db.unsafe_commit, self.db.prefix_db.mempool_tx, _to_put, current_mempool
)
async def check_and_advance_blocks(self, raw_blocks):
"""Process the list of raw blocks passed. Detects and handles
reorgs.
@@ -335,45 +251,23 @@ class BlockProcessor:
total_start = time.perf_counter()
try:
for block in blocks:
if self._stopping:
return
start = time.perf_counter()
await self.run_in_thread(self.advance_block, block)
await self.flush()
await self.run_in_thread_with_lock(self.advance_block, block)
self.logger.info("advanced to %i in %0.3fs", self.height, time.perf_counter() - start)
self.logger.info("writer advanced to %i in %0.3fs", self.height, time.perf_counter() - start)
if self.height == self.coin.nExtendedClaimExpirationForkHeight:
self.logger.warning(
"applying extended claim expiration fork on claims accepted by, %i", self.height
)
await self.run_in_thread_with_lock(self.db.apply_expiration_extension_fork)
if self.db.first_sync:
self.db.search_index.clear_caches()
self.touched_claims_to_send_es.clear()
self.removed_claims_to_send_es.clear()
self.activation_info_to_send_es.clear()
# TODO: we shouldnt wait on the search index updating before advancing to the next block
if not self.db.first_sync:
await self.db.reload_blocking_filtering_streams()
await self.db.search_index.claim_consumer(self.claim_producer())
await self.db.search_index.apply_filters(self.db.blocked_streams, self.db.blocked_channels,
self.db.filtered_streams, self.db.filtered_channels)
await self.db.search_index.update_trending_score(self.activation_info_to_send_es)
self.db.search_index.clear_caches()
self.touched_claims_to_send_es.clear()
self.removed_claims_to_send_es.clear()
self.activation_info_to_send_es.clear()
# print("******************\n")
except:
self.logger.exception("advance blocks failed")
raise
processed_time = time.perf_counter() - total_start
self.block_count_metric.set(self.height)
self.block_update_time_metric.observe(processed_time)
self.status_server.set_height(self.db.fs_height, self.db.db_tip)
if not self.db.first_sync:
s = '' if len(blocks) == 1 else 's'
self.logger.info('processed {:,d} block{} in {:.1f}s'.format(len(blocks), s, processed_time))
if self._caught_up_event.is_set():
await self.mempool.on_block(self.touched_hashXs, self.height)
self.touched_hashXs.clear()
elif hprevs[0] != chain[0]:
min_start_height = max(self.height - self.coin.REORG_LIMIT, 0)
@@ -391,19 +285,11 @@ class BlockProcessor:
try:
assert count > 0, count
for _ in range(count):
await self.backup_block()
await self.run_in_thread_with_lock(self.backup_block)
self.logger.info(f'backed up to height {self.height:,d}')
await self.db._read_claim_txos() # TODO: don't do this
for touched in self.touched_claims_to_send_es:
if not self.db.get_claim_txo(touched):
self.removed_claims_to_send_es.add(touched)
self.touched_claims_to_send_es.difference_update(self.removed_claims_to_send_es)
await self.db.search_index.claim_consumer(self.claim_producer())
self.db.search_index.clear_caches()
self.touched_claims_to_send_es.clear()
self.removed_claims_to_send_es.clear()
self.activation_info_to_send_es.clear()
if self.env.cache_all_claim_txos:
await self.db._read_claim_txos() # TODO: don't do this
await self.prefetcher.reset_height(self.height)
self.reorg_count_metric.inc()
except:
@@ -421,19 +307,6 @@ class BlockProcessor:
'resetting the prefetcher')
await self.prefetcher.reset_height(self.height)
async def flush(self):
save_undo = (self.daemon.cached_height() - self.height) <= self.env.reorg_limit
def flush():
self.db.write_db_state()
if save_undo:
self.db.prefix_db.commit(self.height)
else:
self.db.prefix_db.unsafe_commit()
self.clear_after_advance_or_reorg()
self.db.assert_db_state()
await self.run_in_thread_with_lock(flush)
def _add_claim_or_update(self, height: int, txo: 'Output', tx_hash: bytes, tx_num: int, nout: int,
spent_claims: typing.Dict[bytes, typing.Tuple[int, int, str]]):
try:
@@ -480,9 +353,7 @@ class BlockProcessor:
if signing_channel:
raw_channel_tx = self.db.prefix_db.tx.get(
self.db.prefix_db.tx_hash.get(
signing_channel.tx_num, deserialize_value=False
), deserialize_value=False
self.db.get_tx_hash(signing_channel.tx_num), deserialize_value=False
)
channel_pub_key_bytes = None
try:
@@ -498,7 +369,7 @@ class BlockProcessor:
channel_pub_key_bytes = channel_meta.channel.public_key_bytes
if channel_pub_key_bytes:
channel_signature_is_valid = Output.is_signature_valid(
txo.get_encoded_signature(), txo.get_signature_digest(self.ledger), channel_pub_key_bytes
txo.signable.signature, txo.get_signature_digest(self.ledger), channel_pub_key_bytes
)
if channel_signature_is_valid:
self.pending_channel_counts[signing_channel_hash] += 1
@@ -537,10 +408,11 @@ class BlockProcessor:
previous_amount = previous_claim.amount
self.updated_claims.add(claim_hash)
self.db.claim_to_txo[claim_hash] = ClaimToTXOValue(
tx_num, nout, root_tx_num, root_idx, txo.amount, channel_signature_is_valid, claim_name
)
self.db.txo_to_claim[tx_num][nout] = claim_hash
if self.env.cache_all_claim_txos:
self.db.claim_to_txo[claim_hash] = ClaimToTXOValue(
tx_num, nout, root_tx_num, root_idx, txo.amount, channel_signature_is_valid, claim_name
)
self.db.txo_to_claim[tx_num][nout] = claim_hash
pending = StagedClaimtrieItem(
claim_name, normalized_name, claim_hash, txo.amount, self.coin.get_expiration_height(height), tx_num, nout,
@@ -685,7 +557,7 @@ class BlockProcessor:
if (txin_num, txin.prev_idx) in self.txo_to_claim:
spent = self.txo_to_claim[(txin_num, txin.prev_idx)]
else:
if txin_num not in self.db.txo_to_claim or txin.prev_idx not in self.db.txo_to_claim[txin_num]:
if not self.db.get_cached_claim_exists(txin_num, txin.prev_idx):
# txo is not a claim
return False
spent_claim_hash_and_name = self.db.get_claim_from_txo(
@@ -693,10 +565,12 @@ class BlockProcessor:
)
assert spent_claim_hash_and_name is not None
spent = self._make_pending_claim_txo(spent_claim_hash_and_name.claim_hash)
claim_hash = self.db.txo_to_claim[txin_num].pop(txin.prev_idx)
if not self.db.txo_to_claim[txin_num]:
self.db.txo_to_claim.pop(txin_num)
self.db.claim_to_txo.pop(claim_hash)
if self.env.cache_all_claim_txos:
claim_hash = self.db.txo_to_claim[txin_num].pop(txin.prev_idx)
if not self.db.txo_to_claim[txin_num]:
self.db.txo_to_claim.pop(txin_num)
self.db.claim_to_txo.pop(claim_hash)
if spent.reposted_claim_hash:
self.pending_reposted.add(spent.reposted_claim_hash)
if spent.signing_hash and spent.channel_signature_is_valid and spent.signing_hash not in self.abandoned_claims:
@@ -1014,8 +888,8 @@ class BlockProcessor:
# prepare to activate or delay activation of the pending claims being added this block
for (tx_num, nout), staged in self.txo_to_claim.items():
is_delayed = not staged.is_update
if staged.claim_hash in self.db.claim_to_txo:
prev_txo = self.db.claim_to_txo[staged.claim_hash]
prev_txo = self.db.get_cached_claim_txo(staged.claim_hash)
if prev_txo:
prev_activation = self.db.get_activation(prev_txo.tx_num, prev_txo.position)
if height < prev_activation or prev_activation < 0:
is_delayed = True
@@ -1309,10 +1183,6 @@ class BlockProcessor:
self.touched_claim_hashes.add(controlling.claim_hash)
self.touched_claim_hashes.add(winning)
def _add_claim_activation_change_notification(self, claim_id: str, height: int, added: bool, prev_amount: int,
new_amount: int):
self.activation_info_to_send_es[claim_id].append(TrendingNotification(height, added, prev_amount, new_amount))
def _get_cumulative_update_ops(self, height: int):
# update the last takeover height for names with takeovers
for name in self.taken_over_names:
@@ -1337,6 +1207,8 @@ class BlockProcessor:
set(self.removed_active_support_amount_by_claim.keys())
).union(
set(self.activated_support_amount_by_claim.keys())
).union(
set(self.pending_support_amount_change.keys())
).difference(
self.removed_claim_hashes
)
@@ -1389,25 +1261,17 @@ class BlockProcessor:
(name, prev_effective_amount, amt.tx_num, amt.position), (touched,)
)
if (name, touched) in self.activated_claim_amount_by_name_and_hash:
self._add_claim_activation_change_notification(
touched.hex(), height, True, prev_effective_amount,
self.activated_claim_amount_by_name_and_hash[(name, touched)]
)
if touched in self.activated_support_amount_by_claim:
for support_amount in self.activated_support_amount_by_claim[touched]:
self._add_claim_activation_change_notification(
touched.hex(), height, True, prev_effective_amount, support_amount
)
if touched in self.removed_active_support_amount_by_claim:
for support_amount in self.removed_active_support_amount_by_claim[touched]:
self._add_claim_activation_change_notification(
touched.hex(), height, False, prev_effective_amount, support_amount
)
new_effective_amount = self._get_pending_effective_amount(name, touched)
self.db.prefix_db.effective_amount.stage_put(
(name, new_effective_amount, tx_num, position), (touched,)
)
if touched in self.claim_hash_to_txo or touched in self.removed_claim_hashes \
or touched in self.pending_support_amount_change:
# exclude sending notifications for claims/supports that activated but
# weren't added/spent in this block
self.db.prefix_db.trending_notification.stage_put(
(height, touched), (prev_effective_amount, new_effective_amount)
)
for channel_hash, count in self.pending_channel_counts.items():
if count != 0:
@@ -1423,8 +1287,8 @@ class BlockProcessor:
self.touched_claim_hashes.update(
{k for k, v in self.pending_channel_counts.items() if v != 0 and k not in self.removed_claim_hashes}
)
self.touched_claims_to_send_es.difference_update(self.removed_claim_hashes)
self.touched_claims_to_send_es.update(self.touched_claim_hashes)
self.touched_claims_to_send_es.difference_update(self.removed_claim_hashes)
self.removed_claims_to_send_es.update(self.removed_claim_hashes)
def advance_block(self, block):
@@ -1440,11 +1304,18 @@ class BlockProcessor:
self.db.prefix_db.block_hash.stage_put(key_args=(height,), value_args=(self.coin.header_hash(block.header),))
self.db.prefix_db.header.stage_put(key_args=(height,), value_args=(block.header,))
self.db.prefix_db.block_txs.stage_put(key_args=(height,), value_args=([tx_hash for tx, tx_hash in txs],))
for tx, tx_hash in txs:
spent_claims = {}
txos = Transaction(tx.raw).outputs
# clean up mempool, delete txs that were already in mempool/staged to be added
# leave txs in mempool that werent in the block
mempool_tx = self.db.prefix_db.mempool_tx.get_pending(tx_hash)
if mempool_tx:
self.db.prefix_db.mempool_tx.stage_delete((tx_hash,), mempool_tx)
self.db.prefix_db.tx.stage_put(key_args=(tx_hash,), value_args=(tx.raw,))
self.db.prefix_db.tx_num.stage_put(key_args=(tx_hash,), value_args=(tx_count,))
self.db.prefix_db.tx_hash.stage_put(key_args=(tx_count,), value_args=(tx_hash,))
@@ -1490,6 +1361,9 @@ class BlockProcessor:
self._abandon_claim(abandoned_claim_hash, tx_num, nout, normalized_name)
self.pending_transactions[tx_count] = tx_hash
self.pending_transaction_num_mapping[tx_hash] = tx_count
if self.env.cache_all_tx_hashes:
self.db.total_transactions.append(tx_hash)
self.db.tx_num_mapping[tx_hash] = tx_count
tx_count += 1
# handle expired claims
@@ -1501,6 +1375,8 @@ class BlockProcessor:
# update effective amount and update sets of touched and deleted claims
self._get_cumulative_update_ops(height)
self.db.prefix_db.touched_hashX.stage_put((height,), (list(sorted(self.touched_hashXs)),))
self.db.prefix_db.tx_count.stage_put(key_args=(height,), value_args=(tx_count,))
for hashX, new_history in self.hashXs_by_tx.items():
@@ -1512,26 +1388,16 @@ class BlockProcessor:
self.db.tx_counts.append(self.tx_count)
cached_max_reorg_depth = self.daemon.cached_height() - self.env.reorg_limit
if height >= cached_max_reorg_depth:
self.db.prefix_db.touched_or_deleted.stage_put(
key_args=(height,), value_args=(self.touched_claim_hashes, self.removed_claim_hashes)
)
# if height >= cached_max_reorg_depth:
self.db.prefix_db.touched_or_deleted.stage_put(
key_args=(height,), value_args=(self.touched_claim_hashes, self.removed_claim_hashes)
)
self.height = height
self.db.headers.append(block.header)
self.tip = self.coin.header_hash(block.header)
min_height = self.db.min_undo_height(self.db.db_height)
if min_height > 0: # delete undos for blocks deep enough they can't be reorged
undo_to_delete = list(self.db.prefix_db.undo.iterate(start=(0,), stop=(min_height,)))
for (k, v) in undo_to_delete:
self.db.prefix_db.undo.stage_delete((k,), (v,))
touched_or_deleted_to_delete = list(self.db.prefix_db.touched_or_deleted.iterate(
start=(0,), stop=(min_height,))
)
for (k, v) in touched_or_deleted_to_delete:
self.db.prefix_db.touched_or_deleted.stage_delete(k, v)
self.db.fs_height = self.height
self.db.fs_tx_count = self.tx_count
self.db.hist_flush_count += 1
@@ -1544,9 +1410,18 @@ class BlockProcessor:
now = time.time()
self.db.wall_time += now - self.db.last_flush
self.db.last_flush = now
self.db.write_db_state()
# flush the changes
save_undo = (self.daemon.cached_height() - self.height) <= self.env.reorg_limit
if save_undo:
self.db.prefix_db.commit(self.height, self.tip)
else:
self.db.prefix_db.unsafe_commit()
self.clear_after_advance_or_reorg()
self.db.assert_db_state()
def clear_after_advance_or_reorg(self):
self.txo_to_claim.clear()
self.claim_hash_to_txo.clear()
@@ -1569,8 +1444,6 @@ class BlockProcessor:
self.claim_channels.clear()
self.utxo_cache.clear()
self.hashXs_by_tx.clear()
self.history_cache.clear()
self.mempool.notified_mempool_txs.clear()
self.removed_claim_hashes.clear()
self.touched_claim_hashes.clear()
self.pending_reposted.clear()
@@ -1580,8 +1453,9 @@ class BlockProcessor:
self.pending_transaction_num_mapping.clear()
self.pending_transactions.clear()
self.pending_support_amount_change.clear()
self.touched_hashXs.clear()
async def backup_block(self):
def backup_block(self):
assert len(self.db.prefix_db._op_stack) == 0
touched_and_deleted = self.db.prefix_db.touched_or_deleted.get(self.height)
self.touched_claims_to_send_es.update(touched_and_deleted.touched_claims)
@@ -1592,11 +1466,17 @@ class BlockProcessor:
self.logger.info("backup block %i", self.height)
# Check and update self.tip
self.db.headers.pop()
self.db.tx_counts.pop()
reverted_block_hash = self.coin.header_hash(self.db.headers.pop())
self.tip = self.coin.header_hash(self.db.headers[-1])
self.tx_count = self.db.tx_counts[-1]
if self.env.cache_all_tx_hashes:
while len(self.db.total_transactions) > self.db.tx_counts[-1]:
self.db.tx_num_mapping.pop(self.db.total_transactions.pop())
self.tx_count -= 1
else:
self.tx_count = self.db.tx_counts[-1]
self.height -= 1
# self.touched can include other addresses which is
# harmless, but remove None.
self.touched_hashXs.discard(None)
@@ -1626,8 +1506,14 @@ class BlockProcessor:
self.db.last_flush = now
self.db.last_flush_tx_count = self.db.fs_tx_count
await self.run_in_thread_with_lock(self.db.prefix_db.rollback, self.height + 1)
# rollback
self.db.prefix_db.rollback(self.height + 1, reverted_block_hash)
self.db.es_sync_height = self.height
self.db.write_db_state()
self.db.prefix_db.unsafe_commit()
self.clear_after_advance_or_reorg()
self.db.assert_db_state()
elapsed = self.db.last_flush - start_time
self.logger.warning(f'backup flush #{self.db.hist_flush_count:,d} took {elapsed:.1f}s. '
@@ -1646,7 +1532,7 @@ class BlockProcessor:
if tx_hash in self.pending_transaction_num_mapping:
return self.pending_transaction_num_mapping[tx_hash]
else:
return self.db.prefix_db.tx_num.get(tx_hash).tx_num
return self.db.get_tx_num(tx_hash)
def spend_utxo(self, tx_hash: bytes, nout: int):
hashX, amount = self.utxo_cache.pop((tx_hash, nout), (None, None))
@@ -1674,21 +1560,40 @@ class BlockProcessor:
self.touched_hashXs.add(hashX)
return hashX
async def _process_prefetched_blocks(self):
async def process_blocks_and_mempool_forever(self):
"""Loop forever processing blocks as they arrive."""
while True:
if self.height == self.daemon.cached_height():
if not self._caught_up_event.is_set():
await self._first_caught_up()
self._caught_up_event.set()
await self.blocks_event.wait()
self.blocks_event.clear()
blocks = self.prefetcher.get_prefetched_blocks()
try:
await self.check_and_advance_blocks(blocks)
except Exception:
self.logger.exception("error while processing txs")
raise
try:
while not self._stopping:
if self.height == self.daemon.cached_height():
if not self._caught_up_event.is_set():
await self._first_caught_up()
self._caught_up_event.set()
try:
await asyncio.wait_for(self.blocks_event.wait(), self.wait_for_blocks_duration)
except asyncio.TimeoutError:
pass
self.blocks_event.clear()
blocks = self.prefetcher.get_prefetched_blocks()
if self._stopping:
break
if not blocks:
try:
await self.refresh_mempool()
except asyncio.CancelledError:
raise
except Exception:
self.logger.exception("error while updating mempool txs")
raise
else:
try:
await self.check_and_advance_blocks(blocks)
except asyncio.CancelledError:
raise
except Exception:
self.logger.exception("error while processing txs")
raise
finally:
self._ready_to_stop.set()
async def _first_caught_up(self):
self.logger.info(f'caught up to height {self.height}')
@@ -1709,6 +1614,13 @@ class BlockProcessor:
f'height {self.height:,d}, halting here.')
self.shutdown_event.set()
async def open(self):
self.db.open_db()
self.height = self.db.db_height
self.tip = self.db.db_tip
self.tx_count = self.db.db_tx_count
await self.db.initialize_caches()
async def fetch_and_process_blocks(self, caught_up_event):
"""Fetch, process and index blocks from the daemon.
@@ -1722,16 +1634,13 @@ class BlockProcessor:
could be lost.
"""
await self.open()
self._caught_up_event = caught_up_event
try:
await self.db.open_dbs()
self.height = self.db.db_height
self.tip = self.db.db_tip
self.tx_count = self.db.db_tx_count
self.status_server.set_height(self.db.fs_height, self.db.db_tip)
await asyncio.wait([
self.prefetcher.main_loop(self.height),
self._process_prefetched_blocks()
self.process_blocks_and_mempool_forever()
])
except asyncio.CancelledError:
raise
@@ -1739,8 +1648,50 @@ class BlockProcessor:
self.logger.exception("Block processing failed!")
raise
finally:
self.status_server.stop()
# Shut down block processing
self.logger.info('closing the DB for a clean shutdown...')
self._chain_executor.shutdown(wait=True)
self.db.close()
# self.executor.shutdown(wait=True)
async def start(self):
self._stopping = False
env = self.env
min_str, max_str = env.coin.SESSIONCLS.protocol_min_max_strings()
self.logger.info(f'software version: {lbry.__version__}')
self.logger.info(f'supported protocol versions: {min_str}-{max_str}')
self.logger.info(f'event loop policy: {env.loop_policy}')
self.logger.info(f'reorg limit is {env.reorg_limit:,d} blocks')
await self.daemon.height()
def _start_cancellable(run, *args):
_flag = asyncio.Event()
self.cancellable_tasks.append(asyncio.ensure_future(run(*args, _flag)))
return _flag.wait()
await _start_cancellable(self.fetch_and_process_blocks)
async def stop(self):
self._stopping = True
await self._ready_to_stop.wait()
for task in reversed(self.cancellable_tasks):
task.cancel()
await asyncio.wait(self.cancellable_tasks)
self.shutdown_event.set()
await self.daemon.close()
def run(self):
loop = asyncio.get_event_loop()
loop.set_default_executor(self._chain_executor)
def __exit():
raise SystemExit()
try:
loop.add_signal_handler(signal.SIGINT, __exit)
loop.add_signal_handler(signal.SIGTERM, __exit)
loop.run_until_complete(self.start())
loop.run_until_complete(self.shutdown_event.wait())
except (SystemExit, KeyboardInterrupt):
pass
finally:
loop.run_until_complete(self.stop())
+285
View File
@@ -0,0 +1,285 @@
import signal
import logging
import asyncio
import typing
from concurrent.futures.thread import ThreadPoolExecutor
from prometheus_client import Gauge, Histogram
import lbry
from lbry.wallet.server.mempool import MemPool
from lbry.wallet.server.db.prefixes import DBState
from lbry.wallet.server.udp import StatusServer
from lbry.wallet.server.db.db import HubDB
from lbry.wallet.server.db.elasticsearch.notifier import ElasticNotifierClientProtocol
from lbry.wallet.server.session import LBRYSessionManager
from lbry.prometheus import PrometheusServer
HISTOGRAM_BUCKETS = (
.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0, 7.5, 10.0, 15.0, 20.0, 30.0, 60.0, float('inf')
)
class BlockchainReader:
block_count_metric = Gauge(
"block_count", "Number of processed blocks", namespace="blockchain_reader"
)
block_update_time_metric = Histogram(
"block_time", "Block update times", namespace="blockchain_reader", buckets=HISTOGRAM_BUCKETS
)
reorg_count_metric = Gauge(
"reorg_count", "Number of reorgs", namespace="blockchain_reader"
)
def __init__(self, env, secondary_name: str, thread_workers: int = 1, thread_prefix: str = 'blockchain-reader'):
self.env = env
self.log = logging.getLogger(__name__).getChild(self.__class__.__name__)
self.shutdown_event = asyncio.Event()
self.cancellable_tasks = []
self._thread_workers = thread_workers
self._thread_prefix = thread_prefix
self._executor = ThreadPoolExecutor(thread_workers, thread_name_prefix=thread_prefix)
self.db = HubDB(
env.coin, env.db_dir, env.cache_MB, env.reorg_limit, env.cache_all_claim_txos, env.cache_all_tx_hashes,
secondary_name=secondary_name, max_open_files=-1, blocking_channel_ids=env.blocking_channel_ids,
filtering_channel_ids=env.filtering_channel_ids, executor=self._executor
)
self.last_state: typing.Optional[DBState] = None
self._refresh_interval = 0.1
self._lock = asyncio.Lock()
def _detect_changes(self):
try:
self.db.prefix_db.try_catch_up_with_primary()
except:
self.log.exception('failed to update secondary db')
raise
state = self.db.prefix_db.db_state.get()
if not state or state.height <= 0:
return
if self.last_state and self.last_state.height > state.height:
self.log.warning("reorg detected, waiting until the writer has flushed the new blocks to advance")
return
last_height = 0 if not self.last_state else self.last_state.height
rewound = False
if self.last_state:
while True:
if self.db.headers[-1] == self.db.prefix_db.header.get(last_height, deserialize_value=False):
self.log.debug("connects to block %i", last_height)
break
else:
self.log.warning("disconnect block %i", last_height)
self.unwind()
rewound = True
last_height -= 1
if rewound:
self.reorg_count_metric.inc()
self.db.read_db_state()
if not self.last_state or last_height < state.height:
for height in range(last_height + 1, state.height + 1):
self.log.info("advancing to %i", height)
self.advance(height)
self.clear_caches()
self.last_state = state
self.block_count_metric.set(self.last_state.height)
self.db.blocked_streams, self.db.blocked_channels = self.db.get_streams_and_channels_reposted_by_channel_hashes(
self.db.blocking_channel_hashes
)
self.db.filtered_streams, self.db.filtered_channels = self.db.get_streams_and_channels_reposted_by_channel_hashes(
self.db.filtering_channel_hashes
)
async def poll_for_changes(self):
await asyncio.get_event_loop().run_in_executor(self._executor, self._detect_changes)
async def refresh_blocks_forever(self, synchronized: asyncio.Event):
while True:
try:
async with self._lock:
await self.poll_for_changes()
except asyncio.CancelledError:
raise
except:
self.log.exception("blockchain reader main loop encountered an unexpected error")
raise
await asyncio.sleep(self._refresh_interval)
synchronized.set()
def clear_caches(self):
pass
def advance(self, height: int):
tx_count = self.db.prefix_db.tx_count.get(height).tx_count
assert tx_count not in self.db.tx_counts, f'boom {tx_count} in {len(self.db.tx_counts)} tx counts'
assert len(self.db.tx_counts) == height, f"{len(self.db.tx_counts)} != {height}"
self.db.tx_counts.append(tx_count)
self.db.headers.append(self.db.prefix_db.header.get(height, deserialize_value=False))
def unwind(self):
self.db.tx_counts.pop()
self.db.headers.pop()
async def start(self):
if not self._executor:
self._executor = ThreadPoolExecutor(self._thread_workers, thread_name_prefix=self._thread_prefix)
self.db._executor = self._executor
class BlockchainReaderServer(BlockchainReader):
block_count_metric = Gauge(
"block_count", "Number of processed blocks", namespace="wallet_server"
)
block_update_time_metric = Histogram(
"block_time", "Block update times", namespace="wallet_server", buckets=HISTOGRAM_BUCKETS
)
reorg_count_metric = Gauge(
"reorg_count", "Number of reorgs", namespace="wallet_server"
)
def __init__(self, env):
super().__init__(env, 'lbry-reader', thread_workers=max(1, env.max_query_workers), thread_prefix='hub-worker')
self.history_cache = {}
self.resolve_outputs_cache = {}
self.resolve_cache = {}
self.notifications_to_send = []
self.mempool_notifications = set()
self.status_server = StatusServer()
self.daemon = env.coin.DAEMON(env.coin, env.daemon_url) # only needed for broadcasting txs
self.prometheus_server: typing.Optional[PrometheusServer] = None
self.mempool = MemPool(self.env.coin, self.db)
self.session_manager = LBRYSessionManager(
env, self.db, self.mempool, self.history_cache, self.resolve_cache,
self.resolve_outputs_cache, self.daemon,
self.shutdown_event,
on_available_callback=self.status_server.set_available,
on_unavailable_callback=self.status_server.set_unavailable
)
self.mempool.session_manager = self.session_manager
self.es_notifications = asyncio.Queue()
self.es_notification_client = ElasticNotifierClientProtocol(self.es_notifications)
self.synchronized = asyncio.Event()
self._es_height = None
self._es_block_hash = None
def clear_caches(self):
self.history_cache.clear()
self.resolve_outputs_cache.clear()
self.resolve_cache.clear()
# self.clear_search_cache()
# self.mempool.notified_mempool_txs.clear()
def clear_search_cache(self):
self.session_manager.search_index.clear_caches()
def advance(self, height: int):
super().advance(height)
touched_hashXs = self.db.prefix_db.touched_hashX.get(height).touched_hashXs
self.notifications_to_send.append((set(touched_hashXs), height))
def _detect_changes(self):
super()._detect_changes()
self.mempool_notifications.update(self.mempool.refresh())
async def poll_for_changes(self):
await super().poll_for_changes()
if self.db.fs_height <= 0:
return
self.status_server.set_height(self.db.fs_height, self.db.db_tip)
if self.notifications_to_send:
for (touched, height) in self.notifications_to_send:
await self.mempool.on_block(touched, height)
self.log.info("reader advanced to %i", height)
if self._es_height == self.db.db_height:
self.synchronized.set()
if self.mempool_notifications:
await self.mempool.on_mempool(
set(self.mempool.touched_hashXs), self.mempool_notifications, self.db.db_height
)
self.mempool_notifications.clear()
self.notifications_to_send.clear()
async def receive_es_notifications(self, synchronized: asyncio.Event):
await asyncio.get_event_loop().create_connection(
lambda: self.es_notification_client, '127.0.0.1', self.env.elastic_notifier_port
)
synchronized.set()
try:
while True:
self._es_height, self._es_block_hash = await self.es_notifications.get()
self.clear_search_cache()
if self.last_state and self._es_block_hash == self.last_state.tip:
self.synchronized.set()
self.log.info("es and reader are in sync at block %i", self.last_state.height)
else:
self.log.info("es and reader are not yet in sync (block %s vs %s)", self._es_height,
self.db.db_height)
finally:
self.es_notification_client.close()
async def start(self):
await super().start()
env = self.env
min_str, max_str = env.coin.SESSIONCLS.protocol_min_max_strings()
self.log.info(f'software version: {lbry.__version__}')
self.log.info(f'supported protocol versions: {min_str}-{max_str}')
self.log.info(f'event loop policy: {env.loop_policy}')
self.log.info(f'reorg limit is {env.reorg_limit:,d} blocks')
await self.daemon.height()
def _start_cancellable(run, *args):
_flag = asyncio.Event()
self.cancellable_tasks.append(asyncio.ensure_future(run(*args, _flag)))
return _flag.wait()
self.db.open_db()
await self.db.initialize_caches()
self.last_state = self.db.read_db_state()
await self.start_prometheus()
if self.env.udp_port and int(self.env.udp_port):
await self.status_server.start(
0, bytes.fromhex(self.env.coin.GENESIS_HASH)[::-1], self.env.country,
self.env.host, self.env.udp_port, self.env.allow_lan_udp
)
await _start_cancellable(self.receive_es_notifications)
await _start_cancellable(self.refresh_blocks_forever)
await self.session_manager.search_index.start()
await _start_cancellable(self.session_manager.serve, self.mempool)
async def stop(self):
await self.status_server.stop()
async with self._lock:
while self.cancellable_tasks:
t = self.cancellable_tasks.pop()
if not t.done():
t.cancel()
await self.session_manager.search_index.stop()
self.db.close()
if self.prometheus_server:
await self.prometheus_server.stop()
self.prometheus_server = None
await self.daemon.close()
self._executor.shutdown(wait=True)
self._executor = None
self.shutdown_event.set()
def run(self):
loop = asyncio.get_event_loop()
loop.set_default_executor(self._executor)
def __exit():
raise SystemExit()
try:
loop.add_signal_handler(signal.SIGINT, __exit)
loop.add_signal_handler(signal.SIGTERM, __exit)
loop.run_until_complete(self.start())
loop.run_until_complete(self.shutdown_event.wait())
except (SystemExit, KeyboardInterrupt):
pass
finally:
loop.run_until_complete(self.stop())
async def start_prometheus(self):
if not self.prometheus_server and self.env.prometheus_port:
self.prometheus_server = PrometheusServer()
await self.prometheus_server.start("0.0.0.0", self.env.prometheus_port)
+43 -22
View File
@@ -1,43 +1,64 @@
import logging
import traceback
import argparse
import importlib
from lbry.wallet.server.env import Env
from lbry.wallet.server.server import Server
from lbry.wallet.server.block_processor import BlockProcessor
from lbry.wallet.server.chain_reader import BlockchainReaderServer
from lbry.wallet.server.db.elasticsearch.sync import ElasticWriter
def get_argument_parser():
def get_arg_parser(name):
parser = argparse.ArgumentParser(
prog="lbry-hub"
prog=name
)
parser.add_argument("spvserver", type=str, help="Python class path to SPV server implementation.",
nargs="?", default="lbry.wallet.server.coin.LBC")
Env.contribute_to_arg_parser(parser)
return parser
def get_coin_class(spvserver):
spvserver_path, coin_class_name = spvserver.rsplit('.', 1)
spvserver_module = importlib.import_module(spvserver_path)
return getattr(spvserver_module, coin_class_name)
def main():
parser = get_argument_parser()
args = parser.parse_args()
coin_class = get_coin_class(args.spvserver)
def setup_logging():
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-4s %(name)s:%(lineno)d: %(message)s")
logging.info('lbry.server starting')
logging.getLogger('aiohttp').setLevel(logging.WARNING)
logging.getLogger('elasticsearch').setLevel(logging.WARNING)
def run_writer_forever():
setup_logging()
args = get_arg_parser('lbry-hub-writer').parse_args()
try:
server = Server(Env(coin_class))
block_processor = BlockProcessor(Env.from_arg_parser(args))
block_processor.run()
except Exception:
traceback.print_exc()
logging.critical('block processor terminated abnormally')
else:
logging.info('block processor terminated normally')
def run_server_forever():
setup_logging()
args = get_arg_parser('lbry-hub-server').parse_args()
try:
server = BlockchainReaderServer(Env.from_arg_parser(args))
server.run()
except Exception:
traceback.print_exc()
logging.critical('lbry.server terminated abnormally')
logging.critical('server terminated abnormally')
else:
logging.info('lbry.server terminated normally')
logging.info('server terminated normally')
if __name__ == "__main__":
main()
def run_es_sync_forever():
setup_logging()
parser = get_arg_parser('lbry-hub-elastic-sync')
parser.add_argument('--reindex', type=bool, default=False)
args = parser.parse_args()
try:
server = ElasticWriter(Env.from_arg_parser(args))
server.run(args.reindex)
except Exception:
traceback.print_exc()
logging.critical('es writer terminated abnormally')
else:
logging.info('es writer terminated normally')
-5
View File
@@ -12,9 +12,7 @@ from lbry.wallet.server.util import cachedproperty, subclasses
from lbry.wallet.server.hash import Base58, hash160, double_sha256, hash_to_hex_str, HASHX_LEN
from lbry.wallet.server.daemon import Daemon, LBCDaemon
from lbry.wallet.server.script import ScriptPubKey, OpCodes
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.session import LBRYElectrumX, LBRYSessionManager
from lbry.wallet.server.block_processor import BlockProcessor
Block = namedtuple("Block", "raw header transactions")
@@ -38,9 +36,7 @@ class Coin:
SESSIONCLS = LBRYElectrumX
DESERIALIZER = lib_tx.Deserializer
DAEMON = Daemon
BLOCK_PROCESSOR = BlockProcessor
SESSION_MANAGER = LBRYSessionManager
DB = LevelDB
HEADER_VALUES = [
'version', 'prev_block_hash', 'merkle_root', 'timestamp', 'bits', 'nonce'
]
@@ -243,7 +239,6 @@ class LBC(Coin):
SESSIONCLS = LBRYElectrumX
SESSION_MANAGER = LBRYSessionManager
DESERIALIZER = DeserializerSegWit
DB = LevelDB
NAME = "LBRY"
SHORTNAME = "LBC"
NET = "mainnet"
+7 -37
View File
@@ -53,9 +53,9 @@ class Daemon:
self.max_retry = max_retry
self._height = None
self.available_rpcs = {}
self.connector = aiohttp.TCPConnector()
self.connector = aiohttp.TCPConnector(ssl=False)
self._block_hash_cache = LRUCacheWithMetrics(100000)
self._block_cache = LRUCacheWithMetrics(2 ** 16, metric_name='block', namespace=NAMESPACE)
self._block_cache = LRUCacheWithMetrics(2 ** 13, metric_name='block', namespace=NAMESPACE)
async def close(self):
if self.connector:
@@ -250,14 +250,14 @@ class Daemon:
async def deserialised_block(self, hex_hash):
"""Return the deserialised block with the given hex hash."""
if hex_hash not in self._block_cache:
block = await self._send_single('getblock', (hex_hash, True))
block = await self._send_single('getblock', (hex_hash, 1))
self._block_cache[hex_hash] = block
return block
return self._block_cache[hex_hash]
async def raw_blocks(self, hex_hashes):
"""Return the raw binary blocks with the given hex hashes."""
params_iterable = ((h, False) for h in hex_hashes)
params_iterable = ((h, 0) for h in hex_hashes)
blocks = await self._send_vector('getblock', params_iterable)
# Convert hex string to bytes
return [hex_to_bytes(block) for block in blocks]
@@ -334,42 +334,12 @@ class LBCDaemon(Daemon):
async def getrawtransaction(self, hex_hash, verbose=False):
return await super().getrawtransaction(hex_hash=hex_hash, verbose=verbose)
@handles_errors
async def getclaimbyid(self, claim_id):
'''Given a claim id, retrieves claim information.'''
return await self._send_single('getclaimbyid', (claim_id,))
@handles_errors
async def getclaimsbyids(self, claim_ids):
'''Given a list of claim ids, batches calls to retrieve claim information.'''
return await self._send_vector('getclaimbyid', ((claim_id,) for claim_id in claim_ids))
@handles_errors
async def getclaimsforname(self, name):
'''Given a name, retrieves all claims matching that name.'''
return await self._send_single('getclaimsforname', (name,))
@handles_errors
async def getclaimsfortx(self, txid):
'''Given a txid, returns the claims it make.'''
return await self._send_single('getclaimsfortx', (txid,)) or []
@handles_errors
async def getnameproof(self, name, block_hash=None):
'''Given a name and optional block_hash, returns a name proof and winner, if any.'''
return await self._send_single('getnameproof', (name, block_hash,) if block_hash else (name,))
@handles_errors
async def getvalueforname(self, name):
'''Given a name, returns the winning claim value.'''
return await self._send_single('getvalueforname', (name,))
@handles_errors
async def getnamesintrie(self):
'''Given a name, returns the winning claim value.'''
return await self._send_single('getnamesintrie')
@handles_errors
async def claimname(self, name, hexvalue, amount):
'''Claim a name, used for functional tests only.'''
return await self._send_single('claimname', (name, hexvalue, float(amount)))
async def getbestblockhash(self):
'''Given a name, retrieves all claims matching that name.'''
return await self._send_single('getbestblockhash')
+9 -1
View File
@@ -1,4 +1,5 @@
import enum
from typing import NamedTuple, Optional
@enum.unique
@@ -25,7 +26,7 @@ class DB_PREFIXES(enum.Enum):
reposted_claim = b'W'
undo = b'M'
claim_diff = b'Y'
touched_or_deleted = b'Y'
tx = b'B'
block_hash = b'C'
@@ -39,3 +40,10 @@ class DB_PREFIXES(enum.Enum):
db_state = b's'
channel_count = b'Z'
support_amount = b'a'
block_tx = b'b'
trending_notifications = b'c'
mempool_tx = b'd'
touched_hashX = b'e'
COLUMN_SETTINGS = {} # this is updated by the PrefixRow metaclass
+30
View File
@@ -1,4 +1,6 @@
import typing
from typing import Optional
from lbry.error import ResolveCensoredError
CLAIM_TYPES = {
'stream': 1,
@@ -445,3 +447,31 @@ class ResolveResult(typing.NamedTuple):
channel_hash: typing.Optional[bytes]
reposted_claim_hash: typing.Optional[bytes]
signature_valid: typing.Optional[bool]
class TrendingNotification(typing.NamedTuple):
height: int
prev_amount: int
new_amount: int
class UTXO(typing.NamedTuple):
tx_num: int
tx_pos: int
tx_hash: bytes
height: int
value: int
OptionalResolveResultOrError = Optional[typing.Union[ResolveResult, ResolveCensoredError, LookupError, ValueError]]
class ExpandedResolveResult(typing.NamedTuple):
stream: OptionalResolveResultOrError
channel: OptionalResolveResultOrError
repost: OptionalResolveResultOrError
reposted_channel: OptionalResolveResultOrError
class DBError(Exception):
"""Raised on general DB errors generally indicating corruption."""
+1135 -102
View File
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,150 @@
from decimal import Decimal
from typing import Iterable
from lbry.error import TooManyClaimSearchParametersError
from lbry.schema.tags import clean_tags
from lbry.schema.url import normalize_name
from lbry.wallet.server.db.common import CLAIM_TYPES, STREAM_TYPES
from lbry.wallet.server.db.elasticsearch.constants import REPLACEMENTS, FIELDS, TEXT_FIELDS, RANGE_FIELDS
def expand_query(**kwargs):
if "amount_order" in kwargs:
kwargs["limit"] = 1
kwargs["order_by"] = "effective_amount"
kwargs["offset"] = int(kwargs["amount_order"]) - 1
if 'name' in kwargs:
kwargs['name'] = normalize_name(kwargs.pop('name'))
if kwargs.get('is_controlling') is False:
kwargs.pop('is_controlling')
query = {'must': [], 'must_not': []}
collapse = None
if 'fee_currency' in kwargs and kwargs['fee_currency'] is not None:
kwargs['fee_currency'] = kwargs['fee_currency'].upper()
for key, value in kwargs.items():
key = key.replace('claim.', '')
many = key.endswith('__in') or isinstance(value, list)
if many and len(value) > 2048:
raise TooManyClaimSearchParametersError(key, 2048)
if many:
key = key.replace('__in', '')
value = list(filter(None, value))
if value is None or isinstance(value, list) and len(value) == 0:
continue
key = REPLACEMENTS.get(key, key)
if key in FIELDS:
partial_id = False
if key == 'claim_type':
if isinstance(value, str):
value = CLAIM_TYPES[value]
else:
value = [CLAIM_TYPES[claim_type] for claim_type in value]
elif key == 'stream_type':
value = [STREAM_TYPES[value]] if isinstance(value, str) else list(map(STREAM_TYPES.get, value))
if key == '_id':
if isinstance(value, Iterable):
value = [item[::-1].hex() for item in value]
else:
value = value[::-1].hex()
if not many and key in ('_id', 'claim_id') and len(value) < 20:
partial_id = True
if key in ('signature_valid', 'has_source'):
continue # handled later
if key in TEXT_FIELDS:
key += '.keyword'
ops = {'<=': 'lte', '>=': 'gte', '<': 'lt', '>': 'gt'}
if partial_id:
query['must'].append({"prefix": {"claim_id": value}})
elif key in RANGE_FIELDS and isinstance(value, str) and value[0] in ops:
operator_length = 2 if value[:2] in ops else 1
operator, value = value[:operator_length], value[operator_length:]
if key == 'fee_amount':
value = str(Decimal(value)*1000)
query['must'].append({"range": {key: {ops[operator]: value}}})
elif many:
query['must'].append({"terms": {key: value}})
else:
if key == 'fee_amount':
value = str(Decimal(value)*1000)
query['must'].append({"term": {key: {"value": value}}})
elif key == 'not_channel_ids':
for channel_id in value:
query['must_not'].append({"term": {'channel_id.keyword': channel_id}})
query['must_not'].append({"term": {'_id': channel_id}})
elif key == 'channel_ids':
query['must'].append({"terms": {'channel_id.keyword': value}})
elif key == 'claim_ids':
query['must'].append({"terms": {'claim_id.keyword': value}})
elif key == 'media_types':
query['must'].append({"terms": {'media_type.keyword': value}})
elif key == 'any_languages':
query['must'].append({"terms": {'languages': clean_tags(value)}})
elif key == 'any_languages':
query['must'].append({"terms": {'languages': value}})
elif key == 'all_languages':
query['must'].extend([{"term": {'languages': tag}} for tag in value])
elif key == 'any_tags':
query['must'].append({"terms": {'tags.keyword': clean_tags(value)}})
elif key == 'all_tags':
query['must'].extend([{"term": {'tags.keyword': tag}} for tag in clean_tags(value)])
elif key == 'not_tags':
query['must_not'].extend([{"term": {'tags.keyword': tag}} for tag in clean_tags(value)])
elif key == 'not_claim_id':
query['must_not'].extend([{"term": {'claim_id.keyword': cid}} for cid in value])
elif key == 'limit_claims_per_channel':
collapse = ('channel_id.keyword', value)
if kwargs.get('has_channel_signature'):
query['must'].append({"exists": {"field": "signature"}})
if 'signature_valid' in kwargs:
query['must'].append({"term": {"is_signature_valid": bool(kwargs["signature_valid"])}})
elif 'signature_valid' in kwargs:
query.setdefault('should', [])
query["minimum_should_match"] = 1
query['should'].append({"bool": {"must_not": {"exists": {"field": "signature"}}}})
query['should'].append({"term": {"is_signature_valid": bool(kwargs["signature_valid"])}})
if 'has_source' in kwargs:
query.setdefault('should', [])
query["minimum_should_match"] = 1
is_stream_or_repost = {"terms": {"claim_type": [CLAIM_TYPES['stream'], CLAIM_TYPES['repost']]}}
query['should'].append(
{"bool": {"must": [{"match": {"has_source": kwargs['has_source']}}, is_stream_or_repost]}})
query['should'].append({"bool": {"must_not": [is_stream_or_repost]}})
query['should'].append({"bool": {"must": [{"term": {"reposted_claim_type": CLAIM_TYPES['channel']}}]}})
if kwargs.get('text'):
query['must'].append(
{"simple_query_string":
{"query": kwargs["text"], "fields": [
"claim_name^4", "channel_name^8", "title^1", "description^.5", "author^1", "tags^.5"
]}})
query = {
"_source": {"excludes": ["description", "title"]},
'query': {'bool': query},
"sort": [],
}
if "limit" in kwargs:
query["size"] = kwargs["limit"]
if 'offset' in kwargs:
query["from"] = kwargs["offset"]
if 'order_by' in kwargs:
if isinstance(kwargs["order_by"], str):
kwargs["order_by"] = [kwargs["order_by"]]
for value in kwargs['order_by']:
if 'trending_group' in value:
# fixme: trending_mixed is 0 for all records on variable decay, making sort slow.
continue
is_asc = value.startswith('^')
value = value[1:] if is_asc else value
value = REPLACEMENTS.get(value, value)
if value in TEXT_FIELDS:
value += '.keyword'
query['sort'].append({value: "asc" if is_asc else "desc"})
if collapse:
query["collapse"] = {
"field": collapse[0],
"inner_hits": {
"name": collapse[0],
"size": collapse[1],
"sort": query["sort"]
}
}
return query
@@ -27,6 +27,19 @@ INDEX_DEFAULT_SETTINGS = {
"max_chars": 10
}
},
"sd_hash": {
"fields": {
"keyword": {
"ignore_above": 96,
"type": "keyword"
}
},
"type": "text",
"index_prefixes": {
"min_chars": 1,
"max_chars": 4
}
},
"height": {"type": "integer"},
"claim_type": {"type": "byte"},
"censor_type": {"type": "byte"},
@@ -42,7 +55,7 @@ FIELDS = {
'tx_id', 'tx_nout', 'tx_position',
'short_url', 'canonical_url',
'is_controlling', 'last_take_over_height',
'public_key_bytes', 'public_key_id', 'claims_in_channel', 'channel_join_height',
'public_key_bytes', 'public_key_id', 'claims_in_channel',
'channel_id', 'signature', 'signature_digest', 'is_signature_valid',
'amount', 'effective_amount', 'support_amount',
'fee_amount', 'fee_currency',
@@ -52,19 +65,19 @@ FIELDS = {
'timestamp', 'creation_timestamp',
'duration', 'release_time',
'tags', 'languages', 'has_source', 'reposted_claim_type',
'reposted_claim_id', 'repost_count',
'reposted_claim_id', 'repost_count', 'sd_hash',
'trending_score', 'tx_num'
}
TEXT_FIELDS = {'author', 'canonical_url', 'channel_id', 'description', 'claim_id', 'censoring_channel_id',
'media_type', 'normalized_name', 'public_key_bytes', 'public_key_id', 'short_url', 'signature',
'claim_name', 'signature_digest', 'title', 'tx_id', 'fee_currency', 'reposted_claim_id',
'tags'}
'tags', 'sd_hash'}
RANGE_FIELDS = {
'height', 'creation_height', 'activation_height', 'expiration_height',
'timestamp', 'creation_timestamp', 'duration', 'release_time', 'fee_amount',
'tx_position', 'channel_join', 'repost_count', 'limit_claims_per_channel',
'tx_position', 'repost_count', 'limit_claims_per_channel',
'amount', 'effective_amount', 'support_amount',
'trending_score', 'censor_type', 'tx_num'
}
@@ -76,7 +89,10 @@ REPLACEMENTS = {
'name': 'normalized_name',
'txid': 'tx_id',
'nout': 'tx_nout',
'trending_group': 'trending_score',
'trending_mixed': 'trending_score',
'trending_global': 'trending_score',
'trending_local': 'trending_score',
'reposted': 'repost_count',
'stream_types': 'stream_type',
'media_types': 'media_type',
@@ -0,0 +1,117 @@
FAST_AR_TRENDING_SCRIPT = """
double softenLBC(double lbc) { return (Math.pow(lbc, 1.0 / 3.0)); }
double logsumexp(double x, double y)
{
double top;
if(x > y)
top = x;
else
top = y;
double result = top + Math.log(Math.exp(x-top) + Math.exp(y-top));
return(result);
}
double logdiffexp(double big, double small)
{
return big + Math.log(1.0 - Math.exp(small - big));
}
double squash(double x)
{
if(x < 0.0)
return -Math.log(1.0 - x);
else
return Math.log(x + 1.0);
}
double unsquash(double x)
{
if(x < 0.0)
return 1.0 - Math.exp(-x);
else
return Math.exp(x) - 1.0;
}
double log_to_squash(double x)
{
return logsumexp(x, 0.0);
}
double squash_to_log(double x)
{
//assert x > 0.0;
return logdiffexp(x, 0.0);
}
double squashed_add(double x, double y)
{
// squash(unsquash(x) + unsquash(y)) but avoiding overflow.
// Cases where the signs are the same
if (x < 0.0 && y < 0.0)
return -logsumexp(-x, logdiffexp(-y, 0.0));
if (x >= 0.0 && y >= 0.0)
return logsumexp(x, logdiffexp(y, 0.0));
// Where the signs differ
if (x >= 0.0 && y < 0.0)
if (Math.abs(x) >= Math.abs(y))
return logsumexp(0.0, logdiffexp(x, -y));
else
return -logsumexp(0.0, logdiffexp(-y, x));
if (x < 0.0 && y >= 0.0)
{
// Addition is commutative, hooray for new math
return squashed_add(y, x);
}
return 0.0;
}
double squashed_multiply(double x, double y)
{
// squash(unsquash(x)*unsquash(y)) but avoiding overflow.
int sign;
if(x*y >= 0.0)
sign = 1;
else
sign = -1;
return sign*logsumexp(squash_to_log(Math.abs(x))
+ squash_to_log(Math.abs(y)), 0.0);
}
// Squashed inflated units
double inflateUnits(int height) {
double timescale = 576.0; // Half life of 400 = e-folding time of a day
// by coincidence, so may as well go with it
return log_to_squash(height / timescale);
}
double spikePower(double newAmount) {
if (newAmount < 50.0) {
return(0.5);
} else if (newAmount < 85.0) {
return(newAmount / 100.0);
} else {
return(0.85);
}
}
double spikeMass(double oldAmount, double newAmount) {
double softenedChange = softenLBC(Math.abs(newAmount - oldAmount));
double changeInSoftened = Math.abs(softenLBC(newAmount) - softenLBC(oldAmount));
double power = spikePower(newAmount);
if (oldAmount > newAmount) {
-1.0 * Math.pow(changeInSoftened, power) * Math.pow(softenedChange, 1.0 - power)
} else {
Math.pow(changeInSoftened, power) * Math.pow(softenedChange, 1.0 - power)
}
}
for (i in params.src.changes) {
double units = inflateUnits(i.height);
if (ctx._source.trending_score == null) {
ctx._source.trending_score = 0.0;
}
double bigSpike = squashed_multiply(units, squash(spikeMass(i.prev_amount, i.new_amount)));
ctx._source.trending_score = squashed_add(ctx._source.trending_score, bigSpike);
}
"""
@@ -0,0 +1,55 @@
import struct
import typing
import asyncio
import logging
log = logging.getLogger(__name__)
class ElasticNotifierProtocol(asyncio.Protocol):
"""notifies the reader when ES has written updates"""
def __init__(self, listeners):
self._listeners = listeners
self.transport: typing.Optional[asyncio.Transport] = None
def connection_made(self, transport):
self.transport = transport
self._listeners.append(self)
log.info("got es notifier connection")
def connection_lost(self, exc) -> None:
self._listeners.remove(self)
self.transport = None
def send_height(self, height: int, block_hash: bytes):
log.info("notify es update '%s'", height)
self.transport.write(struct.pack(b'>Q32s', height, block_hash))
class ElasticNotifierClientProtocol(asyncio.Protocol):
"""notifies the reader when ES has written updates"""
def __init__(self, notifications: asyncio.Queue):
self.notifications = notifications
self.transport: typing.Optional[asyncio.Transport] = None
def close(self):
if self.transport and not self.transport.is_closing():
self.transport.close()
def connection_made(self, transport):
self.transport = transport
log.info("connected to es notifier")
def connection_lost(self, exc) -> None:
self.transport = None
def data_received(self, data: bytes) -> None:
try:
height, block_hash = struct.unpack(b'>Q32s', data)
except:
log.exception("failed to decode %s", (data or b'').hex())
raise
self.notifications.put_nowait((height, block_hash))
+111 -52
View File
@@ -42,8 +42,7 @@ class IndexVersionMismatch(Exception):
class SearchIndex:
VERSION = 1
def __init__(self, index_prefix: str, search_timeout=3.0, elastic_host='localhost', elastic_port=9200,
half_life=0.4, whale_threshold=10000, whale_half_life=0.99):
def __init__(self, index_prefix: str, search_timeout=3.0, elastic_host='localhost', elastic_port=9200):
self.search_timeout = search_timeout
self.sync_timeout = 600 # wont hit that 99% of the time, but can hit on a fresh import
self.search_client: Optional[AsyncElasticsearch] = None
@@ -51,14 +50,9 @@ class SearchIndex:
self.index = index_prefix + 'claims'
self.logger = class_logger(__name__, self.__class__.__name__)
self.claim_cache = LRUCache(2 ** 15)
self.short_id_cache = LRUCache(2 ** 17)
self.search_cache = LRUCache(2 ** 17)
self.resolution_cache = LRUCache(2 ** 17)
self._elastic_host = elastic_host
self._elastic_port = elastic_port
self._trending_half_life = half_life
self._trending_whale_threshold = whale_threshold
self._trending_whale_half_life = whale_half_life
async def get_index_version(self) -> int:
try:
@@ -98,10 +92,11 @@ class SearchIndex:
await self.sync_client.indices.refresh(self.index)
return acked
def stop(self):
clients = [self.sync_client, self.search_client]
async def stop(self):
clients = [c for c in (self.sync_client, self.search_client) if c is not None]
self.sync_client, self.search_client = None, None
return asyncio.ensure_future(asyncio.gather(*(client.close() for client in clients)))
if clients:
await asyncio.gather(*(client.close() for client in clients))
def delete_index(self):
return self.sync_client.indices.delete(self.index, ignore_unavailable=True)
@@ -160,21 +155,102 @@ class SearchIndex:
async def update_trending_score(self, params):
update_trending_score_script = """
double softenLBC(double lbc) { Math.pow(lbc, 1.0f / 3.0f) }
double inflateUnits(int height) {
int renormalizationPeriod = 100000;
double doublingRate = 400.0f;
Math.pow(2.0, (height % renormalizationPeriod) / doublingRate)
double softenLBC(double lbc) { return (Math.pow(lbc, 1.0 / 3.0)); }
double logsumexp(double x, double y)
{
double top;
if(x > y)
top = x;
else
top = y;
double result = top + Math.log(Math.exp(x-top) + Math.exp(y-top));
return(result);
}
double logdiffexp(double big, double small)
{
return big + Math.log(1.0 - Math.exp(small - big));
}
double squash(double x)
{
if(x < 0.0)
return -Math.log(1.0 - x);
else
return Math.log(x + 1.0);
}
double unsquash(double x)
{
if(x < 0.0)
return 1.0 - Math.exp(-x);
else
return Math.exp(x) - 1.0;
}
double log_to_squash(double x)
{
return logsumexp(x, 0.0);
}
double squash_to_log(double x)
{
//assert x > 0.0;
return logdiffexp(x, 0.0);
}
double squashed_add(double x, double y)
{
// squash(unsquash(x) + unsquash(y)) but avoiding overflow.
// Cases where the signs are the same
if (x < 0.0 && y < 0.0)
return -logsumexp(-x, logdiffexp(-y, 0.0));
if (x >= 0.0 && y >= 0.0)
return logsumexp(x, logdiffexp(y, 0.0));
// Where the signs differ
if (x >= 0.0 && y < 0.0)
if (Math.abs(x) >= Math.abs(y))
return logsumexp(0.0, logdiffexp(x, -y));
else
return -logsumexp(0.0, logdiffexp(-y, x));
if (x < 0.0 && y >= 0.0)
{
// Addition is commutative, hooray for new math
return squashed_add(y, x);
}
return 0.0;
}
double squashed_multiply(double x, double y)
{
// squash(unsquash(x)*unsquash(y)) but avoiding overflow.
int sign;
if(x*y >= 0.0)
sign = 1;
else
sign = -1;
return sign*logsumexp(squash_to_log(Math.abs(x))
+ squash_to_log(Math.abs(y)), 0.0);
}
// Squashed inflated units
double inflateUnits(int height) {
double timescale = 576.0; // Half life of 400 = e-folding time of a day
// by coincidence, so may as well go with it
return log_to_squash(height / timescale);
}
double spikePower(double newAmount) {
if (newAmount < 50.0) {
0.5
return(0.5);
} else if (newAmount < 85.0) {
newAmount / 100.0
return(newAmount / 100.0);
} else {
0.85
return(0.85);
}
}
double spikeMass(double oldAmount, double newAmount) {
double softenedChange = softenLBC(Math.abs(newAmount - oldAmount));
double changeInSoftened = Math.abs(softenLBC(newAmount) - softenLBC(oldAmount));
@@ -187,19 +263,11 @@ class SearchIndex:
}
for (i in params.src.changes) {
double units = inflateUnits(i.height);
if (i.added) {
if (ctx._source.trending_score == null) {
ctx._source.trending_score = (units * spikeMass(i.prev_amount, i.prev_amount + i.new_amount));
} else {
ctx._source.trending_score += (units * spikeMass(i.prev_amount, i.prev_amount + i.new_amount));
}
} else {
if (ctx._source.trending_score == null) {
ctx._source.trending_score = (units * spikeMass(i.prev_amount, i.prev_amount - i.new_amount));
} else {
ctx._source.trending_score += (units * spikeMass(i.prev_amount, i.prev_amount - i.new_amount));
}
if (ctx._source.trending_score == null) {
ctx._source.trending_score = 0.0;
}
double bigSpike = squashed_multiply(units, squash(spikeMass(i.prev_amount, i.new_amount)));
ctx._source.trending_score = squashed_add(ctx._source.trending_score, bigSpike);
}
"""
start = time.perf_counter()
@@ -217,9 +285,8 @@ class SearchIndex:
'changes': [
{
'height': p.height,
'added': p.added,
'prev_amount': p.prev_amount * 1E-9,
'new_amount': p.new_amount * 1E-9,
'prev_amount': p.prev_amount / 1E8,
'new_amount': p.new_amount / 1E8,
} for p in claim_updates
]
}}
@@ -260,9 +327,7 @@ class SearchIndex:
def clear_caches(self):
self.search_cache.clear()
self.short_id_cache.clear()
self.claim_cache.clear()
self.resolution_cache.clear()
async def cached_search(self, kwargs):
total_referenced = []
@@ -354,21 +419,6 @@ class SearchIndex:
for result in expand_result(filter(lambda doc: doc['found'], results["docs"])):
self.claim_cache.set(result['claim_id'], result)
async def full_id_from_short_id(self, name, short_id, channel_id=None):
key = '#'.join((channel_id or '', name, short_id))
if key not in self.short_id_cache:
query = {'name': name, 'claim_id': short_id}
if channel_id:
query['channel_id'] = channel_id
query['order_by'] = ['^channel_join']
query['signature_valid'] = True
else:
query['order_by'] = '^creation_height'
result, _, _ = await self.search(**query, limit=1)
if len(result) == 1:
result = result[0]['claim_id']
self.short_id_cache[key] = result
return self.short_id_cache.get(key, None)
async def search(self, **kwargs):
try:
@@ -510,7 +560,7 @@ def expand_query(**kwargs):
value = [item[::-1].hex() for item in value]
else:
value = value[::-1].hex()
if not many and key in ('_id', 'claim_id') and len(value) < 20:
if not many and key in ('_id', 'claim_id', 'sd_hash') and len(value) < 20:
partial_id = True
if key in ('signature_valid', 'has_source'):
continue # handled later
@@ -518,13 +568,22 @@ def expand_query(**kwargs):
key += '.keyword'
ops = {'<=': 'lte', '>=': 'gte', '<': 'lt', '>': 'gt'}
if partial_id:
query['must'].append({"prefix": {"claim_id": value}})
query['must'].append({"prefix": {key: value}})
elif key in RANGE_FIELDS and isinstance(value, str) and value[0] in ops:
operator_length = 2 if value[:2] in ops else 1
operator, value = value[:operator_length], value[operator_length:]
if key == 'fee_amount':
value = str(Decimal(value)*1000)
query['must'].append({"range": {key: {ops[operator]: value}}})
elif key in RANGE_FIELDS and isinstance(value, list) and all(v[0] in ops for v in value):
range_constraints = []
for v in value:
operator_length = 2 if v[:2] in ops else 1
operator, stripped_op_v = v[:operator_length], v[operator_length:]
if key == 'fee_amount':
stripped_op_v = str(Decimal(stripped_op_v)*1000)
range_constraints.append((operator, stripped_op_v))
query['must'].append({"range": {key: {ops[operator]: v for operator, v in range_constraints}}})
elif many:
query['must'].append({"terms": {key: value}})
else:
+400 -72
View File
@@ -1,88 +1,416 @@
import argparse
import os
import time
import signal
import json
import typing
from collections import defaultdict
import asyncio
import logging
from elasticsearch import AsyncElasticsearch
from elasticsearch.helpers import async_bulk
from lbry.wallet.server.env import Env
from lbry.wallet.server.coin import LBC
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.db.elasticsearch.search import SearchIndex, IndexVersionMismatch
from lbry.wallet.server.db.elasticsearch.constants import ALL_FIELDS
from elasticsearch import AsyncElasticsearch, NotFoundError
from elasticsearch.helpers import async_streaming_bulk
from prometheus_client import Gauge, Histogram
from lbry.schema.result import Censor
from lbry.wallet.server.db.elasticsearch.search import IndexVersionMismatch
from lbry.wallet.server.db.elasticsearch.constants import ALL_FIELDS, INDEX_DEFAULT_SETTINGS
from lbry.wallet.server.db.elasticsearch.common import expand_query
from lbry.wallet.server.db.elasticsearch.notifier import ElasticNotifierProtocol
from lbry.wallet.server.db.elasticsearch.fast_ar_trending import FAST_AR_TRENDING_SCRIPT
from lbry.wallet.server.chain_reader import BlockchainReader, HISTOGRAM_BUCKETS
from lbry.wallet.server.db.revertable import RevertableOp
from lbry.wallet.server.db.common import TrendingNotification
from lbry.wallet.server.db import DB_PREFIXES
async def get_all_claims(index_name='claims', db=None):
env = Env(LBC)
need_open = db is None
db = db or LevelDB(env)
if need_open:
await db.open_dbs()
try:
cnt = 0
async for claim in db.all_claims_producer():
yield {
'doc': {key: value for key, value in claim.items() if key in ALL_FIELDS},
'_id': claim['claim_id'],
'_index': index_name,
'_op_type': 'update',
'doc_as_upsert': True
}
cnt += 1
if cnt % 10000 == 0:
print(f"{cnt} claims sent")
finally:
if need_open:
db.close()
log = logging.getLogger()
async def make_es_index(index=None):
env = Env(LBC)
if index is None:
index = SearchIndex('', elastic_host=env.elastic_host, elastic_port=env.elastic_port)
class ElasticWriter(BlockchainReader):
VERSION = 1
prometheus_namespace = ""
block_count_metric = Gauge(
"block_count", "Number of processed blocks", namespace="elastic_sync"
)
block_update_time_metric = Histogram(
"block_time", "Block update times", namespace="elastic_sync", buckets=HISTOGRAM_BUCKETS
)
reorg_count_metric = Gauge(
"reorg_count", "Number of reorgs", namespace="elastic_sync"
)
try:
return await index.start()
except IndexVersionMismatch as err:
logging.info(
"dropping ES search index (version %s) for upgrade to version %s", err.got_version, err.expected_version
def __init__(self, env):
super().__init__(env, 'lbry-elastic-writer', thread_workers=1, thread_prefix='lbry-elastic-writer')
# self._refresh_interval = 0.1
self._task = None
self.index = self.env.es_index_prefix + 'claims'
self._elastic_host = env.elastic_host
self._elastic_port = env.elastic_port
self.sync_timeout = 1800
self.sync_client = None
self._es_info_path = os.path.join(env.db_dir, 'es_info')
self._last_wrote_height = 0
self._last_wrote_block_hash = None
self._touched_claims = set()
self._deleted_claims = set()
self._removed_during_undo = set()
self._trending = defaultdict(list)
self._advanced = True
self.synchronized = asyncio.Event()
self._listeners: typing.List[ElasticNotifierProtocol] = []
async def run_es_notifier(self, synchronized: asyncio.Event):
server = await asyncio.get_event_loop().create_server(
lambda: ElasticNotifierProtocol(self._listeners), '127.0.0.1', self.env.elastic_notifier_port
)
await index.delete_index()
await index.stop()
return await index.start()
finally:
index.stop()
self.log.info("ES notifier server listening on TCP localhost:%i", self.env.elastic_notifier_port)
synchronized.set()
async with server:
await server.serve_forever()
def notify_es_notification_listeners(self, height: int, block_hash: bytes):
for p in self._listeners:
p.send_height(height, block_hash)
self.log.info("notify listener %i", height)
async def run_sync(index_name='claims', db=None, clients=32):
env = Env(LBC)
logging.info("ES sync host: %s:%i", env.elastic_host, env.elastic_port)
es = AsyncElasticsearch([{'host': env.elastic_host, 'port': env.elastic_port}])
claim_generator = get_all_claims(index_name=index_name, db=db)
def _read_es_height(self):
info = {}
if os.path.exists(self._es_info_path):
with open(self._es_info_path, 'r') as f:
info.update(json.loads(f.read()))
self._last_wrote_height = int(info.get('height', 0))
self._last_wrote_block_hash = info.get('block_hash', None)
try:
await async_bulk(es, claim_generator, request_timeout=600)
await es.indices.refresh(index=index_name)
finally:
await es.close()
async def read_es_height(self):
await asyncio.get_event_loop().run_in_executor(self._executor, self._read_es_height)
def write_es_height(self, height: int, block_hash: str):
with open(self._es_info_path, 'w') as f:
f.write(json.dumps({'height': height, 'block_hash': block_hash}, indent=2))
self._last_wrote_height = height
self._last_wrote_block_hash = block_hash
def run_elastic_sync():
logging.basicConfig(level=logging.INFO)
logging.getLogger('aiohttp').setLevel(logging.WARNING)
logging.getLogger('elasticsearch').setLevel(logging.WARNING)
async def get_index_version(self) -> int:
try:
template = await self.sync_client.indices.get_template(self.index)
return template[self.index]['version']
except NotFoundError:
return 0
logging.info('lbry.server starting')
parser = argparse.ArgumentParser(prog="lbry-hub-elastic-sync")
# parser.add_argument("db_path", type=str)
parser.add_argument("-c", "--clients", type=int, default=32)
parser.add_argument("-b", "--blocks", type=int, default=0)
parser.add_argument("-f", "--force", default=False, action='store_true')
args = parser.parse_args()
async def set_index_version(self, version):
await self.sync_client.indices.put_template(
self.index, body={'version': version, 'index_patterns': ['ignored']}, ignore=400
)
# if not args.force and not os.path.exists(args.db_path):
# logging.info("DB path doesnt exist")
# return
async def start_index(self) -> bool:
if self.sync_client:
return False
hosts = [{'host': self._elastic_host, 'port': self._elastic_port}]
self.sync_client = AsyncElasticsearch(hosts, timeout=self.sync_timeout)
while True:
try:
await self.sync_client.cluster.health(wait_for_status='yellow')
self.log.info("ES is ready to connect to")
break
except ConnectionError:
self.log.warning("Failed to connect to Elasticsearch. Waiting for it!")
await asyncio.sleep(1)
if not args.force and not asyncio.run(make_es_index()):
logging.info("ES is already initialized")
return
asyncio.run(run_sync(clients=args.clients))
index_version = await self.get_index_version()
res = await self.sync_client.indices.create(self.index, INDEX_DEFAULT_SETTINGS, ignore=400)
acked = res.get('acknowledged', False)
if acked:
await self.set_index_version(self.VERSION)
return True
elif index_version != self.VERSION:
self.log.error("es search index has an incompatible version: %s vs %s", index_version, self.VERSION)
raise IndexVersionMismatch(index_version, self.VERSION)
else:
await self.sync_client.indices.refresh(self.index)
return False
async def stop_index(self):
if self.sync_client:
await self.sync_client.close()
self.sync_client = None
async def delete_index(self):
if self.sync_client:
return await self.sync_client.indices.delete(self.index, ignore_unavailable=True)
def update_filter_query(self, censor_type, blockdict, channels=False):
blockdict = {blocked.hex(): blocker.hex() for blocked, blocker in blockdict.items()}
if channels:
update = expand_query(channel_id__in=list(blockdict.keys()), censor_type=f"<{censor_type}")
else:
update = expand_query(claim_id__in=list(blockdict.keys()), censor_type=f"<{censor_type}")
key = 'channel_id' if channels else 'claim_id'
update['script'] = {
"source": f"ctx._source.censor_type={censor_type}; "
f"ctx._source.censoring_channel_id=params[ctx._source.{key}];",
"lang": "painless",
"params": blockdict
}
return update
async def apply_filters(self, blocked_streams, blocked_channels, filtered_streams, filtered_channels):
if filtered_streams:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.SEARCH, filtered_streams), slices=4)
await self.sync_client.indices.refresh(self.index)
if filtered_channels:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.SEARCH, filtered_channels), slices=4)
await self.sync_client.indices.refresh(self.index)
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.SEARCH, filtered_channels, True), slices=4)
await self.sync_client.indices.refresh(self.index)
if blocked_streams:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_streams), slices=4)
await self.sync_client.indices.refresh(self.index)
if blocked_channels:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_channels), slices=4)
await self.sync_client.indices.refresh(self.index)
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_channels, True), slices=4)
await self.sync_client.indices.refresh(self.index)
@staticmethod
def _upsert_claim_query(index, claim):
return {
'doc': {key: value for key, value in claim.items() if key in ALL_FIELDS},
'_id': claim['claim_id'],
'_index': index,
'_op_type': 'update',
'doc_as_upsert': True
}
@staticmethod
def _delete_claim_query(index, claim_hash: bytes):
return {
'_index': index,
'_op_type': 'delete',
'_id': claim_hash.hex()
}
@staticmethod
def _update_trending_query(index, claim_hash, notifications):
return {
'_id': claim_hash.hex(),
'_index': index,
'_op_type': 'update',
'script': {
'lang': 'painless',
'source': FAST_AR_TRENDING_SCRIPT,
'params': {'src': {
'changes': [
{
'height': notification.height,
'prev_amount': notification.prev_amount / 1E8,
'new_amount': notification.new_amount / 1E8,
} for notification in notifications
]
}}
},
}
async def _claim_producer(self):
for deleted in self._deleted_claims:
yield self._delete_claim_query(self.index, deleted)
for touched in self._touched_claims:
claim = self.db.claim_producer(touched)
if claim:
yield self._upsert_claim_query(self.index, claim)
for claim_hash, notifications in self._trending.items():
yield self._update_trending_query(self.index, claim_hash, notifications)
def advance(self, height: int):
super().advance(height)
touched_or_deleted = self.db.prefix_db.touched_or_deleted.get(height)
for k, v in self.db.prefix_db.trending_notification.iterate((height,)):
self._trending[k.claim_hash].append(TrendingNotification(k.height, v.previous_amount, v.new_amount))
if touched_or_deleted:
readded_after_reorg = self._removed_during_undo.intersection(touched_or_deleted.touched_claims)
self._deleted_claims.difference_update(readded_after_reorg)
self._touched_claims.update(touched_or_deleted.touched_claims)
self._deleted_claims.update(touched_or_deleted.deleted_claims)
self._touched_claims.difference_update(self._deleted_claims)
for to_del in touched_or_deleted.deleted_claims:
if to_del in self._trending:
self._trending.pop(to_del)
self._advanced = True
def unwind(self):
self.db.tx_counts.pop()
reverted_block_hash = self.db.coin.header_hash(self.db.headers.pop())
packed = self.db.prefix_db.undo.get(len(self.db.tx_counts), reverted_block_hash)
touched_or_deleted = None
claims_to_delete = []
# find and apply the touched_or_deleted items in the undos for the reverted blocks
assert packed, f'missing undo information for block {len(self.db.tx_counts)}'
while packed:
op, packed = RevertableOp.unpack(packed)
if op.is_delete and op.key.startswith(DB_PREFIXES.touched_or_deleted.value):
assert touched_or_deleted is None, 'only should have one match'
touched_or_deleted = self.db.prefix_db.touched_or_deleted.unpack_value(op.value)
elif op.is_delete and op.key.startswith(DB_PREFIXES.claim_to_txo.value):
v = self.db.prefix_db.claim_to_txo.unpack_value(op.value)
if v.root_tx_num == v.tx_num and v.root_tx_num > self.db.tx_counts[-1]:
claims_to_delete.append(self.db.prefix_db.claim_to_txo.unpack_key(op.key).claim_hash)
if touched_or_deleted:
self._touched_claims.update(set(touched_or_deleted.deleted_claims).union(
touched_or_deleted.touched_claims.difference(set(claims_to_delete))))
self._deleted_claims.update(claims_to_delete)
self._removed_during_undo.update(claims_to_delete)
self._advanced = True
self.log.warning("delete %i claim and upsert %i from reorg", len(self._deleted_claims), len(self._touched_claims))
async def poll_for_changes(self):
await super().poll_for_changes()
cnt = 0
success = 0
if self._advanced:
if self._touched_claims or self._deleted_claims or self._trending:
async for ok, item in async_streaming_bulk(
self.sync_client, self._claim_producer(),
raise_on_error=False):
cnt += 1
if not ok:
self.log.warning("indexing failed for an item: %s", item)
else:
success += 1
await self.sync_client.indices.refresh(self.index)
await self.db.reload_blocking_filtering_streams()
await self.apply_filters(
self.db.blocked_streams, self.db.blocked_channels, self.db.filtered_streams,
self.db.filtered_channels
)
self.write_es_height(self.db.db_height, self.db.db_tip[::-1].hex())
self.log.info("Indexing block %i done. %i/%i successful", self._last_wrote_height, success, cnt)
self._touched_claims.clear()
self._deleted_claims.clear()
self._removed_during_undo.clear()
self._trending.clear()
self._advanced = False
self.synchronized.set()
self.notify_es_notification_listeners(self._last_wrote_height, self.db.db_tip)
@property
def last_synced_height(self) -> int:
return self._last_wrote_height
async def start(self, reindex=False):
await super().start()
def _start_cancellable(run, *args):
_flag = asyncio.Event()
self.cancellable_tasks.append(asyncio.ensure_future(run(*args, _flag)))
return _flag.wait()
self.db.open_db()
await self.db.initialize_caches()
await self.read_es_height()
await self.start_index()
self.last_state = self.db.read_db_state()
await _start_cancellable(self.run_es_notifier)
if reindex or self._last_wrote_height == 0 and self.db.db_height > 0:
if self._last_wrote_height == 0:
self.log.info("running initial ES indexing of rocksdb at block height %i", self.db.db_height)
else:
self.log.info("reindex (last wrote: %i, db height: %i)", self._last_wrote_height, self.db.db_height)
await self.reindex()
await _start_cancellable(self.refresh_blocks_forever)
async def stop(self, delete_index=False):
async with self._lock:
while self.cancellable_tasks:
t = self.cancellable_tasks.pop()
if not t.done():
t.cancel()
if delete_index:
await self.delete_index()
await self.stop_index()
self._executor.shutdown(wait=True)
self._executor = None
self.shutdown_event.set()
def run(self, reindex=False):
loop = asyncio.get_event_loop()
loop.set_default_executor(self._executor)
def __exit():
raise SystemExit()
try:
loop.add_signal_handler(signal.SIGINT, __exit)
loop.add_signal_handler(signal.SIGTERM, __exit)
loop.run_until_complete(self.start(reindex=reindex))
loop.run_until_complete(self.shutdown_event.wait())
except (SystemExit, KeyboardInterrupt):
pass
finally:
loop.run_until_complete(self.stop())
async def reindex(self):
async with self._lock:
self.log.info("reindexing %i claims (estimate)", self.db.prefix_db.claim_to_txo.estimate_num_keys())
await self.delete_index()
res = await self.sync_client.indices.create(self.index, INDEX_DEFAULT_SETTINGS, ignore=400)
acked = res.get('acknowledged', False)
if acked:
await self.set_index_version(self.VERSION)
await self.sync_client.indices.refresh(self.index)
self.write_es_height(0, self.env.coin.GENESIS_HASH)
await self._sync_all_claims()
await self.sync_client.indices.refresh(self.index)
self.write_es_height(self.db.db_height, self.db.db_tip[::-1].hex())
self.notify_es_notification_listeners(self.db.db_height, self.db.db_tip)
self.log.info("finished reindexing")
async def _sync_all_claims(self, batch_size=100000):
def load_historic_trending():
notifications = self._trending
for k, v in self.db.prefix_db.trending_notification.iterate():
notifications[k.claim_hash].append(TrendingNotification(k.height, v.previous_amount, v.new_amount))
async def all_claims_producer():
async for claim in self.db.all_claims_producer(batch_size=batch_size):
yield self._upsert_claim_query(self.index, claim)
claim_hash = bytes.fromhex(claim['claim_id'])
if claim_hash in self._trending:
yield self._update_trending_query(self.index, claim_hash, self._trending.pop(claim_hash))
self._trending.clear()
self.log.info("loading about %i historic trending updates", self.db.prefix_db.trending_notification.estimate_num_keys())
await asyncio.get_event_loop().run_in_executor(self._executor, load_historic_trending)
self.log.info("loaded historic trending updates for %i claims", len(self._trending))
cnt = 0
success = 0
producer = all_claims_producer()
finished = False
try:
async for ok, item in async_streaming_bulk(self.sync_client, producer, raise_on_error=False):
cnt += 1
if not ok:
self.log.warning("indexing failed for an item: %s", item)
else:
success += 1
if cnt % batch_size == 0:
self.log.info(f"indexed {success} claims")
finished = True
await self.sync_client.indices.refresh(self.index)
self.log.info("indexed %i/%i claims", success, cnt)
finally:
if not finished:
await producer.aclose()
self.shutdown_event.set()
+145
View File
@@ -0,0 +1,145 @@
import struct
import typing
import rocksdb
from typing import Optional
from lbry.wallet.server.db import DB_PREFIXES, COLUMN_SETTINGS
from lbry.wallet.server.db.revertable import RevertableOpStack, RevertablePut, RevertableDelete
class BasePrefixDB:
"""
Base class for a revertable rocksdb database (a rocksdb db where each set of applied changes can be undone)
"""
UNDO_KEY_STRUCT = struct.Struct(b'>Q32s')
PARTIAL_UNDO_KEY_STRUCT = struct.Struct(b'>Q')
def __init__(self, path, max_open_files=64, secondary_path='', max_undo_depth: int = 200, unsafe_prefixes=None):
column_family_options = {}
for prefix in DB_PREFIXES:
settings = COLUMN_SETTINGS[prefix.value]
column_family_options[prefix.value] = rocksdb.ColumnFamilyOptions()
column_family_options[prefix.value].table_factory = rocksdb.BlockBasedTableFactory(
block_cache=rocksdb.LRUCache(settings['cache_size']),
)
self.column_families: typing.Dict[bytes, 'rocksdb.ColumnFamilyHandle'] = {}
options = rocksdb.Options(
create_if_missing=True, use_fsync=False, target_file_size_base=33554432,
max_open_files=max_open_files if not secondary_path else -1, create_missing_column_families=True
)
self._db = rocksdb.DB(
path, options, secondary_name=secondary_path, column_families=column_family_options
)
for prefix in DB_PREFIXES:
cf = self._db.get_column_family(prefix.value)
if cf is None and not secondary_path:
self._db.create_column_family(prefix.value, column_family_options[prefix.value])
cf = self._db.get_column_family(prefix.value)
self.column_families[prefix.value] = cf
self._op_stack = RevertableOpStack(self.get, unsafe_prefixes=unsafe_prefixes)
self._max_undo_depth = max_undo_depth
def unsafe_commit(self):
"""
Write staged changes to the database without keeping undo information
Changes written cannot be undone
"""
try:
if not len(self._op_stack):
return
with self._db.write_batch(sync=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
get_column_family = self.column_families.__getitem__
for staged_change in self._op_stack:
column_family = get_column_family(DB_PREFIXES(staged_change.key[:1]).value)
if staged_change.is_put:
batch_put((column_family, staged_change.key), staged_change.value)
else:
batch_delete((column_family, staged_change.key))
finally:
self._op_stack.clear()
def commit(self, height: int, block_hash: bytes):
"""
Write changes for a block height to the database and keep undo information so that the changes can be reverted
"""
undo_ops = self._op_stack.get_undo_ops()
delete_undos = []
if height > self._max_undo_depth:
delete_undos.extend(self._db.iterator(
start=DB_PREFIXES.undo.value + self.PARTIAL_UNDO_KEY_STRUCT.pack(0),
iterate_upper_bound=DB_PREFIXES.undo.value + self.PARTIAL_UNDO_KEY_STRUCT.pack(height - self._max_undo_depth),
include_value=False
))
try:
undo_c_f = self.column_families[DB_PREFIXES.undo.value]
with self._db.write_batch(sync=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
get_column_family = self.column_families.__getitem__
for staged_change in self._op_stack:
column_family = get_column_family(DB_PREFIXES(staged_change.key[:1]).value)
if staged_change.is_put:
batch_put((column_family, staged_change.key), staged_change.value)
else:
batch_delete((column_family, staged_change.key))
for undo_to_delete in delete_undos:
batch_delete((undo_c_f, undo_to_delete))
batch_put((undo_c_f, DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(height, block_hash)), undo_ops)
finally:
self._op_stack.clear()
def rollback(self, height: int, block_hash: bytes):
"""
Revert changes for a block height
"""
undo_key = DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(height, block_hash)
undo_c_f = self.column_families[DB_PREFIXES.undo.value]
undo_info = self._db.get((undo_c_f, undo_key))
self._op_stack.apply_packed_undo_ops(undo_info)
try:
with self._db.write_batch(sync=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
get_column_family = self.column_families.__getitem__
for staged_change in self._op_stack:
column_family = get_column_family(DB_PREFIXES(staged_change.key[:1]).value)
if staged_change.is_put:
batch_put((column_family, staged_change.key), staged_change.value)
else:
batch_delete((column_family, staged_change.key))
# batch_delete(undo_key)
finally:
self._op_stack.clear()
def get(self, key: bytes, fill_cache: bool = True) -> Optional[bytes]:
cf = self.column_families[key[:1]]
return self._db.get((cf, key), fill_cache=fill_cache)
def iterator(self, start: bytes, column_family: 'rocksdb.ColumnFamilyHandle' = None,
iterate_lower_bound: bytes = None, iterate_upper_bound: bytes = None,
reverse: bool = False, include_key: bool = True, include_value: bool = True,
fill_cache: bool = True, prefix_same_as_start: bool = False, auto_prefix_mode: bool = True):
return self._db.iterator(
start=start, column_family=column_family, iterate_lower_bound=iterate_lower_bound,
iterate_upper_bound=iterate_upper_bound, reverse=reverse, include_key=include_key,
include_value=include_value, fill_cache=fill_cache, prefix_same_as_start=prefix_same_as_start,
auto_prefix_mode=auto_prefix_mode
)
def close(self):
self._db.close()
def try_catch_up_with_primary(self):
self._db.try_catch_up_with_primary()
def stage_raw_put(self, key: bytes, value: bytes):
self._op_stack.append_op(RevertablePut(key, value))
def stage_raw_delete(self, key: bytes, value: bytes):
self._op_stack.append_op(RevertableDelete(key, value))
def estimate_num_keys(self, column_family: 'rocksdb.ColumnFamilyHandle' = None):
return int(self._db.get_property(b'rocksdb.estimate-num-keys', column_family).decode())
+262 -75
View File
@@ -2,9 +2,11 @@ import typing
import struct
import array
import base64
import rocksdb
import rocksdb.interfaces
from typing import Union, Tuple, NamedTuple, Optional
from lbry.wallet.server.db import DB_PREFIXES
from lbry.wallet.server.db.db import KeyValueStorage, PrefixDB
from lbry.wallet.server.db import DB_PREFIXES, COLUMN_SETTINGS
from lbry.wallet.server.db.interface import BasePrefixDB
from lbry.wallet.server.db.revertable import RevertableOpStack, RevertablePut, RevertableDelete
from lbry.schema.url import normalize_name
@@ -29,6 +31,10 @@ class PrefixRowType(type):
klass = super().__new__(cls, name, bases, kwargs)
if name != "PrefixRow":
ROW_TYPES[klass.prefix] = klass
cache_size = klass.cache_size
COLUMN_SETTINGS[klass.prefix] = {
'cache_size': cache_size,
}
return klass
@@ -37,22 +43,34 @@ class PrefixRow(metaclass=PrefixRowType):
key_struct: struct.Struct
value_struct: struct.Struct
key_part_lambdas = []
cache_size: int = 1024 * 1024 * 64
def __init__(self, db: KeyValueStorage, op_stack: RevertableOpStack):
def __init__(self, db: 'rocksdb.DB', op_stack: RevertableOpStack):
self._db = db
self._op_stack = op_stack
self._column_family = self._db.get_column_family(self.prefix)
if not self._column_family.is_valid:
raise RuntimeError('column family is not valid')
def iterate(self, prefix=None, start=None, stop=None,
reverse: bool = False, include_key: bool = True, include_value: bool = True,
fill_cache: bool = True, deserialize_key: bool = True, deserialize_value: bool = True):
def iterate(self, prefix=None, start=None, stop=None, reverse: bool = False, include_key: bool = True,
include_value: bool = True, fill_cache: bool = True, deserialize_key: bool = True,
deserialize_value: bool = True):
if not prefix and not start and not stop:
prefix = ()
if prefix is not None:
prefix = self.pack_partial_key(*prefix)
if start is not None:
start = self.pack_partial_key(*start)
if stop is not None:
stop = self.pack_partial_key(*stop)
if stop is None:
try:
stop = (int.from_bytes(prefix, byteorder='big') + 1).to_bytes(len(prefix), byteorder='big')
except OverflowError:
stop = (int.from_bytes(prefix, byteorder='big') + 1).to_bytes(len(prefix) + 1, byteorder='big')
else:
stop = self.pack_partial_key(*stop)
else:
if start is not None:
start = self.pack_partial_key(*start)
if stop is not None:
stop = self.pack_partial_key(*stop)
if deserialize_key:
key_getter = lambda k: self.unpack_key(k)
@@ -63,25 +81,39 @@ class PrefixRow(metaclass=PrefixRowType):
else:
value_getter = lambda v: v
it = self._db.iterator(
start or prefix, self._column_family, iterate_lower_bound=(start or prefix),
iterate_upper_bound=stop, reverse=reverse, include_key=include_key,
include_value=include_value, fill_cache=fill_cache, prefix_same_as_start=False
)
if include_key and include_value:
for k, v in self._db.iterator(prefix=prefix, start=start, stop=stop, reverse=reverse,
fill_cache=fill_cache):
yield key_getter(k), value_getter(v)
for k, v in it:
yield key_getter(k[1]), value_getter(v)
elif include_key:
for k in self._db.iterator(prefix=prefix, start=start, stop=stop, reverse=reverse, include_value=False,
fill_cache=fill_cache):
yield key_getter(k)
for k in it:
yield key_getter(k[1])
elif include_value:
for v in self._db.iterator(prefix=prefix, start=start, stop=stop, reverse=reverse, include_key=False,
fill_cache=fill_cache):
for v in it:
yield value_getter(v)
else:
for _ in self._db.iterator(prefix=prefix, start=start, stop=stop, reverse=reverse, include_key=False,
include_value=False, fill_cache=fill_cache):
for _ in it:
yield None
def get(self, *key_args, fill_cache=True, deserialize_value=True):
v = self._db.get(self.pack_key(*key_args), fill_cache=fill_cache)
v = self._db.get((self._column_family, self.pack_key(*key_args)), fill_cache=fill_cache)
if v:
return v if not deserialize_value else self.unpack_value(v)
def get_pending(self, *key_args, fill_cache=True, deserialize_value=True):
packed_key = self.pack_key(*key_args)
last_op = self._op_stack.get_last_op_for_key(packed_key)
if last_op:
if last_op.is_put:
return last_op.value if not deserialize_value else self.unpack_value(last_op.value)
else: # it's a delete
return
v = self._db.get((self._column_family, packed_key), fill_cache=fill_cache)
if v:
return v if not deserialize_value else self.unpack_value(v)
@@ -105,7 +137,7 @@ class PrefixRow(metaclass=PrefixRowType):
@classmethod
def unpack_key(cls, key: bytes):
assert key[:1] == cls.prefix
assert key[:1] == cls.prefix, f"prefix should be {cls.prefix}, got {key[:1]}"
return cls.key_struct.unpack(key[1:])
@classmethod
@@ -116,6 +148,9 @@ class PrefixRow(metaclass=PrefixRowType):
def unpack_item(cls, key: bytes, value: bytes):
return cls.unpack_key(key), cls.unpack_value(value)
def estimate_num_keys(self) -> int:
return int(self._db.get_property(b'rocksdb.estimate-num-keys', self._column_family).decode())
class UTXOKey(NamedTuple):
hashX: bytes
@@ -169,6 +204,14 @@ class BlockHashValue(NamedTuple):
return f"{self.__class__.__name__}(block_hash={self.block_hash.hex()})"
class BlockTxsKey(NamedTuple):
height: int
class BlockTxsValue(NamedTuple):
tx_hashes: typing.List[bytes]
class TxCountKey(NamedTuple):
height: int
@@ -185,14 +228,14 @@ class TxHashValue(NamedTuple):
tx_hash: bytes
def __str__(self):
return f"{self.__class__.__name__}(tx_hash={self.tx_hash.hex()})"
return f"{self.__class__.__name__}(tx_hash={self.tx_hash[::-1].hex()})"
class TxNumKey(NamedTuple):
tx_hash: bytes
def __str__(self):
return f"{self.__class__.__name__}(tx_hash={self.tx_hash.hex()})"
return f"{self.__class__.__name__}(tx_hash={self.tx_hash[::-1].hex()})"
class TxNumValue(NamedTuple):
@@ -203,14 +246,14 @@ class TxKey(NamedTuple):
tx_hash: bytes
def __str__(self):
return f"{self.__class__.__name__}(tx_hash={self.tx_hash.hex()})"
return f"{self.__class__.__name__}(tx_hash={self.tx_hash[::-1].hex()})"
class TxValue(NamedTuple):
raw_tx: bytes
def __str__(self):
return f"{self.__class__.__name__}(raw_tx={base64.b64encode(self.raw_tx)})"
return f"{self.__class__.__name__}(raw_tx={base64.b64encode(self.raw_tx).decode()})"
class BlockHeaderKey(NamedTuple):
@@ -514,6 +557,7 @@ class DBState(typing.NamedTuple):
hist_flush_count: int
comp_flush_count: int
comp_cursor: int
es_sync_height: int
class ActiveAmountPrefixRow(PrefixRow):
@@ -528,6 +572,7 @@ class ActiveAmountPrefixRow(PrefixRow):
struct.Struct(b'>20sBLL').pack,
struct.Struct(b'>20sBLLH').pack
]
cache_size = 1024 * 1024 * 128
@classmethod
def pack_key(cls, claim_hash: bytes, txo_type: int, activation_height: int, tx_num: int, position: int):
@@ -558,6 +603,7 @@ class ClaimToTXOPrefixRow(PrefixRow):
lambda: b'',
struct.Struct(b'>20s').pack
]
cache_size = 1024 * 1024 * 128
@classmethod
def pack_key(cls, claim_hash: bytes):
@@ -597,6 +643,7 @@ class TXOToClaimPrefixRow(PrefixRow):
prefix = DB_PREFIXES.txo_to_claim.value
key_struct = struct.Struct(b'>LH')
value_struct = struct.Struct(b'>20s')
cache_size = 1024 * 1024 * 128
@classmethod
def pack_key(cls, tx_num: int, position: int):
@@ -991,6 +1038,7 @@ class EffectiveAmountPrefixRow(PrefixRow):
shortid_key_helper(b'>QL'),
shortid_key_helper(b'>QLH'),
]
cache_size = 1024 * 1024 * 128
@classmethod
def pack_key(cls, name: str, effective_amount: int, tx_num: int, position: int):
@@ -1021,6 +1069,7 @@ class EffectiveAmountPrefixRow(PrefixRow):
class RepostPrefixRow(PrefixRow):
prefix = DB_PREFIXES.repost.value
key_struct = struct.Struct(b'>20s')
key_part_lambdas = [
lambda: b'',
@@ -1029,13 +1078,11 @@ class RepostPrefixRow(PrefixRow):
@classmethod
def pack_key(cls, claim_hash: bytes):
return cls.prefix + claim_hash
return super().pack_key(claim_hash)
@classmethod
def unpack_key(cls, key: bytes) -> RepostKey:
assert key[:1] == cls.prefix
assert len(key) == 21
return RepostKey(key[1:])
return RepostKey(*super().unpack_key(key))
@classmethod
def pack_value(cls, reposted_claim_hash: bytes) -> bytes:
@@ -1082,24 +1129,28 @@ class RepostedPrefixRow(PrefixRow):
return cls.pack_key(reposted_claim_hash, tx_num, position), cls.pack_value(claim_hash)
class UndoKey(NamedTuple):
height: int
block_hash: bytes
class UndoPrefixRow(PrefixRow):
prefix = DB_PREFIXES.undo.value
key_struct = struct.Struct(b'>Q')
key_struct = struct.Struct(b'>Q32s')
key_part_lambdas = [
lambda: b'',
struct.Struct(b'>Q').pack
struct.Struct(b'>Q').pack,
struct.Struct(b'>Q32s').pack
]
@classmethod
def pack_key(cls, height: int):
return super().pack_key(height)
def pack_key(cls, height: int, block_hash: bytes):
return super().pack_key(height, block_hash)
@classmethod
def unpack_key(cls, key: bytes) -> int:
assert key[:1] == cls.prefix
height, = cls.key_struct.unpack(key[1:])
return height
def unpack_key(cls, key: bytes) -> UndoKey:
return UndoKey(*super().unpack_key(key))
@classmethod
def pack_value(cls, undo_ops: bytes) -> bytes:
@@ -1110,8 +1161,8 @@ class UndoPrefixRow(PrefixRow):
return data
@classmethod
def pack_item(cls, height: int, undo_ops: bytes):
return cls.pack_key(height), cls.pack_value(undo_ops)
def pack_item(cls, height: int, block_hash: bytes, undo_ops: bytes):
return cls.pack_key(height, block_hash), cls.pack_value(undo_ops)
class BlockHashPrefixRow(PrefixRow):
@@ -1401,7 +1452,7 @@ class HashXHistoryPrefixRow(PrefixRow):
class TouchedOrDeletedPrefixRow(PrefixRow):
prefix = DB_PREFIXES.claim_diff.value
prefix = DB_PREFIXES.touched_or_deleted.value
key_struct = struct.Struct(b'>L')
value_struct = struct.Struct(b'>LL')
key_part_lambdas = [
@@ -1449,7 +1500,7 @@ class ChannelCountPrefixRow(PrefixRow):
]
@classmethod
def pack_key(cls, channel_hash: int):
def pack_key(cls, channel_hash: bytes):
return super().pack_key(channel_hash)
@classmethod
@@ -1501,7 +1552,7 @@ class SupportAmountPrefixRow(PrefixRow):
class DBStatePrefixRow(PrefixRow):
prefix = DB_PREFIXES.db_state.value
value_struct = struct.Struct(b'>32sLL32sLLBBlll')
value_struct = struct.Struct(b'>32sLL32sLLBBlllL')
key_struct = struct.Struct(b'')
key_part_lambdas = [
@@ -1519,62 +1570,194 @@ class DBStatePrefixRow(PrefixRow):
@classmethod
def pack_value(cls, genesis: bytes, height: int, tx_count: int, tip: bytes, utxo_flush_count: int, wall_time: int,
first_sync: bool, db_version: int, hist_flush_count: int, comp_flush_count: int,
comp_cursor: int) -> bytes:
comp_cursor: int, es_sync_height: int) -> bytes:
return super().pack_value(
genesis, height, tx_count, tip, utxo_flush_count,
wall_time, 1 if first_sync else 0, db_version, hist_flush_count,
comp_flush_count, comp_cursor
comp_flush_count, comp_cursor, es_sync_height
)
@classmethod
def unpack_value(cls, data: bytes) -> DBState:
if len(data) == 94:
# TODO: delete this after making a new snapshot - 10/20/21
# migrate in the es_sync_height if it doesnt exist
data += data[32:36]
return DBState(*super().unpack_value(data))
@classmethod
def pack_item(cls, genesis: bytes, height: int, tx_count: int, tip: bytes, utxo_flush_count: int, wall_time: int,
first_sync: bool, db_version: int, hist_flush_count: int, comp_flush_count: int,
comp_cursor: int):
comp_cursor: int, es_sync_height: int):
return cls.pack_key(), cls.pack_value(
genesis, height, tx_count, tip, utxo_flush_count, wall_time, first_sync, db_version, hist_flush_count,
comp_flush_count, comp_cursor
comp_flush_count, comp_cursor, es_sync_height
)
class LevelDBStore(KeyValueStorage):
def __init__(self, path: str, cache_mb: int, max_open_files: int):
import plyvel
self.db = plyvel.DB(
path, create_if_missing=True, max_open_files=max_open_files,
lru_cache_size=cache_mb * 1024 * 1024, write_buffer_size=64 * 1024 * 1024,
max_file_size=1024 * 1024 * 64, bloom_filter_bits=32
)
class BlockTxsPrefixRow(PrefixRow):
prefix = DB_PREFIXES.block_tx.value
key_struct = struct.Struct(b'>L')
key_part_lambdas = [
lambda: b'',
struct.Struct(b'>L').pack
]
def get(self, key: bytes, fill_cache: bool = True) -> Optional[bytes]:
return self.db.get(key, fill_cache=fill_cache)
@classmethod
def pack_key(cls, height: int):
return super().pack_key(height)
def iterator(self, reverse=False, start=None, stop=None, include_start=True, include_stop=False, prefix=None,
include_key=True, include_value=True, fill_cache=True):
return self.db.iterator(
reverse=reverse, start=start, stop=stop, include_start=include_start, include_stop=include_stop,
prefix=prefix, include_key=include_key, include_value=include_value, fill_cache=fill_cache
)
@classmethod
def unpack_key(cls, key: bytes) -> BlockTxsKey:
return BlockTxsKey(*super().unpack_key(key))
def write_batch(self, transaction: bool = False, sync: bool = False):
return self.db.write_batch(transaction=transaction, sync=sync)
@classmethod
def pack_value(cls, tx_hashes: typing.List[bytes]) -> bytes:
assert all(len(tx_hash) == 32 for tx_hash in tx_hashes)
return b''.join(tx_hashes)
def close(self):
return self.db.close()
@classmethod
def unpack_value(cls, data: bytes) -> BlockTxsValue:
return BlockTxsValue([data[i*32:(i+1)*32] for i in range(len(data) // 32)])
@property
def closed(self) -> bool:
return self.db.closed
@classmethod
def pack_item(cls, height, tx_hashes):
return cls.pack_key(height), cls.pack_value(tx_hashes)
class HubDB(PrefixDB):
def __init__(self, path: str, cache_mb: int = 128, reorg_limit: int = 200, max_open_files: int = 512,
unsafe_prefixes: Optional[typing.Set[bytes]] = None):
db = LevelDBStore(path, cache_mb, max_open_files)
super().__init__(db, reorg_limit, unsafe_prefixes=unsafe_prefixes)
class MempoolTxKey(NamedTuple):
tx_hash: bytes
def __str__(self):
return f"{self.__class__.__name__}(tx_hash={self.tx_hash[::-1].hex()})"
class MempoolTxValue(NamedTuple):
raw_tx: bytes
def __str__(self):
return f"{self.__class__.__name__}(raw_tx={base64.b64encode(self.raw_tx).decode()})"
class MempoolTXPrefixRow(PrefixRow):
prefix = DB_PREFIXES.mempool_tx.value
key_struct = struct.Struct(b'>32s')
key_part_lambdas = [
lambda: b'',
struct.Struct(b'>32s').pack
]
@classmethod
def pack_key(cls, tx_hash: bytes) -> bytes:
return super().pack_key(tx_hash)
@classmethod
def unpack_key(cls, tx_hash: bytes) -> MempoolTxKey:
return MempoolTxKey(*super().unpack_key(tx_hash))
@classmethod
def pack_value(cls, tx: bytes) -> bytes:
return tx
@classmethod
def unpack_value(cls, data: bytes) -> MempoolTxValue:
return MempoolTxValue(data)
@classmethod
def pack_item(cls, tx_hash: bytes, raw_tx: bytes):
return cls.pack_key(tx_hash), cls.pack_value(raw_tx)
class TrendingNotificationKey(typing.NamedTuple):
height: int
claim_hash: bytes
class TrendingNotificationValue(typing.NamedTuple):
previous_amount: int
new_amount: int
class TrendingNotificationPrefixRow(PrefixRow):
prefix = DB_PREFIXES.trending_notifications.value
key_struct = struct.Struct(b'>L20s')
value_struct = struct.Struct(b'>QQ')
key_part_lambdas = [
lambda: b'',
struct.Struct(b'>L').pack,
struct.Struct(b'>L20s').pack
]
@classmethod
def pack_key(cls, height: int, claim_hash: bytes):
return super().pack_key(height, claim_hash)
@classmethod
def unpack_key(cls, key: bytes) -> TrendingNotificationKey:
return TrendingNotificationKey(*super().unpack_key(key))
@classmethod
def pack_value(cls, previous_amount: int, new_amount: int) -> bytes:
return super().pack_value(previous_amount, new_amount)
@classmethod
def unpack_value(cls, data: bytes) -> TrendingNotificationValue:
return TrendingNotificationValue(*super().unpack_value(data))
@classmethod
def pack_item(cls, height, claim_hash, previous_amount, new_amount):
return cls.pack_key(height, claim_hash), cls.pack_value(previous_amount, new_amount)
class TouchedHashXKey(NamedTuple):
height: int
class TouchedHashXValue(NamedTuple):
touched_hashXs: typing.List[bytes]
def __str__(self):
return f"{self.__class__.__name__}(touched_hashXs=[{', '.join(map(lambda x: x.hex(), self.touched_hashXs))}])"
class TouchedHashXPrefixRow(PrefixRow):
prefix = DB_PREFIXES.touched_hashX.value
key_struct = struct.Struct(b'>L')
key_part_lambdas = [
lambda: b'',
struct.Struct(b'>L').pack
]
@classmethod
def pack_key(cls, height: int):
return super().pack_key(height)
@classmethod
def unpack_key(cls, key: bytes) -> TouchedHashXKey:
return TouchedHashXKey(*super().unpack_key(key))
@classmethod
def pack_value(cls, touched: typing.List[bytes]) -> bytes:
assert all(map(lambda item: len(item) == 11, touched))
return b''.join(touched)
@classmethod
def unpack_value(cls, data: bytes) -> TouchedHashXValue:
return TouchedHashXValue([data[idx*11:(idx*11)+11] for idx in range(len(data) // 11)])
@classmethod
def pack_item(cls, height: int, touched: typing.List[bytes]):
return cls.pack_key(height), cls.pack_value(touched)
class PrefixDB(BasePrefixDB):
def __init__(self, path: str, cache_mb: int = 128, reorg_limit: int = 200, max_open_files: int = 64,
secondary_path: str = '', unsafe_prefixes: Optional[typing.Set[bytes]] = None):
super().__init__(path, max_open_files=max_open_files, secondary_path=secondary_path,
max_undo_depth=reorg_limit, unsafe_prefixes=unsafe_prefixes)
db = self._db
self.claim_to_support = ClaimToSupportPrefixRow(db, self._op_stack)
self.support_to_claim = SupportToClaimPrefixRow(db, self._op_stack)
self.claim_to_txo = ClaimToTXOPrefixRow(db, self._op_stack)
@@ -1604,6 +1787,10 @@ class HubDB(PrefixDB):
self.channel_count = ChannelCountPrefixRow(db, self._op_stack)
self.db_state = DBStatePrefixRow(db, self._op_stack)
self.support_amount = SupportAmountPrefixRow(db, self._op_stack)
self.block_txs = BlockTxsPrefixRow(db, self._op_stack)
self.mempool_tx = MempoolTXPrefixRow(db, self._op_stack)
self.trending_notification = TrendingNotificationPrefixRow(db, self._op_stack)
self.touched_hashX = TouchedHashXPrefixRow(db, self._op_stack)
def auto_decode_item(key: bytes, value: bytes) -> Union[Tuple[NamedTuple, NamedTuple], Tuple[bytes, bytes]]:
+30 -2
View File
@@ -83,11 +83,26 @@ class OpStackIntegrity(Exception):
class RevertableOpStack:
def __init__(self, get_fn: Callable[[bytes], Optional[bytes]], unsafe_prefixes=None):
"""
This represents a sequence of revertable puts and deletes to a key-value database that checks for integrity
violations when applying the puts and deletes. The integrity checks assure that keys that do not exist
are not deleted, and that when keys are deleted the current value is correctly known so that the delete
may be undone. When putting values, the integrity checks assure that existing values are not overwritten
without first being deleted. Updates are performed by applying a delete op for the old value and a put op
for the new value.
:param get_fn: getter function from an object implementing `KeyValueStorage`
:param unsafe_prefixes: optional set of prefixes to ignore integrity errors for, violations are still logged
"""
self._get = get_fn
self._items = defaultdict(list)
self._unsafe_prefixes = unsafe_prefixes or set()
def append_op(self, op: RevertableOp):
"""
Apply a put or delete op, checking that it introduces no integrity errors
"""
inverted = op.invert()
if self._items[op.key] and inverted == self._items[op.key][-1]:
self._items[op.key].pop() # if the new op is the inverse of the last op, we can safely null both
@@ -106,19 +121,22 @@ class RevertableOpStack:
elif op.is_delete and has_stored_val and stored_val != op.value and not will_delete_existing_stored:
# there is a value and we're not deleting it in this op
# check that a delete for the stored value is in the stack
raise OpStackIntegrity(f"delete {op}")
raise OpStackIntegrity(f"db op tries to delete with incorrect existing value {op}")
elif op.is_delete and not has_stored_val:
raise OpStackIntegrity(f"db op tries to delete nonexistent key: {op}")
elif op.is_delete and stored_val != op.value:
raise OpStackIntegrity(f"db op tries to delete with incorrect value: {op}")
except OpStackIntegrity as err:
if op.key[:1] in self._unsafe_prefixes:
log.error(f"skipping over integrity error: {err}")
log.debug(f"skipping over integrity error: {err}")
else:
raise err
self._items[op.key].append(op)
def extend_ops(self, ops: Iterable[RevertableOp]):
"""
Apply a sequence of put or delete ops, checking that they introduce no integrity errors
"""
for op in ops:
self.append_op(op)
@@ -139,9 +157,19 @@ class RevertableOpStack:
yield op
def get_undo_ops(self) -> bytes:
"""
Get the serialized bytes to undo all of the changes made by the pending ops
"""
return b''.join(op.invert().pack() for op in reversed(self))
def apply_packed_undo_ops(self, packed: bytes):
"""
Unpack and apply a sequence of undo ops from serialized undo bytes
"""
while packed:
op, packed = RevertableOp.unpack(packed)
self.append_op(op)
def get_last_op_for_key(self, key: bytes) -> Optional[RevertableOp]:
if key in self._items and self._items[key]:
return self._items[key][-1]
+182 -65
View File
@@ -13,7 +13,7 @@ from collections import namedtuple
from ipaddress import ip_address
from lbry.wallet.server.util import class_logger
from lbry.wallet.server.coin import Coin
from lbry.wallet.server.coin import Coin, LBC, LBCTestNet, LBCRegTest
import lbry.wallet.server.util as lib_util
@@ -28,77 +28,89 @@ class Env:
class Error(Exception):
pass
def __init__(self, coin=None):
def __init__(self, coin=None, db_dir=None, daemon_url=None, host=None, rpc_host=None, elastic_host=None,
elastic_port=None, loop_policy=None, max_query_workers=None, websocket_host=None, websocket_port=None,
chain=None, es_index_prefix=None, cache_MB=None, reorg_limit=None, tcp_port=None,
udp_port=None, ssl_port=None, ssl_certfile=None, ssl_keyfile=None, rpc_port=None,
prometheus_port=None, max_subscriptions=None, banner_file=None, anon_logs=None, log_sessions=None,
allow_lan_udp=None, cache_all_tx_hashes=None, cache_all_claim_txos=None, country=None,
payment_address=None, donation_address=None, max_send=None, max_receive=None, max_sessions=None,
session_timeout=None, drop_client=None, description=None, daily_fee=None,
database_query_timeout=None, db_max_open_files=64, elastic_notifier_port=None,
blocking_channel_ids=None, filtering_channel_ids=None, peer_hubs=None, peer_announce=None):
self.logger = class_logger(__name__, self.__class__.__name__)
self.allow_root = self.boolean('ALLOW_ROOT', False)
self.host = self.default('HOST', 'localhost')
self.rpc_host = self.default('RPC_HOST', 'localhost')
self.elastic_host = self.default('ELASTIC_HOST', 'localhost')
self.elastic_port = self.integer('ELASTIC_PORT', 9200)
self.loop_policy = self.set_event_loop_policy()
self.obsolete(['UTXO_MB', 'HIST_MB', 'NETWORK'])
self.db_dir = self.required('DB_DIRECTORY')
self.db_engine = self.default('DB_ENGINE', 'leveldb')
# self.trending_algorithms = [
# trending for trending in set(self.default('TRENDING_ALGORITHMS', 'zscore').split(' ')) if trending
# ]
self.trending_half_life = math.log2(0.1 ** (1 / (3 + self.integer('TRENDING_DECAY_RATE', 48)))) + 1
self.trending_whale_half_life = math.log2(0.1 ** (1 / (3 + self.integer('TRENDING_WHALE_DECAY_RATE', 24)))) + 1
self.trending_whale_threshold = float(self.integer('TRENDING_WHALE_THRESHOLD', 10000)) * 1E8
self.max_query_workers = self.integer('MAX_QUERY_WORKERS', 4)
self.individual_tag_indexes = self.boolean('INDIVIDUAL_TAG_INDEXES', True)
self.track_metrics = self.boolean('TRACK_METRICS', False)
self.websocket_host = self.default('WEBSOCKET_HOST', self.host)
self.websocket_port = self.integer('WEBSOCKET_PORT', None)
self.daemon_url = self.required('DAEMON_URL')
self.db_dir = db_dir if db_dir is not None else self.required('DB_DIRECTORY')
self.daemon_url = daemon_url if daemon_url is not None else self.required('DAEMON_URL')
self.db_max_open_files = db_max_open_files
self.host = host if host is not None else self.default('HOST', 'localhost')
self.rpc_host = rpc_host if rpc_host is not None else self.default('RPC_HOST', 'localhost')
self.elastic_host = elastic_host if elastic_host is not None else self.default('ELASTIC_HOST', 'localhost')
self.elastic_port = elastic_port if elastic_port is not None else self.integer('ELASTIC_PORT', 9200)
self.elastic_notifier_port = elastic_notifier_port if elastic_notifier_port is not None else self.integer('ELASTIC_NOTIFIER_PORT', 19080)
self.loop_policy = self.set_event_loop_policy(
loop_policy if loop_policy is not None else self.default('EVENT_LOOP_POLICY', None)
)
self.obsolete(['UTXO_MB', 'HIST_MB', 'NETWORK'])
self.max_query_workers = max_query_workers if max_query_workers is not None else self.integer('MAX_QUERY_WORKERS', 4)
self.websocket_host = websocket_host if websocket_host is not None else self.default('WEBSOCKET_HOST', self.host)
self.websocket_port = websocket_port if websocket_port is not None else self.integer('WEBSOCKET_PORT', None)
if coin is not None:
assert issubclass(coin, Coin)
self.coin = coin
else:
coin_name = self.required('COIN').strip()
network = self.default('NET', 'mainnet').strip()
self.coin = Coin.lookup_coin_class(coin_name, network)
self.es_index_prefix = self.default('ES_INDEX_PREFIX', '')
self.es_mode = self.default('ES_MODE', 'writer')
self.cache_MB = self.integer('CACHE_MB', 4096)
self.reorg_limit = self.integer('REORG_LIMIT', self.coin.REORG_LIMIT)
chain = chain if chain is not None else self.default('NET', 'mainnet').strip().lower()
if chain == 'mainnet':
self.coin = LBC
elif chain == 'testnet':
self.coin = LBCTestNet
else:
self.coin = LBCRegTest
self.es_index_prefix = es_index_prefix if es_index_prefix is not None else self.default('ES_INDEX_PREFIX', '')
self.cache_MB = cache_MB if cache_MB is not None else self.integer('CACHE_MB', 1024)
self.reorg_limit = reorg_limit if reorg_limit is not None else self.integer('REORG_LIMIT', self.coin.REORG_LIMIT)
# Server stuff
self.tcp_port = self.integer('TCP_PORT', None)
self.udp_port = self.integer('UDP_PORT', self.tcp_port)
self.ssl_port = self.integer('SSL_PORT', None)
self.tcp_port = tcp_port if tcp_port is not None else self.integer('TCP_PORT', None)
self.udp_port = udp_port if udp_port is not None else self.integer('UDP_PORT', self.tcp_port)
self.ssl_port = ssl_port if ssl_port is not None else self.integer('SSL_PORT', None)
if self.ssl_port:
self.ssl_certfile = self.required('SSL_CERTFILE')
self.ssl_keyfile = self.required('SSL_KEYFILE')
self.rpc_port = self.integer('RPC_PORT', 8000)
self.prometheus_port = self.integer('PROMETHEUS_PORT', 0)
self.max_subscriptions = self.integer('MAX_SUBSCRIPTIONS', 10000)
self.banner_file = self.default('BANNER_FILE', None)
self.tor_banner_file = self.default('TOR_BANNER_FILE', self.banner_file)
self.anon_logs = self.boolean('ANON_LOGS', False)
self.log_sessions = self.integer('LOG_SESSIONS', 3600)
self.allow_lan_udp = self.boolean('ALLOW_LAN_UDP', False)
self.country = self.default('COUNTRY', 'US')
self.ssl_certfile = ssl_certfile if ssl_certfile is not None else self.required('SSL_CERTFILE')
self.ssl_keyfile = ssl_keyfile if ssl_keyfile is not None else self.required('SSL_KEYFILE')
self.rpc_port = rpc_port if rpc_port is not None else self.integer('RPC_PORT', 8000)
self.prometheus_port = prometheus_port if prometheus_port is not None else self.integer('PROMETHEUS_PORT', 0)
self.max_subscriptions = max_subscriptions if max_subscriptions is not None else self.integer('MAX_SUBSCRIPTIONS', 10000)
self.banner_file = banner_file if banner_file is not None else self.default('BANNER_FILE', None)
# self.tor_banner_file = self.default('TOR_BANNER_FILE', self.banner_file)
self.anon_logs = anon_logs if anon_logs is not None else self.boolean('ANON_LOGS', False)
self.log_sessions = log_sessions if log_sessions is not None else self.integer('LOG_SESSIONS', 3600)
self.allow_lan_udp = allow_lan_udp if allow_lan_udp is not None else self.boolean('ALLOW_LAN_UDP', False)
self.cache_all_tx_hashes = cache_all_tx_hashes if cache_all_tx_hashes is not None else self.boolean('CACHE_ALL_TX_HASHES', False)
self.cache_all_claim_txos = cache_all_claim_txos if cache_all_claim_txos is not None else self.boolean('CACHE_ALL_CLAIM_TXOS', False)
self.country = country if country is not None else self.default('COUNTRY', 'US')
# Peer discovery
self.peer_discovery = self.peer_discovery_enum()
self.peer_announce = self.boolean('PEER_ANNOUNCE', True)
self.peer_hubs = self.extract_peer_hubs()
self.force_proxy = self.boolean('FORCE_PROXY', False)
self.tor_proxy_host = self.default('TOR_PROXY_HOST', 'localhost')
self.tor_proxy_port = self.integer('TOR_PROXY_PORT', None)
self.peer_announce = peer_announce if peer_announce is not None else self.boolean('PEER_ANNOUNCE', True)
if peer_hubs is not None:
self.peer_hubs = [p.strip("") for p in peer_hubs.split(",")]
else:
self.peer_hubs = self.extract_peer_hubs()
# self.tor_proxy_host = self.default('TOR_PROXY_HOST', 'localhost')
# self.tor_proxy_port = self.integer('TOR_PROXY_PORT', None)
# The electrum client takes the empty string as unspecified
self.payment_address = self.default('PAYMENT_ADDRESS', '')
self.donation_address = self.default('DONATION_ADDRESS', '')
self.payment_address = payment_address if payment_address is not None else self.default('PAYMENT_ADDRESS', '')
self.donation_address = donation_address if donation_address is not None else self.default('DONATION_ADDRESS', '')
# Server limits to help prevent DoS
self.max_send = self.integer('MAX_SEND', 1000000)
self.max_receive = self.integer('MAX_RECEIVE', 1000000)
self.max_subs = self.integer('MAX_SUBS', 250000)
self.max_sessions = self.sane_max_sessions()
self.max_session_subs = self.integer('MAX_SESSION_SUBS', 50000)
self.session_timeout = self.integer('SESSION_TIMEOUT', 600)
self.drop_client = self.custom("DROP_CLIENT", None, re.compile)
self.description = self.default('DESCRIPTION', '')
self.daily_fee = self.string_amount('DAILY_FEE', '0')
self.max_send = max_send if max_send is not None else self.integer('MAX_SEND', 1000000)
self.max_receive = max_receive if max_receive is not None else self.integer('MAX_RECEIVE', 1000000)
# self.max_subs = self.integer('MAX_SUBS', 250000)
self.max_sessions = max_sessions if max_sessions is not None else self.sane_max_sessions()
# self.max_session_subs = self.integer('MAX_SESSION_SUBS', 50000)
self.session_timeout = session_timeout if session_timeout is not None else self.integer('SESSION_TIMEOUT', 600)
self.drop_client = drop_client if drop_client is not None else self.custom("DROP_CLIENT", None, re.compile)
self.description = description if description is not None else self.default('DESCRIPTION', '')
self.daily_fee = daily_fee if daily_fee is not None else self.string_amount('DAILY_FEE', '0')
# Identities
clearnet_identity = self.clearnet_identity()
@@ -106,7 +118,12 @@ class Env:
self.identities = [identity
for identity in (clearnet_identity, tor_identity)
if identity is not None]
self.database_query_timeout = float(self.integer('QUERY_TIMEOUT_MS', 3000)) / 1000.0
self.database_query_timeout = database_query_timeout if database_query_timeout is not None else \
(float(self.integer('QUERY_TIMEOUT_MS', 10000)) / 1000.0)
# Filtering / Blocking
self.blocking_channel_ids = blocking_channel_ids if blocking_channel_ids is not None else self.default('BLOCKING_CHANNEL_IDS', '').split(' ')
self.filtering_channel_ids = filtering_channel_ids if filtering_channel_ids is not None else self.default('FILTERING_CHANNEL_IDS', '').split(' ')
@classmethod
def default(cls, envvar, default):
@@ -158,9 +175,9 @@ class Env:
if bad:
raise cls.Error(f'remove obsolete environment variables {bad}')
def set_event_loop_policy(self):
policy_name = self.default('EVENT_LOOP_POLICY', None)
if not policy_name:
@classmethod
def set_event_loop_policy(cls, policy_name: str = None):
if not policy_name or policy_name == 'default':
import asyncio
return asyncio.get_event_loop_policy()
elif policy_name == 'uvloop':
@@ -169,7 +186,7 @@ class Env:
loop_policy = uvloop.EventLoopPolicy()
asyncio.set_event_loop_policy(loop_policy)
return loop_policy
raise self.Error(f'unknown event loop policy "{policy_name}"')
raise cls.Error(f'unknown event loop policy "{policy_name}"')
def cs_host(self, *, for_rpc):
"""Returns the 'host' argument to pass to asyncio's create_server
@@ -278,3 +295,103 @@ class Env:
def extract_peer_hubs(self):
return [hub.strip() for hub in self.default('PEER_HUBS', '').split(',') if hub.strip()]
@classmethod
def contribute_to_arg_parser(cls, parser):
parser.add_argument('--db_dir', type=str, help='path of the directory containing lbry-leveldb',
default=cls.default('DB_DIRECTORY', None))
parser.add_argument('--daemon_url',
help='URL for rpc from lbrycrd, <rpcuser>:<rpcpassword>@<lbrycrd rpc ip><lbrycrd rpc port>',
default=cls.default('DAEMON_URL', None))
parser.add_argument('--db_max_open_files', type=int, default=64,
help='number of files rocksdb can have open at a time')
parser.add_argument('--host', type=str, default=cls.default('HOST', 'localhost'),
help='Interface for hub server to listen on')
parser.add_argument('--tcp_port', type=int, default=cls.integer('TCP_PORT', 50001),
help='TCP port to listen on for hub server')
parser.add_argument('--udp_port', type=int, default=cls.integer('UDP_PORT', 50001),
help='UDP port to listen on for hub server')
parser.add_argument('--rpc_host', default=cls.default('RPC_HOST', 'localhost'), type=str,
help='Listening interface for admin rpc')
parser.add_argument('--rpc_port', default=cls.integer('RPC_PORT', 8000), type=int,
help='Listening port for admin rpc')
parser.add_argument('--websocket_host', default=cls.default('WEBSOCKET_HOST', 'localhost'), type=str,
help='Listening interface for websocket')
parser.add_argument('--websocket_port', default=cls.integer('WEBSOCKET_PORT', None), type=int,
help='Listening port for websocket')
parser.add_argument('--ssl_port', default=cls.integer('SSL_PORT', None), type=int,
help='SSL port to listen on for hub server')
parser.add_argument('--ssl_certfile', default=cls.default('SSL_CERTFILE', None), type=str,
help='Path to SSL cert file')
parser.add_argument('--ssl_keyfile', default=cls.default('SSL_KEYFILE', None), type=str,
help='Path to SSL key file')
parser.add_argument('--reorg_limit', default=cls.integer('REORG_LIMIT', 200), type=int, help='Max reorg depth')
parser.add_argument('--elastic_host', default=cls.default('ELASTIC_HOST', 'localhost'), type=str,
help='elasticsearch host')
parser.add_argument('--elastic_port', default=cls.integer('ELASTIC_PORT', 9200), type=int,
help='elasticsearch port')
parser.add_argument('--es_index_prefix', default=cls.default('ES_INDEX_PREFIX', ''), type=str)
parser.add_argument('--loop_policy', default=cls.default('EVENT_LOOP_POLICY', 'default'), type=str,
choices=['default', 'uvloop'])
parser.add_argument('--max_query_workers', type=int, default=cls.integer('MAX_QUERY_WORKERS', 4),
help='number of threads used by the request handler to read the database')
parser.add_argument('--cache_MB', type=int, default=cls.integer('CACHE_MB', 1024),
help='size of the leveldb lru cache, in megabytes')
parser.add_argument('--cache_all_tx_hashes', type=bool,
help='Load all tx hashes into memory. This will make address subscriptions and sync, '
'resolve, transaction fetching, and block sync all faster at the expense of higher '
'memory usage')
parser.add_argument('--cache_all_claim_txos', type=bool,
help='Load all claim txos into memory. This will make address subscriptions and sync, '
'resolve, transaction fetching, and block sync all faster at the expense of higher '
'memory usage')
parser.add_argument('--prometheus_port', type=int, default=cls.integer('PROMETHEUS_PORT', 0),
help='port for hub prometheus metrics to listen on, disabled by default')
parser.add_argument('--max_subscriptions', type=int, default=cls.integer('MAX_SUBSCRIPTIONS', 10000),
help='max subscriptions per connection')
parser.add_argument('--banner_file', type=str, default=cls.default('BANNER_FILE', None),
help='path to file containing banner text')
parser.add_argument('--anon_logs', type=bool, default=cls.boolean('ANON_LOGS', False),
help="don't log ip addresses")
parser.add_argument('--allow_lan_udp', type=bool, default=cls.boolean('ALLOW_LAN_UDP', False),
help='reply to hub UDP ping messages from LAN ip addresses')
parser.add_argument('--country', type=str, default=cls.default('COUNTRY', 'US'), help='')
parser.add_argument('--max_send', type=int, default=cls.default('MAX_SEND', 1000000), help='')
parser.add_argument('--max_receive', type=int, default=cls.default('MAX_RECEIVE', 1000000), help='')
parser.add_argument('--max_sessions', type=int, default=cls.default('MAX_SESSIONS', 1000), help='')
parser.add_argument('--session_timeout', type=int, default=cls.default('SESSION_TIMEOUT', 600), help='')
parser.add_argument('--drop_client', type=str, default=cls.default('DROP_CLIENT', None), help='')
parser.add_argument('--description', type=str, default=cls.default('DESCRIPTION', ''), help='')
parser.add_argument('--daily_fee', type=float, default=cls.default('DAILY_FEE', 0.0), help='')
parser.add_argument('--payment_address', type=str, default=cls.default('PAYMENT_ADDRESS', ''), help='')
parser.add_argument('--donation_address', type=str, default=cls.default('DONATION_ADDRESS', ''), help='')
parser.add_argument('--chain', type=str, default=cls.default('NET', 'mainnet'),
help="Which chain to use, default is mainnet")
parser.add_argument('--query_timeout_ms', type=int, default=cls.integer('QUERY_TIMEOUT_MS', 10000),
help="elasticsearch query timeout")
parser.add_argument('--blocking_channel_ids', nargs='*', help='',
default=cls.default('BLOCKING_CHANNEL_IDS', '').split(' '))
parser.add_argument('--filtering_channel_ids', nargs='*', help='',
default=cls.default('FILTERING_CHANNEL_IDS', '').split(' '))
@classmethod
def from_arg_parser(cls, args):
return cls(
db_dir=args.db_dir, daemon_url=args.daemon_url, db_max_open_files=args.db_max_open_files,
host=args.host, rpc_host=args.rpc_host, elastic_host=args.elastic_host, elastic_port=args.elastic_port,
loop_policy=args.loop_policy, max_query_workers=args.max_query_workers, websocket_host=args.websocket_host,
websocket_port=args.websocket_port, chain=args.chain, es_index_prefix=args.es_index_prefix,
cache_MB=args.cache_MB, reorg_limit=args.reorg_limit, tcp_port=args.tcp_port,
udp_port=args.udp_port, ssl_port=args.ssl_port, ssl_certfile=args.ssl_certfile,
ssl_keyfile=args.ssl_keyfile, rpc_port=args.rpc_port, prometheus_port=args.prometheus_port,
max_subscriptions=args.max_subscriptions, banner_file=args.banner_file, anon_logs=args.anon_logs,
log_sessions=None, allow_lan_udp=args.allow_lan_udp,
cache_all_tx_hashes=args.cache_all_tx_hashes, cache_all_claim_txos=args.cache_all_claim_txos,
country=args.country, payment_address=args.payment_address, donation_address=args.donation_address,
max_send=args.max_send, max_receive=args.max_receive, max_sessions=args.max_sessions,
session_timeout=args.session_timeout, drop_client=args.drop_client, description=args.description,
daily_fee=args.daily_fee, database_query_timeout=(args.query_timeout_ms / 1000),
blocking_channel_ids=args.blocking_channel_ids, filtering_channel_ids=args.filtering_channel_ids
)
File diff suppressed because it is too large. Load diff
+112 -265
View File
@@ -11,14 +11,13 @@ import itertools
import time
import attr
import typing
from typing import Set, Optional, Callable, Awaitable
from collections import defaultdict
from prometheus_client import Histogram
from lbry.wallet.server.hash import hash_to_hex_str, hex_str_to_hash
from lbry.wallet.server.util import class_logger, chunks
from lbry.wallet.server.leveldb import UTXO
from lbry.wallet.server.util import class_logger
if typing.TYPE_CHECKING:
from lbry.wallet.server.session import LBRYSessionManager
from wallet.server.db.db import HubDB
@attr.s(slots=True)
@@ -29,6 +28,7 @@ class MemPoolTx:
out_pairs = attr.ib()
fee = attr.ib()
size = attr.ib()
raw_tx = attr.ib()
@attr.s(slots=True)
@@ -49,269 +49,100 @@ mempool_process_time_metric = Histogram(
class MemPool:
def __init__(self, coin, daemon, db, state_lock: asyncio.Lock, refresh_secs=1.0, log_status_secs=120.0):
def __init__(self, coin, db: 'HubDB', refresh_secs=1.0):
self.coin = coin
self._daemon = daemon
self._db = db
self._touched_mp = {}
self._touched_bp = {}
self._highest_block = -1
self.logger = class_logger(__name__, self.__class__.__name__)
self.txs = {}
self.hashXs = defaultdict(set) # None can be a key
self.cached_compact_histogram = []
self.raw_mempool = {}
self.touched_hashXs: typing.DefaultDict[bytes, typing.Set[bytes]] = defaultdict(set) # None can be a key
self.refresh_secs = refresh_secs
self.log_status_secs = log_status_secs
# Prevents mempool refreshes during fee histogram calculation
self.lock = state_lock
self.wakeup = asyncio.Event()
self.mempool_process_time_metric = mempool_process_time_metric
self.notified_mempool_txs = set()
self.notify_sessions: Optional[Callable[[int, Set[bytes], Set[bytes]], Awaitable[None]]] = None
self.session_manager: typing.Optional['LBRYSessionManager'] = None
async def _logging(self, synchronized_event):
"""Print regular logs of mempool stats."""
self.logger.info('beginning processing of daemon mempool. '
'This can take some time...')
start = time.perf_counter()
await synchronized_event.wait()
elapsed = time.perf_counter() - start
self.logger.info(f'synced in {elapsed:.2f}s')
while True:
self.logger.info(f'{len(self.txs):,d} txs '
f'touching {len(self.hashXs):,d} addresses')
await asyncio.sleep(self.log_status_secs)
await synchronized_event.wait()
def refresh(self) -> typing.Set[bytes]: # returns list of new touched hashXs
prefix_db = self._db.prefix_db
new_mempool = {k.tx_hash: v.raw_tx for k, v in prefix_db.mempool_tx.iterate()} # TODO: make this more efficient
self.raw_mempool.clear()
self.raw_mempool.update(new_mempool)
def _accept_transactions(self, tx_map, utxo_map, touched):
"""Accept transactions in tx_map to the mempool if all their inputs
can be found in the existing mempool or a utxo_map from the
DB.
# hashXs = self.hashXs # hashX: [tx_hash, ...]
touched_hashXs = set()
Returns an (unprocessed tx_map, unspent utxo_map) pair.
"""
hashXs = self.hashXs
txs = self.txs
deferred = {}
unspent = set(utxo_map)
# Try to find all prevouts so we can accept the TX
for hash, tx in tx_map.items():
in_pairs = []
try:
for prevout in tx.prevouts:
utxo = utxo_map.get(prevout)
if not utxo:
prev_hash, prev_index = prevout
# Raises KeyError if prev_hash is not in txs
utxo = txs[prev_hash].out_pairs[prev_index]
in_pairs.append(utxo)
except KeyError:
deferred[hash] = tx
continue
# Spend the prevouts
unspent.difference_update(tx.prevouts)
# Save the in_pairs, compute the fee and accept the TX
tx.in_pairs = tuple(in_pairs)
# Avoid negative fees if dealing with generation-like transactions
# because some in_parts would be missing
tx.fee = max(0, (sum(v for _, v in tx.in_pairs) -
sum(v for _, v in tx.out_pairs)))
txs[hash] = tx
for hashX, value in itertools.chain(tx.in_pairs, tx.out_pairs):
touched.add(hashX)
hashXs[hashX].add(hash)
return deferred, {prevout: utxo_map[prevout] for prevout in unspent}
async def _mempool_loop(self, synchronized_event):
try:
return await self._refresh_hashes(synchronized_event)
except asyncio.CancelledError:
raise
except Exception as e:
self.logger.exception("MEMPOOL DIED")
raise e
async def _refresh_hashes(self, synchronized_event):
"""Refresh our view of the daemon's mempool."""
while True:
start = time.perf_counter()
height = self._daemon.cached_height()
hex_hashes = await self._daemon.mempool_hashes()
if height != await self._daemon.height():
continue
hashes = {hex_str_to_hash(hh) for hh in hex_hashes}
async with self.lock:
new_hashes = hashes.difference(self.notified_mempool_txs)
touched = await self._process_mempool(hashes)
self.notified_mempool_txs.update(new_hashes)
new_touched = {
touched_hashx for touched_hashx, txs in self.hashXs.items() if txs.intersection(new_hashes)
}
synchronized_event.set()
synchronized_event.clear()
await self.on_mempool(touched, new_touched, height)
duration = time.perf_counter() - start
self.mempool_process_time_metric.observe(duration)
try:
# we wait up to `refresh_secs` but go early if a broadcast happens (which triggers wakeup event)
await asyncio.wait_for(self.wakeup.wait(), timeout=self.refresh_secs)
except asyncio.TimeoutError:
pass
finally:
self.wakeup.clear()
async def _process_mempool(self, all_hashes):
# Re-sync with the new set of hashes
txs = self.txs
hashXs = self.hashXs # hashX: [tx_hash, ...]
touched = set()
# First handle txs that have disappeared
for tx_hash in set(txs).difference(all_hashes):
tx = txs.pop(tx_hash)
tx_hashXs = {hashX for hashX, value in tx.in_pairs}
tx_hashXs.update(hashX for hashX, value in tx.out_pairs)
# Remove txs that aren't in mempool anymore
for tx_hash in set(self.txs).difference(self.raw_mempool.keys()):
tx = self.txs.pop(tx_hash)
tx_hashXs = {hashX for hashX, value in tx.in_pairs}.union({hashX for hashX, value in tx.out_pairs})
for hashX in tx_hashXs:
hashXs[hashX].remove(tx_hash)
if not hashXs[hashX]:
del hashXs[hashX]
touched.update(tx_hashXs)
# Process new transactions
new_hashes = list(all_hashes.difference(txs))
if new_hashes:
fetches = []
for hashes in chunks(new_hashes, 200):
fetches.append(self._fetch_and_accept(hashes, all_hashes, touched))
tx_map = {}
utxo_map = {}
for fetch in asyncio.as_completed(fetches):
deferred, unspent = await fetch
tx_map.update(deferred)
utxo_map.update(unspent)
prior_count = 0
# FIXME: this is not particularly efficient
while tx_map and len(tx_map) != prior_count:
prior_count = len(tx_map)
tx_map, utxo_map = self._accept_transactions(tx_map, utxo_map, touched)
if tx_map:
self.logger.info(f'{len(tx_map)} txs dropped')
return touched
async def _fetch_and_accept(self, hashes, all_hashes, touched):
"""Fetch a list of mempool transactions."""
raw_txs = await self._daemon.getrawtransactions((hash_to_hex_str(hash) for hash in hashes))
to_hashX = self.coin.hashX_from_script
deserializer = self.coin.DESERIALIZER
if hashX in self.touched_hashXs and tx_hash in self.touched_hashXs[hashX]:
self.touched_hashXs[hashX].remove(tx_hash)
if not self.touched_hashXs[hashX]:
self.touched_hashXs.pop(hashX)
touched_hashXs.update(tx_hashXs)
# Re-sync with the new set of hashes
tx_map = {}
for hash, raw_tx in zip(hashes, raw_txs):
# The daemon may have evicted the tx from its
# mempool or it may have gotten in a block
if not raw_tx:
for tx_hash, raw_tx in self.raw_mempool.items():
if tx_hash in self.txs:
continue
tx, tx_size = deserializer(raw_tx).read_tx_and_vsize()
tx, tx_size = self.coin.DESERIALIZER(raw_tx).read_tx_and_vsize()
# Convert the inputs and outputs into (hashX, value) pairs
# Drop generation-like inputs from MemPoolTx.prevouts
txin_pairs = tuple((txin.prev_hash, txin.prev_idx)
for txin in tx.inputs
if not txin.is_generation())
txout_pairs = tuple((to_hashX(txout.pk_script), txout.value)
txout_pairs = tuple((self.coin.hashX_from_script(txout.pk_script), txout.value)
for txout in tx.outputs)
tx_map[hash] = MemPoolTx(txin_pairs, None, txout_pairs,
0, tx_size)
# Determine all prevouts not in the mempool, and fetch the
# UTXO information from the database. Failed prevout lookups
# return None - concurrent database updates happen - which is
# relied upon by _accept_transactions. Ignore prevouts that are
# generation-like.
prevouts = tuple(prevout for tx in tx_map.values()
for prevout in tx.prevouts
if prevout[0] not in all_hashes)
utxos = await self._db.lookup_utxos(prevouts)
utxo_map = dict(zip(prevouts, utxos))
tx_map[tx_hash] = MemPoolTx(None, txin_pairs, txout_pairs, 0, tx_size, raw_tx)
return self._accept_transactions(tx_map, utxo_map, touched)
for tx_hash, tx in tx_map.items():
prevouts = []
# Look up the prevouts
for prev_hash, prev_index in tx.in_pairs:
if prev_hash in self.txs: # accepted mempool
utxo = self.txs[prev_hash].out_pairs[prev_index]
elif prev_hash in tx_map: # this set of changes
utxo = tx_map[prev_hash].out_pairs[prev_index]
else: # get it from the db
prev_tx_num = prefix_db.tx_num.get(prev_hash)
if not prev_tx_num:
continue
prev_tx_num = prev_tx_num.tx_num
hashX_val = prefix_db.hashX_utxo.get(prev_hash[:4], prev_tx_num, prev_index)
if not hashX_val:
continue
hashX = hashX_val.hashX
utxo_value = prefix_db.utxo.get(hashX, prev_tx_num, prev_index)
utxo = (hashX, utxo_value.amount)
prevouts.append(utxo)
#
# External interface
#
# Save the prevouts, compute the fee and accept the TX
tx.prevouts = tuple(prevouts)
# Avoid negative fees if dealing with generation-like transactions
# because some in_parts would be missing
tx.fee = max(0, (sum(v for _, v in tx.prevouts) -
sum(v for _, v in tx.out_pairs)))
self.txs[tx_hash] = tx
# print(f"added {tx_hash[::-1].hex()} reader to mempool")
async def keep_synchronized(self, synchronized_event):
"""Keep the mempool synchronized with the daemon."""
await asyncio.wait([
self._mempool_loop(synchronized_event),
# self._refresh_histogram(synchronized_event),
self._logging(synchronized_event)
])
async def balance_delta(self, hashX):
"""Return the unconfirmed amount in the mempool for hashX.
Can be positive or negative.
"""
value = 0
if hashX in self.hashXs:
for hash in self.hashXs[hashX]:
tx = self.txs[hash]
value -= sum(v for h168, v in tx.in_pairs if h168 == hashX)
value += sum(v for h168, v in tx.out_pairs if h168 == hashX)
return value
def compact_fee_histogram(self):
"""Return a compact fee histogram of the current mempool."""
return self.cached_compact_histogram
async def potential_spends(self, hashX):
"""Return a set of (prev_hash, prev_idx) pairs from mempool
transactions that touch hashX.
None, some or all of these may be spends of the hashX, but all
actual spends of it (in the DB or mempool) will be included.
"""
result = set()
for tx_hash in self.hashXs.get(hashX, ()):
tx = self.txs[tx_hash]
result.update(tx.prevouts)
return result
for hashX, value in itertools.chain(tx.prevouts, tx.out_pairs):
self.touched_hashXs[hashX].add(tx_hash)
touched_hashXs.add(hashX)
return touched_hashXs
def transaction_summaries(self, hashX):
"""Return a list of MemPoolTxSummary objects for the hashX."""
result = []
for tx_hash in self.hashXs.get(hashX, ()):
for tx_hash in self.touched_hashXs.get(hashX, ()):
if tx_hash not in self.txs:
continue # the tx hash for the touched address is an input that isn't in mempool anymore
tx = self.txs[tx_hash]
has_ui = any(hash in self.txs for hash, idx in tx.prevouts)
has_ui = any(hash in self.txs for hash, idx in tx.in_pairs)
result.append(MemPoolTxSummary(tx_hash, tx.fee, has_ui))
return result
async def unordered_UTXOs(self, hashX):
"""Return an unordered list of UTXO named tuples from mempool
transactions that pay to hashX.
This does not consider if any other mempool transactions spend
the outputs.
"""
utxos = []
for tx_hash in self.hashXs.get(hashX, ()):
tx = self.txs.get(tx_hash)
for pos, (hX, value) in enumerate(tx.out_pairs):
if hX == hashX:
utxos.append(UTXO(-1, pos, tx_hash, 0, value))
return utxos
def get_mempool_height(self, tx_hash):
def get_mempool_height(self, tx_hash: bytes) -> int:
# Height Progression
# -2: not broadcast
# -1: in mempool but has unconfirmed inputs
@@ -320,41 +151,57 @@ class MemPool:
if tx_hash not in self.txs:
return -2
tx = self.txs[tx_hash]
unspent_inputs = sum(1 if hash in self.txs else 0 for hash, idx in tx.prevouts)
unspent_inputs = any(hash in self.raw_mempool for hash, idx in tx.in_pairs)
if unspent_inputs:
return -1
return 0
async def _maybe_notify(self, new_touched):
tmp, tbp = self._touched_mp, self._touched_bp
common = set(tmp).intersection(tbp)
if common:
height = max(common)
elif tmp and max(tmp) == self._highest_block:
height = self._highest_block
else:
# Either we are processing a block and waiting for it to
# come in, or we have not yet had a mempool update for the
# new block height
return
touched = tmp.pop(height)
for old in [h for h in tmp if h <= height]:
del tmp[old]
for old in [h for h in tbp if h <= height]:
touched.update(tbp.pop(old))
# print("notify", height, len(touched), len(new_touched))
await self.notify_sessions(height, touched, new_touched)
async def start(self, height, session_manager: 'LBRYSessionManager'):
self._highest_block = height
self.notify_sessions = session_manager._notify_sessions
await self.notify_sessions(height, set(), set())
self.session_manager = session_manager
await self._notify_sessions(height, set(), set())
async def on_mempool(self, touched, new_touched, height):
self._touched_mp[height] = touched
await self._maybe_notify(new_touched)
await self._notify_sessions(height, touched, new_touched)
async def on_block(self, touched, height):
self._touched_bp[height] = touched
self._highest_block = height
await self._maybe_notify(set())
await self._notify_sessions(height, touched, set())
async def _notify_sessions(self, height, touched, new_touched):
"""Notify sessions about height changes and touched addresses."""
height_changed = height != self.session_manager.notified_height
if height_changed:
await self.session_manager._refresh_hsub_results(height)
if not self.session_manager.sessions:
return
if height_changed:
header_tasks = [
session.send_notification('blockchain.headers.subscribe', (self.session_manager.hsub_results[session.subscribe_headers_raw], ))
for session in self.session_manager.sessions.values() if session.subscribe_headers
]
if header_tasks:
self.logger.info(f'notify {len(header_tasks)} sessions of new header')
asyncio.create_task(asyncio.wait(header_tasks))
for hashX in touched.intersection(self.session_manager.mempool_statuses.keys()):
self.session_manager.mempool_statuses.pop(hashX, None)
# self.bp._chain_executor
await asyncio.get_event_loop().run_in_executor(
self._db._executor, touched.intersection_update, self.session_manager.hashx_subscriptions_by_session.keys()
)
if touched or new_touched or (height_changed and self.session_manager.mempool_statuses):
notified_hashxs = 0
session_hashxes_to_notify = defaultdict(list)
to_notify = touched if height_changed else new_touched
for hashX in to_notify:
if hashX not in self.session_manager.hashx_subscriptions_by_session:
continue
for session_id in self.session_manager.hashx_subscriptions_by_session[hashX]:
session_hashxes_to_notify[session_id].append(hashX)
notified_hashxs += 1
for session_id, hashXes in session_hashxes_to_notify.items():
asyncio.create_task(self.session_manager.sessions[session_id].send_history_notifications(*hashXes))
if session_hashxes_to_notify:
self.logger.info(f'notified {len(session_hashxes_to_notify)} sessions/{notified_hashxs:,d} touched addresses')
-135
View File
@@ -1,135 +0,0 @@
import time
import math
from typing import Tuple
def calculate_elapsed(start) -> int:
return int((time.perf_counter() - start) * 1000)
def calculate_avg_percentiles(data) -> Tuple[int, int, int, int, int, int, int, int]:
if not data:
return 0, 0, 0, 0, 0, 0, 0, 0
data.sort()
size = len(data)
return (
int(sum(data) / size),
data[0],
data[math.ceil(size * .05) - 1],
data[math.ceil(size * .25) - 1],
data[math.ceil(size * .50) - 1],
data[math.ceil(size * .75) - 1],
data[math.ceil(size * .95) - 1],
data[-1]
)
def remove_select_list(sql) -> str:
return sql[sql.index('FROM'):]
class APICallMetrics:
def __init__(self, name):
self.name = name
# total requests received
self.receive_count = 0
self.cache_response_count = 0
# millisecond timings for query based responses
self.query_response_times = []
self.query_intrp_times = []
self.query_error_times = []
self.query_python_times = []
self.query_wait_times = []
self.query_sql_times = [] # aggregate total of multiple SQL calls made per request
self.individual_sql_times = [] # every SQL query run on server
# actual queries
self.errored_queries = set()
self.interrupted_queries = set()
def to_json(self):
return {
# total requests received
"receive_count": self.receive_count,
# sum of these is total responses made
"cache_response_count": self.cache_response_count,
"query_response_count": len(self.query_response_times),
"intrp_response_count": len(self.query_intrp_times),
"error_response_count": len(self.query_error_times),
# millisecond timings for non-cache responses
"response": calculate_avg_percentiles(self.query_response_times),
"interrupt": calculate_avg_percentiles(self.query_intrp_times),
"error": calculate_avg_percentiles(self.query_error_times),
# response, interrupt and error each also report the python, wait and sql stats:
"python": calculate_avg_percentiles(self.query_python_times),
"wait": calculate_avg_percentiles(self.query_wait_times),
"sql": calculate_avg_percentiles(self.query_sql_times),
# extended timings for individual sql executions
"individual_sql": calculate_avg_percentiles(self.individual_sql_times),
"individual_sql_count": len(self.individual_sql_times),
# actual queries
"errored_queries": list(self.errored_queries),
"interrupted_queries": list(self.interrupted_queries),
}
def start(self):
self.receive_count += 1
def cache_response(self):
self.cache_response_count += 1
def _add_query_timings(self, request_total_time, metrics):
if metrics and 'execute_query' in metrics:
sub_process_total = metrics[self.name][0]['total']
individual_query_times = [f['total'] for f in metrics['execute_query']]
aggregated_query_time = sum(individual_query_times)
self.individual_sql_times.extend(individual_query_times)
self.query_sql_times.append(aggregated_query_time)
self.query_python_times.append(sub_process_total - aggregated_query_time)
self.query_wait_times.append(request_total_time - sub_process_total)
@staticmethod
def _add_queries(query_set, metrics):
if metrics and 'execute_query' in metrics:
for execute_query in metrics['execute_query']:
if 'sql' in execute_query:
query_set.add(remove_select_list(execute_query['sql']))
def query_response(self, start, metrics):
self.query_response_times.append(calculate_elapsed(start))
self._add_query_timings(self.query_response_times[-1], metrics)
def query_interrupt(self, start, metrics):
self.query_intrp_times.append(calculate_elapsed(start))
self._add_queries(self.interrupted_queries, metrics)
self._add_query_timings(self.query_intrp_times[-1], metrics)
def query_error(self, start, metrics):
self.query_error_times.append(calculate_elapsed(start))
self._add_queries(self.errored_queries, metrics)
self._add_query_timings(self.query_error_times[-1], metrics)
class ServerLoadData:
def __init__(self):
self._apis = {}
def for_api(self, name) -> APICallMetrics:
if name not in self._apis:
self._apis[name] = APICallMetrics(name)
return self._apis[name]
def to_json_and_reset(self, status):
try:
return {
'api': {name: api.to_json() for name, api in self._apis.items()},
'status': status
}
finally:
self._apis = {}
+119
View File
@@ -0,0 +1,119 @@
import asyncio
import typing
from lbry.wallet.server.util import chunks, class_logger
if typing.TYPE_CHECKING:
from lbry.wallet.server.daemon import LBCDaemon
from lbry.wallet.server.coin import Coin
class Prefetcher:
"""Prefetches blocks (in the forward direction only)."""
def __init__(self, daemon: 'LBCDaemon', coin: 'Coin', blocks_event: asyncio.Event):
self.logger = class_logger(__name__, self.__class__.__name__)
self.daemon = daemon
self.coin = coin
self.blocks_event = blocks_event
self.blocks = []
self.caught_up = False
# Access to fetched_height should be protected by the semaphore
self.fetched_height = None
self.semaphore = asyncio.Semaphore()
self.refill_event = asyncio.Event()
# The prefetched block cache size. The min cache size has
# little effect on sync time.
self.cache_size = 0
self.min_cache_size = 10 * 1024 * 1024
# This makes the first fetch be 10 blocks
self.ave_size = self.min_cache_size // 10
self.polling_delay = 0.5
async def main_loop(self, bp_height):
"""Loop forever polling for more blocks."""
await self.reset_height(bp_height)
try:
while True:
# Sleep a while if there is nothing to prefetch
await self.refill_event.wait()
if not await self._prefetch_blocks():
await asyncio.sleep(self.polling_delay)
finally:
self.logger.info("block pre-fetcher is shutting down")
def get_prefetched_blocks(self):
"""Called by block processor when it is processing queued blocks."""
blocks = self.blocks
self.blocks = []
self.cache_size = 0
self.refill_event.set()
return blocks
async def reset_height(self, height):
"""Reset to prefetch blocks from the block processor's height.
Used in blockchain reorganisations. This coroutine can be
called asynchronously to the _prefetch_blocks coroutine so we
must synchronize with a semaphore.
"""
async with self.semaphore:
self.blocks.clear()
self.cache_size = 0
self.fetched_height = height
self.refill_event.set()
daemon_height = await self.daemon.height()
behind = daemon_height - height
if behind > 0:
self.logger.info(f'catching up to daemon height {daemon_height:,d} '
f'({behind:,d} blocks behind)')
else:
self.logger.info(f'caught up to daemon height {daemon_height:,d}')
async def _prefetch_blocks(self):
"""Prefetch some blocks and put them on the queue.
Repeats until the queue is full or caught up.
"""
daemon = self.daemon
daemon_height = await daemon.height()
async with self.semaphore:
while self.cache_size < self.min_cache_size:
# Try and catch up all blocks but limit to room in cache.
# Constrain fetch count to between 0 and 500 regardless;
# testnet can be lumpy.
cache_room = self.min_cache_size // self.ave_size
count = min(daemon_height - self.fetched_height, cache_room)
count = min(500, max(count, 0))
if not count:
self.caught_up = True
return False
first = self.fetched_height + 1
hex_hashes = await daemon.block_hex_hashes(first, count)
if self.caught_up:
self.logger.info('new block height {:,d} hash {}'
.format(first + count-1, hex_hashes[-1]))
blocks = await daemon.raw_blocks(hex_hashes)
assert count == len(blocks)
# Special handling for genesis block
if first == 0:
blocks[0] = self.coin.genesis_block(blocks[0])
self.logger.info(f'verified genesis block with hash {hex_hashes[0]}')
# Update our recent average block size estimate
size = sum(len(block) for block in blocks)
if count >= 10:
self.ave_size = size // count
else:
self.ave_size = (size + (10 - count) * self.ave_size) // 10
self.blocks.extend(blocks)
self.cache_size += size
self.fetched_height += count
self.blocks_event.set()
self.refill_event.clear()
return True
-91
View File
@@ -1,91 +0,0 @@
import signal
import logging
import asyncio
from concurrent.futures.thread import ThreadPoolExecutor
import typing
import lbry
from lbry.wallet.server.mempool import MemPool
from lbry.wallet.server.block_processor import BlockProcessor
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.session import LBRYSessionManager
from lbry.prometheus import PrometheusServer
class Server:
def __init__(self, env):
self.env = env
self.log = logging.getLogger(__name__).getChild(self.__class__.__name__)
self.shutdown_event = asyncio.Event()
self.cancellable_tasks = []
self.daemon = daemon = env.coin.DAEMON(env.coin, env.daemon_url)
self.db = db = LevelDB(env)
self.bp = bp = BlockProcessor(env, db, daemon, self.shutdown_event)
self.prometheus_server: typing.Optional[PrometheusServer] = None
self.session_mgr = LBRYSessionManager(
env, db, bp, daemon, self.shutdown_event
)
self._indexer_task = None
async def start(self):
env = self.env
min_str, max_str = env.coin.SESSIONCLS.protocol_min_max_strings()
self.log.info(f'software version: {lbry.__version__}')
self.log.info(f'supported protocol versions: {min_str}-{max_str}')
self.log.info(f'event loop policy: {env.loop_policy}')
self.log.info(f'reorg limit is {env.reorg_limit:,d} blocks')
await self.daemon.height()
def _start_cancellable(run, *args):
_flag = asyncio.Event()
self.cancellable_tasks.append(asyncio.ensure_future(run(*args, _flag)))
return _flag.wait()
await self.start_prometheus()
if self.env.udp_port:
await self.bp.status_server.start(
0, bytes.fromhex(self.bp.coin.GENESIS_HASH)[::-1], self.env.country,
self.env.host, self.env.udp_port, self.env.allow_lan_udp
)
await _start_cancellable(self.bp.fetch_and_process_blocks)
await self.db.populate_header_merkle_cache()
await _start_cancellable(self.bp.mempool.keep_synchronized)
await _start_cancellable(self.session_mgr.serve, self.bp.mempool)
async def stop(self):
for task in reversed(self.cancellable_tasks):
task.cancel()
await asyncio.wait(self.cancellable_tasks)
if self.prometheus_server:
await self.prometheus_server.stop()
self.prometheus_server = None
self.shutdown_event.set()
await self.daemon.close()
def run(self):
loop = asyncio.get_event_loop()
executor = ThreadPoolExecutor(self.env.max_query_workers)
loop.set_default_executor(executor)
def __exit():
raise SystemExit()
try:
loop.add_signal_handler(signal.SIGINT, __exit)
loop.add_signal_handler(signal.SIGTERM, __exit)
loop.run_until_complete(self.start())
loop.run_until_complete(self.shutdown_event.wait())
except (SystemExit, KeyboardInterrupt):
pass
finally:
loop.run_until_complete(self.stop())
executor.shutdown(True)
async def start_prometheus(self):
if not self.prometheus_server and self.env.prometheus_port:
self.prometheus_server = PrometheusServer()
await self.prometheus_server.start("0.0.0.0", self.env.prometheus_port)
+208 -257
View File
@@ -2,21 +2,17 @@ import os
import ssl
import math
import time
import json
import base64
import codecs
import typing
import asyncio
import logging
import itertools
import collections
from bisect import bisect_right
from asyncio import Event, sleep
from collections import defaultdict
from functools import partial
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from elasticsearch import ConnectionTimeout
from prometheus_client import Counter, Info, Histogram, Gauge
@@ -24,10 +20,8 @@ import lbry
from lbry.error import ResolveCensoredError, TooManyClaimSearchParametersError
from lbry.build_info import BUILD, COMMIT_HASH, DOCKER_TAG
from lbry.schema.result import Outputs
from lbry.wallet.server.block_processor import BlockProcessor
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.db.db import HubDB
from lbry.wallet.server.websocket import AdminWebSocket
from lbry.wallet.server.metrics import ServerLoadData, APICallMetrics
from lbry.wallet.rpc.framing import NewlineFramer
import lbry.wallet.server.version as VERSION
@@ -36,14 +30,15 @@ from lbry.wallet.rpc import (
RPCSession, JSONRPCAutoDetect, JSONRPCConnection,
handler_invocation, RPCError, Request, JSONRPC, Notification, Batch
)
from lbry.wallet.server import text
from lbry.wallet.server import util
from lbry.wallet.server.hash import sha256, hash_to_hex_str, hex_str_to_hash, HASHX_LEN, Base58Error
from lbry.wallet.server.daemon import DaemonError
from lbry.wallet.server.db.elasticsearch import SearchIndex
if typing.TYPE_CHECKING:
from lbry.wallet.server.env import Env
from lbry.wallet.server.mempool import MemPool
from lbry.wallet.server.daemon import Daemon
from lbry.wallet.server.mempool import MemPool
BAD_REQUEST = 1
DAEMON_ERROR = 2
@@ -177,31 +172,43 @@ class SessionManager:
namespace=NAMESPACE, buckets=HISTOGRAM_BUCKETS
)
def __init__(self, env: 'Env', db: LevelDB, bp: BlockProcessor, daemon: 'Daemon', shutdown_event: asyncio.Event):
def __init__(self, env: 'Env', db: HubDB, mempool: 'MemPool', history_cache, resolve_cache, resolve_outputs_cache,
daemon: 'Daemon', shutdown_event: asyncio.Event,
on_available_callback: typing.Callable[[], None], on_unavailable_callback: typing.Callable[[], None]):
env.max_send = max(350000, env.max_send)
self.env = env
self.db = db
self.bp = bp
self.on_available_callback = on_available_callback
self.on_unavailable_callback = on_unavailable_callback
self.daemon = daemon
self.mempool = bp.mempool
self.mempool = mempool
self.shutdown_event = shutdown_event
self.logger = util.class_logger(__name__, self.__class__.__name__)
self.servers: typing.Dict[str, asyncio.AbstractServer] = {}
self.sessions: typing.Dict[int, 'SessionBase'] = {}
self.sessions: typing.Dict[int, 'LBRYElectrumX'] = {}
self.hashx_subscriptions_by_session: typing.DefaultDict[str, typing.Set[int]] = defaultdict(set)
self.mempool_statuses = {}
self.cur_group = SessionGroup(0)
self.txs_sent = 0
self.start_time = time.time()
self.history_cache = self.bp.history_cache
self.history_cache = history_cache
self.resolve_cache = resolve_cache
self.resolve_outputs_cache = resolve_outputs_cache
self.notified_height: typing.Optional[int] = None
# Cache some idea of room to avoid recounting on each subscription
self.subs_room = 0
self.session_event = Event()
# Search index
self.search_index = SearchIndex(
self.env.es_index_prefix, self.env.database_query_timeout,
elastic_host=env.elastic_host, elastic_port=env.elastic_port
)
async def _start_server(self, kind, *args, **kw_args):
loop = asyncio.get_event_loop()
if kind == 'RPC':
protocol_class = LocalRPC
else:
@@ -250,7 +257,7 @@ class SessionManager:
await self.session_event.wait()
self.session_event.clear()
if not paused and len(self.sessions) >= max_sessions:
self.bp.status_server.set_unavailable()
self.on_unavailable_callback()
self.logger.info(f'maximum sessions {max_sessions:,d} '
f'reached, stopping new connections until '
f'count drops to {low_watermark:,d}')
@@ -259,12 +266,11 @@ class SessionManager:
# Start listening for incoming connections if paused and
# session count has fallen
if paused and len(self.sessions) <= low_watermark:
self.bp.status_server.set_available()
self.on_available_callback()
self.logger.info('resuming listening for incoming connections')
await self._start_external_servers()
paused = False
def _group_map(self):
group_map = defaultdict(list)
for session in self.sessions.values():
@@ -541,16 +547,20 @@ class SessionManager:
await self.start_other()
await self._start_external_servers()
server_listening_event.set()
self.bp.status_server.set_available()
self.on_available_callback()
# Peer discovery should start after the external servers
# because we connect to ourself
await asyncio.wait([
self._clear_stale_sessions(),
self._manage_servers()
])
except Exception as err:
if not isinstance(err, asyncio.CancelledError):
log.exception("hub server died")
raise err
finally:
await self._close_servers(list(self.servers.keys()))
log.warning("disconnect %i sessions", len(self.sessions))
log.info("disconnect %i sessions", len(self.sessions))
if self.sessions:
await asyncio.wait([
session.close(force_after=1) for session in self.sessions.values()
@@ -577,7 +587,7 @@ class SessionManager:
async def raw_header(self, height):
"""Return the binary header at the given height."""
try:
return self.db.raw_header(height)
return await self.db.raw_header(height)
except IndexError:
raise RPCError(BAD_REQUEST, f'height {height:,d} '
'out of range') from None
@@ -612,46 +622,6 @@ class SessionManager:
self.logger.info(f'notify {len(notify_tasks)} sessions of new peers')
asyncio.create_task(asyncio.wait(notify_tasks))
async def _notify_sessions(self, height, touched, new_touched):
"""Notify sessions about height changes and touched addresses."""
height_changed = height != self.notified_height
if height_changed:
await self._refresh_hsub_results(height)
if not self.sessions:
return
if height_changed:
header_tasks = [
session.send_notification('blockchain.headers.subscribe', (self.hsub_results[session.subscribe_headers_raw], ))
for session in self.sessions.values() if session.subscribe_headers
]
if header_tasks:
self.logger.info(f'notify {len(header_tasks)} sessions of new header')
asyncio.create_task(asyncio.wait(header_tasks))
for hashX in touched.intersection(self.mempool_statuses.keys()):
self.mempool_statuses.pop(hashX, None)
await asyncio.get_event_loop().run_in_executor(
None, touched.intersection_update, self.hashx_subscriptions_by_session.keys()
)
if touched or new_touched or (height_changed and self.mempool_statuses):
notified_hashxs = 0
session_hashxes_to_notify = defaultdict(list)
to_notify = touched if height_changed else new_touched
for hashX in to_notify:
if hashX not in self.hashx_subscriptions_by_session:
continue
for session_id in self.hashx_subscriptions_by_session[hashX]:
session_hashxes_to_notify[session_id].append(hashX)
notified_hashxs += 1
for session_id, hashXes in session_hashxes_to_notify.items():
asyncio.create_task(self.sessions[session_id].send_history_notifications(*hashXes))
if session_hashxes_to_notify:
self.logger.info(f'notified {len(session_hashxes_to_notify)} sessions/{notified_hashxs:,d} touched addresses')
def add_session(self, session):
self.sessions[id(session)] = session
self.session_event.set()
@@ -684,12 +654,12 @@ class SessionBase(RPCSession):
request_handlers: typing.Dict[str, typing.Callable] = {}
version = '0.5.7'
def __init__(self, session_mgr, db, mempool, kind):
def __init__(self, session_manager: 'LBRYSessionManager', db: 'HubDB', mempool: 'MemPool', kind: str):
connection = JSONRPCConnection(JSONRPCAutoDetect)
self.env = session_mgr.env
self.env = session_manager.env
super().__init__(connection=connection)
self.logger = util.class_logger(__name__, self.__class__.__name__)
self.session_mgr = session_mgr
self.session_manager = session_manager
self.db = db
self.mempool = mempool
self.kind = kind # 'RPC', 'TCP' etc.
@@ -697,7 +667,7 @@ class SessionBase(RPCSession):
self.anon_logs = self.env.anon_logs
self.txs_sent = 0
self.log_me = False
self.daemon_request = self.session_mgr.daemon_request
self.daemon_request = self.session_manager.daemon_request
# Hijack the connection so we can log messages
self._receive_message_orig = self.connection.receive_message
self.connection.receive_message = self.receive_message
@@ -727,17 +697,17 @@ class SessionBase(RPCSession):
self.session_id = next(self.session_counter)
context = {'conn_id': f'{self.session_id}'}
self.logger = util.ConnectionLogger(self.logger, context)
self.group = self.session_mgr.add_session(self)
self.session_mgr.session_count_metric.labels(version=self.client_version).inc()
self.group = self.session_manager.add_session(self)
self.session_manager.session_count_metric.labels(version=self.client_version).inc()
peer_addr_str = self.peer_address_str()
self.logger.info(f'{self.kind} {peer_addr_str}, '
f'{self.session_mgr.session_count():,d} total')
f'{self.session_manager.session_count():,d} total')
def connection_lost(self, exc):
"""Handle client disconnection."""
super().connection_lost(exc)
self.session_mgr.remove_session(self)
self.session_mgr.session_count_metric.labels(version=self.client_version).dec()
self.session_manager.remove_session(self)
self.session_manager.session_count_metric.labels(version=self.client_version).dec()
msg = ''
if not self._can_send.is_set():
msg += ' whilst paused'
@@ -761,7 +731,7 @@ class SessionBase(RPCSession):
"""Handle an incoming request. ElectrumX doesn't receive
notifications from client sessions.
"""
self.session_mgr.request_count_metric.labels(method=request.method, version=self.client_version).inc()
self.session_manager.request_count_metric.labels(method=request.method, version=self.client_version).inc()
if isinstance(request, Request):
handler = self.request_handlers.get(request.method)
handler = partial(handler, self)
@@ -775,10 +745,9 @@ class LBRYSessionManager(SessionManager):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.query_executor = None
self.websocket = None
# self.metrics = ServerLoadData()
self.metrics_loop = None
# self.metrics_loop = None
self.running = False
if self.env.websocket_host is not None and self.env.websocket_port is not None:
self.websocket = AdminWebSocket(self)
@@ -795,12 +764,6 @@ class LBRYSessionManager(SessionManager):
async def start_other(self):
self.running = True
if self.env.max_query_workers is not None and self.env.max_query_workers == 0:
self.query_executor = ThreadPoolExecutor(max_workers=1)
else:
self.query_executor = ProcessPoolExecutor(
max_workers=self.env.max_query_workers or max(os.cpu_count(), 4)
)
if self.websocket is not None:
await self.websocket.start()
@@ -808,7 +771,6 @@ class LBRYSessionManager(SessionManager):
self.running = False
if self.websocket is not None:
await self.websocket.stop()
self.query_executor.shutdown()
class LBRYElectrumX(SessionBase):
@@ -817,7 +779,7 @@ class LBRYElectrumX(SessionBase):
PROTOCOL_MIN = VERSION.PROTOCOL_MIN
PROTOCOL_MAX = VERSION.PROTOCOL_MAX
max_errors = math.inf # don't disconnect people for errors! let them happen...
session_mgr: LBRYSessionManager
session_manager: LBRYSessionManager
version = lbry.__version__
cached_server_features = {}
@@ -828,17 +790,17 @@ class LBRYElectrumX(SessionBase):
'blockchain.block.get_header': cls.block_get_header,
'blockchain.estimatefee': cls.estimatefee,
'blockchain.relayfee': cls.relayfee,
'blockchain.scripthash.get_balance': cls.scripthash_get_balance,
# 'blockchain.scripthash.get_balance': cls.scripthash_get_balance,
'blockchain.scripthash.get_history': cls.scripthash_get_history,
'blockchain.scripthash.get_mempool': cls.scripthash_get_mempool,
'blockchain.scripthash.listunspent': cls.scripthash_listunspent,
# 'blockchain.scripthash.listunspent': cls.scripthash_listunspent,
'blockchain.scripthash.subscribe': cls.scripthash_subscribe,
'blockchain.transaction.broadcast': cls.transaction_broadcast,
'blockchain.transaction.get': cls.transaction_get,
'blockchain.transaction.get_batch': cls.transaction_get_batch,
'blockchain.transaction.info': cls.transaction_info,
'blockchain.transaction.get_merkle': cls.transaction_merkle,
'server.add_peer': cls.add_peer,
# 'server.add_peer': cls.add_peer,
'server.banner': cls.banner,
'server.payment_address': cls.payment_address,
'server.donation_address': cls.donation_address,
@@ -855,10 +817,10 @@ class LBRYElectrumX(SessionBase):
'blockchain.block.headers': cls.block_headers,
'server.ping': cls.ping,
'blockchain.headers.subscribe': cls.headers_subscribe_False,
'blockchain.address.get_balance': cls.address_get_balance,
# 'blockchain.address.get_balance': cls.address_get_balance,
'blockchain.address.get_history': cls.address_get_history,
'blockchain.address.get_mempool': cls.address_get_mempool,
'blockchain.address.listunspent': cls.address_listunspent,
# 'blockchain.address.listunspent': cls.address_listunspent,
'blockchain.address.subscribe': cls.address_subscribe,
'blockchain.address.unsubscribe': cls.address_unsubscribe,
})
@@ -877,9 +839,8 @@ class LBRYElectrumX(SessionBase):
self.sv_seen = False
self.protocol_tuple = self.PROTOCOL_MIN
self.protocol_string = None
self.daemon = self.session_mgr.daemon
self.bp: BlockProcessor = self.session_mgr.bp
self.db: LevelDB = self.bp.db
self.daemon = self.session_manager.daemon
self.db: HubDB = self.session_manager.db
@classmethod
def protocol_min_max_strings(cls):
@@ -902,7 +863,7 @@ class LBRYElectrumX(SessionBase):
'donation_address': env.donation_address,
'daily_fee': env.daily_fee,
'hash_function': 'sha256',
'trending_algorithm': 'variable_decay'
'trending_algorithm': 'fast_ar'
})
async def server_features_async(self):
@@ -919,6 +880,20 @@ class LBRYElectrumX(SessionBase):
def sub_count(self):
return len(self.hashX_subs)
async def get_hashX_status(self, hashX: bytes):
mempool_history = self.mempool.transaction_summaries(hashX)
history = ''.join(f'{hash_to_hex_str(tx_hash)}:'
f'{height:d}:'
for tx_hash, height in await self.session_manager.limited_history(hashX))
history += ''.join(f'{hash_to_hex_str(tx.hash)}:'
f'{-tx.has_unconfirmed_inputs:d}:'
for tx in mempool_history)
if history:
status = sha256(history.encode()).hex()
else:
status = None
return history, status, len(mempool_history) > 0
async def send_history_notifications(self, *hashXes: typing.Iterable[bytes]):
notifications = []
for hashX in hashXes:
@@ -928,38 +903,26 @@ class LBRYElectrumX(SessionBase):
else:
method = 'blockchain.address.subscribe'
start = time.perf_counter()
db_history = await self.session_mgr.limited_history(hashX)
mempool = self.mempool.transaction_summaries(hashX)
status = ''.join(f'{hash_to_hex_str(tx_hash)}:'
f'{height:d}:'
for tx_hash, height in db_history)
status += ''.join(f'{hash_to_hex_str(tx.hash)}:'
f'{-tx.has_unconfirmed_inputs:d}:'
for tx in mempool)
if status:
status = sha256(status.encode()).hex()
history, status, mempool_status = await self.get_hashX_status(hashX)
if mempool_status:
self.session_manager.mempool_statuses[hashX] = status
else:
status = None
if mempool:
self.session_mgr.mempool_statuses[hashX] = status
else:
self.session_mgr.mempool_statuses.pop(hashX, None)
self.session_manager.mempool_statuses.pop(hashX, None)
self.session_mgr.address_history_metric.observe(time.perf_counter() - start)
self.session_manager.address_history_metric.observe(time.perf_counter() - start)
notifications.append((method, (alias, status)))
start = time.perf_counter()
self.session_mgr.notifications_in_flight_metric.inc()
self.session_manager.notifications_in_flight_metric.inc()
for method, args in notifications:
self.NOTIFICATION_COUNT.labels(method=method, version=self.client_version).inc()
try:
await self.send_notifications(
Batch([Notification(method, (alias, status)) for (method, (alias, status)) in notifications])
)
self.session_mgr.notifications_sent_metric.observe(time.perf_counter() - start)
self.session_manager.notifications_sent_metric.observe(time.perf_counter() - start)
finally:
self.session_mgr.notifications_in_flight_metric.dec()
self.session_manager.notifications_in_flight_metric.dec()
# def get_metrics_or_placeholder_for_api(self, query_name):
# """ Do not hold on to a reference to the metrics
@@ -967,65 +930,48 @@ class LBRYElectrumX(SessionBase):
# you may be working with a stale metrics object.
# """
# if self.env.track_metrics:
# # return self.session_mgr.metrics.for_api(query_name)
# # return self.session_manager.metrics.for_api(query_name)
# else:
# return APICallMetrics(query_name)
async def run_in_executor(self, query_name, func, kwargs):
start = time.perf_counter()
try:
self.session_mgr.pending_query_metric.inc()
result = await asyncio.get_running_loop().run_in_executor(
self.session_mgr.query_executor, func, kwargs
)
except asyncio.CancelledError:
raise
except Exception:
log.exception("dear devs, please handle this exception better")
self.session_mgr.db_error_metric.inc()
raise RPCError(JSONRPC.INTERNAL_ERROR, 'unknown server error')
else:
return base64.b64encode(result).decode()
finally:
self.session_mgr.pending_query_metric.dec()
self.session_mgr.executor_time_metric.observe(time.perf_counter() - start)
# async def run_and_cache_query(self, query_name, kwargs):
# start = time.perf_counter()
# if isinstance(kwargs, dict):
# kwargs['release_time'] = format_release_time(kwargs.get('release_time'))
# try:
# self.session_mgr.pending_query_metric.inc()
# self.session_manager.pending_query_metric.inc()
# return await self.db.search_index.session_query(query_name, kwargs)
# except ConnectionTimeout:
# self.session_mgr.interrupt_count_metric.inc()
# self.session_manager.interrupt_count_metric.inc()
# raise RPCError(JSONRPC.QUERY_TIMEOUT, 'query timed out')
# finally:
# self.session_mgr.pending_query_metric.dec()
# self.session_mgr.executor_time_metric.observe(time.perf_counter() - start)
# self.session_manager.pending_query_metric.dec()
# self.session_manager.executor_time_metric.observe(time.perf_counter() - start)
async def mempool_compact_histogram(self):
return self.mempool.compact_fee_histogram()
return [] #self.mempool.compact_fee_histogram()
async def claimtrie_search(self, **kwargs):
start = time.perf_counter()
if 'release_time' in kwargs:
release_time = kwargs.pop('release_time')
release_times = release_time if isinstance(release_time, list) else [release_time]
try:
kwargs['release_time'] = format_release_time(release_time)
kwargs['release_time'] = [format_release_time(release_time) for release_time in release_times]
except ValueError:
pass
try:
self.session_mgr.pending_query_metric.inc()
self.session_manager.pending_query_metric.inc()
if 'channel' in kwargs:
channel_url = kwargs.pop('channel')
_, channel_claim, _, _ = await self.db.resolve(channel_url)
if not channel_claim or isinstance(channel_claim, (ResolveCensoredError, LookupError, ValueError)):
return Outputs.to_base64([], [], 0, None, None)
kwargs['channel_id'] = channel_claim.claim_hash.hex()
return await self.db.search_index.cached_search(kwargs)
return await self.session_manager.search_index.cached_search(kwargs)
except ConnectionTimeout:
self.session_mgr.interrupt_count_metric.inc()
self.session_manager.interrupt_count_metric.inc()
raise RPCError(JSONRPC.QUERY_TIMEOUT, 'query timed out')
except TooManyClaimSearchParametersError as err:
await asyncio.sleep(2)
@@ -1033,57 +979,69 @@ class LBRYElectrumX(SessionBase):
self.peer_address()[0], err.key, err.limit)
return RPCError(1, str(err))
finally:
self.session_mgr.pending_query_metric.dec()
self.session_mgr.executor_time_metric.observe(time.perf_counter() - start)
self.session_manager.pending_query_metric.dec()
self.session_manager.executor_time_metric.observe(time.perf_counter() - start)
def _claimtrie_resolve(self, *urls):
rows, extra = [], []
for url in urls:
self.session_mgr.urls_to_resolve_count_metric.inc()
stream, channel, repost, reposted_channel = self.db._resolve(url)
if isinstance(channel, ResolveCensoredError):
rows.append(channel)
extra.append(channel.censor_row)
elif isinstance(stream, ResolveCensoredError):
rows.append(stream)
extra.append(stream.censor_row)
elif channel and not stream:
rows.append(channel)
# print("resolved channel", channel.name.decode())
if repost:
extra.append(repost)
if reposted_channel:
extra.append(reposted_channel)
elif stream:
# print("resolved stream", stream.name.decode())
rows.append(stream)
if channel:
# print("and channel", channel.name.decode())
extra.append(channel)
if repost:
extra.append(repost)
if reposted_channel:
extra.append(reposted_channel)
# print("claimtrie resolve %i rows %i extrat" % (len(rows), len(extra)))
return Outputs.to_base64(rows, extra, 0, None, None)
async def _cached_resolve_url(self, url):
if url not in self.session_manager.resolve_cache:
self.session_manager.resolve_cache[url] = await self.loop.run_in_executor(self.db._executor, self.db._resolve, url)
return self.session_manager.resolve_cache[url]
async def claimtrie_resolve(self, *urls):
result = await self.loop.run_in_executor(None, self._claimtrie_resolve, *urls)
self.session_mgr.resolved_url_count_metric.inc(len(urls))
return result
async def claimtrie_resolve(self, *urls) -> str:
sorted_urls = tuple(sorted(urls))
self.session_manager.urls_to_resolve_count_metric.inc(len(sorted_urls))
try:
if sorted_urls in self.session_manager.resolve_outputs_cache:
return self.session_manager.resolve_outputs_cache[sorted_urls]
rows, extra = [], []
for url in urls:
if url not in self.session_manager.resolve_cache:
self.session_manager.resolve_cache[url] = await self._cached_resolve_url(url)
stream, channel, repost, reposted_channel = self.session_manager.resolve_cache[url]
if isinstance(channel, ResolveCensoredError):
rows.append(channel)
extra.append(channel.censor_row)
elif isinstance(stream, ResolveCensoredError):
rows.append(stream)
extra.append(stream.censor_row)
elif channel and not stream:
rows.append(channel)
# print("resolved channel", channel.name.decode())
if repost:
extra.append(repost)
if reposted_channel:
extra.append(reposted_channel)
elif stream:
# print("resolved stream", stream.name.decode())
rows.append(stream)
if channel:
# print("and channel", channel.name.decode())
extra.append(channel)
if repost:
extra.append(repost)
if reposted_channel:
extra.append(reposted_channel)
await asyncio.sleep(0)
self.session_manager.resolve_outputs_cache[sorted_urls] = result = await self.loop.run_in_executor(
None, Outputs.to_base64, rows, extra, 0, None, None
)
return result
finally:
self.session_manager.resolved_url_count_metric.inc(len(sorted_urls))
async def get_server_height(self):
return self.bp.height
return self.db.db_height
async def transaction_get_height(self, tx_hash):
self.assert_tx_hash(tx_hash)
transaction_info = await self.daemon.getrawtransaction(tx_hash, True)
if transaction_info and 'hex' in transaction_info and 'confirmations' in transaction_info:
# an unconfirmed transaction from lbrycrdd will not have a 'confirmations' field
return (self.db.db_height - transaction_info['confirmations']) + 1
elif transaction_info and 'hex' in transaction_info:
return -1
return None
def get_height():
v = self.db.prefix_db.tx_num.get(tx_hash)
if v:
return bisect_right(self.db.tx_counts, v.tx_num)
return self.mempool.get_mempool_height(tx_hash)
return await asyncio.get_event_loop().run_in_executor(self.db._executor, get_height)
async def claimtrie_getclaimbyid(self, claim_id):
rows = []
@@ -1106,7 +1064,7 @@ class LBRYElectrumX(SessionBase):
async def subscribe_headers_result(self):
"""The result of a header subscription or notification."""
return self.session_mgr.hsub_results[self.subscribe_headers_raw]
return self.session_manager.hsub_results[self.subscribe_headers_raw]
async def _headers_subscribe(self, raw):
"""Subscribe to get headers of new blocks."""
@@ -1142,48 +1100,34 @@ class LBRYElectrumX(SessionBase):
"""
# Note history is ordered and mempool unordered in electrum-server
# For mempool, height is -1 if it has unconfirmed inputs, otherwise 0
db_history = await self.session_mgr.limited_history(hashX)
mempool = self.mempool.transaction_summaries(hashX)
status = ''.join(f'{hash_to_hex_str(tx_hash)}:'
f'{height:d}:'
for tx_hash, height in db_history)
status += ''.join(f'{hash_to_hex_str(tx.hash)}:'
f'{-tx.has_unconfirmed_inputs:d}:'
for tx in mempool)
if status:
status = sha256(status.encode()).hex()
_, status, has_mempool_history = await self.get_hashX_status(hashX)
if has_mempool_history:
self.session_manager.mempool_statuses[hashX] = status
else:
status = None
if mempool:
self.session_mgr.mempool_statuses[hashX] = status
else:
self.session_mgr.mempool_statuses.pop(hashX, None)
self.session_manager.mempool_statuses.pop(hashX, None)
return status
async def hashX_listunspent(self, hashX):
"""Return the list of UTXOs of a script hash, including mempool
effects."""
utxos = await self.db.all_utxos(hashX)
utxos = sorted(utxos)
utxos.extend(await self.mempool.unordered_UTXOs(hashX))
spends = await self.mempool.potential_spends(hashX)
return [{'tx_hash': hash_to_hex_str(utxo.tx_hash),
'tx_pos': utxo.tx_pos,
'height': utxo.height, 'value': utxo.value}
for utxo in utxos
if (utxo.tx_hash, utxo.tx_pos) not in spends]
# async def hashX_listunspent(self, hashX):
# """Return the list of UTXOs of a script hash, including mempool
# effects."""
# utxos = await self.db.all_utxos(hashX)
# utxos = sorted(utxos)
# utxos.extend(await self.mempool.unordered_UTXOs(hashX))
# spends = await self.mempool.potential_spends(hashX)
#
# return [{'tx_hash': hash_to_hex_str(utxo.tx_hash),
# 'tx_pos': utxo.tx_pos,
# 'height': utxo.height, 'value': utxo.value}
# for utxo in utxos
# if (utxo.tx_hash, utxo.tx_pos) not in spends]
async def hashX_subscribe(self, hashX, alias):
self.hashX_subs[hashX] = alias
self.session_mgr.hashx_subscriptions_by_session[hashX].add(id(self))
self.session_manager.hashx_subscriptions_by_session[hashX].add(id(self))
return await self.address_status(hashX)
async def hashX_unsubscribe(self, hashX, alias):
sessions = self.session_mgr.hashx_subscriptions_by_session[hashX]
sessions = self.session_manager.hashx_subscriptions_by_session[hashX]
sessions.remove(id(self))
if not sessions:
self.hashX_subs.pop(hashX, None)
@@ -1195,10 +1139,10 @@ class LBRYElectrumX(SessionBase):
pass
raise RPCError(BAD_REQUEST, f'{address} is not a valid address')
async def address_get_balance(self, address):
"""Return the confirmed and unconfirmed balance of an address."""
hashX = self.address_to_hashX(address)
return await self.get_balance(hashX)
# async def address_get_balance(self, address):
# """Return the confirmed and unconfirmed balance of an address."""
# hashX = self.address_to_hashX(address)
# return await self.get_balance(hashX)
async def address_get_history(self, address):
"""Return the confirmed and unconfirmed history of an address."""
@@ -1210,10 +1154,10 @@ class LBRYElectrumX(SessionBase):
hashX = self.address_to_hashX(address)
return self.unconfirmed_history(hashX)
async def address_listunspent(self, address):
"""Return the list of UTXOs of an address."""
hashX = self.address_to_hashX(address)
return await self.hashX_listunspent(hashX)
# async def address_listunspent(self, address):
# """Return the list of UTXOs of an address."""
# hashX = self.address_to_hashX(address)
# return await self.hashX_listunspent(hashX)
async def address_subscribe(self, *addresses):
"""Subscribe to an address.
@@ -1221,9 +1165,11 @@ class LBRYElectrumX(SessionBase):
address: the address to subscribe to"""
if len(addresses) > 1000:
raise RPCError(BAD_REQUEST, f'too many addresses in subscription request: {len(addresses)}')
return [
await self.hashX_subscribe(self.address_to_hashX(address), address) for address in addresses
]
results = []
for address in addresses:
results.append(await self.hashX_subscribe(self.address_to_hashX(address), address))
await asyncio.sleep(0)
return results
async def address_unsubscribe(self, address):
"""Unsubscribe an address.
@@ -1232,16 +1178,16 @@ class LBRYElectrumX(SessionBase):
hashX = self.address_to_hashX(address)
return await self.hashX_unsubscribe(hashX, address)
async def get_balance(self, hashX):
utxos = await self.db.all_utxos(hashX)
confirmed = sum(utxo.value for utxo in utxos)
unconfirmed = await self.mempool.balance_delta(hashX)
return {'confirmed': confirmed, 'unconfirmed': unconfirmed}
# async def get_balance(self, hashX):
# utxos = await self.db.all_utxos(hashX)
# confirmed = sum(utxo.value for utxo in utxos)
# unconfirmed = await self.mempool.balance_delta(hashX)
# return {'confirmed': confirmed, 'unconfirmed': unconfirmed}
async def scripthash_get_balance(self, scripthash):
"""Return the confirmed and unconfirmed balance of a scripthash."""
hashX = scripthash_to_hashX(scripthash)
return await self.get_balance(hashX)
# async def scripthash_get_balance(self, scripthash):
# """Return the confirmed and unconfirmed balance of a scripthash."""
# hashX = scripthash_to_hashX(scripthash)
# return await self.get_balance(hashX)
def unconfirmed_history(self, hashX):
# Note unconfirmed history is unordered in electrum-server
@@ -1253,7 +1199,7 @@ class LBRYElectrumX(SessionBase):
async def confirmed_and_unconfirmed_history(self, hashX):
# Note history is ordered but unconfirmed is unordered in e-s
history = await self.session_mgr.limited_history(hashX)
history = await self.session_manager.limited_history(hashX)
conf = [{'tx_hash': hash_to_hex_str(tx_hash), 'height': height}
for tx_hash, height in history]
return conf + self.unconfirmed_history(hashX)
@@ -1268,10 +1214,10 @@ class LBRYElectrumX(SessionBase):
hashX = scripthash_to_hashX(scripthash)
return self.unconfirmed_history(hashX)
async def scripthash_listunspent(self, scripthash):
"""Return the list of UTXOs of a scripthash."""
hashX = scripthash_to_hashX(scripthash)
return await self.hashX_listunspent(hashX)
# async def scripthash_listunspent(self, scripthash):
# """Return the list of UTXOs of a scripthash."""
# hashX = scripthash_to_hashX(scripthash)
# return await self.hashX_listunspent(hashX)
async def scripthash_subscribe(self, scripthash):
"""Subscribe to a script hash.
@@ -1306,7 +1252,7 @@ class LBRYElectrumX(SessionBase):
max_size = self.MAX_CHUNK_SIZE
count = min(count, max_size)
headers, count = self.db.read_headers(start_height, count)
headers, count = await self.db.read_headers(start_height, count)
if b64:
headers = self.db.encode_headers(start_height, count, headers)
@@ -1329,7 +1275,7 @@ class LBRYElectrumX(SessionBase):
index = non_negative_integer(index)
size = self.coin.CHUNK_SIZE
start_height = index * size
headers, _ = self.db.read_headers(start_height, size)
headers, _ = await self.db.read_headers(start_height, size)
return headers.hex()
async def block_get_header(self, height):
@@ -1337,7 +1283,7 @@ class LBRYElectrumX(SessionBase):
height: the header's height"""
height = non_negative_integer(height)
return await self.session_mgr.electrum_header(height)
return await self.session_manager.electrum_header(height)
def is_tor(self):
"""Try to detect if the connection is to a tor hidden service we are
@@ -1427,10 +1373,10 @@ class LBRYElectrumX(SessionBase):
self.close_after_send = True
raise RPCError(BAD_REQUEST, f'unsupported client: {client_name}')
if self.client_version != client_name[:17]:
self.session_mgr.session_count_metric.labels(version=self.client_version).dec()
self.session_manager.session_count_metric.labels(version=self.client_version).dec()
self.client_version = client_name[:17]
self.session_mgr.session_count_metric.labels(version=self.client_version).inc()
self.session_mgr.client_version_metric.labels(version=self.client_version).inc()
self.session_manager.session_count_metric.labels(version=self.client_version).inc()
self.session_manager.client_version_metric.labels(version=self.client_version).inc()
# Find the highest common protocol version. Disconnect if
# that protocol version in unsupported.
@@ -1451,9 +1397,10 @@ class LBRYElectrumX(SessionBase):
raw_tx: the raw transaction as a hexadecimal string"""
# This returns errors as JSON RPC errors, as is natural
try:
hex_hash = await self.session_mgr.broadcast_transaction(raw_tx)
hex_hash = await self.session_manager.broadcast_transaction(raw_tx)
self.txs_sent += 1
self.mempool.wakeup.set()
# self.mempool.wakeup.set()
# await asyncio.sleep(0.5)
self.logger.info(f'sent tx: {hex_hash}')
return hex_hash
except DaemonError as e:
@@ -1467,20 +1414,25 @@ class LBRYElectrumX(SessionBase):
return (await self.transaction_get_batch(tx_hash))[tx_hash]
async def transaction_get_batch(self, *tx_hashes):
self.session_mgr.tx_request_count_metric.inc(len(tx_hashes))
self.session_manager.tx_request_count_metric.inc(len(tx_hashes))
if len(tx_hashes) > 100:
raise RPCError(BAD_REQUEST, f'too many tx hashes in request: {len(tx_hashes)}')
for tx_hash in tx_hashes:
assert_tx_hash(tx_hash)
batch_result = await self.db.fs_transactions(tx_hashes)
batch_result = await self.db.get_transactions_and_merkles(tx_hashes)
needed_merkles = {}
for tx_hash in tx_hashes:
if tx_hash in batch_result and batch_result[tx_hash][0]:
continue
tx_info = await self.daemon_request('getrawtransaction', tx_hash, True)
raw_tx = tx_info['hex']
block_hash = tx_info.get('blockhash')
tx_hash_bytes = bytes.fromhex(tx_hash)[::-1]
mempool_tx = self.mempool.txs.get(tx_hash_bytes, None)
if mempool_tx:
raw_tx, block_hash = mempool_tx.raw_tx.hex(), None
else:
tx_info = await self.daemon_request('getrawtransaction', tx_hash, 1)
raw_tx = tx_info['hex']
block_hash = tx_info.get('blockhash')
if block_hash:
block = await self.daemon.deserialised_block(block_hash)
height = block['height']
@@ -1501,8 +1453,7 @@ class LBRYElectrumX(SessionBase):
'block_height': block_height
}
await asyncio.sleep(0) # heavy call, give other tasks a chance
self.session_mgr.tx_replied_count_metric.inc(len(tx_hashes))
self.session_manager.tx_replied_count_metric.inc(len(tx_hashes))
return batch_result
async def transaction_get(self, tx_hash, verbose=False):
@@ -1515,7 +1466,7 @@ class LBRYElectrumX(SessionBase):
if verbose not in (True, False):
raise RPCError(BAD_REQUEST, f'"verbose" must be a boolean')
return await self.daemon_request('getrawtransaction', tx_hash, verbose)
return await self.daemon_request('getrawtransaction', tx_hash, int(verbose))
def _get_merkle_branch(self, tx_hashes, tx_pos):
"""Return a merkle branch to a transaction.
-82
View File
@@ -1,82 +0,0 @@
import time
from lbry.wallet.server import util
def sessions_lines(data):
"""A generator returning lines for a list of sessions.
data is the return value of rpc_sessions()."""
fmt = ('{:<6} {:<5} {:>17} {:>5} {:>5} {:>5} '
'{:>7} {:>7} {:>7} {:>7} {:>7} {:>9} {:>21}')
yield fmt.format('ID', 'Flags', 'Client', 'Proto',
'Reqs', 'Txs', 'Subs',
'Recv', 'Recv KB', 'Sent', 'Sent KB', 'Time', 'Peer')
for (id_, flags, peer, client, proto, reqs, txs_sent, subs,
recv_count, recv_size, send_count, send_size, time) in data:
yield fmt.format(id_, flags, client, proto,
f'{reqs:,d}',
f'{txs_sent:,d}',
f'{subs:,d}',
f'{recv_count:,d}',
'{:,d}'.format(recv_size // 1024),
f'{send_count:,d}',
'{:,d}'.format(send_size // 1024),
util.formatted_time(time, sep=''), peer)
def groups_lines(data):
"""A generator returning lines for a list of groups.
data is the return value of rpc_groups()."""
fmt = ('{:<6} {:>9} {:>9} {:>6} {:>6} {:>8}'
'{:>7} {:>9} {:>7} {:>9}')
yield fmt.format('ID', 'Sessions', 'Bwidth KB', 'Reqs', 'Txs', 'Subs',
'Recv', 'Recv KB', 'Sent', 'Sent KB')
for (id_, session_count, bandwidth, reqs, txs_sent, subs,
recv_count, recv_size, send_count, send_size) in data:
yield fmt.format(id_,
f'{session_count:,d}',
'{:,d}'.format(bandwidth // 1024),
f'{reqs:,d}',
f'{txs_sent:,d}',
f'{subs:,d}',
f'{recv_count:,d}',
'{:,d}'.format(recv_size // 1024),
f'{send_count:,d}',
'{:,d}'.format(send_size // 1024))
def peers_lines(data):
"""A generator returning lines for a list of peers.
data is the return value of rpc_peers()."""
def time_fmt(t):
if not t:
return 'Never'
return util.formatted_time(now - t)
now = time.time()
fmt = ('{:<30} {:<6} {:>5} {:>5} {:<17} {:>4} '
'{:>4} {:>8} {:>11} {:>11} {:>5} {:>20} {:<15}')
yield fmt.format('Host', 'Status', 'TCP', 'SSL', 'Server', 'Min',
'Max', 'Pruning', 'Last Good', 'Last Try',
'Tries', 'Source', 'IP Address')
for item in data:
features = item['features']
hostname = item['host']
host = features['hosts'][hostname]
yield fmt.format(hostname[:30],
item['status'],
host.get('tcp_port') or '',
host.get('ssl_port') or '',
features['server_version'] or 'unknown',
features['protocol_min'],
features['protocol_max'],
features['pruning'] or '',
time_fmt(item['last_good']),
time_fmt(item['last_try']),
item['try_count'],
item['source'][:20],
item['ip_addr'] or '')
+1 -1
View File
@@ -98,7 +98,7 @@ class Deserializer:
TX_HASH_FN = staticmethod(double_sha256)
def __init__(self, binary, start=0):
assert isinstance(binary, bytes)
assert isinstance(binary, bytes), f"type {type(binary)} is not 'bytes'"
self.binary = binary
self.binary_length = len(binary)
self.cursor = start
+7 -3
View File
@@ -110,6 +110,7 @@ class SPVServerStatusProtocol(asyncio.DatagramProtocol):
self._min_delay = 1 / throttle_reqs_per_sec
self._allow_localhost = allow_localhost
self._allow_lan = allow_lan
self.closed = asyncio.Event()
def update_cached_response(self):
self._left_cache, self._right_cache = SPVPong.make_sans_source_address(
@@ -160,13 +161,16 @@ class SPVServerStatusProtocol(asyncio.DatagramProtocol):
def connection_made(self, transport) -> None:
self.transport = transport
self.closed.clear()
def connection_lost(self, exc: Optional[Exception]) -> None:
self.transport = None
self.closed.set()
def close(self):
async def close(self):
if self.transport:
self.transport.close()
await self.closed.wait()
class StatusServer:
@@ -184,9 +188,9 @@ class StatusServer:
await loop.create_datagram_endpoint(lambda: self._protocol, (interface, port))
log.info("started udp status server on %s:%i", interface, port)
def stop(self):
async def stop(self):
if self.is_running:
self._protocol.close()
await self._protocol.close()
self._protocol = None
@property
+20 -40
View File
@@ -1,19 +1,9 @@
import struct
import hashlib
import logging
import typing
import asyncio
from binascii import hexlify, unhexlify
from typing import List, Iterable, Optional, Tuple
import ecdsa
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.serialization import load_der_public_key
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.asymmetric.utils import Prehashed
from cryptography.exceptions import InvalidSignature
from lbry.error import InsufficientFundsError
from lbry.crypto.hash import hash160, sha256
from lbry.crypto.base58 import Base58
@@ -24,10 +14,11 @@ from lbry.schema.purchase import Purchase
from lbry.schema.support import Support
from .script import InputScript, OutputScript
from .constants import COIN, NULL_HASH32
from .constants import COIN, DUST, NULL_HASH32
from .bcd_data_stream import BCDataStream
from .hash import TXRef, TXRefImmutable
from .util import ReadOnlyList
from .bip32 import PrivateKey, PublicKey
if typing.TYPE_CHECKING:
from lbry.wallet.account import Account
@@ -222,7 +213,8 @@ class Output(InputOutput):
is_my_output: Optional[bool] = None, is_my_input: Optional[bool] = None,
sent_supports: Optional[int] = None, sent_tips: Optional[int] = None,
received_tips: Optional[int] = None,
channel: Optional['Output'] = None, private_key: Optional[str] = None
channel: Optional['Output'] = None,
private_key: Optional[PrivateKey] = None
) -> None:
super().__init__(tx_ref, position)
self.amount = amount
@@ -235,7 +227,7 @@ class Output(InputOutput):
self.sent_tips = sent_tips
self.received_tips = received_tips
self.channel = channel
self.private_key = private_key
self.private_key: PrivateKey = private_key
self.purchase: 'Output' = None # txo containing purchase metadata
self.purchased_claim: 'Output' = None # resolved claim pointed to by purchase
self.purchase_receipt: 'Output' = None # txo representing purchase receipt for this claim
@@ -425,25 +417,15 @@ class Output(InputOutput):
]
return sha256(b''.join(pieces))
def get_encoded_signature(self):
signature = hexlify(self.signable.signature)
r = int(signature[:int(len(signature)/2)], 16)
s = int(signature[int(len(signature)/2):], 16)
return ecdsa.util.sigencode_der(r, s, len(signature)*4)
@staticmethod
def is_signature_valid(encoded_signature, signature_digest, public_key_bytes):
try:
public_key = load_der_public_key(public_key_bytes, default_backend())
public_key.verify(encoded_signature, signature_digest, ec.ECDSA(Prehashed(hashes.SHA256())))
return True
except (ValueError, InvalidSignature):
pass
return False
def is_signature_valid(signature, digest, public_key_bytes):
return PublicKey\
.from_compressed(public_key_bytes)\
.verify(signature, digest)
def is_signed_by(self, channel: 'Output', ledger=None):
return self.is_signature_valid(
self.get_encoded_signature(),
self.signable.signature,
self.get_signature_digest(ledger),
channel.claim.channel.public_key_bytes
)
@@ -456,29 +438,27 @@ class Output(InputOutput):
self.signable.signing_channel_hash,
self.signable.to_message_bytes()
]))
self.signable.signature = channel.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
self.signable.signature = channel.private_key.sign_compact(digest)
self.script.generate()
def sign_data(self, data:bytes, timestamp:str) -> str:
def sign_data(self, data: bytes, timestamp: str) -> str:
pieces = [timestamp.encode(), self.claim_hash, data]
digest = sha256(b''.join(pieces))
signature = self.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
signature = self.private_key.sign_compact(digest)
return hexlify(signature).decode()
def clear_signature(self):
self.channel = None
self.signable.clear_signature()
async def generate_channel_private_key(self):
self.private_key = await asyncio.get_event_loop().run_in_executor(
None, ecdsa.SigningKey.generate, ecdsa.SECP256k1, None, hashlib.sha256
)
self.claim.channel.public_key_bytes = self.private_key.get_verifying_key().to_der()
def set_channel_private_key(self, private_key: PrivateKey):
self.private_key = private_key
self.claim.channel.public_key_bytes = private_key.public_key.pubkey_bytes
self.script.generate()
return self.private_key
def is_channel_private_key(self, private_key):
return self.claim.channel.public_key_bytes == private_key.get_verifying_key().to_der()
def is_channel_private_key(self, private_key: PrivateKey):
return self.claim.channel.public_key_bytes == private_key.public_key.pubkey_bytes
@classmethod
def pay_claim_name_pubkey_hash(
@@ -838,10 +818,10 @@ class Transaction:
)
if payment > cost:
change = payment - cost
if change > cost_of_change:
change_amount = change - cost_of_change
if change_amount > DUST:
change_address = await change_account.change.get_or_create_usable_address()
change_hash160 = change_account.ledger.address_to_hash160(change_address)
change_amount = change - cost_of_change
change_output = Output.pay_pubkey_hash(change_amount, change_hash160)
change_output.is_internal_transfer = True
tx.add_outputs([Output.pay_pubkey_hash(change_amount, change_hash160)])
+4 -9
View File
@@ -40,22 +40,17 @@ def checkrecord(record, expected_winner, expected_claim):
async def checkcontrolling(daemon: Daemon, db: SQLDB):
records, claim_ids, names, futs = [], [], [], []
records, names, futs = [], [], []
for record in db.get_claims('claimtrie.claim_hash as is_controlling, claim.*', is_controlling=True):
records.append(record)
claim_id = hex_reverted(record['claim_hash'])
claim_ids.append((claim_id,))
names.append((record['normalized'],))
names.append((record['normalized'], (claim_id,), "", True)) # last parameter is IncludeValues
if len(names) > 50000:
futs.append(daemon._send_vector('getvalueforname', names[:]))
futs.append(daemon._send_vector('getclaimbyid', claim_ids[:]))
futs.append(daemon._send_vector('getclaimsfornamebyid', names))
names.clear()
claim_ids.clear()
if names:
futs.append(daemon._send_vector('getvalueforname', names[:]))
futs.append(daemon._send_vector('getclaimbyid', claim_ids[:]))
futs.append(daemon._send_vector('getclaimsfornamebyid', names))
names.clear()
claim_ids.clear()
while futs:
winners, claims = futs.pop(0), futs.pop(0)
+67 -3
View File
@@ -1,7 +1,12 @@
import asyncio
import argparse
import logging
import csv
import os.path
from io import StringIO
from typing import Optional
from aiohttp import web
from prometheus_client import generate_latest as prom_generate_latest
from lbry.dht.constants import generate_id
from lbry.dht.node import Node
@@ -13,9 +18,62 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-4s %(na
log = logging.getLogger(__name__)
async def main(host: str, port: int, db_file_path: str, bootstrap_node: Optional[str]):
class SimpleMetrics:
def __init__(self, port, node):
self.prometheus_port = port
self.dht_node: Node = node
async def handle_metrics_get_request(self, _):
try:
return web.Response(
text=prom_generate_latest().decode(),
content_type='text/plain; version=0.0.4'
)
except Exception:
log.exception('could not generate prometheus data')
raise
async def handle_peers_csv(self, _):
out = StringIO()
writer = csv.DictWriter(out, fieldnames=["ip", "port", "dht_id"])
writer.writeheader()
for peer in self.dht_node.protocol.routing_table.get_peers():
writer.writerow({"ip": peer.address, "port": peer.udp_port, "dht_id": peer.node_id.hex()})
return web.Response(text=out.getvalue(), content_type='text/csv')
async def handle_blobs_csv(self, _):
out = StringIO()
writer = csv.DictWriter(out, fieldnames=["blob_hash"])
writer.writeheader()
for blob in self.dht_node.protocol.data_store.keys():
writer.writerow({"blob_hash": blob.hex()})
return web.Response(text=out.getvalue(), content_type='text/csv')
async def start(self):
prom_app = web.Application()
prom_app.router.add_get('/metrics', self.handle_metrics_get_request)
if self.dht_node:
prom_app.router.add_get('/peers.csv', self.handle_peers_csv)
prom_app.router.add_get('/blobs.csv', self.handle_blobs_csv)
metrics_runner = web.AppRunner(prom_app)
await metrics_runner.setup()
prom_site = web.TCPSite(metrics_runner, "0.0.0.0", self.prometheus_port)
await prom_site.start()
async def main(host: str, port: int, db_file_path: str, bootstrap_node: Optional[str], prometheus_port: int, export: bool):
loop = asyncio.get_event_loop()
conf = Config()
if not db_file_path.startswith(':memory:'):
node_id_file_path = db_file_path + 'node_id'
if os.path.exists(node_id_file_path):
with open(node_id_file_path, 'rb') as node_id_file:
node_id = node_id_file.read()
else:
with open(node_id_file_path, 'wb') as node_id_file:
node_id = generate_id()
node_id_file.write(node_id)
storage = SQLiteStorage(conf, db_file_path, loop, loop.time)
if bootstrap_node:
nodes = bootstrap_node.split(':')
@@ -24,10 +82,14 @@ async def main(host: str, port: int, db_file_path: str, bootstrap_node: Optional
nodes = conf.known_dht_nodes
await storage.open()
node = Node(
loop, PeerManager(loop), generate_id(), port, port, 3333, None,
loop, PeerManager(loop), node_id, port, port, 3333, None,
storage=storage
)
if prometheus_port > 0:
metrics = SimpleMetrics(prometheus_port, node if export else None)
await metrics.start()
node.start(host, nodes)
log.info("Peer with id %s started", node_id.hex())
while True:
await asyncio.sleep(10)
log.info("Known peers: %d. Storing contact information for %d blobs from %d peers.",
@@ -44,5 +106,7 @@ if __name__ == '__main__':
parser.add_argument("--bootstrap_node", default=None, type=str,
help="Node to connect for bootstraping this node. Leave unset to use the default ones. "
"Format: host:port Example: lbrynet1.lbry.com:4444")
parser.add_argument("--metrics_port", default=0, type=int, help="Port for Prometheus metrics. 0 to disable. Default: 0")
parser.add_argument("--enable_csv_export", action='store_true', help="Enable CSV endpoints on metrics server.")
args = parser.parse_args()
asyncio.run(main(args.host, args.port, args.db_file, args.bootstrap_node))
asyncio.run(main(args.host, args.port, args.db_file, args.bootstrap_node, args.metrics_port, args.enable_csv_export))
+33
View File
@@ -0,0 +1,33 @@
#!/bin/bash
SNAPSHOT_HEIGHT="1072108"
HUB_VOLUME_PATH="/var/lib/docker/volumes/${USER}_wallet_server"
ES_VOLUME_PATH="/var/lib/docker/volumes/${USER}_es01"
SNAPSHOT_TAR_NAME="wallet_server_snapshot_${SNAPSHOT_HEIGHT}.tar.gz"
ES_SNAPSHOT_TAR_NAME="es_snapshot_${SNAPSHOT_HEIGHT}.tar.gz"
SNAPSHOT_URL="https://snapshots.lbry.com/hub/${SNAPSHOT_TAR_NAME}"
ES_SNAPSHOT_URL="https://snapshots.lbry.com/hub/${ES_SNAPSHOT_TAR_NAME}"
echo "fetching wallet server snapshot"
wget $SNAPSHOT_URL
echo "decompressing wallet server snapshot"
tar -xf $SNAPSHOT_TAR_NAME
sudo mkdir -p $HUB_VOLUME_PATH
sudo rm -rf "${HUB_VOLUME_PATH}/_data"
sudo chown -R 999:999 "snapshot_${SNAPSHOT_HEIGHT}"
sudo mv "snapshot_${SNAPSHOT_HEIGHT}" "${HUB_VOLUME_PATH}/_data"
echo "finished setting up wallet server snapshot"
echo "fetching elasticsearch snapshot"
wget $ES_SNAPSHOT_URL
echo "decompressing elasticsearch snapshot"
tar -xf $ES_SNAPSHOT_TAR_NAME
sudo chown -R $USER:root "snapshot_es_${SNAPSHOT_HEIGHT}"
sudo chmod -R 775 "snapshot_es_${SNAPSHOT_HEIGHT}"
sudo mkdir -p $ES_VOLUME_PATH
sudo rm -rf "${ES_VOLUME_PATH}/_data"
sudo mv "snapshot_es_${SNAPSHOT_HEIGHT}" "${ES_VOLUME_PATH}/_data"
echo "finished setting up elasticsearch snapshot"
+1 -1
View File
@@ -18,7 +18,7 @@ ignore=words,server,rpc,schema,winpaths.py,migrator,undecorated.py
max-parents=10
max-args=10
max-line-length=120
good-names=T,t,n,i,j,k,x,y,s,f,d,h,c,e,op,db,tx,io,cachedproperty,log,id,r,iv,ts,l
good-names=T,t,n,i,j,k,x,y,s,f,d,h,c,e,op,db,tx,io,cachedproperty,log,id,r,iv,ts,l,pk
valid-metaclass-classmethod-first-arg=mcs
disable=
c-extension-no-member,
+12 -8
View File
@@ -7,9 +7,11 @@ BASE = os.path.dirname(__file__)
with open(os.path.join(BASE, 'README.md'), encoding='utf-8') as fh:
long_description = fh.read()
PLYVEL = []
if sys.platform.startswith('linux'):
PLYVEL.append('plyvel==1.3.0')
ROCKSDB = []
if sys.platform.startswith('linux') or sys.platform.startswith('darwin'):
ROCKSDB.append('lbry-rocksdb==0.8.2')
setup(
name=__name__,
@@ -28,9 +30,10 @@ setup(
entry_points={
'console_scripts': [
'lbrynet=lbry.extras.cli:main',
'lbry-hub=lbry.wallet.server.cli:main',
'orchstr8=lbry.wallet.orchstr8.cli:main',
'lbry-hub-elastic-sync=lbry.wallet.server.db.elasticsearch.sync:run_elastic_sync'
'lbry-hub-writer=lbry.wallet.server.cli:run_writer_forever',
'lbry-hub-server=lbry.wallet.server.cli:run_server_forever',
'lbry-hub-elastic-sync=lbry.wallet.server.cli:run_es_sync_forever',
'orchstr8=lbry.wallet.orchstr8.cli:main'
],
},
install_requires=[
@@ -56,8 +59,9 @@ setup(
'attrs==18.2.0',
'pylru==1.1.0',
'elasticsearch==7.10.1',
'grpcio==1.38.0'
] + PLYVEL,
'grpcio==1.38.0',
'filetype==1.0.9'
] + ROCKSDB,
extras_require={
'torrent': ['lbry-libtorrent'],
'lint': ['pylint==2.10.0'],
+47
View File
@@ -0,0 +1,47 @@
#! python
import os
import shutil
import rocksdb
import tempfile
import logging
log = logging.getLogger()
log.addHandler(logging.StreamHandler())
log.setLevel(logging.INFO)
def _main(db_loc):
opts = rocksdb.Options(create_if_missing=True)
db = rocksdb.DB(os.path.join(db_loc, "test"), opts)
secondary_location = os.path.join(db_loc, "secondary")
secondary = rocksdb.DB(
os.path.join(db_loc, "test"),
rocksdb.Options(create_if_missing=True, max_open_files=-1),
secondary_name=secondary_location
)
try:
assert secondary.get(b"a") is None
db.put(b"a", b"b")
assert db.get(b"a") == b"b"
assert secondary.get(b"a") is None
secondary.try_catch_up_with_primary()
assert secondary.get(b"a") == b"b"
finally:
secondary.close()
db.close()
def main():
db_dir = tempfile.mkdtemp()
try:
_main(db_dir)
log.info("rocksdb %s (%s) works!", rocksdb.__version__, rocksdb.ROCKSDB_VERSION)
except:
log.exception("boom")
finally:
shutil.rmtree(db_dir)
if __name__ == "__main__":
main()
@@ -1,5 +1,8 @@
from binascii import unhexlify
from lbry.testcase import CommandTestCase
from lbry.wallet.dewies import dewies_to_lbc
from lbry.wallet.account import DeterministicChannelKeyManager
def extract(d, keys):
@@ -60,15 +63,30 @@ class AccountManagement(CommandTestCase):
self.assertEqual(accounts['items'][0]['name'], 'recreated account')
async def test_wallet_migration(self):
old_id, new_id, valid_key = (
'mi9E8KqFfW5ngktU22pN2jpgsdf81ZbsGY',
'mqs77XbdnuxWN4cXrjKbSoGLkvAHa4f4B8',
'-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIBZRTZ7tHnYCH3IE9mCo95'
'466L/ShYFhXGrjmSMFJw8eoAcGBSuBBAAK\noUQDQgAEmucoPz9nI+ChZrfhnh'
'0RZ/bcX0r2G0pYBmoNKovtKzXGa8y07D66MWsW\nqXptakqO/9KddIkBu5eJNS'
'UZzQCxPQ==\n-----END EC PRIVATE KEY-----\n'
)
# null certificates should get deleted
await self.channel_create('@foo1')
await self.channel_create('@foo2')
await self.channel_create('@foo3')
keys = list(self.account.channel_keys.keys())
self.account.channel_keys[keys[0]] = None
self.account.channel_keys[keys[1]] = "some invalid junk"
self.account.channel_keys = {
new_id: 'not valid key',
'foo': 'bar',
}
await self.account.maybe_migrate_certificates()
self.assertEqual(list(self.account.channel_keys.keys()), [keys[2]])
self.assertEqual(self.account.channel_keys, {})
self.account.channel_keys = {
new_id: 'not valid key',
'foo': 'bar',
'invalid address': valid_key,
}
await self.account.maybe_migrate_certificates()
self.assertEqual(self.account.channel_keys, {
new_id: valid_key
})
async def assertFindsClaims(self, claim_names, awaitable):
self.assertEqual(claim_names, [txo.claim_name for txo in (await awaitable)['items']])
@@ -85,7 +103,7 @@ class AccountManagement(CommandTestCase):
second_account = await self.daemon.jsonrpc_account_create('second account')
tx = await self.daemon.jsonrpc_account_send(
'0.05', await self.daemon.jsonrpc_address_unused(account_id=second_account.id)
'0.05', await self.daemon.jsonrpc_address_unused(account_id=second_account.id), blocking=True
)
await self.confirm_tx(tx.id)
await self.assertOutputAmount(['0.05', '9.949876'], utxo_list())
@@ -174,3 +192,100 @@ class AccountManagement(CommandTestCase):
bad_address = address[0:20] + '9999999' + address[27:]
with self.assertRaisesRegex(Exception, f"'{bad_address}' is not a valid address"):
await self.daemon.jsonrpc_account_send('0.1', addresses=[bad_address])
async def test_hybrid_channel_keys(self):
# non-deterministic channel
self.account.channel_keys = {
'mqs77XbdnuxWN4cXrjKbSoGLkvAHa4f4B8':
'-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIBZRTZ7tHnYCH3IE9mCo95'
'466L/ShYFhXGrjmSMFJw8eoAcGBSuBBAAK\noUQDQgAEmucoPz9nI+ChZrfhnh'
'0RZ/bcX0r2G0pYBmoNKovtKzXGa8y07D66MWsW\nqXptakqO/9KddIkBu5eJNS'
'UZzQCxPQ==\n-----END EC PRIVATE KEY-----\n'
}
channel1 = await self.create_nondeterministic_channel('@foo1', '1.0', unhexlify(
'3056301006072a8648ce3d020106052b8104000a034200049ae7283f3f6723e0a1'
'66b7e19e1d1167f6dc5f4af61b4a58066a0d2a8bed2b35c66bccb4ec3eba316b16'
'a97a6d6a4a8effd29d748901bb9789352519cd00b13d'
))
await self.confirm_tx(channel1['txid'])
# deterministic channel
channel2 = await self.channel_create('@foo2')
await self.stream_create('stream-in-channel1', '0.01', channel_id=self.get_claim_id(channel1))
await self.stream_create('stream-in-channel2', '0.01', channel_id=self.get_claim_id(channel2))
resolved_stream1 = await self.resolve('@foo1/stream-in-channel1')
self.assertEqual('stream-in-channel1', resolved_stream1['name'])
self.assertTrue(resolved_stream1['is_channel_signature_valid'])
resolved_stream2 = await self.resolve('@foo2/stream-in-channel2')
self.assertEqual('stream-in-channel2', resolved_stream2['name'])
self.assertTrue(resolved_stream2['is_channel_signature_valid'])
async def test_deterministic_channel_keys(self):
seed = self.account.seed
keys = self.account.deterministic_channel_keys
# create two channels and make sure they have different keys
channel1a = await self.channel_create('@foo1')
channel2a = await self.channel_create('@foo2')
self.assertNotEqual(
channel1a['outputs'][0]['value']['public_key'],
channel2a['outputs'][0]['value']['public_key'],
)
# start another daemon from the same seed
self.daemon2 = await self.add_daemon(seed=seed)
channel2b, channel1b = (await self.daemon2.jsonrpc_channel_list())['items']
# both daemons end up with the same channel signing keys automagically
self.assertTrue(channel1b.has_private_key)
self.assertEqual(
channel1a['outputs'][0]['value']['public_key_id'],
channel1b.private_key.address
)
self.assertTrue(channel2b.has_private_key)
self.assertEqual(
channel2a['outputs'][0]['value']['public_key_id'],
channel2b.private_key.address
)
# repeatedly calling next channel key returns the same key when not used
current_known = keys.last_known
next_key = await keys.generate_next_key()
self.assertEqual(current_known, keys.last_known)
self.assertEqual(next_key.address, (await keys.generate_next_key()).address)
# again, should be idempotent
next_key = await keys.generate_next_key()
self.assertEqual(current_known, keys.last_known)
self.assertEqual(next_key.address, (await keys.generate_next_key()).address)
# create third channel while both daemons running, second daemon should pick it up
channel3a = await self.channel_create('@foo3')
self.assertEqual(current_known+1, keys.last_known)
self.assertNotEqual(next_key.address, (await keys.generate_next_key()).address)
channel3b, = (await self.daemon2.jsonrpc_channel_list(name='@foo3'))['items']
self.assertTrue(channel3b.has_private_key)
self.assertEqual(
channel3a['outputs'][0]['value']['public_key_id'],
channel3b.private_key.address
)
# channel key cache re-populated after simulated restart
# reset cache
self.account.deterministic_channel_keys = DeterministicChannelKeyManager(self.account)
channel3c, channel2c, channel1c = (await self.daemon.jsonrpc_channel_list())['items']
self.assertFalse(channel1c.has_private_key)
self.assertFalse(channel2c.has_private_key)
self.assertFalse(channel3c.has_private_key)
# repopulate cache
await self.account.deterministic_channel_keys.ensure_cache_primed()
self.assertEqual(self.account.deterministic_channel_keys.last_known, keys.last_known)
channel3c, channel2c, channel1c = (await self.daemon.jsonrpc_channel_list())['items']
self.assertTrue(channel1c.has_private_key)
self.assertTrue(channel2c.has_private_key)
self.assertTrue(channel3c.has_private_key)
@@ -9,7 +9,7 @@ class BlockchainReorganizationTests(CommandTestCase):
VERBOSITY = logging.WARN
async def assertBlockHash(self, height):
bp = self.conductor.spv_node.server.bp
bp = self.conductor.spv_node.writer
def get_txids():
return [
@@ -23,21 +23,22 @@ class BlockchainReorganizationTests(CommandTestCase):
self.assertEqual(block_hash, (await bp.db.fs_block_hashes(height, 1))[0][::-1].hex())
txids = await asyncio.get_event_loop().run_in_executor(None, get_txids)
txs = await bp.db.fs_transactions(txids)
txs = await bp.db.get_transactions_and_merkles(txids)
block_txs = (await bp.daemon.deserialised_block(block_hash))['tx']
self.assertSetEqual(set(block_txs), set(txs.keys()), msg='leveldb/lbrycrd is missing transactions')
self.assertListEqual(block_txs, list(txs.keys()), msg='leveldb/lbrycrd transactions are of order')
async def test_reorg(self):
bp = self.conductor.spv_node.server.bp
bp = self.conductor.spv_node.writer
bp.reorg_count_metric.set(0)
# invalidate current block, move forward 2
height = 206
self.assertEqual(self.ledger.headers.height, height)
await self.assertBlockHash(height)
await self.blockchain.invalidate_block((await self.ledger.headers.hash(206)).decode())
block_hash = (await self.ledger.headers.hash(206)).decode()
await self.blockchain.invalidate_block(block_hash)
await self.blockchain.generate(2)
await self.ledger.on_header.where(lambda e: e.height == 207)
await asyncio.wait_for(self.on_header(207), 3.0)
self.assertEqual(self.ledger.headers.height, 207)
await self.assertBlockHash(206)
await self.assertBlockHash(207)
@@ -46,14 +47,14 @@ class BlockchainReorganizationTests(CommandTestCase):
# invalidate current block, move forward 3
await self.blockchain.invalidate_block((await self.ledger.headers.hash(206)).decode())
await self.blockchain.generate(3)
await self.ledger.on_header.where(lambda e: e.height == 208)
await asyncio.wait_for(self.on_header(208), 3.0)
self.assertEqual(self.ledger.headers.height, 208)
await self.assertBlockHash(206)
await self.assertBlockHash(207)
await self.assertBlockHash(208)
self.assertEqual(2, bp.reorg_count_metric._samples()[0][2])
await self.blockchain.generate(3)
await self.ledger.on_header.where(lambda e: e.height == 211)
await asyncio.wait_for(self.on_header(211), 3.0)
await self.assertBlockHash(209)
await self.assertBlockHash(210)
await self.assertBlockHash(211)
@@ -62,7 +63,7 @@ class BlockchainReorganizationTests(CommandTestCase):
)
await self.ledger.wait(still_valid)
await self.blockchain.generate(1)
await self.ledger.on_header.where(lambda e: e.height == 212)
await asyncio.wait_for(self.on_header(212), 1.0)
claim_id = still_valid.outputs[0].claim_id
c1 = (await self.resolve(f'still-valid#{claim_id}'))['claim_id']
c2 = (await self.resolve(f'still-valid#{claim_id[:2]}'))['claim_id']
@@ -71,7 +72,7 @@ class BlockchainReorganizationTests(CommandTestCase):
abandon_tx = await self.daemon.jsonrpc_stream_abandon(claim_id=claim_id)
await self.blockchain.generate(1)
await self.ledger.on_header.where(lambda e: e.height == 213)
await asyncio.wait_for(self.on_header(213), 1.0)
c1 = await self.resolve(f'still-valid#{still_valid.outputs[0].claim_id}')
c2 = await self.daemon.jsonrpc_resolve([f'still-valid#{claim_id[:2]}'])
c3 = await self.daemon.jsonrpc_resolve([f'still-valid'])
@@ -112,11 +113,10 @@ class BlockchainReorganizationTests(CommandTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.blockchain.clear_mempool()
await self.conductor.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
await asyncio.wait_for(self.on_header(209), 3.0)
await self.assertBlockHash(207)
await self.assertBlockHash(208)
await self.assertBlockHash(209)
@@ -142,9 +142,8 @@ class BlockchainReorganizationTests(CommandTestCase):
# broadcast the claim in a different block
new_txid = await self.blockchain.sendrawtransaction(hexlify(broadcast_tx.raw).decode())
self.assertEqual(broadcast_tx.id, new_txid)
await self.blockchain.generate(1)
# wait for the client to catch up
await self.blockchain.generate(1)
await asyncio.wait_for(self.on_header(210), 1.0)
# verify the claim is in the new block and that it is returned by claim_search
@@ -191,7 +190,7 @@ class BlockchainReorganizationTests(CommandTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.blockchain.clear_mempool()
await self.conductor.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -222,8 +221,6 @@ class BlockchainReorganizationTests(CommandTestCase):
new_txid = await self.blockchain.sendrawtransaction(hexlify(broadcast_tx.raw).decode())
self.assertEqual(broadcast_tx.id, new_txid)
await self.blockchain.generate(1)
# wait for the client to catch up
await asyncio.wait_for(self.on_header(210), 1.0)
# verify the claim is in the new block and that it is returned by claim_search
+23 -14
View File
@@ -33,10 +33,16 @@ class NetworkTests(IntegrationTestCase):
'donation_address': '',
'daily_fee': '0',
'server_version': lbry.__version__,
'trending_algorithm': 'variable_decay',
'trending_algorithm': 'fast_ar',
}, await self.ledger.network.get_server_features())
# await self.conductor.spv_node.stop()
payment_address, donation_address = await self.account.get_addresses(limit=2)
original_address = self.conductor.spv_node.server.env.payment_address
original_donation_address = self.conductor.spv_node.server.env.donation_address
original_description = self.conductor.spv_node.server.env.description
original_daily_fee = self.conductor.spv_node.server.env.daily_fee
self.conductor.spv_node.server.env.payment_address = payment_address
self.conductor.spv_node.server.env.donation_address = donation_address
self.conductor.spv_node.server.env.description = 'Fastest server in the west.'
@@ -58,9 +64,16 @@ class NetworkTests(IntegrationTestCase):
'donation_address': donation_address,
'daily_fee': '42',
'server_version': lbry.__version__,
'trending_algorithm': 'variable_decay',
'trending_algorithm': 'fast_ar',
}, await self.ledger.network.get_server_features())
# cleanup the changes since the attributes are set on the class
self.conductor.spv_node.server.env.payment_address = original_address
self.conductor.spv_node.server.env.donation_address = original_donation_address
self.conductor.spv_node.server.env.description = original_description
self.conductor.spv_node.server.env.daily_fee = original_daily_fee
LBRYElectrumX.set_server_features(self.conductor.spv_node.server.env)
class ReconnectTests(IntegrationTestCase):
@@ -86,7 +99,7 @@ class ReconnectTests(IntegrationTestCase):
await self.ledger.stop()
initial_height = self.ledger.local_height_including_downloaded_height
await self.blockchain.generate(100)
while self.conductor.spv_node.server.session_mgr.notified_height < initial_height + 99: # off by 1
while self.conductor.spv_node.server.session_manager.notified_height < initial_height + 99: # off by 1
await asyncio.sleep(0.1)
self.assertEqual(initial_height, self.ledger.local_height_including_downloaded_height)
await self.ledger.headers.open()
@@ -101,12 +114,7 @@ class ReconnectTests(IntegrationTestCase):
self.ledger.network.client.transport.close()
self.assertFalse(self.ledger.network.is_connected)
await self.ledger.resolve([], 'derp')
sendtxid = await self.blockchain.send_to_address(address1, 1.1337)
# await self.ledger.resolve([], 'derp')
# self.assertTrue(self.ledger.network.is_connected)
await asyncio.wait_for(self.on_transaction_id(sendtxid), 10.0) # mempool
await self.blockchain.generate(1)
await self.on_transaction_id(sendtxid) # confirmed
sendtxid = await self.send_to_address_and_wait(address1, 1.1337, 1)
self.assertLess(self.ledger.network.client.response_time, 1) # response time properly set lower, we are fine
await self.assertBalance(self.account, '1.1337')
@@ -135,7 +143,7 @@ class ReconnectTests(IntegrationTestCase):
await self.conductor.spv_node.stop()
self.assertFalse(self.ledger.network.is_connected)
await asyncio.sleep(0.2) # let it retry and fail once
await self.conductor.spv_node.start(self.conductor.blockchain_node)
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.ledger.network.on_connected.first
self.assertTrue(self.ledger.network.is_connected)
@@ -161,15 +169,16 @@ class ReconnectTests(IntegrationTestCase):
class UDPServerFailDiscoveryTest(AsyncioTestCase):
async def test_wallet_connects_despite_lack_of_udp(self):
conductor = Conductor()
conductor.spv_node.udp_port = '0'
await conductor.start_blockchain()
self.addCleanup(conductor.stop_blockchain)
await conductor.start_lbcd()
self.addCleanup(conductor.stop_lbcd)
await conductor.start_lbcwallet()
self.addCleanup(conductor.stop_lbcwallet)
await conductor.start_spv()
self.addCleanup(conductor.stop_spv)
self.assertFalse(conductor.spv_node.server.bp.status_server.is_running)
self.assertFalse(conductor.spv_node.server.status_server.is_running)
await asyncio.wait_for(conductor.start_wallet(), timeout=5)
self.addCleanup(conductor.stop_wallet)
self.assertTrue(conductor.wallet_node.ledger.network.is_connected)
@@ -174,8 +174,7 @@ class PurchaseCommandTests(CommandTestCase):
self.merchant_address = await self.account.receiving.get_or_create_usable_address()
daemon2 = await self.add_daemon()
address2 = await daemon2.wallet_manager.default_account.receiving.get_or_create_usable_address()
sendtxid = await self.blockchain.send_to_address(address2, 2)
await self.confirm_tx(sendtxid, daemon2.ledger)
await self.send_to_address_and_wait(address2, 2, 1, ledger=daemon2.ledger)
stream = await self.priced_stream('a', '1.0')
await self.assertBalance(self.account, '9.987893')
+3 -3
View File
@@ -63,7 +63,7 @@ class SyncTests(IntegrationTestCase):
await self.assertBalance(account1, '1.0')
await self.assertBalance(account2, '1.0')
await self.blockchain.generate(1)
await self.generate(1)
# pay 0.01 from main node to receiving node, would have increased change addresses
address0 = (await account0.receiving.get_addresses())[0]
@@ -79,7 +79,7 @@ class SyncTests(IntegrationTestCase):
account1.ledger.wait(tx),
account2.ledger.wait(tx),
])
await self.blockchain.generate(1)
await self.generate(1)
await asyncio.wait([
account0.ledger.wait(tx),
account1.ledger.wait(tx),
@@ -92,7 +92,7 @@ class SyncTests(IntegrationTestCase):
await self.assertBalance(account1, '0.989876')
await self.assertBalance(account2, '0.989876')
await self.blockchain.generate(1)
await self.generate(1)
# create a new mirror node and see if it syncs to same balance from scratch
node3 = await self.make_wallet_node(account1.seed)
@@ -1,5 +1,6 @@
import asyncio
import json
from binascii import unhexlify
from lbry.wallet import ENCRYPT_ON_DISK
from lbry.error import InvalidPasswordError
@@ -10,7 +11,7 @@ from lbry.wallet.dewies import dict_values_to_lbc
class WalletCommands(CommandTestCase):
async def test_wallet_create_and_add_subscribe(self):
session = next(iter(self.conductor.spv_node.server.session_mgr.sessions.values()))
session = next(iter(self.conductor.spv_node.server.session_manager.sessions.values()))
self.assertEqual(len(session.hashX_subs), 27)
wallet = await self.daemon.jsonrpc_wallet_create('foo', create_account=True, single_key=True)
self.assertEqual(len(session.hashX_subs), 28)
@@ -22,7 +23,7 @@ class WalletCommands(CommandTestCase):
async def test_wallet_syncing_status(self):
address = await self.daemon.jsonrpc_address_unused()
self.assertFalse(self.daemon.jsonrpc_wallet_status()['is_syncing'])
await self.blockchain.send_to_address(address, 1)
await self.send_to_address_and_wait(address, 1)
await self.ledger._update_tasks.started.wait()
self.assertTrue(self.daemon.jsonrpc_wallet_status()['is_syncing'])
await self.ledger._update_tasks.done.wait()
@@ -46,9 +47,9 @@ class WalletCommands(CommandTestCase):
status = await self.daemon.jsonrpc_status()
self.assertEqual(len(status['wallet']['servers']), 1)
self.assertEqual(status['wallet']['servers'][0]['port'], 50002)
await self.conductor.spv_node.stop(True)
await self.conductor.spv_node.stop()
self.conductor.spv_node.port = 54320
await self.conductor.spv_node.start(self.conductor.blockchain_node)
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
status = await self.daemon.jsonrpc_status()
self.assertEqual(len(status['wallet']['servers']), 0)
self.daemon.jsonrpc_settings_set('lbryum_servers', ['localhost:54320'])
@@ -58,23 +59,22 @@ class WalletCommands(CommandTestCase):
self.assertEqual(status['wallet']['servers'][0]['port'], 54320)
async def test_sending_to_scripthash_address(self):
self.assertEqual(await self.blockchain.get_balance(), '95.99973580')
bal = await self.blockchain.get_balance()
await self.assertBalance(self.account, '10.0')
p2sh_address1 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
tx = await self.account_send('2.0', p2sh_address1)
self.assertEqual(tx['outputs'][0]['address'], p2sh_address1)
self.assertEqual(await self.blockchain.get_balance(), '98.99973580') # +1 lbc for confirm block
self.assertEqual(await self.blockchain.get_balance(), str(float(bal)+3)) # +1 lbc for confirm block
await self.assertBalance(self.account, '7.999877')
await self.wallet_send('3.0', p2sh_address1)
self.assertEqual(await self.blockchain.get_balance(), '102.99973580') # +1 lbc for confirm block
self.assertEqual(await self.blockchain.get_balance(), str(float(bal)+7)) # +1 lbc for confirm block
await self.assertBalance(self.account, '4.999754')
async def test_balance_caching(self):
account2 = await self.daemon.jsonrpc_account_create("Tip-er")
address2 = await self.daemon.jsonrpc_address_unused(account2.id)
sendtxid = await self.blockchain.send_to_address(address2, 10)
await self.confirm_tx(sendtxid)
await self.generate(1)
await self.send_to_address_and_wait(address2, 10, 2)
await self.ledger.tasks_are_done() # don't mess with the query count while we need it
wallet_balance = self.daemon.jsonrpc_wallet_balance
ledger = self.ledger
@@ -89,14 +89,16 @@ class WalletCommands(CommandTestCase):
self.assertIsNone(ledger._balance_cache.get(self.account.id))
query_count += 2
self.assertEqual(await wallet_balance(), expected)
balance = await wallet_balance()
self.assertEqual(self.ledger.db.db.query_count, query_count)
self.assertEqual(balance, expected)
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(self.account.id))['total'], '10.0')
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(account2.id))['total'], '10.0')
# calling again uses cache
self.assertEqual(await wallet_balance(), expected)
balance = await wallet_balance()
self.assertEqual(self.ledger.db.db.query_count, query_count)
self.assertEqual(balance, expected)
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(self.account.id))['total'], '10.0')
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(account2.id))['total'], '10.0')
@@ -122,8 +124,7 @@ class WalletCommands(CommandTestCase):
wallet2 = await self.daemon.jsonrpc_wallet_create('foo', create_account=True)
account3 = wallet2.default_account
address3 = await self.daemon.jsonrpc_address_unused(account3.id, wallet2.id)
await self.confirm_tx(await self.blockchain.send_to_address(address3, 1))
await self.generate(1)
await self.send_to_address_and_wait(address3, 1, 1)
account_balance = self.daemon.jsonrpc_account_balance
wallet_balance = self.daemon.jsonrpc_wallet_balance
@@ -153,7 +154,7 @@ class WalletCommands(CommandTestCase):
address2 = await self.daemon.jsonrpc_address_unused(account2.id)
# send lbc to someone else
tx = await self.daemon.jsonrpc_account_send('1.0', address2)
tx = await self.daemon.jsonrpc_account_send('1.0', address2, blocking=True)
await self.confirm_tx(tx.id)
self.assertEqual(await account_balance(), {
'total': '8.97741',
@@ -186,7 +187,7 @@ class WalletCommands(CommandTestCase):
})
# tip claimed
tx = await self.daemon.jsonrpc_support_abandon(txid=support1['txid'], nout=0)
tx = await self.daemon.jsonrpc_support_abandon(txid=support1['txid'], nout=0, blocking=True)
await self.confirm_tx(tx.id)
self.assertEqual(await account_balance(), {
'total': '9.277303',
@@ -237,8 +238,7 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
"carbon smart garage balance margin twelve"
)
address = (await self.daemon2.wallet_manager.default_account.receiving.get_addresses(limit=1, only_usable=True))[0]
sendtxid = await self.blockchain.send_to_address(address, 1)
await self.confirm_tx(sendtxid, self.daemon2.ledger)
await self.send_to_address_and_wait(address, 1, 1, ledger=self.daemon2.ledger)
def assertWalletEncrypted(self, wallet_path, encrypted):
with open(wallet_path) as opened:
@@ -282,8 +282,19 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
)
# Channel Certificate
channel = await daemon2.jsonrpc_channel_create('@foo', '0.1')
await self.confirm_tx(channel.id, self.daemon2.ledger)
# non-deterministic channel
self.daemon2.wallet_manager.default_account.channel_keys['mqs77XbdnuxWN4cXrjKbSoGLkvAHa4f4B8'] = (
'-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIBZRTZ7tHnYCH3IE9mCo95'
'466L/ShYFhXGrjmSMFJw8eoAcGBSuBBAAK\noUQDQgAEmucoPz9nI+ChZrfhnh'
'0RZ/bcX0r2G0pYBmoNKovtKzXGa8y07D66MWsW\nqXptakqO/9KddIkBu5eJNS'
'UZzQCxPQ==\n-----END EC PRIVATE KEY-----\n'
)
channel = await self.create_nondeterministic_channel('@foo', '0.1', unhexlify(
'3056301006072a8648ce3d020106052b8104000a034200049ae7283f3f6723e0a1'
'66b7e19e1d1167f6dc5f4af61b4a58066a0d2a8bed2b35c66bccb4ec3eba316b16'
'a97a6d6a4a8effd29d748901bb9789352519cd00b13d'
), self.daemon2, blocking=True)
await self.confirm_tx(channel['txid'], self.daemon2.ledger)
# both daemons will have the channel but only one has the cert so far
self.assertItemCount(await daemon.jsonrpc_channel_list(), 1)
@@ -353,6 +364,13 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
self.assertEqual(daemon2.jsonrpc_wallet_status(),
{'is_locked': False, 'is_encrypted': True, 'is_syncing': False})
async def test_locking_unlocking_does_not_break_deterministic_channels(self):
self.assertTrue(self.daemon.jsonrpc_wallet_encrypt("password"))
self.assertTrue(self.daemon.jsonrpc_wallet_lock())
self.account.deterministic_channel_keys._private_key = None
self.assertTrue(self.daemon.jsonrpc_wallet_unlock("password"))
await self.channel_create()
async def test_sync_with_encryption_and_password_change(self):
daemon, daemon2 = self.daemon, self.daemon2
wallet, wallet2 = daemon.wallet_manager.default_wallet, daemon2.wallet_manager.default_wallet
@@ -5,7 +5,6 @@ import lbry.wallet
from lbry.error import ServerPaymentFeeAboveMaxAllowedError
from lbry.wallet.network import ClientSession
from lbry.wallet.rpc import RPCError
from lbry.wallet.server.db.elasticsearch.sync import run_sync, make_es_index
from lbry.wallet.server.session import LBRYElectrumX
from lbry.testcase import IntegrationTestCase, CommandTestCase
from lbry.wallet.orchstr8.node import SPVNode
@@ -25,13 +24,13 @@ class TestSessions(IntegrationTestCase):
)
await session.create_connection()
await session.send_request('server.banner', ())
self.assertEqual(len(self.conductor.spv_node.server.session_mgr.sessions), 1)
self.assertEqual(len(self.conductor.spv_node.server.session_manager.sessions), 1)
self.assertFalse(session.is_closing())
await asyncio.sleep(1.1)
with self.assertRaises(asyncio.TimeoutError):
await session.send_request('server.banner', ())
self.assertTrue(session.is_closing())
self.assertEqual(len(self.conductor.spv_node.server.session_mgr.sessions), 0)
self.assertEqual(len(self.conductor.spv_node.server.session_manager.sessions), 0)
async def test_proper_version(self):
info = await self.ledger.network.get_server_features()
@@ -46,7 +45,7 @@ class TestSessions(IntegrationTestCase):
class TestUsagePayment(CommandTestCase):
async def _test_single_server_payment(self):
async def test_single_server_payment(self):
wallet_pay_service = self.daemon.component_manager.get_component('wallet_server_payments')
wallet_pay_service.payment_period = 1
# only starts with a positive max key fee
@@ -64,7 +63,7 @@ class TestUsagePayment(CommandTestCase):
self.assertEqual(history, [])
node = SPVNode(self.conductor.spv_module, node_number=2)
await node.start(self.blockchain, extraconf={"PAYMENT_ADDRESS": address, "DAILY_FEE": "1.1"})
await node.start(self.blockchain, extraconf={"payment_address": address, "daily_fee": "1.1"})
self.addCleanup(node.stop)
self.daemon.jsonrpc_settings_set('lbryum_servers', [f"{node.hostname}:{node.port}"])
await self.daemon.jsonrpc_wallet_reconnect()
@@ -90,52 +89,77 @@ class TestUsagePayment(CommandTestCase):
class TestESSync(CommandTestCase):
async def test_es_sync_utility(self):
es_writer = self.conductor.spv_node.es_writer
server_search_client = self.conductor.spv_node.server.session_manager.search_index
for i in range(10):
await self.stream_create(f"stream{i}", bid='0.001')
await self.generate(1)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
db = self.conductor.spv_node.server.db
await db.search_index.delete_index()
db.search_index.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
await db.search_index.stop()
self.assertTrue(await make_es_index(db.search_index))
async def resync():
await db.search_index.start()
db.search_index.clear_caches()
await run_sync(index_name=db.search_index.index, db=db)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# delete the index and verify nothing is returned by claim search
await es_writer.delete_index()
server_search_client.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
await resync()
# reindex, 10 claims should be returned
await es_writer.reindex()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
server_search_client.clear_caches()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# this time we will test a migration from unversioned to v1
await db.search_index.sync_client.indices.delete_template(db.search_index.index)
await db.search_index.stop()
self.assertTrue(await make_es_index(db.search_index))
await db.search_index.start()
await resync()
# reindex again, this should not appear to do anything but will delete and reinsert the same 10 claims
await es_writer.reindex()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
server_search_client.clear_caches()
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# delete the index again and stop the writer, upon starting it the writer should reindex automatically
await es_writer.stop(delete_index=True)
server_search_client.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
await es_writer.start(reindex=True)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
# stop the es writer and advance the chain by 1, adding a new claim. upon resuming the es writer, it should
# add the new claim
await es_writer.stop()
await self.stream_create(f"stream11", bid='0.001', confirm=False)
generate_block_task = asyncio.create_task(self.generate(1))
await es_writer.start()
await generate_block_task
self.assertEqual(11, len(await self.claim_search(order_by=['height'])))
# # this time we will test a migration from unversioned to v1
# await db.search_index.sync_client.indices.delete_template(db.search_index.index)
# await db.search_index.stop()
#
# await make_es_index_and_run_sync(env, db=db, index_name=db.search_index.index, force=True)
# await db.search_index.start()
#
# await es_writer.reindex()
# self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
class TestHubDiscovery(CommandTestCase):
async def test_hub_discovery(self):
us_final_node = SPVNode(self.conductor.spv_module, node_number=2)
await us_final_node.start(self.blockchain, extraconf={"COUNTRY": "US"})
await us_final_node.start(self.blockchain, extraconf={"country": "US"})
self.addCleanup(us_final_node.stop)
final_node_host = f"{us_final_node.hostname}:{us_final_node.port}"
kp_final_node = SPVNode(self.conductor.spv_module, node_number=3)
await kp_final_node.start(self.blockchain, extraconf={"COUNTRY": "KP"})
await kp_final_node.start(self.blockchain, extraconf={"country": "KP"})
self.addCleanup(kp_final_node.stop)
kp_final_node_host = f"{kp_final_node.hostname}:{kp_final_node.port}"
relay_node = SPVNode(self.conductor.spv_module, node_number=4)
await relay_node.start(self.blockchain, extraconf={
"COUNTRY": "FR",
"PEER_HUBS": ",".join([kp_final_node_host, final_node_host])
"country": "FR",
"peer_hubs": ",".join([kp_final_node_host, final_node_host])
})
relay_node_host = f"{relay_node.hostname}:{relay_node.port}"
self.addCleanup(relay_node.stop)
@@ -182,7 +206,7 @@ class TestHubDiscovery(CommandTestCase):
self.daemon.ledger.network.client.server_address_and_port, ('127.0.0.1', kp_final_node.port)
)
kp_final_node.server.session_mgr._notify_peer('127.0.0.1:9988')
kp_final_node.server.session_manager._notify_peer('127.0.0.1:9988')
await self.daemon.ledger.network.on_hub.first
await asyncio.sleep(0.5) # wait for above event to be processed by other listeners
self.assertEqual(
+42 -37
View File
@@ -12,18 +12,13 @@ from lbry.error import InsufficientFundsError
from lbry.extras.daemon.daemon import DEFAULT_PAGE_SIZE
from lbry.testcase import CommandTestCase
from lbry.wallet.orchstr8.node import SPVNode
from lbry.wallet.server.db.common import STREAM_TYPES
from lbry.wallet.transaction import Transaction, Output
from lbry.wallet.util import satoshis_to_coins as lbc
from lbry.crypto.hash import sha256
log = logging.getLogger(__name__)
def get_encoded_signature(signature):
signature = signature.encode() if isinstance(signature, str) else signature
r = int(signature[:int(len(signature) / 2)], 16)
s = int(signature[int(len(signature) / 2):], 16)
return ecdsa.util.sigencode_der(r, s, len(signature) * 4)
def verify(channel, data, signature, channel_hash=None):
pieces = [
@@ -32,7 +27,7 @@ def verify(channel, data, signature, channel_hash=None):
data
]
return Output.is_signature_valid(
get_encoded_signature(signature['signature']),
unhexlify(signature['signature']),
sha256(b''.join(pieces)),
channel.claim.channel.public_key_bytes
)
@@ -130,18 +125,6 @@ class ClaimSearchCommand(ClaimTestCase):
with self.assertRaises(ConnectionResetError):
await self.claim_search(claim_ids=claim_ids)
async def test_claim_search_as_reader_server(self):
node2 = SPVNode(self.conductor.spv_module, node_number=2)
current_prefix = self.conductor.spv_node.server.bp.env.es_index_prefix
await node2.start(self.blockchain, extraconf={'ES_MODE': 'reader', 'ES_INDEX_PREFIX': current_prefix})
self.addCleanup(node2.stop)
self.ledger.network.config['default_servers'] = [(node2.hostname, node2.port)]
await self.ledger.stop()
await self.ledger.start()
channel2 = await self.channel_create('@abc', '0.1', allow_duplicate_name=True)
await asyncio.sleep(1) # fixme: find a way to block on the writer
await self.assertFindsClaims([channel2], name='@abc')
async def test_basic_claim_search(self):
await self.create_channel()
channel_txo = self.channel['outputs'][0]
@@ -213,6 +196,10 @@ class ClaimSearchCommand(ClaimTestCase):
await self.assertFindsClaims([three, two], claim_ids=[self.get_claim_id(three), self.get_claim_id(two)])
await self.assertFindsClaims([three], claim_id=self.get_claim_id(three))
await self.assertFindsClaims([three], claim_id=self.get_claim_id(three), text='*')
# resolve by sd hash
two_sd_hash = two['outputs'][0]['value']['source']['sd_hash']
await self.assertFindsClaims([two], sd_hash=two_sd_hash)
await self.assertFindsClaims([two], sd_hash=two_sd_hash[:2])
async def test_source_filter(self):
channel = await self.channel_create('@abc')
@@ -227,6 +214,7 @@ class ClaimSearchCommand(ClaimTestCase):
await self.assertListsClaims([channel_repost, no_source_repost, normal_repost, normal], has_source=True)
await self.assertFindsClaims([channel_repost, no_source_repost, normal_repost, normal, no_source, channel])
await self.assertListsClaims([channel_repost, no_source_repost, normal_repost, normal, no_source, channel])
await self.assertFindsClaims([normal_repost, normal], stream_types=list(STREAM_TYPES.keys()))
async def test_pagination(self):
await self.create_channel()
@@ -494,8 +482,7 @@ class ClaimSearchCommand(ClaimTestCase):
tx = await Transaction.claim_create(
'unknown', b'{"sources":{"lbry_sd_hash":""}}', 1, address, [self.account], self.account)
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
await self.broadcast_and_confirm(tx)
octet = await self.stream_create()
video = await self.stream_create('chrome', file_path=self.video_file_name)
@@ -1121,17 +1108,17 @@ class ChannelCommands(CommandTestCase):
tx = await self.channel_update(claim_id, bid='4.0')
self.assertEqual(tx['outputs'][0]['amount'], '4.0')
await self.assertBalance(self.account, '5.991447')
await self.assertBalance(self.account, '5.991503')
# not enough funds
with self.assertRaisesRegex(
InsufficientFundsError, "Not enough funds to cover this transaction."):
await self.channel_create('@foo2', '9.0')
self.assertItemCount(await self.daemon.jsonrpc_channel_list(), 1)
await self.assertBalance(self.account, '5.991447')
await self.assertBalance(self.account, '5.991503')
# spend exactly amount available, no change
tx = await self.channel_create('@foo3', '5.981266')
tx = await self.channel_create('@foo3', '5.981322')
await self.assertBalance(self.account, '0.0')
self.assertEqual(len(tx['outputs']), 1) # no change
self.assertItemCount(await self.daemon.jsonrpc_channel_list(), 2)
@@ -1226,13 +1213,15 @@ class ChannelCommands(CommandTestCase):
data_to_sign = "CAFEBABE"
# claim new name
await self.channel_create('@someotherchan')
channel_tx = await self.daemon.jsonrpc_channel_create('@signer', '0.1')
channel_tx = await self.daemon.jsonrpc_channel_create('@signer', '0.1', blocking=True)
await self.confirm_tx(channel_tx.id)
channel = channel_tx.outputs[0]
signature1 = await self.out(self.daemon.jsonrpc_channel_sign(channel_name='@signer', hexdata=data_to_sign))
signature2 = await self.out(self.daemon.jsonrpc_channel_sign(channel_id=channel.claim_id, hexdata=data_to_sign))
self.assertTrue(verify(channel, unhexlify(data_to_sign), signature1))
self.assertTrue(verify(channel, unhexlify(data_to_sign), signature2))
signature3 = await self.out(self.daemon.jsonrpc_channel_sign(channel_id=channel.claim_id, hexdata=99))
self.assertTrue(verify(channel, unhexlify('99'), signature3))
async def test_channel_export_import_before_sending_channel(self):
# export
@@ -1247,7 +1236,7 @@ class ChannelCommands(CommandTestCase):
await daemon2.jsonrpc_channel_import(exported_data)
channels = (await daemon2.jsonrpc_channel_list())['items']
self.assertEqual(1, len(channels))
self.assertEqual(channel_private_key.to_string(), channels[0].private_key.to_string())
self.assertEqual(channel_private_key.private_key_bytes, channels[0].private_key.private_key_bytes)
# second wallet can't update until channel is sent to it
with self.assertRaisesRegex(AssertionError, 'Cannot find private key for signing output.'):
@@ -1371,7 +1360,7 @@ class StreamCommands(ClaimTestCase):
self.assertEqual('8.989893', (await self.daemon.jsonrpc_account_balance())['available'])
result = await self.out(self.daemon.jsonrpc_account_send(
'5.0', await self.daemon.jsonrpc_address_unused(account2_id)
'5.0', await self.daemon.jsonrpc_address_unused(account2_id), blocking=True
))
await self.confirm_tx(result['txid'])
@@ -1497,6 +1486,11 @@ class StreamCommands(ClaimTestCase):
self.assertEqual(resolved['@reposting-goodies/repost-on-channel'], search)
self.assertEqual(resolved['newstuff-again']['reposted_claim']['name'], 'newstuff')
await self.stream_update(repost_id, bid='0.42')
searched_repost = (await self.claim_search(claim_id=repost_id))[0]
self.assertEqual(searched_repost['amount'], '0.42')
self.assertEqual(searched_repost['signing_channel']['claim_id'], spam_claim_id)
async def test_filtering_channels_for_removing_content(self):
await self.channel_create('@some_channel', '0.1')
await self.stream_create('good_content', '0.1', channel_name='@some_channel', tags=['good'])
@@ -1507,9 +1501,11 @@ class StreamCommands(ClaimTestCase):
await self.channel_create('@filtering', '0.1')
)
self.conductor.spv_node.server.db.filtering_channel_hashes.add(bytes.fromhex(filtering_channel_id))
self.assertEqual(0, len(self.conductor.spv_node.server.db.filtered_streams))
self.conductor.spv_node.es_writer.db.filtering_channel_hashes.add(bytes.fromhex(filtering_channel_id))
self.assertEqual(0, len(self.conductor.spv_node.es_writer.db.filtered_streams))
await self.stream_repost(bad_content_id, 'filter1', '0.1', channel_name='@filtering')
self.assertEqual(1, len(self.conductor.spv_node.server.db.filtered_streams))
self.assertEqual(1, len(self.conductor.spv_node.es_writer.db.filtered_streams))
# search for filtered content directly
result = await self.out(self.daemon.jsonrpc_claim_search(name='bad_content'))
@@ -1553,16 +1549,17 @@ class StreamCommands(ClaimTestCase):
)
# test setting from env vars and starting from scratch
await self.conductor.spv_node.stop(False)
await self.conductor.spv_node.start(self.conductor.blockchain_node,
extraconf={'BLOCKING_CHANNEL_IDS': blocking_channel_id,
'FILTERING_CHANNEL_IDS': filtering_channel_id})
await self.conductor.spv_node.start(self.conductor.lbcwallet_node,
extraconf={'blocking_channel_ids': [blocking_channel_id],
'filtering_channel_ids': [filtering_channel_id]})
await self.daemon.wallet_manager.reset()
self.assertEqual(0, len(self.conductor.spv_node.server.db.blocked_streams))
self.assertEqual(0, len(self.conductor.spv_node.es_writer.db.blocked_streams))
await self.stream_repost(bad_content_id, 'block1', '0.1', channel_name='@blocking')
self.assertEqual(1, len(self.conductor.spv_node.server.db.blocked_streams))
self.assertEqual(1, len(self.conductor.spv_node.es_writer.db.blocked_streams))
# blocked content is not resolveable
print((await self.resolve('lbry://@some_channel/bad_content')))
error = (await self.resolve('lbry://@some_channel/bad_content'))['error']
self.assertEqual(error['name'], 'BLOCKED')
self.assertTrue(error['text'].startswith("Resolve of 'lbry://@some_channel/bad_content' was censored"))
@@ -1787,13 +1784,21 @@ class StreamCommands(ClaimTestCase):
self.assertEqual(3, len(await self.claim_search(release_time='>0', order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(release_time='>=0', order_by=['release_time'])))
self.assertEqual(4, len(await self.claim_search(order_by=['release_time'])))
self.assertEqual(4, len(await self.claim_search(release_time='<derp', order_by=['release_time'])))
self.assertEqual(3, len(await self.claim_search(claim_type='stream', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(claim_type='channel', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(release_time='>=123456', order_by=['release_time'])))
self.assertEqual(1, len(await self.claim_search(release_time='>123456', order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time='<123457', order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time=['<123457'], order_by=['release_time'])))
self.assertEqual(2, len(await self.claim_search(release_time=['>0', '<123457'], order_by=['release_time'])))
self.assertEqual(
2, len(await self.claim_search(release_time=['>=123097', '<123457'], order_by=['release_time']))
)
self.assertEqual(
2, len(await self.claim_search(release_time=['<123457', '>0'], order_by=['release_time']))
)
async def test_setting_fee_fields(self):
tx = await self.out(self.stream_create('paid-stream'))
txo = tx['outputs'][0]
@@ -2162,7 +2167,7 @@ class SupportCommands(CommandTestCase):
tip = await self.out(
self.daemon.jsonrpc_support_create(
claim_id, '1.0', True, account_id=account2.id, wallet_id='wallet2',
funding_account_ids=[account2.id])
funding_account_ids=[account2.id], blocking=True)
)
await self.confirm_tx(tip['txid'])
@@ -2194,7 +2199,7 @@ class SupportCommands(CommandTestCase):
support = await self.out(
self.daemon.jsonrpc_support_create(
claim_id, '2.0', False, account_id=account2.id, wallet_id='wallet2',
funding_account_ids=[account2.id])
funding_account_ids=[account2.id], blocking=True)
)
await self.confirm_tx(support['txid'])
@@ -1,11 +1,14 @@
import unittest
from unittest import skipIf
import asyncio
import os
from binascii import hexlify
from lbry.schema import Claim
from lbry.stream.background_downloader import BackgroundDownloader
from lbry.stream.descriptor import StreamDescriptor
from lbry.testcase import CommandTestCase
from lbry.extras.daemon.components import TorrentSession
from lbry.extras.daemon.components import TorrentSession, BACKGROUND_DOWNLOADER_COMPONENT
from lbry.wallet import Transaction
@@ -34,8 +37,7 @@ class FileCommands(CommandTestCase):
tx_to_update.outputs[0], claim, 1, address, [self.account], self.account
)
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
await self.broadcast_and_confirm(tx)
self.client_session = self.daemon.file_manager.source_managers['torrent'].torrent_session
self.client_session._session.add_dht_node(('localhost', 4040))
self.client_session.wait_start = False # fixme: this is super slow on tests
@@ -105,18 +107,12 @@ class FileCommands(CommandTestCase):
self.assertEqual(await self.daemon.storage.get_blobs_to_announce(), [])
await self.stream_create('foo', '0.01')
stream = (await self.daemon.jsonrpc_file_list())["items"][0]
self.assertSetEqual(
set(await self.daemon.storage.get_blobs_to_announce()),
{stream.sd_hash, stream.descriptor.blobs[0].blob_hash}
)
self.assertSetEqual(set(await self.daemon.storage.get_blobs_to_announce()), {stream.sd_hash})
self.assertTrue(await self.daemon.jsonrpc_file_delete(delete_all=True))
# announces on download
self.assertEqual(await self.daemon.storage.get_blobs_to_announce(), [])
stream = await self.daemon.jsonrpc_get('foo')
self.assertSetEqual(
set(await self.daemon.storage.get_blobs_to_announce()),
{stream.sd_hash, stream.descriptor.blobs[0].blob_hash}
)
self.assertSetEqual(set(await self.daemon.storage.get_blobs_to_announce()), {stream.sd_hash})
async def _purge_file(self, claim_name, full_path):
self.assertTrue(
@@ -220,6 +216,7 @@ class FileCommands(CommandTestCase):
await self.wait_files_to_complete()
self.assertNotEqual(first_path, second_path)
@unittest.SkipTest # FIXME: claimname/updateclaim is gone. #3480 wip, unblock #3479"
async def test_file_list_updated_metadata_on_resolve(self):
await self.stream_create('foo', '0.01')
txo = (await self.daemon.resolve(self.wallet.accounts, ['lbry://foo']))['lbry://foo']
@@ -322,10 +319,11 @@ class FileCommands(CommandTestCase):
tx = await self.stream_create('foo', '0.01', data=b'deadbeef' * 1000000)
sd_hash = tx['outputs'][0]['value']['source']['sd_hash']
file_info = (await self.file_list())[0]
await self.daemon.jsonrpc_file_delete(claim_name='foo')
blobs = await self.server_storage.get_blobs_for_stream(
await self.server_storage.get_stream_hash_for_sd_hash(sd_hash)
blobs = await self.daemon.storage.get_blobs_for_stream(
await self.daemon.storage.get_stream_hash_for_sd_hash(sd_hash)
)
await self.daemon.jsonrpc_file_delete(claim_name='foo')
self.assertEqual(5, len(blobs))
all_except_sd_and_head = [
blob.blob_hash for blob in blobs[1:-1]
]
@@ -342,10 +340,11 @@ class FileCommands(CommandTestCase):
async def test_incomplete_downloads_retry(self):
tx = await self.stream_create('foo', '0.01', data=b'deadbeef' * 1000000)
sd_hash = tx['outputs'][0]['value']['source']['sd_hash']
await self.daemon.jsonrpc_file_delete(claim_name='foo')
blobs = await self.server_storage.get_blobs_for_stream(
await self.server_storage.get_stream_hash_for_sd_hash(sd_hash)
blobs = await self.daemon.storage.get_blobs_for_stream(
await self.daemon.storage.get_stream_hash_for_sd_hash(sd_hash)
)
self.assertEqual(5, len(blobs))
await self.daemon.jsonrpc_file_delete(claim_name='foo')
all_except_sd_and_head = [
blob.blob_hash for blob in blobs[1:-1]
]
@@ -363,7 +362,6 @@ class FileCommands(CommandTestCase):
self.assertNotIn('error', resp)
self.assertItemCount(await self.daemon.jsonrpc_file_list(), 1)
self.assertEqual('running', (await self.file_list())[0]['status'])
await self.daemon.jsonrpc_file_set_status('stop', claim_name='foo')
# recover blobs
for blob_hash in all_except_sd_and_head:
@@ -372,7 +370,6 @@ class FileCommands(CommandTestCase):
self.server_blob_manager.blobs.clear()
await self.server_blob_manager.blob_completed(self.server_blob_manager.get_blob(blob_hash))
await self.daemon.jsonrpc_file_set_status('start', claim_name='foo')
await asyncio.wait_for(self.wait_files_to_complete(), timeout=5)
file_info = (await self.file_list())[0]
self.assertEqual(file_info['blobs_completed'], file_info['blobs_in_stream'])
@@ -508,8 +505,7 @@ class FileCommands(CommandTestCase):
tx.outputs[0].claim.stream.fee.address_bytes = b''
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
await self.broadcast_and_confirm(tx)
async def __raw_value_update_no_fee_amount(self, tx, claim_address):
tx = await self.daemon.jsonrpc_stream_update(
@@ -519,8 +515,7 @@ class FileCommands(CommandTestCase):
tx.outputs[0].claim.stream.fee.message.ClearField('amount')
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
await self.broadcast_and_confirm(tx)
class DiskSpaceManagement(CommandTestCase):
@@ -535,7 +530,7 @@ class DiskSpaceManagement(CommandTestCase):
async def test_file_management(self):
status = await self.status()
self.assertIn('disk_space', status)
self.assertEqual('0', status['disk_space']['space_used'])
self.assertEqual(0, status['disk_space']['total_used_mb'])
self.assertEqual(True, status['disk_space']['running'])
sd_hash1, blobs1 = await self.get_referenced_blobs(
await self.stream_create('foo1', '0.01', data=('0' * 2 * 1024 * 1024).encode())
@@ -553,21 +548,108 @@ class DiskSpaceManagement(CommandTestCase):
await self.daemon.storage.update_blob_ownership(sd_hash1, False)
await self.daemon.storage.update_blob_ownership(sd_hash3, False)
await self.daemon.storage.update_blob_ownership(sd_hash4, False)
await self.blob_clean() # just to refresh caches, has no effect
self.assertEqual('10', (await self.status())['disk_space']['space_used'])
self.assertEqual(7, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(10, (await self.status())['disk_space']['total_used_mb'])
self.assertEqual(blobs1 | blobs2 | blobs3 | blobs4, set(await self.blob_list()))
await self.blob_clean()
self.assertEqual('10', (await self.status())['disk_space']['space_used'])
self.assertEqual(10, (await self.status())['disk_space']['total_used_mb'])
self.assertEqual(7, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(3, (await self.status())['disk_space']['published_blobs_storage_used_mb'])
self.assertEqual(blobs1 | blobs2 | blobs3 | blobs4, set(await self.blob_list()))
self.daemon.conf.blob_storage_limit = 6
await self.blob_clean()
self.assertEqual('5', (await self.status())['disk_space']['space_used'])
self.assertEqual(5, (await self.status())['disk_space']['total_used_mb'])
self.assertEqual(2, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(3, (await self.status())['disk_space']['published_blobs_storage_used_mb'])
blobs = set(await self.blob_list())
self.assertFalse(blobs1.issubset(blobs))
self.assertTrue(blobs2.issubset(blobs))
self.assertFalse(blobs3.issubset(blobs))
self.assertTrue(blobs4.issubset(blobs))
class TestBackgroundDownloaderComponent(CommandTestCase):
async def get_blobs_from_sd_blob(self, sd_blob):
descriptor = await StreamDescriptor.from_stream_descriptor_blob(
asyncio.get_running_loop(), self.daemon.blob_manager.blob_dir, sd_blob
)
return descriptor.blobs
async def assertBlobs(self, *sd_hashes, no_files=True):
# checks that we have ony the finished blobs needed for the the referenced streams
seen = set(sd_hashes)
for sd_hash in sd_hashes:
sd_blob = self.daemon.blob_manager.get_blob(sd_hash)
self.assertTrue(sd_blob.get_is_verified())
blobs = await self.get_blobs_from_sd_blob(sd_blob)
for blob in blobs[:-1]:
self.assertTrue(self.daemon.blob_manager.get_blob(blob.blob_hash).get_is_verified())
seen.update(blob.blob_hash for blob in blobs if blob.blob_hash)
if no_files:
self.assertEqual(seen, self.daemon.blob_manager.completed_blob_hashes)
self.assertEqual(0, len(await self.file_list()))
async def clear(self):
await self.daemon.jsonrpc_file_delete(delete_all=True)
self.assertEqual(0, len(await self.file_list()))
await self.daemon.blob_manager.delete_blobs(list(self.daemon.blob_manager.completed_blob_hashes), True)
self.assertEqual(0, len((await self.daemon.jsonrpc_blob_list())['items']))
async def test_download(self):
content1 = await self.stream_create('content1', '0.01', data=bytes([0] * 32 * 1024 * 1024))
content1 = content1['outputs'][0]['value']['source']['sd_hash']
content2 = await self.stream_create('content2', '0.01', data=bytes([0] * 16 * 1024 * 1024))
content2 = content2['outputs'][0]['value']['source']['sd_hash']
self.assertEqual(48, (await self.status())['disk_space']['published_blobs_storage_used_mb'])
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
background_downloader = BackgroundDownloader(self.daemon.conf, self.daemon.storage, self.daemon.blob_manager)
self.daemon.conf.network_storage_limit = 32
await self.clear()
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['total_used_mb'])
await background_downloader.download_blobs(content1)
await self.assertBlobs(content1)
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(32, (await self.status())['disk_space']['seed_blobs_storage_used_mb'])
self.daemon.conf.network_storage_limit = 48
await background_downloader.download_blobs(content2)
await self.assertBlobs(content1, content2)
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(48, (await self.status())['disk_space']['seed_blobs_storage_used_mb'])
await self.clear()
await background_downloader.download_blobs(content2)
await self.assertBlobs(content2)
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(16, (await self.status())['disk_space']['seed_blobs_storage_used_mb'])
# tests that an attempt to download something that isn't a sd blob will download the single blob and stop
blobs = await self.get_blobs_from_sd_blob(self.reflector.blob_manager.get_blob(content1))
await self.clear()
await background_downloader.download_blobs(blobs[0].blob_hash)
self.assertEqual({blobs[0].blob_hash}, self.daemon.blob_manager.completed_blob_hashes)
# test that disk space manager doesn't delete orphan network blobs
await background_downloader.download_blobs(content1)
await self.daemon.storage.db.execute_fetchall("update blob set added_on=0") # so it is preferred for cleaning
await self.daemon.jsonrpc_get("content2", save_file=False)
while (await self.file_list())[0]['status'] != 'stopped':
await asyncio.sleep(0.5)
await self.assertBlobs(content1, no_files=False)
self.daemon.conf.blob_storage_limit = 1
await self.blob_clean()
await self.assertBlobs(content1, no_files=False)
self.daemon.conf.network_storage_limit = 0
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['seed_blobs_storage_used_mb'])
+8 -7
View File
@@ -80,7 +80,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# After some soul searching Chris decides that his story needs more
# heart and a better ending. He takes down the story and begins the rewrite.
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(claim_id, blocking=False))
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(claim_id, blocking=True))
self.assertEqual(abandon['inputs'][0]['claim_id'], claim_id)
await self.confirm_tx(abandon['txid'])
@@ -103,7 +103,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# 1 LBC to which Chris readily obliges
ramsey_account_id = (await self.out(self.daemon.jsonrpc_account_create("Ramsey")))['id']
ramsey_address = await self.daemon.jsonrpc_address_unused(ramsey_account_id)
result = await self.out(self.daemon.jsonrpc_account_send('1.0', ramsey_address))
result = await self.out(self.daemon.jsonrpc_account_send('1.0', ramsey_address, blocking=True))
self.assertIn("txid", result)
await self.confirm_tx(result['txid'])
@@ -133,7 +133,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# And voila, and bravo and encore! His Best Friend Ramsey read the story and immediately knew this was a hit
# Now to keep this claim winning on the lbry blockchain he immediately supports the claim
tx = await self.out(self.daemon.jsonrpc_support_create(
claim_id2, '0.2', account_id=ramsey_account_id
claim_id2, '0.2', account_id=ramsey_account_id, blocking=True
))
await self.confirm_tx(tx['txid'])
@@ -147,7 +147,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# Now he also wanted to support the original creator of the Award Winning Novel
# So he quickly decides to send a tip to him
tx = await self.out(
self.daemon.jsonrpc_support_create(claim_id2, '0.3', tip=True, account_id=ramsey_account_id)
self.daemon.jsonrpc_support_create(claim_id2, '0.3', tip=True, account_id=ramsey_account_id, blocking=True)
)
await self.confirm_tx(tx['txid'])
@@ -158,7 +158,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
await self.generate(5)
# Seeing the ravishing success of his novel Chris adds support to his claim too
tx = await self.out(self.daemon.jsonrpc_support_create(claim_id2, '0.4'))
tx = await self.out(self.daemon.jsonrpc_support_create(claim_id2, '0.4', blocking=True))
await self.confirm_tx(tx['txid'])
# And check if his support showed up
@@ -183,7 +183,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# But sadly Ramsey wasn't so pleased. It was hard for him to tell Chris...
# Chris, though a bit heartbroken, abandoned the claim for now, but instantly started working on new hit lyrics
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(txid=tx['txid'], nout=0, blocking=False))
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(txid=tx['txid'], nout=0, blocking=True))
self.assertTrue(abandon['inputs'][0]['txid'], tx['txid'])
await self.confirm_tx(abandon['txid'])
@@ -199,5 +199,6 @@ class EpicAdventuresOfChris45(CommandTestCase):
# He closes and opens the wallet server databases to see how horribly they break
db = self.conductor.spv_node.server.db
db.close()
await db.open_dbs()
db.open_db()
await db.initialize_caches()
# They didn't! (error would be AssertionError: 276 vs 266 (264 counts) on startup)
+10 -2
View File
@@ -1,4 +1,6 @@
import contextlib
import os
import tempfile
from io import StringIO
from lbry.testcase import AsyncioTestCase
@@ -8,7 +10,7 @@ from lbry.extras.daemon.components import (
DATABASE_COMPONENT, DISK_SPACE_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT,
HASH_ANNOUNCER_COMPONENT, FILE_MANAGER_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT,
UPNP_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, WALLET_SERVER_PAYMENTS_COMPONENT,
LIBTORRENT_COMPONENT
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT
)
from lbry.extras.daemon.daemon import Daemon
@@ -24,7 +26,7 @@ class CLIIntegrationTest(AsyncioTestCase):
DATABASE_COMPONENT, DISK_SPACE_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT,
HASH_ANNOUNCER_COMPONENT, FILE_MANAGER_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT,
UPNP_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, WALLET_SERVER_PAYMENTS_COMPONENT,
LIBTORRENT_COMPONENT
LIBTORRENT_COMPONENT, BACKGROUND_DOWNLOADER_COMPONENT
)
Daemon.component_attributes = {}
self.daemon = Daemon(conf)
@@ -37,3 +39,9 @@ class CLIIntegrationTest(AsyncioTestCase):
cli.main(["--api", "localhost:5299", "status"])
actual_output = actual_output.getvalue()
self.assertIn("is_running", actual_output)
def test_when_download_dir_non_writable_on_start_then_daemon_dies_with_helpful_msg(self):
with tempfile.TemporaryDirectory() as download_dir:
os.chmod(download_dir, mode=0o555) # makes download dir non-writable, readable and executable
with self.assertRaisesRegex(PermissionError, f"The following directory is not writable: {download_dir}"):
cli.main(["start", "--download-dir", download_dir])
@@ -1,6 +1,7 @@
import asyncio
import json
import hashlib
import sys
from bisect import bisect_right
from binascii import hexlify, unhexlify
from collections import defaultdict
@@ -20,116 +21,162 @@ class ClaimStateValue(NamedTuple):
class BaseResolveTestCase(CommandTestCase):
def assertMatchESClaim(self, claim_from_es, claim_from_db):
self.assertEqual(claim_from_es['claim_hash'][::-1].hex(), claim_from_db.claim_hash.hex())
self.assertEqual(claim_from_es['claim_id'], claim_from_db.claim_hash.hex())
self.assertEqual(claim_from_es['activation_height'], claim_from_db.activation_height, f"es height: {claim_from_es['activation_height']}, rocksdb height: {claim_from_db.activation_height}")
self.assertEqual(claim_from_es['last_take_over_height'], claim_from_db.last_takeover_height)
self.assertEqual(claim_from_es['tx_id'], claim_from_db.tx_hash[::-1].hex())
self.assertEqual(claim_from_es['tx_nout'], claim_from_db.position)
self.assertEqual(claim_from_es['amount'], claim_from_db.amount)
self.assertEqual(claim_from_es['effective_amount'], claim_from_db.effective_amount)
def assertMatchDBClaim(self, expected, claim):
self.assertEqual(expected['claimid'], claim.claim_hash.hex())
self.assertEqual(expected['validatheight'], claim.activation_height)
self.assertEqual(expected['lasttakeoverheight'], claim.last_takeover_height)
self.assertEqual(expected['txid'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['n'], claim.position)
self.assertEqual(expected['amount'], claim.amount)
self.assertEqual(expected['effectiveamount'], claim.effective_amount)
async def assertResolvesToClaimId(self, name, claim_id):
other = await self.resolve(name)
if claim_id is None:
self.assertIn('error', other)
self.assertEqual(other['error']['name'], 'NOT_FOUND')
claims_from_es = (await self.conductor.spv_node.server.bp.db.search_index.search(name=name))[0]
claims_from_es = (await self.conductor.spv_node.server.session_manager.search_index.search(name=name))[0]
claims_from_es = [c['claim_hash'][::-1].hex() for c in claims_from_es]
self.assertNotIn(claim_id, claims_from_es)
else:
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(claim_id=claim_id)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(claim_id=claim_id)
self.assertEqual(claim_id, other['claim_id'])
self.assertEqual(claim_id, claim_from_es[0][0]['claim_hash'][::-1].hex())
async def assertNoClaimForName(self, name: str):
lbrycrd_winning = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.bp.db.resolve(name)
self.assertNotIn('claimId', lbrycrd_winning)
lbrycrd_winning = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.db.resolve(name)
if 'claims' in lbrycrd_winning and lbrycrd_winning['claims'] is not None:
self.assertEqual(len(lbrycrd_winning['claims']), 0)
if stream is not None:
self.assertIsInstance(stream, LookupError)
else:
self.assertIsInstance(channel, LookupError)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(name=name)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(name=name)
self.assertListEqual([], claim_from_es[0])
async def assertNoClaim(self, claim_id: str):
self.assertDictEqual(
{}, json.loads(await self.blockchain._cli_cmnd('getclaimbyid', claim_id))
)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(claim_id=claim_id)
async def assertNoClaim(self, name: str, claim_id: str):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebyid', name, '["' + claim_id + '"]'))
if 'claims' in expected and expected['claims'] is not None:
# ensure that if we do have the matching claim that it is not active
self.assertEqual(expected['claims'][0]['effectiveamount'], 0)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(claim_id=claim_id)
self.assertListEqual([], claim_from_es[0])
claim = await self.conductor.spv_node.server.bp.db.fs_getclaimbyid(claim_id)
claim = await self.conductor.spv_node.server.db.fs_getclaimbyid(claim_id)
self.assertIsNone(claim)
async def assertMatchWinningClaim(self, name):
expected = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.bp.db.resolve(name)
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebybid', name, "[0]"))
stream, channel, _, _ = await self.conductor.spv_node.server.db.resolve(name)
claim = stream if stream else channel
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
expected['claims'][0]['lasttakeoverheight'] = expected['lasttakeoverheight']
await self._assertMatchClaim(expected['claims'][0], claim)
return claim
async def _assertMatchClaim(self, expected, claim):
self.assertMatchDBClaim(expected, claim)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(
claim_id=claim.claim_hash.hex()
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertMatchESClaim(claim_from_es[0][0], claim)
self._check_supports(claim.claim_hash.hex(), expected.get('supports', []),
claim_from_es[0][0]['support_amount'])
async def assertMatchClaim(self, name, claim_id, is_active_in_lbrycrd=True):
claim = await self.conductor.spv_node.server.db.fs_getclaimbyid(claim_id)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(
claim_id=claim.claim_hash.hex()
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim.claim_hash.hex())
self.assertEqual(expected['claimId'], claim.claim_hash.hex())
self.assertEqual(expected['validAtHeight'], claim.activation_height)
self.assertEqual(expected['lastTakeoverHeight'], claim.last_takeover_height)
self.assertEqual(expected['txId'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['n'], claim.position)
self.assertEqual(expected['amount'], claim.amount)
self.assertEqual(expected['effectiveAmount'], claim.effective_amount)
return claim
self.assertMatchESClaim(claim_from_es[0][0], claim)
async def assertMatchClaim(self, claim_id, is_active_in_lbrycrd=True):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimbyid', claim_id))
claim = await self.conductor.spv_node.server.bp.db.fs_getclaimbyid(claim_id)
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebyid', name, '["' + claim_id + '"]'))
if is_active_in_lbrycrd:
if not expected:
self.assertIsNone(claim)
return
self.assertEqual(expected['claimId'], claim.claim_hash.hex())
self.assertEqual(expected['validAtHeight'], claim.activation_height)
self.assertEqual(expected['lastTakeoverHeight'], claim.last_takeover_height)
self.assertEqual(expected['txId'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['n'], claim.position)
self.assertEqual(expected['amount'], claim.amount)
self.assertEqual(expected['effectiveAmount'], claim.effective_amount)
expected['claims'][0]['lasttakeoverheight'] = expected['lasttakeoverheight']
self.assertMatchDBClaim(expected['claims'][0], claim)
self._check_supports(claim.claim_hash.hex(), expected['claims'][0].get('supports', []),
claim_from_es[0][0]['support_amount'])
else:
self.assertDictEqual({}, expected)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=claim.claim_hash.hex()
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim.claim_hash.hex())
self.assertEqual(claim_from_es[0][0]['claim_id'], claim.claim_hash.hex())
self.assertEqual(claim_from_es[0][0]['activation_height'], claim.activation_height)
self.assertEqual(claim_from_es[0][0]['last_take_over_height'], claim.last_takeover_height)
self.assertEqual(claim_from_es[0][0]['tx_id'], claim.tx_hash[::-1].hex())
self.assertEqual(claim_from_es[0][0]['tx_nout'], claim.position)
self.assertEqual(claim_from_es[0][0]['amount'], claim.amount)
self.assertEqual(claim_from_es[0][0]['effective_amount'], claim.effective_amount)
if 'claims' in expected and expected['claims'] is not None:
# ensure that if we do have the matching claim that it is not active
self.assertEqual(expected['claims'][0]['effectiveamount'], 0)
return claim
async def assertMatchClaimIsWinning(self, name, claim_id):
self.assertEqual(claim_id, (await self.assertMatchWinningClaim(name)).claim_hash.hex())
await self.assertMatchClaim(claim_id)
await self.assertMatchClaimsForName(name)
def _check_supports(self, claim_id, lbrycrd_supports, es_support_amount):
total_lbrycrd_amount = 0.0
total_es_amount = 0.0
active_es_amount = 0.0
db = self.conductor.spv_node.server.db
es_supports = db.get_supports(bytes.fromhex(claim_id))
# we're only concerned about active supports here, and they should match
self.assertTrue(len(es_supports) >= len(lbrycrd_supports))
for i, (tx_num, position, amount) in enumerate(es_supports):
total_es_amount += amount
valid_height = db.get_activation(tx_num, position, is_support=True)
if valid_height > db.db_height:
continue
active_es_amount += amount
txid = db.prefix_db.tx_hash.get(tx_num, deserialize_value=False)[::-1].hex()
support = next(filter(lambda s: s['txid'] == txid and s['n'] == position, lbrycrd_supports))
total_lbrycrd_amount += support['amount']
self.assertEqual(support['height'], bisect_right(db.tx_counts, tx_num))
self.assertEqual(support['validatheight'], valid_height)
self.assertEqual(total_es_amount, es_support_amount)
self.assertEqual(active_es_amount, total_lbrycrd_amount)
async def assertMatchClaimsForName(self, name):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name))
db = self.conductor.spv_node.server.bp.db
def check_supports(claim_id, lbrycrd_supports):
for i, (tx_num, position, amount) in enumerate(db.get_supports(bytes.fromhex(claim_id))):
support = lbrycrd_supports[i]
self.assertEqual(support['txId'], db.prefix_db.tx_hash.get(tx_num, deserialize_value=False)[::-1].hex())
self.assertEqual(support['n'], position)
self.assertEqual(support['height'], bisect_right(db.tx_counts, tx_num))
self.assertEqual(support['validAtHeight'], db.get_activation(tx_num, position, is_support=True))
# self.assertEqual(len(expected['claims']), len(db_claims.claims))
# self.assertEqual(expected['lastTakeoverHeight'], db_claims.lastTakeoverHeight)
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name, "", "true"))
db = self.conductor.spv_node.server.db
for c in expected['claims']:
check_supports(c['claimId'], c['supports'])
claim_hash = bytes.fromhex(c['claimId'])
self.assertEqual(c['validAtHeight'], db.get_activation(
db.prefix_db.tx_num.get(bytes.fromhex(c['txId'])[::-1]).tx_num, c['n']
))
self.assertEqual(c['effectiveAmount'], db.get_effective_amount(claim_hash))
c['lasttakeoverheight'] = expected['lasttakeoverheight']
claim_id = c['claimid']
claim_hash = bytes.fromhex(claim_id)
claim = db._fs_get_claim_by_hash(claim_hash)
self.assertMatchDBClaim(c, claim)
claim_from_es = await self.conductor.spv_node.server.session_manager.search_index.search(
claim_id=claim_id
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim_id)
self.assertMatchESClaim(claim_from_es[0][0], claim)
self._check_supports(claim_id, c.get('supports', []),
claim_from_es[0][0]['support_amount'])
async def assertNameState(self, height: int, name: str, winning_claim_id: str, last_takeover_height: int,
non_winning_claims: List[ClaimStateValue]):
self.assertEqual(height, self.conductor.spv_node.server.db.db_height)
await self.assertMatchClaimIsWinning(name, winning_claim_id)
for non_winning in non_winning_claims:
claim = await self.assertMatchClaim(
name, non_winning.claim_id, is_active_in_lbrycrd=non_winning.active_in_lbrycrd
)
self.assertEqual(non_winning.activation_height, claim.activation_height)
self.assertEqual(last_takeover_height, claim.last_takeover_height)
class ResolveCommand(BaseResolveTestCase):
@@ -153,11 +200,17 @@ class ResolveCommand(BaseResolveTestCase):
await self.assertResolvesToClaimId(
f'@abc#{colliding_claim_ids[0][:1]}', first_claim
)
collision_depth = 0
for c1, c2 in zip(colliding_claim_ids[0], colliding_claim_ids[1]):
if c1 == c2:
collision_depth += 1
else:
break
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[0][:2]}', colliding_claim_ids[0])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[0][:7]}', colliding_claim_ids[0])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[0][:17]}', colliding_claim_ids[0])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[0]}', colliding_claim_ids[0])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[1][:3]}', colliding_claim_ids[1])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[1][:collision_depth + 1]}', colliding_claim_ids[1])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[1][:7]}', colliding_claim_ids[1])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[1][:17]}', colliding_claim_ids[1])
await self.assertResolvesToClaimId(f'@abc#{colliding_claim_ids[1]}', colliding_claim_ids[1])
@@ -235,19 +288,20 @@ class ResolveCommand(BaseResolveTestCase):
tx_details = await self.blockchain.get_raw_transaction(claim['txid'])
self.assertEqual(claim['confirmations'], json.loads(tx_details)['confirmations'])
# FIXME : claimname/updateclaim is gone. #3480 wip, unblock #3479"
# resolve handles invalid data
await self.blockchain_claim_name("gibberish", hexlify(b"{'invalid':'json'}").decode(), "0.1")
await self.generate(1)
response = await self.out(self.daemon.jsonrpc_resolve("lbry://gibberish"))
self.assertSetEqual({'lbry://gibberish'}, set(response))
claim = response['lbry://gibberish']
self.assertEqual(claim['name'], 'gibberish')
self.assertNotIn('value', claim)
# await self.blockchain_claim_name("gibberish", hexlify(b"{'invalid':'json'}").decode(), "0.1")
# await self.generate(1)
# response = await self.out(self.daemon.jsonrpc_resolve("lbry://gibberish"))
# self.assertSetEqual({'lbry://gibberish'}, set(response))
# claim = response['lbry://gibberish']
# self.assertEqual(claim['name'], 'gibberish')
# self.assertNotIn('value', claim)
# resolve retries
await self.conductor.spv_node.stop()
resolve_task = asyncio.create_task(self.resolve('foo'))
await self.conductor.spv_node.start(self.conductor.blockchain_node)
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
self.assertIsNotNone((await resolve_task)['claim_id'])
async def test_winning_by_effective_amount(self):
@@ -417,16 +471,16 @@ class ResolveCommand(BaseResolveTestCase):
self.assertEqual(one, claim6['name'])
async def test_resolve_old_claim(self):
channel = await self.daemon.jsonrpc_channel_create('@olds', '1.0')
channel = await self.daemon.jsonrpc_channel_create('@olds', '1.0', blocking=True)
await self.confirm_tx(channel.id)
address = channel.outputs[0].get_address(self.account.ledger)
claim = generate_signed_legacy(address, channel.outputs[0])
tx = await Transaction.claim_create('example', claim.SerializeToString(), 1, address, [self.account], self.account)
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
await self.broadcast_and_confirm(tx)
response = await self.resolve('@olds/example')
self.assertTrue('is_channel_signature_valid' in response, str(response))
self.assertTrue(response['is_channel_signature_valid'])
claim.publisherSignature.signature = bytes(reversed(claim.publisherSignature.signature))
@@ -434,8 +488,7 @@ class ResolveCommand(BaseResolveTestCase):
'bad_example', claim.SerializeToString(), 1, address, [self.account], self.account
)
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
await self.broadcast_and_confirm(tx)
response = await self.resolve('bad_example')
self.assertFalse(response['is_channel_signature_valid'])
@@ -617,10 +670,10 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
async def assertNameState(self, height: int, name: str, winning_claim_id: str, last_takeover_height: int,
non_winning_claims: List[ClaimStateValue]):
self.assertEqual(height, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(height, self.conductor.spv_node.server.db.db_height)
await self.assertMatchClaimIsWinning(name, winning_claim_id)
for non_winning in non_winning_claims:
claim = await self.assertMatchClaim(
claim = await self.assertMatchClaim(name,
non_winning.claim_id, is_active_in_lbrycrd=non_winning.active_in_lbrycrd
)
self.assertEqual(non_winning.activation_height, claim.activation_height)
@@ -935,7 +988,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
)
greater_than_or_equal_to_zero = [
claim['claim_id'] for claim in (
await self.conductor.spv_node.server.bp.db.search_index.search(
await self.conductor.spv_node.server.session_manager.search_index.search(
channel_id=channel_id, fee_amount=">=0"
))[0]
]
@@ -943,7 +996,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertSetEqual(set(greater_than_or_equal_to_zero), {stream_with_no_fee, stream_with_fee})
greater_than_zero = [
claim['claim_id'] for claim in (
await self.conductor.spv_node.server.bp.db.search_index.search(
await self.conductor.spv_node.server.session_manager.search_index.search(
channel_id=channel_id, fee_amount=">0"
))[0]
]
@@ -951,7 +1004,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertSetEqual(set(greater_than_zero), {stream_with_fee})
equal_to_zero = [
claim['claim_id'] for claim in (
await self.conductor.spv_node.server.bp.db.search_index.search(
await self.conductor.spv_node.server.session_manager.search_index.search(
channel_id=channel_id, fee_amount="<=0"
))[0]
]
@@ -966,10 +1019,10 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
name = 'test'
await self.generate(494)
address = (await self.account.receiving.get_addresses(True))[0]
await self.blockchain.send_to_address(address, 400.0)
await self.send_to_address_and_wait(address, 400.0)
await self.account.ledger.on_address.first
await self.generate(100)
self.assertEqual(800, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(800, self.conductor.spv_node.server.db.db_height)
# Block 801: Claim A for 10 LBC is accepted.
# It is the first claim, so it immediately becomes active and controlling.
@@ -981,10 +1034,10 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
# Its activation height is 1121 + min(4032, floor((1121-801) / 32)) = 1121 + 10 = 1131.
# State: A(10) is controlling, B(20) is accepted.
await self.generate(32 * 10 - 1)
self.assertEqual(1120, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(1120, self.conductor.spv_node.server.db.db_height)
claim_id_B = (await self.stream_create(name, '20.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
claim_B, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_B}")
self.assertEqual(1121, self.conductor.spv_node.server.bp.db.db_height)
claim_B, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_B}")
self.assertEqual(1121, self.conductor.spv_node.server.db.db_height)
self.assertEqual(1131, claim_B.activation_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
@@ -992,33 +1045,33 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
# Since it is a support for the controlling claim, it activates immediately.
# State: A(10+14) is controlling, B(20) is accepted.
await self.support_create(claim_id_A, bid='14.0')
self.assertEqual(1122, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(1122, self.conductor.spv_node.server.db.db_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
# Block 1123: Claim C for 50 LBC is accepted.
# The activation height is 1123 + min(4032, floor((1123-801) / 32)) = 1123 + 10 = 1133.
# State: A(10+14) is controlling, B(20) is accepted, C(50) is accepted.
claim_id_C = (await self.stream_create(name, '50.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
self.assertEqual(1123, self.conductor.spv_node.server.bp.db.db_height)
claim_C, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_C}")
self.assertEqual(1123, self.conductor.spv_node.server.db.db_height)
claim_C, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_C}")
self.assertEqual(1133, claim_C.activation_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
await self.generate(7)
self.assertEqual(1130, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(1130, self.conductor.spv_node.server.db.db_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
await self.generate(1)
# Block 1131: Claim B activates. It has 20 LBC, while claim A has 24 LBC (10 original + 14 from support X). There is no takeover, and claim A remains controlling.
# State: A(10+14) is controlling, B(20) is active, C(50) is accepted.
self.assertEqual(1131, self.conductor.spv_node.server.bp.db.db_height)
self.assertEqual(1131, self.conductor.spv_node.server.db.db_height)
await self.assertMatchClaimIsWinning(name, claim_id_A)
# Block 1132: Claim D for 300 LBC is accepted. The activation height is 1132 + min(4032, floor((1132-801) / 32)) = 1132 + 10 = 1142.
# State: A(10+14) is controlling, B(20) is active, C(50) is accepted, D(300) is accepted.
claim_id_D = (await self.stream_create(name, '300.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
self.assertEqual(1132, self.conductor.spv_node.server.bp.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(1132, self.conductor.spv_node.server.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(False, claim_D.is_controlling)
self.assertEqual(801, claim_D.last_takeover_height)
self.assertEqual(1142, claim_D.activation_height)
@@ -1027,8 +1080,8 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
# Block 1133: Claim C activates. It has 50 LBC, while claim A has 24 LBC, so a takeover is initiated. The takeover height for this name is set to 1133, and therefore the activation delay for all the claims becomes min(4032, floor((1133-1133) / 32)) = 0. All the claims become active. The totals for each claim are recalculated, and claim D becomes controlling because it has the highest total.
# State: A(10+14) is active, B(20) is active, C(50) is active, D(300) is controlling
await self.generate(1)
self.assertEqual(1133, self.conductor.spv_node.server.bp.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.bp.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(1133, self.conductor.spv_node.server.db.db_height)
claim_D, _, _, _ = await self.conductor.spv_node.server.db.resolve(f"{name}:{claim_id_D}")
self.assertEqual(True, claim_D.is_controlling)
self.assertEqual(1133, claim_D.last_takeover_height)
self.assertEqual(1133, claim_D.activation_height)
@@ -1170,10 +1223,12 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertNotEqual(first_claim_id, second_claim_id)
# takeover should not have happened yet
await self.assertMatchClaimIsWinning(name, first_claim_id)
await self.generate(8)
await self.assertMatchClaimIsWinning(name, first_claim_id)
for _ in range(8):
await self.generate(1)
await self.assertMatchClaimIsWinning(name, first_claim_id)
# prevent the takeover by adding a support one block before the takeover happens
await self.support_create(first_claim_id, bid='1.0')
await self.assertMatchClaimIsWinning(name, first_claim_id)
# one more block until activation
await self.generate(1)
await self.assertMatchClaimIsWinning(name, first_claim_id)
@@ -1299,15 +1354,15 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
await self.generate(8)
await self.assertMatchClaimIsWinning(name, first_claim_id)
# abandon the support that causes the winning claim to have the highest staked
tx = await self.daemon.jsonrpc_txo_spend(type='support', txid=controlling_support_tx.id)
tx = await self.daemon.jsonrpc_txo_spend(type='support', txid=controlling_support_tx.id, blocking=True)
await self.generate(1)
await self.assertMatchClaimIsWinning(name, first_claim_id)
# await self.assertMatchClaim(second_claim_id)
await self.assertNameState(538, name, first_claim_id, last_takeover_height=207, non_winning_claims=[
ClaimStateValue(second_claim_id, activation_height=539, active_in_lbrycrd=False)
])
await self.generate(1)
await self.assertMatchClaim(first_claim_id)
await self.assertMatchClaimIsWinning(name, second_claim_id)
await self.assertNameState(539, name, second_claim_id, last_takeover_height=539, non_winning_claims=[
ClaimStateValue(first_claim_id, activation_height=207, active_in_lbrycrd=True)
])
async def test_remove_controlling_support(self):
name = 'derp'
@@ -1316,27 +1371,35 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
first_support_tx = await self.daemon.jsonrpc_support_create(first_claim_id, '0.9')
await self.ledger.wait(first_support_tx)
await self.assertMatchClaimIsWinning(name, first_claim_id)
await self.generate(320) # give the first claim long enough for a 10 block takeover delay
await self.assertMatchClaimIsWinning(name, first_claim_id)
await self.assertNameState(527, name, first_claim_id, last_takeover_height=207, non_winning_claims=[])
# make a second claim which will take over the name
second_claim_id = (await self.stream_create(name, '0.1', allow_duplicate_name=True))['outputs'][0]['claim_id']
self.assertNotEqual(first_claim_id, second_claim_id)
await self.assertNameState(528, name, first_claim_id, last_takeover_height=207, non_winning_claims=[
ClaimStateValue(second_claim_id, activation_height=538, active_in_lbrycrd=False)
])
second_claim_support_tx = await self.daemon.jsonrpc_support_create(second_claim_id, '1.5')
await self.ledger.wait(second_claim_support_tx)
await self.generate(1) # neither the second claim or its support have activated yet
await self.assertMatchClaimIsWinning(name, first_claim_id)
await self.assertNameState(529, name, first_claim_id, last_takeover_height=207, non_winning_claims=[
ClaimStateValue(second_claim_id, activation_height=538, active_in_lbrycrd=False)
])
await self.generate(9) # claim activates, but is not yet winning
await self.assertMatchClaimIsWinning(name, first_claim_id)
await self.assertNameState(538, name, first_claim_id, last_takeover_height=207, non_winning_claims=[
ClaimStateValue(second_claim_id, activation_height=538, active_in_lbrycrd=True)
])
await self.generate(1) # support activates, takeover happens
await self.assertMatchClaimIsWinning(name, second_claim_id)
await self.assertNameState(539, name, second_claim_id, last_takeover_height=539, non_winning_claims=[
ClaimStateValue(first_claim_id, activation_height=207, active_in_lbrycrd=True)
])
await self.daemon.jsonrpc_txo_spend(type='support', claim_id=second_claim_id, blocking=True)
await self.generate(1) # support activates, takeover happens
await self.assertMatchClaimIsWinning(name, first_claim_id)
await self.assertNameState(540, name, first_claim_id, last_takeover_height=540, non_winning_claims=[
ClaimStateValue(second_claim_id, activation_height=538, active_in_lbrycrd=True)
])
async def test_claim_expiration(self):
name = 'derp'
@@ -1369,14 +1432,14 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
await self.generate(32)
second_claim_id = (await self.stream_create(name, '0.01', allow_duplicate_name=True))['outputs'][0]['claim_id']
await self.assertNoClaim(second_claim_id)
await self.assertNoClaim(name, second_claim_id)
self.assertEqual(
len((await self.conductor.spv_node.server.bp.db.search_index.search(claim_name=name))[0]), 1
len((await self.conductor.spv_node.server.session_manager.search_index.search(claim_name=name))[0]), 1
)
await self.generate(1)
await self.assertMatchClaim(second_claim_id)
await self.assertMatchClaim(name, second_claim_id)
self.assertEqual(
len((await self.conductor.spv_node.server.bp.db.search_index.search(claim_name=name))[0]), 2
len((await self.conductor.spv_node.server.session_manager.search_index.search(claim_name=name))[0]), 2
)
async def test_abandon_controlling_same_block_as_new_claim(self):
@@ -1392,74 +1455,80 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
async def test_trending(self):
async def get_trending_score(claim_id):
return (await self.conductor.spv_node.server.bp.db.search_index.search(
return (await self.conductor.spv_node.server.session_manager.search_index.search(
claim_id=claim_id
))[0][0]['trending_score']
claim_id1 = (await self.stream_create('derp', '1.0'))['outputs'][0]['claim_id']
claim_id2 = (await self.stream_create('derp', '1.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
claim_id3 = (await self.stream_create('derp', '1.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
claim_id4 = (await self.stream_create('derp', '1.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
claim_id5 = (await self.stream_create('derp', '1.0', allow_duplicate_name=True))['outputs'][0]['claim_id']
COIN = int(1E8)
COIN = 1E9
height = 99000
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id1, height, True, 1 * COIN, 1_000_000 * COIN
)
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id2, height, True, 1 * COIN, 100_000 * COIN
)
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id2, height + 1, False, 100_001 * COIN, 100_000 * COIN
)
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id3, height, True, 1 * COIN, 1_000 * COIN
)
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id4, height, True, 1 * COIN, 10 * COIN
self.assertEqual(self.conductor.spv_node.writer.height, 207)
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(208, bytes.fromhex(claim_id1)), (0, 10 * COIN)
)
await self.generate(1)
self.assertEqual(self.conductor.spv_node.writer.height, 208)
self.assertEqual(3.1711298570548195e+76, await get_trending_score(claim_id1))
self.assertEqual(-1.369652719234026e+74, await get_trending_score(claim_id2))
self.assertEqual(2.925275298842502e+75, await get_trending_score(claim_id3))
self.assertEqual(5.193711055804491e+74, await get_trending_score(claim_id4))
self.assertEqual(0.6690521635580086, await get_trending_score(claim_id5))
self.conductor.spv_node.server.bp._add_claim_activation_change_notification(
claim_id5, height + 100, True, 2 * COIN, 10 * COIN
self.assertEqual(1.7090807854206793, await get_trending_score(claim_id1))
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(209, bytes.fromhex(claim_id1)), (10 * COIN, 100 * COIN)
)
await self.generate(1)
self.assertEqual(5.664516565750028e+74, await get_trending_score(claim_id5))
self.assertEqual(self.conductor.spv_node.writer.height, 209)
self.assertEqual(2.2437974397778886, await get_trending_score(claim_id1))
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(309, bytes.fromhex(claim_id1)), (100 * COIN, 1000000 * COIN)
)
await self.generate(100)
self.assertEqual(self.conductor.spv_node.writer.height, 309)
self.assertEqual(5.157053472135866, await get_trending_score(claim_id1))
search_results = (await self.conductor.spv_node.server.bp.db.search_index.search(claim_name="derp"))[0]
self.assertEqual(5, len(search_results))
self.assertListEqual([claim_id1, claim_id3, claim_id4, claim_id2, claim_id5], [c['claim_id'] for c in search_results])
self.conductor.spv_node.writer.db.prefix_db.trending_notification.stage_put(
(409, bytes.fromhex(claim_id1)), (1000000 * COIN, 1 * COIN)
)
await self.generate(99)
self.assertEqual(self.conductor.spv_node.writer.height, 408)
self.assertEqual(5.157053472135866, await get_trending_score(claim_id1))
await self.generate(1)
self.assertEqual(self.conductor.spv_node.writer.height, 409)
self.assertEqual(-3.4256156592205627, await get_trending_score(claim_id1))
search_results = (await self.conductor.spv_node.server.session_manager.search_index.search(claim_name="derp"))[0]
self.assertEqual(1, len(search_results))
self.assertListEqual([claim_id1], [c['claim_id'] for c in search_results])
class ResolveAfterReorg(BaseResolveTestCase):
async def reorg(self, start):
blocks = self.ledger.headers.height - start
self.blockchain.block_expected = start - 1
prepare = self.ledger.on_header.where(self.blockchain.is_expected_block)
self.conductor.spv_node.server.synchronized.clear()
# go back to start
await self.blockchain.invalidate_block((await self.ledger.headers.hash(start)).decode())
# go to previous + 1
await self.generate(blocks + 2)
await self.blockchain.generate(blocks + 2)
await prepare # no guarantee that it didn't happen already, so start waiting from before calling generate
await self.conductor.spv_node.server.synchronized.wait()
# await asyncio.wait_for(self.on_header(self.blockchain.block_expected), 30.0)
async def assertBlockHash(self, height):
bp = self.conductor.spv_node.server.bp
reader_db = self.conductor.spv_node.server.db
block_hash = await self.blockchain.get_block_hash(height)
self.assertEqual(block_hash, (await self.ledger.headers.hash(height)).decode())
self.assertEqual(block_hash, (await bp.db.fs_block_hashes(height, 1))[0][::-1].hex())
self.assertEqual(block_hash, (await reader_db.fs_block_hashes(height, 1))[0][::-1].hex())
txids = [
tx_hash[::-1].hex() for tx_hash in bp.db.get_block_txs(height)
tx_hash[::-1].hex() for tx_hash in reader_db.get_block_txs(height)
]
txs = await bp.db.fs_transactions(txids)
block_txs = (await bp.daemon.deserialised_block(block_hash))['tx']
txs = await reader_db.get_transactions_and_merkles(txids)
block_txs = (await self.conductor.spv_node.server.daemon.deserialised_block(block_hash))['tx']
self.assertSetEqual(set(block_txs), set(txs.keys()), msg='leveldb/lbrycrd is missing transactions')
self.assertListEqual(block_txs, list(txs.keys()), msg='leveldb/lbrycrd transactions are of order')
@@ -1470,9 +1539,18 @@ class ResolveAfterReorg(BaseResolveTestCase):
channel_id = self.get_claim_id(
await self.channel_create(channel_name, '0.01')
)
self.assertEqual(channel_id, (await self.assertMatchWinningClaim(channel_name)).claim_hash.hex())
await self.assertNameState(
height=207, name='@abc', winning_claim_id=channel_id, last_takeover_height=207,
non_winning_claims=[]
)
await self.reorg(206)
self.assertEqual(channel_id, (await self.assertMatchWinningClaim(channel_name)).claim_hash.hex())
await self.assertNameState(
height=208, name='@abc', winning_claim_id=channel_id, last_takeover_height=207,
non_winning_claims=[]
)
# await self.assertNoClaimForName(channel_name)
# self.assertNotIn('error', await self.resolve(channel_name))
@@ -1481,16 +1559,29 @@ class ResolveAfterReorg(BaseResolveTestCase):
stream_id = self.get_claim_id(
await self.stream_create(stream_name, '0.01', channel_id=channel_id)
)
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.assertNameState(
height=209, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
await self.reorg(206)
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.assertNameState(
height=210, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
await self.support_create(stream_id, '0.01')
self.assertNotIn('error', await self.resolve(stream_name))
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.assertNameState(
height=211, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
await self.reorg(206)
# self.assertNotIn('error', await self.resolve(stream_name))
self.assertEqual(stream_id, (await self.assertMatchWinningClaim(stream_name)).claim_hash.hex())
await self.assertNameState(
height=212, name=stream_name, winning_claim_id=stream_id, last_takeover_height=209,
non_winning_claims=[]
)
await self.stream_abandon(stream_id)
self.assertNotIn('error', await self.resolve(channel_name))
@@ -1532,7 +1623,6 @@ class ResolveAfterReorg(BaseResolveTestCase):
await self.ledger.wait(broadcast_tx)
await self.support_create(still_valid.outputs[0].claim_id, '0.01')
# await self.generate(1)
await self.ledger.wait(broadcast_tx, self.blockchain.block_expected)
self.assertEqual(self.ledger.headers.height, 208)
await self.assertBlockHash(208)
@@ -1549,7 +1639,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.blockchain.clear_mempool()
await self.conductor.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -1582,7 +1672,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
await self.blockchain.generate(1)
# wait for the client to catch up
await asyncio.wait_for(self.on_header(210), 1.0)
await asyncio.wait_for(self.on_header(210), 3.0)
# verify the claim is in the new block and that it is returned by claim_search
republished = await self.resolve('hovercraft')
@@ -1628,11 +1718,11 @@ class ResolveAfterReorg(BaseResolveTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.blockchain.clear_mempool()
await self.conductor.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
await asyncio.wait_for(self.on_header(209), 3.0)
await asyncio.wait_for(self.on_header(209), 30.0)
await self.assertBlockHash(207)
await self.assertBlockHash(208)
await self.assertBlockHash(209)
@@ -1700,7 +1790,7 @@ def generate_signed_legacy(address: bytes, output: Output):
claim.SerializeToString(),
output.claim_hash[::-1]
]))
signature = output.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
signature = output.private_key.sign_compact(digest)
claim.publisherSignature.version = 1
claim.publisherSignature.signatureType = 1
claim.publisherSignature.signature = signature
@@ -21,9 +21,8 @@ class BasicTransactionTest(IntegrationTestCase):
[asyncio.ensure_future(self.on_address_update(address1)),
asyncio.ensure_future(self.on_address_update(address2))]
))
sendtxid1 = await self.blockchain.send_to_address(address1, 5)
sendtxid2 = await self.blockchain.send_to_address(address2, 5)
await self.blockchain.generate(1)
await self.send_to_address_and_wait(address1, 5)
await self.send_to_address_and_wait(address2, 5, 1)
await notifications
self.assertEqual(d2l(await self.account.get_balance()), '10.0')
@@ -32,7 +31,9 @@ class BasicTransactionTest(IntegrationTestCase):
channel_txo = Output.pay_claim_name_pubkey_hash(
l2d('1.0'), '@bar', channel, self.account.ledger.address_to_hash160(address1)
)
await channel_txo.generate_channel_private_key()
channel_txo.set_channel_private_key(
await self.account.generate_channel_private_key()
)
channel_txo.script.generate()
channel_tx = await Transaction.create([], [channel_txo], [self.account], self.account)
@@ -55,7 +56,7 @@ class BasicTransactionTest(IntegrationTestCase):
notifications = asyncio.create_task(asyncio.wait(
[asyncio.ensure_future(self.ledger.wait(channel_tx)), asyncio.ensure_future(self.ledger.wait(stream_tx))]
))
await self.blockchain.generate(1)
await self.generate(1)
await notifications
self.assertEqual(d2l(await self.account.get_balance()), '7.985786')
self.assertEqual(d2l(await self.account.get_balance(include_claims=True)), '9.985786')
@@ -68,7 +69,7 @@ class BasicTransactionTest(IntegrationTestCase):
await self.broadcast(abandon_tx)
await notify
notify = asyncio.create_task(self.ledger.wait(abandon_tx))
await self.blockchain.generate(1)
await self.generate(1)
await notify
response = await self.ledger.resolve([], ['lbry://@bar/foo'])
@@ -1,12 +1,25 @@
import unittest
from lbry.testcase import CommandTestCase
class TransactionCommandsTestCase(CommandTestCase):
async def test_txo_dust_prevention(self):
address = await self.daemon.jsonrpc_address_unused(self.account.id)
tx = await self.account_send('9.9997758', address)
# dust prevention threshold not reached, small txo created
self.assertEqual(2, len(tx['outputs']))
self.assertEqual(tx['outputs'][1]['amount'], '0.0001002')
tx = await self.account_send('9.999706', address)
# dust prevention prevented dust
self.assertEqual(1, len(tx['outputs']))
self.assertEqual(tx['outputs'][0]['amount'], '9.999706')
async def test_transaction_show(self):
# local tx
result = await self.out(self.daemon.jsonrpc_account_send(
'5.0', await self.daemon.jsonrpc_address_unused(self.account.id)
'5.0', await self.daemon.jsonrpc_address_unused(self.account.id), blocking=True
))
await self.confirm_tx(result['txid'])
tx = await self.daemon.jsonrpc_transaction_show(result['txid'])
@@ -27,10 +40,9 @@ class TransactionCommandsTestCase(CommandTestCase):
self.assertFalse(result['success'])
async def test_utxo_release(self):
sendtxid = await self.blockchain.send_to_address(
await self.account.receiving.get_or_create_usable_address(), 1
await self.send_to_address_and_wait(
await self.account.receiving.get_or_create_usable_address(), 1, 1
)
await self.confirm_tx(sendtxid)
await self.assertBalance(self.account, '11.0')
await self.ledger.reserve_outputs(await self.account.get_utxos())
await self.assertBalance(self.account, '0.0')
@@ -40,6 +52,7 @@ class TransactionCommandsTestCase(CommandTestCase):
class TestSegwit(CommandTestCase):
@unittest.SkipTest
async def test_segwit(self):
p2sh_address1 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
p2sh_address2 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
@@ -53,14 +66,13 @@ class TestSegwit(CommandTestCase):
p2sh_txid2 = await self.blockchain.send_to_address(p2sh_address2, '1.0')
bech32_txid1 = await self.blockchain.send_to_address(bech32_address1, '1.0')
bech32_txid2 = await self.blockchain.send_to_address(bech32_address2, '1.0')
await self.generate(1)
# P2SH & BECH32 can pay to P2SH address
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid1, "vout": 0},
{"txid": bech32_txid1, "vout": 0},
], [{p2sh_address3: '1.9'}]
], {p2sh_address3: 1.9}
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
p2sh_txid3 = await self.blockchain.send_raw_transaction(tx)
@@ -71,7 +83,7 @@ class TestSegwit(CommandTestCase):
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid2, "vout": 0},
{"txid": bech32_txid2, "vout": 0},
], [{bech32_address3: '1.9'}]
], {bech32_address3: 1.9}
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
bech32_txid3 = await self.blockchain.send_raw_transaction(tx)
@@ -83,12 +95,9 @@ class TestSegwit(CommandTestCase):
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid3, "vout": 0},
{"txid": bech32_txid3, "vout": 0},
], [{address: '3.5'}]
], {address: 3.5}
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
txid = await self.blockchain.send_raw_transaction(tx)
await self.on_transaction_id(txid)
await self.generate(1)
await self.on_transaction_id(txid)
await self.generate_and_wait(1, [txid])
await self.assertBalance(self.account, '13.5')
@@ -1,7 +1,7 @@
import asyncio
import random
from itertools import chain
import lbry.wallet.rpc.jsonrpc
from lbry.wallet.transaction import Transaction, Output, Input
from lbry.testcase import IntegrationTestCase
from lbry.wallet.util import satoshis_to_coins, coins_to_satoshis
@@ -9,9 +9,8 @@ from lbry.wallet.manager import WalletManager
class BasicTransactionTests(IntegrationTestCase):
async def test_variety_of_transactions_and_longish_history(self):
await self.blockchain.generate(300)
await self.generate(300)
await self.assertBalance(self.account, '0.0')
addresses = await self.account.receiving.get_addresses()
@@ -19,10 +18,10 @@ class BasicTransactionTests(IntegrationTestCase):
# to the 10th receiving address for a total of 30 UTXOs on the entire account
for i in range(10):
notification = asyncio.ensure_future(self.on_address_update(addresses[i]))
txid = await self.blockchain.send_to_address(addresses[i], 10)
_ = await self.send_to_address_and_wait(addresses[i], 10)
await notification
notification = asyncio.ensure_future(self.on_address_update(addresses[9]))
txid = await self.blockchain.send_to_address(addresses[9], 10)
_ = await self.send_to_address_and_wait(addresses[9], 10)
await notification
# use batching to reduce issues with send_to_address on cli
@@ -57,7 +56,7 @@ class BasicTransactionTests(IntegrationTestCase):
for tx in await self.ledger.db.get_transactions(txid__in=[tx.id for tx in txs])
]))
await self.blockchain.generate(1)
await self.generate(1)
await asyncio.wait([self.ledger.wait(tx) for tx in txs])
await self.assertBalance(self.account, '199.99876')
@@ -74,7 +73,7 @@ class BasicTransactionTests(IntegrationTestCase):
)
await self.broadcast(tx)
await self.ledger.wait(tx)
await self.blockchain.generate(1)
await self.generate(1)
await self.ledger.wait(tx)
self.assertEqual(2, await self.account.get_utxo_count()) # 199 + change
@@ -88,12 +87,10 @@ class BasicTransactionTests(IntegrationTestCase):
await self.assertBalance(account2, '0.0')
addresses = await account1.receiving.get_addresses()
txids = await asyncio.gather(*(
self.blockchain.send_to_address(address, 1.1) for address in addresses[:5]
))
await asyncio.wait([self.on_transaction_id(txid) for txid in txids]) # mempool
await self.blockchain.generate(1)
await asyncio.wait([self.on_transaction_id(txid) for txid in txids]) # confirmed
txids = []
for address in addresses[:5]:
txids.append(await self.send_to_address_and_wait(address, 1.1))
await self.generate_and_wait(1, txids)
await self.assertBalance(account1, '5.5')
await self.assertBalance(account2, '0.0')
@@ -107,7 +104,7 @@ class BasicTransactionTests(IntegrationTestCase):
)
await self.broadcast(tx)
await self.ledger.wait(tx) # mempool
await self.blockchain.generate(1)
await self.generate(1)
await self.ledger.wait(tx) # confirmed
await self.assertBalance(account1, '3.499802')
@@ -121,7 +118,7 @@ class BasicTransactionTests(IntegrationTestCase):
)
await self.broadcast(tx)
await self.ledger.wait(tx) # mempool
await self.blockchain.generate(1)
await self.generate(1)
await self.ledger.wait(tx) # confirmed
tx = (await account1.get_transactions(include_is_my_input=True, include_is_my_output=True))[1]
@@ -133,11 +130,11 @@ class BasicTransactionTests(IntegrationTestCase):
self.assertTrue(tx.outputs[1].is_internal_transfer)
async def test_history_edge_cases(self):
await self.blockchain.generate(300)
await self.generate(300)
await self.assertBalance(self.account, '0.0')
address = await self.account.receiving.get_or_create_usable_address()
# evil trick: mempool is unsorted on real life, but same order between python instances. reproduce it
original_summary = self.conductor.spv_node.server.bp.mempool.transaction_summaries
original_summary = self.conductor.spv_node.server.mempool.transaction_summaries
def random_summary(*args, **kwargs):
summary = original_summary(*args, **kwargs)
@@ -146,13 +143,10 @@ class BasicTransactionTests(IntegrationTestCase):
while summary == ordered:
random.shuffle(summary)
return summary
self.conductor.spv_node.server.bp.mempool.transaction_summaries = random_summary
self.conductor.spv_node.server.mempool.transaction_summaries = random_summary
# 10 unconfirmed txs, all from blockchain wallet
sends = [self.blockchain.send_to_address(address, 10) for _ in range(10)]
# use batching to reduce issues with send_to_address on cli
for batch in range(0, len(sends), 10):
txids = await asyncio.gather(*sends[batch:batch + 10])
await asyncio.wait([self.on_transaction_id(txid) for txid in txids])
for i in range(10):
await self.send_to_address_and_wait(address, 10)
remote_status = await self.ledger.network.subscribe_address(address)
self.assertTrue(await self.ledger.update_history(address, remote_status))
# 20 unconfirmed txs, 10 from blockchain, 10 from local to local
@@ -170,8 +164,7 @@ class BasicTransactionTests(IntegrationTestCase):
remote_status = await self.ledger.network.subscribe_address(address)
self.assertTrue(await self.ledger.update_history(address, remote_status))
# server history grows unordered
txid = await self.blockchain.send_to_address(address, 1)
await self.on_transaction_id(txid)
await self.send_to_address_and_wait(address, 1)
self.assertTrue(await self.ledger.update_history(address, remote_status))
self.assertEqual(21, len((await self.ledger.get_local_status_and_history(address))[1]))
self.assertEqual(0, len(self.ledger._known_addresses_out_of_sync))
@@ -195,37 +188,37 @@ class BasicTransactionTests(IntegrationTestCase):
self.ledger, 2000000000000, [self.account], set_reserved=False, return_insufficient_funds=True
)
got_amounts = [estimator.effective_amount for estimator in spendable]
self.assertListEqual(amounts, got_amounts)
self.assertListEqual(sorted(amounts), sorted(got_amounts))
async def test_sqlite_coin_chooser(self):
wallet_manager = WalletManager([self.wallet], {self.ledger.get_id(): self.ledger})
await self.blockchain.generate(300)
await self.generate(300)
await self.assertBalance(self.account, '0.0')
address = await self.account.receiving.get_or_create_usable_address()
other_account = self.wallet.generate_account(self.ledger)
other_address = await other_account.receiving.get_or_create_usable_address()
self.ledger.coin_selection_strategy = 'sqlite'
await self.ledger.subscribe_account(self.account)
await self.ledger.subscribe_account(other_account)
accepted = asyncio.ensure_future(self.on_address_update(address))
txid = await self.blockchain.send_to_address(address, 1.0)
_ = await self.send_to_address_and_wait(address, 1.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
txid = await self.blockchain.send_to_address(address, 1.0)
_ = await self.send_to_address_and_wait(address, 1.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
txid = await self.blockchain.send_to_address(address, 3.0)
_ = await self.send_to_address_and_wait(address, 3.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
txid = await self.blockchain.send_to_address(address, 5.0)
_ = await self.send_to_address_and_wait(address, 5.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
txid = await self.blockchain.send_to_address(address, 10.0)
_ = await self.send_to_address_and_wait(address, 10.0)
await accepted
await self.assertBalance(self.account, '20.0')
@@ -266,6 +259,12 @@ class BasicTransactionTests(IntegrationTestCase):
async def broadcast(tx):
try:
return await real_broadcast(tx)
except lbry.wallet.rpc.jsonrpc.RPCError as err:
# this is expected in tests where we try to double spend.
if 'the transaction was rejected by network rules.' in str(err):
pass
else:
raise err
finally:
e.set()
+35 -1
View File
@@ -1,19 +1,22 @@
import asyncio
import tempfile
from io import BytesIO
from unittest import mock
import shutil
import os
import copy
from lbry.blob_exchange.serialization import BlobRequest
from lbry.testcase import AsyncioTestCase
from lbry.conf import Config
from lbry.extras.daemon.storage import SQLiteStorage
from lbry.extras.daemon.daemon import Daemon
from lbry.blob.blob_manager import BlobManager
from lbry.blob_exchange.server import BlobServer, BlobServerProtocol
from lbry.blob_exchange.client import request_blob
from lbry.dht.peer import PeerManager, make_kademlia_peer
from lbry.dht.node import Node
# import logging
# logging.getLogger("lbry").setLevel(logging.DEBUG)
@@ -326,3 +329,34 @@ class TestBlobExchange(BlobExchangeTestBase):
with self.assertRaises(asyncio.CancelledError):
await request_blob(self.loop, client_blob, self.server_from_client.address,
self.server_from_client.tcp_port, 2, 3)
async def test_download_blob_using_jsonrpc_blob_get(self):
blob_hash = "7f5ab2def99f0ddd008da71db3a3772135f4002b19b7605840ed1034c8955431bd7079549e65e6b2a3b9c17c773073ed"
mock_blob_bytes = b'1' * ((2 * 2 ** 20) - 1)
await self._add_blob_to_server(blob_hash, mock_blob_bytes)
# setup RPC Daemon
daemon_config = copy.deepcopy(self.client_config)
daemon_config.fixed_peers = [(self.server_from_client.address, self.server_from_client.tcp_port)]
daemon = Daemon(daemon_config)
mock_node = mock.Mock(spec=Node)
def _mock_accumulate_peers(q1, q2=None):
async def _task():
pass
q2 = q2 or asyncio.Queue(loop=self.loop)
return q2, self.loop.create_task(_task())
mock_node.accumulate_peers = _mock_accumulate_peers
with mock.patch('lbry.extras.daemon.componentmanager.ComponentManager.all_components_running',
return_value=True):
with mock.patch('lbry.extras.daemon.daemon.Daemon.dht_node', new_callable=mock.PropertyMock) \
as daemon_mock_dht:
with mock.patch('lbry.extras.daemon.daemon.Daemon.blob_manager', new_callable=mock.PropertyMock) \
as daemon_mock_blob_manager:
daemon_mock_dht.return_value = mock_node
daemon_mock_blob_manager.return_value = self.client_blob_manager
result = await daemon.jsonrpc_blob_get(blob_hash, read=True)
self.assertIsNotNone(result)
self.assertEqual(mock_blob_bytes.decode(), result, "Downloaded blob is different than server blob")
Loaded 100 of 111 files, more files were not shown because too many files have changed in this diff. Show more