Compare commits

..
1029 Commits
Author SHA1 Message Date
Victor Shyba 574c2ce794 adds integration test for release_time 2019-01-23 11:16:43 -03:00
Victor Shyba 7f5541eb5c adds release_time parameter to RPC publish 2019-01-23 11:16:43 -03:00
Victor Shyba 7b49241c38 adjust balance assertions for integration test using new signature model due new fees from new size 2019-01-23 11:16:43 -03:00
Victor Shyba da716b2e3d remove new signature model integration test as its the default now 2019-01-23 11:16:43 -03:00
Victor Shyba a144fae8b2 activate new signature model 2019-01-23 11:16:43 -03:00
Victor Shyba 0d90b82287 clear releaseTime when not set 2019-01-23 11:16:43 -03:00
Victor Shyba 567c23e10f regen for metadata releaseTime field 2019-01-23 11:16:43 -03:00
Victor Shyba 16970b7d37 regen proto from lbry/types, removes local .proto files 2019-01-23 11:16:43 -03:00
Lex Berezhny 8593cd362a check darwin before windows because darwin has win, face plam! 2019-01-22 18:04:17 -05:00
Lex Berezhny 2dbf96da81 blob downer test fixes 2019-01-22 18:04:17 -05:00
Lex Berezhny 83c7797187 pylint 2019-01-22 18:04:17 -05:00
Lex Berezhny 6606deafa5 fixed after testing with desktop app 2019-01-22 18:04:17 -05:00
Lex Berezhny 19b0d59159 config options api_host and api_port changed to just api 2019-01-22 18:04:17 -05:00
Lex Berezhny 9ccb3fa2a3 unit/test_cli tests 2019-01-22 18:04:17 -05:00
Lex Berezhny 48533df523 fix integration tests 2019-01-22 18:04:17 -05:00
Lex Berezhny 1ae1137005 fixing integration tests 2019-01-22 18:04:17 -05:00
Lex Berezhny d5f2832b2f pylint 2019-01-22 18:04:17 -05:00
Lex Berezhny d0230b4893 command line fixes 2019-01-22 18:04:17 -05:00
Lex Berezhny a6cd53b97c test_Daemon.py uses /tmp for data_dir 2019-01-22 18:04:17 -05:00
Lex Berezhny 7a28171a72 conf no longer global 2019-01-22 18:04:17 -05:00
Lex Berezhny 150bcb1116 new configuration manager, first pass 2019-01-22 18:04:17 -05:00
Jack Robison 6fe9c5bf00 Merge pull request #1794 from lbryio/async-exchange-rate-manager
refactor exchange rate manager to use asyncio
2019-01-22 12:05:07 -05:00
Jack Robison 3a86fc6680 refactor exchange rate manager to use asyncio 2019-01-22 11:34:49 -05:00
Jack Robison 2805cbb331 refactor loggly handler to use asyncio 2019-01-22 10:52:50 -05:00
Jack Robison 68d7a7014c Merge pull request #1792 from lbryio/delete-unused-scripts
delete unused scripts
2019-01-22 09:53:24 -05:00
Jack Robison 883fa267a5 delete unused scripts 2019-01-22 09:33:42 -05:00
Lex Berezhny 5f2c3ac965 dont run customlogger tests 2019-01-19 16:32:11 -05:00
Lex Berezhny bf8e32ebd2 remove custom logging 2019-01-19 16:30:19 -05:00
Lex Berezhny 5971be5bef v0.30.5 2019-01-18 16:14:48 -05:00
Lex Berezhny 677d76a3e4 v0.30.5rc4 2019-01-17 23:26:12 -05:00
Victor Shyba 6ed3207cb7 phase1: legacy claim format, ability to decode new format 2019-01-17 23:24:58 -05:00
Victor Shyba be6448b74f fix str/bytes confusion from resolve/claimdict on certificate_id 2019-01-17 23:24:58 -05:00
Victor Shyba b707ee4844 encode and decode detached signatures from claim dict 2019-01-17 23:24:58 -05:00
Victor Shyba 0e9888be3b fixes from review 2019-01-17 23:24:58 -05:00
Victor Shyba 68b56b7858 adds test for claim_show 2019-01-17 23:24:58 -05:00
Victor Shyba d32697f611 adjust assertions for fee changes 2019-01-17 23:24:58 -05:00
Victor Shyba 3c54b2bcd7 activate detached sigs 2019-01-17 23:24:58 -05:00
Victor Shyba 899b182811 fix getclaimsfortx in server 2019-01-17 23:24:58 -05:00
Victor Shyba 0dfb5e008d add integration test with unserializable claim 2019-01-17 23:24:58 -05:00
Victor Shyba abf94357eb tests and fixes for sig validation on undecode-able claims 2019-01-17 23:24:58 -05:00
Victor Shyba 36aded3830 store original payload for signing and verifying 2019-01-17 23:24:58 -05:00
Victor Shyba d7ebf50602 detached as a parameter instead of class attribute 2019-01-17 23:24:58 -05:00
Victor Shyba 523c4c86a9 change default curve to SECP256k1, pass name parameter 2019-01-17 23:24:58 -05:00
Victor Shyba 74cccdbfc7 simplify detached signing and verifying 2019-01-17 23:24:58 -05:00
Victor Shyba 70471eebfa wallet server and resolve working, functional test 2019-01-17 23:24:58 -05:00
Victor Shyba 6339224980 tests for detached signatures 2019-01-17 23:24:58 -05:00
Victor Shyba fe97f7a119 adds a detached signature attribute to ClaimDict 2019-01-17 23:24:58 -05:00
Victor Shyba f8ed605da2 adds a way to verify detached signatures 2019-01-17 23:24:58 -05:00
Victor Shyba e38ef2af1f adds a way to sign detached 2019-01-17 23:24:58 -05:00
Victor Shyba 5d8e627d02 adds a separate representation for a detached signature 2019-01-17 23:24:58 -05:00
Lex Berezhny c9d4ad5450 pylint unused import 2019-01-17 19:04:52 -05:00
Lex Berezhny 31586bf588 error out when user tries --http-auth 2019-01-17 19:04:52 -05:00
Lex Berezhny 974d3c83e3 keep wallet_balance to test deprecation with, until we find better way to do this 2019-01-17 17:35:15 -05:00
Lex Berezhny 8dc8ca382b removed previously deprecated commands 2019-01-17 17:35:15 -05:00
Lex Berezhny 1dbb250413 fix bytes vs str error in migrate5to6 script 2019-01-15 21:44:16 -05:00
Lex Berezhny 9f9ec692c7 orchstr8 docker for lbry 2019-01-15 17:15:24 -05:00
Thomas Zarebczan 2c1b54c76c Add note about wallet needing to be unlocked (grin was confused about this) 2019-01-15 15:59:45 -05:00
Thomas Zarebczan 7314494fb9 add m4v
It's a container type very similar to mp4. Someone recently added one in the community top bids section.
2019-01-15 15:55:59 -05:00
Jack Robison e74a5e8ef3 cleanup 2019-01-11 16:36:55 -05:00
Jack Robison b9af999739 rename -> time_to_first_byte.py 2019-01-11 16:36:55 -05:00
hackrush 2de155f90f Profile get using rpc calls 2019-01-11 16:36:55 -05:00
hackrush ab3db2e626 Get paths from args 2019-01-11 16:36:55 -05:00
hackrush f9d26b04d8 Get averages! 2019-01-11 16:36:55 -05:00
hackrush 24e0673198 Profile downloading of sd blobs 2019-01-11 16:36:55 -05:00
hackrush 833261801f Profile downloading of sd blobs 2019-01-11 16:36:55 -05:00
hackrush 6c8cbc1b9e added additional fields to uri parser 2019-01-11 15:10:22 -05:00
Lex Berezhny 6ae4e68d1c looping_call shouldnt need to mess with cancellation 2019-01-11 14:33:45 -05:00
Oleg Silkin d22a2dee33 Removes await from shutdown call 2019-01-11 14:33:45 -05:00
Oleg Silkin 0f1ed589a5 Modifies the unit tests for the analytics manager to have async routines
Signed-off-by: Oleg Silkin <o.silkin98@gmail.com>
2019-01-11 14:33:45 -05:00
Oleg Silkin b032e99dd1 Removes the twisted dependency from the analytics file and replaces it
with `asyncio` and `aiohttp`. Also removes the calling loop from the
analytics in favor of a `dict` that stores the names of the methods
and their coroutines that are wrapped inside a looping task. The tasks
are canceled when the analytics manager is asked to shutdown

Signed-off-by: Oleg Silkin <o.silkin98@gmail.com>
2019-01-11 14:33:45 -05:00
Lex Berezhny fd7811f38d v0.30.5rc3 2019-01-11 01:17:43 -05:00
Lex Berezhny 071db3a73d release.py script now extracts backwards-incompatible: value from PR description and adds it to changelog under Backwards Incompatible Changes 2019-01-11 01:16:59 -05:00
Lex Berezhny 950c122850 fix blob announce command argument constraints 2019-01-11 00:54:15 -05:00
Thomas Zarebczan e0386ea9dc rename lbnyum_wallet_dir to wallet_dir 2019-01-11 00:02:52 -05:00
Thomas Zarebczan 6c3eee0f0e Update conf.py 2019-01-10 23:50:59 -05:00
Thomas Zarebczan 4c23add193 fix unavailable download directory
If an uploaded file had a directory that was removed, or if the old download directory doesn't exist, use the default.
2019-01-10 23:05:43 -05:00
Lex Berezhny 248d1a18c7 pylint 2019-01-10 22:38:27 -05:00
Lex Berezhny 17d1b85423 run brew upgrade twice and ignore result 2019-01-10 22:38:27 -05:00
Lex Berezhny b0deb63129 blob_reflect fix 2019-01-10 22:38:27 -05:00
Lex Berezhny ebdb33bd11 blob_availability fix 2019-01-10 22:38:27 -05:00
Lex Berezhny 69ccbda168 fixed error handling when calling commands 2019-01-10 22:38:27 -05:00
Lex Berezhny a38643cca6 remove d2f from storage function call 2019-01-10 22:38:27 -05:00
Lex Berezhny 712b19605e Revert "fix breaking integration tests after fees changed due to content_type modification"
This reverts commit 0613cf2d59.
2019-01-10 10:22:37 -05:00
Alex Grintsvayg 8a5f3842fc fix media type for files without extension 2019-01-10 10:22:37 -05:00
Lex Berezhny db3e578bcd v0.30.5rc2 2019-01-08 23:56:07 -05:00
Lex Berezhny 7e22adff7c added contributor username to each PR listed in changelog 2019-01-08 23:55:34 -05:00
Lex Berezhny 76fbaac658 do not log aiohttp.access 2019-01-08 23:54:32 -05:00
hackrush c4fac29ecb actually get the certificate_id 2019-01-08 23:45:20 -05:00
Lex Berezhny 33392b3351 migration fix 2019-01-08 23:15:25 -05:00
Lex Berezhny 0613cf2d59 fix breaking integration tests after fees changed due to content_type modification 2019-01-08 17:54:40 -05:00
Alex Grintsvayg a930ca0631 if media type is not listed, default to a type that includes the file extension 2019-01-08 16:06:44 -05:00
Lex Berezhny 63a7271008 v0.30.5rc1 2019-01-07 22:15:37 -05:00
Lex Berezhny 5ca0ebcbb2 store loop on storage 2019-01-07 21:56:05 -05:00
Lex Berezhny 2c84b7a7dd get_running_loop() -> get_event_loop() 2019-01-07 21:56:05 -05:00
Lex Berezhny 71a0c704bd storage.clock.seconds() -> asyncio.get_running_loop().time() 2019-01-07 21:56:05 -05:00
Lex Berezhny d30e7cd29e use loop.time() instead of reactor.seconds() in SQLite storage 2019-01-07 21:56:05 -05:00
Lex Berezhny aeb7c6fe76 pylint, removed unused import 2019-01-07 21:56:05 -05:00
Lex Berezhny 134c20da93 stream cost estimate command refactor 2019-01-07 21:56:05 -05:00
Lex Berezhny 893b4fc3ca fixes 2019-01-07 21:56:05 -05:00
Lex Berezhny 9b27e81940 fix logging 2019-01-07 21:56:05 -05:00
Lex Berezhny 72c0acf269 test_HTTPBlobDownloader 2019-01-07 21:56:05 -05:00
Lex Berezhny 7be53ad692 more bug fixes 2019-01-07 21:56:05 -05:00
Lex Berezhny 16e596ec87 lots o bugs fixed discovered while using desktop app 2019-01-07 21:56:05 -05:00
Lex Berezhny 3fdcb80960 minor test fixes 2019-01-07 21:56:05 -05:00
Lex Berezhny 26a3221bac hash announcer fix 2019-01-07 21:56:05 -05:00
Lex Berezhny eb16cb0aa8 fix test_HashBlob 2019-01-07 21:56:05 -05:00
Lex Berezhny b5340505e5 fix blobmanager tests and dont run conf tests 2019-01-07 21:56:05 -05:00
Lex Berezhny 4bb8129bb7 fixing tests to work with asyncio daemon 2019-01-07 21:56:05 -05:00
Lex Berezhny 7f838d3a31 all integration tests run with plain unittest now 2019-01-07 21:56:05 -05:00
Lex Berezhny c186c31bf7 simplified component setup 2019-01-07 21:56:05 -05:00
Lex Berezhny 86fb99a37c fully asyncio daemon api 2019-01-07 21:56:05 -05:00
Lex Berezhny 6c52450858 temporary module providing d2f and f2d for converting between twisted and asyncio 2019-01-07 21:56:05 -05:00
Lex Berezhny fb4e94b04a asyncifying minimal amount of the old blob/p2p code 2019-01-07 21:56:05 -05:00
Lex Berezhny 248baf58b4 asyncio daemon 2019-01-07 21:56:05 -05:00
Lex Berezhny 0845d65f4e rebase master 2019-01-07 21:56:05 -05:00
Lex Berezhny dcc5415648 v0.30.4 2019-01-04 22:03:39 -05:00
Lex Berezhny 3c38a88b22 v0.30.4rc2 2019-01-04 11:53:54 -05:00
Victor Shyba 187b7670b1 adds a dockerfile for wallet server 2019-01-04 11:10:52 -05:00
Victor Shyba 049c17ff08 adds testenet to wallet server 2019-01-04 11:10:52 -05:00
Thomas Zarebczan 8de97bb441 Add lbrynet4 dht seed node (#1734)
added `lbrynet4.lbry.io` to list of default DHT nodes
2019-01-04 11:07:02 -05:00
Lex Berezhny 97c0092923 v0.30.4rc1 2019-01-04 03:14:56 -05:00
Lex Berezhny e65e1fb5a2 added utxo_release command 2019-01-04 03:14:37 -05:00
Victor Shyba 20dd1e55f5 fix data_dir parameter parsing 2019-01-04 01:25:35 -05:00
Lex Berezhny 6aa709e34e v0.30.3 2018-12-21 18:45:34 -05:00
Lex Berezhny e73f121c08 v0.30.3rc1 2018-12-19 09:12:56 -05:00
Victor Shyba 3b9d6c4f8e Instantiate resolver before the request to hold the selected height 2018-12-19 09:04:30 -05:00
Alyssa Callahan 69b60c19c3 Merge pull request #1712 from lbryio/readme-update
Update README.md
2018-12-17 10:47:29 -05:00
Lex Berezhny 04278cb1ab release script automatically figures out prerelease status 2018-12-14 22:25:06 -05:00
Lex Berezhny e8dec5b440 v0.30.2 2018-12-14 21:50:48 -05:00
Lex Berezhny 75f82b107e updated release script with unittests and ability to go from rc to final release 2018-12-14 21:50:30 -05:00
Jack Robison 0e90c4b611 Merge pull request #1717 from lbryio/fix-directory-args
fix directory cli args and update help message
2018-12-14 17:09:15 -05:00
Jack Robison fb66b68bc8 fix directory cli args, update help message 2018-12-14 15:51:11 -05:00
Victor Shyba c601677386 run schema unit tests on travis 2018-12-14 09:21:31 -05:00
Lex Berezhny 1fbd1cd04e v0.30.2rc2 2018-12-12 19:52:11 -05:00
Jack Robison 4a71b91fbf Merge pull request #1671 from lbryio/remove-android-helpers
remove `lbrynet.androidhelpers` module
2018-12-12 18:35:31 -05:00
Jack Robison b8b0d8b1ac refactor default directory setup, remove lbrynet.androidhelpers
fixes https://github.com/lbryio/lbry/issues/1606
2018-12-12 17:46:10 -05:00
Jack Robison 8ef73d3985 update docs 2018-12-12 12:19:26 -05:00
Jack Robison 05e556a5e3 fix generate_json_api script 2018-12-12 12:18:32 -05:00
Lex Berezhny d1da42bea8 Update README.md
* updated description and added bullet points for main features of this SDK
* lex (@eukreign ) is the official product owner of lbrysdk
2018-12-12 11:47:28 -05:00
Lex Berezhny 90685e8357 deploy release for tags 2018-12-12 00:10:50 -05:00
Lex Berezhny a471d5d167 v0.30.2rc1 2018-12-11 23:44:02 -05:00
Lex Berezhny 16122299e4 revert version back 0.30.1 2018-12-11 23:39:08 -05:00
Lex Berezhny 9f18da4bae release script (#1697)
* travis deploying to github
* release script
2018-12-11 23:24:19 -05:00
Lex Berezhny 1abce52867 v0.30.2rc1 2018-12-11 23:09:31 -05:00
Lex Berezhny 072e2b65d3 revert version change 2018-12-11 22:31:45 -05:00
Lex Berezhny f6d11e5467 0.30.1rc1 2018-12-11 22:29:40 -05:00
Oleg Silkin 458bddcb65 Removes Faker from Dependencies (#1687)
* Removes faker import and its usage
* Replaces data generated with faker with static data
* Drops faker as a dependency & removes extras require section
* Removes installation instruction for the test dependencies in setup.py
2018-12-11 20:19:58 -05:00
Victor Shyba 5fad43ff69 fix test tear down and unify test_commands 2018-12-11 16:34:37 -05:00
Victor Shyba 6992d0d947 add new test to tox.ini 2018-12-11 16:34:37 -05:00
Victor Shyba d1b0180437 add tests for claim_list_by_channel 2018-12-11 16:34:37 -05:00
Victor Shyba ba59e126cd forward errors outside of uris on channel listing 2018-12-11 16:34:37 -05:00
Victor Shyba 4e3467bb7e fix windows release type set script 2018-12-11 16:30:35 -05:00
Lex Berezhny 4d4c705eb9 0.30.1 changelog updates 2018-12-09 23:30:09 -05:00
Lex Berezhny ce7b3184c4 0.30.1 2018-12-09 23:15:11 -05:00
Lex Berezhny b71faddaac upload zip files into the same directory 2018-12-08 19:01:13 -05:00
Lex Berezhny 1328cb4cce check file exists before reading config 2018-12-08 19:01:13 -05:00
Lex Berezhny 946488bce9 correctly save daemon_settings.yml on fresh install 2018-12-08 19:01:13 -05:00
Victor Shyba 7713881e71 additional tests for resolution border cases 2018-12-08 14:40:37 -05:00
Victor Shyba 26fcc48c5a fix for name collision on channel path 2018-12-08 14:40:37 -05:00
Thomas Zarebczan bf12fb68a2 use xcode8.3 for mac build (#1692)
use xcode8.3 for mac build
2018-12-08 14:38:35 -05:00
Jack Robison 3a4c9ff4f9 bump to 0.30.1rc11 2018-12-07 13:07:06 -05:00
Jack Robison 6180a2917d add epub and wmv content types 2018-12-07 13:06:36 -05:00
Lex Berezhny 014cbb4623 v0.30.1rc10 2018-12-07 12:04:28 -05:00
Jack Robison ee98251d79 Merge pull request #1700 from lbryio/mimetypes
improve contentType detection
2018-12-07 12:02:41 -05:00
Jack Robison e3d08ec773 improve contentType detection
fixes https://github.com/lbryio/lbry/issues/1690
2018-12-07 11:31:05 -05:00
Lex Berezhny f03e68e63e 0.30.1rc9 2018-12-06 00:51:07 -05:00
Lex Berezhny 72090b45dd confirmations is returned as int 2018-12-06 00:49:07 -05:00
Lex Berezhny 93f9622798 fix json_response_encoder to also return correct confirmations 2018-12-06 00:49:07 -05:00
Lex Berezhny 16ff4b4e2c confirmations fix 2018-12-06 00:49:07 -05:00
Lex Berezhny c90cca4b20 fix #1683 2018-12-05 14:38:35 -05:00
Lex Berezhny 98d83ea74f fix for integration test 2018-12-05 12:14:43 -05:00
Lex Berezhny a234f92d2c 0.30.1rc8 2018-12-05 11:53:37 -05:00
Lex Berezhny 8ba3575901 dropped aiorpcx 2018-12-05 09:17:36 -05:00
Jack Robison a857c83c59 Merge pull request #1685 from lbryio/fix-1674
fix int too large to convert to integer when updating claim
2018-12-04 18:16:24 -05:00
Jack Robison a6128ac8a2 fix https://github.com/lbryio/lbry/compare/fix-1674 2018-12-04 17:35:40 -05:00
hackrush 2693301ad0 Fix python and pip3 not available in xcode8 2018-12-04 15:12:07 -05:00
Victor Shyba f7c7cb9535 fix transaction_show 2018-12-04 12:12:08 -05:00
Jack Robison b36a5f9b6b Merge pull request #1678 from lbryio/warn_migration_bad_claim_id
check and log claim id validity during migration
2018-11-30 15:58:26 -05:00
Jack Robison 4826eb9969 migrate double hex encoded certificates 2018-11-30 14:00:35 -05:00
Victor Shyba c34b150794 check and log claim id validity during migration 2018-11-30 14:00:35 -05:00
Lex Berezhny c4c7670a7c do not run brew update 2018-11-30 12:20:07 -05:00
Lex Berezhny 0918bac86e mac build stuff 2018-11-30 11:50:32 -05:00
Victor Shyba a60a921706 resolve claims on claim_list (#1483) 2018-11-30 11:46:56 -05:00
Lex Berezhny b3ee29b66d maybe python is already installed on mac 2018-11-30 11:30:03 -05:00
Lex Berezhny 3a1a9b988c on mac brew, python 3 is installed with plain python while python2 now requires python@2 2018-11-30 10:55:15 -05:00
Lex Berezhny 12f2bebc46 build for mac using xcode8 2018-11-30 10:34:45 -05:00
Jack Robison 31a3812c8a bump 2018-11-29 17:23:32 -05:00
Lex Berezhny 70ec08e30b icons! 2018-11-29 17:15:18 -05:00
Jack Robison 23e8f0d849 Merge pull request #1676 from lbryio/fix-1573
unlock wallet after encrypting it
2018-11-29 16:13:21 -05:00
Jack Robison 1cdfcd5b4e unlock wallet after encrypting it 2018-11-29 15:16:53 -05:00
Lex Berezhny dc2786fbc0 assert instead of if 2018-11-28 18:03:33 -05:00
Lex Berezhny 7e39b0c4fe run ClaimManagement.test_create_update_and_abandon_claim 2018-11-28 18:03:33 -05:00
Lex Berezhny 4af830c405 tests for transaction_list going through claim creation -> claim update -> claim abandon 2018-11-28 18:03:33 -05:00
Lex Berezhny fe844581fd correctly split value/fee in transaction_list 2018-11-28 18:03:33 -05:00
Lex Berezhny 443ce6352b typo 2018-11-28 18:03:33 -05:00
Lex Berezhny 158c354a27 split out fee from value when possible 2018-11-28 18:03:33 -05:00
Lex Berezhny 92d1230a90 remove fees from value, abandon is positive 2018-11-28 18:03:33 -05:00
Jack Robison d64916a061 bump 2018-11-27 16:28:55 -05:00
Alyssa Callahan 20a0568253 Merge pull request #1567 from preethamvishy/master
Update pull request template
2018-11-27 15:45:54 -05:00
Alyssa Callahan 5a1f42ee54 Merge branch 'master' into master 2018-11-27 15:39:48 -05:00
Alyssa Callahan ab95f14e21 Update PULL_REQUEST_TEMPLATE.md 2018-11-27 15:38:06 -05:00
Jack Robison bae2676b69 Merge pull request #1669 from lbryio/fix_invalid_sigs_from_name
allow invalid sigs claims to show outside a channel
2018-11-27 14:41:28 -05:00
Victor Shyba ea6869a63e allow invalid sigs claims to show outside a channel 2018-11-27 15:36:53 -03:00
Jack Robison 953327c150 bump 2018-11-27 09:52:33 -05:00
Lex Berezhny e01ad6bc85 disable travis cache on mac build 2018-11-26 20:55:03 -05:00
Victor Shyba 47a33009ff remove python 3.6 steps from travis 2018-11-26 20:55:03 -05:00
Victor Shyba bc844013a1 (win build) add pywin32 and netifaces wheels 2018-11-26 20:55:03 -05:00
Victor Shyba 2fd2319cdb use lbry dockerhub image 2018-11-26 20:55:03 -05:00
Victor Shyba a50657c472 add dockerfile and readme on how win docker builder is generated 2018-11-26 20:55:03 -05:00
Victor Shyba ef5f6e06ba use vshyba/temporary_pyinstaller_32bits_py37:latest 2018-11-26 20:55:03 -05:00
Victor Shyba 977cf9f506 log python version on windows build script 2018-11-26 20:55:03 -05:00
Lex Berezhny ffae10d11f drop pytest 2018-11-26 18:02:12 -05:00
Thomas Zarebczan 221107e76e test mac build, do not merge (#1659)
* Update .travis.yml to use xcode 6.4 for mac build
2018-11-26 14:13:45 -05:00
Jack Robison e74c78b607 Merge pull request #1432 from lbryio/get-block-on-bytes-written
[API] block get on bytes being written to the output file
2018-11-26 12:16:42 -05:00
Jack Robison 434c7b57b1 tests 2018-11-26 11:36:28 -05:00
Jack Robison 9dc4608a62 combine StreamProgressManager and FullStreamProgressManager
-add wrote_first_data Deferred to FullStreamProgressManager
2018-11-26 11:36:28 -05:00
Jack Robison fc69366092 set the download manager before creating the first deferred 2018-11-26 11:36:28 -05:00
Victor Shyba 9f0b1f3e25 fix pylint failures unveiled after astroid pin 2018-11-26 11:32:41 -05:00
Victor Shyba 3f03a845ec only return invalid signatures if its a direct resolution 2018-11-26 11:32:41 -05:00
Victor Shyba bc28976e12 adds test for claims on abandoned channels 2018-11-26 11:32:41 -05:00
Victor Shyba 8f826098f9 validate claim id before resolving 2018-11-26 11:32:41 -05:00
Victor Shyba 0a765e6207 handle errors on resolve 2018-11-26 11:32:41 -05:00
Lex Berezhny f83aa61923 force astroid==2.0.4 in .travis.yml 2018-11-26 10:47:48 -05:00
Oleg Silkin f9ff3e5d14 Removes third-party mock dependency (#1661)
* Removes mock from setup.py dependencies
* Refactors name resetTime to reset_time
* Replaces import mock with built-in unittest.mock
* Refactors imports to separate built-in, external, and project modules
2018-11-25 11:12:30 -05:00
Oleg Silkin b3fde9d78d Removes six From Project (#1660)
* Replaces `six` urllib with python 3's urllib
* Replaces `six`'s int2byte method with native `bytes` class
* Removes edge case testing for python2 vs python3
* Removes useless object inheritance
* Uses native io.BytesIO instead of six.BytesIO
* Removes six from dependencies
2018-11-22 23:24:21 -05:00
Lex Berezhny 0c999083cc added jsonrpc_account_send and dropped torba[server] 2018-11-21 01:14:39 -05:00
Victor Shyba 561215db8a fix download script 2018-11-21 00:25:43 -05:00
Jack Robison 8dbb7f2c78 bump 2018-11-20 21:37:49 -05:00
Lex Berezhny 6f7f3bb316 fixes #1653 2018-11-20 16:12:51 -05:00
Jack Robison 8d507c4b73 bump 2018-11-19 21:12:39 -05:00
Lex Berezhny 2beb0eda94 SupportingSupports test does not exist 2018-11-19 20:34:38 -05:00
Lex Berezhny b7e07b72ca update tox.ini to run correct tests 2018-11-19 20:34:38 -05:00
Lex Berezhny fde8718c8c make sure only actual claims/updates are found when updating a claim or sending it to someone 2018-11-19 20:34:38 -05:00
Lex Berezhny b2d1bddcfe update to latest torba 2018-11-19 20:34:38 -05:00
Lex Berezhny dd1d1be994 working claim_send_to_address 2018-11-19 20:34:38 -05:00
Jack Robison a93957178a bump 2018-11-19 17:31:31 -05:00
Jack Robison 5026d0b2f3 Merge branch 'fix-get-bad-claim-error' 2018-11-19 17:31:01 -05:00
Jack Robison cc8b47d882 improve error when get fails to decode the resolved claim 2018-11-19 17:30:02 -05:00
Jack Robison db0b5961b3 Merge pull request #1642 from jessopb/readmeStuff
adds daemon config to and updates links in readme
2018-11-19 14:37:15 -05:00
Jessop Breth cb6f7ada42 adds sample config and links to settings documentation 2018-11-19 10:10:02 -05:00
Jack Robison 59464099b4 add example daemon_settings.yml 2018-11-19 09:42:13 -05:00
preserveddarnell aed00b53fd Update various typos & edits 2018-11-18 20:52:18 -05:00
YULIUS KURNIAWAN KRISTIANTO cdb0f97cb4 added more social links 2018-11-18 10:29:47 -05:00
Jack Robison 33bcc8256c Merge branch 'fix-1624' 2018-11-16 11:45:05 -05:00
Jack Robison 0a8d44f12f Merge branch 'populate-buckets' 2018-11-16 11:44:25 -05:00
Jack Robison 1b74594f5b don't create ssl/auth keys if disabled
fixes https://github.com/lbryio/lbry/issues/1624
2018-11-16 11:12:33 -05:00
Jack Robison b0718d295e fix empty buckets in routing_table_get 2018-11-16 10:59:12 -05:00
Jack Robison 9bb91b126d use midpoint id instead of random for refresh list 2018-11-16 10:59:12 -05:00
Jack Robison 6dbb00d8d5 populate buckets when joining the dht 2018-11-16 10:59:12 -05:00
Jack Robison 3a27a4509a Merge pull request #1639 from lbryio/fix-1638
fix bytes/str in iterativeFindValue
2018-11-14 16:22:30 -05:00
Jack Robison 1d438f5d23 fixes https://github.com/lbryio/lbry/issues/1638 2018-11-14 15:35:59 -05:00
YULIUS KURNIAWAN KRISTIANTO 3840f19869 fix mistake code 2018-11-14 14:20:02 -05:00
Lex Berezhny ec0483b975 header progress percent starts at 0 instead of None 2018-11-12 21:16:05 -05:00
Jack Robison 2fba1f417f less verbose logging 2018-11-12 14:47:11 -05:00
Jack Robison 5b419b55c5 fix peer_list 2018-11-12 14:45:41 -05:00
Victor Shyba b31379e51c log and handle misformed certificate claims from #1609 2018-11-10 20:49:36 -05:00
Victor Shyba 8394dcb663 log and handle misformed claims from #1611 2018-11-10 20:49:36 -05:00
Lex Berezhny 457a56ce04 Merge pull request #1604 from lbryio/fix-external-ip
Fix setting the external ip
2018-11-10 20:48:40 -05:00
Jack Robison 7c8016d807 remove accidentally added lbrynet.schema.signature 2018-11-09 16:52:46 -05:00
Jack Robison d704336f20 absolute imports 2018-11-09 16:52:46 -05:00
Jack Robison 1b4230b06e move is_valid_blobhash to lbrynet.blob.blob_file 2018-11-09 16:52:46 -05:00
Jack Robison 88095af1cd move system_info.py into lbrynet.extras 2018-11-09 16:52:46 -05:00
Jack Robison e9efd9f68a improve get_external_ip 2018-11-09 16:52:46 -05:00
Jack Robison 7b32e2ff7c don't block the dht component on populating the routing table 2018-11-09 16:52:46 -05:00
Jack Robison 572777305e reduce analytics event frequency 2018-11-09 16:52:46 -05:00
Jack Robison 7e27a6e37d fix 2018-11-09 16:52:46 -05:00
Jack Robison b98621d7ea avoid name errors 2018-11-09 16:52:46 -05:00
Jack Robison bc5824b8bc better setup of custom logger class 2018-11-09 16:52:46 -05:00
Jack Robison a6b643bb34 move files 2018-11-09 16:52:42 -05:00
Jack Robison 488b976734 move HashAnnouncer.py to extras 2018-11-09 14:16:33 -05:00
Jack Robison 253912b52e fix skipping dht, peer_protocol_server, and hash_announcer components
-move PeerFinder and PeerManager initialization to the ComponentManager

-remove dht component requirement from all but the hash_announcer component. This allows running the file manager component without either of the server components.
2018-11-09 14:16:33 -05:00
Jack Robison 0bf8416d9f fix components_to_skip setting 2018-11-09 14:16:33 -05:00
Jack Robison 2e921437d8 fix external ip setup
fixes https://github.com/lbryio/lbry/issues/1561
2018-11-09 14:16:33 -05:00
Jack Robison b6f9b60683 remove commit in version for source installs 2018-11-09 14:16:33 -05:00
Lex Berezhny 57430e9f5f fixed #1616 2018-11-09 12:03:55 -05:00
Victor Shyba 1e13fbee8d await missing on resolve.py 2018-11-09 11:11:57 -05:00
Lex Berezhny 3d8b9e2b66 block=False for abandon during erics epic adventure tests 2018-11-07 18:28:02 -05:00
Lex Berezhny 64898d43f1 tearDown() should run before daemon shutdown 2018-11-07 18:28:02 -05:00
Lex Berezhny b2117fddab on_transaction -> ledger.wait 2018-11-07 18:28:02 -05:00
Lex Berezhny 5658623f29 claims include supports 2018-11-07 18:28:02 -05:00
Lex Berezhny 934682bd0f --blocking support in abandon command 2018-11-07 18:28:02 -05:00
Victor Shyba daecc582f3 simplify bind statement 2018-11-07 11:29:56 -05:00
Victor Shyba 3cb4012dbf unify loop 2018-11-07 11:29:56 -05:00
Victor Shyba f7bd44a4f5 fix query for windows (999 vars limit) 2018-11-07 11:29:56 -05:00
hackrush 52317b05bb Better help message instructions 2018-11-07 11:02:20 -05:00
Lex Berezhny 83c0baa3cb test abandoning claim at a loss, required a fix in torba 2018-11-07 10:57:11 -05:00
Lex Berezhny 340c5addab insufficientfundserror test 2018-11-07 10:57:11 -05:00
Lex Berezhny e3fd94dbf0 pylint fix 2018-11-07 10:57:11 -05:00
Lex Berezhny 923fcfb9e9 run publish integration tests separately 2018-11-07 10:57:11 -05:00
Lex Berezhny 0ae5f1a4c6 added fix for #1533 2018-11-07 10:57:11 -05:00
Lex Berezhny 90e06abba2 fixes for #1569 2018-11-07 10:57:11 -05:00
Victor Shyba 6ecfc94ff1 add balance to startup report 2018-11-07 10:50:13 -05:00
Alyssa Callahan 60c4a727d3 Merge pull request #1574 from czakarian/patch-1
Fix typos / grammar changes
2018-11-07 10:19:59 -05:00
Alyssa Callahan 9fbb9f2e98 Merge branch 'master' into patch-1 2018-11-07 10:18:58 -05:00
Lex Berezhny f1feec66c0 fixed path to cli.py in setup.py 2018-11-06 13:55:22 -05:00
Lex Berezhny 73b9b983b1 pinned protobuf to specific version, fixes #1602 2018-11-06 00:05:40 -05:00
Victor Shyba 588779e3a4 makefile for using types repo 2018-11-05 20:48:21 -05:00
Jack Robison b7ae446059 Merge pull request #1495 from lbryio/retry_dht
[DHT] retry on protocol level
2018-11-05 15:16:53 -05:00
Victor Shyba db898307b5 obey the rpcAttempts constant on retries 2018-11-05 15:16:39 -03:00
Victor Shyba 7e3512cbf2 retry on protocol level 2018-11-05 15:16:39 -03:00
Lex Berezhny cdd65c6783 updated pyinstaller build instructions 2018-11-04 20:25:24 -05:00
Lex Berezhny 320ad717cd pylint cleanup 2018-11-04 20:25:24 -05:00
Lex Berezhny 264e6744e1 updated references after moving conf to lbrynet.extras.daemon.conf 2018-11-04 20:25:24 -05:00
Lex Berezhny 64e2b79c07 moved conf into daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny 9255bb5abe updated references after moving analytics to lbrynet.extras.daemon.analytics 2018-11-04 20:25:24 -05:00
Lex Berezhny d7525dacc3 moved analytics to lbrynet/extra/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny 5cb07100f8 deleted unused interfaces file 2018-11-04 20:25:24 -05:00
Lex Berezhny 171c080d33 updated references lbrynet.file_manager -> lbrynet.blob 2018-11-04 20:25:24 -05:00
Lex Berezhny cfc3ec2a05 merged lbrynet/file_manager into lbrynet/blob 2018-11-04 20:25:24 -05:00
Lex Berezhny 1392b8f83b removed lbrynet/lbry_file/__init__.py 2018-11-04 20:25:24 -05:00
Lex Berezhny 8b265801df updated references lbrynet.lbry_file.client -> lbrynet.file_manager.client 2018-11-04 20:25:24 -05:00
Lex Berezhny 6c4fb8dab1 moved lbrynet/lbry_file/client into lbrynet/file_manager/client 2018-11-04 20:25:24 -05:00
Lex Berezhny c1a9cd93ee updated references lbrynet.crypstream -> lbrynet.blob 2018-11-04 20:25:24 -05:00
Lex Berezhny 0d4e83e167 merged lbrynet/cryptstream into lbrynet/blob 2018-11-04 20:25:24 -05:00
Lex Berezhny 1292eb345d fix dbmigrator pylint issue 2018-11-04 20:25:24 -05:00
Lex Berezhny f4c6dc90c9 updated references lbrynet.core -> lbrynet.p2p 2018-11-04 20:25:24 -05:00
Lex Berezhny 8c8f5d8f58 renamed lbrynet/core -> lbrynet/p2p 2018-11-04 20:25:24 -05:00
Lex Berezhny b6d986cb40 updated references after moving lbrynet/database into lbrynet/extras/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny 6cc134c7c8 merged lbrynet/database with lbrynet/extras/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny eee964cfa0 updated references to daemon after moving it to extras 2018-11-04 20:25:24 -05:00
Lex Berezhny a03c94bfad moved daemon to extras/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny f1f0917063 updated all references to point to new reflector location 2018-11-04 20:25:24 -05:00
Lex Berezhny fe2476b5aa moved reflector into extras 2018-11-04 20:25:24 -05:00
Lex Berezhny 9c5fa8bf76 winhelpers/winpaths.py -> extras/winpaths.py 2018-11-04 20:25:24 -05:00
Lex Berezhny cd43a71172 update winpaths.py references 2018-11-04 20:25:24 -05:00
Lex Berezhny e2c6de76f4 renamed knownpaths.py -> winpaths.py 2018-11-04 20:25:24 -05:00
Lex Berezhny 05aeadc9fe extraneous LICENSE file since the whole project is already MIT, moved copyright directly into file header 2018-11-04 20:25:24 -05:00
Lex Berezhny 7153417372 #egg=torba[server] 2018-11-04 10:50:15 -05:00
Lex Berezhny fceb2ae0a2 hmm 2018-11-04 10:50:15 -05:00
Lex Berezhny 0cef2fdd82 install extras from github 2018-11-04 10:50:15 -05:00
Lex Berezhny 9c91b135bd pylint requires [wallet-server] 2018-11-04 10:50:15 -05:00
Lex Berezhny e3b127192c install .[test] includes the [wallet-server] 2018-11-04 10:50:15 -05:00
Lex Berezhny ee1738da6e lbrynet.extras.wallet.server pylint fixes 2018-11-04 10:50:15 -05:00
Lex Berezhny 8ca2b75c80 lbryumx -> lbrynet.extras.wallet.server 2018-11-04 10:50:15 -05:00
Lex Berezhny dc81b6def1 install lbryumx after lbrynet so that we dont get the old protobuf 2018-11-04 10:50:15 -05:00
Lex Berezhny 22f5aaee70 regenerated protobufs 2018-11-04 10:50:15 -05:00
Lex Berezhny 4bd6569c9c lbrynet.schema.schema 2018-11-04 10:50:15 -05:00
Lex Berezhny dbf39baeb1 fix lbrynet.schema import 2018-11-04 10:50:15 -05:00
Lex Berezhny 3adf3c26f2 add set_build.py back in 2018-11-04 10:50:15 -05:00
Lex Berezhny fa0d13e630 protobuf==3.2.0 2018-11-04 10:50:15 -05:00
Lex Berezhny 227a82d6d3 update protobuf version 2018-11-04 10:50:15 -05:00
Lex Berezhny 65cfc1f63c removed lbryschema references from scripts and copied lbryschema requirements into local setup.py 2018-11-04 10:50:15 -05:00
Victor Shyba e46f0e2444 fix namespace lbryschema->lbrynet.schema 2018-11-04 10:50:15 -05:00
Victor Shyba 00bc7995c4 move lbryschema/tests/ into tests/unit/schema/ 2018-11-04 10:50:15 -05:00
Victor Shyba 2dcebe48b7 move lbryschema/lbryschema into lbrynet/schema 2018-11-04 10:50:15 -05:00
Lex Berezhny d12b6cddb3 fix integration tests 2018-11-04 02:28:36 -05:00
Lex Berezhny 699c057f2d upgrading to new torba 2018-11-04 02:28:36 -05:00
Lex Berezhny 3ce9f31f4b recursive channel lookup 2018-11-04 02:28:36 -05:00
Lex Berezhny 485d5d55e8 pylint fix 2018-11-04 02:28:36 -05:00
Lex Berezhny 6ee9f2513b add extra debug logging when a channel is not a claim errors 2018-11-04 02:28:36 -05:00
Daniel Brereton 81a6e14a5c added alt text 2018-11-02 19:38:30 -04:00
czakarian 496ffe0096 Fix typos / grammar changes 2018-10-31 16:35:39 -07:00
Preetham Viswanathan 216c760179 Merge pull request #1 from preethamvishy/preethamvishy/pr-template
Update pull request template
2018-10-30 17:16:56 -04:00
Preetham Viswanathan 01eca3004f Update PULL_REQUEST_TEMPLATE.md 2018-10-30 16:57:41 -04:00
GwanYeong Kim 2f47a3758e Remove unused Python modules in tests/unit/test_cli.py 2018-10-30 10:32:34 -04:00
pushkyn 73bcc1796f Add GitHub pull request template 2018-10-30 10:27:47 -04:00
Jack Robison e03e41ad31 bump 2018-10-26 16:51:23 -04:00
Jack Robison 6cb30f8e05 Merge branch 'fix-update-stream'
[API] add `aioupnp_version` to `upnp` field in the response to `status`
[API] fix uncaught claim signature errors
2018-10-26 16:49:51 -04:00
Jack Robison d9b1be6214 add aioupnp version to upnp status
-set non-blank not found message if upnp fails
2018-10-26 16:48:52 -04:00
Jack Robison 5f2f06c269 fix loggly 2018-10-26 13:01:50 -04:00
Jack Robison e1dcdcf4db catch claim signature error 2018-10-26 13:01:50 -04:00
Jack Robison 416472c5d8 log stream update failure 2018-10-26 13:01:46 -04:00
Alyssa Callahan 196f3ba92d added Internal use section 2018-10-24 15:47:07 -04:00
Lex Berezhny 6feae93c58 Update lbrynet/wallet/claim_proofs.py 2018-10-23 12:02:10 -04:00
Lex Berezhny 203d30d4f7 Update lbrynet/wallet/claim_proofs.py 2018-10-23 12:02:10 -04:00
Hugo d0a7ca841b Upgrade unit tests to use more useful asserts 2018-10-23 12:02:10 -04:00
Hugo 2e6c848c39 Upgrade Python syntax with pyupgrade --py36-plus 2018-10-23 12:02:10 -04:00
Jack Robison 18a0204105 set_build regex 2018-10-22 22:27:57 -04:00
Jack Robison 01baa193d5 fix set_build 2018-10-22 20:58:52 -04:00
Jack Robison ba57af5502 fix set_build 2018-10-22 17:55:32 -04:00
Jack Robison 8139f93ff1 set build type 2018-10-22 16:31:05 -04:00
Jack Robison 4596d5c26f bump 2018-10-22 09:12:49 -04:00
Nikita Titov 810a63782d Fixed long description of package 2018-10-20 10:15:29 -04:00
Nikita Titov ea83827d57 Update macOS naming 2018-10-20 10:02:24 -04:00
Jack Robison e19f8f3d95 bump 2018-10-19 12:28:08 -04:00
Jack Robison 7af501e0bc Merge branch 'temp-blob-pass-sqlite-object' 2018-10-19 12:27:27 -04:00
hackrush a692b6ff7d Pass SQLite storage object to DiskBlobManager for single(temp) blob download 2018-10-19 10:39:39 -04:00
Jack Robison 3364ac0976 bump 2018-10-18 18:22:46 -04:00
Jack Robison cf1157f02a Merge branch 'upnp-status' 2018-10-18 18:21:53 -04:00
Jack Robison 0ed56bbe1a tests 2018-10-18 18:12:50 -04:00
Jack Robison 213bdac55f check upnp redirects and external ip in a looping call
-add upnp setup success/fail analytics event
2018-10-18 17:41:49 -04:00
Jack Robison 6d8ab97ca9 fix analytics 2018-10-18 17:41:17 -04:00
Jack Robison fb34476cd7 add upnp status dictionary to status response
-log upnp debugging information as an error
2018-10-18 15:10:00 -04:00
Jack Robison 07d353a947 bump 2018-10-18 13:19:25 -04:00
Jack Robison 3c5fb6490d catch errors 2018-10-18 13:00:27 -04:00
Jack Robison a5d21081b9 Merge pull request #1549 from hugovk/link-to-key
Add link to GPG key
2018-10-18 12:41:42 -04:00
Jack Robison ed98a3cc21 Merge branch 'master' into link-to-key 2018-10-18 12:39:39 -04:00
Jack Robison 45c2cd2986 Merge pull request #1550 from hugovk/fix-spelling
Fix spelling
2018-10-18 12:37:56 -04:00
Hugo 75f21950da Add link to GPG key
And fix some typos.
2018-10-18 14:44:48 +03:00
Hugo 3f704be85c Fix spelling 2018-10-18 14:40:37 +03:00
Jack Robison 67a15ea788 bump 2018-10-18 00:34:58 -04:00
Jack Robison 55bcd2dfee Merge branch 'fix-asyncio-build' 2018-10-18 00:34:04 -04:00
Jack Robison d9757fdae8 remove asyncio from requirements 2018-10-18 00:32:44 -04:00
Jack Robison a4ae8d7c53 bump 2018-10-17 21:36:54 -04:00
Jack Robison 78078b4c9f Merge remote-tracking branch 'origin/fix_dht_logging' 2018-10-17 21:26:46 -04:00
Victor Shyba 1331e2bbe6 fix DHT logging 2018-10-17 22:25:41 -03:00
Lex Berezhny 4c4d652f1b fixes for #1526 and #1483 2018-10-17 21:10:23 -04:00
Lex Berezhny 73bd76b631 format publish output bid value in lbc instead of dewies 2018-10-17 19:50:31 -04:00
Lex Berezhny 59a0f9f602 removed the max response size from Known Issues as it was fixed 2018-10-17 19:38:14 -04:00
Lex Berezhny 1b1847bc31 added Known Issues section 2018-10-17 19:38:14 -04:00
Lex Berezhny 21a569edcc updated CHANGELOG for v0.30 2018-10-17 19:38:14 -04:00
Lex Berezhny 2ddf1a08f6 publish command supports custom account list to lookup channels 2018-10-17 19:37:54 -04:00
Victor Shyba 8754c6fabe fix resolve_name for asyncio 2018-10-17 17:17:25 -04:00
Victor Shyba f049109a38 fix claim_show 2018-10-17 17:17:25 -04:00
Victor Shyba 8db7c15aa1 fix address query handler 2018-10-17 15:33:22 -04:00
Victor Shyba 90e0694d2e fix download from peer script 2018-10-17 15:33:22 -04:00
Victor Shyba 4748cbf654 fix tests for cost estimate 2018-10-17 14:16:00 -04:00
Victor Shyba 74eb991983 fix paid downloads 2018-10-17 14:16:00 -04:00
Victor Shyba e11bae2dda stream cost estimate fix for uri without size 2018-10-17 14:16:00 -04:00
Victor Shyba 8462f56b13 fix blob_list for uris 2018-10-17 14:16:00 -04:00
Victor Shyba 0bec3c7294 fix claim_list_by_channel 2018-10-17 14:16:00 -04:00
Victor Shyba 2bb4052193 fix claim_list 2018-10-17 14:16:00 -04:00
Lex Berezhny 101a3f2b22 respect --show_seed argument to account_list command 2018-10-17 13:45:01 -04:00
Lex Berezhny 04a2bde8dd removed extraneous asyncio import 2018-10-17 13:29:28 -04:00
Jack Robison a1368b05da txupnp --> aioupnp
-update INSTALL.md
2018-10-17 13:29:28 -04:00
Victor Shyba 99ea7a26b0 fix get for asyncio torba 2018-10-17 10:27:57 -04:00
Lex Berezhny 8b3a457925 improve integration test reliability 2018-10-16 21:28:47 -04:00
Lex Berezhny e02e2a249a fix wallet component 2018-10-16 17:09:21 -04:00
Lex Berezhny 3019a1c222 revert tox config 2018-10-16 17:09:21 -04:00
Lex Berezhny 1f3a19c100 pylint 2018-10-16 17:09:21 -04:00
Lex Berezhny 5eac4247e1 working integration tests 2018-10-16 17:09:21 -04:00
Lex Berezhny 1ca82b9d52 major facepalm 2018-10-16 17:09:21 -04:00
Lex Berezhny 21702ec43c asyncio progress for victor 2018-10-16 17:09:21 -04:00
Lex Berezhny d2083bc17e for victor 2018-10-16 17:09:21 -04:00
Lex Berezhny 52edf880e0 fix typo in .travis.yml file 2018-10-16 17:09:21 -04:00
Lex Berezhny 1f45af991f install dependencies needed for asyncio unit tests 2018-10-16 17:09:21 -04:00
Lex Berezhny 8afdd0b2e9 pylint fix 2018-10-16 17:09:21 -04:00
Lex Berezhny 99efc70ba3 run wallet unit tests separate from the other twisted trial based tests 2018-10-16 17:09:21 -04:00
Lex Berezhny 64e306801d updated wallet to use asyncio 2018-10-16 17:09:21 -04:00
Victor Shyba 330db61b36 report balance and summary on startup 2018-10-12 20:53:31 -04:00
Lex Berezhny 357763ff61 respect blockchain_name in daemon settings when creating initial wallet 2018-10-12 12:28:39 -04:00
Lex Berezhny 8a6e6b4d77 add category attribute in JSON RPC output for backwards compatibility 2018-10-12 11:08:01 -04:00
Lex Berezhny 64913c4288 JSON RPC output for TXOs now includes: height, confirmations and valid_signature 2018-10-12 10:41:49 -04:00
Victor Shyba f01ad0e4bd use smart_decode on transaction parsing to support legacy claims 2018-10-11 17:38:23 -04:00
Victor Shyba d9fcc458dc refactor part of peer logic out of try/catch 2018-10-10 15:27:03 -04:00
Victor Shyba 24a11d3f21 make peer search asynchronous 2018-10-10 15:27:03 -04:00
Victor Shyba 234eaf8890 fix download_blob_from_peer script 2018-10-10 15:27:03 -04:00
Tomasz Kopeć 9e2e53147e --conf option improvements (#1455)
* Conf file improvements
* Add test for loading config file
2018-10-10 13:44:51 -04:00
Victor Shyba fc41af5889 set wallet max_gap after sync 2018-10-10 13:38:14 -04:00
Victor Shyba 72ce0cd0b0 explicitly define a default for signatures as expected on 0.21 2018-10-10 11:36:20 -04:00
Jack Robison e4497c1327 Merge remote-tracking branch 'origin/fix-1414' 2018-10-10 11:16:45 -04:00
hackrush a1bf832ea6 fix blob_availability 2018-10-10 16:21:54 +05:30
Lex Berezhny d4e76500fb removed unused import 2018-10-10 00:02:50 -04:00
Lex Berezhny f898bb3d0e database index fix 2018-10-09 23:58:32 -04:00
Lex Berezhny ad13282494 txo (claim_id) column index 2018-10-09 23:40:02 -04:00
Lex Berezhny 73519f8c6a claim_name updated to use new account.get_claims() method 2018-10-09 21:46:54 -04:00
Lex Berezhny b2c95f35c7 fixed test_transaction reliability 2018-10-09 21:32:32 -04:00
Lex Berezhny 0bfccb86ff refactored pagination per @seanyesmunt feedback 2018-10-09 20:46:41 -04:00
Lex Berezhny 5118240897 fix log message in header downloader to log local_height as integer instead of float 2018-10-09 17:39:28 -04:00
Lex Berezhny 427fc179ac fix for #1523 2018-10-09 16:48:12 -04:00
Lex Berezhny 55bcc520e4 pagination method fixes and not showing fee for recieved TX in get_history 2018-10-09 14:46:44 -04:00
Lex Berezhny bc8251696e fix cli version test 2018-10-09 11:37:49 -04:00
Lex Berezhny 677ba25ccd use .format() instead of string concatenation 2018-10-09 11:15:11 -04:00
Alex Grintsvayg 95436b233c fix lbrynet --version 2018-10-09 11:15:11 -04:00
Lex Berezhny 82871b8f6b fix command line argument for account_balance and other paginated commands 2018-10-09 10:30:30 -04:00
Lex Berezhny 9525b87224 inlineCallbacks fix 2018-10-08 22:44:00 -04:00
Lex Berezhny f06098978c ledger.get_transaction dropped in favor of ledger.db.get_transaction 2018-10-08 19:51:44 -04:00
Jack Robison 141491f2da bump version 2018-10-08 16:11:36 -04:00
Victor Shyba 38006f7d29 migrate addresses before starting the migrated wallet 2018-10-08 15:29:57 -04:00
Lex Berezhny 3664c25d98 changed lbry account.get_balance() to have same confirmations default as baseaccount.get_balance() 2018-10-08 15:02:52 -04:00
Lex Berezhny 1747c9d866 removed --include_reserved from account_list command 2018-10-08 15:02:52 -04:00
Lex Berezhny 1066fbaa7c pylint fix 2018-10-08 15:02:52 -04:00
Lex Berezhny 7f7961f4cc updated integration tests after reducing required confirmations to 0 2018-10-08 15:02:52 -04:00
Lex Berezhny 9d9916548b added channel/signature annotation to all TXOs, support for pagination 2018-10-08 15:02:52 -04:00
Lex Berezhny ada5b55f22 remove unused import 2018-10-08 15:02:52 -04:00
Lex Berezhny a2f28e7788 + permanent_url and TXO annotations 2018-10-08 15:02:52 -04:00
Jack Robison c6092a4c79 bump version 2018-10-08 09:16:54 -04:00
Jack Robison 315a557019 Merge remote-tracking branch 'origin/1439' 2018-10-08 09:15:46 -04:00
Jack Robison 20221e1186 Merge remote-tracking branch 'origin/reduce_routing_table_ping_flood' 2018-10-08 09:15:30 -04:00
Victor Shyba 50003d2600 restore delay on ping queue 2018-10-03 19:21:48 -03:00
Victor Shyba ea6b2b98fb refactor ping queue 2018-10-03 19:21:48 -03:00
Victor Shyba 4890fdfb50 remove semaphore from ping queue 2018-10-03 19:21:48 -03:00
Lex Berezhny 2f95c3a9d1 wallet integration test fixes 2018-10-03 18:06:22 -04:00
Lex Berezhny 6039159ecc use d2l/l2d in integration tests 2018-10-03 18:06:22 -04:00
Lex Berezhny 8e4d8345bd removed unused import 2018-10-03 18:06:22 -04:00
Lex Berezhny 22df26eeb6 dewies_to_lbc and lbc_to_dewies 2018-10-03 18:06:22 -04:00
Lex Berezhny 7b9ff3e8b5 updated with torba refactoring and working claim_send_to_address 2018-10-03 12:32:01 -04:00
Jack Robison 1a5b2c08ee update INSTALL.md 2018-10-02 13:31:29 -04:00
Jack Robison b6babb1c9d bump 2018-10-02 11:07:51 -04:00
Jack Robison a88485c5a0 Merge branch 'fix-missing-wallet-dir' 2018-10-02 11:06:59 -04:00
Jack Robison 0cefa07e97 Merge branch 'fix-1501' 2018-10-02 11:06:35 -04:00
Jack Robison ca9d48d574 fix wallet directory not being created 2018-10-02 10:34:05 -04:00
Jack Robison e10e1de92f [API] fix https://github.com/lbryio/lbry/issues/1501 2018-10-02 10:26:02 -04:00
Jack Robison bdf6887791 Merge branch 'fix-reactor-tests' 2018-10-02 09:09:34 -04:00
Jack Robison 295f2c1115 update INSTALL.md and README.md 2018-10-01 19:42:23 -04:00
Jack Robison 78fcf6a44a add asyncio to requirements 2018-10-01 18:42:28 -04:00
Jack Robison 7dcebf99e0 don't delete the reactor from sys.modules unless in a build 2018-10-01 18:41:45 -04:00
Victor Shyba 463cbde4b1 cast dict_keys to list before using as shortlist 2018-10-01 15:23:50 -03:00
Victor Shyba 1adce8b227 reduce routing table ping flood 2018-10-01 15:17:10 -03:00
Jack Robison 673d259d7a Merge branch '1446' 2018-10-01 08:34:57 -04:00
Jack Robison d0b739a0b9 Merge branch 'fix_peer_ping' 2018-10-01 08:32:30 -04:00
Victor Shyba b4aa51444c fix peer_ping for inexistent peers 2018-10-01 02:09:47 -03:00
Victor Shyba 9178ca701c nullify and refresh tokens 2018-09-28 14:19:51 -03:00
Victor Shyba 7eb9f344f4 test recover from invalid tokens 2018-09-28 13:47:37 -03:00
Victor Shyba 9f8d3e69ae implement get_best_blockhash 2018-09-27 15:30:53 -04:00
Victor Shyba e1130147b5 fix format of resolve on inexistent URI 2018-09-27 15:30:53 -04:00
Jack Robison 0fb6f50aba bump 2018-09-26 20:23:49 -04:00
Jack Robison 91fc772e3c travis 2018-09-26 19:18:53 -04:00
Jack Robison 30b3d31af0 one more 2018-09-26 17:52:21 -04:00
Jack Robison 1a9f81ea14 travis 2018-09-26 17:20:24 -04:00
Jack Robison 5813df74a5 travis 2018-09-26 16:44:01 -04:00
Jack Robison dc6d803fed travis 2018-09-26 15:54:40 -04:00
Jack Robison f9cba69d03 Merge remote-tracking branch 'origin/fix_master' 2018-09-26 15:53:48 -04:00
Victor Shyba 17f9d7cfb2 fix analytics encoding 2018-09-26 16:45:53 -03:00
Jack Robison 42c43e86ef bump version 2018-09-26 14:38:44 -04:00
Jack Robison 8cd230eb6e changelog 2018-09-26 12:23:13 -04:00
Jack Robison dcc71713c3 Merge branch 'fix-1402-1411' 2018-09-26 11:47:42 -04:00
hackrush 3d72b2adf3 Review Fixes 2018-09-26 11:21:32 -04:00
hackrush 655ece18de Changelog update 2018-09-26 11:21:32 -04:00
hackrush cda14f7c69 Deprecated wallet_public_key and claim_renew commands 2018-09-26 11:21:32 -04:00
Jack Robison 3d36e36173 Merge branch 'ssl' 2018-09-26 11:19:48 -04:00
Jack Robison 2dd0221711 [API] support ssl, add use_https setting 2018-09-26 09:56:26 -04:00
Lex Berezhny 1fec9f52ef temporarily return floats until we fix desktop app to recieve string decimals 2018-09-26 07:24:25 -04:00
Lex Berezhny ca9ca09a00 wallet unit test fixes 2018-09-26 07:24:25 -04:00
Lex Berezhny b16c36b55f transaction_list produces same output as old version 2018-09-26 07:24:25 -04:00
Victor Shyba 1c2849235c send startup event right after setup 2018-09-25 17:49:23 -04:00
Jack Robison 202a957a1a Merge branch 'torba-encryption' 2018-09-25 14:59:54 -04:00
Jack Robison 66f33202c7 [API] add account_lock 2018-09-25 13:10:57 -04:00
Jack Robison f310bb6825 [API] fix account_unlock, account_decrypt, and account_encrypt commands 2018-09-25 13:10:55 -04:00
Lex Berezhny 1fb7ffd759 fix integration test to pass amount as decimal string 2018-09-24 01:25:23 -04:00
Mark Beamer Jr 3fcca4bc47 adjusted test to pass string instead of number 2018-09-24 01:25:23 -04:00
Mark Beamer Jr 12c73cf50c adjusted e2e test to use 1.0 instead of 1 per eukreign which came from lyoshenko. 2018-09-24 01:25:23 -04:00
Mark Beamer Jr 02d398613d removed unused imports 2018-09-24 01:25:23 -04:00
Mark Beamer Jr 65e13898c2 fixed pylint 2018-09-24 01:25:23 -04:00
Mark Beamer Jr 559afa465a Implemented removal of float usage. Floats converted to int internally. 2018-09-24 01:25:23 -04:00
Lex Berezhny c753a5825a switched all --account args to --account_id, fixed issues reported by @osilkin98 and regenerated api.json 2018-09-23 23:48:53 -04:00
Victor Shyba 4185a3b9e3 add a pyinstaller hook for coincurve 2018-09-23 22:42:41 -04:00
Lex Berezhny 2f4e6911d6 daemon api json serialization imporovements 2018-09-21 23:24:50 -04:00
Lex Berezhny bcf7218b66 fix integration test 2018-09-21 23:24:50 -04:00
Lex Berezhny e21ade7360 pylint 2018-09-21 23:24:50 -04:00
Lex Berezhny 53d4c149bc added --account to most commands, also claim_list_mine, transaction_list, address_is_mine and other cleanup 2018-09-21 23:24:50 -04:00
Lex Berezhny 0adcaa6a5d utxo_list, transaction_show 2018-09-21 23:24:50 -04:00
Victor Shyba ce4d734250 adds get_claims_for_name 2018-09-21 17:38:07 -04:00
hackrush 04e0933d36 Fixes report bug on python 3 2018-09-21 17:37:01 -04:00
Jack Robison bcf52c3f24 Merge branch 'download_analytics_v2' 2018-09-21 16:51:33 -04:00
Alex Grintsvayg a702afd010 track all downloads in a single db (in parallel with existing metrics) 2018-09-21 15:19:12 -04:00
hackrush ad492a09f7 Added integration tests and other review fixes 2018-09-20 19:45:06 -04:00
hackrush 790991ecad Use account and reduce code + raise exception on claim not found 2018-09-20 19:45:06 -04:00
hackrush f5789775a2 Change sql to use less code 2018-09-20 19:45:06 -04:00
hackrush 5cc0d365c2 Added support to abandon claim using txid and nout 2018-09-20 19:45:06 -04:00
Victor Shyba 5430a0f289 supress git version print statement to debug 2018-09-20 12:49:00 -04:00
Victor Shyba cc7656faa4 adapt content payment calls to new wallet 2018-09-19 10:13:14 -04:00
Victor Shyba 03415415ed improve error message when daemon is already running 2018-09-18 23:09:43 -04:00
Lex Berezhny 1e47ff5a83 argument name changed to avoid redefining variable 2018-09-18 23:08:58 -04:00
Lex Berezhny 88bad1f868 publish accepts channel_id or channel_name 2018-09-18 23:08:58 -04:00
Victor Shyba 91405ef232 explicitly convert page and page_size to int 2018-09-18 22:12:23 -04:00
Jack Robison 570c1163c1 Merge branch 'author-docstring' 2018-09-18 10:12:23 -04:00
Jack Robison 17827036ce update docstring 2018-09-18 10:10:36 -04:00
Jack Robison 338e6c2116 Merge branch 'publish-command-doc-fix' 2018-09-13 12:38:06 -04:00
hackrush cdf6155102 Added which characters are acceptable in the name field of publish command 2018-09-13 12:36:26 -04:00
hackrush e47ef741f7 Review Fixes 2018-09-09 22:00:37 -04:00
hackrush 2878716381 Tests for support and tip 2018-09-09 22:00:37 -04:00
hackrush 6d9947779e Added command to send a tip to a claim id 2018-09-09 22:00:37 -04:00
hackrush c7e5bdd3ae claim_new_support working in the new wallet 2018-09-09 22:00:37 -04:00
Victor Shyba 46dcf8a899 fix wallet reference bug 2018-09-09 21:10:51 -04:00
Victor Shyba c050231ac5 remove file_status flag 2018-09-09 21:10:51 -04:00
Victor Shyba ee5be5adc8 fail graceful when get is called on inexistent URI 2018-09-03 12:48:54 -04:00
hackrush d99ce5f3d6 Integration test for wallet_send (#1395) 2018-09-03 11:41:30 -04:00
Lex Berezhny e7634b74c8 improved test for setting account attributes 2018-09-03 09:09:28 -04:00
Lex Berezhny 26af01a5cc test account list for specific account 2018-09-03 09:09:28 -04:00
Lex Berezhny 36989f1974 removed unnecessary defer.returnValue() 2018-09-03 09:09:28 -04:00
Lex Berezhny 1d40ec34a2 account_balance returns Decimal intead of rounded float 2018-09-03 09:09:28 -04:00
Lex Berezhny f0f30fd75c link to torba repo 2018-09-03 09:09:28 -04:00
Lex Berezhny be2ef747dc deleted commented out code 2018-09-03 09:09:28 -04:00
Lex Berezhny 268a01dc23 run AccountManagement tests on travis 2018-09-03 09:09:28 -04:00
Lex Berezhny 8399f2ab33 account management 2018-09-03 09:09:28 -04:00
Lex Berezhny 7698ab7053 catch InvalidOperation exception when parsing decimals 2018-09-03 09:09:28 -04:00
Lex Berezhny eff66bdf80 use get_dewies_or_error in jsonrpc_publish 2018-09-03 09:09:28 -04:00
Lex Berezhny 70e68c6361 handle amount passed as float 2018-09-03 09:09:28 -04:00
Lex Berezhny 2a53afbfd8 dont fail migrating certs if lbryumx doesnt know about a cert claim 2018-09-03 09:09:28 -04:00
Lex Berezhny a42bf67800 look in all accounts for certificates 2018-09-03 09:09:28 -04:00
Lex Berezhny b783b4daed publish with certificates fix 2018-09-03 09:09:28 -04:00
Lex Berezhny c38b463bf6 added default_wallet and default_account properties to daemon 2018-09-03 09:09:28 -04:00
Lex Berezhny 8696cd3c27 fix for chris45 integration test 2018-09-03 09:09:28 -04:00
Lex Berezhny 0abbc20899 fix for cli test 2018-09-03 09:09:28 -04:00
Lex Berezhny 2fca29144e pylint: line too long fixes 2018-09-03 09:09:28 -04:00
Lex Berezhny 1675cc4580 renamed a bunch of wallet commands and split up the big account command 2018-09-03 09:09:28 -04:00
shyba 93d134f0ce [bug] remove get_block_info as it doesnt make sense, unify to get_block (#1425) 2018-08-31 13:12:13 -04:00
hackrush 1c6ce49b7b Removed unnecessary yield and inlineCallbacks, rectified command behaviour (#1421)
* wallet_prefill_addresses command follows specifications(docs)
* unnecessary to document bugs in changelog which were created and fixed without having ever been released
2018-08-30 15:26:04 -04:00
Jack Robison a7f34e2913 Merge remote-tracking branch 'origin/fix_rerun' 2018-08-28 11:54:33 -04:00
Victor Shyba 2a285d139a bump attempts to 5 and properly check failure 2018-08-27 22:51:16 -03:00
shyba 5e371e8e6a [API] Fixes for block_show and claim_show (#1390)
block_show using block hash
block_show using height
claim_show from claim_id
claim_show from outpoint
2018-08-27 20:03:08 -04:00
Lex Berezhny fdc98fc746 Update README.md
Fixed download links which were pointing to very outdated versions of daemon.
2018-08-26 00:10:38 -04:00
Lex Berezhny 0d8e55bc95 Remove icons directory (#1389)
* removed icons directory, saved 48x48.png icon to be the README.md icon
* removed extra whitespace in lbry.png
* point README to new icon
2018-08-24 17:36:52 -04:00
Lex Berezhny 69851a6588 README and other post lbryum-refactor merge cleanup (#1388)
* updated README with correct Python version, point code coverage to codecov, added header icon and fixed jacks email link
* removed .appveyor.yml, requirements.txt, requirements_testing.txt
2018-08-24 16:44:26 -04:00
Jack Robison b101fafd39 Merge branch 'lbryum-refactor' 2018-08-24 15:04:35 -04:00
Victor Shyba d2aaf7bd09 revert travis to older lbryumx for now 2018-08-24 15:15:57 -03:00
Jack Robison 5a1f551ce5 requirements and bytes/string 2018-08-24 12:21:11 -04:00
Lex Berezhny 584433e16c updated CHANGELOG 2018-08-24 11:42:35 -04:00
Lex Berezhny 0c9c6c2705 create wallet directory if it doesnt exist 2018-08-24 11:42:35 -04:00
Lex Berezhny afae649a9e stubbed out buy/sell claim 2018-08-24 11:42:35 -04:00
Lex Berezhny 2238e1f802 wallet_send command working (#1382)
* wallet_send command
2018-08-24 11:42:35 -04:00
Victor Shyba 518c447fef standardize distance.py to int.from_bytes 2018-08-24 11:42:35 -04:00
Lex Berezhny cb9b348828 WalletComponent expects manager to have is_wallet_unlocked, added a stubbed method for now 2018-08-24 11:42:35 -04:00
shyba eab95a6246 DHT fixes from review and an attempt at removing hashing and equals (#1370)
* use int to_bytes/from_bytes instead of struct
* fix ping queue bug and dht functional tests
* run functional tests on travis
* re-add contact comparison unit test
* dont need __ne__ if its just inverting __eq__ result
2018-08-24 11:42:35 -04:00
Lex Berezhny 593d0046bd Revert "bytes/string"
This reverts commit 594dd61
2018-08-24 11:42:35 -04:00
Lex Berezhny c5d1478c02 changed default maximum number of times an address should get used to 1 from 2, per lbrynaut review 2018-08-24 11:42:35 -04:00
Lex Berezhny 1c57293e9c amount comparison should be 0, per lbrynaut review 2018-08-24 11:42:35 -04:00
hackrush 14a067a4c4 wallet_prefill_command working 2018-08-24 11:42:35 -04:00
hackrush 707ab831fd Fake Analytics for test_cli 2018-08-24 11:42:35 -04:00
hackrush ceb25f917a Use get_dewies_or_error method instead 2018-08-24 11:42:35 -04:00
hackrush 002faf7a58 Convert str to Decimals 2018-08-24 11:42:35 -04:00
Jack Robison d1bbb7af74 update required twisted version 2018-08-24 11:42:35 -04:00
Jack Robison ea603dce13 back up original lbryum wallet before migration
-use same directory and file name for new wallet as the old wallet
-catch error when trying to migrate an abandoned certificate
2018-08-24 11:42:35 -04:00
Jack Robison f2a768c9c4 bytes/string 2018-08-24 11:42:35 -04:00
Jack Robison 12384a89a1 remove wsgiref from requirements 2018-08-24 11:42:35 -04:00
Jack Robison c21956b4f3 handle distro import error 2018-08-24 11:42:35 -04:00
Lex Berezhny 4a8776be10 wallet integration tests fixed 2018-08-24 11:42:35 -04:00
Lex Berezhny d6d69a46d4 added missing yield 2018-08-24 11:42:35 -04:00
Jack Robison 4384a5f022 fix json encoding of bytes 2018-08-24 11:42:35 -04:00
hackrush aa83bbd637 sq close the goddamn session 2018-08-24 11:42:35 -04:00
hackrush daae0b2836 sq return not yield, run on clear field 2018-08-24 11:42:35 -04:00
Lex Berezhny 5afd446bb3 fixing wallet unit tests test_ledger.py and test_transactions.py 2018-08-24 11:42:35 -04:00
hackrush 2b5e3204c0 Refactor client, cli and test_cli, fix delayedCalls not expiring 2018-08-24 11:42:35 -04:00
Lex Berezhny 84c91c480f fix for unit/wallet/test_accounts.py 2018-08-24 11:42:35 -04:00
Lex Berezhny cbc98e72a7 pylint fixes 2018-08-24 11:42:35 -04:00
Lex Berezhny d1f00255c1 test fixes related to headers 2018-08-24 11:42:35 -04:00
Lex Berezhny 434d98c6a6 restore formatting 2018-08-24 11:42:35 -04:00
Lex Berezhny e51c98ca8c formatting fixes 2018-08-24 11:42:35 -04:00
Lex Berezhny f719aa1db0 old command is called daemon_stop 2018-08-24 11:42:35 -04:00
Lex Berezhny 3686b1d970 changes from jacks review 2018-08-24 11:42:35 -04:00
Lex Berezhny e6fd8cc0f2 wip new header stuff 2018-08-24 11:42:35 -04:00
Lex Berezhny 8739dc5d2c pin faker to 0.8.17 2018-08-24 11:42:33 -04:00
hackrush d6c04d3586 sq Fix tests 2018-08-24 11:42:08 -04:00
hackrush 7236203f1e sq Add new integration tests to tox.ini 2018-08-24 11:42:08 -04:00
hackrush 8c6c442fdd Fixed regression in unauthenticated API client w/ integration tests 2018-08-24 11:42:08 -04:00
Lex Berezhny 49c659b832 assert integration test is actually using authentication 2018-08-24 11:42:08 -04:00
hackrush 2590523fa1 Rename client and add test_cli to tox.ini 2018-08-24 11:42:08 -04:00
hackrush 08b4fd6942 stop listening port only if it is not None 2018-08-24 11:42:08 -04:00
hackrush 6a8963d807 Added Authenticated API Client Integration Test 2018-08-24 11:42:08 -04:00
hackrush 083bf4f61d aiohttp based Async Authenticated API Client 2018-08-24 11:42:08 -04:00
Lex Berezhny ab3b0134b5 Purchase Transaction Type 2018-08-24 11:42:08 -04:00
Victor Shyba 3b88d2465d fixes and refactors from review 2018-08-24 11:42:08 -04:00
Victor Shyba 0841c90e6c fix encoding on download 2018-08-24 11:42:06 -04:00
Victor Shyba 3f6e928cc2 fix findValue result parsing 2018-08-24 11:40:43 -04:00
Victor Shyba a32a0c6401 fix peer_ping 2018-08-24 11:40:43 -04:00
Victor Shyba 95cb29d3c8 fix routing_table_get 2018-08-24 11:40:43 -04:00
Victor Shyba e85f8b245b fix conf.py 2018-08-24 11:40:43 -04:00
Victor Shyba 04836ea0d9 fixes from review 2018-08-24 11:40:43 -04:00
Victor Shyba 451823f33e test_misc and test_streamify functionals on py3 2018-08-24 11:40:43 -04:00
Victor Shyba 8c3552dd34 port reflector 2018-08-24 11:40:43 -04:00
Lex Berezhny a204f0d3e6 - replaced old lbryum function with struct.pack, per @BrannonKing review
- make copy of keys so we can modify the dictionary
2018-08-24 11:40:43 -04:00
Lex Berezhny 892758be65 account.from_dict() takes wallet as second arg 2018-08-24 11:40:43 -04:00
Lex Berezhny 3594c8976d improved output when migrating certificates 2018-08-24 11:40:43 -04:00
Lex Berezhny 23ede44bff fix blob downloader test on py3
not tying lbrynet db to wallet db just yet
2018-08-24 11:40:43 -04:00
Lex Berezhny 0006a68bf2 fix import in blob downloder tests 2018-08-24 11:40:43 -04:00
Lex Berezhny 0349e68201 moved unit tests into unit test directory 2018-08-24 11:40:43 -04:00
Lex Berezhny be248dc448 account command can now configure the default account 2018-08-24 11:40:43 -04:00
Lex Berezhny d0d5d0340d unit tests fixed 2018-08-24 11:40:43 -04:00
Lex Berezhny f79a49bbf4 added account management command 2018-08-24 11:40:43 -04:00
Lex Berezhny 6a5d88a0d5 new fund command and automatic account creation 2018-08-24 11:40:43 -04:00
hackrush 4165881d5f Authenticated CLI works again 2018-08-24 11:40:43 -04:00
Lex Berezhny 8dc4e3be43 integration tests working again and daemon starts normally 2018-08-24 11:40:43 -04:00
Lex Berezhny 0badea874d test_Downloder tests fixed 2018-08-24 11:40:43 -04:00
Lex Berezhny 0dd6193eaa FakeComponent was missing __lt__ comparison operator and couldnt be put in set() 2018-08-24 11:40:43 -04:00
Lex Berezhny d10778430f upnpclient -> txupnp 2018-08-24 11:40:43 -04:00
Lex Berezhny da8e09846d pylint fixed 2018-08-24 11:40:43 -04:00
Lex Berezhny 310fe4a42c updated Headers component to use lbrynet.wallet 2018-08-24 11:40:43 -04:00
hackrush a7ef8889dd Unified CLI, python 3(WIP) (#1330)
* Added new custom cli class using aiohttp
* Proper error handling in CLI based on RPC error codes(PoC)
* Auth API working
* UnitTests
2018-08-24 11:40:43 -04:00
Lex Berezhny f41229cb5b tx.get_claim_id() -> txo.claim_id, claim update works now
channel_list encodes claim name and also added claim_id
fixed BlobManager foreign key error handling
2018-08-24 11:40:43 -04:00
Lex Berezhny 8ab4e3ca49 + channel_list command works again but test_commands.py integration test is failing 2018-08-24 11:40:43 -04:00
Victor Shyba ff8d37443e more checks on test_transactions 2018-08-24 11:40:43 -04:00
Lex Berezhny fcd46629c4 refactored how transactions are created, fixed list addresses command 2018-08-24 11:40:43 -04:00
Victor Shyba 9ad9eb083b fix integration test NBO encoding 2018-08-24 11:40:43 -04:00
Lex Berezhny 4669507880 Transaction.abandon() now requires list of outputs 2018-08-24 11:40:43 -04:00
Lex Berezhny 9348f4f366 add missing new line at end of file 2018-08-24 11:40:43 -04:00
Lex Berezhny 47bb634035 abandon claims and chris45 epic adventure 2018-08-24 11:40:43 -04:00
Lex Berezhny 10b34d6b33 unnecessary list() added during py3 port
instead of recursive bytes2unicode use a proper JSONEncoder to conver bytes->unicode for json.dumps()
removing excessive isinstance(data, bytes) checks
py3: / -> // and list() around .items() that gets modified in loop
moved lbrynet.undecorated to where its actually used and hopefully we can delete it eventually
removed build/upload_assets.py, travis can do all this now
2018-08-24 11:40:43 -04:00
Lex Berezhny f061ca2b15 integration tests no longer mock Blob, BlobManager and Session 2018-08-24 11:40:43 -04:00
Lex Berezhny bc24dbea29 refactoring of DHT tests and fixed encoding bug when dealing with bytearray 2018-08-24 11:40:43 -04:00
Lex Berezhny 2d4bf73632 working on unit tests 2018-08-24 11:40:43 -04:00
Lex Berezhny c21d8cc8dc update wallet migration 2018-08-24 11:40:43 -04:00
Lex Berezhny cc6a91a477 windows travis build is lagnuage generic 2018-08-24 11:40:43 -04:00
Lex Berezhny 21a2725215 travis 2018-08-24 11:40:43 -04:00
Lex Berezhny 411b8c74cc travis yaml conslidation via anchors 2018-08-24 11:40:43 -04:00
Lex Berezhny 91bf312e4d fix for file open during publish 2018-08-24 11:40:43 -04:00
Lex Berezhny 277b8b122c run integration tests with 3.7 2018-08-24 11:40:43 -04:00
Lex Berezhny 59c49d2d31 dont run DHT tests, until we can make them run faster 2018-08-24 11:40:43 -04:00
Lex Berezhny 16024c95aa py2->py3 unhexlify() 2018-08-24 11:40:43 -04:00
Lex Berezhny 923ed80fcf run DHT tests as their own job 2018-08-24 11:40:43 -04:00
Lex Berezhny 2ee2916f78 run functional DHT tests on travis 2018-08-24 11:40:43 -04:00
Lex Berezhny 79bf1b427f work around for a bug in travis artifact uploader 2018-08-24 11:40:43 -04:00
Lex Berezhny c21c329aa0 debug artifact uploading 2018-08-24 11:40:43 -04:00
Lex Berezhny 6b3af4bc21 fix for travis buid to put build binary in right place 2018-08-24 11:40:43 -04:00
Lex Berezhny ffbdf03847 switched from testing on 3.7 to 3.6 2018-08-24 11:40:43 -04:00
Lex Berezhny 8d08bfb3f9 moved build/icons up one level and deleted build directory 2018-08-24 11:40:43 -04:00
hackrush 1af6326b97 Fix account_balance's doc strings 2018-08-24 11:40:43 -04:00
Victor Shyba 31630a84be reduce entropy on DHT test suite 2018-08-24 11:40:43 -04:00
hackrush 0601bf3197 sq Learning better coding practices with Lex 2 2018-08-24 11:40:43 -04:00
hackrush 826d2866fa sq Learning better coding practices with Lex 2018-08-24 11:40:43 -04:00
hackrush ba00498d83 Make commands work from terminal 2018-08-24 11:40:43 -04:00
Lex Berezhny 4da40e8d52 renamed build binary to lbrynet and other cleanup 2018-08-24 11:40:41 -04:00
Lex Berezhny d54d989c40 change stop command from daemon_stop to just stop lbrynet 2018-08-24 11:38:20 -04:00
Lex Berezhny 18ecbba87c fix cli formatting 2018-08-24 11:38:20 -04:00
Lex Berezhny 9b417486a9 altogether now! 2018-08-24 11:38:20 -04:00
Lex Berezhny 6d208ac496 trying pip3 on mac 2018-08-24 11:38:20 -04:00
Lex Berezhny 8990c145fb python already installed on mac 2018-08-24 11:38:20 -04:00
Lex Berezhny e5b1c3c829 dont manually install python on mac 2018-08-24 11:38:20 -04:00
Lex Berezhny 6c7128c0f6 trying mac build 2018-08-24 11:38:20 -04:00
Lex Berezhny f31f1fa40b switching down to python 3.6 until pyinstaller and wine support 3.7 2018-08-24 11:38:20 -04:00
Lex Berezhny 2ceba79b9f linux build added and uploading to S3 2018-08-24 11:38:19 -04:00
Lex Berezhny 1c8a59c1a9 upgrade upload assets script to py3 2018-08-24 11:37:53 -04:00
Lex Berezhny 298bf91293 moved build/upload_assets.py -> scripts/upload_assets.py 2018-08-24 11:37:53 -04:00
Victor Shyba b25d592d99 more dht fixes and most of functional tests 2018-08-24 11:37:53 -04:00
Lex Berezhny 875edb5f76 + aiohttp dependency added 2018-08-24 11:37:53 -04:00
Lex Berezhny 84d97ab323 + account_max_gap command 2018-08-24 11:37:53 -04:00
Lex Berezhny aeaffd620e simple cli implementation 2018-08-24 11:37:53 -04:00
hackrush 3f7091cbf2 Make curl work in py3 again 2018-08-24 11:37:53 -04:00
hackrush a95f791215 Make daemon work in py3 again 2018-08-24 11:37:53 -04:00
Lex Berezhny af2aeaa66b fixing unit tests 2018-08-24 11:37:53 -04:00
Lex Berezhny 855fd8bf9a moved test_customLogger.py 2018-08-24 11:37:53 -04:00
Lex Berezhny a937aff80f pylint fixes 2018-08-24 11:37:53 -04:00
Lex Berezhny 5597d45aed got integration tests to work again with rebased branch 2018-08-24 11:37:53 -04:00
hackrush 473d2eabfa curl works with python3 now 2018-08-24 11:37:53 -04:00
Victor Shyba 78c8c8e64d more porting, plus some functional tests working 2018-08-24 11:37:53 -04:00
Lex Berezhny 7335d012ef replace miniupnpc with upnpclient
docker build with wine
2018-08-24 11:37:53 -04:00
Victor Shyba 99be38604a attempt to fix tests isolation 2018-08-24 11:37:53 -04:00
Lex Berezhny d35d340613 fixing unit tests
fixing integration tests
skip running functional tests on travis until they are fixed
2018-08-24 11:37:53 -04:00
Lex Berezhny f05ca137be adding build stages to .travis.yml 2018-08-24 11:37:53 -04:00
Lex Berezhny e718caca77 final pylint fixes (hopefuly) 2018-08-24 11:37:53 -04:00
Lex Berezhny 693a3346d2 pylint fixes 2018-08-24 11:37:53 -04:00
Lex Berezhny 1f75b6a49e pylint skips assignment-from-no-return, useless-return 2018-08-24 11:37:53 -04:00
Lex Berezhny 4ece422f48 No longer inheriting from object and added proper use of super(). 2018-08-24 11:37:53 -04:00
Lex Berezhny 607677f98f iterkeys() -> keys() 2018-08-24 11:37:53 -04:00
Victor Shyba e1e7be63b8 more fixes on dht functionals 2018-08-24 11:37:53 -04:00
Victor Shyba e1314a9d1e working functional test_contact_rpc + more string bans 2018-08-24 11:37:53 -04:00
Victor Shyba 1ee682f06f make bencoding right for both py3+py2 2018-08-24 11:37:53 -04:00
Lex Berezhny ba80c0e594 py2(iteritems, itervalues) -> py3(items, values) 2018-08-24 11:37:53 -04:00
Lex Berezhny fbdbcc8070 assertEquals() -> assertEqual() 2018-08-24 11:37:53 -04:00
Lex Berezhny 24a872885a xrange() -> range() 2018-08-24 11:37:53 -04:00
Lex Berezhny 005a131440 require python 3.7 2018-08-24 11:37:53 -04:00
Lex Berezhny 0b1cb16353 windows related fixes, in py3 everything is unicode, no need to convert paths to bytes 2018-08-24 11:37:53 -04:00
Lex Berezhny 1a3eaf7e6a unhexlify/hexlify some values in database/storage.py 2018-08-24 11:37:53 -04:00
Lex Berezhny 971252d5d1 undo simplejson commit 2018-08-24 11:37:53 -04:00
Lex Berezhny c5ba9b263e use twisted[tls] instead of specifying twisted dependencies in our own setup.py 2018-08-24 11:37:53 -04:00
hackrush c01716a6c0 Using simplejson and fixed some encodes and decodes 2018-08-24 11:37:53 -04:00
hackrush 0fbf131df3 default certificates is a dict now 2018-08-24 11:37:53 -04:00
hackrush 4bd0806e58 Fix metaclass not registering objects 2018-08-24 11:37:53 -04:00
hackrush 4cbb71705e Added pyopenssl and service-identity dependency to setup.py 2018-08-24 11:37:53 -04:00
Lex Berezhny 8ab4dd6b2c build py37, using ubuntu xenail on traivs 2018-08-24 11:37:53 -04:00
Lex Berezhny 318c369752 made tests into a package again and moved mocks.py back to root tests dir 2018-08-24 11:37:53 -04:00
Victor Shyba cea3b7630c partial dht functional tests porting to py3 2018-08-24 11:37:53 -04:00
Victor Shyba 19211d4417 make sure bencoding works for bytes, not strings 2018-08-24 11:37:53 -04:00
Victor Shyba c312d1b3a6 all unit/dht green on py3 2018-08-24 11:37:53 -04:00
Victor Shyba e4ea1ccbfb test_node green on py3 2018-08-24 11:37:53 -04:00
Victor Shyba 5b35c4e8f0 port dht.distance ~> py3 2018-08-24 11:37:53 -04:00
Victor Shyba 9967857a57 port dht.encoding ~> py3 2018-08-24 11:37:53 -04:00
Victor Shyba 78c560a3be general incomplete porting of daemon booting ~> py3 2018-08-24 11:37:53 -04:00
Victor Shyba 07a12b66e8 port dht.contacts ~> py3 2018-08-24 11:37:53 -04:00
Lex Berezhny a28c9d09c8 - fix and improvements related to the two balance command account_balance and wallet_balance
- working CommonWorkflowTests integration test
- pylint, unit and integration test fixes
- switch integration tests to use async/await
2018-08-24 11:37:53 -04:00
Victor Shyba b2ab13187f fix get_certificates 2018-08-24 11:37:53 -04:00
Victor Shyba 5ec696fbfd fix lookup helper by not calling hexlify twice 2018-08-24 11:37:53 -04:00
Victor Shyba 44bd0ae847 fix another forgotten claim id nbo conversion 2018-08-24 11:37:53 -04:00
Victor Shyba 5809ba093d convert amount on channel_new 2018-08-24 11:37:53 -04:00
Lex Berezhny 17aee92dc9 wallet_balance --include-unconfirmed works now 2018-08-24 11:37:53 -04:00
Victor Shyba 6f1d2e7690 fix RPC publish claim serialization 2018-08-24 11:37:53 -04:00
Victor Shyba 75f57fc517 fix claim id handling during publish from the latest changes 2018-08-24 11:37:53 -04:00
Lex Berezhny f589410496 tests 2018-08-24 11:37:53 -04:00
Lex Berezhny aecc7c6643 resolve works in py3!!!!!11111oneoneone 2018-08-24 11:37:53 -04:00
Victor Shyba c5c1b62939 fix resolve from tx.id rename 2018-08-24 11:37:53 -04:00
Lex Berezhny adc92a2b52 fix for daemon.get_est_cost 2018-08-24 11:37:53 -04:00
Lex Berezhny 9636ca22e7 fixing unit tests 2018-08-24 11:37:53 -04:00
Lex Berezhny 0442770c9d pylint fixes 2018-08-24 11:37:53 -04:00
Lex Berezhny 3b6a879fc9 add is_hd=true for default migrated wallet 2018-08-24 11:37:53 -04:00
Lex Berezhny a1b4c9acd6 claim_id is hexlified before saving to sqlite 2018-08-24 11:37:53 -04:00
Lex Berezhny 0f90dee224 faililng test 2018-08-24 11:37:53 -04:00
Lex Berezhny 159e84468c py3 support in resolve 2018-08-24 11:37:53 -04:00
Lex Berezhny 59e4ac30c2 fixing py2 stuff that broke during py3 fixes 2018-08-24 11:37:53 -04:00
Lex Berezhny b5ce948bc1 convert buffer to string 2018-08-24 11:37:53 -04:00
Lex Berezhny 076af7ef43 py3 compatibility updates 2018-08-24 11:37:53 -04:00
Lex Berezhny 43bef9447c progress on publish command: py3 porting and integration tests 2018-08-24 11:37:53 -04:00
Victor Shyba b1c5fe0b4d cryptblob write needs bytes on py3 2018-08-24 11:37:53 -04:00
Victor Shyba 39d7f2e46e port cryptblob test to py3 2018-08-24 11:37:53 -04:00
Lex Berezhny 315661208d integration test publishes actual file instead of just sources 2018-08-24 11:37:53 -04:00
Lex Berezhny e175e43082 .decode() string for sqlite 2018-08-24 11:37:53 -04:00
Lex Berezhny cb9b320ed7 dropped make_claim in favor of wallet.claim_name 2018-08-24 11:37:53 -04:00
Lex Berezhny 1d906fb0c7 create_and_publish_stream fix 2018-08-24 11:37:53 -04:00
Lex Berezhny c544e26206 publish integration test works 2018-08-24 11:37:53 -04:00
Lex Berezhny 1eaac35a3e claim_name does not take change_address anymore 2018-08-24 11:37:53 -04:00
Lex Berezhny cc7962fc2e stringify address from sqlite 2018-08-24 11:37:53 -04:00
Lex Berezhny 3638984d67 fixed getting unused address 2018-08-24 11:37:53 -04:00
Lex Berezhny 62e77c69f5 jsonrpc_publish takes amount in lbc, not satoshi 2018-08-24 11:37:53 -04:00
Lex Berezhny 774d813576 fixed encoding issue on saving wallet 2018-08-24 11:37:53 -04:00
Lex Berezhny 688caf4453 + save wallet after creating a channel 2018-08-24 11:37:53 -04:00
Lex Berezhny eeb5b3c139 publishing 2018-08-24 11:37:53 -04:00
Lex Berezhny b5e6dd1060 + maybe_migrate_certificates 2018-08-24 11:37:53 -04:00
Lex Berezhny 272d818050 saving txo bug fix 2018-08-24 11:37:53 -04:00
Lex Berezhny 0fc1bd7a8c store claim_id 2018-08-24 11:37:53 -04:00
Lex Berezhny 2b234e0713 fix syntax error 2018-08-24 11:37:53 -04:00
Victor Shyba 9e8cb17ecb insane working version of paged resolve, to be refactored 2018-08-24 11:37:53 -04:00
Victor Shyba e0230864ce creates a Resolve class for moving resolve methods out of ledger 2018-08-24 11:37:53 -04:00
Victor Shyba 9e89b3e40e start a new module for isolating resolve related code before we can refactor it 2018-08-24 11:37:53 -04:00
Victor Shyba 036663ae62 adds get_claims_by_ids and fix a test name 2018-08-24 11:37:53 -04:00
Victor Shyba b62321689d test proofs 2018-08-24 11:37:53 -04:00
Victor Shyba bdd271e78f adds resolve 2018-08-24 11:37:53 -04:00
Lex Berezhny 4a56d38782 certificate.py 2018-08-24 11:37:53 -04:00
Lex Berezhny 70a7ca95fe wip 2018-08-24 11:37:53 -04:00
Lex Berezhny f85e61d8ed run functional tests first
run two integration tests on travis 😱
reduce integration test verbosity
2018-08-24 11:37:53 -04:00
Lex Berezhny ac8c641d64 use lbryumx branch of electrumx 2018-08-24 11:37:53 -04:00
Lex Berezhny d81502e191 unit test fixes, balance/utxos filters out claims, abandoning claims 2018-08-24 11:37:53 -04:00
Lex Berezhny 8c35d355e5 added is_claim,is_support,is_update and claim_name attributes to txo able 2018-08-24 11:37:53 -04:00
Lex Berezhny 72b004664f config fix 2018-08-24 11:37:53 -04:00
Lex Berezhny 49a7408968 travis-ci stuff: only run pylint for py27 in tox and checkout lbryschema 2018-08-24 11:37:53 -04:00
Lex Berezhny 17738ad248 import fixes due to tests directory moving 2018-08-24 11:37:53 -04:00
Lex Berezhny 425c952293 added tox 2018-08-24 11:37:53 -04:00
Lex Berezhny 03d2d0e237 fixed import paths as a result of moving the tests 2018-08-24 11:37:53 -04:00
Lex Berezhny edfc2c6124 fix to make sd_hash more deterministic 2018-08-24 11:37:53 -04:00
Lex Berezhny 545451a829 moved ./lbry/tests to ./tests 2018-08-24 11:37:53 -04:00
Lex Berezhny 12ff7015cd wallet tests and py3 compatibility 2018-08-24 11:37:53 -04:00
Lex Berezhny f2f212edba switch component wallet to use the new wallet 2018-08-24 11:37:53 -04:00
Lex Berezhny 69446491b8 pylint and unit test fixes 2018-08-24 11:37:50 -04:00
Lex Berezhny d5beaa0937 jsonrpc_channel_new uses new wallet 2018-08-24 11:37:25 -04:00
Lex Berezhny cc970c499b daemon/auth py3 conversion, this breaks the client for now 2018-08-24 11:37:25 -04:00
Lex Berezhny 0087b16912 deleted old broken test 2018-08-24 11:37:25 -04:00
Lex Berezhny 5520d518b5 DHT py3 compatibility, mostly commenting out implements() and fixing imports
cryptstream py3 support, mostly commenting out implements()
lbry_file py3 support, mostly commenting out implements()
file_manager py3 support, mostly commenting out implements()
core py3 support, mostly commenting out implements() and fixing imports
2018-08-24 11:37:25 -04:00
Lex Berezhny b3e9240aa8 wip, valuesforuris 2018-08-24 11:37:25 -04:00
Lex Berezhny 137a270488 * channel certificate creation and taking advantage of is_reserved feature on txos to exclude output from being used in new transactions 2018-08-24 11:37:25 -04:00
Lex Berezhny e36610d63c fix resolve 2018-08-24 11:37:25 -04:00
Lex Berezhny c4273fbef0 claim test 2018-08-24 11:37:25 -04:00
Lex Berezhny a4e9f3100f updating wallet tests 2018-08-24 11:37:25 -04:00
Lex Berezhny 2668ceac0e updated to newer torba 2018-08-24 11:37:25 -04:00
Lex Berezhny fb2fb5c38c recovered an old stash 2018-08-24 11:37:25 -04:00
Lex Berezhny 56175df121 wip 2018-08-24 11:37:25 -04:00
Lex Berezhny 7d3daa9fe6 set the lbryschema BLOCKCHAIN_NAME global variable 2018-08-24 11:37:25 -04:00
Lex Berezhny 1d2de27536 import fix 2018-08-24 11:37:25 -04:00
Lex Berezhny aab42e56bd removing lbryum 2018-08-24 11:37:25 -04:00
Lex Berezhny 1368a6e074 wallet manager refactoring
test uses manager.send_amount_to_address instead of manually creating tx
2018-08-24 11:37:25 -04:00
Lex Berezhny 80990b959c moved files around 2018-08-24 11:37:25 -04:00
Lex Berezhny 053b6d1c80 removing lbryum from requirements 2018-08-24 11:37:25 -04:00
Lex Berezhny 692d8e8e1a new wallet becoming drop-in replacement for old lbryum wallet 2018-08-24 11:37:25 -04:00
Lex Berezhny 5e71dcbaf0 wip: implementation is now generic and supports multiple currencies 2018-08-24 11:37:25 -04:00
Lex Berezhny 83958604d5 finally generating fully signed tx submittable to lbrycrd 2018-08-24 11:37:25 -04:00
Lex Berezhny dcd8a6bb0e branch and bound based coin selection with random draw fallback 2018-08-24 11:37:25 -04:00
Lex Berezhny 7161a0edb9 new bitcoin script parser 2018-08-24 11:37:25 -04:00
Lex Berezhny 0fd160e6e6 a lot more stuff 2018-08-24 11:37:25 -04:00
Lex Berezhny ca8b2dd83e wip: initial import of twisted based wallet 2018-08-24 11:37:25 -04:00
Jack Robison 79d1da5ff8 Bump version 0.21.2rc3 --> 0.21.2
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-23 23:43:35 -04:00
Jack Robison d73252b11a Bump version 0.21.2rc2 --> 0.21.2rc3
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-23 22:40:16 -04:00
Jack Robison 1f5c56cac3 Merge branch 'fix-mirror-lc' 2018-08-23 22:40:04 -04:00
Jack Robison 18db2aa970 changelog 2018-08-23 22:38:55 -04:00
Jack Robison a7a53f9381 fix mirror downloader looping call 2018-08-23 22:36:30 -04:00
Jack Robison f4b2a05fff fix a typo 2018-08-23 18:35:47 -04:00
Jack Robison cdd2b6bfc1 Bump version 0.21.2rc1 --> 0.21.2rc2
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-23 11:57:49 -04:00
Jack Robison 26bd27cd32 Bump version 0.21.1 --> 0.21.2rc1
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-23 11:45:07 -04:00
Jack Robison 613feb0808 Merge branch 'fix-mirror-download-of-partial-hosted' 2018-08-23 11:44:48 -04:00
Jack Robison d4849443e5 changelog 2018-08-23 11:22:03 -04:00
Jack Robison db70f7e98a requirements
-bump txupnp requirement
-set Faker requirement
2018-08-23 11:21:53 -04:00
Jack Robison 37ddaeca12 fix ping queue bug 2018-08-23 11:18:28 -04:00
Jack Robison 2c546b2cde retry mirror downloads 2018-08-23 11:18:28 -04:00
Jack Robison c48d195951 docs 2018-08-22 00:20:30 -04:00
Alex Grintsvayg 149fa625e0 back to reflector 1 2018-08-15 16:11:34 -04:00
Jack Robison 16cda60f31 fix test 2018-08-13 17:16:48 -04:00
Jack Robison 0d54f6215f Bump version 0.21.1rc2 --> 0.21.1
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-13 16:20:16 -04:00
Jack Robison a9774a293f docs 2018-08-13 16:18:31 -04:00
Jack Robison ee8cf08c0e Bump version 0.21.1rc1 --> 0.21.1rc2
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-13 14:04:39 -04:00
Jack Robison ea481e6c5a fix download_progress in blockchain_headers status
-use a 0-100 range for percentage regardless of how much we have to download (if we're resuming or starting fresh)
-prevent negative blocks or blocks_behind counts
2018-08-13 14:03:15 -04:00
Jack Robison 36df9e8283 Bump version 0.21.0 --> 0.21.1rc1
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-12 22:16:00 -04:00
Jack Robison 0467b65e69 docs 2018-08-12 22:15:43 -04:00
Jack Robison 68f1dda7b0 initialize headers download progress 2018-08-12 22:05:07 -04:00
Jack Robison 5b7f7c87bf fix stop_lbry_files 2018-08-12 22:04:55 -04:00
Jack Robison 153022a1a7 add is_locked to status 2018-08-12 22:04:48 -04:00
Lex Berezhny 1098ca0494 + convert float to decimal for wallet_send 2018-08-09 19:14:56 -04:00
Jack Robison d4a7dbf805 Bump version 0.21.0rc13 --> 0.21.0
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 13:44:28 -04:00
Jack Robison 56ee815eec Bump version 0.21.0rc12 --> 0.21.0rc13
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 13:20:15 -04:00
Jack Robison fe43764bcb fix announcement of blobs downloaded from a http mirror 2018-08-09 13:19:59 -04:00
Jack Robison 93be8b9e31 more changelog cleanup 2018-08-09 12:18:25 -04:00
Jack Robison 2448d51ed5 Bump version 0.21.0rc11 --> 0.21.0rc12
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 12:07:35 -04:00
Jack Robison 2804fa4448 fix disabling the hash announcer 2018-08-09 12:04:14 -04:00
Jack Robison 90a119bb2c changelog 2018-08-09 11:56:28 -04:00
Jack Robison 69ae60b4bb Bump version 0.21.0rc10 --> 0.21.0rc11
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 11:29:49 -04:00
Jack Robison dca4af942f also try/except around upnp.discover
-catches edge case where miniupnpc fallback is used and a device is found but selectigd fails
2018-08-09 11:28:07 -04:00
Jack Robison 2b7e135839 fix error in blob_delete 2018-08-09 11:26:57 -04:00
Jack Robison d565aa0b3f Bump version 0.21.0rc9 --> 0.21.0rc10
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 09:24:39 -04:00
Jack Robison 8b53140581 set downloading_headers to False after the download finishes or fails 2018-08-09 09:24:04 -04:00
Jack Robison 917bae568d fix disabling blob announcement by setting concurrent announcers to 0 2018-08-09 09:22:55 -04:00
Jack Robison efb6af4d51 Bump version 0.21.0rc8 --> 0.21.0rc9
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 09:01:57 -04:00
Jack Robison 2361d8bdc1 clean up changelog 2018-08-09 09:01:33 -04:00
Jack Robison 5fe8022b15 don't attempt to announce blobs until we we have contacts in routing 2018-08-09 09:00:36 -04:00
Jack Robison 7931397595 Bump version 0.21.0rc7 --> 0.21.0rc8
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-09 08:44:23 -04:00
Jack Robison 0501f52805 fix upnp edge case
-fixes https://github.com/lbryio/lbry/issues/1352
2018-08-09 08:42:51 -04:00
Jack Robison 108d2a033b fix attribute error when reflector component is disabled 2018-08-09 08:32:43 -04:00
Jack Robison 17a9cfd65c Bump version 0.21.0rc6 --> 0.21.0rc7
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-08 14:12:01 -04:00
Jack Robison 7069ffa37d fix pip 2018-08-08 14:09:33 -04:00
Jack Robison 8eb740cf67 Bump version 0.21.0rc5 --> 0.21.0rc6
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-08 13:58:17 -04:00
Jack Robison 7803eb2e5a bump txupnp 2018-08-08 13:57:41 -04:00
Jack Robison b73e50e64b Bump version 0.21.0rc4 --> 0.21.0rc5
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-07 19:27:50 -04:00
Jack Robison c73148a1fb Merge remote-tracking branch 'origin/fix_1346' 2018-08-07 19:27:28 -04:00
Jack Robison 65a02190dc Merge remote-tracking branch 'origin/fix_1344' 2018-08-07 19:26:15 -04:00
Jack Robison 74b49fe69f Merge branch 'upnp-fixes' 2018-08-07 19:25:06 -04:00
Jack Robison 26e32472d5 fix upnp and peer port setup
-bump txupnp requirement
2018-08-07 19:04:54 -04:00
Victor Shyba 20d8b2963c adds a CHANGELOG 2018-08-07 14:01:46 -03:00
Victor Shyba d21313e590 allow mirrors if conf is set, but clear it for tests 2018-08-07 13:55:45 -03:00
Victor Shyba 81e9c9bf74 adds a CHANGELOG 2018-08-07 13:53:15 -03:00
Victor Shyba 382226ff9a tells that an address is wrong earlier on publishing 2018-08-07 13:52:01 -03:00
Jack Robison 9511040d5a fix upnp and peer port setup 2018-08-05 13:12:39 -04:00
Jack Robison 049936f4a6 docs 2018-08-03 13:32:57 -04:00
Jack Robison 15161dd429 fix docstring 2018-08-03 13:31:51 -04:00
Jack Robison efc5982860 Bump version 0.21.0rc3 --> 0.21.0rc4
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-08-03 13:22:06 -04:00
Jack Robison acb95fdcdf Merge remote-tracking branch 'origin/greedy_search' 2018-08-03 13:21:29 -04:00
Jack Robison f1b19f10cc Merge branch 'master' into greedy_search 2018-08-03 13:08:31 -04:00
Jack Robison 6eaf1ac2a0 Merge branch 'update-cryptography' 2018-08-03 12:52:47 -04:00
Jack Robison 01ee1e1468 Merge branch 'component-status' 2018-08-03 12:51:49 -04:00
Jack Robison 83dc295b7b update cryptography requirement 2018-08-03 12:38:28 -04:00
Jack Robison 4a28c620a4 changelog 2018-08-03 12:35:13 -04:00
Jack Robison e50ac69358 tests and pylint 2018-08-03 12:35:10 -04:00
Jack Robison aef561d78b refactor status results 2018-08-03 12:33:31 -04:00
Jack Robison 4ef4a4816f update component tests 2018-08-03 11:37:10 -04:00
Jack Robison 8533bfa7c5 refactor test_misc, remove Session
-use only the one process
2018-08-03 11:37:10 -04:00
Jack Robison 6389e961b9 update (and refactor) tests to not use Session 2018-08-03 11:37:10 -04:00
Jack Robison 6359fc608e set download_mirrors upon making the downloader instead in the file manager globally 2018-08-03 11:37:10 -04:00
Jack Robison 314400a1bd move looping call to check announcement status to SQLiteStorage 2018-08-03 11:37:10 -04:00
Jack Robison a3de065c93 use txupnp instead of miniupnpc 2018-08-03 11:37:07 -04:00
Jack Robison 99207b7221 delete Session.py
split Session into Components
2018-07-30 18:00:33 -04:00
Jack Robison e3c3fafa1e split headers download into own component
-add component statuses
2018-07-30 18:00:33 -04:00
Jack Robison 5b3103e41b Merge pull request #1315 from lbryio/mirroring_refactor
refactor mirroring
2018-07-30 16:00:02 -04:00
Victor Shyba 9742da348b do not consume errors on DeferredList 2018-07-30 13:53:58 -03:00
Victor Shyba 94a31922f7 fix unnecessary logging tag 2018-07-30 11:22:31 -03:00
Victor Shyba 1937eb17c4 adds a test for stopping 2018-07-30 11:22:31 -03:00
Victor Shyba f291ea515a add test for invalid content 2018-07-30 11:22:31 -03:00
Victor Shyba 56522f39c0 adds a CHANGELOG 2018-07-30 11:22:31 -03:00
Victor Shyba ff4aba9423 refactor mirroring 2018-07-30 11:22:31 -03:00
hackrush 9ab256df30 fixed improper error handling when data is not valid JSON (#1326) 2018-07-30 09:38:36 -04:00
Lex Berezhny 88b8ca8a49 Revert "negate wallet lock check in components"
This reverts commit 32507d7
2018-07-25 14:59:10 -04:00
Jack Robison 09f637496c Bump version 0.21.0rc2 --> 0.21.0rc3
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-25 12:25:24 -04:00
Jack Robison 826d269e0e Merge branch 'blobs_mirroring' 2018-07-25 12:24:37 -04:00
Victor Shyba 5163baf9c1 improve changelog on changes section 2018-07-25 13:09:25 -03:00
Victor Shyba ab27203100 improve exception logging and add a docstring on the interaction between downloaders 2018-07-25 13:01:13 -03:00
Victor Shyba 88c2051605 set download_mirror conf from components change 2018-07-25 12:23:15 -03:00
Victor Shyba f510c2a433 improve logging from review 2018-07-25 12:15:52 -03:00
Victor Shyba 66982c86a9 adds changelog 2018-07-25 12:15:52 -03:00
Victor Shyba ec140d5d8a changes from code review 2018-07-25 12:15:52 -03:00
Victor Shyba ad96b006f9 adds http download support for blobs mirroring 2018-07-25 12:15:52 -03:00
Lex Berezhny 32507d74f7 negate wallet lock check in components 2018-07-25 01:28:19 -04:00
Jack Robison 2a00d00e44 docs 2018-07-24 21:56:15 -04:00
Jack Robison 40c5f6e3ab Bump version 0.21.0rc1 --> 0.21.0rc2
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-24 21:41:58 -04:00
Jack Robison ae67918c8a Merge branch 'better-peer-ping' 2018-07-24 21:34:15 -04:00
Jack Robison 96e73f3476 changelog 2018-07-24 21:31:51 -04:00
Jack Robison c1b05402ef add address and port arguments to peer_ping
allows directly pinging the peer without first doing an iterative find
2018-07-24 21:30:54 -04:00
Jack Robison 1dd1bad9c2 Bump version 0.20.4 --> 0.21.0rc1
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-24 21:21:52 -04:00
Jack Robison 9b873d2f48 Merge branch 'daemon-components' 2018-07-24 21:17:51 -04:00
Jack Robison 5984ae7ce9 fix 2018-07-24 21:10:53 -04:00
hackrush 903cd86cdd changelog 2018-07-24 20:02:08 -04:00
Jack Robison 5d9a187659 refactor required conditions and @requires decorator 2018-07-24 20:01:38 -04:00
Jack Robison a9c94ca22d move setup and _shutdown to AuthJSONRPCServer 2018-07-24 20:01:38 -04:00
Jack Robison a285db1b08 pylint 2018-07-24 20:01:38 -04:00
Jack Robison 13bf8125e3 status doc, better determination of is_first_run
-add skipped_components to status response
-re-add run_reflector_server to settings
2018-07-24 20:01:38 -04:00
Jack Robison b06dcf0a0d cancel starting components if the reactor is stopped before startup has finished
-don't block starting the dht component on having found enough peers, only on setting up the protocol
2018-07-24 20:01:38 -04:00
Jack Robison a800f6ddf0 update status command 2018-07-24 20:01:38 -04:00
Jack Robison a89306b6bf reorganize daemon startup
-fix loggly not using the share usage setting

-delete more
2018-07-24 20:01:38 -04:00
Jack Robison c3120e93cf delete unneeded daemon attributes 2018-07-24 20:01:38 -04:00
Jack Robison defe9506bb set daemon attribute for each component as they start 2018-07-24 20:01:38 -04:00
Jack Robison 55d3bb0ec3 remove auto_renew 2018-07-24 20:01:38 -04:00
hackrush 005a8b3008 allow utf-8 characters in Daemon.py
-amit really wants the shrug emoji ¯\_(ツ)_/¯
2018-07-24 20:01:38 -04:00
hackrush a741fdfc44 update daemon unit tests 2018-07-24 20:01:38 -04:00
hackrush 944b94aae8 update logging tests 2018-07-24 20:01:38 -04:00
hackrush 37eaf634c4 update functional tests 2018-07-24 20:01:38 -04:00
hackrush edcb06a415 update mocks, add test_Component_Manager 2018-07-24 20:01:38 -04:00
hackrush 62b50dc0ae move custom logger to lbrynet directory
-import on module level __init__
2018-07-24 20:01:38 -04:00
hackrush e7c57dcabc add components_to_skip setting 2018-07-24 20:01:38 -04:00
hackrush 094d9c6497 update lbrynet-cli 2018-07-24 20:01:38 -04:00
hackrush 3dc5a9de7b whitespace 2018-07-24 20:01:38 -04:00
hackrush 4e29041291 update status command 2018-07-24 20:01:38 -04:00
hackrush 6b6a29fdb7 update settings_set 2018-07-24 20:01:38 -04:00
hackrush 2d61ba629f add requires decorator 2018-07-24 20:01:38 -04:00
hackrush 5a2075019c update Wallet.check_locked 2018-07-24 20:01:38 -04:00
hackrush 944200ca8c add all the daemon components 2018-07-24 20:01:38 -04:00
hackrush 40d8e96811 remove functions and attributes in Session and Daemon that are now part of components
-rename attributes in daemon to use components
2018-07-24 20:01:38 -04:00
Jack Robison 7e8ca842a2 change ComponentManager to use instance methods rather than class methods
-add get_component method to ComponentManager

-add override_components kwargs to ComponentManager

-add skip_components to ComponentManager

-change component_manager attribute to exist on the AuthJSONRPCServer instance instead of the class
2018-07-24 20:01:38 -04:00
Jack Robison b808d08eb3 support callbacks to component setups 2018-07-24 20:01:38 -04:00
Jack Robison eb11da9b19 convert directory and SQLiteStorage setup to be a Component 2018-07-24 20:01:38 -04:00
Jack Robison 68b31a09b4 add daemon Component and ComponentManager classes 2018-07-24 20:01:38 -04:00
Lex Berezhny cab8416596 api doc generator (#1290)
script to generate docs/api.json
2018-07-24 19:36:09 -04:00
Electron - Mark Firth 94f4e6714a A More Informative Error Message (#1309)
Additional information added to the balance error message when editing a claim.
2018-07-20 15:41:24 -04:00
Jack Robison 2ddd6b051d Bump version 0.20.4rc4 --> 0.20.4
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-18 15:36:21 -04:00
Jack Robison c506fc5c75 lower peer finder timeout message to debug from warning 2018-07-18 15:32:45 -04:00
Jack Robison d306ece0d4 Bump version 0.20.4rc3 --> 0.20.4rc4
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-17 17:38:27 -04:00
Jack Robison d29f103910 pylint 2018-07-17 17:38:19 -04:00
Jack Robison 2386a3a79b Bump version 0.20.4rc2 --> 0.20.4rc3
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-17 17:14:07 -04:00
Jack Robison ec51333b56 allow node udp interface to be configured 2018-07-17 17:13:33 -04:00
Jack Robison 75829812f3 Merge pull request #1305 from lbryio/fix_score_call
fix where it called the wrong score method
2018-07-17 17:02:35 -04:00
Victor Shyba 4d720370b1 fix where it called the wrong score method 2018-07-17 17:48:25 -03:00
Jack Robison 7d7dd665aa Bump version 0.20.4rc1 --> 0.20.4rc2
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-17 13:31:37 -04:00
Jack Robison b6bee47481 changelog 2018-07-17 13:31:20 -04:00
Jack Robison 0ec8739d5a Merge branch 'fix_on_high_cpu_usage_on_incomplete_streams' 2018-07-17 13:24:06 -04:00
Victor Shyba 7d09d1402f changelog 2018-07-17 13:22:08 -04:00
Victor Shyba 5492ab9081 stop requesting peers which doesnt have a blob 2018-07-17 13:22:08 -04:00
Jack Robison 7f6bbc0648 Merge branch 'dht_shortlist_without_ignored' 2018-07-17 13:01:17 -04:00
Victor Shyba b1d4072c0b changelog 2018-07-17 12:57:57 -04:00
Victor Shyba 562d23df11 remove nodes that went ignored during iteration 2018-07-16 17:22:46 -03:00
Jack Robison 2c13c04f86 Bump version 0.20.3 --> 0.20.4rc1
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-13 12:02:50 -04:00
Jack Robison 7365bfad9f Merge branch 'dht_failure_pruning' 2018-07-13 12:02:30 -04:00
Victor Shyba 1af1cf212c changelog 2018-07-13 11:39:44 -04:00
Victor Shyba 5c7d279104 prune failures during is_ignored calls 2018-07-12 18:24:59 -03:00
Victor Shyba cfe8e17223 filter out the bad node in results instead of punishing the node we trusted to contact 2018-07-12 18:24:18 -03:00
Thomas Zarebczan 91d4bf3cd9 Update ISSUE_TEMPLATE.md 2018-07-12 16:15:19 -04:00
Lex Berezhny df9642c325 removed slack notification from .travis.yml 2018-07-10 00:34:59 -04:00
Victor Shyba af98b0eb4d adds a changelog 2018-07-05 22:03:52 -03:00
Victor Shyba c589e89bf0 greedy search with exclude filtering 2018-07-05 22:03:52 -03:00
Lex Berezhny 284264962d reflector test fix 2018-07-05 19:51:15 -04:00
Lex Berezhny 6952c2c07e Revert "refactor lbrynet-daemon into modular components" (#1286)
* Revert "fix reflector test"

This reverts commit 6a15b51ac3.

* Revert "refactor lbrynet-daemon into modular components (#1164)"

This reverts commit 75a6ff269e.
2018-07-05 19:05:48 -04:00
Jack Robison 6a15b51ac3 fix reflector test 2018-07-05 15:40:13 -04:00
Jack Robison 75a6ff269e refactor lbrynet-daemon into modular components (#1164)
* add daemon Component and ComponentManager classes

* convert directory and SQLiteStorage setup to be a Component

* support callbacks to component setups

* Fixed typo in ComponentManager

* convert wallet to be Component

* Use storage from session.

* Remove create_session internal function and PEP8

* Starting to convert session to its own component. Removed ref to `self.storage` from Daemon.py

* Making DHT component(broken)

* Refactored classes to reduce redundancy in getting config setting

* DHT is now it's own component

* Fixed `test_streamify` test

* Fixed regression caused by removing `peer_manager` from session

* refactor ComponentManager and Component to use instance instead of class methods

* Hash announcer, file manager, stream identifier components

* Query Handler and server components

* Reflector Component

* Fixed test_streamify(well Jack did, but ¯\_(ツ)_/¯)

* All tests now passing

* Pylint fixes

* Oops(That's all you're gonna get :-P)

* Making decorators(WIP, commit so that I don't lose work)

* Decorator made and decorating of functions done(some other changes)

* import fixes and removed temporary test function

* Fixed new broken tests from daemon refactor

* Sanitization of modules

* Reworded errors

* wallet unlock condition checks, fixed breaking changes

* Rebased on amster and other crazy stuff

* Started writing tests

* Tests for component manager

* Fix Daemon Tests

* Fixed passing mutable args in init

* Using constants instead of strings. Added CHANGELOG.md

* Now components can be skipped by setting relevant config in file.

* P-Y-L-I-N-T #angry_emoji
2018-07-05 15:21:52 -04:00
ocornoc 148cc96025 Changed "cliam" to "claim" and added accurate info for nsfw. (#1285)
* Changed "cliam" to "claim" and added accurate info for nsfw.
* Update CHANGELOG.md
2018-07-05 12:00:43 -04:00
hackrush 14efa94382 Started to standardize the README.md file.
Closes #1252
2018-07-05 11:09:57 -04:00
Jack Robison da7e7503b9 Bump version 0.20.3rc2 --> 0.20.3
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-03 14:32:38 -04:00
Jack Robison 67bedfafcb changelog 2018-07-03 14:29:46 -04:00
Jack Robison 5adcc86f40 Bump version 0.20.3rc1 --> 0.20.3rc2
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-03 13:10:35 -04:00
Jack Robison a9971677f8 Merge branch 'stop_conditions_find_value' 2018-07-03 13:09:58 -04:00
Victor Shyba 8b6902fb00 adds a changelog 2018-07-03 13:49:35 -03:00
Victor Shyba 1cc7ce69ab set the peer search timeout as configured and raise it to 30s 2018-07-03 13:47:00 -03:00
Victor Shyba 7e0784d8e0 log a warning when peer search times out for real 2018-07-03 12:20:33 -03:00
Victor Shyba 023846703b adds a changelog 2018-07-03 11:53:21 -03:00
Victor Shyba 56613a71d5 let find value run until it finds a value 2018-07-03 11:51:39 -03:00
Jack Robison 2b0d4c1603 Bump version 0.20.2 --> 0.20.3rc1
Signed-off-by: Jack Robison <jackrobison@lbry.io>
2018-07-02 17:24:30 -04:00
Jack Robison a3e17f7916 Merge branch 'fix-find-close-nodes' 2018-07-02 17:24:09 -04:00
Jack Robison 929a0cc5af exclude self and querying node from closest contacts 2018-07-02 14:54:03 -04:00
Jack Robison b6289d101d fix improper sorting when getting the closest peers to a hash 2018-07-02 14:54:03 -04:00
Jack Robison 396542dc26 Merge branch 'self-store' 2018-07-02 14:52:38 -04:00
Jack Robison 7fe92d2df0 load all finished blobs into the node datastore 2018-07-02 14:31:09 -04:00
Jack Robison 03769b94b8 accept invalid store tokens for the first expire-time after startup
this is to better handle nodes storing to us after we've restarted our node
2018-07-02 13:26:38 -04:00
Jack Robison 07292363f9 Merge branch 'faster-channel-id-file-startup' 2018-07-02 10:49:13 -04:00
Jack Robison c2dc872a54 fix file manager startup locking up when there are many files in a channel 2018-06-29 21:45:30 -04:00
Jack Robison 8a0d0b44dd Merge branch 'fix-blob-list-keyerror' 2018-06-29 11:49:06 -04:00
Lex Berezhny accf161355 Merge pull request #1274 from lbryio/req-update
Removed jsonrpclib from requirements
2018-06-26 10:06:47 -05:00
hackrush a5411c7b29 Removed jsonrpclib from requirements 2018-06-26 10:31:04 -04:00
Jack Robison e0ceaba655 fix blob_list raising an error when blobs in a stream haven't yet been created 2018-06-25 16:47:23 -04:00
Jack Robison a1f3fc9f3c Merge pull request #1269 from lbryio/fix_stop_nonetype
Fix stop raising `NoneType' object has no attribute 'finished_deferred'
2018-06-25 16:42:07 -04:00
Victor Shyba 238a07e8e2 adds a changelog 2018-06-23 18:26:45 -03:00
Victor Shyba c5f4f58221 collect finished deferred before cancel call 2018-06-23 18:23:25 -03:00
342 changed files with 23606 additions and 16133 deletions

No files matched your search

-48
View File
@@ -1,48 +0,0 @@
version: 1.0.{build}
branches:
only:
- master
environment:
GH_TOKEN:
secure: LiI5jyuHUw6XbH4kC3gP1HX4P/v4rwD/gCNtaFhQu2AvJz1/1wALkp5ECnIxRySN
key_pass:
secure: u6DydPcdrUJlxGL9uc7yQRYG8+5rY6aAEE9nfCSzFyNzZlX9NniOp8Uh5ZKQqX7bGEngLI6ipbLfiJvn0XFnhbn2iTkOuMqOXVJVOehvwlQ=
pfx_key:
secure: 1mwqyRy7hDqDjDK+TIAoaXyXzpNgwruFNA6TPkinUcVM7A+NLD33RQLnfnwVy+R5ovD2pUfhQ6+N0Fqebv6tZh436LIEsock+6IOdpgFwrg=
AWS_ACCESS_KEY_ID:
secure: iVGwoJ7ogspjSmuqr+haVPLglSgQsp6tUZx6mIlKH7Q=
AWS_SECRET_ACCESS_KEY:
secure: zKaqdZGPl0exDL5YhJkb33prSemC9Rzg9S7Lw2wFy1WnJ6ffgl6mQH7jqJDUTqsY
#notifications:
# - provider: Slack
# incoming_webhook:
# secure: LuxwG5OZnnA//gmSXzCKu8/FRqYjCgGfVFqajSsGHeQ1HQNp7rYNhQpsO8/3PK63xKJj3wzt86DJekf9q9Q5OcHa9AHXUQbEveX0psd7elw=
clone_folder: c:\projects\lbry
test_script:
- cd C:\projects\lbry\
- pip install cython
- pip install mock pylint unqlite Faker
- pip install .
- pylint lbrynet
# disable tests for now so that appveyor can build the app
- set PYTHONPATH=.
- trial lbrynet.tests
# TODO: integration tests do not work
#- python -m unittest discover tests/integration
#- rvm use 2.3.1 && gem install danger --version '~> 4.0' && danger
build_script:
- cd C:\projects\lbry\build\
- ps: .\build.ps1
artifacts:
- path: build\dist\*.zip
name: lbrynet-daemon
+7
View File
@@ -0,0 +1,7 @@
.git
.tox
__pycache__
dist
lbrynet.egg-info
docs
tests
+22 -1
View File
@@ -4,7 +4,7 @@ Thanks for reporting an issue to LBRY and helping us improve!
To make it possible for us to help you, please fill out below information carefully.
Before reporting any issues, please make sure that you're using the latest version.
- App: https://github.com/lbryio/lbry-app/releases
- App: https://github.com/lbryio/lbry-desktop/releases
- Daemon: https://github.com/lbryio/lbry/releases
We are also available on Discord at https://chat.lbry.io
@@ -51,3 +51,24 @@ Tell us what happens instead
## Screenshots
<!-- If a screenshot would help explain the bug, please include one or two here -->
## Internal Use
### Acceptance Criteria
1.
2.
3.
### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed
+38
View File
@@ -0,0 +1,38 @@
## PR Checklist
Please check all that apply to this PR using "x":
- [ ] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
- [ ] I have checked that this PR does not introduce a breaking change
- [ ] This PR introduces breaking changes and I have provided a detailed explanation below
## PR Type
What kind of change does this PR introduce?
Why is this change necessary?
<!-- Please check all that apply to this PR using "x". -->
- [ ] Bugfix
- [ ] Feature
- [ ] Breaking changes (bugfix or feature that introduces breaking changes)
- [ ] Code style update (formatting)
- [ ] Refactoring (no functional changes)
- [ ] Documentation changes
- [ ] Other - Please describe:
## Fixes
Issue Number: N/A
## What is the current behavior?
## What is the new behavior?
## Other information
<!-- If this PR contains a breaking change, please describe the impact and solution strategy for existing applications below. -->
+7 -23
View File
@@ -1,25 +1,9 @@
*.pyc
*.egg
*.so
*.xml
*.iml
*.log
*.pem
*.decTest
*.prof
.#*
/build
/dist
/.tox
/.idea
/.coverage
/build/build
/build/dist
/bulid/requirements_base.txt
/lbrynet.egg-info
/docs_build
/lbry-venv
.idea/
.coverage
.DS_Store
# temporary files from the twisted.trial test runner
lbrynet.egg-info
__pycache__
_trial_temp/
+9 -5
View File
@@ -9,7 +9,7 @@
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=CVS
ignore=CVS,schema
# Add files or directories matching the regex patterns to the
# blacklist. The regex matches against base names, not paths.
@@ -62,7 +62,7 @@ confidence=
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then reenable specific checks. For example, if
# disable everything first and then re-enable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
@@ -121,7 +121,11 @@ disable=
unidiomatic-typecheck,
global-at-module-level,
inconsistent-return-statements,
keyword-arg-before-vararg
keyword-arg-before-vararg,
assignment-from-no-return,
useless-return,
assignment-from-none,
stop-iteration-return
[REPORTS]
@@ -386,7 +390,7 @@ int-import-graph=
[DESIGN]
# Maximum number of arguments for function / method
max-args=5
max-args=10
# Argument names that match this expression will be ignored. Default to name
# with leading underscore
@@ -405,7 +409,7 @@ max-branches=12
max-statements=50
# Maximum number of parents for a class (see R0901).
max-parents=7
max-parents=8
# Maximum number of attributes for a class (see R0902).
max-attributes=7
+122 -38
View File
@@ -1,46 +1,130 @@
os: linux
dist: trusty
sudo: required
dist: xenial
language: python
python: 2.7
python: "3.7"
branches:
except:
- gh-pages
jobs:
include:
notifications:
slack:
secure: "Am13HPtpgCMljh0MDVuoFHvQXB8yhf4Kvf/qAeSp5N0vsHGL70CSF9Ahccw8dVPE6mbuak1OGtSUb6/UaErLHkpz3ztaRLkDa9x7CmBB3Kynnh8oO2VbB7b/2ROULqkhF4VZmAnNfwrQrbC3gs8Sybp261Nyc7y4ww15xDYBrk2fyq4ds2DCaJdRxfJUJFonrZ6KXr3fVaXosO6cjuyS8eRodcmrqsT4cCtinjNTD1hGWoH107E4ObSmpVelxQO193KhNJMRiLlEcVkvYUOqIWBtwdGHbNE/6Yeuq1TXgKJ0KeJWAmW3wTfUYNngGXNAsyCnrhul5TKNevNzfIAQZHvRsczYiWPJV6LtohHT0CcUiCXJtvEPOyahEBfwK3etY/xxFqny7N9OEmpdW2sgsEPNPX2LJynJti2rQA9SuAD1ogR3ZpDy/NXoaAZf8PTdPcuNUMULV9PGG7tVrLBecO/W1qO6hdFxwlLdgqGLxAENZgGp++v/DhPk/WvtmHj7iTbRq0nxaTWyX5uKOn2ADH+k/yfutjv6BsQU9xNyPeZEEtuEpc6X6waiYn/8G9vl9PecvKC5H0MgsZ6asAxmg7mZ3VSMFG7mo8ENeOhSZ0Oz6ZTBILL3wFccZA9uJIq7NWmqC9dRiGiuKXBB62No7sINoHg3114e2xYa9qvNmGg="
- stage: code quality
name: "pylint lbrynet"
install:
- pip install astroid==2.0.4 aiohttp==3.4.4
# newer astroid and aiohttp fails in pylint so we pre-install older version
- pip install pylint
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .
script: pylint lbrynet
- &tests
stage: test
name: "Unit Tests"
install:
- pip install coverage
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .[test]
script:
- HOME=/tmp coverage run -p --source=lbrynet -m twisted.trial --reactor=asyncio tests.unit.core tests.unit.cryptstream tests.unit.database tests.unit.dht tests.unit.lbryfilemanager tests.unit.lbrynet_daemon tests.unit.schema tests.unit.wallet tests.unit.components tests.unit.test_conf
- HOME=/tmp coverage run -p --source=lbrynet -m twisted.trial --reactor=asyncio tests.unit.test_cli
#- HOME=/tmp coverage run -p --source=lbrynet -m twisted.trial --reactor=asyncio tests.unit.analytics
after_success:
- coverage combine
- bash <(curl -s https://codecov.io/bash)
- <<: *tests
name: "DHT Tests"
script: HOME=/tmp coverage run --source=lbrynet -m twisted.trial --reactor=asyncio tests.functional
- name: "Integration Tests"
install:
- pip install tox-travis coverage
- pushd .. && git clone https://github.com/lbryio/torba.git && popd
script: tox
after_success:
- coverage combine tests/
- bash <(curl -s https://codecov.io/bash)
- stage: build
name: "Windows"
language: generic
services:
- docker
install:
- docker pull lbry/pyinstaller34_32bits:py371
script:
- python scripts/set_build.py
- docker run -v "$(pwd):/src/lbry" lbry/pyinstaller34_32bits:py371 lbry/scripts/wine_build.sh
- sudo zip -j dist/lbrynet-windows.zip dist/lbrynet.exe
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-windows.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-windows.zip
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- &build
name: "Linux"
env: OS=linux
install:
- pip3 install pyinstaller
- pip3 install git+https://github.com/lbryio/torba.git
- python scripts/set_build.py
- pip3 install -e .
script:
- pyinstaller -F -n lbrynet lbrynet/extras/cli.py
- chmod +x dist/lbrynet
- zip -j dist/lbrynet-${OS}.zip dist/lbrynet
- ./dist/lbrynet --version
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-${OS}.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-${OS}.zip
# artifact uploader thinks lbrynet is a directory, https://github.com/travis-ci/artifacts/issues/78
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- <<: *build
name: "Mac"
os: osx
osx_image: xcode8.3
language: generic
env: OS=mac
cache: false
before_install:
- brew upgrade python || true
- brew upgrade python || true
install:
- python3 --version
- pip3 --version
- pip3 install pyinstaller
- git clone https://github.com/lbryio/torba.git --depth 1
- sed -i -e "s/'plyvel',//" torba/setup.py
- cd torba && pip3 install -e . && cd ..
- python3 scripts/set_build.py
- pip3 install -e .
cache:
directories:
- $HOME/.cache/pip
- $HOME/Library/Caches/pip
- $TRAVIS_BUILD_DIR/cache/wheel
addons:
#srcclr:
# debug: true
apt:
packages:
- libgmp3-dev
- build-essential
- git
- libssl-dev
- libffi-dev
before_install:
- virtualenv venv
- source venv/bin/activate
install:
- pip install -U pip==9.0.3
- pip install -r requirements.txt
- pip install -r requirements_testing.txt
- pip install .
script:
- pip install mock pylint
- pylint lbrynet
- PYTHONPATH=. trial lbrynet.tests
- rvm install ruby-2.3.1
- rvm use 2.3.1 && gem install danger --version '~> 4.0' && danger
- $TRAVIS_BUILD_DIR/.tox
+202 -20
View File
@@ -4,33 +4,215 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/) with
regard to the json-rpc api. As we're currently pre-1.0 release, we
can and probably will change functionality and break backwards compatability
can and probably will change functionality and break backwards compatibility
at anytime.
## [Unreleased]
## [0.30.1] - 2018-12-09
### API
* add `aioupnp_version` to `upnp` field in the response to `status`
* fixed uncaught claim signature errors
* `--blocking` support for `abandon` command (https://github.com/lbryio/lbry/pull/1618)
* don't create SSL keys if SSL is disabled (https://github.com/lbryio/lbry/pull/1644)
* added `account_send` command (https://github.com/lbryio/lbry/pull/1654)
* fixes for claim send to address (https://github.com/lbryio/lbry/pull/1651)
* better handle errors on resolve (https://github.com/lbryio/lbry/pull/1662)
* fix `transaction_show` (https://github.com/lbryio/lbry/pull/1689)
### Schema
* merged `lbryschema` repository into `lbrynet` (https://github.com/lbryio/lbry/pull/1444)
* updated `protobuf` to `3.2.0` (https://github.com/lbryio/lbry/issues/1602)
* use `types` repo for generating python protobufs (https://github.com/lbryio/lbry/pull/1605)
### Wallet
* fixed resolving names with invalid channel signatures
* fixed wallet sync issues
* fixed wallet locking/encryption issues
### DHT
* fixed retrying kademlia rpc calls (https://github.com/lbryio/lbry/pull/1495)
* fixed `KeyError` in `iterativeFindValue`, this is used by `peer_list` and when finding peers to download blobs from (https://github.com/lbryio/lbry/issues/1638)
* settings external ip address and ability to disable more DHT components (https://github.com/lbryio/lbry/pull/1604)
* improved routing table bucket initialization (https://github.com/lbryio/lbry/pull/1633)
### Database
* resolved too many variables in sqlite issue (https://github.com/lbryio/lbry/pull/1612)
### Misc
* switch to Python 3.7
* brought back icon for windows binary
* using custom list of mime types with more types (https://github.com/lbryio/lbry/pull/1700)
* add balance to startup report (https://github.com/lbryio/lbry/pull/1613)
* drop `six` dependency (https://github.com/lbryio/lbry/pull/1660)
* drop `mock` dependency (https://github.com/lbryio/lbry/pull/1661)
## [0.30.0] - 2018-10-22
This release is the most ambitious upgrade to lbrynet yet; including a brand
new wallet implementation, switch to Python 3 of the entire code base and
many changes to the API such as pagination and a more consistent output from
most commands.
### Security
*
*
* upgraded `cryptography` package.
* switch to `coincurve` for private / public key generation (addresses)
* support for connecting to daemon via SSL
* improved workflow for encrypt/decrypt of wallet
### API
* unified all command line executables into a single `lbrynet` executable.
* added `use_https` configuration setting (off by default), if true the api
server will generate a self signed ssl certificate when started and only
handle api requests on the https address (default of `https://localhost:5279/lbryapi`)
* deprecated `daemon_stop` command, use `stop` instead.
* deprecated `wallet_balance` command, use `account_balance` instead.
* deprecated `wallet_unlock` command, use `account_unlock` instead.
* deprecated `wallet_decrypt` command, use `account_decrypt` instead.
* deprecated `wallet_encrypt` command, use `account_encrypt` instead.
* deprecated `wallet_prefill_addresses` command, use `account_fund` instead.
* deprecated `wallet_list` command, use `address_list` instead.
* deprecated `wallet_is_address_mine` command, use `address_is_mine` instead.
* deprecated `wallet_public_key` command.
* deprecated `wallet_new_address` command.
* deprecated `wallet_unused_address` command, use `address_unused` instead.
* deprecated `claim_renew` command.
* added `account_list` command to list accounts including their balance.
* added `account_add` command to add a previously created account from seed or private key.
* added `account_create` command to generate a new account.
* added `account_remove` command to remove an account from wallet.
* added `account_set` command to change a setting on an account.
* added `account_balance` command to get just the account balance.
* added `account_unlock` command to unlock an encrypted account.
* added `account_lock` command to lock an encrypted account.
* added `account_encrypt` command to encrypt an account.
* added `account_decrypt` command to decrypt an account.
* added `account_fund` command to move funds between or within an account in various ways.
* added `account_max_address_gap` command to find large gaps of unused addresses.
* added `address_list` command to list addresses.
* added `address_is_mine` command to check if an address is one of your addresses.
* added `address_unused` command to get existing or generate a new unused address.
* added pagination support for `address_list`, `channel_list`, `claim_list_mine`,
`transaction_list` and `utxo_list`.
* added `upnp` field to `status` response
* removed `send_amount_to_address` command previously marked as deprecated
* removed `channel_list_mine` command previously marked as deprecated
* removed `get_availability` command previously marked as deprecated
* updated `txupnp` dependency to `aioupnp` (UPnP for asyncio)
### Wallet
* changed to a new wallet implementation: [torba](https://github.com/lbryio/torba).
* changed wallet file format to support multiple accounts in one wallet.
* moved transaction data from wallet file into an sqlite database.
* changed channel certificates to be keyed by txid:nout instead of claim_id which
makes it possible to recover old certificates.
### File Manager
* Extensive internal changes as a result of porting to Python 3.
### DHT
* Extensive internal changes as a result of porting to Python 3.
### P2P
* Extensive internal changes as a result of porting to Python 3.
### Reflector
* No major changes to reflector.
### Database
* No changes to existing storage module.
### Known Issues
* Channel certificate import/export is currently not working.
## [0.21.2] - 2018-08-23
### Fixed
*
*
* issue in dht ping queue where enqueued pings that aren't yet due wouldn't be rescheduled
* blob mirror downloader not finishing streams that were partially uploaded at the time of the download attempt (https://github.com/lbryio/lbry/issues/1376)
### Deprecated
*
*
### Changed
*
*
## [0.21.1] - 2018-08-13
### Fixed
* `download_progress` field in `blockchain_headers` section of `status` not initializing correctly when resuming a download (https://github.com/lbryio/lbry/issues/1355)
* `wallet_send` not accepting decimal amounts (https://github.com/lbryio/lbry/issues/1356 commit https://github.com/lbryio/lbry/commit/1098ca0494ece420c70fc57f69d6d388715a99b8)
### Added
*
*
* `is_locked` to `wallet` in `status` response (https://github.com/lbryio/lbry/issues/1354, commit https://github.com/lbryio/lbry/commit/153022a1a7122ab6d31d3db433dccbe469bbcb3c)
### Changed
* Bumped `lbryum` requirement to 3.2.4 [see changelog](https://github.com/lbryio/lbryum/blob/master/CHANGELOG.md#324---2018-08-13)
## [0.21.0] - 2018-08-09
### Fixed
* check `claim_address` and `change_address` earlier on publishing, to avoid hard to understand errors later in the process (pr https://github.com/lbryio/lbry/pull/1347)
* loggly error reporting not following `share_usage_data` (pr https://github.com/lbryio/lbry/pull/1328)
* improper error handling when data is not valid JSON (pr https://github.com/lbryio/lbry/pull/1326)
* blob mirroring being set in unrelated tests, making them fail (pr https://github.com/lbryio/lbry/pull/1348)
* http blob mirroring edge cases (pr https://github.com/lbryio/lbry/pull/1315)
* external ports in log messages not showing the correct external port from the upnp redirects (https://github.com/lbryio/lbry/issues/1338) (pr https://github.com/lbryio/lbry/pull/1349)
* miniupnpc fallback issues in txupnp (https://github.com/lbryio/lbry/issues/1341) (pr https://github.com/lbryio/lbry/pull/1349)
* upnp error when disabled on router and a non-gateway is found, such as chromecast (https://github.com/lbryio/lbry/issues/1352) (https://github.com/lbryio/lbry/commit/dca4af942fbe95547794213775f0a62cd04a393f)
### Deprecated
* automatic claim renew, this is no longer needed
### Changed
* api server class to use components, and for all JSONRPC API commands to be callable so long as the required components are available. (pr https://github.com/lbryio/lbry/pull/1294)
* return error messages when required conditions on components are not met for API calls (pr https://github.com/lbryio/lbry/pull/1328)
* `status` to no longer return a base58 encoded `lbry_id`, instead return this as the hex encoded `node_id` in a new `dht` field. (pr https://github.com/lbryio/lbry/pull/1328)
* `startup_status` field in the response to `status` to be a dict of component names to status booleans (pr https://github.com/lbryio/lbry/pull/1328)
* renamed the `blockchain_status` field in the response to `status` to `wallet` (pr https://github.com/lbryio/lbry/pull/1328)
* moved and renamed `wallet_is_encrypted` to `is_encrypted` in the `wallet` field in the response to `status` (pr https://github.com/lbryio/lbry/pull/1328)
* moved wallet, upnp and dht startup code from `Session` to `Components` (pr https://github.com/lbryio/lbry/pull/1328)
* attempt blob downloads from http mirror sources (by default) concurrently to p2p sources (pr https://github.com/lbryio/lbry/pull/1233)
* replace miniupnpc with [txupnp](https://github.com/lbryio/txupnp). Since txupnp is still under development, it will internally fall back to miniupnpc. (pr https://github.com/lbryio/lbry/pull/1328)
* simplified test_misc.py in the functional tests (pr https://github.com/lbryio/lbry/pull/1328)
* update `cryptography` requirement to 2.3 (pr https://github.com/lbryio/lbry/pull/1333)
### Added
* `skipped_components` list to the response from `status` (pr https://github.com/lbryio/lbry/pull/1328)
* component statuses (`blockchain_headers`, `dht`, `wallet`, `blob_manager` `hash_announcer`, and `file_manager`) to the response to `status` (pr https://github.com/lbryio/lbry/pull/1328)
* `skipped_components` config setting, accepts a list of names of components to not run (pr https://github.com/lbryio/lbry/pull/1294)
* `ComponentManager` for managing the life-cycles of dependencies (pr https://github.com/lbryio/lbry/pull/1294)
* `requires` decorator to register the components required by a `jsonrpc_` command, to facilitate commands registering asynchronously (pr https://github.com/lbryio/lbry/pull/1294)
* unit tests for `ComponentManager` (pr https://github.com/lbryio/lbry/pull/1294)
* script to generate docs/api.json file (https://github.com/lbryio/lbry.tech/issues/42)
* additional information to the balance error message when editing a claim (pr https://github.com/lbryio/lbry/pull/1309)
* `address` and `port` arguments to `peer_ping` (https://github.com/lbryio/lbry/issues/1313) (pr https://github.com/lbryio/lbry/pull/1299)
* ability to download from HTTP mirrors by setting `download_mirrors` (prs https://github.com/lbryio/lbry/pull/1233 and https://github.com/lbryio/lbry/pull/1315)
* ability to filter peers from an iterative find value operation (finding peers for a blob). This is used to filter peers we've already found for a blob when accumulating the list of peers. (pr https://github.com/lbryio/lbry/pull/1287)
### Removed
*
*
* `session_status` argument and response field from `status` (pr https://github.com/lbryio/lbry/pull/1328)
* most of the internal attributes from `Daemon` (pr https://github.com/lbryio/lbry/pull/1294)
## [0.20.4] - 2018-07-18
### Fixed
* spelling errors in messages printed by `lbrynet-cli`
* high CPU usage when a stream is incomplete and the peers we're requesting from have no more blobs to send us (https://github.com/lbryio/lbry/pull/1301)
### Changed
* keep track of failures for DHT peers for up to ten minutes instead of indefinitely (https://github.com/lbryio/lbry/pull/1300)
* skip ignored peers from iterative lookups instead of blocking the peer who returned them to us too (https://github.com/lbryio/lbry/pull/1300)
* if a node becomes ignored during an iterative find cycle remove it from the shortlist so that we can't return it as a result nor try to probe it anyway (https://github.com/lbryio/lbry/pull/1303)
## [0.20.3] - 2018-07-03
### Fixed
* `blob_list` raising an error when blobs in a stream haven't yet been created (8a0d0b44ddf9cbeb2a9074eed39d6064ce21df64)
* stopping a download potentially raising an attribute error (https://github.com/lbryio/lbry/pull/1269)
* file manager startup locking up when there are many files for some channels (https://github.com/lbryio/lbry/pull/1281)
* improper sorting when getting the closest peers to a hash (https://github.com/lbryio/lbry/pull/1282)
### Changed
* raised the default `peer_search_timeout` setting from 3 to 30 and added logging for when it happens (https://github.com/lbryio/lbry/pull/1283)
* change iterative find stop condition on find value to allow it to continue until a value is found or it times out (https://github.com/lbryio/lbry/pull/1283)
* include all of our own blobs in the local dht datastore (as if we had announced them to ourselves) (https://github.com/lbryio/lbry/pull/1280)
* ignore dht `store` token validation errors for the first expiration-time after startup (fixes failed `store` requests after a restart) (https://github.com/lbryio/lbry/pull/1280)
### Removed
* `jsonrpclib` as a requirement for the project (https://github.com/lbryio/lbry/pull/1274)
## [0.20.2] - 2018-06-23
@@ -304,7 +486,7 @@ at anytime.
* Fixed https://github.com/lbryio/lbry/issues/923
* Fixed concurrent reflects opening too many files
* Fixed cases when reflecting would fail on error conditions
* Fixed deadlocks from occuring during blob writes
* Fixed deadlocks from occurring during blob writes
* Fixed and updated`lbrynet.tests.dht`
* Fixed redundant dht id
* Fixed dht `ping` method
@@ -691,7 +873,7 @@ at anytime.
* Removed check_pending logic from Daemon
* Switched to txrequests so requests can use twisted event loop
* Renamed API command file_seed to file_set_status
* Dont add expected payment to wallet when payment rate is 0
* Don't add expected payment to wallet when payment rate is 0
### Fixed
* Fix restart procedure in DaemonControl
* Create download directory if it doesn't exist
@@ -722,7 +904,7 @@ at anytime.
* Added string comparison to ClaimOutpoint (needed to look things up by outpoint)
* Remove unused API commands from daemon
* Fix file filter `outpoint`
* Made dictionary key names in API commmand outputs to be more consistent
* Made dictionary key names in API command outputs to be more consistent
### Added
* Add file filters: `claim_id`, `outpoint`, and `rowid`
* Make loggly logs less verbose
@@ -774,7 +956,7 @@ at anytime.
* Include download url in version check
### Fixed
* add misssing traceback to logging
* add missing traceback to logging
## [0.8.3] - 2017-02-15
### Fixed
+36 -65
View File
@@ -1,70 +1,48 @@
# Installing LBRY
The recommended way to install LBRY is to use an installer. We provide installers for all major operating systems. See the [README](README.md).
If only the JSON-RPC API server is needed, the recommended way to install LBRY is to use a pre-built binary. We provide binaries for all major operating systems. See the [README](README.md).
These instructions are for installing from source code, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
These instructions are for installing LBRY from source, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
Here's a video walkthrough of this setup, which is itself hosted by the LBRY network and provided via [spee.ch](https://github.com/lbryio/spee.ch):
[![Setup for development](https://spee.ch/2018-10-04-17-13-54-017046806.png)](https://spee.ch/967f99344308f1e90f0620d91b6c93e4dfb240e0/lbrynet-dev-setup.mp4)
## Prerequisites
Before starting, you'll need to install some additional libraries depending on your operating system.
Running `lbrynet` from source requires Python 3.6 or higher (3.7 is preferred). Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/)
### Python
After installing python 3, you'll need to install some additional libraries depending on your operating system.
LBRY requires Python 2.7. Python 3 is not supported.
### Virtualenv
Once python 3 is installed run `python3 -m pip install virtualenv` to install virtualenv.
### Windows
Windows users will need to install `Visual C++ Build Tools`, which can be installed by [Microsoft Build Tools](Microsoft Build Tools 2015)
### OS X Prerequisites
### macOS
Install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
macOS users will need to install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
Remaining dependencies can then be installed by running:
```
brew install mpfr libmpc protobuf
sudo easy_install pip
sudo pip install virtualenv
brew install python3 protobuf
```
### Linux Prerequisites
### Linux
On Ubuntu (we recommend 16.04), install the following:
On Ubuntu (we recommend 18.04), install the following:
```
sudo apt-get install build-essential python2.7 python2.7-dev python-pip git python-virtualenv libssl-dev libffi-dev python-protobuf
sudo apt-get install build-essential python3.7 python3.7-dev git python-virtualenv libssl-dev python-protobuf
```
On Raspbian, you will also need to install `python-pyparsing`.
If you're running another Linux flavor, install the equivalent of the above packages for your system.
### Windows Prerequisites
Install [mingw32](http://www.mingw.org/) base and c++ compiler.
Add C:\MinGW\bin to the windows PATH.
Enable distutils to compile with mingw32 by creating a distutils.cfg file in *PYTHONPATH\Lib\distutils* containing:
```
[build]
compiler = mingw32
```
If using virtualenv, which is recommended, copy the *PYTHONPATH\Lib\distutils* directory to the virtualenv.
It's recommended to use [Unofficial Windows Binaries for Python Extension Packages](http://www.lfd.uci.edu/~gohlke/pythonlibs/) for as many of the required packages as possible.
Currently, available binaries include:
- Cython
- Twisted
- Zope.interface
- pywin32
- requests
Install the above binaries with `pip install *.whl`
Install pywin32 system files by run `python.exe Scripts\pywin32_postinstall.py -install` from an elevated command prompt.
Finally, you'll need [miniupnpc](https://pypi.python.org/pypi/miniupnpc/1.9). Download the source and compile with MinGW by running `mingw32make.bat`. Then install the module by running `python setupmingw32.py install`.
If you're running another Linux distro, install the equivalent of the above packages for your system.
## Installation
@@ -73,34 +51,27 @@ To install:
```
git clone https://github.com/lbryio/lbry.git
cd lbry
virtualenv lbry-venv
virtualenv lbry-venv --python=python3.7
source lbry-venv/bin/activate
python --version # Python 3 is not supported. Make sure you're on Python 2.7
pip install -U -r requirements.txt
pip install --editable .
python --version # Python 2 is not supported. Make sure you're on Python 3.7
pip install -e .
```
To start LBRY, run `lbrynet-daemon` in a terminal.
To verify your installation, `which lbrynet` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
To verify your install, `which lbrynet-daemon` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
## Run the tests
To run the unit tests from the repo directory:
```
trial --reactor=asyncio tests.unit
```
## Making a Change
## Usage
If you are not already inside the virtual environment, run:
To start the API server:
`lbrynet start`
```
source lbry-venv/bin/activate
```
Then run:
```
pip install --editable .
```
This will install `lbrynet-daemon` in such a way that the changes you make to the code will be used as soon as you restart the daemon.
Happy hacking!
+38 -34
View File
@@ -1,52 +1,56 @@
[![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry)
[![Coverage Status](https://coveralls.io/repos/github/lbryio/lbry/badge.svg)](https://coveralls.io/github/lbryio/lbry)
# <img src="https://raw.githubusercontent.com/lbryio/lbry/master/lbry.png" alt="LBRY" width="48" height="36" /> LBRY SDK [![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry) [![Test Coverage](https://codecov.io/gh/lbryio/lbry/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry)
# LBRY
LBRY is a decentralized peer-to-peer network providing distribution, discovery, and purchase of digital content (data). It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, rights and access rules. LBRY also provides a data network that consists of peers (seeders) uploading and downloading data from other peers, possibly in exchange for payments, as well as a distributed hash table used by peers to discover other peers.
LBRY is an open-source protocol providing distribution, discovery, and purchase of digital content (data) via a decentralized network.
LBRY SDK for Python is currently the most full featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components:
This repo is a reference implementation of the [LBRY API](https://lbry.io/api).
* Built on Python 3.7+ and `asyncio`.
* DHT (Distributed Hash Table) implementation for finding peers ([lbrynet.dht](https://github.com/lbryio/lbry/tree/master/lbrynet/dht)).
* Blob exchange protocol for downloading content and negotiating payments ([lbrynet.blob_exchange](https://github.com/lbryio/lbry/tree/master/lbrynet/blob_exchange)).
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([lbrynet.schema](https://github.com/lbryio/lbry/tree/master/lbrynet/schema)).
* Wallet implementation for the LBRY blockchain ([lbrynet.extras.wallet](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/wallet)).
* Daemon with a JSON-RPC API to ease building end user applications in any language and for automating various tasks ([lbrynet.extras.daemon](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/daemon)).
It provides a daemon that can interact with the network via a json-rpc interface over HTTP.
## Installation
## Installing
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
**Note**: This project no longer bundles a graphic interface (browser). If you want to use LBRY via a GUI, [use the LBRY App](https://github.com/lbryio/lbry-app).
## Usage
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases, for macOS, Debian-based Linux, and Windows.
Run `lbrynet start` to launch the API server.
Installing from source is also relatively painless, full instructions are in [INSTALL.md](INSTALL.md)
By default, `lbrynet` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
## Running
Our [quickstart guide](https://lbry.tech/playground) provides a simple walkthrough and examples for learning.
Run `lbrynet-daemon` to launch the daemon.
With the daemon running, `lbrynet commands` will show you a list of commands.
## Using
The full API is documented [here](https://lbry.tech/api/sdk).
By default, `lbrynet-daemon` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
## Running from source
Our [quickstart guide](http://lbry.io/quickstart) provides a simple walkthrough and examples for learning.
Installing from source is also relatively painless. Full instructions are in [INSTALL.md](INSTALL.md)
The full API is documented [here](https://lbryio.github.io/lbry/cli).
## Contributing
## What is LBRY?
LBRY is a protocol that provides a fully decentralized network for the discovery, distribution, and payment of data.
It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, and rights and access rules.
LBRY also provides a data network that consists of peers uploading and downloading data from other peers, possibly in exchange for payments, and a distributed hash table, used by peers to discover other peers.
## Contributions
To contribute, [join us on Discord](https://chat.lbry.io/) or contact jeremy@lbry.io. Pull requests are also welcome.
## Support
Please open an issue and describe your situation in detail. We will respond as soon as we can.
For private issues, contact jeremy@lbry.io.
Contributions to this project are welcome, encouraged, and compensated. For more details, please check [this](https://lbry.io/faq/contributing) link.
## License
See [LICENSE](LICENSE)
This project is MIT licensed. For the full license, see [LICENSE](LICENSE).
## Security
We take security seriously. Please contact security@lbry.io regarding any security issues. [Our GPG key is here](https://lbry.io/faq/gpg-key) if you need it.
## Contact
The primary contact for this project is [@eukreign](mailto:lex@lbry.io).
## Additional information and links
The documentation for the API can be found [here](https://lbry.tech/api/sdk).
Daemon defaults, ports, and other settings are documented [here](https://lbry.tech/resources/daemon-settings).
Settings can be configured using a daemon-settings.yml file. An example can be found [here](https://github.com/lbryio/lbry/blob/master/example_daemon_settings.yml).
-33
View File
@@ -1,33 +0,0 @@
$env:Path += ";C:\MinGW\bin\"
$env:Path += ";C:\Program Files (x86)\Windows Kits\10\bin\x86\"
gcc --version
mingw32-make --version
# build/install miniupnpc manually
tar zxf miniupnpc-1.9.tar.gz
cd miniupnpc-1.9
mingw32-make -f Makefile.mingw
python setupmingw32.py build --compiler=mingw32
python setupmingw32.py install
cd ..\
Remove-Item -Recurse -Force miniupnpc-1.9
# copy requirements from lbry, but remove miniupnpc (installed manually)
Get-Content ..\requirements.txt | Select-String -Pattern 'miniupnpc' -NotMatch | Out-File requirements_base.txt
python set_build.py
pip install -r requirements.txt
pip install ..\.
pyinstaller -y daemon.onefile.spec
pyinstaller -y cli.onefile.spec
pyinstaller -y console.onefile.spec
nuget install secure-file -ExcludeVersion
secure-file\tools\secure-file -decrypt .\lbry2.pfx.enc -secret "$env:pfx_key"
signtool.exe sign /f .\lbry2.pfx /p "$env:key_pass" /tr http://tsa.starfieldtech.com /td SHA256 /fd SHA256 dist\*.exe
python zip_daemon.py
python upload_assets.py
-59
View File
@@ -1,59 +0,0 @@
#!/bin/bash
set -euo pipefail
set -x
ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
cd "$ROOT"
BUILD_DIR="$ROOT/build"
FULL_BUILD="${FULL_BUILD:-false}"
if [ -n "${TEAMCITY_VERSION:-}" -o -n "${APPVEYOR:-}" ]; then
FULL_BUILD="true"
fi
[ -d "$BUILD_DIR/bulid" ] && rm -rf "$BUILD_DIR/build"
[ -d "$BUILD_DIR/dist" ] && rm -rf "$BUILD_DIR/dist"
if [ "$FULL_BUILD" == "true" ]; then
# install dependencies
$BUILD_DIR/prebuild.sh
VENV="$BUILD_DIR/venv"
if [ -d "$VENV" ]; then
rm -rf "$VENV"
fi
virtualenv "$VENV"
set +u
source "$VENV/bin/activate"
set -u
# must set build before installing lbrynet. otherwise it has no effect
python "$BUILD_DIR/set_build.py"
fi
cp "$ROOT/requirements.txt" "$BUILD_DIR/requirements_base.txt"
(
cd "$BUILD_DIR"
pip install -r requirements.txt
)
(
cd "$BUILD_DIR"
pyinstaller -y daemon.onefile.spec
pyinstaller -y cli.onefile.spec
pyinstaller -y console.onefile.spec
)
python "$BUILD_DIR/zip_daemon.py"
if [ "$FULL_BUILD" == "true" ]; then
# electron-build has a publish feature, but I had a hard time getting
# it to reliably work and it also seemed difficult to configure. Not proud of
# this, but it seemed better to write my own.
python "$BUILD_DIR/upload_assets.py"
deactivate
fi
echo 'Build complete.'
-42
View File
@@ -1,42 +0,0 @@
# -*- mode: python -*-
import platform
import os
dir = 'build';
cwd = os.getcwd()
if os.path.basename(cwd) != dir:
raise Exception('pyinstaller build needs to be run from the ' + dir + ' directory')
repo_base = os.path.abspath(os.path.join(cwd, '..'))
execfile(os.path.join(cwd, "entrypoint.py")) # ghetto import
system = platform.system()
if system == 'Darwin':
icns = os.path.join(repo_base, 'build', 'icon.icns')
elif system == 'Linux':
icns = os.path.join(repo_base, 'build', 'icons', '256x256.png')
elif system == 'Windows':
icns = os.path.join(repo_base, 'build', 'icons', 'lbry256.ico')
else:
print 'Warning: System {} has no icons'.format(system)
icns = None
a = Entrypoint('lbrynet', 'console_scripts', 'lbrynet-cli', pathex=[cwd])
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='lbrynet-cli',
debug=False,
strip=False,
upx=True,
console=True,
icon=icns
)
-50
View File
@@ -1,50 +0,0 @@
# -*- mode: python -*-
import platform
import os
import lbryum
dir = 'build';
cwd = os.getcwd()
if os.path.basename(cwd) != dir:
raise Exception('pyinstaller build needs to be run from the ' + dir + ' directory')
repo_base = os.path.abspath(os.path.join(cwd, '..'))
execfile(os.path.join(cwd, "entrypoint.py")) # ghetto import
system = platform.system()
if system == 'Darwin':
icns = os.path.join(repo_base, 'build', 'icon.icns')
elif system == 'Linux':
icns = os.path.join(repo_base, 'build', 'icons', '256x256.png')
elif system == 'Windows':
icns = os.path.join(repo_base, 'build', 'icons', 'lbry256.ico')
else:
print 'Warning: System {} has no icons'.format(system)
icns = None
datas = [
(os.path.join(os.path.dirname(lbryum.__file__), 'wordlist', language + '.txt'), 'lbryum/wordlist')
for language in ('chinese_simplified', 'japanese', 'spanish','english', 'portuguese')
]
a = Entrypoint('lbrynet', 'console_scripts', 'lbrynet-console', pathex=[cwd], datas=datas)
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='lbrynet-console',
debug=False,
strip=False,
upx=True,
console=True,
icon=icns
)
-50
View File
@@ -1,50 +0,0 @@
# -*- mode: python -*-
import platform
import os
import lbryum
dir = 'build';
cwd = os.getcwd()
if os.path.basename(cwd) != dir:
raise Exception('pyinstaller build needs to be run from the ' + dir + ' directory')
repo_base = os.path.abspath(os.path.join(cwd, '..'))
execfile(os.path.join(cwd, "entrypoint.py")) # ghetto import
system = platform.system()
if system == 'Darwin':
icns = os.path.join(repo_base, 'build', 'icon.icns')
elif system == 'Linux':
icns = os.path.join(repo_base, 'build', 'icons', '256x256.png')
elif system == 'Windows':
icns = os.path.join(repo_base, 'build', 'icons', 'lbry256.ico')
else:
print 'Warning: System {} has no icons'.format(system)
icns = None
datas = [
(os.path.join(os.path.dirname(lbryum.__file__), 'wordlist', language + '.txt'), 'lbryum/wordlist')
for language in ('chinese_simplified', 'japanese', 'spanish','english', 'portuguese')
]
a = Entrypoint('lbrynet', 'console_scripts', 'lbrynet-daemon', pathex=[cwd], datas=datas)
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='lbrynet-daemon',
debug=False,
strip=False,
upx=True,
console=True,
icon=icns
)
-47
View File
@@ -1,47 +0,0 @@
# https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point
def Entrypoint(dist, group, name,
scripts=None, pathex=None, binaries=None, datas=None,
hiddenimports=None, hookspath=None, excludes=None, runtime_hooks=None,
cipher=None, win_no_prefer_redirects=False, win_private_assemblies=False):
import pkg_resources
# get toplevel packages of distribution from metadata
def get_toplevel(dist):
distribution = pkg_resources.get_distribution(dist)
if distribution.has_metadata('top_level.txt'):
return list(distribution.get_metadata('top_level.txt').split())
else:
return []
hiddenimports = hiddenimports or []
packages = []
for distribution in hiddenimports:
packages += get_toplevel(distribution)
scripts = scripts or []
pathex = pathex or []
# get the entry point
ep = pkg_resources.get_entry_info(dist, group, name)
# insert path of the egg at the verify front of the search path
pathex = [ep.dist.location] + pathex
# script name must not be a valid module name to avoid name clashes on import
script_path = os.path.join(workpath, name + '-script.py')
print "creating script for entry point", dist, group, name
with open(script_path, 'w') as fh:
fh.write("import {0}\n".format(ep.module_name))
fh.write("{0}.{1}()\n".format(ep.module_name, '.'.join(ep.attrs)))
for package in packages:
fh.write("import {0}\n".format(package))
return Analysis([script_path] + scripts,
pathex=pathex,
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=hookspath,
excludes=excludes,
runtime_hooks=runtime_hooks,
cipher=cipher,
win_no_prefer_redirects=win_no_prefer_redirects,
win_private_assemblies=win_private_assemblies
)
Binary file not shown.
Binary file not shown.
-81
View File
@@ -1,81 +0,0 @@
#!/bin/bash
set -euo pipefail
set -x
LINUX=false
OSX=false
if [ "$(uname)" == "Darwin" ]; then
OSX=true
elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
LINUX=true
else
echo "Platform detection failed"
exit 1
fi
SUDO=''
if $LINUX && (( $EUID != 0 )); then
SUDO='sudo'
fi
cmd_exists() {
command -v "$1" >/dev/null 2>&1
return $?
}
set +eu
GITUSERNAME=$(git config --global --get user.name)
if [ -z "$GITUSERNAME" ]; then
git config --global user.name "$(whoami)"
fi
GITEMAIL=$(git config --global --get user.email)
if [ -z "$GITEMAIL" ]; then
git config --global user.email "$(whoami)@lbry.io"
fi
set -eu
if $LINUX; then
INSTALL="$SUDO apt-get install --no-install-recommends -y"
$INSTALL build-essential libssl-dev libffi-dev python2.7-dev wget
elif $OSX && ! cmd_exists brew ; then
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
fi
if ! cmd_exists python; then
if $LINUX; then
$INSTALL python2.7
elif $OSX; then
brew install python
fi
fi
PYTHON_VERSION=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
if [ "$PYTHON_VERSION" != "2.7" ]; then
echo "Python 2.7 required"
exit 1
fi
if ! cmd_exists pip; then
if $LINUX; then
$INSTALL python-pip
$SUDO pip install --upgrade pip
else
echo "Pip required"
exit 1
fi
fi
if $LINUX && [ "$(pip list --format=columns | grep setuptools | wc -l)" -ge 1 ]; then
#$INSTALL python-setuptools
$SUDO pip install setuptools
fi
if ! cmd_exists virtualenv; then
$SUDO pip install virtualenv
fi
-11
View File
@@ -1,11 +0,0 @@
# install daemon requirements (created by build script. see build.sh, build.ps1)
-r requirements_base.txt
# install daemon itself. make sure you run `pip install` from this dir. this is how you do relative file paths with pip
file:../.
# install other build requirements
PyInstaller==3.2.1
requests[security]==2.13.0
uritemplate==3.0.0
boto3==1.4.4
-143
View File
@@ -1,143 +0,0 @@
import glob
import json
import os
import subprocess
import sys
import github
import uritemplate
import boto3
def main():
upload_to_github_if_tagged('lbryio/lbry')
upload_to_s3('daemon')
def get_asset_filename():
this_dir = os.path.dirname(os.path.realpath(__file__))
return glob.glob(this_dir + '/dist/*.zip')[0]
def upload_to_s3(folder):
tag = subprocess.check_output(['git', 'describe', '--always', '--abbrev=8', 'HEAD']).strip()
commit_date = subprocess.check_output([
'git', 'show', '-s', '--format=%cd', '--date=format:%Y%m%d-%H%I%S', 'HEAD']).strip()
asset_path = get_asset_filename()
bucket = 'releases.lbry.io'
key = folder + '/' + commit_date + '-' + tag + '/' + os.path.basename(asset_path)
print "Uploading " + asset_path + " to s3://" + bucket + '/' + key + ''
if 'AWS_ACCESS_KEY_ID' not in os.environ or 'AWS_SECRET_ACCESS_KEY' not in os.environ:
print 'Must set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to publish assets to s3'
return 1
s3 = boto3.resource(
's3',
aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],
config=boto3.session.Config(signature_version='s3v4')
)
s3.meta.client.upload_file(asset_path, bucket, key)
def upload_to_github_if_tagged(repo_name):
try:
current_tag = subprocess.check_output(
['git', 'describe', '--exact-match', 'HEAD']).strip()
except subprocess.CalledProcessError:
print 'Not uploading to GitHub as we are not currently on a tag'
return 1
print "Current tag: " + current_tag
if 'GH_TOKEN' not in os.environ:
print 'Must set GH_TOKEN in order to publish assets to a release'
return 1
gh_token = os.environ['GH_TOKEN']
auth = github.Github(gh_token)
repo = auth.get_repo(repo_name)
if not check_repo_has_tag(repo, current_tag):
print 'Tag {} is not in repo {}'.format(current_tag, repo)
# TODO: maybe this should be an error
return 1
asset_path = get_asset_filename()
print "Uploading " + asset_path + " to Github tag " + current_tag
release = get_github_release(repo, current_tag)
upload_asset_to_github(release, asset_path, gh_token)
def check_repo_has_tag(repo, target_tag):
tags = repo.get_tags().get_page(0)
for tag in tags:
if tag.name == target_tag:
return True
return False
def get_github_release(repo, current_tag):
for release in repo.get_releases():
if release.tag_name == current_tag:
return release
raise Exception('No release for {} was found'.format(current_tag))
def upload_asset_to_github(release, asset_to_upload, token):
basename = os.path.basename(asset_to_upload)
for asset in release.raw_data['assets']:
if asset['name'] == basename:
print 'File {} has already been uploaded to {}'.format(basename, release.tag_name)
return
upload_uri = uritemplate.expand(release.upload_url, {'name': basename})
count = 0
while count < 10:
try:
output = _curl_uploader(upload_uri, asset_to_upload, token)
if 'errors' in output:
raise Exception(output)
else:
print 'Successfully uploaded to {}'.format(output['browser_download_url'])
except Exception:
print 'Failed uploading on attempt {}'.format(count + 1)
count += 1
def _curl_uploader(upload_uri, asset_to_upload, token):
# using requests.post fails miserably with SSL EPIPE errors. I spent
# half a day trying to debug before deciding to switch to curl.
#
# TODO: actually set the content type
print 'Using curl to upload {} to {}'.format(asset_to_upload, upload_uri)
cmd = [
'curl',
'-sS',
'-X', 'POST',
'-u', ':{}'.format(os.environ['GH_TOKEN']),
'--header', 'Content-Type: application/octet-stream',
'--data-binary', '@-',
upload_uri
]
# '-d', '{"some_key": "some_value"}',
print 'Calling curl:'
print cmd
print
with open(asset_to_upload, 'rb') as fp:
p = subprocess.Popen(cmd, stdin=fp, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
stdout, stderr = p.communicate()
print 'curl return code:', p.returncode
if stderr:
print 'stderr output from curl:'
print stderr
print 'stdout from curl:'
print stdout
return json.loads(stdout)
if __name__ == '__main__':
sys.exit(main())
-29
View File
@@ -1,29 +0,0 @@
import os
import platform
import subprocess
import sys
import zipfile
def main():
this_dir = os.path.dirname(os.path.realpath(__file__))
tag = subprocess.check_output(['git', 'describe']).strip()
zipfilename = 'lbrynet-daemon-{}-{}.zip'.format(tag, get_system_label())
full_filename = os.path.join(this_dir, 'dist', zipfilename)
executables = ['lbrynet-daemon', 'lbrynet-cli', 'lbrynet-console']
ext = '.exe' if platform.system() == 'Windows' else ''
with zipfile.ZipFile(full_filename, 'w') as myzip:
for executable in executables:
myzip.write(os.path.join(this_dir, 'dist', executable + ext), executable + ext)
def get_system_label():
system = platform.system()
if system == 'Darwin':
return 'macos'
else:
return system.lower()
if __name__ == '__main__':
sys.exit(main())
+3 -3
View File
@@ -100,7 +100,7 @@
<div class="md-flex__cell md-flex__cell--shrink">
<a href="/" title="LBRY" class="md-header-nav__button md-logo">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" width="24" height="24">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" alt="LBRY logo" width="24" height="24">
</a>
</div>
@@ -196,7 +196,7 @@
<label class="md-nav__title md-nav__title--site" for="drawer">
<span class="md-nav__button md-logo">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" width="48" height="48">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" alt="LBRY logo" width="48" height="48">
</span>
LBRY
@@ -304,4 +304,4 @@
</body>
</html>
</html>
+1605
View File
File diff suppressed because it is too large. Load diff
+61 -38
View File
@@ -1454,7 +1454,7 @@ Returns:
'is_spent': (bool) true if claim is abandoned, false otherwise
'name': (str) name of the claim
'permanent_url': (str) permanent url of the claim,
'txid': (str) txid of the cliam
'txid': (str) txid of the claim
'nout': (int) nout of the claim
'value': (str) value of the claim
},
@@ -1579,12 +1579,12 @@ Usage:
Options:
--a_arg : a arg
--b_arg : b arg
--pos_arg=&lt;pos_arg&gt; : pos arg
--pos_args=&lt;pos_args&gt; : pos args
--pos_arg2=&lt;pos_arg2&gt; : pos arg 2
--pos_arg3=&lt;pos_arg3&gt; : pos arg 3
--a_arg : (bool) a arg
--b_arg : (bool) b arg
--pos_arg=&lt;pos_arg&gt; : (int) pos arg
--pos_args=&lt;pos_args&gt; : (int) pos args
--pos_arg2=&lt;pos_arg2&gt; : (int) pos arg 2
--pos_arg3=&lt;pos_arg3&gt; : (int) pos arg 3
Returns:
pos args
@@ -1828,14 +1828,16 @@ Returns:
</code></pre>
<h2 id="peer_ping">peer_ping</h2>
<pre><code class="text">Find and ping a peer by node id
<pre><code class="text">Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,
if not provided the peer is located first.
Usage:
peer_ping (&lt;node_id&gt; | --node_id=&lt;node_id&gt;)
peer_ping (&lt;node_id&gt; | --node_id=&lt;node_id&gt;) [&lt;address&gt; | --address=&lt;address&gt;] [&lt;port&gt; | --port=&lt;port&gt;]
Options:
None
--address=&lt;address&gt; : (str) ip address of the peer
--port=&lt;port&gt; : (int) udp port of the peer
Returns:
(str) pong, or {'error': &lt;error message&gt;} if an error is encountered
@@ -1893,7 +1895,7 @@ Options:
--license_url=&lt;license_url&gt; : (str) publication license url
--thumbnail=&lt;thumbnail&gt; : (str) thumbnail url
--preview=&lt;preview&gt; : (str) preview url
--nsfw=&lt;nsfw&gt; : (bool) title of the publication
--nsfw=&lt;nsfw&gt; : (bool) whether the content is nsfw
--sources=&lt;sources&gt; : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file
--channel_name=&lt;channel_name&gt; : (str) name of the publisher channel name in the wallet
--channel_id=&lt;channel_id&gt; : (str) claim id of the publisher channel, does not check
@@ -1901,7 +1903,7 @@ Options:
publishing to a channel where only the certificate
private key is in the wallet.
--claim_address=&lt;claim_address&gt; : (str) address where the claim is sent to, if not specified
new address wil automatically be created
new address will automatically be created
Returns:
(dict) Dictionary containing result of the claim
@@ -2110,41 +2112,62 @@ Returns:
<pre><code class="text">Get daemon status
Usage:
status [--session_status]
status
Options:
--session_status : (bool) include session status in results
None
Returns:
(dict) lbrynet-daemon status
{
'lbry_id': lbry peer id, base58,
'installation_id': installation id, base58,
'is_running': bool,
'installation_id': (str) installation id - base58,
'is_running': (bool),
'is_first_run': bool,
'startup_status': {
'code': status code,
'message': status message
'skipped_components': (list) [names of skipped components (str)],
'startup_status': { Does not include components which have been skipped
'database': (bool),
'wallet': (bool),
'session': (bool),
'dht': (bool),
'hash_announcer': (bool),
'stream_identifier': (bool),
'file_manager': (bool),
'blob_manager': (bool),
'blockchain_headers': (bool),
'peer_protocol_server': (bool),
'reflector': (bool),
'upnp': (bool),
'exchange_rate_manager': (bool),
},
'connection_status': {
'code': connection status code,
'message': connection status message
'code': (str) connection status code,
'message': (str) connection status message
},
'blockchain_status': {
'blocks': local blockchain height,
'blocks_behind': remote_height - local_height,
'best_blockhash': block hash of most recent block,
'blockchain_headers': {
'downloading_headers': (bool),
'download_progress': (float) 0-100.0
},
'wallet_is_encrypted': bool,
If given the session status option:
'session_status': {
'managed_blobs': count of blobs in the blob manager,
'managed_streams': count of streams in the file manager
'announce_queue_size': number of blobs currently queued to be announced
'should_announce_blobs': number of blobs that should be announced
}
'wallet': {
'blocks': (int) local blockchain height,
'blocks_behind': (int) remote_height - local_height,
'best_blockhash': (str) block hash of most recent block,
'is_encrypted': (bool),
'is_locked': (bool),
},
'dht': {
'node_id': (str) lbry dht node id - hex encoded,
'peers_in_routing_table': (int) the number of peers in the routing table,
},
'blob_manager': {
'finished_blobs': (int) number of finished blobs in the blob manager,
},
'hash_announcer': {
'announce_queue_size': (int) number of blobs currently queued to be announced
},
'file_manager': {
'managed_files': (int) count of files in the file manager,
}
}
</code></pre>
@@ -2452,9 +2475,9 @@ Usage:
Options:
--amount=&lt;amount&gt; : (float) amount of credit to send
--address=&lt;address&gt; : (str) address to send credits to
--claim_id=&lt;claim_id&gt; : (float) claim_id of the claim to send to tip to
--amount=&lt;amount&gt; : (Decimal) amount of credit to send
--address=&lt;address&gt; : (str) address to send credits to
--claim_id=&lt;claim_id&gt; : (str) claim_id of the claim to send to tip to
Returns:
If sending to an address:
+59 -36
View File
@@ -1380,7 +1380,7 @@ Returns:
'is_spent': (bool) true if claim is abandoned, false otherwise
'name': (str) name of the claim
'permanent_url': (str) permanent url of the claim,
'txid': (str) txid of the cliam
'txid': (str) txid of the claim
'nout': (int) nout of the claim
'value': (str) value of the claim
},
@@ -1479,12 +1479,12 @@ Returns:
<h2 id="cli_test_command">cli_test_command</h2>
<pre><code class="text">This command is only for testing the CLI argument parsing
Args:
'a_arg' : a arg
'b_arg' : b arg
'pos_arg' (required) : pos arg
'pos_args' : pos args
'pos_arg2' : pos arg 2
'pos_arg3' : pos arg 3
'a_arg' : (bool) a arg
'b_arg' : (bool) b arg
'pos_arg' (required) : (int) pos arg
'pos_args' : (int) pos args
'pos_arg2' : (int) pos arg 2
'pos_arg3' : (int) pos arg 3
Returns:
pos args
@@ -1682,10 +1682,12 @@ Returns:
</code></pre>
<h2 id="peer_ping">peer_ping</h2>
<pre><code class="text">Find and ping a peer by node id
<pre><code class="text">Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,
if not provided the peer is located first.
Args:
None
'address' : (str) ip address of the peer
'port' : (int) udp port of the peer
Returns:
(str) pong, or {'error': &lt;error message&gt;} if an error is encountered
@@ -1733,7 +1735,7 @@ Args:
'license_url' : (str) publication license url
'thumbnail' : (str) thumbnail url
'preview' : (str) preview url
'nsfw' : (bool) title of the publication
'nsfw' : (bool) whether the content is nsfw
'sources' : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file
'channel_name' : (str) name of the publisher channel name in the wallet
'channel_id' : (str) claim id of the publisher channel, does not check
@@ -1741,7 +1743,7 @@ Args:
publishing to a channel where only the certificate
private key is in the wallet.
'claim_address' : (str) address where the claim is sent to, if not specified
new address wil automatically be created
new address will automatically be created
Returns:
(dict) Dictionary containing result of the claim
@@ -1913,37 +1915,58 @@ Returns:
<pre><code class="text">Get daemon status
Args:
'session_status' : (bool) include session status in results
None
Returns:
(dict) lbrynet-daemon status
{
'lbry_id': lbry peer id, base58,
'installation_id': installation id, base58,
'is_running': bool,
'installation_id': (str) installation id - base58,
'is_running': (bool),
'is_first_run': bool,
'startup_status': {
'code': status code,
'message': status message
'skipped_components': (list) [names of skipped components (str)],
'startup_status': { Does not include components which have been skipped
'database': (bool),
'wallet': (bool),
'session': (bool),
'dht': (bool),
'hash_announcer': (bool),
'stream_identifier': (bool),
'file_manager': (bool),
'blob_manager': (bool),
'blockchain_headers': (bool),
'peer_protocol_server': (bool),
'reflector': (bool),
'upnp': (bool),
'exchange_rate_manager': (bool),
},
'connection_status': {
'code': connection status code,
'message': connection status message
'code': (str) connection status code,
'message': (str) connection status message
},
'blockchain_status': {
'blocks': local blockchain height,
'blocks_behind': remote_height - local_height,
'best_blockhash': block hash of most recent block,
'blockchain_headers': {
'downloading_headers': (bool),
'download_progress': (float) 0-100.0
},
'wallet_is_encrypted': bool,
If given the session status option:
'session_status': {
'managed_blobs': count of blobs in the blob manager,
'managed_streams': count of streams in the file manager
'announce_queue_size': number of blobs currently queued to be announced
'should_announce_blobs': number of blobs that should be announced
}
'wallet': {
'blocks': (int) local blockchain height,
'blocks_behind': (int) remote_height - local_height,
'best_blockhash': (str) block hash of most recent block,
'is_encrypted': (bool),
'is_locked': (bool),
},
'dht': {
'node_id': (str) lbry dht node id - hex encoded,
'peers_in_routing_table': (int) the number of peers in the routing table,
},
'blob_manager': {
'finished_blobs': (int) number of finished blobs in the blob manager,
},
'hash_announcer': {
'announce_queue_size': (int) number of blobs currently queued to be announced
},
'file_manager': {
'managed_files': (int) count of files in the file manager,
}
}
</code></pre>
@@ -2186,9 +2209,9 @@ to the owner of a claim specified by uri. A tip is a claim support where the rec
of the support is the claim address for the claim being supported.
Args:
'amount' (required) : (float) amount of credit to send
'address' (required) : (str) address to send credits to
'claim_id' (required) : (float) claim_id of the claim to send to tip to
'amount' (required) : (Decimal) amount of credit to send
'address' (required) : (str) address to send credits to
'claim_id' (required) : (str) claim_id of the claim to send to tip to
Returns:
If sending to an address:
+14 -14
View File
@@ -2,7 +2,7 @@
"docs": [
{
"location": "/",
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nArgs:\n 'blob_hash' : (str) announce a blob, specified by blob_hash\n 'stream_hash' : (str) announce all blobs associated with\n stream_hash\n 'sd_hash' : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nArgs:\n 'blob_hash' : (str) check availability for this blob hash\n 'search_timeout' : (int) how long to search for peers for the blob\n in the dht\n 'blob_timeout' : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nArgs:\n 'blob_hash' : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nArgs:\n 'blob_hash' (required) : (str) blob hash of the blob to get\n 'timeout' : (int) timeout in number of seconds\n 'encoding' : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n 'payment_rate_manager' : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nArgs:\n 'needed' : (bool) only return needed blobs\n 'finished' : (bool) only return finished blobs\n 'uri' : (str) filter blobs by stream in a uri\n 'stream_hash' : (str) filter blobs by stream hash\n 'sd_hash' : (str) filter blobs by sd hash\n 'page_size' : (int) results page size\n 'page' : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nArgs:\n 'reflector_server' : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nArgs:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nArgs:\n 'blockhash' (required) : (str) hash of the block to look up\n 'height' (required) : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nArgs:\n 'claim_id' (required) : (str) Claim ID to export information about\n\nReturns:\n (str) Serialized certificate information\n\n\n\n\nchannel_import\n\n\nImport serialized channel signing information (to allow signing new claims to the channel)\n\nArgs:\n 'serialized_certificate_info' (required) : (str) certificate info\n\nReturns:\n (dict) Result dictionary\n\n\n\n\nchannel_list\n\n\nGet certificate claim infos for channels that can be published to\n\nArgs:\n None\n\nReturns:\n (list) ClaimDict, includes 'is_mine' field to indicate if the certificate claim\n is in the wallet.\n\n\n\n\nchannel_new\n\n\nGenerate a publisher key and create a new '@' prefixed certificate claim\n\nArgs:\n 'channel_name' (required) : (str) name of the channel prefixed with '@'\n 'amount' (required) : (float) bid amount on the channel\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (float) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }\n\n\n\n\nclaim_abandon\n\n\nAbandon a name and reclaim credits from the claim\n\nArgs:\n 'claim_id' : (str) claim_id of the claim to abandon\n 'txid' : (str) txid of the claim to abandon\n 'nout' : (int) nout of the claim to abandon\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n txid : (str) txid of resulting transaction\n fee : (float) fee paid for the transaction\n }\n\n\n\n\nclaim_list\n\n\nList current claims and information about them for a given name\n\nArgs:\n 'name' (required) : (str) name of the claim to list info about\n\nReturns:\n (dict) State Line truncated
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nArgs:\n 'blob_hash' : (str) announce a blob, specified by blob_hash\n 'stream_hash' : (str) announce all blobs associated with\n stream_hash\n 'sd_hash' : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nArgs:\n 'blob_hash' : (str) check availability for this blob hash\n 'search_timeout' : (int) how long to search for peers for the blob\n in the dht\n 'blob_timeout' : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nArgs:\n 'blob_hash' : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nArgs:\n 'blob_hash' (required) : (str) blob hash of the blob to get\n 'timeout' : (int) timeout in number of seconds\n 'encoding' : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n 'payment_rate_manager' : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nArgs:\n 'needed' : (bool) only return needed blobs\n 'finished' : (bool) only return finished blobs\n 'uri' : (str) filter blobs by stream in a uri\n 'stream_hash' : (str) filter blobs by stream hash\n 'sd_hash' : (str) filter blobs by sd hash\n 'page_size' : (int) results page size\n 'page' : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nArgs:\n 'reflector_server' : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nArgs:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nArgs:\n 'blockhash' (required) : (str) hash of the block to look up\n 'height' (required) : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nArgs:\n 'claim_id' (required) : (str) Claim ID to export information about\n\nReturns:\n (str) Serialized certificate information\n\n\n\n\nchannel_import\n\n\nImport serialized channel signing information (to allow signing new claims to the channel)\n\nArgs:\n 'serialized_certificate_info' (required) : (str) certificate info\n\nReturns:\n (dict) Result dictionary\n\n\n\n\nchannel_list\n\n\nGet certificate claim infos for channels that can be published to\n\nArgs:\n None\n\nReturns:\n (list) ClaimDict, includes 'is_mine' field to indicate if the certificate claim\n is in the wallet.\n\n\n\n\nchannel_new\n\n\nGenerate a publisher key and create a new '@' prefixed certificate claim\n\nArgs:\n 'channel_name' (required) : (str) name of the channel prefixed with '@'\n 'amount' (required) : (float) bid amount on the channel\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (float) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }\n\n\n\n\nclaim_abandon\n\n\nAbandon a name and reclaim credits from the claim\n\nArgs:\n 'claim_id' : (str) claim_id of the claim to abandon\n 'txid' : (str) txid of the claim to abandon\n 'nout' : (int) nout of the claim to abandon\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n txid : (str) txid of resulting transaction\n fee : (float) fee paid for the transaction\n }\n\n\n\n\nclaim_list\n\n\nList current claims and information about them for a given name\n\nArgs:\n 'name' (required) : (str) name of the claim to list info about\n\nReturns:\n (dict) State Line truncated
"title": "API"
},
{
@@ -87,7 +87,7 @@
},
{
"location": "/#claim_list_mine",
"text": "List my name claims\n\nArgs:\n None\n\nReturns:\n (list) List of name claims owned by user\n [\n {\n 'address': (str) address that owns the claim\n 'amount': (float) amount assigned to the claim\n 'blocks_to_expiration': (int) number of blocks until it expires\n 'category': (str) claim , update , or support \n 'claim_id': (str) claim ID of the claim\n 'confirmations': (int) number of blocks of confirmations for the claim\n 'expiration_height': (int) the block height which the claim will expire\n 'expired': (bool) true if expired, false otherwise\n 'height': (int) height of the block containing the claim\n 'is_spent': (bool) true if claim is abandoned, false otherwise\n 'name': (str) name of the claim\n 'permanent_url': (str) permanent url of the claim,\n 'txid': (str) txid of the cliam\n 'nout': (int) nout of the claim\n 'value': (str) value of the claim\n },\n ]",
"text": "List my name claims\n\nArgs:\n None\n\nReturns:\n (list) List of name claims owned by user\n [\n {\n 'address': (str) address that owns the claim\n 'amount': (float) amount assigned to the claim\n 'blocks_to_expiration': (int) number of blocks until it expires\n 'category': (str) claim , update , or support \n 'claim_id': (str) claim ID of the claim\n 'confirmations': (int) number of blocks of confirmations for the claim\n 'expiration_height': (int) the block height which the claim will expire\n 'expired': (bool) true if expired, false otherwise\n 'height': (int) height of the block containing the claim\n 'is_spent': (bool) true if claim is abandoned, false otherwise\n 'name': (str) name of the claim\n 'permanent_url': (str) permanent url of the claim,\n 'txid': (str) txid of the claim\n 'nout': (int) nout of the claim\n 'value': (str) value of the claim\n },\n ]",
"title": "claim_list_mine"
},
{
@@ -112,7 +112,7 @@
},
{
"location": "/#cli_test_command",
"text": "This command is only for testing the CLI argument parsing\nArgs:\n 'a_arg' : a arg\n 'b_arg' : b arg\n 'pos_arg' (required) : pos arg\n 'pos_args' : pos args\n 'pos_arg2' : pos arg 2\n 'pos_arg3' : pos arg 3\n\nReturns:\n pos args",
"text": "This command is only for testing the CLI argument parsing\nArgs:\n 'a_arg' : (bool) a arg\n 'b_arg' : (bool) b arg\n 'pos_arg' (required) : (int) pos arg\n 'pos_args' : (int) pos args\n 'pos_arg2' : (int) pos arg 2\n 'pos_arg3' : (int) pos arg 3\n\nReturns:\n pos args",
"title": "cli_test_command"
},
{
@@ -162,12 +162,12 @@
},
{
"location": "/#peer_ping",
"text": "Find and ping a peer by node id\n\nArgs:\n None\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"text": "Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,\nif not provided the peer is located first.\n\nArgs:\n 'address' : (str) ip address of the peer\n 'port' : (int) udp port of the peer\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"title": "peer_ping"
},
{
"location": "/#publish",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nArgs:\n 'name' (required) : (str) name of the content\n 'bid' (required) : (decimal) amount to back the claim\n 'metadata' : (dict) ClaimDict to associate with the claim.\n 'file_path' : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n 'fee' : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n 'title' : (str) title of the publication\n 'description' : (str) description of the publication\n 'author' : (str) author of the publication\n 'language' : (str) language of the publication\n 'license' : (str) publication license\n 'license_url' : (str) publication license url\n 'thumbnail' : (str) thumbnail url\n 'preview' : (str) preview url\n 'nsfw' : (bool) title of the publication\n 'sources' : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n 'channel_name' : (str) name of the publisher channel name in the wallet\n 'channel_id' : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n 'claim_address' : (str) address where the claim is sent to, if not specified\n new address wil automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nArgs:\n 'name' (required) : (str) name of the content\n 'bid' (required) : (decimal) amount to back the claim\n 'metadata' : (dict) ClaimDict to associate with the claim.\n 'file_path' : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n 'fee' : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n 'title' : (str) title of the publication\n 'description' : (str) description of the publication\n 'author' : (str) author of the publication\n 'language' : (str) language of the publication\n 'license' : (str) publication license\n 'license_url' : (str) publication license url\n 'thumbnail' : (str) thumbnail url\n 'preview' : (str) preview url\n 'nsfw' : (bool) whether the content is nsfw\n 'sources' : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n 'channel_name' : (str) name of the publisher channel name in the wallet\n 'channel_id' : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n 'claim_address' : (str) address where the claim is sent to, if not specified\n new address will automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"title": "publish"
},
{
@@ -202,7 +202,7 @@
},
{
"location": "/#status",
"text": "Get daemon status\n\nArgs:\n 'session_status' : (bool) include session status in results\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'lbry_id': lbry peer id, base58,\n 'installation_id': installation id, base58,\n 'is_running': bool,\n 'is_first_run': bool,\n 'startup_status': {\n 'code': status code,\n 'message': status message\n },\n 'connection_status': {\n 'code': connection status code,\n 'message': connection status message\n },\n 'blockchain_status': {\n 'blocks': local blockchain height,\n 'blocks_behind': remote_height - local_height,\n 'best_blockhash': block hash of most recent block,\n },\n 'wallet_is_encrypted': bool,\n\n If given the session status option:\n 'session_status': {\n 'managed_blobs': count of blobs in the blob manager,\n 'managed_streams': count of streams in the file manager\n 'announce_queue_size': number of blobs currently queued to be announced\n 'should_announce_blobs': number of blobs that should be announced\n }\n }",
"text": "Get daemon status\n\nArgs:\n None\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'is_first_run': bool,\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'database': (bool),\n 'wallet': (bool),\n 'session': (bool),\n 'dht': (bool),\n 'hash_announcer': (bool),\n 'stream_identifier': (bool),\n 'file_manager': (bool),\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'peer_protocol_server': (bool),\n 'reflector': (bool),\n 'upnp': (bool),\n 'exchange_rate_manager': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the file manager,\n }\n }",
"title": "status"
},
{
@@ -277,7 +277,7 @@
},
{
"location": "/#wallet_send",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nArgs:\n 'amount' (required) : (float) amount of credit to send\n 'address' (required) : (str) address to send credits to\n 'claim_id' (required) : (float) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nArgs:\n 'amount' (required) : (Decimal) amount of credit to send\n 'address' (required) : (str) address to send credits to\n 'claim_id' (required) : (str) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"title": "wallet_send"
},
{
@@ -292,7 +292,7 @@
},
{
"location": "/cli/",
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nUsage:\n blob_announce [\nblob_hash\n | --blob_hash=\nblob_hash\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n] | [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) announce a blob, specified by blob_hash\n --stream_hash=\nstream_hash\n : (str) announce all blobs associated with\n stream_hash\n --sd_hash=\nsd_hash\n : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nUsage:\n blob_availability (\nblob_hash\n) [\nsearch_timeout\n | --search_timeout=\nsearch_timeout\n]\n [\nblob_timeout\n | --blob_timeout=\nblob_timeout\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) check availability for this blob hash\n --search_timeout=\nsearch_timeout\n : (int) how long to search for peers for the blob\n in the dht\n --blob_timeout=\nblob_timeout\n : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nUsage:\n blob_delete (\nblob_hash\n | --blob_hash=\nblob_hash)\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nUsage:\n blob_get (\nblob_hash\n | --blob_hash=\nblob_hash\n) [--timeout=\ntimeout\n]\n [--encoding=\nencoding\n] [--payment_rate_manager=\npayment_rate_manager\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to get\n --timeout=\ntimeout\n : (int) timeout in number of seconds\n --encoding=\nencoding\n : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n --payment_rate_manager=\npayment_rate_manager\n : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nUsage:\n blob_list [--needed] [--finished] [\nuri\n | --uri=\nuri\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n]\n [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n [\npage_size\n | --page_size=\npage_size\n]\n [\npage\n | --page=\npage\n]\n\n\nOptions:\n --needed : (bool) only return needed blobs\n --finished : (bool) only return finished blobs\n --uri=\nuri\n : (str) filter blobs by stream in a uri\n --stream_hash=\nstream_hash\n : (str) filter blobs by stream hash\n --sd_hash=\nsd_hash\n : (str) filter blobs by sd hash\n --page_size=\npage_size\n : (int) results page size\n --page=\npage\n : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nUsage:\n blob_reflect (\nblob_hashes\n...) [--reflector_server=\nreflector_server\n]\n\n\nOptions:\n --reflector_server=\nreflector_server\n : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nUsage:\n blob_reflect_all\n\n\nOptions:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nUsage:\n block_show (\nblockhash\n | --blockhash=\nblockhash\n) | (\nheight\n | --height=\nheight\n)\n\n\nOptions:\n --blockhash=\nblockhash\n : (str) hash of the block to look up\n --height=\nheight\n : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nUsage:\n channel_export (\nclaim_id\n | --claim_id=\nclaim_id\n)\n\n\nOptions:\n --claim_id=\nclaim_id\n : (str) Claim ID to export informatiLine truncated
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nUsage:\n blob_announce [\nblob_hash\n | --blob_hash=\nblob_hash\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n] | [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) announce a blob, specified by blob_hash\n --stream_hash=\nstream_hash\n : (str) announce all blobs associated with\n stream_hash\n --sd_hash=\nsd_hash\n : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nUsage:\n blob_availability (\nblob_hash\n) [\nsearch_timeout\n | --search_timeout=\nsearch_timeout\n]\n [\nblob_timeout\n | --blob_timeout=\nblob_timeout\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) check availability for this blob hash\n --search_timeout=\nsearch_timeout\n : (int) how long to search for peers for the blob\n in the dht\n --blob_timeout=\nblob_timeout\n : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nUsage:\n blob_delete (\nblob_hash\n | --blob_hash=\nblob_hash)\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nUsage:\n blob_get (\nblob_hash\n | --blob_hash=\nblob_hash\n) [--timeout=\ntimeout\n]\n [--encoding=\nencoding\n] [--payment_rate_manager=\npayment_rate_manager\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to get\n --timeout=\ntimeout\n : (int) timeout in number of seconds\n --encoding=\nencoding\n : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n --payment_rate_manager=\npayment_rate_manager\n : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nUsage:\n blob_list [--needed] [--finished] [\nuri\n | --uri=\nuri\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n]\n [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n [\npage_size\n | --page_size=\npage_size\n]\n [\npage\n | --page=\npage\n]\n\n\nOptions:\n --needed : (bool) only return needed blobs\n --finished : (bool) only return finished blobs\n --uri=\nuri\n : (str) filter blobs by stream in a uri\n --stream_hash=\nstream_hash\n : (str) filter blobs by stream hash\n --sd_hash=\nsd_hash\n : (str) filter blobs by sd hash\n --page_size=\npage_size\n : (int) results page size\n --page=\npage\n : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nUsage:\n blob_reflect (\nblob_hashes\n...) [--reflector_server=\nreflector_server\n]\n\n\nOptions:\n --reflector_server=\nreflector_server\n : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nUsage:\n blob_reflect_all\n\n\nOptions:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nUsage:\n block_show (\nblockhash\n | --blockhash=\nblockhash\n) | (\nheight\n | --height=\nheight\n)\n\n\nOptions:\n --blockhash=\nblockhash\n : (str) hash of the block to look up\n --height=\nheight\n : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nUsage:\n channel_export (\nclaim_id\n | --claim_id=\nclaim_id\n)\n\n\nOptions:\n --claim_id=\nclaim_id\n : (str) Claim ID to export informatiLine truncated
"title": "CLI"
},
{
@@ -377,7 +377,7 @@
},
{
"location": "/cli/#claim_list_mine",
"text": "List my name claims\n\nUsage:\n claim_list_mine\n\n\nOptions:\n None\n\nReturns:\n (list) List of name claims owned by user\n [\n {\n 'address': (str) address that owns the claim\n 'amount': (float) amount assigned to the claim\n 'blocks_to_expiration': (int) number of blocks until it expires\n 'category': (str) claim , update , or support \n 'claim_id': (str) claim ID of the claim\n 'confirmations': (int) number of blocks of confirmations for the claim\n 'expiration_height': (int) the block height which the claim will expire\n 'expired': (bool) true if expired, false otherwise\n 'height': (int) height of the block containing the claim\n 'is_spent': (bool) true if claim is abandoned, false otherwise\n 'name': (str) name of the claim\n 'permanent_url': (str) permanent url of the claim,\n 'txid': (str) txid of the cliam\n 'nout': (int) nout of the claim\n 'value': (str) value of the claim\n },\n ]",
"text": "List my name claims\n\nUsage:\n claim_list_mine\n\n\nOptions:\n None\n\nReturns:\n (list) List of name claims owned by user\n [\n {\n 'address': (str) address that owns the claim\n 'amount': (float) amount assigned to the claim\n 'blocks_to_expiration': (int) number of blocks until it expires\n 'category': (str) claim , update , or support \n 'claim_id': (str) claim ID of the claim\n 'confirmations': (int) number of blocks of confirmations for the claim\n 'expiration_height': (int) the block height which the claim will expire\n 'expired': (bool) true if expired, false otherwise\n 'height': (int) height of the block containing the claim\n 'is_spent': (bool) true if claim is abandoned, false otherwise\n 'name': (str) name of the claim\n 'permanent_url': (str) permanent url of the claim,\n 'txid': (str) txid of the claim\n 'nout': (int) nout of the claim\n 'value': (str) value of the claim\n },\n ]",
"title": "claim_list_mine"
},
{
@@ -402,7 +402,7 @@
},
{
"location": "/cli/#cli_test_command",
"text": "This command is only for testing the CLI argument parsing\nUsage:\n cli_test_command [--a_arg] [--b_arg] ( pos_arg | --pos_arg= pos_arg )\n [ pos_args ...] [--pos_arg2= pos_arg2 ]\n [--pos_arg3= pos_arg3 ]\n\n\nOptions:\n --a_arg : a arg\n --b_arg : b arg\n --pos_arg= pos_arg : pos arg\n --pos_args= pos_args : pos args\n --pos_arg2= pos_arg2 : pos arg 2\n --pos_arg3= pos_arg3 : pos arg 3\n\nReturns:\n pos args",
"text": "This command is only for testing the CLI argument parsing\nUsage:\n cli_test_command [--a_arg] [--b_arg] ( pos_arg | --pos_arg= pos_arg )\n [ pos_args ...] [--pos_arg2= pos_arg2 ]\n [--pos_arg3= pos_arg3 ]\n\n\nOptions:\n --a_arg : (bool) a arg\n --b_arg : (bool) b arg\n --pos_arg= pos_arg : (int) pos arg\n --pos_args= pos_args : (int) pos args\n --pos_arg2= pos_arg2 : (int) pos arg 2\n --pos_arg3= pos_arg3 : (int) pos arg 3\n\nReturns:\n pos args",
"title": "cli_test_command"
},
{
@@ -452,12 +452,12 @@
},
{
"location": "/cli/#peer_ping",
"text": "Find and ping a peer by node id\n\nUsage:\n peer_ping ( node_id | --node_id= node_id )\n\n\nOptions:\n None\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"text": "Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,\nif not provided the peer is located first.\n\nUsage:\n peer_ping ( node_id | --node_id= node_id ) [ address | --address= address ] [ port | --port= port ]\n\n\nOptions:\n --address= address : (str) ip address of the peer\n --port= port : (int) udp port of the peer\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"title": "peer_ping"
},
{
"location": "/cli/#publish",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nUsage:\n publish ( name | --name= name ) ( bid | --bid= bid ) [--metadata= metadata ]\n [--file_path= file_path ] [--fee= fee ] [--title= title ]\n [--description= description ] [--author= author ] [--language= language ]\n [--license= license ] [--license_url= license_url ] [--thumbnail= thumbnail ]\n [--preview= preview ] [--nsfw= nsfw ] [--sources= sources ]\n [--channel_name= channel_name ] [--channel_id= channel_id ]\n [--claim_address= claim_address ] [--change_address= change_address ]\n\n\nOptions:\n --name= name : (str) name of the content\n --bid= bid : (decimal) amount to back the claim\n --metadata= metadata : (dict) ClaimDict to associate with the claim.\n --file_path= file_path : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n --fee= fee : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n --title= title : (str) title of the publication\n --description= description : (str) description of the publication\n --author= author : (str) author of the publication\n --language= language : (str) language of the publication\n --license= license : (str) publication license\n --license_url= license_url : (str) publication license url\n --thumbnail= thumbnail : (str) thumbnail url\n --preview= preview : (str) preview url\n --nsfw= nsfw : (bool) title of the publication\n --sources= sources : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n --channel_name= channel_name : (str) name of the publisher channel name in the wallet\n --channel_id= channel_id : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n --claim_address= claim_address : (str) address where the claim is sent to, if not specified\n new address wil automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nUsage:\n publish ( name | --name= name ) ( bid | --bid= bid ) [--metadata= metadata ]\n [--file_path= file_path ] [--fee= fee ] [--title= title ]\n [--description= description ] [--author= author ] [--language= language ]\n [--license= license ] [--license_url= license_url ] [--thumbnail= thumbnail ]\n [--preview= preview ] [--nsfw= nsfw ] [--sources= sources ]\n [--channel_name= channel_name ] [--channel_id= channel_id ]\n [--claim_address= claim_address ] [--change_address= change_address ]\n\n\nOptions:\n --name= name : (str) name of the content\n --bid= bid : (decimal) amount to back the claim\n --metadata= metadata : (dict) ClaimDict to associate with the claim.\n --file_path= file_path : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n --fee= fee : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n --title= title : (str) title of the publication\n --description= description : (str) description of the publication\n --author= author : (str) author of the publication\n --language= language : (str) language of the publication\n --license= license : (str) publication license\n --license_url= license_url : (str) publication license url\n --thumbnail= thumbnail : (str) thumbnail url\n --preview= preview : (str) preview url\n --nsfw= nsfw : (bool) whether the content is nsfw\n --sources= sources : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n --channel_name= channel_name : (str) name of the publisher channel name in the wallet\n --channel_id= channel_id : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n --claim_address= claim_address : (str) address where the claim is sent to, if not specified\n new address will automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"title": "publish"
},
{
@@ -492,7 +492,7 @@
},
{
"location": "/cli/#status",
"text": "Get daemon status\n\nUsage:\n status [--session_status]\n\n\nOptions:\n --session_status : (bool) include session status in results\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'lbry_id': lbry peer id, base58,\n 'installation_id': installation id, base58,\n 'is_running': bool,\n 'is_first_run': bool,\n 'startup_status': {\n 'code': status code,\n 'message': status message\n },\n 'connection_status': {\n 'code': connection status code,\n 'message': connection status message\n },\n 'blockchain_status': {\n 'blocks': local blockchain height,\n 'blocks_behind': remote_height - local_height,\n 'best_blockhash': block hash of most recent block,\n },\n 'wallet_is_encrypted': bool,\n\n If given the session status option:\n 'session_status': {\n 'managed_blobs': count of blobs in the blob manager,\n 'managed_streams': count of streams in the file manager\n 'announce_queue_size': number of blobs currently queued to be announced\n 'should_announce_blobs': number of blobs that should be announced\n }\n }",
"text": "Get daemon status\n\nUsage:\n status\n\n\nOptions:\n None\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'is_first_run': bool,\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'database': (bool),\n 'wallet': (bool),\n 'session': (bool),\n 'dht': (bool),\n 'hash_announcer': (bool),\n 'stream_identifier': (bool),\n 'file_manager': (bool),\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'peer_protocol_server': (bool),\n 'reflector': (bool),\n 'upnp': (bool),\n 'exchange_rate_manager': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the file manager,\n }\n }",
"title": "status"
},
{
@@ -567,7 +567,7 @@
},
{
"location": "/cli/#wallet_send",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nUsage:\n wallet_send ( amount | --amount= amount )\n (( address | --address= address ) | ( claim_id | --claim_id= claim_id ))\n\n\nOptions:\n --amount= amount : (float) amount of credit to send\n --address= address : (str) address to send credits to\n --claim_id= claim_id : (float) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nUsage:\n wallet_send ( amount | --amount= amount )\n (( address | --address= address ) | ( claim_id | --claim_id= claim_id ))\n\n\nOptions:\n --amount= amount : (Decimal) amount of credit to send\n --address= address : (str) address to send credits to\n --claim_id= claim_id : (str) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"title": "wallet_send"
},
{
+23
View File
@@ -0,0 +1,23 @@
# This is an example daemon_settings.yml file.
# See https://lbry.tech/resources/daemon-settings for all configuration keys and values
share_usage_data: True
lbryum_servers:
- lbryumx1.lbry.io:50001
- lbryumx2.lbry.io:50001
blockchain_name: lbrycrd_main
data_dir: /home/lbry/.lbrynet
download_directory: /home/lbry/downloads
delete_blobs_on_remove: True
dht_node_port: 4444
peer_port: 3333
use_upnp: True
#components_to_skip:
# - peer_protocol_server
# - hash_announcer
# - dht
File renamed without changes.

Before

Width:  |  Height:  |  Size: 7.4 KiB

After

Width:  |  Height:  |  Size: 7.4 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 19 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.2 KiB

After

Width:  |  Height:  |  Size: 1.2 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 2.6 KiB

After

Width:  |  Height:  |  Size: 2.6 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 6.1 KiB

After

Width:  |  Height:  |  Size: 6.1 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 97 KiB

After

Width:  |  Height:  |  Size: 97 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 1.1 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 361 KiB

After

Width:  |  Height:  |  Size: 361 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 5.3 KiB

After

Width:  |  Height:  |  Size: 5.3 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 15 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 31 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.6 KiB

+2 -4
View File
@@ -1,6 +1,4 @@
import logging
__version__ = "0.20.2"
__name__ = "lbrynet"
__version__ = "0.30.5"
version = tuple(__version__.split('.'))
logging.getLogger(__name__).addHandler(logging.NullHandler())
-274
View File
@@ -1,274 +0,0 @@
import collections
import logging
import treq
from twisted.internet import defer, task
from lbrynet import conf
from lbrynet.core import looping_call_manager, utils, system_info
# Things We Track
SERVER_STARTUP = 'Server Startup'
SERVER_STARTUP_SUCCESS = 'Server Startup Success'
SERVER_STARTUP_ERROR = 'Server Startup Error'
DOWNLOAD_STARTED = 'Download Started'
DOWNLOAD_ERRORED = 'Download Errored'
DOWNLOAD_FINISHED = 'Download Finished'
HEARTBEAT = 'Heartbeat'
CLAIM_ACTION = 'Claim Action' # publish/create/update/abandon
NEW_CHANNEL = 'New Channel'
CREDITS_SENT = 'Credits Sent'
BLOB_BYTES_UPLOADED = 'Blob Bytes Uploaded'
log = logging.getLogger(__name__)
class Manager(object):
def __init__(self, analytics_api, context=None, installation_id=None, session_id=None):
self.analytics_api = analytics_api
self._tracked_data = collections.defaultdict(list)
self.looping_call_manager = self._setup_looping_calls()
self.context = context or self._make_context(
system_info.get_platform(), conf.settings['wallet'])
self.installation_id = installation_id or conf.settings.installation_id
self.session_id = session_id or conf.settings.get_session_id()
self.is_started = False
@classmethod
def new_instance(cls, enabled=None):
api = Api.new_instance(enabled)
return cls(api)
# Things We Track
def send_server_startup(self):
self.analytics_api.track(self._event(SERVER_STARTUP))
def send_server_startup_success(self):
self.analytics_api.track(self._event(SERVER_STARTUP_SUCCESS))
def send_server_startup_error(self, message):
self.analytics_api.track(self._event(SERVER_STARTUP_ERROR, {'message': message}))
def send_download_started(self, id_, name, claim_dict=None):
self.analytics_api.track(
self._event(DOWNLOAD_STARTED, self._download_properties(id_, name, claim_dict))
)
def send_download_errored(self, err, id_, name, claim_dict, report):
download_error_properties = self._download_error_properties(err, id_, name, claim_dict,
report)
self.analytics_api.track(self._event(DOWNLOAD_ERRORED, download_error_properties))
def send_download_finished(self, id_, name, report, claim_dict=None):
download_properties = self._download_properties(id_, name, claim_dict, report)
self.analytics_api.track(self._event(DOWNLOAD_FINISHED, download_properties))
def send_claim_action(self, action):
self.analytics_api.track(self._event(CLAIM_ACTION, {'action': action}))
def send_new_channel(self):
self.analytics_api.track(self._event(NEW_CHANNEL))
def send_credits_sent(self):
self.analytics_api.track(self._event(CREDITS_SENT))
def _send_heartbeat(self):
self.analytics_api.track(self._event(HEARTBEAT))
def _update_tracked_metrics(self):
should_send, value = self.summarize_and_reset(BLOB_BYTES_UPLOADED)
if should_send:
self.analytics_api.track(self._metric_event(BLOB_BYTES_UPLOADED, value))
# Setup / Shutdown
def start(self):
if not self.is_started:
for name, _, interval in self._get_looping_calls():
self.looping_call_manager.start(name, interval)
self.is_started = True
def shutdown(self):
self.looping_call_manager.shutdown()
def register_repeating_metric(self, event_name, value_generator, frequency=300):
lcall = task.LoopingCall(self._send_repeating_metric, event_name, value_generator)
self.looping_call_manager.register_looping_call(event_name, lcall)
lcall.start(frequency)
def _get_looping_calls(self):
return [
('send_heartbeat', self._send_heartbeat, 60),
('update_tracked_metrics', self._update_tracked_metrics, 300),
]
def _setup_looping_calls(self):
call_manager = looping_call_manager.LoopingCallManager()
for name, fn, _ in self._get_looping_calls():
call_manager.register_looping_call(name, task.LoopingCall(fn))
return call_manager
def _send_repeating_metric(self, event_name, value_generator):
result = value_generator()
self._if_deferred(result, self._send_repeating_metric_value, event_name)
def _send_repeating_metric_value(self, result, event_name):
should_send, value = result
if should_send:
self.analytics_api.track(self._metric_event(event_name, value))
def add_observation(self, metric, value):
self._tracked_data[metric].append(value)
def summarize_and_reset(self, metric, op=sum):
"""Apply `op` on the current values for `metric`.
This operation also resets the metric.
Returns:
a tuple (should_send, value)
"""
try:
values = self._tracked_data.pop(metric)
return True, op(values)
except KeyError:
return False, None
def _event(self, event, event_properties=None):
return {
'userId': 'lbry',
'event': event,
'properties': self._event_properties(event_properties),
'context': self.context,
'timestamp': utils.isonow()
}
def _metric_event(self, metric_name, value):
return self._event(metric_name, {'value': value})
def _event_properties(self, event_properties=None):
properties = {
'lbry_id': self.installation_id,
'session_id': self.session_id,
}
properties.update(event_properties or {})
return properties
@staticmethod
def _download_properties(id_, name, claim_dict=None, report=None):
sd_hash = None if not claim_dict else claim_dict.source_hash
p = {
'download_id': id_,
'name': name,
'stream_info': sd_hash
}
if report:
p['report'] = report
return p
@staticmethod
def _download_error_properties(error, id_, name, claim_dict, report):
def error_name(err):
if not hasattr(type(err), "__name__"):
return str(type(err))
return type(err).__name__
return {
'download_id': id_,
'name': name,
'stream_info': claim_dict.source_hash,
'error': error_name(error),
'reason': error.message,
'report': report
}
@staticmethod
def _make_context(platform, wallet):
context = {
'app': {
'name': 'lbrynet',
'version': platform['lbrynet_version'],
'python_version': platform['python_version'],
'build': platform['build'],
'wallet': {
'name': wallet,
'version': platform['lbryum_version'] if wallet == conf.LBRYUM_WALLET else None
},
},
# TODO: expand os info to give linux/osx specific info
'os': {
'name': platform['os_system'],
'version': platform['os_release']
},
'library': {
'name': 'lbrynet-analytics',
'version': '1.0.0'
},
}
if 'desktop' in platform and 'distro' in platform:
context['os']['desktop'] = platform['desktop']
context['os']['distro'] = platform['distro']
return context
@staticmethod
def _if_deferred(maybe_deferred, callback, *args, **kwargs):
if isinstance(maybe_deferred, defer.Deferred):
maybe_deferred.addCallback(callback, *args, **kwargs)
else:
callback(maybe_deferred, *args, **kwargs)
class Api(object):
def __init__(self, cookies, url, write_key, enabled):
self.cookies = cookies
self.url = url
self._write_key = write_key
self._enabled = enabled
def _post(self, endpoint, data):
# there is an issue with a timing condition with keep-alive
# that is best explained here: https://github.com/mikem23/keepalive-race
#
# If you make a request, wait just the right amount of time,
# then make another request, the requests module may opt to
# reuse the connection, but by the time the server gets it the
# timeout will have expired.
#
# by forcing the connection to close, we will disable the keep-alive.
def update_cookies(response):
self.cookies.update(response.cookies())
return response
assert endpoint[0] == '/'
headers = {b"Connection": b"close"}
d = treq.post(self.url + endpoint, auth=(self._write_key, ''), json=data,
headers=headers, cookies=self.cookies)
d.addCallback(update_cookies)
return d
def track(self, event):
"""Send a single tracking event"""
if not self._enabled:
return defer.succeed('Analytics disabled')
def _log_error(failure, event):
log.warning('Failed to send track event. %s (%s)', failure.getTraceback(), str(event))
log.debug('Sending track event: %s', event)
d = self._post('/track', event)
d.addErrback(_log_error, event)
return d
@classmethod
def new_instance(cls, enabled=None):
"""Initialize an instance using values from the configuration"""
if enabled is None:
enabled = conf.settings['share_usage_data']
return cls(
{},
conf.settings['ANALYTICS_ENDPOINT'],
utils.deobfuscate(conf.settings['ANALYTICS_TOKEN']),
enabled,
)
-1
View File
@@ -1 +0,0 @@
import paths
-26
View File
@@ -1,26 +0,0 @@
# App root files dir
# https://developer.android.com/reference/android/content/ContextWrapper.html#getFilesDir%28%29
def android_files_dir():
return None
# Base internal storage path
def android_internal_storage_dir():
return None
# Base external (SD card, if present) storage path
def android_external_storage_dir():
return None
# Internal device storage (private app folder)
# https://developer.android.com/reference/android/content/ContextWrapper.html#getExternalFilesDirs(java.lang.String)
def android_app_internal_storage_dir():
return None
# External (app folder on SD card, if present) storage
# https://developer.android.com/reference/android/content/ContextWrapper.html#getExternalFilesDirs(java.lang.String)
def android_app_external_storage_dir():
return None
@@ -7,7 +7,7 @@ from cryptography.hazmat.primitives.ciphers import Cipher, modes
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from cryptography.hazmat.primitives.padding import PKCS7
from cryptography.hazmat.backends import default_backend
from lbrynet.core.BlobInfo import BlobInfo
from lbrynet.p2p.BlobInfo import BlobInfo
from lbrynet.blob.blob_file import MAX_BLOB_SIZE
log = logging.getLogger(__name__)
@@ -16,21 +16,21 @@ backend = default_backend()
class CryptBlobInfo(BlobInfo):
def __init__(self, blob_hash, blob_num, length, iv):
BlobInfo.__init__(self, blob_hash, blob_num, length)
super().__init__(blob_hash, blob_num, length)
self.iv = iv
def get_dict(self):
info = {
"blob_num": self.blob_num,
"length": self.length,
"iv": self.iv
"iv": self.iv.decode()
}
if self.blob_hash:
info['blob_hash'] = self.blob_hash
return info
class StreamBlobDecryptor(object):
class StreamBlobDecryptor:
def __init__(self, blob, key, iv, length):
"""
This class decrypts blob
@@ -53,10 +53,10 @@ class StreamBlobDecryptor(object):
def decrypt(self, write_func):
"""
Decrypt blob and write its content useing write_func
Decrypt blob and write its content using write_func
write_func - function that takes decrypted string as
arugment and writes it somewhere
argument and writes it somewhere
Returns:
@@ -68,14 +68,14 @@ class StreamBlobDecryptor(object):
def write_bytes():
if self.len_read < self.length:
num_bytes_to_decrypt = greatest_multiple(len(self.buff), (AES.block_size / 8))
num_bytes_to_decrypt = greatest_multiple(len(self.buff), (AES.block_size // 8))
data_to_decrypt, self.buff = split(self.buff, num_bytes_to_decrypt)
write_func(self.cipher.update(data_to_decrypt))
def finish_decrypt():
bytes_left = len(self.buff) % (AES.block_size / 8)
bytes_left = len(self.buff) % (AES.block_size // 8)
if bytes_left != 0:
log.warning(self.buff[-1 * (AES.block_size / 8):].encode('hex'))
log.warning(binascii.hexlify(self.buff[-1 * (AES.block_size // 8):]).decode())
raise Exception("blob %s has incorrect padding: %i bytes left" %
(self.blob.blob_hash, bytes_left))
data_to_decrypt, self.buff = self.buff, b''
@@ -99,7 +99,7 @@ class StreamBlobDecryptor(object):
return d
class CryptStreamBlobMaker(object):
class CryptStreamBlobMaker:
def __init__(self, key, iv, blob_num, blob):
"""
This class encrypts data and writes it to a new blob
@@ -146,7 +146,7 @@ class CryptStreamBlobMaker(object):
def close(self):
log.debug("closing blob %s with plaintext len %s", str(self.blob_num), str(self.length))
if self.length != 0:
self.length += (AES.block_size / 8) - (self.length % (AES.block_size / 8))
self.length += (AES.block_size // 8) - (self.length % (AES.block_size // 8))
padded_data = self.padder.finalize()
encrypted_data = self.cipher.update(padded_data) + self.cipher.finalize()
self.blob.write(encrypted_data)
@@ -5,15 +5,14 @@ import os
import logging
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from twisted.internet import interfaces, defer
from zope.interface import implements
from lbrynet.cryptstream.CryptBlob import CryptStreamBlobMaker
from twisted.internet import defer
from lbrynet.blob.CryptBlob import CryptStreamBlobMaker
log = logging.getLogger(__name__)
class CryptStreamCreator(object):
class CryptStreamCreator:
"""
Create a new stream with blobs encrypted by a symmetric cipher.
@@ -22,7 +21,7 @@ class CryptStreamCreator(object):
the blob is associated with the stream.
"""
implements(interfaces.IConsumer)
#implements(interfaces.IConsumer)
def __init__(self, blob_manager, name=None, key=None, iv_generator=None):
"""@param blob_manager: Object that stores and provides access to blobs.
@@ -101,13 +100,13 @@ class CryptStreamCreator(object):
@staticmethod
def random_iv_generator():
while 1:
yield os.urandom(AES.block_size / 8)
yield os.urandom(AES.block_size // 8)
def setup(self):
"""Create the symmetric key if it wasn't provided"""
if self.key is None:
self.key = os.urandom(AES.block_size / 8)
self.key = os.urandom(AES.block_size // 8)
return defer.succeed(True)
@@ -122,7 +121,7 @@ class CryptStreamCreator(object):
yield defer.DeferredList(self.finished_deferreds)
self.blob_count += 1
iv = self.iv_generator.next()
iv = next(self.iv_generator)
final_blob = self._get_blob_maker(iv, self.blob_manager.get_blob_creator())
stream_terminator = yield final_blob.close()
terminator_info = yield self._blob_finished(stream_terminator)
@@ -133,7 +132,7 @@ class CryptStreamCreator(object):
if self.current_blob is None:
self.next_blob_creator = self.blob_manager.get_blob_creator()
self.blob_count += 1
iv = self.iv_generator.next()
iv = next(self.iv_generator)
self.current_blob = self._get_blob_maker(iv, self.next_blob_creator)
done, num_bytes_written = self.current_blob.write(data)
data = data[num_bytes_written:]
@@ -2,16 +2,17 @@
Utilities for turning plain files into LBRY Files.
"""
import binascii
import logging
import os
import logging
from binascii import hexlify
from twisted.internet import defer
from twisted.protocols.basic import FileSender
from lbrynet.core.StreamDescriptor import BlobStreamDescriptorWriter, EncryptedFileStreamType
from lbrynet.core.StreamDescriptor import format_sd_info, get_stream_hash, validate_descriptor
from lbrynet.cryptstream.CryptStreamCreator import CryptStreamCreator
from lbrynet.extras.compat import f2d
from lbrynet.p2p.StreamDescriptor import BlobStreamDescriptorWriter, EncryptedFileStreamType
from lbrynet.p2p.StreamDescriptor import format_sd_info, get_stream_hash, validate_descriptor
from lbrynet.blob.CryptStreamCreator import CryptStreamCreator
log = logging.getLogger(__name__)
@@ -23,7 +24,7 @@ class EncryptedFileStreamCreator(CryptStreamCreator):
def __init__(self, blob_manager, lbry_file_manager, stream_name=None,
key=None, iv_generator=None):
CryptStreamCreator.__init__(self, blob_manager, stream_name, key, iv_generator)
super().__init__(blob_manager, stream_name, key, iv_generator)
self.lbry_file_manager = lbry_file_manager
self.stream_hash = None
self.blob_infos = []
@@ -37,14 +38,14 @@ class EncryptedFileStreamCreator(CryptStreamCreator):
def _finished(self):
# calculate the stream hash
self.stream_hash = get_stream_hash(
hexlify(self.name), hexlify(self.key), hexlify(self.name),
hexlify(self.name.encode()).decode(), hexlify(self.key).decode(), hexlify(self.name.encode()).decode(),
self.blob_infos
)
# generate the sd info
self.sd_info = format_sd_info(
EncryptedFileStreamType, hexlify(self.name), hexlify(self.key),
hexlify(self.name), self.stream_hash, self.blob_infos
EncryptedFileStreamType, hexlify(self.name.encode()).decode(), hexlify(self.key).decode(),
hexlify(self.name.encode()).decode(), self.stream_hash, self.blob_infos
)
# sanity check
@@ -59,7 +60,8 @@ class EncryptedFileStreamCreator(CryptStreamCreator):
# we can simply read the file from the disk without needing to
# involve reactor.
@defer.inlineCallbacks
def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=None, iv_generator=None):
def create_lbry_file(blob_manager, storage, payment_rate_manager, lbry_file_manager, file_name, file_handle,
key=None, iv_generator=None):
"""Turn a plain file into an LBRY File.
An LBRY File is a collection of encrypted blobs of data and the metadata that binds them
@@ -98,7 +100,7 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
file_directory = os.path.dirname(file_handle.name)
lbry_file_creator = EncryptedFileStreamCreator(
session.blob_manager, lbry_file_manager, base_file_name, key, iv_generator
blob_manager, lbry_file_manager, base_file_name, key, iv_generator
)
yield lbry_file_creator.setup()
@@ -114,25 +116,17 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
log.debug("making the sd blob")
sd_info = lbry_file_creator.sd_info
descriptor_writer = BlobStreamDescriptorWriter(session.blob_manager)
descriptor_writer = BlobStreamDescriptorWriter(blob_manager)
sd_hash = yield descriptor_writer.create_descriptor(sd_info)
log.debug("saving the stream")
yield session.storage.store_stream(
yield f2d(storage.store_stream(
sd_info['stream_hash'], sd_hash, sd_info['stream_name'], sd_info['key'],
sd_info['suggested_file_name'], sd_info['blobs']
)
))
log.debug("adding to the file manager")
lbry_file = yield lbry_file_manager.add_published_file(
sd_info['stream_hash'], sd_hash, binascii.hexlify(file_directory), session.payment_rate_manager,
session.payment_rate_manager.min_blob_data_payment_rate
)
lbry_file = yield f2d(lbry_file_manager.add_published_file(
sd_info['stream_hash'], sd_hash, hexlify(file_directory.encode()), payment_rate_manager,
payment_rate_manager.min_blob_data_payment_rate
))
defer.returnValue(lbry_file)
def hexlify(str_or_unicode):
if isinstance(str_or_unicode, unicode):
strng = str_or_unicode.encode('utf-8')
else:
strng = str_or_unicode
return binascii.hexlify(strng)
@@ -2,18 +2,17 @@
Download LBRY Files from LBRYnet and save them to disk.
"""
import logging
import binascii
from binascii import hexlify, unhexlify
from zope.interface import implements
from twisted.internet import defer
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.utils import short_hash
from lbrynet.lbry_file.client.EncryptedFileDownloader import EncryptedFileSaver
from lbrynet.lbry_file.client.EncryptedFileDownloader import EncryptedFileDownloader
from lbrynet.file_manager.EncryptedFileStatusReport import EncryptedFileStatusReport
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.core.StreamDescriptor import save_sd_info
from lbrynet.conf import Config
from lbrynet.extras.compat import f2d
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.p2p.HTTPBlobDownloader import HTTPBlobDownloader
from lbrynet.utils import short_hash
from lbrynet.blob.client.EncryptedFileDownloader import EncryptedFileSaver
from lbrynet.blob.EncryptedFileStatusReport import EncryptedFileStatusReport
from lbrynet.p2p.StreamDescriptor import save_sd_info
log = logging.getLogger(__name__)
@@ -35,16 +34,16 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
STATUS_STOPPED = "stopped"
STATUS_FINISHED = "finished"
def __init__(self, rowid, stream_hash, peer_finder, rate_limiter, blob_manager, storage, lbry_file_manager,
payment_rate_manager, wallet, download_directory, file_name, stream_name, sd_hash, key,
suggested_file_name):
EncryptedFileSaver.__init__(
self, stream_hash, peer_finder, rate_limiter, blob_manager, storage, payment_rate_manager, wallet,
download_directory, key, stream_name, file_name
def __init__(self, conf: Config, rowid, stream_hash, peer_finder, rate_limiter, blob_manager, storage,
lbry_file_manager, payment_rate_manager, wallet, download_directory, file_name, stream_name,
sd_hash, key, suggested_file_name, download_mirrors=None):
super().__init__(
conf, stream_hash, peer_finder, rate_limiter, blob_manager, storage, payment_rate_manager,
wallet, download_directory, key, stream_name, file_name
)
self.sd_hash = sd_hash
self.rowid = rowid
self.suggested_file_name = binascii.unhexlify(suggested_file_name)
self.suggested_file_name = unhexlify(suggested_file_name).decode()
self.lbry_file_manager = lbry_file_manager
self._saving_status = False
self.claim_id = None
@@ -55,6 +54,11 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
self.channel_claim_id = None
self.channel_name = None
self.metadata = None
self.mirror = None
if download_mirrors or conf.download_mirrors:
self.mirror = HTTPBlobDownloader(
self.blob_manager, servers=download_mirrors or conf.download_mirrors
)
def set_claim_info(self, claim_info):
self.claim_id = claim_info['claim_id']
@@ -66,13 +70,11 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
self.channel_name = claim_info['channel_name']
self.metadata = claim_info['value']['stream']['metadata']
@defer.inlineCallbacks
def get_claim_info(self, include_supports=True):
claim_info = yield self.storage.get_content_claim(self.stream_hash, include_supports)
async def get_claim_info(self, include_supports=True):
claim_info = await self.storage.get_content_claim(self.stream_hash, include_supports)
if claim_info:
self.set_claim_info(claim_info)
defer.returnValue(claim_info)
return claim_info
@property
def saving_status(self):
@@ -89,22 +91,23 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
elif status == ManagedEncryptedFileDownloader.STATUS_FINISHED:
self.completed = True
else:
raise Exception("Unknown status for stream %s: %s" % (self.stream_hash, status))
raise Exception(f"Unknown status for stream {self.stream_hash}: {status}")
@defer.inlineCallbacks
def stop(self, err=None, change_status=True):
log.debug('Stopping download for stream %s', short_hash(self.stream_hash))
if self.mirror:
self.mirror.stop()
# EncryptedFileSaver deletes metadata when it's stopped. We don't want that here.
yield EncryptedFileDownloader.stop(self, err=err)
yield super().stop(err)
if change_status is True:
status = yield self._save_status()
defer.returnValue(status)
defer.returnValue(status)
@defer.inlineCallbacks
def status(self):
blobs = yield self.storage.get_blobs_for_stream(self.stream_hash)
async def status(self):
blobs = await self.storage.get_blobs_for_stream(self.stream_hash)
blob_hashes = [b.blob_hash for b in blobs if b.blob_hash is not None]
completed_blobs = yield self.blob_manager.completed_blobs(blob_hashes)
completed_blobs = self.blob_manager.completed_blobs(blob_hashes)
num_blobs_completed = len(completed_blobs)
num_blobs_known = len(blob_hashes)
@@ -114,15 +117,18 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
status = "stopped"
else:
status = "running"
defer.returnValue(EncryptedFileStatusReport(
return EncryptedFileStatusReport(
self.file_name, num_blobs_completed, num_blobs_known, status
))
)
@defer.inlineCallbacks
def _start(self):
yield EncryptedFileSaver._start(self)
status = yield self._save_status()
log_status(self.sd_hash, status)
if self.mirror:
self.mirror.download_stream(self.stream_hash, self.sd_hash)
defer.returnValue(status)
def _get_finished_deferred_callback_value(self):
@@ -142,7 +148,7 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
status = ManagedEncryptedFileDownloader.STATUS_RUNNING
status = yield self.lbry_file_manager.change_lbry_file_status(self, status)
self._saving_status = False
defer.returnValue(status)
return status
def save_status(self):
return self._save_status()
@@ -152,27 +158,30 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
self.blob_manager, download_manager)
class ManagedEncryptedFileDownloaderFactory(object):
implements(IStreamDownloaderFactory)
class ManagedEncryptedFileDownloaderFactory:
#implements(IStreamDownloaderFactory)
def __init__(self, lbry_file_manager):
def __init__(self, lbry_file_manager, blob_manager):
self.lbry_file_manager = lbry_file_manager
self.blob_manager = blob_manager
def can_download(self, sd_validator):
# TODO: add a sd_validator for non live streams, use it
return True
@defer.inlineCallbacks
def make_downloader(self, metadata, data_rate, payment_rate_manager, download_directory, file_name=None):
stream_hash = yield save_sd_info(self.lbry_file_manager.session.blob_manager,
def make_downloader(self, metadata, data_rate, payment_rate_manager, download_directory, file_name=None,
download_mirrors=None):
stream_hash = yield save_sd_info(self.blob_manager,
metadata.source_blob_hash,
metadata.validator.raw_info)
if file_name:
file_name = binascii.hexlify(file_name)
lbry_file = yield self.lbry_file_manager.add_downloaded_file(
stream_hash, metadata.source_blob_hash, binascii.hexlify(download_directory), payment_rate_manager,
data_rate, file_name=file_name
)
file_name = hexlify(file_name.encode())
hex_download_directory = hexlify(download_directory.encode())
lbry_file = yield f2d(self.lbry_file_manager.add_downloaded_file(
stream_hash, metadata.source_blob_hash, hex_download_directory, payment_rate_manager,
data_rate, file_name=file_name, download_mirrors=download_mirrors
))
defer.returnValue(lbry_file)
@staticmethod
@@ -3,24 +3,25 @@ Keep track of which LBRY Files are downloading and store their LBRY File specifi
"""
import os
import logging
import random
from binascii import hexlify, unhexlify
from twisted.internet import defer, task, reactor
from twisted.python.failure import Failure
from lbrynet.reflector.reupload import reflect_file
# from lbrynet.core.PaymentRateManager import NegotiatedPaymentRateManager
from lbrynet.file_manager.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet.file_manager.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet.core.StreamDescriptor import EncryptedFileStreamType, get_sd_info
from lbrynet.cryptstream.client.CryptStreamDownloader import AlreadyStoppedError
from lbrynet.cryptstream.client.CryptStreamDownloader import CurrentlyStoppingError
from lbrynet.core.utils import safe_start_looping_call, safe_stop_looping_call
from lbrynet import conf
from lbrynet.conf import Config
from lbrynet.extras.compat import f2d
from lbrynet.extras.reflector.reupload import reflect_file
from lbrynet.blob.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet.blob.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamType, get_sd_info
from lbrynet.blob.client.CryptStreamDownloader import AlreadyStoppedError
from lbrynet.blob.client.CryptStreamDownloader import CurrentlyStoppingError
from lbrynet.utils import safe_start_looping_call, safe_stop_looping_call
log = logging.getLogger(__name__)
class EncryptedFileManager(object):
class EncryptedFileManager:
"""
Keeps track of currently opened LBRY Files, their options, and
their LBRY File specific metadata.
@@ -28,33 +29,35 @@ class EncryptedFileManager(object):
# when reflecting files, reflect up to this many files at a time
CONCURRENT_REFLECTS = 5
def __init__(self, session, sd_identifier):
self.auto_re_reflect = conf.settings['reflect_uploads'] and conf.settings['auto_re_reflect_interval'] > 0
self.auto_re_reflect_interval = conf.settings['auto_re_reflect_interval']
self.session = session
self.storage = session.storage
def __init__(self, conf: Config, peer_finder, rate_limiter, blob_manager, wallet,
payment_rate_manager, storage, sd_identifier):
self.conf = conf
self.auto_re_reflect = conf.reflect_uploads and conf.auto_re_reflect_interval > 0
self.auto_re_reflect_interval = conf.auto_re_reflect_interval
self.peer_finder = peer_finder
self.rate_limiter = rate_limiter
self.blob_manager = blob_manager
self.wallet = wallet
self.payment_rate_manager = payment_rate_manager
self.storage = storage
# TODO: why is sd_identifier part of the file manager?
self.sd_identifier = sd_identifier
assert sd_identifier
self.lbry_files = []
self.lbry_file_reflector = task.LoopingCall(self.reflect_lbry_files)
@defer.inlineCallbacks
def setup(self):
yield self._add_to_sd_identifier()
yield self._start_lbry_files()
log.info("Started file manager")
self._add_to_sd_identifier()
return self._start_lbry_files()
def get_lbry_file_status(self, lbry_file):
return self.session.storage.get_lbry_file_status(lbry_file.rowid)
return self.storage.get_lbry_file_status(lbry_file.rowid)
def set_lbry_file_data_payment_rate(self, lbry_file, new_rate):
return self.session.storage(lbry_file.rowid, new_rate)
return self.storage(lbry_file.rowid, new_rate)
def change_lbry_file_status(self, lbry_file, status):
log.debug("Changing status of %s to %s", lbry_file.stream_hash, status)
return self.session.storage.change_file_status(lbry_file.rowid, status)
return f2d(self.storage.change_file_status(lbry_file.rowid, status))
def get_lbry_file_status_reports(self):
ds = []
@@ -71,35 +74,37 @@ class EncryptedFileManager(object):
return dl
def _add_to_sd_identifier(self):
downloader_factory = ManagedEncryptedFileDownloaderFactory(self)
downloader_factory = ManagedEncryptedFileDownloaderFactory(self, self.blob_manager)
self.sd_identifier.add_stream_downloader_factory(
EncryptedFileStreamType, downloader_factory)
def _get_lbry_file(self, rowid, stream_hash, payment_rate_manager, sd_hash, key,
stream_name, file_name, download_directory, suggested_file_name):
stream_name, file_name, download_directory, suggested_file_name, download_mirrors=None):
return ManagedEncryptedFileDownloader(
self.conf,
rowid,
stream_hash,
self.session.peer_finder,
self.session.rate_limiter,
self.session.blob_manager,
self.session.storage,
self.peer_finder,
self.rate_limiter,
self.blob_manager,
self.storage,
self,
payment_rate_manager,
self.session.wallet,
self.wallet,
download_directory,
file_name,
stream_name=stream_name,
sd_hash=sd_hash,
key=key,
suggested_file_name=suggested_file_name
suggested_file_name=suggested_file_name,
download_mirrors=download_mirrors
)
def _start_lbry_file(self, file_info, payment_rate_manager, claim_info):
def _start_lbry_file(self, file_info, payment_rate_manager, claim_info, download_mirrors=None):
lbry_file = self._get_lbry_file(
file_info['row_id'], file_info['stream_hash'], payment_rate_manager, file_info['sd_hash'],
file_info['key'], file_info['stream_name'], file_info['file_name'], file_info['download_directory'],
file_info['suggested_file_name']
file_info['suggested_file_name'], download_mirrors
)
if claim_info:
lbry_file.set_claim_info(claim_info)
@@ -113,11 +118,10 @@ class EncryptedFileManager(object):
except Exception:
log.warning("Failed to start %i", file_info.get('rowid'))
@defer.inlineCallbacks
def _start_lbry_files(self):
files = yield self.session.storage.get_all_lbry_files()
claim_infos = yield self.session.storage.get_claims_from_stream_hashes([file['stream_hash'] for file in files])
prm = self.session.payment_rate_manager
async def _start_lbry_files(self):
files = await self.storage.get_all_lbry_files()
claim_infos = await self.storage.get_claims_from_stream_hashes([file['stream_hash'] for file in files])
prm = self.payment_rate_manager
log.info("Starting %i files", len(files))
for file_info in files:
@@ -144,58 +148,55 @@ class EncryptedFileManager(object):
finally:
defer.returnValue(None)
@defer.inlineCallbacks
def _stop_lbry_files(self):
log.info("Stopping %i lbry files", len(self.lbry_files))
lbry_files = self.lbry_files
for lbry_file in lbry_files:
yield self._stop_lbry_file(lbry_file)
yield defer.DeferredList([self._stop_lbry_file(lbry_file) for lbry_file in list(self.lbry_files)])
@defer.inlineCallbacks
def add_published_file(self, stream_hash, sd_hash, download_directory, payment_rate_manager, blob_data_rate):
async def add_published_file(self, stream_hash, sd_hash, download_directory, payment_rate_manager, blob_data_rate):
status = ManagedEncryptedFileDownloader.STATUS_FINISHED
stream_metadata = yield get_sd_info(self.session.storage, stream_hash, include_blobs=False)
stream_metadata = await get_sd_info(self.storage, stream_hash, include_blobs=False)
key = stream_metadata['key']
stream_name = stream_metadata['stream_name']
file_name = stream_metadata['suggested_file_name']
rowid = yield self.storage.save_published_file(
rowid = await self.storage.save_published_file(
stream_hash, file_name, download_directory, blob_data_rate, status
)
lbry_file = self._get_lbry_file(
rowid, stream_hash, payment_rate_manager, sd_hash, key, stream_name, file_name, download_directory,
stream_metadata['suggested_file_name']
stream_metadata['suggested_file_name'], download_mirrors=None
)
lbry_file.restore(status)
yield lbry_file.get_claim_info()
await lbry_file.get_claim_info()
self.storage.content_claim_callbacks[stream_hash] = lbry_file.get_claim_info
self.lbry_files.append(lbry_file)
defer.returnValue(lbry_file)
return lbry_file
@defer.inlineCallbacks
def add_downloaded_file(self, stream_hash, sd_hash, download_directory, payment_rate_manager=None,
blob_data_rate=None, status=None, file_name=None):
async def add_downloaded_file(self, stream_hash, sd_hash, download_directory, payment_rate_manager=None,
blob_data_rate=None, status=None, file_name=None, download_mirrors=None):
status = status or ManagedEncryptedFileDownloader.STATUS_STOPPED
payment_rate_manager = payment_rate_manager or self.session.payment_rate_manager
payment_rate_manager = payment_rate_manager or self.payment_rate_manager
blob_data_rate = blob_data_rate or payment_rate_manager.min_blob_data_payment_rate
stream_metadata = yield get_sd_info(self.session.storage, stream_hash, include_blobs=False)
stream_metadata = await get_sd_info(self.storage, stream_hash, include_blobs=False)
key = stream_metadata['key']
stream_name = stream_metadata['stream_name']
file_name = file_name or stream_metadata['suggested_file_name']
# when we save the file we'll atomic touch the nearest file to the suggested file name
# that doesn't yet exist in the download directory
rowid = yield self.storage.save_downloaded_file(
stream_hash, os.path.basename(file_name.decode('hex')).encode('hex'), download_directory, blob_data_rate
rowid = await self.storage.save_downloaded_file(
stream_hash, hexlify(os.path.basename(unhexlify(file_name))), download_directory, blob_data_rate
)
file_name = yield self.session.storage.get_filename_for_rowid(rowid)
file_name = (await self.storage.get_filename_for_rowid(rowid)).decode()
lbry_file = self._get_lbry_file(
rowid, stream_hash, payment_rate_manager, sd_hash, key, stream_name, file_name, download_directory,
stream_metadata['suggested_file_name']
stream_metadata['suggested_file_name'], download_mirrors
)
lbry_file.restore(status)
yield lbry_file.get_claim_info(include_supports=False)
await lbry_file.get_claim_info(include_supports=False)
self.storage.content_claim_callbacks[stream_hash] = lbry_file.get_claim_info
self.lbry_files.append(lbry_file)
defer.returnValue(lbry_file)
return lbry_file
@defer.inlineCallbacks
def delete_lbry_file(self, lbry_file, delete_file=False):
@@ -221,7 +222,7 @@ class EncryptedFileManager(object):
del self.storage.content_claim_callbacks[lbry_file.stream_hash]
yield lbry_file.delete_data()
yield self.session.storage.delete_stream(lbry_file.stream_hash)
yield f2d(self.storage.delete_stream(lbry_file.stream_hash))
if delete_file and os.path.isfile(full_path):
os.remove(full_path)
@@ -239,15 +240,15 @@ class EncryptedFileManager(object):
def reflect_lbry_files(self):
sem = defer.DeferredSemaphore(self.CONCURRENT_REFLECTS)
ds = []
sd_hashes_to_reflect = yield self.storage.get_streams_to_re_reflect()
sd_hashes_to_reflect = yield f2d(self.storage.get_streams_to_re_reflect())
for lbry_file in self.lbry_files:
if lbry_file.sd_hash in sd_hashes_to_reflect:
ds.append(sem.run(reflect_file, lbry_file))
ds.append(sem.run(reflect_file, lbry_file, random.choice(self.conf.reflector_servers)))
yield defer.DeferredList(ds)
@defer.inlineCallbacks
def stop(self):
safe_stop_looping_call(self.lbry_file_reflector)
yield defer.DeferredList(list(self._stop_lbry_files()))
yield self._stop_lbry_files()
log.info("Stopped encrypted file manager")
defer.returnValue(True)
@@ -1,4 +1,4 @@
class EncryptedFileStatusReport(object):
class EncryptedFileStatusReport:
def __init__(self, name, num_completed, num_known, running_status):
self.name = name
self.num_completed = num_completed
-4
View File
@@ -1,4 +0,0 @@
from blob_file import BlobFile
from creator import BlobFileCreator
from writer import HashBlobWriter
from reader import HashBlobReader
+34 -26
View File
@@ -1,10 +1,10 @@
import logging
import os
from twisted.internet import defer, threads
import logging
from twisted.internet import defer
from twisted.web.client import FileBodyProducer
from twisted.python.failure import Failure
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError, InvalidBlobHashError
from lbrynet.core.utils import is_valid_blobhash
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbrynet.p2p.Error import DownloadCanceledError, InvalidDataError, InvalidBlobHashError
from lbrynet.blob.writer import HashBlobWriter
from lbrynet.blob.reader import HashBlobReader
@@ -12,8 +12,26 @@ log = logging.getLogger(__name__)
MAX_BLOB_SIZE = 2 * 2 ** 20
# digest_size is in bytes, and blob hashes are hex encoded
blobhash_length = get_lbry_hash_obj().digest_size * 2
class BlobFile(object):
def is_valid_hashcharacter(char):
return char in "0123456789abcdef"
def is_valid_blobhash(blobhash):
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: True/False
"""
return len(blobhash) == blobhash_length and all(is_valid_hashcharacter(l) for l in blobhash)
class BlobFile:
"""
A chunk of data available on the network which is specified by a hashsum
@@ -52,7 +70,7 @@ class BlobFile(object):
def open_for_writing(self, peer):
"""
open a blob file to be written by peer, supports concurrent
writers, as long as they are from differnt peers.
writers, as long as they are from different peers.
returns tuple of (writer, finished_deferred)
@@ -60,12 +78,12 @@ class BlobFile(object):
finished_deferred - deferred that is fired when write is finished and returns
a instance of itself as HashBlob
"""
if not peer in self.writers:
if peer not in self.writers:
log.debug("Opening %s to be written by %s", str(self), str(peer))
finished_deferred = defer.Deferred()
writer = HashBlobWriter(self.get_length, self.writer_finished)
self.writers[peer] = (writer, finished_deferred)
return (writer, finished_deferred)
return writer, finished_deferred
log.warning("Tried to download the same file twice simultaneously from the same peer")
return None, None
@@ -93,29 +111,19 @@ class BlobFile(object):
if not self.writers and not self.readers:
self._verified = False
self.saved_verified_blob = False
def delete_from_file_system():
try:
if os.path.isfile(self.file_path):
os.remove(self.file_path)
d = threads.deferToThread(delete_from_file_system)
def log_error(err):
log.warning("An error occurred deleting %s: %s",
str(self.file_path), err.getErrorMessage())
return err
d.addErrback(log_error)
return d
except Exception as e:
log.exception("An error occurred deleting %s:", str(self.file_path), exc_info=e)
else:
return defer.fail(Failure(
ValueError("File is currently being read or written and cannot be deleted")))
raise ValueError("File is currently being read or written and cannot be deleted")
@property
def verified(self):
"""
Protect verified from being modified by other classes.
verified is True if a write to a blob has completed succesfully,
verified is True if a write to a blob has completed successfully,
or a blob has been read to have the same length as specified
in init
"""
@@ -149,7 +157,7 @@ class BlobFile(object):
def writer_finished(self, writer, err=None):
def fire_finished_deferred():
self._verified = True
for p, (w, finished_deferred) in self.writers.items():
for p, (w, finished_deferred) in list(self.writers.items()):
if w == writer:
del self.writers[p]
finished_deferred.callback(self)
@@ -160,7 +168,7 @@ class BlobFile(object):
return False
def errback_finished_deferred(err):
for p, (w, finished_deferred) in self.writers.items():
for p, (w, finished_deferred) in list(self.writers.items()):
if w == writer:
del self.writers[p]
finished_deferred.errback(err)
@@ -182,7 +190,7 @@ class BlobFile(object):
if writer.len_so_far != self.length:
err_string = "blob length is %i vs expected %i" % (writer.len_so_far, self.length)
else:
err_string = "blob hash is %s vs expected %s" % (writer.blob_hash, self.blob_hash)
err_string = f"blob hash is {writer.blob_hash} vs expected {self.blob_hash}"
errback_finished_deferred(Failure(InvalidDataError(err_string)))
d = defer.succeed(None)
else:
@@ -1,12 +1,10 @@
import binascii
from zope.interface import implements
from twisted.internet import defer
from lbrynet.cryptstream.CryptBlob import StreamBlobDecryptor
from lbrynet.interfaces import IBlobHandler
from lbrynet.blob.CryptBlob import StreamBlobDecryptor
class CryptBlobHandler(object):
implements(IBlobHandler)
class CryptBlobHandler:
#implements(IBlobHandler)
def __init__(self, key, write_func):
self.key = key
@@ -1,15 +1,15 @@
import binascii
import logging
from zope.interface import implements
from lbrynet.interfaces import IStreamDownloader
from lbrynet.core.client.BlobRequester import BlobRequester
from lbrynet.core.client.ConnectionManager import ConnectionManager
from lbrynet.core.client.DownloadManager import DownloadManager
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.cryptstream.client.CryptBlobHandler import CryptBlobHandler
from binascii import unhexlify
from twisted.internet import defer
from twisted.python.failure import Failure
from lbrynet.conf import Config
from lbrynet.p2p.client.BlobRequester import BlobRequester
from lbrynet.p2p.client.ConnectionManager import ConnectionManager
from lbrynet.p2p.client.DownloadManager import DownloadManager
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.blob.client.CryptBlobHandler import CryptBlobHandler
log = logging.getLogger(__name__)
@@ -34,11 +34,11 @@ class CurrentlyStartingError(Exception):
pass
class CryptStreamDownloader(object):
class CryptStreamDownloader:
implements(IStreamDownloader)
#implements(IStreamDownloader)
def __init__(self, peer_finder, rate_limiter, blob_manager, payment_rate_manager, wallet,
def __init__(self, conf: Config, peer_finder, rate_limiter, blob_manager, payment_rate_manager, wallet,
key, stream_name):
"""Initialize a CryptStreamDownloader
@@ -56,14 +56,14 @@ class CryptStreamDownloader(object):
@return:
"""
self.conf = conf
self.peer_finder = peer_finder
self.rate_limiter = rate_limiter
self.blob_manager = blob_manager
self.payment_rate_manager = payment_rate_manager
self.wallet = wallet
self.key = binascii.unhexlify(key)
self.stream_name = binascii.unhexlify(stream_name)
self.key = unhexlify(key)
self.stream_name = unhexlify(stream_name).decode()
self.completed = False
self.stopped = True
self.stopping = False
@@ -1,16 +1,16 @@
import binascii
from zope.interface import implements
from lbrynet.core.StreamDescriptor import save_sd_info
from lbrynet.cryptstream.client.CryptStreamDownloader import CryptStreamDownloader
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbry_file.client.EncryptedFileMetadataHandler import EncryptedFileMetadataHandler
import os
from twisted.internet import defer, threads
import logging
import traceback
from binascii import hexlify, unhexlify
from twisted.internet import defer, threads
from lbrynet.conf import Config
from lbrynet.extras.compat import f2d
from lbrynet.p2p.StreamDescriptor import save_sd_info
from lbrynet.blob.client.CryptStreamDownloader import CryptStreamDownloader
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.p2p.Error import FileOpenError
from lbrynet.blob.client.EncryptedFileMetadataHandler import EncryptedFileMetadataHandler
log = logging.getLogger(__name__)
@@ -19,35 +19,45 @@ log = logging.getLogger(__name__)
class EncryptedFileDownloader(CryptStreamDownloader):
"""Classes which inherit from this class download LBRY files"""
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
def __init__(self, conf: Config, stream_hash, peer_finder, rate_limiter, blob_manager,
storage, payment_rate_manager, wallet, key, stream_name, file_name):
CryptStreamDownloader.__init__(self, peer_finder, rate_limiter, blob_manager,
payment_rate_manager, wallet, key, stream_name)
super().__init__(conf, peer_finder, rate_limiter, blob_manager,
payment_rate_manager, wallet, key, stream_name)
self.stream_hash = stream_hash
self.storage = storage
self.file_name = binascii.unhexlify(os.path.basename(file_name))
self.file_name = os.path.basename(unhexlify(file_name).decode())
self._calculated_total_bytes = None
@defer.inlineCallbacks
def delete_data(self):
crypt_infos = yield self.storage.get_blobs_for_stream(self.stream_hash)
crypt_infos = yield f2d(self.storage.get_blobs_for_stream(self.stream_hash))
blob_hashes = [b.blob_hash for b in crypt_infos if b.blob_hash]
sd_hash = yield self.storage.get_sd_blob_hash_for_stream(self.stream_hash)
sd_hash = yield f2d(self.storage.get_sd_blob_hash_for_stream(self.stream_hash))
blob_hashes.append(sd_hash)
yield self.blob_manager.delete_blobs(blob_hashes)
def stop(self, err=None):
d = self._close_output()
d.addCallback(lambda _: CryptStreamDownloader.stop(self, err=err))
return d
self._close_output()
return super().stop(err=err)
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager, download_manager)
def _start(self):
def check_start_succeeded(success):
if success:
self.starting = False
self.stopped = False
self.completed = False
return True
else:
return self._start_failed()
self.download_manager = self._get_download_manager()
d = self._setup_output()
d.addCallback(lambda _: CryptStreamDownloader._start(self))
d.addCallback(lambda _: self.download_manager.start_downloading())
d.addCallbacks(check_start_succeeded)
return d
def _setup_output(self):
@@ -56,14 +66,9 @@ class EncryptedFileDownloader(CryptStreamDownloader):
def _close_output(self):
pass
def get_total_bytes(self):
d = self.storage.get_blobs_for_stream(self.stream_hash)
def calculate_size(blobs):
return sum([b.length for b in blobs])
d.addCallback(calculate_size)
return d
async def get_total_bytes(self):
blobs = await self.storage.get_blobs_for_stream(self.stream_hash)
return sum([b.length for b in blobs])
def get_total_bytes_cached(self):
if self._calculated_total_bytes is None or self._calculated_total_bytes == 0:
@@ -90,10 +95,11 @@ class EncryptedFileDownloader(CryptStreamDownloader):
self.storage, download_manager)
class EncryptedFileDownloaderFactory(object):
implements(IStreamDownloaderFactory)
class EncryptedFileDownloaderFactory:
#implements(IStreamDownloaderFactory)
def __init__(self, peer_finder, rate_limiter, blob_manager, storage, wallet):
def __init__(self, conf: Config, peer_finder, rate_limiter, blob_manager, storage, wallet):
self.conf = conf
self.peer_finder = peer_finder
self.rate_limiter = rate_limiter
self.blob_manager = blob_manager
@@ -126,22 +132,18 @@ class EncryptedFileDownloaderFactory(object):
class EncryptedFileSaver(EncryptedFileDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, storage, payment_rate_manager, wallet,
download_directory, key, stream_name, file_name):
EncryptedFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter,
blob_manager, storage, payment_rate_manager,
wallet, key, stream_name, file_name)
self.download_directory = binascii.unhexlify(download_directory)
self.file_written_to = os.path.join(self.download_directory, binascii.unhexlify(file_name))
def __init__(self, conf: Config, stream_hash, peer_finder, rate_limiter, blob_manager, storage,
payment_rate_manager, wallet, download_directory, key, stream_name, file_name):
super().__init__(conf, stream_hash, peer_finder, rate_limiter,
blob_manager, storage, payment_rate_manager,
wallet, key, stream_name, file_name)
self.download_directory = unhexlify(download_directory).decode()
self.file_written_to = os.path.join(self.download_directory, unhexlify(file_name).decode())
self.file_handle = None
def __str__(self):
return str(self.file_written_to)
def stop(self, err=None):
d = EncryptedFileDownloader.stop(self, err=err)
return d
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager,
@@ -156,7 +158,7 @@ class EncryptedFileSaver(EncryptedFileDownloader):
self.file_written_to = file_written_to
except IOError:
log.error(traceback.format_exc())
raise ValueError(
raise FileOpenError(
"Failed to open %s. Make sure you have permission to save files to that"
" location." % file_written_to
)
@@ -164,15 +166,11 @@ class EncryptedFileSaver(EncryptedFileDownloader):
def _close_output(self):
self.file_handle, file_handle = None, self.file_handle
def close_file():
if file_handle is not None:
name = file_handle.name
file_handle.close()
if self.completed is False:
os.remove(name)
return threads.deferToThread(close_file)
if file_handle is not None:
name = file_handle.name
file_handle.close()
if self.completed is False:
os.remove(name)
def _get_write_func(self):
def write_func(data):
@@ -182,17 +180,18 @@ class EncryptedFileSaver(EncryptedFileDownloader):
class EncryptedFileSaverFactory(EncryptedFileDownloaderFactory):
def __init__(self, peer_finder, rate_limiter, blob_manager, storage, wallet, download_directory):
EncryptedFileDownloaderFactory.__init__(self, peer_finder, rate_limiter, blob_manager, storage, wallet)
self.download_directory = binascii.hexlify(download_directory)
def __init__(self, conf: Config, peer_finder, rate_limiter, blob_manager, storage, wallet, download_directory):
super().__init__(conf, peer_finder, rate_limiter, blob_manager, storage, wallet)
self.download_directory = hexlify(download_directory.encode())
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info):
stream_name = stream_info.raw_info['stream_name']
key = stream_info.raw_info['key']
suggested_file_name = stream_info.raw_info['suggested_file_name']
return EncryptedFileSaver(
stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager, self.storage, payment_rate_manager,
self.wallet, self.download_directory, key=key, stream_name=stream_name, file_name=suggested_file_name
self.conf, stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager, self.storage,
payment_rate_manager, self.wallet, self.download_directory, key=key, stream_name=stream_name,
file_name=suggested_file_name
)
@staticmethod
@@ -1,14 +1,12 @@
import logging
from zope.interface import implements
from twisted.internet import defer
from lbrynet.interfaces import IMetadataHandler
from lbrynet.extras.compat import f2d
log = logging.getLogger(__name__)
class EncryptedFileMetadataHandler(object):
implements(IMetadataHandler)
class EncryptedFileMetadataHandler:
def __init__(self, stream_hash, storage, download_manager):
self.stream_hash = stream_hash
@@ -20,9 +18,8 @@ class EncryptedFileMetadataHandler(object):
@defer.inlineCallbacks
def get_initial_blobs(self):
blob_infos = yield self.storage.get_blobs_for_stream(self.stream_hash)
formatted_infos = self._format_initial_blobs_for_download_manager(blob_infos)
defer.returnValue(formatted_infos)
blob_infos = yield f2d(self.storage.get_blobs_for_stream(self.stream_hash))
return self._format_initial_blobs_for_download_manager(blob_infos)
def final_blob_num(self):
return self._final_blob_num
@@ -1,6 +1,6 @@
from lbrynet.core.StreamDescriptor import EncryptedFileStreamType
from lbrynet.core.StreamDescriptor import EncryptedFileStreamDescriptorValidator
from lbrynet.core.DownloadOption import DownloadOption, DownloadOptionChoice
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamType
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamDescriptorValidator
from lbrynet.p2p.DownloadOption import DownloadOption, DownloadOptionChoice
def add_lbry_file_to_sd_identifier(sd_identifier):
@@ -8,7 +8,7 @@ def add_lbry_file_to_sd_identifier(sd_identifier):
EncryptedFileOptions())
class EncryptedFileOptions(object):
class EncryptedFileOptions:
def __init__(self):
pass
File renamed without changes.
+2 -2
View File
@@ -3,12 +3,12 @@ import logging
from io import BytesIO
from twisted.internet import defer
from twisted.web.client import FileBodyProducer
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
class BlobFileCreator(object):
class BlobFileCreator:
"""
This class is used to create blobs on the local filesystem
when we do not know the blob hash beforehand (i.e, when creating
+2 -4
View File
@@ -3,7 +3,7 @@ import logging
log = logging.getLogger(__name__)
class HashBlobReader(object):
class HashBlobReader:
"""
This is a file like reader class that supports
read(size) and close()
@@ -15,7 +15,7 @@ class HashBlobReader(object):
def __del__(self):
if self.finished_cb_d is None:
log.warn("Garbage collection was called, but reader for %s was not closed yet",
log.warning("Garbage collection was called, but reader for %s was not closed yet",
self.read_handle.name)
self.close()
@@ -28,5 +28,3 @@ class HashBlobReader(object):
return
self.read_handle.close()
self.finished_cb_d = self.finished_cb(self)
+5 -5
View File
@@ -1,13 +1,13 @@
import logging
from io import BytesIO
from twisted.python.failure import Failure
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.p2p.Error import DownloadCanceledError, InvalidDataError
from lbrynet.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
class HashBlobWriter(object):
class HashBlobWriter:
def __init__(self, length_getter, finished_cb):
self.write_handle = BytesIO()
self.length_getter = length_getter
@@ -18,7 +18,7 @@ class HashBlobWriter(object):
def __del__(self):
if self.finished_cb_d is None:
log.warn("Garbage collection was called, but writer was not closed yet")
log.warning("Garbage collection was called, but writer was not closed yet")
self.close()
@property
@@ -27,7 +27,7 @@ class HashBlobWriter(object):
def write(self, data):
if self.write_handle is None:
log.exception("writer has already been closed")
log.warning("writer has already been closed")
raise IOError('I/O operation on closed file')
self._hashsum.update(data)
+1 -1
View File
@@ -1,2 +1,2 @@
# dont touch this. CI server changes this during build/deployment
# don't touch this. CI server changes this during build/deployment
BUILD = "dev"
+478 -614
View File
File diff suppressed because it is too large. Load diff
-282
View File
@@ -1,282 +0,0 @@
import logging
import miniupnpc
from twisted.internet import threads, defer
from lbrynet.core.BlobManager import DiskBlobManager
from lbrynet.dht import node, hashannouncer
from lbrynet.database.storage import SQLiteStorage
from lbrynet.core.RateLimiter import RateLimiter
from lbrynet.core.utils import generate_id
from lbrynet.core.PaymentRateManager import BasePaymentRateManager, OnlyFreePaymentsManager
log = logging.getLogger(__name__)
class Session(object):
"""This class manages all important services common to any application that uses the network.
the hash announcer, which informs other peers that this peer is
associated with some hash. Usually, this means this peer has a
blob identified by the hash in question, but it can be used for
other purposes.
the peer finder, which finds peers that are associated with some
hash.
the blob manager, which keeps track of which blobs have been
downloaded and provides access to them,
the rate limiter, which attempts to ensure download and upload
rates stay below a set maximum
upnp, which opens holes in compatible firewalls so that remote
peers can connect to this peer.
"""
def __init__(self, blob_data_payment_rate, db_dir=None, node_id=None, peer_manager=None, dht_node_port=None,
known_dht_nodes=None, peer_finder=None, hash_announcer=None, blob_dir=None, blob_manager=None,
peer_port=None, use_upnp=True, rate_limiter=None, wallet=None, dht_node_class=node.Node,
blob_tracker_class=None, payment_rate_manager_class=None, is_generous=True, external_ip=None,
storage=None):
"""@param blob_data_payment_rate: The default payment rate for blob data
@param db_dir: The directory in which levelDB files should be stored
@param node_id: The unique ID of this node
@param peer_manager: An object which keeps track of all known
peers. If None, a PeerManager will be created
@param dht_node_port: The port on which the dht node should
listen for incoming connections
@param known_dht_nodes: A list of nodes which the dht node
should use to bootstrap into the dht
@param peer_finder: An object which is used to look up peers
that are associated with some hash. If None, a
DHTPeerFinder will be used, which looks for peers in the
distributed hash table.
@param hash_announcer: An object which announces to other
peers that this peer is associated with some hash. If
None, and peer_port is not None, a DHTHashAnnouncer will
be used. If None and peer_port is None, a
DummyHashAnnouncer will be used, which will not actually
announce anything.
@param blob_dir: The directory in which blobs will be
stored. If None and blob_manager is None, blobs will be
stored in memory only.
@param blob_manager: An object which keeps track of downloaded
blobs and provides access to them. If None, and blob_dir
is not None, a DiskBlobManager will be used, with the
given blob_dir. If None and blob_dir is None, a
TempBlobManager will be used, which stores blobs in memory
only.
@param peer_port: The port on which other peers should connect
to this peer
@param use_upnp: Whether or not to try to open a hole in the
firewall so that outside peers can connect to this peer's
peer_port and dht_node_port
@param rate_limiter: An object which keeps track of the amount
of data transferred to and from this peer, and can limit
that rate if desired
@param wallet: An object which will be used to keep track of
expected payments and which will pay peers. If None, a
wallet which uses the Point Trader system will be used,
which is meant for testing only
"""
self.db_dir = db_dir
self.node_id = node_id
self.peer_manager = peer_manager
self.peer_finder = peer_finder
self.hash_announcer = hash_announcer
self.dht_node_port = dht_node_port
self.known_dht_nodes = known_dht_nodes
if self.known_dht_nodes is None:
self.known_dht_nodes = []
self.blob_dir = blob_dir
self.blob_manager = blob_manager
# self.blob_tracker = None
# self.blob_tracker_class = blob_tracker_class or BlobAvailabilityTracker
self.peer_port = peer_port
self.use_upnp = use_upnp
self.rate_limiter = rate_limiter
self.external_ip = external_ip
self.upnp_redirects = []
self.wallet = wallet
self.dht_node_class = dht_node_class
self.dht_node = None
self.base_payment_rate_manager = BasePaymentRateManager(blob_data_payment_rate)
self.payment_rate_manager = OnlyFreePaymentsManager()
# self.payment_rate_manager_class = payment_rate_manager_class or NegotiatedPaymentRateManager
# self.is_generous = is_generous
self.storage = storage or SQLiteStorage(self.db_dir)
def setup(self):
"""Create the blob directory and database if necessary, start all desired services"""
log.debug("Starting session.")
if self.node_id is None:
self.node_id = generate_id()
if self.use_upnp is True:
d = self._try_upnp()
else:
d = defer.succeed(True)
d.addCallback(lambda _: self.storage.setup())
d.addCallback(lambda _: self._setup_dht())
d.addCallback(lambda _: self._setup_other_components())
return d
def shut_down(self):
"""Stop all services"""
log.info('Stopping session.')
ds = []
if self.hash_announcer:
self.hash_announcer.stop()
# if self.blob_tracker is not None:
# ds.append(defer.maybeDeferred(self.blob_tracker.stop))
if self.dht_node is not None:
ds.append(defer.maybeDeferred(self.dht_node.stop))
if self.rate_limiter is not None:
ds.append(defer.maybeDeferred(self.rate_limiter.stop))
if self.wallet is not None:
ds.append(defer.maybeDeferred(self.wallet.stop))
if self.blob_manager is not None:
ds.append(defer.maybeDeferred(self.blob_manager.stop))
if self.use_upnp is True:
ds.append(defer.maybeDeferred(self._unset_upnp))
return defer.DeferredList(ds)
def _try_upnp(self):
log.debug("In _try_upnp")
def get_free_port(upnp, port, protocol):
# returns an existing mapping if it exists
mapping = upnp.getspecificportmapping(port, protocol)
if not mapping:
return port
if upnp.lanaddr == mapping[0]:
return mapping[1]
return get_free_port(upnp, port + 1, protocol)
def get_port_mapping(upnp, port, protocol, description):
# try to map to the requested port, if there is already a mapping use the next external
# port available
if protocol not in ['UDP', 'TCP']:
raise Exception("invalid protocol")
port = get_free_port(upnp, port, protocol)
if isinstance(port, tuple):
log.info("Found existing UPnP redirect %s:%i (%s) to %s:%i, using it",
self.external_ip, port, protocol, upnp.lanaddr, port)
return port
upnp.addportmapping(port, protocol, upnp.lanaddr, port,
description, '')
log.info("Set UPnP redirect %s:%i (%s) to %s:%i", self.external_ip, port,
protocol, upnp.lanaddr, port)
return port
def threaded_try_upnp():
if self.use_upnp is False:
log.debug("Not using upnp")
return False
u = miniupnpc.UPnP()
num_devices_found = u.discover()
if num_devices_found > 0:
u.selectigd()
external_ip = u.externalipaddress()
if external_ip != '0.0.0.0' and not self.external_ip:
# best not to rely on this external ip, the router can be behind layers of NATs
self.external_ip = external_ip
if self.peer_port:
self.peer_port = get_port_mapping(u, self.peer_port, 'TCP', 'LBRY peer port')
self.upnp_redirects.append((self.peer_port, 'TCP'))
if self.dht_node_port:
self.dht_node_port = get_port_mapping(u, self.dht_node_port, 'UDP', 'LBRY DHT port')
self.upnp_redirects.append((self.dht_node_port, 'UDP'))
return True
return False
def upnp_failed(err):
log.warning("UPnP failed. Reason: %s", err.getErrorMessage())
return False
d = threads.deferToThread(threaded_try_upnp)
d.addErrback(upnp_failed)
return d
def _setup_dht(self): # does not block startup, the dht will re-attempt if necessary
self.dht_node = self.dht_node_class(
node_id=self.node_id,
udpPort=self.dht_node_port,
externalIP=self.external_ip,
peerPort=self.peer_port,
peer_manager=self.peer_manager,
peer_finder=self.peer_finder,
)
if not self.hash_announcer:
self.hash_announcer = hashannouncer.DHTHashAnnouncer(self.dht_node, self.storage)
self.peer_manager = self.dht_node.peer_manager
self.peer_finder = self.dht_node.peer_finder
d = self.dht_node.start(self.known_dht_nodes)
d.addCallback(lambda _: log.info("Joined the dht"))
d.addCallback(lambda _: self.hash_announcer.start())
def _setup_other_components(self):
log.debug("Setting up the rest of the components")
if self.rate_limiter is None:
self.rate_limiter = RateLimiter()
if self.blob_manager is None:
if self.blob_dir is None:
raise Exception(
"TempBlobManager is no longer supported, specify BlobManager or db_dir")
else:
self.blob_manager = DiskBlobManager(self.blob_dir, self.storage)
# if self.blob_tracker is None:
# self.blob_tracker = self.blob_tracker_class(
# self.blob_manager, self.dht_node.peer_finder, self.dht_node
# )
# if self.payment_rate_manager is None:
# self.payment_rate_manager = self.payment_rate_manager_class(
# self.base_payment_rate_manager, self.blob_tracker, self.is_generous
# )
self.rate_limiter.start()
d = self.blob_manager.setup()
d.addCallback(lambda _: self.wallet.start())
# d.addCallback(lambda _: self.blob_tracker.start())
return d
def _unset_upnp(self):
log.info("Unsetting upnp for session")
def threaded_unset_upnp():
u = miniupnpc.UPnP()
num_devices_found = u.discover()
if num_devices_found > 0:
u.selectigd()
for port, protocol in self.upnp_redirects:
if u.getspecificportmapping(port, protocol) is None:
log.warning(
"UPnP redirect for %s %d was removed by something else.",
protocol, port)
else:
u.deleteportmapping(port, protocol)
log.info("Removed UPnP redirect for %s %d.", protocol, port)
self.upnp_redirects = []
d = threads.deferToThread(threaded_unset_upnp)
d.addErrback(lambda err: str(err))
return d
File diff suppressed because it is too large. Load diff
-17
View File
@@ -1,17 +0,0 @@
import os
from contextlib import contextmanager
@contextmanager
def get_read_handle(path):
"""
Get os independent read handle for a file
"""
if os.name == "nt":
file_mode = 'rb'
else:
file_mode = 'r'
read_handle = open(path, file_mode)
yield read_handle
read_handle.close()
-380
View File
@@ -1,380 +0,0 @@
import inspect
import json
import logging
import logging.handlers
import os
import sys
import traceback
import treq
from twisted.internet import defer
import twisted.python.log
from lbrynet import __version__ as lbrynet_version, build_type, conf
from lbrynet.core import utils
####
# This code is copied from logging/__init__.py in the python source code
####
#
# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
if hasattr(sys, 'frozen'): # support for py2exe
_srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
_srcfile = __file__[:-4] + '.py'
else:
_srcfile = __file__
_srcfile = os.path.normcase(_srcfile)
#####
TRACE = 5
class HTTPSHandler(logging.Handler):
def __init__(self, url, fqdn=False, localname=None, facility=None, cookies=None):
logging.Handler.__init__(self)
self.url = url
self.fqdn = fqdn
self.localname = localname
self.facility = facility
self.cookies = cookies or {}
def get_full_message(self, record):
if record.exc_info:
return '\n'.join(traceback.format_exception(*record.exc_info))
else:
return record.getMessage()
@defer.inlineCallbacks
def _emit(self, record):
payload = self.format(record)
response = yield treq.post(self.url, data=payload, cookies=self.cookies)
self.cookies.update(response.cookies())
def emit(self, record):
return self._emit(record)
DEFAULT_FORMAT = "%(asctime)s %(levelname)-8s %(name)s:%(lineno)d: %(message)s"
DEFAULT_FORMATTER = logging.Formatter(DEFAULT_FORMAT)
LOGGLY_URL = "https://logs-01.loggly.com/inputs/{token}/tag/{tag}"
def remove_handlers(log, handler_name):
for handler in log.handlers:
if handler.name == handler_name:
log.removeHandler(handler)
def _log_decorator(fn):
"""Configure a logging handler.
`fn` is a function that returns a logging handler. The returned
handler has its log-level set and is attached to the specified
logger or the root logger.
"""
def helper(*args, **kwargs):
log = kwargs.pop('log', logging.getLogger())
level = kwargs.pop('level', logging.INFO)
if not isinstance(level, int):
# despite the name, getLevelName returns
# the numeric level when passed a text level
level = logging.getLevelName(level)
handler = fn(*args, **kwargs)
configure_handler(handler, log, level)
return handler
return helper
def configure_handler(handler, log, level):
if handler.name:
remove_handlers(log, handler.name)
handler.setLevel(level)
log.addHandler(handler)
# need to reduce the logger's level down to the
# handler's level or else the handler won't
# get those messages
if log.level > level:
log.setLevel(level)
return handler
def disable_third_party_loggers():
logging.getLogger('requests').setLevel(logging.CRITICAL)
logging.getLogger('urllib3').setLevel(logging.CRITICAL)
logging.getLogger('BitcoinRPC').setLevel(logging.INFO)
logging.getLogger('lbryum').setLevel(logging.WARNING)
logging.getLogger('twisted').setLevel(logging.CRITICAL)
@_log_decorator
def configure_console(**kwargs):
"""Convenience function to configure a log-handler that outputs to stdout"""
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(DEFAULT_FORMATTER)
handler.name = 'console'
return handler
@_log_decorator
def configure_file_handler(file_name, **kwargs):
"""Convenience function to configure a log-handler that writes to `file_name`"""
handler = logging.handlers.RotatingFileHandler(file_name, maxBytes=2097152, backupCount=5)
handler.setFormatter(DEFAULT_FORMATTER)
handler.name = 'file'
return handler
def get_loggly_url(token=None, version=None):
token = token or utils.deobfuscate(conf.settings['LOGGLY_TOKEN'])
version = version or lbrynet_version
return LOGGLY_URL.format(token=token, tag='lbrynet-' + version)
def configure_loggly_handler():
if build_type.BUILD == 'dev':
return
level = logging.ERROR
handler = get_loggly_handler(level=level, installation_id=conf.settings.installation_id,
session_id=conf.settings.get_session_id())
log = logging.getLogger("lbrynet")
if handler.name:
remove_handlers(log, handler.name)
handler.setLevel(level)
log.addHandler(handler)
# need to reduce the logger's level down to the
# handler's level or else the handler won't
# get those messages
if log.level > level:
log.setLevel(level)
def get_loggly_handler(level, installation_id, session_id):
formatter = JsonFormatter(level=level, installation_id=installation_id, session_id=session_id)
handler = HTTPSHandler(get_loggly_url())
handler.setFormatter(formatter)
handler.name = 'loggly'
return handler
class JsonFormatter(logging.Formatter):
"""Format log records using json serialization"""
def __init__(self, **kwargs):
self.attributes = kwargs
def format(self, record):
data = {
'loggerName': record.name,
'asciTime': self.formatTime(record),
'fileName': record.filename,
'functionName': record.funcName,
'levelNo': record.levelno,
'lineNo': record.lineno,
'levelName': record.levelname,
'message': record.getMessage(),
}
data.update(self.attributes)
if record.exc_info:
data['exc_info'] = self.formatException(record.exc_info)
return json.dumps(data)
####
# This code is copied from logging/__init__.py in the python source code
####
def findCaller(srcfile=None):
"""Returns the filename, line number and function name of the caller"""
srcfile = srcfile or _srcfile
f = inspect.currentframe()
# On some versions of IronPython, currentframe() returns None if
# IronPython isn't run with -X:Frames.
if f is not None:
f = f.f_back
rv = "(unknown file)", 0, "(unknown function)"
while hasattr(f, "f_code"):
co = f.f_code
filename = os.path.normcase(co.co_filename)
# ignore any function calls that are in this file
if filename == srcfile:
f = f.f_back
continue
rv = (filename, f.f_lineno, co.co_name)
break
return rv
###
def failure(failure, log, msg, *args):
"""Log a failure message from a deferred.
Args:
failure: twisted.python.failure.Failure
log: a python logger instance
msg: the message to log. Can use normal logging string interpolation.
the last argument will be set to the error message from the failure.
args: values to substitute into `msg`
"""
args += (failure.getErrorMessage(),)
exc_info = (failure.type, failure.value, failure.getTracebackObject())
log.error(msg, *args, exc_info=exc_info)
def convert_verbose(verbose):
"""Convert the results of the --verbose flag into a list of logger names
if --verbose is not provided, args.verbose will be None and logging
should be at the info level.
if --verbose is provided, but not followed by any arguments, then
args.verbose = [] and debug logging should be enabled for all of lbrynet
along with info logging on lbryum.
if --verbose is provided and followed by arguments, those arguments
will be in a list
"""
if verbose is None:
return []
if verbose == []:
return ['lbrynet', 'lbryum']
return verbose
def configure_logging(file_name, console, verbose=None):
"""Apply the default logging configuration.
Enables two log-handlers at the INFO level: a file logger and a loggly logger.
Optionally turns on a console logger that defaults to the INFO level, with
specified loggers being set to the DEBUG level.
Args:
file_name: the file to which logs should be saved
console: If true, enable a console logger
verbose: a list of loggers to set to debug level.
See `convert_verbose` for more details.
"""
verbose = convert_verbose(verbose)
configure_twisted()
configure_file_handler(file_name)
disable_third_party_loggers()
if console:
# if there are some loggers at the debug level, we need
# to enable the console to allow debug. Otherwise, only
# allow info.
level = 'DEBUG' if verbose else 'INFO'
handler = configure_console(level=level)
if 'lbryum' in verbose:
# TODO: this enables lbryum logging on the other handlers
# too which isn't consistent with how verbose logging
# happens with other loggers. Should change the configuration
# so that its only logging at the INFO level for the console.
logging.getLogger('lbryum').setLevel(logging.INFO)
verbose.remove('lbryum')
if verbose:
handler.addFilter(LoggerNameFilter(verbose))
def configure_twisted():
"""Setup twisted logging to output events to the python stdlib logger"""
# I tried using the new logging api
# https://twistedmatrix.com/documents/current/core/howto/logger.html#compatibility-with-standard-library-logging
# and it simply didn't work
observer = twisted.python.log.PythonLoggingObserver()
observer.start()
class LoggerNameFilter(object):
"""Filter a log record based on its name.
Allows all info level and higher records to pass thru.
Debug records pass if the log record name (or a parent) match
the input list of logger names.
"""
def __init__(self, logger_names):
self.logger_names = logger_names
def filter(self, record):
if record.levelno >= logging.INFO:
return True
name = record.name
while name:
if name in self.logger_names:
return True
name = get_parent(name)
return False
def get_parent(logger_name):
names = logger_name.split('.')
if len(names) == 1:
return ''
names = names[:-1]
return '.'.join(names)
class Logger(logging.Logger):
"""A logger that has an extra `fail` method useful for handling twisted failures."""
def fail(self, callback=None, *args, **kwargs):
"""Returns a function to log a failure from an errback.
The returned function appends the error message and extracts
the traceback from `err`.
Example usage:
d.addErrback(log.fail(), 'This is an error message')
Although odd, making the method call is necessary to extract
out useful filename and line number information; otherwise the
reported values are from inside twisted's deferred handling
code.
Args:
callback: callable to call after making the log. The first argument
will be the `err` from the deferred
args: extra arguments to pass into `callback`
Returns: a function that takes the following arguments:
err: twisted.python.failure.Failure
msg: the message to log, using normal logging string iterpolation.
msg_args: the values to subtitute into `msg`
msg_kwargs: set `level` to change from the default ERROR severity. Other
keywoards are treated as normal log kwargs.
"""
fn, lno, func = findCaller()
def _fail(err, msg, *msg_args, **msg_kwargs):
level = msg_kwargs.pop('level', logging.ERROR)
msg += ": %s"
msg_args += (err.getErrorMessage(),)
exc_info = (err.type, err.value, err.getTracebackObject())
record = self.makeRecord(
self.name, level, fn, lno, msg, msg_args, exc_info, func, msg_kwargs)
self.handle(record)
if callback:
try:
return callback(err, *args, **kwargs)
except Exception:
# log.fail is almost always called within an
# errback. If callback fails and we didn't catch
# the exception we would need to attach a second
# errback to deal with that, which we will almost
# never do and then we end up with an unhandled
# error that will get swallowed by twisted
self.exception('Failed to run callback')
return _fail
def trace(self, msg, *args, **kwargs):
if self.isEnabledFor(TRACE):
self._log(TRACE, msg, args, **kwargs)
logging.setLoggerClass(Logger)
logging.addLevelName(TRACE, 'TRACE')
-54
View File
@@ -1,54 +0,0 @@
import platform
import json
import subprocess
import os
from urllib2 import urlopen, URLError
from lbryschema import __version__ as lbryschema_version
from lbryum import __version__ as LBRYUM_VERSION
from lbrynet import build_type, __version__ as lbrynet_version
from lbrynet.conf import ROOT_DIR
def get_lbrynet_version():
if build_type.BUILD == "dev":
try:
with open(os.devnull, 'w') as devnull:
git_dir = ROOT_DIR + '/.git'
return subprocess.check_output(
['git', '--git-dir='+git_dir, 'describe', '--dirty', '--always'],
stderr=devnull
).strip().lstrip('v')
except (subprocess.CalledProcessError, OSError):
print "failed to get version from git"
return lbrynet_version
def get_platform(get_ip=True):
p = {
"processor": platform.processor(),
"python_version": platform.python_version(),
"platform": platform.platform(),
"os_release": platform.release(),
"os_system": platform.system(),
"lbrynet_version": get_lbrynet_version(),
"lbryum_version": LBRYUM_VERSION,
"lbryschema_version": lbryschema_version,
"build": build_type.BUILD, # CI server sets this during build step
}
if p["os_system"] == "Linux":
import distro
p["distro"] = distro.info()
p["desktop"] = os.environ.get('XDG_CURRENT_DESKTOP', 'Unknown')
# TODO: remove this from get_platform and add a get_external_ip function using treq
if get_ip:
try:
response = json.loads(urlopen("https://api.lbry.io/ip").read())
if not response['success']:
raise URLError("failed to get external ip")
p['ip'] = response['data']['ip']
except (URLError, AssertionError):
p['ip'] = "Could not determine IP"
return p
File renamed without changes.
-8
View File
@@ -1,8 +0,0 @@
"""
Classes and functions for dealing with Crypt Streams.
Crypt Streams are encrypted blobs and metadata tying those blobs together. At least some of the
metadata is generally stored in a Stream Descriptor File, for example containing a public key
used to bind blobs to the stream and a symmetric key used to encrypt the blobs. The list of blobs
may or may not be present.
"""
File diff suppressed because it is too large. Load diff
-246
View File
@@ -1,246 +0,0 @@
import json
import os
import sys
import colorama
from docopt import docopt
from collections import OrderedDict
from lbrynet import conf
from lbrynet.core import utils
from lbrynet.daemon.auth.client import JSONRPCException, LBRYAPIClient, AuthAPIClient
from lbrynet.daemon.Daemon import LOADING_WALLET_CODE, Daemon
from lbrynet.core.system_info import get_platform
from jsonrpc.common import RPCError
from requests.exceptions import ConnectionError
from urllib2 import URLError, HTTPError
from httplib import UNAUTHORIZED
def remove_brackets(key):
if key.startswith("<") and key.endswith(">"):
return str(key[1:-1])
return key
def set_flag_vals(flag_names, parsed_args):
kwargs = OrderedDict()
for key, arg in parsed_args.iteritems():
if arg is None:
continue
elif key.startswith("--"):
if remove_brackets(key[2:]) not in kwargs:
k = remove_brackets(key[2:])
elif key in flag_names:
if remove_brackets(flag_names[key]) not in kwargs:
k = remove_brackets(flag_names[key])
elif remove_brackets(key) not in kwargs:
k = remove_brackets(key)
kwargs[k] = guess_type(arg, k)
return kwargs
def main():
argv = sys.argv[1:]
# check if a config file has been specified. If so, shift
# all the arguments so that the parsing can continue without
# noticing
if len(argv) and argv[0] == "--conf":
if len(argv) < 2:
print_error("No config file specified for --conf option")
print_help()
return
conf.conf_file = argv[1]
argv = argv[2:]
if len(argv):
method, args = argv[0], argv[1:]
else:
print_help()
return
if method in ['help', '--help', '-h']:
if len(args) == 1:
print_help_for_command(args[0])
else:
print_help()
return
elif method in ['version', '--version']:
print utils.json_dumps_pretty(get_platform(get_ip=False))
return
if method not in Daemon.callable_methods:
if method not in Daemon.deprecated_methods:
print_error("\"%s\" is not a valid command." % method)
return
new_method = Daemon.deprecated_methods[method]._new_command
print_error("\"%s\" is deprecated, using \"%s\"." % (method, new_method))
method = new_method
fn = Daemon.callable_methods[method]
if hasattr(fn, "_flags"):
flag_names = fn._flags
else:
flag_names = {}
parsed = docopt(fn.__doc__, args)
kwargs = set_flag_vals(flag_names, parsed)
colorama.init()
conf.initialize_settings()
try:
api = LBRYAPIClient.get_client()
status = api.status()
except (URLError, ConnectionError) as err:
if isinstance(err, HTTPError) and err.code == UNAUTHORIZED:
api = AuthAPIClient.config()
# this can happen if the daemon is using auth with the --http-auth flag
# when the config setting is to not use it
try:
status = api.status()
except:
print_error("Daemon requires authentication, but none was provided.",
suggest_help=False)
return 1
else:
print_error("Could not connect to daemon. Are you sure it's running?",
suggest_help=False)
return 1
status_code = status['startup_status']['code']
if status_code != "started" and method not in Daemon.allowed_during_startup:
print "Daemon is in the process of starting. Please try again in a bit."
message = status['startup_status']['message']
if message:
if (
status['startup_status']['code'] == LOADING_WALLET_CODE
and status['blockchain_status']['blocks_behind'] > 0
):
message += '. Blocks left: ' + str(status['blockchain_status']['blocks_behind'])
print " Status: " + message
return 1
# TODO: check if port is bound. Error if its not
try:
result = api.call(method, kwargs)
if isinstance(result, basestring):
# printing the undumped string is prettier
print result
else:
print utils.json_dumps_pretty(result)
except (RPCError, KeyError, JSONRPCException, HTTPError) as err:
if isinstance(err, HTTPError):
error_body = err.read()
try:
error_data = json.loads(error_body)
except ValueError:
print (
"There was an error, and the response was not valid JSON.\n" +
"Raw JSONRPC response:\n" + error_body
)
return 1
print_error(error_data['error']['message'] + "\n", suggest_help=False)
if 'data' in error_data['error'] and 'traceback' in error_data['error']['data']:
print "Here's the traceback for the error you encountered:"
print "\n".join(error_data['error']['data']['traceback'])
print_help_for_command(method)
elif isinstance(err, RPCError):
print_error(err.msg, suggest_help=False)
# print_help_for_command(method)
else:
print_error("Something went wrong\n", suggest_help=False)
print str(err)
return 1
def guess_type(x, key=None):
if not isinstance(x, (unicode, str)):
return x
if key in ('uri', 'channel_name', 'name', 'file_name', 'download_directory'):
return x
if x in ('true', 'True', 'TRUE'):
return True
if x in ('false', 'False', 'FALSE'):
return False
if '.' in x:
try:
return float(x)
except ValueError:
# not a float
pass
try:
return int(x)
except ValueError:
return x
def print_help_suggestion():
print "See `{} help` for more information.".format(os.path.basename(sys.argv[0]))
def print_error(message, suggest_help=True):
error_style = colorama.Style.BRIGHT + colorama.Fore.RED
print error_style + "ERROR: " + message + colorama.Style.RESET_ALL
if suggest_help:
print_help_suggestion()
def print_help():
print "\n".join([
"NAME",
" lbrynet-cli - LBRY command line client.",
"",
"USAGE",
" lbrynet-cli [--conf <config file>] <command> [<args>]",
"",
"EXAMPLES",
" lbrynet-cli commands # list available commands",
" lbrynet-cli status # get daemon status",
" lbrynet-cli --conf ~/l1.conf status # like above but using ~/l1.conf as config file",
" lbrynet-cli resolve_name what # resolve a name",
" lbrynet-cli help resolve_name # get help for a command",
])
def print_help_for_command(command):
fn = Daemon.callable_methods.get(command)
if fn:
print "Help for %s method:\n%s" % (command, fn.__doc__)
def wrap_list_to_term_width(l, width=None, separator=', ', prefix=''):
if width is None:
try:
_, width = os.popen('stty size', 'r').read().split()
width = int(width)
except:
pass
if not width:
width = 80
lines = []
curr_line = ''
for item in l:
new_line = curr_line + item + separator
if len(new_line) + len(prefix) > width:
lines.append(curr_line)
curr_line = item + separator
else:
curr_line = new_line
lines.append(curr_line)
ret = prefix + ("\n" + prefix).join(lines)
if ret.endswith(separator):
ret = ret[:-len(separator)]
return ret
if __name__ == '__main__':
sys.exit(main())
-245
View File
@@ -1,245 +0,0 @@
# -*- coding: utf-8 -*-
import sys
import code
import argparse
import logging.handlers
from exceptions import SystemExit
from twisted.internet import defer, reactor, threads
from lbrynet import analytics
from lbrynet import conf
from lbrynet.core import utils
from lbrynet.core import log_support
from lbrynet.daemon.DaemonServer import DaemonServer
from lbrynet.daemon.auth.client import LBRYAPIClient
from lbrynet.daemon.Daemon import Daemon
get_client = LBRYAPIClient.get_client
log = logging.getLogger(__name__)
if sys.platform.startswith('darwin') or sys.platform.startswith('linux'):
def color(msg, c="white"):
_colors = {
"normal": (0, 37),
"underlined": (2, 37),
"red": (1, 31),
"green": (1, 32),
"yellow": (1, 33),
"blue": (1, 33),
"magenta": (1, 34),
"cyan": (1, 35),
"white": (1, 36),
"grey": (1, 37)
}
i, j = _colors[c]
return "\033[%i;%i;40m%s\033[0m" % (i, j, msg)
logo = """\
╓▄█▄ç
,▄█▓▓▀▀▀▓▓▓▌▄,
▄▄▓▓▓▀¬ ╙▀█▓▓▓▄▄
,▄█▓▓▀▀ ^▀▀▓▓▓▌▄,
▄█▓▓█▀` ╙▀█▓▓▓▄▄
╓▄▓▓▓▀╙ ▀▀▓▓▓▌▄,
▄█▓▓█▀ ╙▀▓▓
╓▄▓▓▓▀` ▄█▓▓▓▀
▓▓█▀ ,▄▓▓▓▀╙
▓▓m ╟▌▄, ▄█▓▓█▀ ,,╓µ
▓▓m ^▀█▓▓▓▄▄ ╓▄▓▓▓▀╙ █▓▓▓▓▓▀
▓▓Q '▀▀▓▓▓▌▄, ,▄█▓▓█▀ ▄█▓▓▓▓▓▀
▀▓▓▓▌▄, ╙▀█▓▓█▄╗ ╓▄▓▓▓▀ ╓▄▓▓▓▀▀ ▀▀
╙▀█▓▓█▄╗ ^▀▀▓▓▓▌▄▄█▓▓▀▀ ▄█▓▓█▀`
'▀▀▓▓▓▌▄, ╙▀██▀` ╓▄▓▓▓▀╙
╙▀█▓▓█▄╥ ,▄█▓▓▀▀
└▀▀▓▓▓▌▄ ▄▒▓▓▓▀╙
╙▀█▓▓█▓▓▓▀▀
╙▀▀`
"""
else:
def color(msg, c=None):
return msg
logo = """\
'.
++++.
+++,;+++,
:+++ :+++:
+++ ,+++;
'++; .+++'
`+++ `++++
+++. `++++
;+++ ++++
+++ +++
+++: '+
,+++ +++
+++` +++:
`+' ,+++
`+ + +++
`+ +++ '++' :'+++:
`+ ++++ `+++ ++++
`+ ++++ +++. :+++'
`+, ++++ ;+++ +++++
`+++, ++++ +++ +++; +
,+++, ++++ +++: .+++
,+++: '++++++ +++`
,+++: '++ '++'
,+++: `+++
.+++; +++,
.+++; ;+++
.+++; +++
`+++++:
`++
"""
welcometext = """\
For a list of available commands:
>>>help()
To see the documentation for a given command:
>>>help("resolve")
To exit:
>>>exit()
"""
welcome = "{:*^60}\n".format(" Welcome to the lbrynet interactive console! ")
welcome += "\n".join(["{:<60}".format(w) for w in welcometext.splitlines()])
welcome += "\n%s" % ("*" * 60)
welcome = color(welcome, "grey")
banner = color(logo, "green") + color(welcome, "grey")
def get_methods(daemon):
locs = {}
def wrapped(name, fn):
client = get_client()
_fn = getattr(client, name)
_fn.__doc__ = fn.__doc__
return {name: _fn}
for method_name, method in daemon.callable_methods.iteritems():
locs.update(wrapped(method_name, method))
return locs
def run_terminal(callable_methods, started_daemon, quiet=False):
locs = {}
locs.update(callable_methods)
def help(method_name=None):
if not method_name:
print "Available api functions: "
for name in callable_methods:
print "\t%s" % name
return
if method_name not in callable_methods:
print "\"%s\" is not a recognized api function"
return
print callable_methods[method_name].__doc__
return
locs.update({'help': help})
if started_daemon:
def exit(status=None):
if not quiet:
print "Stopping lbrynet-daemon..."
callable_methods['daemon_stop']()
return sys.exit(status)
locs.update({'exit': exit})
else:
def exit(status=None):
try:
reactor.callLater(0, reactor.stop)
except Exception as err:
print "error stopping reactor: ", err
return sys.exit(status)
locs.update({'exit': exit})
code.interact(banner if not quiet else "", local=locs)
@defer.inlineCallbacks
def start_server_and_listen(use_auth, analytics_manager, quiet):
log_support.configure_console()
logging.getLogger("lbrynet").setLevel(logging.CRITICAL)
logging.getLogger("lbryum").setLevel(logging.CRITICAL)
logging.getLogger("requests").setLevel(logging.CRITICAL)
analytics_manager.send_server_startup()
daemon_server = DaemonServer(analytics_manager)
try:
yield daemon_server.start(use_auth)
analytics_manager.send_server_startup_success()
if not quiet:
print "Started lbrynet-daemon!"
defer.returnValue(True)
except Exception as e:
log.exception('Failed to start lbrynet-daemon')
analytics_manager.send_server_startup_error(str(e))
daemon_server.stop()
raise
def threaded_terminal(started_daemon, quiet):
callable_methods = get_methods(Daemon)
d = threads.deferToThread(run_terminal, callable_methods, started_daemon, quiet)
d.addErrback(lambda err: err.trap(SystemExit))
d.addErrback(log.exception)
def start_lbrynet_console(quiet, use_existing_daemon, useauth):
if not utils.check_connection():
print "Not connected to internet, unable to start"
raise Exception("Not connected to internet, unable to start")
if not quiet:
print "Starting lbrynet-console..."
try:
get_client().status()
d = defer.succeed(False)
if not quiet:
print "lbrynet-daemon is already running, connecting to it..."
except:
if not use_existing_daemon:
if not quiet:
print "Starting lbrynet-daemon..."
analytics_manager = analytics.Manager.new_instance()
d = start_server_and_listen(useauth, analytics_manager, quiet)
else:
raise Exception("cannot connect to an existing daemon instance, "
"and set to not start a new one")
d.addCallback(threaded_terminal, quiet)
d.addErrback(log.exception)
def main():
conf.initialize_settings()
parser = argparse.ArgumentParser(description="Launch lbrynet-daemon")
parser.add_argument(
"--use_existing_daemon",
help="Start lbrynet-daemon if it isn't already running",
action="store_true",
default=False,
dest="use_existing_daemon"
)
parser.add_argument(
"--quiet", dest="quiet", action="store_true", default=False
)
parser.add_argument(
"--http-auth", dest="useauth", action="store_true", default=conf.settings['use_auth_http']
)
args = parser.parse_args()
start_lbrynet_console(args.quiet, args.use_existing_daemon, args.useauth)
reactor.run()
if __name__ == "__main__":
main()
-124
View File
@@ -1,124 +0,0 @@
import os
import sys
# Set SSL_CERT_FILE env variable for Twisted SSL verification on Windows
# This needs to happen before anything else
if 'win' in sys.platform:
import certifi
os.environ['SSL_CERT_FILE'] = certifi.where()
from lbrynet.core import log_support
import argparse
import logging.handlers
from twisted.internet import defer, reactor
from jsonrpc.proxy import JSONRPCProxy
from lbrynet import analytics
from lbrynet import conf
from lbrynet.core import utils, system_info
from lbrynet.daemon.DaemonServer import DaemonServer
log = logging.getLogger(__name__)
def test_internet_connection():
return utils.check_connection()
def start():
"""The primary entry point for launching the daemon."""
# postpone loading the config file to after the CLI arguments
# have been parsed, as they may contain an alternate config file location
conf.initialize_settings(load_conf_file=False)
parser = argparse.ArgumentParser(description="Launch lbrynet-daemon")
parser.add_argument(
"--conf",
help="specify an alternative configuration file",
type=str,
default=None
)
parser.add_argument(
"--http-auth", dest="useauth", action="store_true", default=conf.settings['use_auth_http']
)
parser.add_argument(
'--quiet', dest='quiet', action="store_true",
help='Disable all console output.'
)
parser.add_argument(
'--verbose', nargs="*",
help=('Enable debug output. Optionally specify loggers for which debug output '
'should selectively be applied.')
)
parser.add_argument(
'--version', action="store_true",
help='Show daemon version and quit'
)
args = parser.parse_args()
update_settings_from_args(args)
conf.settings.load_conf_file_settings()
if args.version:
version = system_info.get_platform(get_ip=False)
version['installation_id'] = conf.settings.installation_id
print utils.json_dumps_pretty(version)
return
lbrynet_log = conf.settings.get_log_filename()
log_support.configure_logging(lbrynet_log, not args.quiet, args.verbose)
log.debug('Final Settings: %s', conf.settings.get_current_settings_dict())
try:
log.debug('Checking for an existing lbrynet daemon instance')
JSONRPCProxy.from_url(conf.settings.get_api_connection_string()).status()
log.info("lbrynet-daemon is already running")
return
except Exception:
log.debug('No lbrynet instance found, continuing to start')
log.info("Starting lbrynet-daemon from command line")
if test_internet_connection():
analytics_manager = analytics.Manager.new_instance()
start_server_and_listen(analytics_manager)
reactor.run()
else:
log.info("Not connected to internet, unable to start")
def update_settings_from_args(args):
if args.conf:
conf.conf_file = args.conf
if args.useauth:
conf.settings.update({
'use_auth_http': args.useauth,
}, data_types=(conf.TYPE_CLI,))
@defer.inlineCallbacks
def start_server_and_listen(analytics_manager):
"""
Args:
use_auth: set to true to enable http authentication
analytics_manager: to send analytics
"""
analytics_manager.send_server_startup()
daemon_server = DaemonServer(analytics_manager)
try:
yield daemon_server.start(conf.settings['use_auth_http'])
analytics_manager.send_server_startup_success()
except Exception as e:
log.exception('Failed to start lbrynet-daemon')
analytics_manager.send_server_startup_error(str(e))
daemon_server.stop()
if __name__ == "__main__":
start()
-77
View File
@@ -1,77 +0,0 @@
import logging
import os
from twisted.web import server, guard, resource
from twisted.internet import defer, reactor, error
from twisted.cred import portal
from lbrynet import conf
from lbrynet.daemon.Daemon import Daemon
from lbrynet.daemon.auth.auth import PasswordChecker, HttpPasswordRealm
from lbrynet.daemon.auth.util import initialize_api_key_file
log = logging.getLogger(__name__)
class IndexResource(resource.Resource):
def getChild(self, name, request):
request.setHeader('cache-control', 'no-cache, no-store, must-revalidate')
request.setHeader('expires', '0')
return self if name == '' else resource.Resource.getChild(self, name, request)
class DaemonServer(object):
def __init__(self, analytics_manager=None):
self._daemon = None
self.root = None
self.server_port = None
self.analytics_manager = analytics_manager
def _setup_server(self, use_auth):
self.root = IndexResource()
self._daemon = Daemon(self.analytics_manager)
self.root.putChild("", self._daemon)
# TODO: DEPRECATED, remove this and just serve the API at the root
self.root.putChild(conf.settings['API_ADDRESS'], self._daemon)
lbrynet_server = get_site_base(use_auth, self.root)
try:
self.server_port = reactor.listenTCP(
conf.settings['api_port'], lbrynet_server, interface=conf.settings['api_host'])
log.info("lbrynet API listening on TCP %s:%i", conf.settings['api_host'], conf.settings['api_port'])
except error.CannotListenError:
log.info('Daemon already running, exiting app')
raise
return defer.succeed(True)
@defer.inlineCallbacks
def start(self, use_auth):
yield self._setup_server(use_auth)
yield self._daemon.setup()
def stop(self):
if reactor.running:
log.info("Stopping the reactor")
reactor.fireSystemEvent("shutdown")
def get_site_base(use_auth, root):
if use_auth:
log.info("Using authenticated API")
root = create_auth_session(root)
else:
log.info("Using non-authenticated API")
return server.Site(root)
def create_auth_session(root):
pw_path = os.path.join(conf.settings['data_dir'], ".api_keys")
initialize_api_key_file(pw_path)
checker = PasswordChecker.load_file(pw_path)
realm = HttpPasswordRealm(root)
portal_to_realm = portal.Portal(realm, [checker, ])
factory = guard.BasicCredentialFactory('Login to lbrynet api')
_lbrynet_server = guard.HTTPAuthSessionWrapper(portal_to_realm, [factory, ])
return _lbrynet_server
-78
View File
@@ -1,78 +0,0 @@
import logging
import mimetypes
import os
from twisted.internet import defer
from lbrynet.core import file_utils
from lbrynet.file_manager.EncryptedFileCreator import create_lbry_file
log = logging.getLogger(__name__)
class Publisher(object):
def __init__(self, session, lbry_file_manager, wallet, certificate_id):
self.session = session
self.lbry_file_manager = lbry_file_manager
self.wallet = wallet
self.certificate_id = certificate_id
self.lbry_file = None
@defer.inlineCallbacks
def create_and_publish_stream(self, name, bid, claim_dict, file_path, claim_address=None,
change_address=None):
"""Create lbry file and make claim"""
log.info('Starting publish for %s', name)
if not os.path.isfile(file_path):
raise Exception("File {} not found".format(file_path))
if os.path.getsize(file_path) == 0:
raise Exception("Cannot publish empty file {}".format(file_path))
file_name = os.path.basename(file_path)
with file_utils.get_read_handle(file_path) as read_handle:
self.lbry_file = yield create_lbry_file(self.session, self.lbry_file_manager, file_name,
read_handle)
if 'source' not in claim_dict['stream']:
claim_dict['stream']['source'] = {}
claim_dict['stream']['source']['source'] = self.lbry_file.sd_hash
claim_dict['stream']['source']['sourceType'] = 'lbry_sd_hash'
claim_dict['stream']['source']['contentType'] = get_content_type(file_path)
claim_dict['stream']['source']['version'] = "_0_0_1" # need current version here
claim_out = yield self.make_claim(name, bid, claim_dict, claim_address, change_address)
# check if we have a file already for this claim (if this is a publish update with a new stream)
old_stream_hashes = yield self.session.storage.get_old_stream_hashes_for_claim_id(claim_out['claim_id'],
self.lbry_file.stream_hash)
if old_stream_hashes:
for lbry_file in filter(lambda l: l.stream_hash in old_stream_hashes,
list(self.lbry_file_manager.lbry_files)):
yield self.lbry_file_manager.delete_lbry_file(lbry_file, delete_file=False)
log.info("Removed old stream for claim update: %s", lbry_file.stream_hash)
yield self.session.storage.save_content_claim(
self.lbry_file.stream_hash, "%s:%i" % (claim_out['txid'], claim_out['nout'])
)
defer.returnValue(claim_out)
@defer.inlineCallbacks
def publish_stream(self, name, bid, claim_dict, stream_hash, claim_address=None, change_address=None):
"""Make a claim without creating a lbry file"""
claim_out = yield self.make_claim(name, bid, claim_dict, claim_address, change_address)
if stream_hash: # the stream_hash returned from the db will be None if this isn't a stream we have
yield self.session.storage.save_content_claim(stream_hash, "%s:%i" % (claim_out['txid'],
claim_out['nout']))
self.lbry_file = [f for f in self.lbry_file_manager.lbry_files if f.stream_hash == stream_hash][0]
defer.returnValue(claim_out)
@defer.inlineCallbacks
def make_claim(self, name, bid, claim_dict, claim_address=None, change_address=None):
claim_out = yield self.wallet.claim_name(name, bid, claim_dict,
certificate_id=self.certificate_id,
claim_address=claim_address,
change_address=change_address)
defer.returnValue(claim_out)
def get_content_type(filename):
return mimetypes.guess_type(filename)[0] or 'application/octet-stream'
-3
View File
@@ -1,3 +0,0 @@
from lbrynet.daemon.auth.client import LBRYAPIClient
get_client = LBRYAPIClient.get_client
-48
View File
@@ -1,48 +0,0 @@
import logging
from zope.interface import implementer
from twisted.cred import portal, checkers, credentials, error as cred_error
from twisted.internet import defer
from twisted.web import resource
from lbrynet.daemon.auth.util import load_api_keys
log = logging.getLogger(__name__)
@implementer(portal.IRealm)
class HttpPasswordRealm(object):
def __init__(self, resource):
self.resource = resource
def requestAvatar(self, avatarId, mind, *interfaces):
log.debug("Processing request for %s", avatarId)
if resource.IResource in interfaces:
return (resource.IResource, self.resource, lambda: None)
raise NotImplementedError()
@implementer(checkers.ICredentialsChecker)
class PasswordChecker(object):
credentialInterfaces = (credentials.IUsernamePassword,)
def __init__(self, passwords):
self.passwords = passwords
@classmethod
def load_file(cls, key_path):
keys = load_api_keys(key_path)
return cls.load(keys)
@classmethod
def load(cls, password_dict):
passwords = {key: password_dict[key].secret for key in password_dict}
log.info("Loaded %i api key(s)", len(passwords))
return cls(passwords)
def requestAvatarId(self, creds):
if creds.username in self.passwords:
pw = self.passwords.get(creds.username)
pw_match = creds.checkPassword(pw)
if pw_match:
return defer.succeed(creds.username)
log.warning('Incorrect username or password')
return defer.fail(cred_error.UnauthorizedLogin('Incorrect username or password'))
-139
View File
@@ -1,139 +0,0 @@
import os
import json
import urlparse
import requests
from requests.cookies import RequestsCookieJar
import logging
from jsonrpc.proxy import JSONRPCProxy
from lbrynet import conf
from lbrynet.daemon.auth.util import load_api_keys, APIKey, API_KEY_NAME, get_auth_message
log = logging.getLogger(__name__)
USER_AGENT = "AuthServiceProxy/0.1"
TWISTED_SESSION = "TWISTED_SESSION"
LBRY_SECRET = "LBRY_SECRET"
HTTP_TIMEOUT = 30
def copy_cookies(cookies):
result = RequestsCookieJar()
result.update(cookies)
return result
class JSONRPCException(Exception):
def __init__(self, rpc_error):
Exception.__init__(self)
self.error = rpc_error
class AuthAPIClient(object):
def __init__(self, key, timeout, connection, count, cookies, url, login_url):
self.__api_key = key
self.__service_url = login_url
self.__id_count = count
self.__url = url
self.__conn = connection
self.__cookies = copy_cookies(cookies)
def __getattr__(self, name):
if name.startswith('__') and name.endswith('__'):
raise AttributeError(name)
def f(*args, **kwargs):
return self.call(name, [args, kwargs])
return f
def call(self, method, params=None):
params = params or {}
self.__id_count += 1
pre_auth_post_data = {
'version': '2',
'method': method,
'params': params,
'id': self.__id_count
}
to_auth = get_auth_message(pre_auth_post_data)
pre_auth_post_data.update({'hmac': self.__api_key.get_hmac(to_auth)})
post_data = json.dumps(pre_auth_post_data)
cookies = copy_cookies(self.__cookies)
req = requests.Request(
method='POST', url=self.__service_url, data=post_data, cookies=cookies,
headers={
'Host': self.__url.hostname,
'User-Agent': USER_AGENT,
'Content-type': 'application/json'
}
)
http_response = self.__conn.send(req.prepare())
if http_response is None:
raise JSONRPCException({
'code': -342, 'message': 'missing HTTP response from server'})
http_response.raise_for_status()
next_secret = http_response.headers.get(LBRY_SECRET, False)
if next_secret:
self.__api_key.secret = next_secret
self.__cookies = copy_cookies(http_response.cookies)
response = http_response.json()
if response.get('error') is not None:
raise JSONRPCException(response['error'])
elif 'result' not in response:
raise JSONRPCException({
'code': -343, 'message': 'missing JSON-RPC result'})
else:
return response['result']
@classmethod
def config(cls, key_name=None, key=None, pw_path=None, timeout=HTTP_TIMEOUT, connection=None, count=0,
cookies=None, auth=None, url=None, login_url=None):
api_key_name = key_name or API_KEY_NAME
pw_path = os.path.join(conf.settings['data_dir'], ".api_keys") if not pw_path else pw_path
if not key:
keys = load_api_keys(pw_path)
api_key = keys.get(api_key_name, False)
else:
api_key = APIKey(name=api_key_name, secret=key)
if login_url is None:
service_url = "http://%s:%s@%s:%i/%s" % (api_key_name,
api_key.secret,
conf.settings['api_host'],
conf.settings['api_port'],
conf.settings['API_ADDRESS'])
else:
service_url = login_url
id_count = count
if auth is None and connection is None and cookies is None and url is None:
# This is a new client instance, start an authenticated session
url = urlparse.urlparse(service_url)
conn = requests.Session()
req = requests.Request(method='POST',
url=service_url,
headers={'Host': url.hostname,
'User-Agent': USER_AGENT,
'Content-type': 'application/json'},)
r = req.prepare()
http_response = conn.send(r)
cookies = RequestsCookieJar()
cookies.update(http_response.cookies)
uid = cookies.get(TWISTED_SESSION)
api_key = APIKey.new(seed=uid)
else:
# This is a client that already has a session, use it
conn = connection
if not cookies.get(LBRY_SECRET):
raise Exception("Missing cookie")
secret = cookies.get(LBRY_SECRET)
api_key = APIKey(secret, api_key_name)
return cls(api_key, timeout, conn, id_count, cookies, url, service_url)
class LBRYAPIClient(object):
@staticmethod
def get_client():
if not conf.settings:
conf.initialize_settings()
return AuthAPIClient.config() if conf.settings['use_auth_http'] else \
JSONRPCProxy.from_url(conf.settings.get_api_connection_string())
-493
View File
@@ -1,493 +0,0 @@
import logging
import urlparse
import json
import inspect
from decimal import Decimal
from zope.interface import implements
from twisted.web import server, resource
from twisted.internet import defer
from twisted.python.failure import Failure
from twisted.internet.error import ConnectionDone, ConnectionLost
from txjsonrpc import jsonrpclib
from traceback import format_exc
from lbrynet import conf
from lbrynet.core.Error import InvalidAuthenticationToken
from lbrynet.core import utils
from lbrynet.daemon.auth.util import APIKey, get_auth_message
from lbrynet.daemon.auth.client import LBRY_SECRET
from lbrynet.undecorated import undecorated
log = logging.getLogger(__name__)
EMPTY_PARAMS = [{}]
class JSONRPCError(object):
# http://www.jsonrpc.org/specification#error_object
CODE_PARSE_ERROR = -32700 # Invalid JSON. Error while parsing the JSON text.
CODE_INVALID_REQUEST = -32600 # The JSON sent is not a valid Request object.
CODE_METHOD_NOT_FOUND = -32601 # The method does not exist / is not available.
CODE_INVALID_PARAMS = -32602 # Invalid method parameter(s).
CODE_INTERNAL_ERROR = -32603 # Internal JSON-RPC error (I think this is like a 500?)
CODE_APPLICATION_ERROR = -32500 # Generic error with our app??
CODE_AUTHENTICATION_ERROR = -32501 # Authentication failed
MESSAGES = {
CODE_PARSE_ERROR: "Parse Error. Data is not valid JSON.",
CODE_INVALID_REQUEST: "JSON data is not a valid Request",
CODE_METHOD_NOT_FOUND: "Method Not Found",
CODE_INVALID_PARAMS: "Invalid Params",
CODE_INTERNAL_ERROR: "Internal Error",
CODE_AUTHENTICATION_ERROR: "Authentication Failed",
}
HTTP_CODES = {
CODE_INVALID_REQUEST: 400,
CODE_PARSE_ERROR: 400,
CODE_INVALID_PARAMS: 400,
CODE_METHOD_NOT_FOUND: 404,
CODE_INTERNAL_ERROR: 500,
CODE_APPLICATION_ERROR: 500,
CODE_AUTHENTICATION_ERROR: 401,
}
def __init__(self, message, code=CODE_APPLICATION_ERROR, traceback=None, data=None):
assert isinstance(code, (int, long)), "'code' must be an int"
assert (data is None or isinstance(data, dict)), "'data' must be None or a dict"
self.code = code
if message is None:
message = self.MESSAGES[code] if code in self.MESSAGES else "API Error"
self.message = message
self.data = {} if data is None else data
self.traceback = []
if traceback is not None:
trace_lines = traceback.split("\n")
for i, t in enumerate(trace_lines):
if "--- <exception caught here> ---" in t:
if len(trace_lines) > i + 1:
self.traceback = [j for j in trace_lines[i+1:] if j]
break
def to_dict(self):
return {
'code': self.code,
'message': self.message,
'data': self.traceback
}
@classmethod
def create_from_exception(cls, exception, code=CODE_APPLICATION_ERROR, traceback=None):
return cls(exception.message, code=code, traceback=traceback)
def default_decimal(obj):
if isinstance(obj, Decimal):
return float(obj)
class UnknownAPIMethodError(Exception):
pass
class NotAllowedDuringStartupError(Exception):
pass
def trap(err, *to_trap):
err.trap(*to_trap)
def jsonrpc_dumps_pretty(obj, **kwargs):
try:
id_ = kwargs.pop("id")
except KeyError:
id_ = None
if isinstance(obj, JSONRPCError):
data = {"jsonrpc": "2.0", "error": obj.to_dict(), "id": id_}
else:
data = {"jsonrpc": "2.0", "result": obj, "id": id_}
return json.dumps(data, cls=jsonrpclib.JSONRPCEncoder, sort_keys=True, indent=2,
separators=(',', ': '), **kwargs) + "\n"
class JSONRPCServerType(type):
def __new__(mcs, name, bases, newattrs):
klass = type.__new__(mcs, name, bases, newattrs)
klass.callable_methods = {}
klass.deprecated_methods = {}
for methodname in dir(klass):
if methodname.startswith("jsonrpc_"):
method = getattr(klass, methodname)
if not hasattr(method, '_deprecated'):
klass.callable_methods.update({methodname.split("jsonrpc_")[1]: method})
else:
klass.deprecated_methods.update({methodname.split("jsonrpc_")[1]: method})
return klass
class AuthorizedBase(object):
__metaclass__ = JSONRPCServerType
@staticmethod
def deprecated(new_command=None):
def _deprecated_wrapper(f):
f._new_command = new_command
f._deprecated = True
return f
return _deprecated_wrapper
class AuthJSONRPCServer(AuthorizedBase):
"""
Authorized JSONRPC server used as the base class for the LBRY API
API methods are named with a leading "jsonrpc_"
Attributes:
allowed_during_startup (list): list of api methods that are callable before the server has finished startup
sessions (dict): (dict): {<session id>: <lbrynet.daemon.auth.util.APIKey>}
callable_methods (dict): {<api method name>: <api method>}
Authentication:
If use_authentication is true, basic HTTP and HMAC authentication will be used for all requests and the
service url will require a username and password.
To start an authenticated session a client sends an HTTP POST to <user>:<password>@<api host>:<api port>.
If accepted, the server replies with a TWISTED_SESSION cookie containing a session id and the message "OK".
The client initializes their shared secret for hmac to be the b64 encoded sha256 of their session id.
To send an authenticated request a client sends an HTTP POST to the auth api url with the TWISTED_SESSION
cookie and includes a hmac token in the message using the previously set shared secret. If the token is valid
the server will randomize the shared secret and return the new value under the LBRY_SECRET header, which the
client uses to generate the token for their next request.
"""
implements(resource.IResource)
isLeaf = True
allowed_during_startup = []
def __init__(self, use_authentication=None):
self._use_authentication = use_authentication or conf.settings['use_auth_http']
self.announced_startup = False
self.sessions = {}
def setup(self):
return NotImplementedError()
def _set_headers(self, request, data, update_secret=False):
if conf.settings['allowed_origin']:
request.setHeader("Access-Control-Allow-Origin", conf.settings['allowed_origin'])
request.setHeader("Content-Type", "application/json")
request.setHeader("Accept", "application/json-rpc")
request.setHeader("Content-Length", str(len(data)))
if update_secret:
session_id = request.getSession().uid
request.setHeader(LBRY_SECRET, self.sessions.get(session_id).secret)
@staticmethod
def _render_message(request, message):
request.write(message)
request.finish()
def _render_error(self, failure, request, id_):
if isinstance(failure, JSONRPCError):
error = failure
elif isinstance(failure, Failure):
# maybe failure is JSONRPCError wrapped in a twisted Failure
error = failure.check(JSONRPCError)
if error is None:
# maybe its a twisted Failure with another type of error
error = JSONRPCError(failure.getErrorMessage() or failure.type.__name__,
traceback=failure.getTraceback())
else:
# last resort, just cast it as a string
error = JSONRPCError(str(failure))
log.warning("error processing api request: %s\ntraceback: %s", error.message,
"\n".join(error.traceback))
response_content = jsonrpc_dumps_pretty(error, id=id_)
self._set_headers(request, response_content)
request.setResponseCode(200)
self._render_message(request, response_content)
@staticmethod
def _handle_dropped_request(result, d, function_name):
if not d.called:
log.warning("Cancelling dropped api request %s", function_name)
d.cancel()
def render(self, request):
try:
return self._render(request)
except BaseException as e:
log.error(e)
error = JSONRPCError.create_from_exception(e, traceback=format_exc())
self._render_error(error, request, None)
return server.NOT_DONE_YET
def _render(self, request):
time_in = utils.now()
# if not self._check_headers(request):
# self._render_error(Failure(InvalidHeaderError()), request, None)
# return server.NOT_DONE_YET
session = request.getSession()
session_id = session.uid
finished_deferred = request.notifyFinish()
if self._use_authentication:
# if this is a new session, send a new secret and set the expiration
# otherwise, session.touch()
if self._initialize_session(session_id):
def expire_session():
self._unregister_user_session(session_id)
session.startCheckingExpiration()
session.notifyOnExpire(expire_session)
message = "OK"
request.setResponseCode(200)
self._set_headers(request, message, True)
self._render_message(request, message)
return server.NOT_DONE_YET
else:
session.touch()
request.content.seek(0, 0)
content = request.content.read()
try:
parsed = jsonrpclib.loads(content)
except ValueError:
log.warning("Unable to decode request json")
self._render_error(JSONRPCError(None, JSONRPCError.CODE_PARSE_ERROR), request, None)
return server.NOT_DONE_YET
request_id = None
try:
function_name = parsed.get('method')
args = parsed.get('params', {})
request_id = parsed.get('id', None)
token = parsed.pop('hmac', None)
except AttributeError as err:
log.warning(err)
self._render_error(
JSONRPCError(None, code=JSONRPCError.CODE_INVALID_REQUEST), request, request_id
)
return server.NOT_DONE_YET
reply_with_next_secret = False
if self._use_authentication:
try:
self._verify_token(session_id, parsed, token)
except InvalidAuthenticationToken as err:
log.warning("API validation failed")
self._render_error(
JSONRPCError.create_from_exception(
err, code=JSONRPCError.CODE_AUTHENTICATION_ERROR,
traceback=format_exc()
),
request, request_id
)
return server.NOT_DONE_YET
request.addCookie("TWISTED_SESSION", session_id)
self._update_session_secret(session_id)
reply_with_next_secret = True
try:
fn = self._get_jsonrpc_method(function_name)
except UnknownAPIMethodError as err:
log.warning('Failed to get function %s: %s', function_name, err)
self._render_error(
JSONRPCError(None, JSONRPCError.CODE_METHOD_NOT_FOUND),
request, request_id
)
return server.NOT_DONE_YET
except NotAllowedDuringStartupError:
log.warning('Function not allowed during startup: %s', function_name)
self._render_error(
JSONRPCError("This method is unavailable until the daemon is fully started",
code=JSONRPCError.CODE_INVALID_REQUEST),
request, request_id
)
return server.NOT_DONE_YET
if args == EMPTY_PARAMS or args == []:
_args, _kwargs = (), {}
elif isinstance(args, dict):
_args, _kwargs = (), args
elif len(args) == 1 and isinstance(args[0], dict):
# TODO: this is for backwards compatibility. Remove this once API and UI are updated
# TODO: also delete EMPTY_PARAMS then
_args, _kwargs = (), args[0]
elif len(args) == 2 and isinstance(args[0], list) and isinstance(args[1], dict):
_args, _kwargs = args
else:
raise ValueError('invalid args format')
params_error, erroneous_params = self._check_params(fn, _args, _kwargs)
if params_error is not None:
params_error_message = '{} for {} command: {}'.format(
params_error, function_name, ', '.join(erroneous_params)
)
log.warning(params_error_message)
self._render_error(
JSONRPCError(params_error_message, code=JSONRPCError.CODE_INVALID_PARAMS),
request, request_id
)
return server.NOT_DONE_YET
d = defer.maybeDeferred(fn, self, *_args, **_kwargs)
# finished_deferred will callback when the request is finished
# and errback if something went wrong. If the errback is
# called, cancel the deferred stack. This is to prevent
# request.finish() from being called on a closed request.
finished_deferred.addErrback(self._handle_dropped_request, d, function_name)
d.addCallback(self._callback_render, request, request_id, reply_with_next_secret)
d.addErrback(trap, ConnectionDone, ConnectionLost, defer.CancelledError)
d.addErrback(self._render_error, request, request_id)
d.addBoth(lambda _: log.debug("%s took %f",
function_name,
(utils.now() - time_in).total_seconds()))
return server.NOT_DONE_YET
def _register_user_session(self, session_id):
"""
Add or update a HMAC secret for a session
@param session_id:
@return: secret
"""
log.info("Started new api session")
token = APIKey.new(seed=session_id)
self.sessions.update({session_id: token})
def _unregister_user_session(self, session_id):
log.info("Unregister API session")
del self.sessions[session_id]
def _check_headers(self, request):
return (
self._check_header_source(request, 'Origin') and
self._check_header_source(request, 'Referer')
)
def _check_header_source(self, request, header):
"""Check if the source of the request is allowed based on the header value."""
source = request.getHeader(header)
if not self._check_source_of_request(source):
log.warning("Attempted api call from invalid %s: %s", header, source)
return False
return True
def _check_source_of_request(self, source):
if source is None:
return True
if conf.settings['api_host'] == '0.0.0.0':
return True
server, port = self.get_server_port(source)
return self._check_server_port(server, port)
def _check_server_port(self, server, port):
api = (conf.settings['api_host'], conf.settings['api_port'])
return (server, port) == api or self._is_from_allowed_origin(server, port)
def _is_from_allowed_origin(self, server, port):
allowed_origin = conf.settings['allowed_origin']
if not allowed_origin:
return False
if allowed_origin == '*':
return True
allowed_server, allowed_port = self.get_server_port(allowed_origin)
return (allowed_server, allowed_port) == (server, port)
def get_server_port(self, origin):
parsed = urlparse.urlparse(origin)
server_port = parsed.netloc.split(':')
assert len(server_port) <= 2
if len(server_port) == 2:
return server_port[0], int(server_port[1])
else:
return server_port[0], 80
def _verify_method_is_callable(self, function_path):
if function_path not in self.callable_methods:
raise UnknownAPIMethodError(function_path)
if not self.announced_startup:
if function_path not in self.allowed_during_startup:
raise NotAllowedDuringStartupError(function_path)
def _get_jsonrpc_method(self, function_path):
if function_path in self.deprecated_methods:
new_command = self.deprecated_methods[function_path]._new_command
log.warning('API function \"%s\" is deprecated, please update to use \"%s\"',
function_path, new_command)
function_path = new_command
self._verify_method_is_callable(function_path)
return self.callable_methods.get(function_path)
@staticmethod
def _check_params(function, args_tup, args_dict):
argspec = inspect.getargspec(undecorated(function))
num_optional_params = 0 if argspec.defaults is None else len(argspec.defaults)
duplicate_params = [
duplicate_param
for duplicate_param in argspec.args[1:len(args_tup) + 1]
if duplicate_param in args_dict
]
if duplicate_params:
return 'Duplicate parameters', duplicate_params
missing_required_params = [
required_param
for required_param in argspec.args[len(args_tup)+1:-num_optional_params]
if required_param not in args_dict
]
if len(missing_required_params):
return 'Missing required parameters', missing_required_params
extraneous_params = [] if argspec.keywords is not None else [
extra_param
for extra_param in args_dict
if extra_param not in argspec.args[1:]
]
if len(extraneous_params):
return 'Extraneous parameters', extraneous_params
return None, None
def _initialize_session(self, session_id):
if not self.sessions.get(session_id):
self._register_user_session(session_id)
return True
return False
def _verify_token(self, session_id, message, token):
if token is None:
raise InvalidAuthenticationToken('Authentication token not found')
to_auth = get_auth_message(message)
api_key = self.sessions.get(session_id)
if not api_key.compare_hmac(to_auth, token):
raise InvalidAuthenticationToken('Invalid authentication token')
def _update_session_secret(self, session_id):
self.sessions.update({session_id: APIKey.new(name=session_id)})
def _callback_render(self, result, request, id_, auth_required=False):
try:
encoded_message = jsonrpc_dumps_pretty(result, id=id_, default=default_decimal)
request.setResponseCode(200)
self._set_headers(request, encoded_message, auth_required)
self._render_message(request, encoded_message)
except Exception as err:
log.exception("Failed to render API response: %s", result)
self._render_error(err, request, id_)
@staticmethod
def _render_response(result):
return defer.succeed(result)
-91
View File
@@ -1,91 +0,0 @@
import base58
import hmac
import hashlib
import yaml
import os
import json
import logging
log = logging.getLogger(__name__)
API_KEY_NAME = "api"
def sha(x):
h = hashlib.sha256(x).digest()
return base58.b58encode(h)
def generate_key(x=None):
if x is None:
return sha(os.urandom(256))
else:
return sha(x)
class APIKey(object):
def __init__(self, secret, name, expiration=None):
self.secret = secret
self.name = name
self.expiration = expiration
@classmethod
def new(cls, seed=None, name=None, expiration=None):
secret = generate_key(seed)
key_name = name if name else sha(secret)
return APIKey(secret, key_name, expiration)
def _raw_key(self):
return base58.b58decode(self.secret)
def get_hmac(self, message):
decoded_key = self._raw_key()
signature = hmac.new(decoded_key, message, hashlib.sha256)
return base58.b58encode(signature.digest())
def compare_hmac(self, message, token):
decoded_token = base58.b58decode(token)
target = base58.b58decode(self.get_hmac(message))
try:
if len(decoded_token) != len(target):
return False
return hmac.compare_digest(decoded_token, target)
except:
return False
def load_api_keys(path):
if not os.path.isfile(path):
raise Exception("Invalid api key path")
with open(path, "r") as f:
data = yaml.load(f.read())
keys_for_return = {}
for key_name in data:
key = data[key_name]
secret = key['secret']
expiration = key['expiration']
keys_for_return.update({key_name: APIKey(secret, key_name, expiration)})
return keys_for_return
def save_api_keys(keys, path):
with open(path, "w") as f:
key_dict = {keys[key_name].name: {'secret': keys[key_name].secret,
'expiration': keys[key_name].expiration}
for key_name in keys}
data = yaml.safe_dump(key_dict)
f.write(data)
def initialize_api_key_file(key_path):
keys = {}
new_api_key = APIKey.new(name=API_KEY_NAME)
keys.update({new_api_key.name: new_api_key})
save_api_keys(keys, key_path)
def get_auth_message(message_dict):
return json.dumps(message_dict, sort_keys=True)
@@ -8,7 +8,7 @@ DELAY_INCREMENT = 0.0001
QUEUE_SIZE_THRESHOLD = 100
class CallLaterManager(object):
class CallLaterManager:
def __init__(self, callLater):
"""
:param callLater: (IReactorTime.callLater)
+2
View File
@@ -29,6 +29,8 @@ rpcTimeout = 5
# number of rpc attempts to make before a timeout results in the node being removed as a contact
rpcAttempts = 5
# time window to count failures (in seconds)
rpcAttemptsPruningTimeWindow = 600
# Delay between iterations of iterative node lookups (for loose parallelism) (in seconds)
iterativeLookupDelay = rpcTimeout / 2
+26 -26
View File
@@ -1,16 +1,18 @@
import ipaddress
from binascii import hexlify
from functools import reduce
from lbrynet.dht import constants
def is_valid_ipv4(address):
try:
ip = ipaddress.ip_address(address.decode()) # this needs to be unicode, thus the decode()
ip = ipaddress.ip_address(address)
return ip.version == 4
except ipaddress.AddressValueError:
return False
class _Contact(object):
class _Contact:
""" Encapsulation for remote contact
This class contains information on a single remote contact, and also
@@ -19,8 +21,8 @@ class _Contact(object):
def __init__(self, contactManager, id, ipAddress, udpPort, networkProtocol, firstComm):
if id is not None:
if not len(id) == constants.key_bits / 8:
raise ValueError("invalid node id: %s" % id.encode('hex'))
if not len(id) == constants.key_bits // 8:
raise ValueError("invalid node id: {}".format(hexlify(id).decode()))
if not 0 <= udpPort <= 65536:
raise ValueError("invalid port")
if not is_valid_ipv4(ipAddress):
@@ -38,7 +40,7 @@ class _Contact(object):
self._token = (None, 0) # token, timestamp
def update_token(self, token):
self._token = token, self.getTime()
self._token = token, self.getTime() if token else 0
@property
def token(self):
@@ -56,7 +58,7 @@ class _Contact(object):
def log_id(self, short=True):
if not self.id:
return "not initialized"
id_hex = self.id.encode('hex')
id_hex = hexlify(self.id)
return id_hex if not short else id_hex[:8]
@property
@@ -65,7 +67,7 @@ class _Contact(object):
@property
def lastFailed(self):
return self._contactManager._rpc_failures.get((self.address, self.port), [None])[-1]
return (self.failures or [None])[-1]
@property
def failures(self):
@@ -95,25 +97,17 @@ class _Contact(object):
return None
def __eq__(self, other):
if isinstance(other, _Contact):
return self.id == other.id
elif isinstance(other, str):
return self.id == other
else:
return False
if not isinstance(other, _Contact):
raise TypeError("invalid type to compare with Contact: %s" % str(type(other)))
return (self.id, self.address, self.port) == (other.id, other.address, other.port)
def __ne__(self, other):
if isinstance(other, _Contact):
return self.id != other.id
elif isinstance(other, str):
return self.id != other
else:
return True
def __hash__(self):
return hash((self.id, self.address, self.port))
def compact_ip(self):
compact_ip = reduce(
lambda buff, x: buff + bytearray([int(x)]), self.address.split('.'), bytearray())
return str(compact_ip)
return compact_ip
def set_id(self, id):
if not self._id:
@@ -156,12 +150,12 @@ class _Contact(object):
raise AttributeError("unknown command: %s" % name)
def _sendRPC(*args, **kwargs):
return self._networkProtocol.sendRPC(self, name, args)
return self._networkProtocol.sendRPC(self, name.encode(), args)
return _sendRPC
class ContactManager(object):
class ContactManager:
def __init__(self, get_time=None):
if not get_time:
from twisted.internet import reactor
@@ -171,12 +165,11 @@ class ContactManager(object):
self._rpc_failures = {}
def get_contact(self, id, address, port):
for contact in self._contacts.itervalues():
for contact in self._contacts.values():
if contact.id == id and contact.address == address and contact.port == port:
return contact
def make_contact(self, id, ipAddress, udpPort, networkProtocol, firstComm=0):
ipAddress = str(ipAddress)
contact = self.get_contact(id, ipAddress, udpPort)
if contact:
return contact
@@ -185,5 +178,12 @@ class ContactManager(object):
return contact
def is_ignored(self, origin_tuple):
failed_rpc_count = len(self._rpc_failures.get(origin_tuple, []))
failed_rpc_count = len(self._prune_failures(origin_tuple))
return failed_rpc_count > constants.rpcAttempts
def _prune_failures(self, origin_tuple):
# Prunes recorded failures to the last time window of attempts
pruning_limit = self._get_time() - constants.rpcAttemptsPruningTimeWindow
pruned = list(filter(lambda t: t >= pruning_limit, self._rpc_failures.get(origin_tuple, [])))
self._rpc_failures[origin_tuple] = pruned
return pruned
+23 -25
View File
@@ -1,25 +1,20 @@
import UserDict
import constants
from interface import IDataStore
from zope.interface import implements
from collections import UserDict
from lbrynet.dht import constants
class DictDataStore(UserDict.DictMixin):
class DictDataStore(UserDict):
""" A datastore using an in-memory Python dictionary """
implements(IDataStore)
#implements(IDataStore)
def __init__(self, getTime=None):
# Dictionary format:
# { <key>: (<contact>, <value>, <lastPublished>, <originallyPublished> <originalPublisherID>) }
self._dict = {}
super().__init__()
if not getTime:
from twisted.internet import reactor
getTime = reactor.seconds
self._getTime = getTime
def keys(self):
""" Return a list of the keys in this data store """
return self._dict.keys()
self.completed_blobs = set()
def filter_bad_and_expired_peers(self, key):
"""
@@ -28,41 +23,44 @@ class DictDataStore(UserDict.DictMixin):
return filter(
lambda peer:
self._getTime() - peer[3] < constants.dataExpireTimeout and peer[0].contact_is_good is not False,
self._dict[key]
self[key]
)
def filter_expired_peers(self, key):
"""
Returns only non-expired peers
"""
return filter(lambda peer: self._getTime() - peer[3] < constants.dataExpireTimeout, self._dict[key])
return filter(lambda peer: self._getTime() - peer[3] < constants.dataExpireTimeout, self[key])
def removeExpiredPeers(self):
for key in self._dict.keys():
unexpired_peers = self.filter_expired_peers(key)
expired_keys = []
for key in self.keys():
unexpired_peers = list(self.filter_expired_peers(key))
if not unexpired_peers:
del self._dict[key]
expired_keys.append(key)
else:
self._dict[key] = unexpired_peers
self[key] = unexpired_peers
for key in expired_keys:
del self[key]
def hasPeersForBlob(self, key):
return True if key in self._dict and len(self.filter_bad_and_expired_peers(key)) else False
return bool(key in self and len(tuple(self.filter_bad_and_expired_peers(key))))
def addPeerToBlob(self, contact, key, compact_address, lastPublished, originallyPublished, originalPublisherID):
if key in self._dict:
if compact_address not in map(lambda store_tuple: store_tuple[1], self._dict[key]):
self._dict[key].append(
if key in self:
if compact_address not in map(lambda store_tuple: store_tuple[1], self[key]):
self[key].append(
(contact, compact_address, lastPublished, originallyPublished, originalPublisherID)
)
else:
self._dict[key] = [(contact, compact_address, lastPublished, originallyPublished, originalPublisherID)]
self[key] = [(contact, compact_address, lastPublished, originallyPublished, originalPublisherID)]
def getPeersForBlob(self, key):
return [] if key not in self._dict else [val[1] for val in self.filter_bad_and_expired_peers(key)]
return [] if key not in self else [val[1] for val in self.filter_bad_and_expired_peers(key)]
def getStoringContacts(self):
contacts = set()
for key in self._dict:
for values in self._dict[key]:
for key in self:
for values in self[key]:
contacts.add(values[0])
return list(contacts)
+5 -5
View File
@@ -1,21 +1,21 @@
from lbrynet.dht import constants
class Distance(object):
class Distance:
"""Calculate the XOR result between two string variables.
Frequently we re-use one of the points so as an optimization
we pre-calculate the long value of that point.
we pre-calculate the value of that point.
"""
def __init__(self, key):
if len(key) != constants.key_bits / 8:
if len(key) != constants.key_bits // 8:
raise ValueError("invalid key length: %i" % len(key))
self.key = key
self.val_key_one = long(key.encode('hex'), 16)
self.val_key_one = int.from_bytes(key, 'big')
def __call__(self, key_two):
val_key_two = long(key_two.encode('hex'), 16)
val_key_two = int.from_bytes(key_two, 'big')
return self.val_key_one ^ val_key_two
def is_closer(self, a, b):
+68 -127
View File
@@ -1,134 +1,75 @@
from error import DecodeError
from lbrynet.dht.error import DecodeError
class Encoding(object):
""" Interface for RPC message encoders/decoders
All encoding implementations used with this library should inherit and
implement this.
"""
def encode(self, data):
""" Encode the specified data
@param data: The data to encode
This method has to support encoding of the following
types: C{str}, C{int} and C{long}
Any additional data types may be supported as long as the
implementing class's C{decode()} method can successfully
decode them.
@return: The encoded data
@rtype: str
"""
def decode(self, data):
""" Decode the specified data string
@param data: The data (byte string) to decode.
@type data: str
@return: The decoded data (in its correct type)
"""
def bencode(data):
""" Encoder implementation of the Bencode algorithm (Bittorrent). """
if isinstance(data, int):
return b'i%de' % data
elif isinstance(data, (bytes, bytearray)):
return b'%d:%s' % (len(data), data)
elif isinstance(data, str):
return b'%d:%s' % (len(data), data.encode())
elif isinstance(data, (list, tuple)):
encoded_list_items = b''
for item in data:
encoded_list_items += bencode(item)
return b'l%se' % encoded_list_items
elif isinstance(data, dict):
encoded_dict_items = b''
keys = data.keys()
for key in sorted(keys):
encoded_dict_items += bencode(key)
encoded_dict_items += bencode(data[key])
return b'd%se' % encoded_dict_items
else:
raise TypeError("Cannot bencode '%s' object" % type(data))
class Bencode(Encoding):
""" Implementation of a Bencode-based algorithm (Bencode is the encoding
algorithm used by Bittorrent).
def bdecode(data):
""" Decoder implementation of the Bencode algorithm. """
assert type(data) == bytes # fixme: _maybe_ remove this after porting
if len(data) == 0:
raise DecodeError('Cannot decode empty string')
try:
return _decode_recursive(data)[0]
except ValueError as e:
raise DecodeError(str(e))
@note: This algorithm differs from the "official" Bencode algorithm in
that it can encode/decode floating point values in addition to
integers.
"""
def encode(self, data):
""" Encoder implementation of the Bencode algorithm
@param data: The data to encode
@type data: int, long, tuple, list, dict or str
@return: The encoded data
@rtype: str
"""
if isinstance(data, (int, long)):
return 'i%de' % data
elif isinstance(data, str):
return '%d:%s' % (len(data), data)
elif isinstance(data, (list, tuple)):
encodedListItems = ''
for item in data:
encodedListItems += self.encode(item)
return 'l%se' % encodedListItems
elif isinstance(data, dict):
encodedDictItems = ''
keys = data.keys()
keys.sort()
for key in keys:
encodedDictItems += self.encode(key) # TODO: keys should always be bytestrings
encodedDictItems += self.encode(data[key])
return 'd%se' % encodedDictItems
else:
print data
raise TypeError("Cannot bencode '%s' object" % type(data))
def decode(self, data):
""" Decoder implementation of the Bencode algorithm
@param data: The encoded data
@type data: str
@note: This is a convenience wrapper for the recursive decoding
algorithm, C{_decodeRecursive}
@return: The decoded data, as a native Python type
@rtype: int, list, dict or str
"""
if len(data) == 0:
raise DecodeError('Cannot decode empty string')
def _decode_recursive(data, start_index=0):
if data[start_index] == ord('i'):
end_pos = data[start_index:].find(b'e') + start_index
return int(data[start_index + 1:end_pos]), end_pos + 1
elif data[start_index] == ord('l'):
start_index += 1
decoded_list = []
while data[start_index] != ord('e'):
list_data, start_index = _decode_recursive(data, start_index)
decoded_list.append(list_data)
return decoded_list, start_index + 1
elif data[start_index] == ord('d'):
start_index += 1
decoded_dict = {}
while data[start_index] != ord('e'):
key, start_index = _decode_recursive(data, start_index)
value, start_index = _decode_recursive(data, start_index)
decoded_dict[key] = value
return decoded_dict, start_index
elif data[start_index] == ord('f'):
# This (float data type) is a non-standard extension to the original Bencode algorithm
end_pos = data[start_index:].find(b'e') + start_index
return float(data[start_index + 1:end_pos]), end_pos + 1
elif data[start_index] == ord('n'):
# This (None/NULL data type) is a non-standard extension
# to the original Bencode algorithm
return None, start_index + 1
else:
split_pos = data[start_index:].find(b':') + start_index
try:
return self._decodeRecursive(data)[0]
except ValueError as e:
raise DecodeError(e.message)
@staticmethod
def _decodeRecursive(data, startIndex=0):
""" Actual implementation of the recursive Bencode algorithm
Do not call this; use C{decode()} instead
"""
if data[startIndex] == 'i':
endPos = data[startIndex:].find('e') + startIndex
return int(data[startIndex + 1:endPos]), endPos + 1
elif data[startIndex] == 'l':
startIndex += 1
decodedList = []
while data[startIndex] != 'e':
listData, startIndex = Bencode._decodeRecursive(data, startIndex)
decodedList.append(listData)
return decodedList, startIndex + 1
elif data[startIndex] == 'd':
startIndex += 1
decodedDict = {}
while data[startIndex] != 'e':
key, startIndex = Bencode._decodeRecursive(data, startIndex)
value, startIndex = Bencode._decodeRecursive(data, startIndex)
decodedDict[key] = value
return decodedDict, startIndex
elif data[startIndex] == 'f':
# This (float data type) is a non-standard extension to the original Bencode algorithm
endPos = data[startIndex:].find('e') + startIndex
return float(data[startIndex + 1:endPos]), endPos + 1
elif data[startIndex] == 'n':
# This (None/NULL data type) is a non-standard extension
# to the original Bencode algorithm
return None, startIndex + 1
else:
splitPos = data[startIndex:].find(':') + startIndex
try:
length = int(data[startIndex:splitPos])
except ValueError, e:
raise DecodeError, e
startIndex = splitPos + 1
endPos = startIndex + length
bytes = data[startIndex:endPos]
return bytes, endPos
length = int(data[start_index:split_pos])
except ValueError:
raise DecodeError()
start_index = split_pos + 1
end_pos = start_index + length
b = data[start_index:end_pos]
return b, end_pos
+3 -5
View File
@@ -1,10 +1,10 @@
import binascii
import exceptions
#import exceptions
# this is a dict of {"exceptions.<exception class name>": exception class} items used to raise
# remote built-in exceptions locally
BUILTIN_EXCEPTIONS = {
"exceptions.%s" % e: getattr(exceptions, e) for e in dir(exceptions) if not e.startswith("_")
# "exceptions.%s" % e: getattr(exceptions, e) for e in dir(exceptions) if not e.startswith("_")
}
@@ -13,14 +13,12 @@ class DecodeError(Exception):
Should be raised by an C{Encoding} implementation if decode operation
fails
"""
pass
class BucketFull(Exception):
"""
Raised when the bucket is full
"""
pass
class UnknownRemoteException(Exception):
@@ -37,7 +35,7 @@ class TimeoutError(Exception):
msg = 'Timeout connecting to {}'.format(binascii.hexlify(remote_contact_id))
else:
msg = 'Timeout connecting to uninitialized node'
Exception.__init__(self, msg)
super().__init__(msg)
self.remote_contact_id = remote_contact_id
+1 -2
View File
@@ -10,7 +10,6 @@ class IDataStore(Interface):
def keys(self):
""" Return a list of the keys in this data store """
pass
def removeExpiredPeers(self):
pass
@@ -59,7 +58,7 @@ class IRoutingTable(Interface):
@param count: the amount of contacts to return
@type count: int
@param _rpcNodeID: Used during RPC, this is be the sender's Node ID
Whatever ID is passed in the paramater will get
Whatever ID is passed in the parameter will get
excluded from the list of returned contacts.
@type _rpcNodeID: str
+43 -22
View File
@@ -1,8 +1,8 @@
import logging
from twisted.internet import defer
from distance import Distance
from error import TimeoutError
import constants
from lbrynet.dht.distance import Distance
from lbrynet.dht.error import TimeoutError
from lbrynet.dht import constants
log = logging.getLogger(__name__)
@@ -14,10 +14,18 @@ def get_contact(contact_list, node_id, address, port):
raise IndexError(node_id)
class _IterativeFind(object):
def expand_peer(compact_peer_info):
host = "{}.{}.{}.{}".format(*compact_peer_info[:4])
port = int.from_bytes(compact_peer_info[4:6], 'big')
peer_node_id = compact_peer_info[6:]
return (peer_node_id, host, port)
class _IterativeFind:
# TODO: use polymorphism to search for a value or node
# instead of using a find_value flag
def __init__(self, node, shortlist, key, rpc):
def __init__(self, node, shortlist, key, rpc, exclude=None):
self.exclude = set(exclude or [])
self.node = node
self.finished_deferred = defer.Deferred()
# all distance operations in this class only care about the distance
@@ -29,7 +37,7 @@ class _IterativeFind(object):
# Shortlist of contact objects (the k closest known contacts to the key from the routing table)
self.shortlist = shortlist
# The search key
self.key = str(key)
self.key = key
# The rpc method name (findValue or findNode)
self.rpc = rpc
# List of active queries; len() indicates number of active probes
@@ -59,28 +67,28 @@ class _IterativeFind(object):
def getContactTriples(self, result):
if self.is_find_value_request:
contact_triples = result['contacts']
contact_triples = result[b'contacts']
else:
contact_triples = result
for contact_tup in contact_triples:
if not isinstance(contact_tup, (list, tuple)) or len(contact_tup) != 3:
raise ValueError("invalid contact triple")
contact_tup[1] = contact_tup[1].decode() # ips are strings
return contact_triples
def sortByDistance(self, contact_list):
"""Sort the list of contacts in order by distance from key"""
contact_list.sort(key=lambda c: self.distance(c.id))
@defer.inlineCallbacks
def extendShortlist(self, contact, result):
# The "raw response" tuple contains the response message and the originating address info
originAddress = (contact.address, contact.port)
if self.finished_deferred.called:
defer.returnValue(contact.id)
return contact.id
if self.node.contact_manager.is_ignored(originAddress):
raise ValueError("contact is ignored")
if contact.id == self.node.node_id:
defer.returnValue(contact.id)
return contact.id
if contact not in self.active_contacts:
self.active_contacts.append(contact)
@@ -93,24 +101,28 @@ class _IterativeFind(object):
# we are looking for before treating it as a list of contact triples
if self.is_find_value_request and self.key in result:
# We have found the value
self.find_value_result[self.key] = result[self.key]
self.finished_deferred.callback(self.find_value_result)
for peer in result[self.key]:
node_id, host, port = expand_peer(peer)
if (host, port) not in self.exclude:
self.find_value_result.setdefault(self.key, []).append((node_id, host, port))
if self.find_value_result:
self.finished_deferred.callback(self.find_value_result)
else:
if self.is_find_value_request:
# We are looking for a value, and the remote node didn't have it
# - mark it as the closest "empty" node, if it is
# TODO: store to this peer after finding the value as per the kademlia spec
if 'closestNodeNoValue' in self.find_value_result:
if b'closestNodeNoValue' in self.find_value_result:
if self.is_closer(contact):
self.find_value_result['closestNodeNoValue'] = contact
self.find_value_result[b'closestNodeNoValue'] = contact
else:
self.find_value_result['closestNodeNoValue'] = contact
self.find_value_result[b'closestNodeNoValue'] = contact
contactTriples = self.getContactTriples(result)
for contactTriple in contactTriples:
if (contactTriple[1], contactTriple[2]) in ((c.address, c.port) for c in self.already_contacted):
continue
elif self.node.contact_manager.is_ignored((contactTriple[1], contactTriple[2])):
raise ValueError("contact is ignored")
continue
else:
found_contact = self.node.contact_manager.make_contact(contactTriple[0], contactTriple[1],
contactTriple[2], self.node._protocol)
@@ -121,19 +133,22 @@ class _IterativeFind(object):
self.sortByDistance(self.active_contacts)
self.finished_deferred.callback(self.active_contacts[:min(constants.k, len(self.active_contacts))])
defer.returnValue(contact.id)
return contact.id
@defer.inlineCallbacks
def probeContact(self, contact):
fn = getattr(contact, self.rpc)
try:
response = yield fn(self.key)
result = yield self.extendShortlist(contact, response)
result = self.extendShortlist(contact, response)
defer.returnValue(result)
except (TimeoutError, defer.CancelledError, ValueError, IndexError):
defer.returnValue(contact.id)
def should_stop(self):
if self.is_find_value_request:
# search stops when it finds a value, let it run
return False
if self.prev_closest_node and self.closest_node and self.distance.is_closer(self.prev_closest_node.id,
self.closest_node.id):
# we're getting further away
@@ -157,6 +172,9 @@ class _IterativeFind(object):
already_contacted_addresses = {(c.address, c.port) for c in self.already_contacted}
to_remove = []
for contact in self.shortlist:
if self.node.contact_manager.is_ignored((contact.address, contact.port)):
to_remove.append(contact) # a contact became bad during iteration
continue
if (contact.address, contact.port) not in already_contacted_addresses:
self.already_contacted.append(contact)
to_remove.append(contact)
@@ -185,8 +203,11 @@ class _IterativeFind(object):
elif not self.finished_deferred.called and not self.active_probes or self.should_stop():
# If no probes were sent, there will not be any improvement, so we're done
self.sortByDistance(self.active_contacts)
self.finished_deferred.callback(self.active_contacts[:min(constants.k, len(self.active_contacts))])
if self.is_find_value_request:
self.finished_deferred.callback(self.find_value_result)
else:
self.sortByDistance(self.active_contacts)
self.finished_deferred.callback(self.active_contacts[:min(constants.k, len(self.active_contacts))])
elif not self.finished_deferred.called:
# Force the next iteration
self.searchIteration()
@@ -203,7 +224,7 @@ class _IterativeFind(object):
self.pending_iteration_calls.append(cancel)
def iterativeFind(node, shortlist, key, rpc):
helper = _IterativeFind(node, shortlist, key, rpc)
def iterativeFind(node, shortlist, key, rpc, exclude=None):
helper = _IterativeFind(node, shortlist, key, rpc, exclude)
helper.searchIteration(0)
return helper.finished_deferred
+7 -6
View File
@@ -1,12 +1,13 @@
import logging
import constants
from distance import Distance
from error import BucketFull
from lbrynet.dht import constants
from lbrynet.dht.distance import Distance
from lbrynet.dht.error import BucketFull
log = logging.getLogger(__name__)
class KBucket(object):
class KBucket:
""" Description - later
"""
@@ -135,8 +136,8 @@ class KBucket(object):
if not.
@rtype: bool
"""
if isinstance(key, str):
key = long(key.encode('hex'), 16)
if isinstance(key, bytes):
key = int.from_bytes(key, 'big')
return self.rangeMin <= key < self.rangeMax
def __len__(self):
+2 -2
View File
@@ -7,10 +7,10 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import msgtypes
from lbrynet.dht import msgtypes
class MessageTranslator(object):
class MessageTranslator:
""" Interface for RPC message translators/formatters
Classes inheriting from this should provide a translation services between
+9 -10
View File
@@ -7,17 +7,17 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
from lbrynet.core.utils import generate_id
import constants
from lbrynet.utils import generate_id
from lbrynet.dht import constants
class Message(object):
class Message:
""" Base class for messages - all "unknown" messages use this class """
def __init__(self, rpcID, nodeID):
if len(rpcID) != constants.rpc_id_length:
raise ValueError("invalid rpc id: %i bytes (expected 20)" % len(rpcID))
if len(nodeID) != constants.key_bits / 8:
if len(nodeID) != constants.key_bits // 8:
raise ValueError("invalid node id: %i bytes (expected 48)" % len(nodeID))
self.id = rpcID
self.nodeID = nodeID
@@ -29,7 +29,7 @@ class RequestMessage(Message):
def __init__(self, nodeID, method, methodArgs, rpcID=None):
if rpcID is None:
rpcID = generate_id()[:constants.rpc_id_length]
Message.__init__(self, rpcID, nodeID)
super().__init__(rpcID, nodeID)
self.request = method
self.args = methodArgs
@@ -38,7 +38,7 @@ class ResponseMessage(Message):
""" Message containing the result from a successful RPC request """
def __init__(self, rpcID, nodeID, response):
Message.__init__(self, rpcID, nodeID)
super().__init__(rpcID, nodeID)
self.response = response
@@ -46,8 +46,7 @@ class ErrorMessage(ResponseMessage):
""" Message containing the error from an unsuccessful RPC request """
def __init__(self, rpcID, nodeID, exceptionType, errorMessage):
ResponseMessage.__init__(self, rpcID, nodeID, errorMessage)
super().__init__(rpcID, nodeID, errorMessage)
if isinstance(exceptionType, type):
self.exceptionType = '%s.%s' % (exceptionType.__module__, exceptionType.__name__)
else:
self.exceptionType = exceptionType
exceptionType = (f'{exceptionType.__module__}.{exceptionType.__name__}').encode()
self.exceptionType = exceptionType
+100 -125
View File
@@ -1,40 +1,20 @@
#!/usr/bin/env python
#
# This library is free software, distributed under the terms of
# the GNU Lesser General Public License Version 3, or any later version.
# See the COPYING file included in this archive
#
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import binascii
import hashlib
import struct
import logging
from functools import reduce
from twisted.internet import defer, error, task
from lbrynet.core.utils import generate_id, DeferredDict
from lbrynet.core.call_later_manager import CallLaterManager
from lbrynet.core.PeerManager import PeerManager
from error import TimeoutError
import constants
import routingtable
import datastore
import protocol
from peerfinder import DHTPeerFinder
from contact import ContactManager
from iterativefind import iterativeFind
from lbrynet.utils import generate_id, DeferredDict
from lbrynet.dht.call_later_manager import CallLaterManager
from lbrynet.dht.error import TimeoutError
from lbrynet.dht import constants, routingtable, datastore, protocol
from lbrynet.dht.contact import ContactManager
from lbrynet.dht.iterativefind import iterativeFind
log = logging.getLogger(__name__)
def expand_peer(compact_peer_info):
host = ".".join([str(ord(d)) for d in compact_peer_info[:4]])
port, = struct.unpack('>H', compact_peer_info[4:6])
peer_node_id = compact_peer_info[6:]
return (peer_node_id, host, port)
def rpcmethod(func):
""" Decorator to expose Node methods as remote procedure calls
@@ -45,7 +25,7 @@ def rpcmethod(func):
return func
class MockKademliaHelper(object):
class MockKademliaHelper:
def __init__(self, clock=None, callLater=None, resolve=None, listenUDP=None):
if not listenUDP or not resolve or not callLater or not clock:
from twisted.internet import reactor
@@ -98,8 +78,8 @@ class Node(MockKademliaHelper):
def __init__(self, node_id=None, udpPort=4000, dataStore=None,
routingTableClass=None, networkProtocol=None,
externalIP=None, peerPort=3333, listenUDP=None,
callLater=None, resolve=None, clock=None, peer_finder=None,
peer_manager=None):
callLater=None, resolve=None, clock=None,
interface='', externalUDPPort=None):
"""
@param dataStore: The data store to use. This must be class inheriting
from the C{DataStore} interface (or providing the
@@ -125,9 +105,10 @@ class Node(MockKademliaHelper):
@param peerPort: the port at which this node announces it has a blob for
"""
MockKademliaHelper.__init__(self, clock, callLater, resolve, listenUDP)
super().__init__(clock, callLater, resolve, listenUDP)
self.node_id = node_id or self._generateID()
self.port = udpPort
self._listen_interface = interface
self._change_token_lc = self.get_looping_call(self.change_token)
self._refresh_node_lc = self.get_looping_call(self._refreshNode)
self._refresh_contacts_lc = self.get_looping_call(self._refreshContacts)
@@ -138,25 +119,23 @@ class Node(MockKademliaHelper):
else:
self._routingTable = routingTableClass(self.node_id, self.clock.seconds)
# Initialize this node's network access mechanisms
if networkProtocol is None:
self._protocol = protocol.KademliaProtocol(self)
else:
self._protocol = networkProtocol
# Initialize the data storage mechanism used by this node
self._protocol = networkProtocol or protocol.KademliaProtocol(self)
self.token_secret = self._generateID()
self.old_token_secret = None
self.externalIP = externalIP
self.peerPort = peerPort
self.externalUDPPort = externalUDPPort or self.port
self._dataStore = dataStore or datastore.DictDataStore(self.clock.seconds)
self.peer_manager = peer_manager or PeerManager()
self.peer_finder = peer_finder or DHTPeerFinder(self, self.peer_manager)
self._join_deferred = None
def __del__(self):
log.warning("unclean shutdown of the dht node")
if hasattr(self, "_listeningPort") and self._listeningPort is not None:
self._listeningPort.stopListening()
#def __del__(self):
# log.warning("unclean shutdown of the dht node")
# if hasattr(self, "_listeningPort") and self._listeningPort is not None:
# self._listeningPort.stopListening()
def __str__(self):
return '<%s.%s object; ID: %s, IP address: %s, UDP port: %d>' % (
self.__module__, self.__class__.__name__, binascii.hexlify(self.node_id), self.externalIP, self.port)
@defer.inlineCallbacks
def stop(self):
@@ -171,7 +150,8 @@ class Node(MockKademliaHelper):
def start_listening(self):
if not self._listeningPort:
try:
self._listeningPort = self.reactor_listenUDP(self.port, self._protocol)
self._listeningPort = self.reactor_listenUDP(self.port, self._protocol,
interface=self._listen_interface)
except error.CannotListenError as e:
import traceback
log.error("Couldn't bind to port %d. %s", self.port, traceback.format_exc())
@@ -200,7 +180,7 @@ class Node(MockKademliaHelper):
if not known_node_resolution:
known_node_resolution = yield _resolve_seeds()
# we are one of the seed nodes, don't add ourselves
if (self.externalIP, self.port) in known_node_resolution.itervalues():
if (self.externalIP, self.port) in known_node_resolution.values():
del known_node_resolution[(self.externalIP, self.port)]
known_node_addresses.remove((self.externalIP, self.port))
@@ -213,7 +193,7 @@ class Node(MockKademliaHelper):
def _initialize_routing():
bootstrap_contacts = []
contact_addresses = {(c.address, c.port): c for c in self.contacts}
for (host, port), ip_address in known_node_resolution.iteritems():
for (host, port), ip_address in known_node_resolution.items():
if (host, port) not in contact_addresses:
# Create temporary contact information for the list of addresses of known nodes
# The contact node id will be set with the responding node id when we initialize it to None
@@ -228,7 +208,7 @@ class Node(MockKademliaHelper):
if not bootstrap_contacts:
log.warning("no bootstrap contacts to ping")
ping_result = yield _ping_contacts(bootstrap_contacts)
shortlist = ping_result.keys()
shortlist = list(ping_result.keys())
if not shortlist:
log.warning("failed to ping %i bootstrap contacts", len(bootstrap_contacts))
defer.returnValue(None)
@@ -236,10 +216,10 @@ class Node(MockKademliaHelper):
# find the closest peers to us
closest = yield self._iterativeFind(self.node_id, shortlist if not self.contacts else None)
yield _ping_contacts(closest)
# # query random hashes in our bucket key ranges to fill or split them
# random_ids_in_range = self._routingTable.getRefreshList()
# while random_ids_in_range:
# yield self.iterativeFindNode(random_ids_in_range.pop())
# query random hashes in our bucket key ranges to fill or split them
random_ids_in_range = self._routingTable.getRefreshList()
while random_ids_in_range:
yield self.iterativeFindNode(random_ids_in_range.pop())
defer.returnValue(None)
@defer.inlineCallbacks
@@ -263,7 +243,7 @@ class Node(MockKademliaHelper):
yield _iterative_join()
@defer.inlineCallbacks
def start(self, known_node_addresses=None):
def start(self, known_node_addresses=None, block_on_join=False):
""" Causes the Node to attempt to join the DHT network by contacting the
known DHT nodes. This can be called multiple times if the previous attempt
has failed or if the Node has lost all the contacts.
@@ -278,8 +258,13 @@ class Node(MockKademliaHelper):
self.start_listening()
yield self._protocol._listening
# TODO: Refresh all k-buckets further away than this node's closest neighbour
yield self.joinNetwork(known_node_addresses or [])
d = self.joinNetwork(known_node_addresses or [])
d.addCallback(lambda _: self.start_looping_calls())
d.addCallback(lambda _: log.info("Joined the dht"))
if block_on_join:
yield d
def start_looping_calls(self):
self.safe_start_looping_call(self._change_token_lc, constants.tokenSecretChangeInterval)
# Start refreshing k-buckets periodically, if necessary
self.safe_start_looping_call(self._refresh_node_lc, constants.checkRefreshInterval)
@@ -305,25 +290,24 @@ class Node(MockKademliaHelper):
@defer.inlineCallbacks
def storeToContact(self, blob_hash, contact):
try:
token = contact.token
if not token:
find_value_response = yield contact.findValue(blob_hash)
token = find_value_response['token']
contact.update_token(token)
res = yield contact.store(blob_hash, token, self.peerPort, self.node_id, 0)
if res != "OK":
if not contact.token:
yield contact.findValue(blob_hash)
res = yield contact.store(blob_hash, contact.token, self.peerPort, self.node_id, 0)
if res != b"OK":
raise ValueError(res)
defer.returnValue(True)
log.debug("Stored %s to %s (%s)", binascii.hexlify(blob_hash), contact.log_id(), contact.address)
return True
except protocol.TimeoutError:
log.debug("Timeout while storing blob_hash %s at %s",
binascii.hexlify(blob_hash), contact.log_id())
except ValueError as err:
log.error("Unexpected response: %s" % err.message)
log.error("Unexpected response: %s" % err)
except Exception as err:
if 'Invalid token' in str(err):
contact.update_token(None)
log.error("Unexpected error while storing blob_hash %s at %s: %s",
binascii.hexlify(blob_hash), contact, err)
defer.returnValue(False)
return False
@defer.inlineCallbacks
def announceHaveBlob(self, blob_hash):
@@ -332,9 +316,7 @@ class Node(MockKademliaHelper):
if not self.externalIP:
raise Exception("Cannot determine external IP: %s" % self.externalIP)
stored_to = yield DeferredDict({contact: self.storeToContact(blob_hash, contact) for contact in contacts})
contacted_node_ids = map(
lambda contact: contact.id.encode('hex'), filter(lambda contact: stored_to[contact], stored_to.keys())
)
contacted_node_ids = [binascii.hexlify(contact.id) for contact in stored_to.keys() if stored_to[contact]]
log.debug("Stored %s to %i of %i attempted peers", binascii.hexlify(blob_hash),
len(contacted_node_ids), len(contacts))
defer.returnValue(contacted_node_ids)
@@ -375,7 +357,7 @@ class Node(MockKademliaHelper):
return self._iterativeFind(key)
@defer.inlineCallbacks
def iterativeFindValue(self, key):
def iterativeFindValue(self, key, exclude=None):
""" The Kademlia search operation (deterministic)
Call this to retrieve data from the DHT.
@@ -396,11 +378,11 @@ class Node(MockKademliaHelper):
@rtype: twisted.internet.defer.Deferred
"""
if len(key) != constants.key_bits / 8:
if len(key) != constants.key_bits // 8:
raise ValueError("invalid key length!")
# Execute the search
find_result = yield self._iterativeFind(key, rpc='findValue')
find_result = yield self._iterativeFind(key, rpc='findValue', exclude=exclude)
if isinstance(find_result, dict):
# We have found the value; now see who was the closest contact without it...
# ...and store the key/value pair
@@ -418,22 +400,15 @@ class Node(MockKademliaHelper):
else:
pass
expanded_peers = []
if find_result:
if key in find_result:
for peer in find_result[key]:
expanded = expand_peer(peer)
if expanded not in expanded_peers:
expanded_peers.append(expanded)
# TODO: get this working
# if 'closestNodeNoValue' in find_result:
# closest_node_without_value = find_result['closestNodeNoValue']
# try:
# response, address = yield closest_node_without_value.findValue(key, rawResponse=True)
# yield closest_node_without_value.store(key, response.response['token'], self.peerPort)
# except TimeoutError:
# pass
defer.returnValue(expanded_peers)
defer.returnValue(list(set(find_result.get(key, []) if find_result else [])))
# TODO: get this working
# if 'closestNodeNoValue' in find_result:
# closest_node_without_value = find_result['closestNodeNoValue']
# try:
# response, address = yield closest_node_without_value.findValue(key, rawResponse=True)
# yield closest_node_without_value.store(key, response.response['token'], self.peerPort)
# except TimeoutError:
# pass
def addContact(self, contact):
""" Add/update the given contact; simple wrapper for the same method
@@ -466,20 +441,10 @@ class Node(MockKademliaHelper):
@rtype: twisted.internet.defer.Deferred
"""
try:
contact = self._routingTable.getContact(contactID)
df = defer.Deferred()
df.callback(contact)
df = defer.succeed(self._routingTable.getContact(contactID))
except (ValueError, IndexError):
def parseResults(nodes):
node_ids = [c.id for c in nodes]
if contactID in nodes:
contact = nodes[node_ids.index(contactID)]
return contact
else:
return None
df = self.iterativeFindNode(contactID)
df.addCallback(parseResults)
df.addCallback(lambda nodes: ([node for node in nodes if node.id == contactID] or (None,))[0])
return df
@rpcmethod
@@ -488,7 +453,7 @@ class Node(MockKademliaHelper):
@rtype: str
"""
return 'pong'
return b'pong'
@rpcmethod
def store(self, rpc_contact, blob_hash, token, port, originalPublisherID, age):
@@ -518,18 +483,20 @@ class Node(MockKademliaHelper):
if originalPublisherID is None:
originalPublisherID = rpc_contact.id
compact_ip = rpc_contact.compact_ip()
if not self.verify_token(token, compact_ip):
if self.clock.seconds() - self._protocol.started_listening_time < constants.tokenSecretChangeInterval:
pass
elif not self.verify_token(token, compact_ip):
raise ValueError("Invalid token")
if 0 <= port <= 65536:
compact_port = str(struct.pack('>H', port))
compact_port = port.to_bytes(2, 'big')
else:
raise TypeError('Invalid port')
raise TypeError(f'Invalid port: {port}')
compact_address = compact_ip + compact_port + rpc_contact.id
now = int(self.clock.seconds())
originallyPublished = now - age
self._dataStore.addPeerToBlob(rpc_contact, blob_hash, compact_address, now, originallyPublished,
originalPublisherID)
return 'OK'
return b'OK'
@rpcmethod
def findNode(self, rpc_contact, key):
@@ -545,10 +512,10 @@ class Node(MockKademliaHelper):
node is returning all of the contacts that it knows of.
@rtype: list
"""
if len(key) != constants.key_bits / 8:
if len(key) != constants.key_bits // 8:
raise ValueError("invalid contact id length: %i" % len(key))
contacts = self._routingTable.findCloseNodes(key, constants.k, rpc_contact.id)
contacts = self._routingTable.findCloseNodes(key, sender_node_id=rpc_contact.id)
contact_triples = []
for contact in contacts:
contact_triples.append((contact.id, contact.address, contact.port))
@@ -567,20 +534,33 @@ class Node(MockKademliaHelper):
@rtype: dict or list
"""
if len(key) != constants.key_bits / 8:
if len(key) != constants.key_bits // 8:
raise ValueError("invalid blob hash length: %i" % len(key))
response = {
'token': self.make_token(rpc_contact.compact_ip()),
b'token': self.make_token(rpc_contact.compact_ip()),
}
if self._protocol._protocolVersion:
response['protocolVersion'] = self._protocol._protocolVersion
response[b'protocolVersion'] = self._protocol._protocolVersion
if self._dataStore.hasPeersForBlob(key):
response[key] = self._dataStore.getPeersForBlob(key)
# get peers we have stored for this blob
has_other_peers = self._dataStore.hasPeersForBlob(key)
peers = []
if has_other_peers:
peers.extend(self._dataStore.getPeersForBlob(key))
# if we don't have k storing peers to return and we have this hash locally, include our contact information
if len(peers) < constants.k and key in self._dataStore.completed_blobs:
compact_ip = reduce(lambda buff, x: buff + bytearray([int(x)]), self.externalIP.split('.'), bytearray())
compact_port = self.peerPort.to_bytes(2, 'big')
compact_address = compact_ip + compact_port + self.node_id
peers.append(compact_address)
if peers:
response[key] = peers
else:
response['contacts'] = self.findNode(rpc_contact, key)
response[b'contacts'] = self.findNode(rpc_contact, key)
return response
def _generateID(self):
@@ -591,10 +571,10 @@ class Node(MockKademliaHelper):
"""
return generate_id()
# from lbrynet.core.utils import profile_deferred
# from lbrynet.p2p.utils import profile_deferred
# @profile_deferred()
@defer.inlineCallbacks
def _iterativeFind(self, key, startupShortlist=None, rpc='findNode'):
def _iterativeFind(self, key, startupShortlist=None, rpc='findNode', exclude=None):
""" The basic Kademlia iterative lookup operation (for nodes/values)
This builds a list of k "closest" contacts through iterative use of
@@ -623,16 +603,16 @@ class Node(MockKademliaHelper):
@rtype: twisted.internet.defer.Deferred
"""
if len(key) != constants.key_bits / 8:
if len(key) != constants.key_bits // 8:
raise ValueError("invalid key length: %i" % len(key))
if startupShortlist is None:
shortlist = self._routingTable.findCloseNodes(key, constants.k)
shortlist = self._routingTable.findCloseNodes(key)
# if key != self.node_id:
# # Update the "last accessed" timestamp for the appropriate k-bucket
# self._routingTable.touchKBucket(key)
if len(shortlist) == 0:
log.warning("This node doesnt know any other nodes")
log.warning("This node doesn't know any other nodes")
# This node doesn't know of any other nodes
fakeDf = defer.Deferred()
fakeDf.callback([])
@@ -642,7 +622,7 @@ class Node(MockKademliaHelper):
# This is used during the bootstrap process
shortlist = startupShortlist
result = yield iterativeFind(self, shortlist, key, rpc)
result = yield iterativeFind(self, shortlist, key, rpc, exclude=exclude)
defer.returnValue(result)
@defer.inlineCallbacks
@@ -651,19 +631,14 @@ class Node(MockKademliaHelper):
replication/republishing as necessary """
yield self._refreshRoutingTable()
self._dataStore.removeExpiredPeers()
yield self._refreshStoringPeers()
self._refreshStoringPeers()
defer.returnValue(None)
def _refreshContacts(self):
return defer.DeferredList(
[self._protocol._ping_queue.enqueue_maybe_ping(contact, delay=0) for contact in self.contacts]
)
self._protocol._ping_queue.enqueue_maybe_ping(*self.contacts, delay=0)
def _refreshStoringPeers(self):
storing_contacts = self._dataStore.getStoringContacts()
return defer.DeferredList(
[self._protocol._ping_queue.enqueue_maybe_ping(contact, delay=0) for contact in storing_contacts]
)
self._protocol._ping_queue.enqueue_maybe_ping(*self._dataStore.getStoringContacts(), delay=0)
@defer.inlineCallbacks
def _refreshRoutingTable(self):
-59
View File
@@ -1,59 +0,0 @@
import binascii
import logging
from zope.interface import implements
from twisted.internet import defer
from lbrynet.interfaces import IPeerFinder
log = logging.getLogger(__name__)
class DummyPeerFinder(object):
"""This class finds peers which have announced to the DHT that they have certain blobs"""
def find_peers_for_blob(self, blob_hash, timeout=None, filter_self=True):
return defer.succeed([])
class DHTPeerFinder(DummyPeerFinder):
"""This class finds peers which have announced to the DHT that they have certain blobs"""
implements(IPeerFinder)
def __init__(self, dht_node, peer_manager):
"""
dht_node - an instance of dht.Node class
peer_manager - an instance of PeerManager class
"""
self.dht_node = dht_node
self.peer_manager = peer_manager
self.peers = []
@defer.inlineCallbacks
def find_peers_for_blob(self, blob_hash, timeout=None, filter_self=True):
"""
Find peers for blob in the DHT
blob_hash (str): blob hash to look for
timeout (int): seconds to timeout after
filter_self (bool): if True, and if a peer for a blob is itself, filter it
from the result
Returns:
list of peers for the blob
"""
bin_hash = binascii.unhexlify(blob_hash)
finished_deferred = self.dht_node.iterativeFindValue(bin_hash)
if timeout:
finished_deferred.addTimeout(timeout, self.dht_node.clock)
try:
peer_list = yield finished_deferred
except defer.TimeoutError:
peer_list = []
peers = set(peer_list)
results = []
for node_id, host, port in peers:
if filter_self and (host, port) == (self.dht_node.externalIP, self.dht_node.peerPort):
continue
results.append(self.peer_manager.get_peer(host, port))
defer.returnValue(results)
+100 -101
View File
@@ -1,20 +1,15 @@
import logging
import socket
import errno
from collections import deque
from binascii import hexlify
from twisted.internet import protocol, defer
from error import BUILTIN_EXCEPTIONS, UnknownRemoteException, TimeoutError, TransportNotConnected
import constants
import encoding
import msgtypes
import msgformat
from lbrynet.dht import constants, encoding, msgformat, msgtypes
from lbrynet.dht.error import BUILTIN_EXCEPTIONS, UnknownRemoteException, TimeoutError, TransportNotConnected
log = logging.getLogger(__name__)
class PingQueue(object):
class PingQueue:
"""
Schedules a 15 minute delayed ping after a new node sends us a query. This is so the new node gets added to the
routing table after having been given enough time for a pinhole to expire.
@@ -22,71 +17,53 @@ class PingQueue(object):
def __init__(self, node):
self._node = node
self._get_time = self._node.clock.seconds
self._queue = deque()
self._enqueued_contacts = {}
self._semaphore = defer.DeferredSemaphore(1)
self._ping_semaphore = defer.DeferredSemaphore(constants.alpha)
self._process_lc = node.get_looping_call(self._semaphore.run, self._process)
self._pending_contacts = {}
self._process_lc = node.get_looping_call(self._process)
def _add_contact(self, contact, delay=None):
if contact in self._enqueued_contacts:
return defer.succeed(None)
delay = delay or constants.checkRefreshInterval
self._enqueued_contacts[contact] = self._get_time() + delay
self._queue.append(contact)
return defer.succeed(None)
def enqueue_maybe_ping(self, *contacts, **kwargs):
delay = kwargs.get('delay', constants.checkRefreshInterval)
no_op = (defer.succeed(None), lambda: None)
for contact in contacts:
if delay and contact not in self._enqueued_contacts:
self._pending_contacts.setdefault(contact, self._node.clock.seconds() + delay)
else:
self._enqueued_contacts.setdefault(contact, no_op)
@defer.inlineCallbacks
def _process(self):
if not len(self._queue):
defer.returnValue(None)
contact = self._queue.popleft()
now = self._get_time()
# if the oldest contact in the queue isn't old enough to be pinged, add it back to the queue and return
if now < self._enqueued_contacts[contact]:
self._queue.appendleft(contact)
defer.returnValue(None)
pinged = []
checked = []
while now > self._enqueued_contacts[contact]:
checked.append(contact)
if not contact.contact_is_good:
pinged.append(contact)
if not len(self._queue):
break
contact = self._queue.popleft()
if not now > self._enqueued_contacts[contact]:
checked.append(contact)
@defer.inlineCallbacks
def _ping(contact):
try:
yield contact.ping()
except TimeoutError:
pass
except Exception as err:
log.warning("unexpected error: %s", err)
yield defer.DeferredList([_ping(contact) for contact in pinged])
for contact in checked:
def _ping(self, contact):
if contact.contact_is_good:
return
try:
yield contact.ping()
except TimeoutError:
pass
except Exception as err:
log.warning("unexpected error: %s", err)
finally:
if contact in self._enqueued_contacts:
del self._enqueued_contacts[contact]
defer.returnValue(None)
def _process(self):
# move contacts that are scheduled to join the queue
if self._pending_contacts:
now = self._node.clock.seconds()
for contact in [contact for contact, schedule in self._pending_contacts.items() if schedule <= now]:
del self._pending_contacts[contact]
self._enqueued_contacts.setdefault(contact, (defer.succeed(None), lambda: None))
# spread pings across 60 seconds to avoid flood and/or false negatives
step = 60.0/float(len(self._enqueued_contacts)) if self._enqueued_contacts else 0
for index, (contact, (call, _)) in enumerate(self._enqueued_contacts.items()):
if call.called and not contact.contact_is_good:
self._enqueued_contacts[contact] = self._node.reactor_callLater(index*step, self._ping, contact)
def start(self):
return self._node.safe_start_looping_call(self._process_lc, 60)
def stop(self):
map(None, (cancel() for _, (call, cancel) in self._enqueued_contacts.items() if not call.called))
return self._node.safe_stop_looping_call(self._process_lc)
def enqueue_maybe_ping(self, contact, delay=None):
return self._semaphore.run(self._add_contact, contact, delay)
class KademliaProtocol(protocol.DatagramProtocol):
""" Implements all low-level network-related functions of a Kademlia node """
@@ -95,7 +72,6 @@ class KademliaProtocol(protocol.DatagramProtocol):
def __init__(self, node):
self._node = node
self._encoder = encoding.Bencode()
self._translator = msgformat.DefaultFormat()
self._sentMessages = {}
self._partialMessages = {}
@@ -103,14 +79,15 @@ class KademliaProtocol(protocol.DatagramProtocol):
self._listening = defer.Deferred(None)
self._ping_queue = PingQueue(self._node)
self._protocolVersion = constants.protocolVersion
self.started_listening_time = 0
def _migrate_incoming_rpc_args(self, contact, method, *args):
if method == 'store' and contact.protocolVersion == 0:
if method == b'store' and contact.protocolVersion == 0:
if isinstance(args[1], dict):
blob_hash = args[0]
token = args[1].pop('token', None)
port = args[1].pop('port', -1)
originalPublisherID = args[1].pop('lbryid', None)
token = args[1].pop(b'token', None)
port = args[1].pop(b'port', -1)
originalPublisherID = args[1].pop(b'lbryid', None)
age = 0
return (blob_hash, token, port, originalPublisherID, age), {}
return args, {}
@@ -121,18 +98,36 @@ class KademliaProtocol(protocol.DatagramProtocol):
protocol version keyword argument to calls to contacts who will accept it
"""
if contact.protocolVersion == 0:
if method == 'store':
if method == b'store':
blob_hash, token, port, originalPublisherID, age = args
args = (blob_hash, {'token': token, 'port': port, 'lbryid': originalPublisherID}, originalPublisherID,
False)
args = (
blob_hash, {
b'token': token,
b'port': port,
b'lbryid': originalPublisherID
}, originalPublisherID, False
)
return args
return args
if args and isinstance(args[-1], dict):
args[-1]['protocolVersion'] = self._protocolVersion
args[-1][b'protocolVersion'] = self._protocolVersion
return args
return args + ({'protocolVersion': self._protocolVersion},)
return args + ({b'protocolVersion': self._protocolVersion},)
@defer.inlineCallbacks
def sendRPC(self, contact, method, args):
for _ in range(constants.rpcAttempts):
try:
response = yield self._sendRPC(contact, method, args)
return response
except TimeoutError:
if contact.contact_is_good:
log.debug("RETRY %s ON %s", method, contact)
continue
else:
raise
def _sendRPC(self, contact, method, args):
"""
Sends an RPC to the specified contact
@@ -155,11 +150,11 @@ class KademliaProtocol(protocol.DatagramProtocol):
msg = msgtypes.RequestMessage(self._node.node_id, method, self._migrate_outgoing_rpc_args(contact, method,
*args))
msgPrimitive = self._translator.toPrimitive(msg)
encodedMsg = self._encoder.encode(msgPrimitive)
encodedMsg = encoding.bencode(msgPrimitive)
if args:
log.debug("%s:%i SEND CALL %s(%s) TO %s:%i", self._node.externalIP, self._node.port, method,
args[0].encode('hex'), contact.address, contact.port)
hexlify(args[0]), contact.address, contact.port)
else:
log.debug("%s:%i SEND CALL %s TO %s:%i", self._node.externalIP, self._node.port, method,
contact.address, contact.port)
@@ -167,20 +162,23 @@ class KademliaProtocol(protocol.DatagramProtocol):
df = defer.Deferred()
def _remove_contact(failure): # remove the contact from the routing table and track the failure
contact.update_last_failed()
try:
self._node.removeContact(contact)
if not contact.contact_is_good:
self._node.removeContact(contact)
except (ValueError, IndexError):
pass
contact.update_last_failed()
return failure
def _update_contact(result): # refresh the contact in the routing table
contact.update_last_replied()
if method == 'findValue':
if 'protocolVersion' not in result:
if method == b'findValue':
if b'token' in result:
contact.update_token(result[b'token'])
if b'protocolVersion' not in result:
contact.update_protocol_version(0)
else:
contact.update_protocol_version(result.pop('protocolVersion'))
contact.update_protocol_version(result.pop(b'protocolVersion'))
d = self._node.addContact(contact)
d.addCallback(lambda _: result)
return d
@@ -198,10 +196,11 @@ class KademliaProtocol(protocol.DatagramProtocol):
return df
def startProtocol(self):
log.info("DHT listening on UDP %s:%i", self._node.externalIP, self._node.port)
log.info("DHT listening on UDP %i (ext port %i)", self._node.port, self._node.externalUDPPort)
if self._listening.called:
self._listening = defer.Deferred()
self._listening.callback(True)
self.started_listening_time = self._node.clock.seconds()
return self._ping_queue.start()
def datagramReceived(self, datagram, address):
@@ -210,18 +209,17 @@ class KademliaProtocol(protocol.DatagramProtocol):
@note: This is automatically called by Twisted when the protocol
receives a UDP datagram
"""
if datagram[0] == '\x00' and datagram[25] == '\x00':
totalPackets = (ord(datagram[1]) << 8) | ord(datagram[2])
if chr(datagram[0]) == '\x00' and chr(datagram[25]) == '\x00':
totalPackets = (datagram[1] << 8) | datagram[2]
msgID = datagram[5:25]
seqNumber = (ord(datagram[3]) << 8) | ord(datagram[4])
seqNumber = (datagram[3] << 8) | datagram[4]
if msgID not in self._partialMessages:
self._partialMessages[msgID] = {}
self._partialMessages[msgID][seqNumber] = datagram[26:]
if len(self._partialMessages[msgID]) == totalPackets:
keys = self._partialMessages[msgID].keys()
keys.sort()
data = ''
data = b''
for key in keys:
data += self._partialMessages[msgID][key]
datagram = data
@@ -229,11 +227,11 @@ class KademliaProtocol(protocol.DatagramProtocol):
else:
return
try:
msgPrimitive = self._encoder.decode(datagram)
msgPrimitive = encoding.bdecode(datagram)
message = self._translator.fromPrimitive(msgPrimitive)
except (encoding.DecodeError, ValueError) as err:
# We received some rubbish here
log.warning("Error decoding datagram %s from %s:%i - %s", datagram.encode('hex'),
log.warning("Error decoding datagram %s from %s:%i - %s", hexlify(datagram),
address[0], address[1], err)
return
except (IndexError, KeyError):
@@ -303,7 +301,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
# the node id of the node we sent a message to (these messages are treated as an error)
if remoteContact.id and remoteContact.id != message.nodeID: # sent_to_id will be None for bootstrap
log.debug("mismatch: (%s) %s:%i (%s vs %s)", method, remoteContact.address, remoteContact.port,
remoteContact.log_id(False), message.nodeID.encode('hex'))
remoteContact.log_id(False), hexlify(message.nodeID))
df.errback(TimeoutError(remoteContact.id))
return
elif not remoteContact.id:
@@ -323,7 +321,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
If the data is spread over multiple UDP datagrams, the packets have the
following structure::
| | | | | |||||||||||| 0x00 |
|Transmision|Total number|Sequence number| RPC ID |Header end|
|Transmission|Total number|Sequence number| RPC ID |Header end|
| type ID | of packets |of this packet | | indicator|
| (1 byte) | (2 bytes) | (2 bytes) |(20 bytes)| (1 byte) |
| | | | | |||||||||||| |
@@ -341,7 +339,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
# 1st byte is transmission type id, bytes 2 & 3 are the
# total number of packets in this transmission, bytes 4 &
# 5 are the sequence number for this specific packet
totalPackets = len(data) / self.msgSizeLimit
totalPackets = len(data) // self.msgSizeLimit
if len(data) % self.msgSizeLimit > 0:
totalPackets += 1
encTotalPackets = chr(totalPackets >> 8) + chr(totalPackets & 0xff)
@@ -350,7 +348,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
while seqNumber < totalPackets:
packetData = data[startPos:startPos + self.msgSizeLimit]
encSeqNumber = chr(seqNumber >> 8) + chr(seqNumber & 0xff)
txData = '\x00%s%s%s\x00%s' % (encTotalPackets, encSeqNumber, rpcID, packetData)
txData = f'\x00{encTotalPackets}{encSeqNumber}{rpcID}\x00{packetData}'
self._scheduleSendNext(txData, address)
startPos += self.msgSizeLimit
@@ -366,10 +364,10 @@ class KademliaProtocol(protocol.DatagramProtocol):
if self.transport:
try:
self.transport.write(txData, address)
except socket.error as err:
except OSError as err:
if err.errno == errno.EWOULDBLOCK:
# i'm scared this may swallow important errors, but i get a million of these
# on Linux and it doesnt seem to affect anything -grin
# on Linux and it doesn't seem to affect anything -grin
log.warning("Can't send data to dht: EWOULDBLOCK")
elif err.errno == errno.ENETUNREACH:
# this should probably try to retransmit when the network connection is back
@@ -386,15 +384,16 @@ class KademliaProtocol(protocol.DatagramProtocol):
"""
msg = msgtypes.ResponseMessage(rpcID, self._node.node_id, response)
msgPrimitive = self._translator.toPrimitive(msg)
encodedMsg = self._encoder.encode(msgPrimitive)
encodedMsg = encoding.bencode(msgPrimitive)
self._send(encodedMsg, rpcID, (contact.address, contact.port))
def _sendError(self, contact, rpcID, exceptionType, exceptionMessage):
""" Send an RPC error message to the specified contact
"""
exceptionMessage = exceptionMessage.encode()
msg = msgtypes.ErrorMessage(rpcID, self._node.node_id, exceptionType, exceptionMessage)
msgPrimitive = self._translator.toPrimitive(msg)
encodedMsg = self._encoder.encode(msgPrimitive)
encodedMsg = encoding.bencode(msgPrimitive)
self._send(encodedMsg, rpcID, (contact.address, contact.port))
def _handleRPC(self, senderContact, rpcID, method, args):
@@ -412,7 +411,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
df.addErrback(handleError)
# Execute the RPC
func = getattr(self._node, method, None)
func = getattr(self._node, method.decode(), None)
if callable(func) and hasattr(func, "rpcmethod"):
# Call the exposed Node method and return the result to the deferred callback chain
# if args:
@@ -421,19 +420,19 @@ class KademliaProtocol(protocol.DatagramProtocol):
# else:
log.debug("%s:%i RECV CALL %s %s:%i", self._node.externalIP, self._node.port, method,
senderContact.address, senderContact.port)
if args and isinstance(args[-1], dict) and 'protocolVersion' in args[-1]: # args don't need reformatting
senderContact.update_protocol_version(int(args[-1].pop('protocolVersion')))
if args and isinstance(args[-1], dict) and b'protocolVersion' in args[-1]: # args don't need reformatting
senderContact.update_protocol_version(int(args[-1].pop(b'protocolVersion')))
a, kw = tuple(args[:-1]), args[-1]
else:
senderContact.update_protocol_version(0)
a, kw = self._migrate_incoming_rpc_args(senderContact, method, *args)
try:
if method != 'ping':
if method != b'ping':
result = func(senderContact, *a)
else:
result = func()
except Exception, e:
log.exception("error handling request for %s:%i %s", senderContact.address, senderContact.port, method)
except Exception as e:
log.error("error handling request for %s:%i %s", senderContact.address, senderContact.port, method)
df.errback(e)
else:
df.callback(result)
@@ -450,7 +449,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
log.error("deferred timed out, but is not present in sent messages list!")
return
remoteContact, df, timeout_call, timeout_canceller, method, args = self._sentMessages[messageID]
if self._partialMessages.has_key(messageID):
if messageID in self._partialMessages:
# We are still receiving this message
self._msgTimeoutInProgress(messageID, timeout_canceller, remoteContact, df, method, args)
return
@@ -476,7 +475,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
def _hasProgressBeenMade(self, messageID):
return (
self._partialMessagesProgress.has_key(messageID) and
messageID in self._partialMessagesProgress and
(
len(self._partialMessagesProgress[messageID]) !=
len(self._partialMessages[messageID])
+45 -70
View File
@@ -6,19 +6,17 @@
# may be created by processing this file with epydoc: http://epydoc.sf.net
import random
from zope.interface import implements
from twisted.internet import defer
import constants
import kbucket
from error import TimeoutError
from distance import Distance
from interface import IRoutingTable
import logging
from twisted.internet import defer
from lbrynet.dht import constants, kbucket
from lbrynet.dht.error import TimeoutError
from lbrynet.dht.distance import Distance
log = logging.getLogger(__name__)
class TreeRoutingTable(object):
class TreeRoutingTable:
""" This class implements a routing table used by a Node class.
The Kademlia routing table is a binary tree whFose leaves are k-buckets,
@@ -33,7 +31,7 @@ class TreeRoutingTable(object):
C{PING} RPC-based k-bucket eviction algorithm described in section 2.2 of
that paper.
"""
implements(IRoutingTable)
#implements(IRoutingTable)
def __init__(self, parentNodeID, getTime=None):
"""
@@ -48,6 +46,7 @@ class TreeRoutingTable(object):
from twisted.internet import reactor
getTime = reactor.seconds
self._getTime = getTime
self._ongoing_replacements = set()
def get_contacts(self):
contacts = []
@@ -134,23 +133,29 @@ class TreeRoutingTable(object):
to_replace = not_good_contacts[0]
else:
to_replace = self._buckets[bucketIndex]._contacts[0]
df = to_replace.ping()
df.addErrback(replaceContact, to_replace)
if to_replace not in self._ongoing_replacements:
log.debug("pinging %s:%s", to_replace.address, to_replace.port)
self._ongoing_replacements.add(to_replace)
df = to_replace.ping()
df.addErrback(replaceContact, to_replace)
df.addBoth(lambda _: self._ongoing_replacements.remove(to_replace))
else:
df = defer.succeed(None)
return df
else:
self.touchKBucketByIndex(bucketIndex)
return defer.succeed(None)
def findCloseNodes(self, key, count, sender_node_id=None):
def findCloseNodes(self, key, count=None, sender_node_id=None):
""" Finds a number of known nodes closest to the node/value with the
specified key.
@param key: the n-bit key (i.e. the node or value ID) to search for
@type key: str
@param count: the amount of contacts to return
@param count: the amount of contacts to return, default of k (8)
@type count: int
@param sender_node_id: Used during RPC, this is be the sender's Node ID
Whatever ID is passed in the paramater will get
Whatever ID is passed in the parameter will get
excluded from the list of returned contacts.
@type sender_node_id: str
@@ -161,45 +166,17 @@ class TreeRoutingTable(object):
node is returning all of the contacts that it knows of.
@rtype: list
"""
bucketIndex = self._kbucketIndex(key)
if bucketIndex < len(self._buckets):
# sort these
closestNodes = self._buckets[bucketIndex].getContacts(count, sender_node_id, sort_distance_to=key)
else:
closestNodes = []
# This method must return k contacts (even if we have the node
# with the specified key as node ID), unless there is less
# than k remote nodes in the routing table
i = 1
canGoLower = bucketIndex - i >= 0
canGoHigher = bucketIndex + i < len(self._buckets)
def get_remain(closest):
return min(count, constants.k) - len(closest)
exclude = [self._parentNodeID]
if sender_node_id:
exclude.append(sender_node_id)
if key in exclude:
exclude.remove(key)
count = count or constants.k
distance = Distance(key)
while len(closestNodes) < min(count, constants.k) and (canGoLower or canGoHigher):
iteration_contacts = []
# get contacts from lower and/or higher buckets without sorting them
if canGoLower and len(closestNodes) < min(count, constants.k):
lower_bucket = self._buckets[bucketIndex - i]
contacts = lower_bucket.getContacts(get_remain(closestNodes), sender_node_id, sort_distance_to=False)
iteration_contacts.extend(contacts)
canGoLower = bucketIndex - (i + 1) >= 0
if canGoHigher and len(closestNodes) < min(count, constants.k):
higher_bucket = self._buckets[bucketIndex + i]
contacts = higher_bucket.getContacts(get_remain(closestNodes), sender_node_id, sort_distance_to=False)
iteration_contacts.extend(contacts)
canGoHigher = bucketIndex + (i + 1) < len(self._buckets)
i += 1
# sort the combined contacts and add as many as possible/needed to the combined contact list
iteration_contacts.sort(key=lambda c: distance(c.id), reverse=True)
while len(iteration_contacts) and len(closestNodes) < min(count, constants.k):
closestNodes.append(iteration_contacts.pop())
return closestNodes
contacts = self.get_contacts()
contacts = [c for c in contacts if c.id not in exclude]
contacts.sort(key=lambda c: distance(c.id))
return contacts[:min(count, len(contacts))]
def getContact(self, contactID):
""" Returns the (known) contact with the specified node ID
@@ -208,12 +185,7 @@ class TreeRoutingTable(object):
by this node
"""
bucketIndex = self._kbucketIndex(contactID)
try:
contact = self._buckets[bucketIndex].getContact(contactID)
except ValueError:
raise
else:
return contact
return self._buckets[bucketIndex].getContact(contactID)
def getRefreshList(self, startIndex=0, force=False):
""" Finds all k-buckets that need refreshing, starting at the
@@ -241,7 +213,7 @@ class TreeRoutingTable(object):
now = int(self._getTime())
for bucket in self._buckets[startIndex:]:
if force or now - bucket.lastAccessed >= constants.refreshTimeout:
searchID = self._randomIDInBucketRange(bucketIndex)
searchID = self.midpoint_id_in_bucket_range(bucketIndex)
refreshIDs.append(searchID)
bucketIndex += 1
return refreshIDs
@@ -289,22 +261,25 @@ class TreeRoutingTable(object):
i += 1
return i
def _randomIDInBucketRange(self, bucketIndex):
def random_id_in_bucket_range(self, bucketIndex):
""" Returns a random ID in the specified k-bucket's range
@param bucketIndex: The index of the k-bucket to use
@type bucketIndex: int
"""
idValue = random.randrange(
self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax)
randomID = hex(idValue)[2:]
if randomID[-1] == 'L':
randomID = randomID[:-1]
if len(randomID) % 2 != 0:
randomID = '0' + randomID
randomID = randomID.decode('hex')
randomID = (constants.key_bits / 8 - len(randomID)) * '\x00' + randomID
return randomID
random_id = int(random.randrange(self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax))
return random_id.to_bytes(constants.key_bits // 8, 'big')
def midpoint_id_in_bucket_range(self, bucketIndex):
""" Returns the middle ID in the specified k-bucket's range
@param bucketIndex: The index of the k-bucket to use
@type bucketIndex: int
"""
half = int((self._buckets[bucketIndex].rangeMax - self._buckets[bucketIndex].rangeMin) // 2)
return int(self._buckets[bucketIndex].rangeMin + half).to_bytes(constants.key_bits // 8, 'big')
def _splitBucket(self, oldBucketIndex):
""" Splits the specified k-bucket into two new buckets which together
File renamed without changes.
+236
View File
@@ -0,0 +1,236 @@
import sys
import json
import asyncio
import argparse
import logging
from docopt import docopt
from textwrap import dedent
import aiohttp
from lbrynet.extras.compat import force_asyncioreactor_install
force_asyncioreactor_install()
from lbrynet import log_support, __name__ as lbrynet_name, __version__ as lbrynet_version
from lbrynet.extras.daemon.loggly_handler import get_loggly_handler
from lbrynet.conf import Config, CLIConfig
from lbrynet.utils import check_connection
from lbrynet.extras.daemon.Daemon import Daemon
log = logging.getLogger(lbrynet_name)
log.addHandler(logging.NullHandler())
def display(data):
print(json.dumps(data, indent=2))
async def execute_command(conf, method, params):
async with aiohttp.ClientSession() as session:
try:
message = {'method': method, 'params': params}
async with session.get(conf.api_connection_url, json=message) as resp:
try:
data = await resp.json()
if 'result' in data:
display(data['result'])
elif 'error' in data:
if 'message' in data['error']:
display(data['error']['message'])
else:
display(data['error'])
except Exception as e:
log.exception('Could not process response from server:', exc_info=e)
except aiohttp.ClientConnectionError:
print("Could not connect to daemon. Are you sure it's running?")
def print_help():
print(dedent("""
NAME
lbrynet - LBRY command line client.
USAGE
lbrynet [--data_dir=<blob and database directory>] [--wallet_dir=<wallet directory>]
[--download_dir=<downloads directory>] <command> [<args>]
EXAMPLES
lbrynet start # starts the daemon and listens for jsonrpc commands
lbrynet help # display this message
lbrynet help <command_name> # get help for a command(doesn't need the daemon to be running)
lbrynet commands # list available commands
lbrynet status # get the running status of the daemon
lbrynet resolve what # resolve a name
lbrynet --wallet_dir=~/wallet2 start # start the daemon using an alternative wallet directory
lbrynet --data_dir=~/lbry start # start the daemon using an alternative data directory
lbrynet --data_dir=~/lbry <command_name> # run a command on a daemon using an alternative data directory,
# which can contain a full daemon_settings.yml config file.
# Note: since the daemon is what runs the wallet and
# downloads files, only the --data_dir setting is needed when
# running commands. The wallet_dir and download_dir would only
# by used when starting the daemon.
"""))
def print_help_for_command(command):
fn = Daemon.callable_methods.get(command)
if fn:
print(dedent(fn.__doc__))
else:
print("Invalid command name")
def normalize_value(x, key=None):
if not isinstance(x, str):
return x
if key in ('uri', 'channel_name', 'name', 'file_name', 'download_directory'):
return x
if x.lower() == 'true':
return True
if x.lower() == 'false':
return False
if x.isdigit():
return int(x)
return x
def remove_brackets(key):
if key.startswith("<") and key.endswith(">"):
return str(key[1:-1])
return key
def set_kwargs(parsed_args):
kwargs = {}
for key, arg in parsed_args.items():
k = None
if arg is None:
continue
elif key.startswith("--") and remove_brackets(key[2:]) not in kwargs:
k = remove_brackets(key[2:])
elif remove_brackets(key) not in kwargs:
k = remove_brackets(key)
kwargs[k] = normalize_value(arg, k)
return kwargs
def get_argument_parser():
main = argparse.ArgumentParser('lbrynet', add_help=False)
main.add_argument(
'--version', dest='cli_version', action="store_true",
help='Show lbrynet CLI version and exit.'
)
main.add_argument(
'-h', '--help', dest='help', action="store_true",
help='Show this help message and exit'
)
CLIConfig.contribute_args(main)
sub = main.add_subparsers(dest='command')
help = sub.add_parser('help', help='Detailed help for remote commands.')
help.add_argument('help_command', nargs='*')
start = sub.add_parser('start', help='Start lbrynet server.')
start.add_argument(
'--quiet', dest='quiet', action="store_true",
help='Disable all console output.'
)
start.add_argument(
'--verbose', nargs="*",
help=('Enable debug output. Optionally specify loggers for which debug output '
'should selectively be applied.')
)
Config.contribute_args(start)
api = Daemon.get_api_definitions()
for group in sorted(api):
group_command = sub.add_parser(group, help=api[group]['doc'])
group_command.set_defaults(group_doc=group_command)
if group in ('status', 'publish', 'version', 'help', 'wallet_balance', 'get'):
continue
commands = group_command.add_subparsers(dest='subcommand')
for command in api[group]['commands']:
commands.add_parser(command['name'], help=command['doc'].strip().splitlines()[0])
for deprecated in Daemon.deprecated_methods:
group_command = sub.add_parser(deprecated)
group_command.add_subparsers(dest='subcommand')
return main
def main(argv=None):
argv = argv or sys.argv[1:]
parser = get_argument_parser()
args, command_args = parser.parse_known_args(argv)
conf = Config.create_from_arguments(args)
if args.cli_version:
print(f"{lbrynet_name} {lbrynet_version}")
return 0
elif args.command == 'start':
log_support.configure_logging(conf.log_file_path, not args.quiet, args.verbose)
if conf.share_usage_data:
loggly_handler = get_loggly_handler()
loggly_handler.setLevel(logging.ERROR)
log.addHandler(loggly_handler)
log.debug('Final Settings: %s', conf.settings_dict)
log.info("Starting lbrynet-daemon from command line")
daemon = Daemon(conf)
if check_connection():
from twisted.internet import reactor
reactor._asyncioEventloop.create_task(daemon.start())
reactor.run()
else:
log.info("Not connected to internet, unable to start")
elif args.command == 'help':
if args.help_command:
method = '_'.join(args.help_command)
else:
parser.print_help()
return 0
if method not in Daemon.callable_methods:
print('Invalid command name: {method}')
return 1
fn = Daemon.callable_methods[method]
print(fn.__doc__)
elif args.command is not None:
if args.command in ('status', 'publish', 'version', 'help', 'wallet_balance', 'get'):
method = args.command
elif args.subcommand is not None:
method = f'{args.command}_{args.subcommand}'
else:
args.group_doc.print_help()
return 0
if method in Daemon.deprecated_methods:
new_method = Daemon.deprecated_methods[method].new_command
if new_method is None:
print(f"{method} is permanently deprecated and does not have a replacement command.")
return 0
print(f"{method} is deprecated, using {new_method}.")
method = new_method
fn = Daemon.callable_methods[method]
parsed = docopt(fn.__doc__, command_args)
params = set_kwargs(parsed)
loop = asyncio.get_event_loop()
loop.run_until_complete(execute_command(conf, method, params))
else:
parser.print_help()
return 0
if __name__ == "__main__":
sys.exit(main())
+25
View File
@@ -0,0 +1,25 @@
import asyncio
def force_asyncioreactor_install():
import sys
from twisted.internet import asyncioreactor
if 'twisted.internet.reactor' not in sys.modules:
asyncioreactor.install()
else:
from twisted.internet import reactor
if not isinstance(reactor, asyncioreactor.AsyncioSelectorReactor) and getattr(sys, 'frozen', False):
# pyinstaller hooks install the default reactor before
# any of our code runs, see kivy for similar problem:
# https://github.com/kivy/kivy/issues/4182
del sys.modules['twisted.internet.reactor']
asyncioreactor.install()
def d2f(deferred):
return deferred.asFuture(asyncio.get_event_loop())
def f2d(future):
from twisted.internet import defer
return defer.Deferred.fromFuture(asyncio.ensure_future(future))
Loaded 100 of 342 files, more files were not shown because too many files have changed in this diff. Show more