Compare commits

...
246 Commits
Author SHA1 Message Date
hackrush ba8bd627a8 return cached resolve result for 1 block 2019-01-05 00:29:19 +05:30
hackrush e6528963ad Actually get the certificate_id 2018-12-25 12:16:06 +05:30
Lex Berezhny 6aa709e34e v0.30.3 2018-12-21 18:45:34 -05:00
Lex Berezhny e73f121c08 v0.30.3rc1 2018-12-19 09:12:56 -05:00
Victor Shyba 3b9d6c4f8e Instantiate resolver before the request to hold the selected height 2018-12-19 09:04:30 -05:00
Alyssa Callahan 69b60c19c3 Merge pull request #1712 from lbryio/readme-update
Update README.md
2018-12-17 10:47:29 -05:00
Lex Berezhny 04278cb1ab release script automatically figures out prerelease status 2018-12-14 22:25:06 -05:00
Lex Berezhny e8dec5b440 v0.30.2 2018-12-14 21:50:48 -05:00
Lex Berezhny 75f82b107e updated release script with unittests and ability to go from rc to final release 2018-12-14 21:50:30 -05:00
Jack Robison 0e90c4b611 Merge pull request #1717 from lbryio/fix-directory-args
fix directory cli args and update help message
2018-12-14 17:09:15 -05:00
Jack Robison fb66b68bc8 fix directory cli args, update help message 2018-12-14 15:51:11 -05:00
Victor Shyba c601677386 run schema unit tests on travis 2018-12-14 09:21:31 -05:00
Lex Berezhny 1fbd1cd04e v0.30.2rc2 2018-12-12 19:52:11 -05:00
Jack Robison 4a71b91fbf Merge pull request #1671 from lbryio/remove-android-helpers
remove `lbrynet.androidhelpers` module
2018-12-12 18:35:31 -05:00
Jack Robison b8b0d8b1ac refactor default directory setup, remove lbrynet.androidhelpers
fixes https://github.com/lbryio/lbry/issues/1606
2018-12-12 17:46:10 -05:00
Jack Robison 8ef73d3985 update docs 2018-12-12 12:19:26 -05:00
Jack Robison 05e556a5e3 fix generate_json_api script 2018-12-12 12:18:32 -05:00
Lex Berezhny d1da42bea8 Update README.md
* updated description and added bullet points for main features of this SDK
* lex (@eukreign ) is the official product owner of lbrysdk
2018-12-12 11:47:28 -05:00
Lex Berezhny 90685e8357 deploy release for tags 2018-12-12 00:10:50 -05:00
Lex Berezhny a471d5d167 v0.30.2rc1 2018-12-11 23:44:02 -05:00
Lex Berezhny 16122299e4 revert version back 0.30.1 2018-12-11 23:39:08 -05:00
Lex Berezhny 9f18da4bae release script (#1697)
* travis deploying to github
* release script
2018-12-11 23:24:19 -05:00
Lex Berezhny 1abce52867 v0.30.2rc1 2018-12-11 23:09:31 -05:00
Lex Berezhny 072e2b65d3 revert version change 2018-12-11 22:31:45 -05:00
Lex Berezhny f6d11e5467 0.30.1rc1 2018-12-11 22:29:40 -05:00
Oleg Silkin 458bddcb65 Removes Faker from Dependencies (#1687)
* Removes faker import and its usage
* Replaces data generated with faker with static data
* Drops faker as a dependency & removes extras require section
* Removes installation instruction for the test dependencies in setup.py
2018-12-11 20:19:58 -05:00
Victor Shyba 5fad43ff69 fix test tear down and unify test_commands 2018-12-11 16:34:37 -05:00
Victor Shyba 6992d0d947 add new test to tox.ini 2018-12-11 16:34:37 -05:00
Victor Shyba d1b0180437 add tests for claim_list_by_channel 2018-12-11 16:34:37 -05:00
Victor Shyba ba59e126cd forward errors outside of uris on channel listing 2018-12-11 16:34:37 -05:00
Victor Shyba 4e3467bb7e fix windows release type set script 2018-12-11 16:30:35 -05:00
Lex Berezhny 4d4c705eb9 0.30.1 changelog updates 2018-12-09 23:30:09 -05:00
Lex Berezhny ce7b3184c4 0.30.1 2018-12-09 23:15:11 -05:00
Lex Berezhny b71faddaac upload zip files into the same directory 2018-12-08 19:01:13 -05:00
Lex Berezhny 1328cb4cce check file exists before reading config 2018-12-08 19:01:13 -05:00
Lex Berezhny 946488bce9 correctly save daemon_settings.yml on fresh install 2018-12-08 19:01:13 -05:00
Victor Shyba 7713881e71 additional tests for resolution border cases 2018-12-08 14:40:37 -05:00
Victor Shyba 26fcc48c5a fix for name collision on channel path 2018-12-08 14:40:37 -05:00
Thomas Zarebczan bf12fb68a2 use xcode8.3 for mac build (#1692)
use xcode8.3 for mac build
2018-12-08 14:38:35 -05:00
Jack Robison 3a4c9ff4f9 bump to 0.30.1rc11 2018-12-07 13:07:06 -05:00
Jack Robison 6180a2917d add epub and wmv content types 2018-12-07 13:06:36 -05:00
Lex Berezhny 014cbb4623 v0.30.1rc10 2018-12-07 12:04:28 -05:00
Jack Robison ee98251d79 Merge pull request #1700 from lbryio/mimetypes
improve contentType detection
2018-12-07 12:02:41 -05:00
Jack Robison e3d08ec773 improve contentType detection
fixes https://github.com/lbryio/lbry/issues/1690
2018-12-07 11:31:05 -05:00
Lex Berezhny f03e68e63e 0.30.1rc9 2018-12-06 00:51:07 -05:00
Lex Berezhny 72090b45dd confirmations is returned as int 2018-12-06 00:49:07 -05:00
Lex Berezhny 93f9622798 fix json_response_encoder to also return correct confirmations 2018-12-06 00:49:07 -05:00
Lex Berezhny 16ff4b4e2c confirmations fix 2018-12-06 00:49:07 -05:00
Lex Berezhny c90cca4b20 fix #1683 2018-12-05 14:38:35 -05:00
Lex Berezhny 98d83ea74f fix for integration test 2018-12-05 12:14:43 -05:00
Lex Berezhny a234f92d2c 0.30.1rc8 2018-12-05 11:53:37 -05:00
Lex Berezhny 8ba3575901 dropped aiorpcx 2018-12-05 09:17:36 -05:00
Jack Robison a857c83c59 Merge pull request #1685 from lbryio/fix-1674
fix int too large to convert to integer when updating claim
2018-12-04 18:16:24 -05:00
Jack Robison a6128ac8a2 fix https://github.com/lbryio/lbry/compare/fix-1674 2018-12-04 17:35:40 -05:00
hackrush 2693301ad0 Fix python and pip3 not available in xcode8 2018-12-04 15:12:07 -05:00
Victor Shyba f7c7cb9535 fix transaction_show 2018-12-04 12:12:08 -05:00
Jack Robison b36a5f9b6b Merge pull request #1678 from lbryio/warn_migration_bad_claim_id
check and log claim id validity during migration
2018-11-30 15:58:26 -05:00
Jack Robison 4826eb9969 migrate double hex encoded certificates 2018-11-30 14:00:35 -05:00
Victor Shyba c34b150794 check and log claim id validity during migration 2018-11-30 14:00:35 -05:00
Lex Berezhny c4c7670a7c do not run brew update 2018-11-30 12:20:07 -05:00
Lex Berezhny 0918bac86e mac build stuff 2018-11-30 11:50:32 -05:00
Victor Shyba a60a921706 resolve claims on claim_list (#1483) 2018-11-30 11:46:56 -05:00
Lex Berezhny b3ee29b66d maybe python is already installed on mac 2018-11-30 11:30:03 -05:00
Lex Berezhny 3a1a9b988c on mac brew, python 3 is installed with plain python while python2 now requires python@2 2018-11-30 10:55:15 -05:00
Lex Berezhny 12f2bebc46 build for mac using xcode8 2018-11-30 10:34:45 -05:00
Jack Robison 31a3812c8a bump 2018-11-29 17:23:32 -05:00
Lex Berezhny 70ec08e30b icons! 2018-11-29 17:15:18 -05:00
Jack Robison 23e8f0d849 Merge pull request #1676 from lbryio/fix-1573
unlock wallet after encrypting it
2018-11-29 16:13:21 -05:00
Jack Robison 1cdfcd5b4e unlock wallet after encrypting it 2018-11-29 15:16:53 -05:00
Lex Berezhny dc2786fbc0 assert instead of if 2018-11-28 18:03:33 -05:00
Lex Berezhny 7e39b0c4fe run ClaimManagement.test_create_update_and_abandon_claim 2018-11-28 18:03:33 -05:00
Lex Berezhny 4af830c405 tests for transaction_list going through claim creation -> claim update -> claim abandon 2018-11-28 18:03:33 -05:00
Lex Berezhny fe844581fd correctly split value/fee in transaction_list 2018-11-28 18:03:33 -05:00
Lex Berezhny 443ce6352b typo 2018-11-28 18:03:33 -05:00
Lex Berezhny 158c354a27 split out fee from value when possible 2018-11-28 18:03:33 -05:00
Lex Berezhny 92d1230a90 remove fees from value, abandon is positive 2018-11-28 18:03:33 -05:00
Jack Robison d64916a061 bump 2018-11-27 16:28:55 -05:00
Alyssa Callahan 20a0568253 Merge pull request #1567 from preethamvishy/master
Update pull request template
2018-11-27 15:45:54 -05:00
Alyssa Callahan 5a1f42ee54 Merge branch 'master' into master 2018-11-27 15:39:48 -05:00
Alyssa Callahan ab95f14e21 Update PULL_REQUEST_TEMPLATE.md 2018-11-27 15:38:06 -05:00
Jack Robison bae2676b69 Merge pull request #1669 from lbryio/fix_invalid_sigs_from_name
allow invalid sigs claims to show outside a channel
2018-11-27 14:41:28 -05:00
Victor Shyba ea6869a63e allow invalid sigs claims to show outside a channel 2018-11-27 15:36:53 -03:00
Jack Robison 953327c150 bump 2018-11-27 09:52:33 -05:00
Lex Berezhny e01ad6bc85 disable travis cache on mac build 2018-11-26 20:55:03 -05:00
Victor Shyba 47a33009ff remove python 3.6 steps from travis 2018-11-26 20:55:03 -05:00
Victor Shyba bc844013a1 (win build) add pywin32 and netifaces wheels 2018-11-26 20:55:03 -05:00
Victor Shyba 2fd2319cdb use lbry dockerhub image 2018-11-26 20:55:03 -05:00
Victor Shyba a50657c472 add dockerfile and readme on how win docker builder is generated 2018-11-26 20:55:03 -05:00
Victor Shyba ef5f6e06ba use vshyba/temporary_pyinstaller_32bits_py37:latest 2018-11-26 20:55:03 -05:00
Victor Shyba 977cf9f506 log python version on windows build script 2018-11-26 20:55:03 -05:00
Lex Berezhny ffae10d11f drop pytest 2018-11-26 18:02:12 -05:00
Thomas Zarebczan 221107e76e test mac build, do not merge (#1659)
* Update .travis.yml to use xcode 6.4 for mac build
2018-11-26 14:13:45 -05:00
Jack Robison e74c78b607 Merge pull request #1432 from lbryio/get-block-on-bytes-written
[API] block get on bytes being written to the output file
2018-11-26 12:16:42 -05:00
Jack Robison 434c7b57b1 tests 2018-11-26 11:36:28 -05:00
Jack Robison 9dc4608a62 combine StreamProgressManager and FullStreamProgressManager
-add wrote_first_data Deferred to FullStreamProgressManager
2018-11-26 11:36:28 -05:00
Jack Robison fc69366092 set the download manager before creating the first deferred 2018-11-26 11:36:28 -05:00
Victor Shyba 9f0b1f3e25 fix pylint failures unveiled after astroid pin 2018-11-26 11:32:41 -05:00
Victor Shyba 3f03a845ec only return invalid signatures if its a direct resolution 2018-11-26 11:32:41 -05:00
Victor Shyba bc28976e12 adds test for claims on abandoned channels 2018-11-26 11:32:41 -05:00
Victor Shyba 8f826098f9 validate claim id before resolving 2018-11-26 11:32:41 -05:00
Victor Shyba 0a765e6207 handle errors on resolve 2018-11-26 11:32:41 -05:00
Lex Berezhny f83aa61923 force astroid==2.0.4 in .travis.yml 2018-11-26 10:47:48 -05:00
Oleg Silkin f9ff3e5d14 Removes third-party mock dependency (#1661)
* Removes mock from setup.py dependencies
* Refactors name resetTime to reset_time
* Replaces import mock with built-in unittest.mock
* Refactors imports to separate built-in, external, and project modules
2018-11-25 11:12:30 -05:00
Oleg Silkin b3fde9d78d Removes six From Project (#1660)
* Replaces `six` urllib with python 3's urllib
* Replaces `six`'s int2byte method with native `bytes` class
* Removes edge case testing for python2 vs python3
* Removes useless object inheritance
* Uses native io.BytesIO instead of six.BytesIO
* Removes six from dependencies
2018-11-22 23:24:21 -05:00
Lex Berezhny 0c999083cc added jsonrpc_account_send and dropped torba[server] 2018-11-21 01:14:39 -05:00
Victor Shyba 561215db8a fix download script 2018-11-21 00:25:43 -05:00
Jack Robison 8dbb7f2c78 bump 2018-11-20 21:37:49 -05:00
Lex Berezhny 6f7f3bb316 fixes #1653 2018-11-20 16:12:51 -05:00
Jack Robison 8d507c4b73 bump 2018-11-19 21:12:39 -05:00
Lex Berezhny 2beb0eda94 SupportingSupports test does not exist 2018-11-19 20:34:38 -05:00
Lex Berezhny b7e07b72ca update tox.ini to run correct tests 2018-11-19 20:34:38 -05:00
Lex Berezhny fde8718c8c make sure only actual claims/updates are found when updating a claim or sending it to someone 2018-11-19 20:34:38 -05:00
Lex Berezhny b2d1bddcfe update to latest torba 2018-11-19 20:34:38 -05:00
Lex Berezhny dd1d1be994 working claim_send_to_address 2018-11-19 20:34:38 -05:00
Jack Robison a93957178a bump 2018-11-19 17:31:31 -05:00
Jack Robison 5026d0b2f3 Merge branch 'fix-get-bad-claim-error' 2018-11-19 17:31:01 -05:00
Jack Robison cc8b47d882 improve error when get fails to decode the resolved claim 2018-11-19 17:30:02 -05:00
Jack Robison db0b5961b3 Merge pull request #1642 from jessopb/readmeStuff
adds daemon config to and updates links in readme
2018-11-19 14:37:15 -05:00
Jessop Breth cb6f7ada42 adds sample config and links to settings documentation 2018-11-19 10:10:02 -05:00
Jack Robison 59464099b4 add example daemon_settings.yml 2018-11-19 09:42:13 -05:00
preserveddarnell aed00b53fd Update various typos & edits 2018-11-18 20:52:18 -05:00
YULIUS KURNIAWAN KRISTIANTO cdb0f97cb4 added more social links 2018-11-18 10:29:47 -05:00
Jack Robison 33bcc8256c Merge branch 'fix-1624' 2018-11-16 11:45:05 -05:00
Jack Robison 0a8d44f12f Merge branch 'populate-buckets' 2018-11-16 11:44:25 -05:00
Jack Robison 1b74594f5b don't create ssl/auth keys if disabled
fixes https://github.com/lbryio/lbry/issues/1624
2018-11-16 11:12:33 -05:00
Jack Robison b0718d295e fix empty buckets in routing_table_get 2018-11-16 10:59:12 -05:00
Jack Robison 9bb91b126d use midpoint id instead of random for refresh list 2018-11-16 10:59:12 -05:00
Jack Robison 6dbb00d8d5 populate buckets when joining the dht 2018-11-16 10:59:12 -05:00
Jack Robison 3a27a4509a Merge pull request #1639 from lbryio/fix-1638
fix bytes/str in iterativeFindValue
2018-11-14 16:22:30 -05:00
Jack Robison 1d438f5d23 fixes https://github.com/lbryio/lbry/issues/1638 2018-11-14 15:35:59 -05:00
YULIUS KURNIAWAN KRISTIANTO 3840f19869 fix mistake code 2018-11-14 14:20:02 -05:00
Lex Berezhny ec0483b975 header progress percent starts at 0 instead of None 2018-11-12 21:16:05 -05:00
Jack Robison 2fba1f417f less verbose logging 2018-11-12 14:47:11 -05:00
Jack Robison 5b419b55c5 fix peer_list 2018-11-12 14:45:41 -05:00
Victor Shyba b31379e51c log and handle misformed certificate claims from #1609 2018-11-10 20:49:36 -05:00
Victor Shyba 8394dcb663 log and handle misformed claims from #1611 2018-11-10 20:49:36 -05:00
Lex Berezhny 457a56ce04 Merge pull request #1604 from lbryio/fix-external-ip
Fix setting the external ip
2018-11-10 20:48:40 -05:00
Jack Robison 7c8016d807 remove accidentally added lbrynet.schema.signature 2018-11-09 16:52:46 -05:00
Jack Robison d704336f20 absolute imports 2018-11-09 16:52:46 -05:00
Jack Robison 1b4230b06e move is_valid_blobhash to lbrynet.blob.blob_file 2018-11-09 16:52:46 -05:00
Jack Robison 88095af1cd move system_info.py into lbrynet.extras 2018-11-09 16:52:46 -05:00
Jack Robison e9efd9f68a improve get_external_ip 2018-11-09 16:52:46 -05:00
Jack Robison 7b32e2ff7c don't block the dht component on populating the routing table 2018-11-09 16:52:46 -05:00
Jack Robison 572777305e reduce analytics event frequency 2018-11-09 16:52:46 -05:00
Jack Robison 7e27a6e37d fix 2018-11-09 16:52:46 -05:00
Jack Robison b98621d7ea avoid name errors 2018-11-09 16:52:46 -05:00
Jack Robison bc5824b8bc better setup of custom logger class 2018-11-09 16:52:46 -05:00
Jack Robison a6b643bb34 move files 2018-11-09 16:52:42 -05:00
Jack Robison 488b976734 move HashAnnouncer.py to extras 2018-11-09 14:16:33 -05:00
Jack Robison 253912b52e fix skipping dht, peer_protocol_server, and hash_announcer components
-move PeerFinder and PeerManager initialization to the ComponentManager

-remove dht component requirement from all but the hash_announcer component. This allows running the file manager component without either of the server components.
2018-11-09 14:16:33 -05:00
Jack Robison 0bf8416d9f fix components_to_skip setting 2018-11-09 14:16:33 -05:00
Jack Robison 2e921437d8 fix external ip setup
fixes https://github.com/lbryio/lbry/issues/1561
2018-11-09 14:16:33 -05:00
Jack Robison b6f9b60683 remove commit in version for source installs 2018-11-09 14:16:33 -05:00
Lex Berezhny 57430e9f5f fixed #1616 2018-11-09 12:03:55 -05:00
Victor Shyba 1e13fbee8d await missing on resolve.py 2018-11-09 11:11:57 -05:00
Lex Berezhny 3d8b9e2b66 block=False for abandon during erics epic adventure tests 2018-11-07 18:28:02 -05:00
Lex Berezhny 64898d43f1 tearDown() should run before daemon shutdown 2018-11-07 18:28:02 -05:00
Lex Berezhny b2117fddab on_transaction -> ledger.wait 2018-11-07 18:28:02 -05:00
Lex Berezhny 5658623f29 claims include supports 2018-11-07 18:28:02 -05:00
Lex Berezhny 934682bd0f --blocking support in abandon command 2018-11-07 18:28:02 -05:00
Victor Shyba daecc582f3 simplify bind statement 2018-11-07 11:29:56 -05:00
Victor Shyba 3cb4012dbf unify loop 2018-11-07 11:29:56 -05:00
Victor Shyba f7bd44a4f5 fix query for windows (999 vars limit) 2018-11-07 11:29:56 -05:00
hackrush 52317b05bb Better help message instructions 2018-11-07 11:02:20 -05:00
Lex Berezhny 83c0baa3cb test abandoning claim at a loss, required a fix in torba 2018-11-07 10:57:11 -05:00
Lex Berezhny 340c5addab insufficientfundserror test 2018-11-07 10:57:11 -05:00
Lex Berezhny e3fd94dbf0 pylint fix 2018-11-07 10:57:11 -05:00
Lex Berezhny 923fcfb9e9 run publish integration tests separately 2018-11-07 10:57:11 -05:00
Lex Berezhny 0ae5f1a4c6 added fix for #1533 2018-11-07 10:57:11 -05:00
Lex Berezhny 90e06abba2 fixes for #1569 2018-11-07 10:57:11 -05:00
Victor Shyba 6ecfc94ff1 add balance to startup report 2018-11-07 10:50:13 -05:00
Alyssa Callahan 60c4a727d3 Merge pull request #1574 from czakarian/patch-1
Fix typos / grammar changes
2018-11-07 10:19:59 -05:00
Alyssa Callahan 9fbb9f2e98 Merge branch 'master' into patch-1 2018-11-07 10:18:58 -05:00
Lex Berezhny f1feec66c0 fixed path to cli.py in setup.py 2018-11-06 13:55:22 -05:00
Lex Berezhny 73b9b983b1 pinned protobuf to specific version, fixes #1602 2018-11-06 00:05:40 -05:00
Victor Shyba 588779e3a4 makefile for using types repo 2018-11-05 20:48:21 -05:00
Jack Robison b7ae446059 Merge pull request #1495 from lbryio/retry_dht
[DHT] retry on protocol level
2018-11-05 15:16:53 -05:00
Victor Shyba db898307b5 obey the rpcAttempts constant on retries 2018-11-05 15:16:39 -03:00
Victor Shyba 7e3512cbf2 retry on protocol level 2018-11-05 15:16:39 -03:00
Lex Berezhny cdd65c6783 updated pyinstaller build instructions 2018-11-04 20:25:24 -05:00
Lex Berezhny 320ad717cd pylint cleanup 2018-11-04 20:25:24 -05:00
Lex Berezhny 264e6744e1 updated references after moving conf to lbrynet.extras.daemon.conf 2018-11-04 20:25:24 -05:00
Lex Berezhny 64e2b79c07 moved conf into daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny 9255bb5abe updated references after moving analytics to lbrynet.extras.daemon.analytics 2018-11-04 20:25:24 -05:00
Lex Berezhny d7525dacc3 moved analytics to lbrynet/extra/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny 5cb07100f8 deleted unused interfaces file 2018-11-04 20:25:24 -05:00
Lex Berezhny 171c080d33 updated references lbrynet.file_manager -> lbrynet.blob 2018-11-04 20:25:24 -05:00
Lex Berezhny cfc3ec2a05 merged lbrynet/file_manager into lbrynet/blob 2018-11-04 20:25:24 -05:00
Lex Berezhny 1392b8f83b removed lbrynet/lbry_file/__init__.py 2018-11-04 20:25:24 -05:00
Lex Berezhny 8b265801df updated references lbrynet.lbry_file.client -> lbrynet.file_manager.client 2018-11-04 20:25:24 -05:00
Lex Berezhny 6c4fb8dab1 moved lbrynet/lbry_file/client into lbrynet/file_manager/client 2018-11-04 20:25:24 -05:00
Lex Berezhny c1a9cd93ee updated references lbrynet.crypstream -> lbrynet.blob 2018-11-04 20:25:24 -05:00
Lex Berezhny 0d4e83e167 merged lbrynet/cryptstream into lbrynet/blob 2018-11-04 20:25:24 -05:00
Lex Berezhny 1292eb345d fix dbmigrator pylint issue 2018-11-04 20:25:24 -05:00
Lex Berezhny f4c6dc90c9 updated references lbrynet.core -> lbrynet.p2p 2018-11-04 20:25:24 -05:00
Lex Berezhny 8c8f5d8f58 renamed lbrynet/core -> lbrynet/p2p 2018-11-04 20:25:24 -05:00
Lex Berezhny b6d986cb40 updated references after moving lbrynet/database into lbrynet/extras/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny 6cc134c7c8 merged lbrynet/database with lbrynet/extras/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny eee964cfa0 updated references to daemon after moving it to extras 2018-11-04 20:25:24 -05:00
Lex Berezhny a03c94bfad moved daemon to extras/daemon 2018-11-04 20:25:24 -05:00
Lex Berezhny f1f0917063 updated all references to point to new reflector location 2018-11-04 20:25:24 -05:00
Lex Berezhny fe2476b5aa moved reflector into extras 2018-11-04 20:25:24 -05:00
Lex Berezhny 9c5fa8bf76 winhelpers/winpaths.py -> extras/winpaths.py 2018-11-04 20:25:24 -05:00
Lex Berezhny cd43a71172 update winpaths.py references 2018-11-04 20:25:24 -05:00
Lex Berezhny e2c6de76f4 renamed knownpaths.py -> winpaths.py 2018-11-04 20:25:24 -05:00
Lex Berezhny 05aeadc9fe extraneous LICENSE file since the whole project is already MIT, moved copyright directly into file header 2018-11-04 20:25:24 -05:00
Lex Berezhny 7153417372 #egg=torba[server] 2018-11-04 10:50:15 -05:00
Lex Berezhny fceb2ae0a2 hmm 2018-11-04 10:50:15 -05:00
Lex Berezhny 0cef2fdd82 install extras from github 2018-11-04 10:50:15 -05:00
Lex Berezhny 9c91b135bd pylint requires [wallet-server] 2018-11-04 10:50:15 -05:00
Lex Berezhny e3b127192c install .[test] includes the [wallet-server] 2018-11-04 10:50:15 -05:00
Lex Berezhny ee1738da6e lbrynet.extras.wallet.server pylint fixes 2018-11-04 10:50:15 -05:00
Lex Berezhny 8ca2b75c80 lbryumx -> lbrynet.extras.wallet.server 2018-11-04 10:50:15 -05:00
Lex Berezhny dc81b6def1 install lbryumx after lbrynet so that we dont get the old protobuf 2018-11-04 10:50:15 -05:00
Lex Berezhny 22f5aaee70 regenerated protobufs 2018-11-04 10:50:15 -05:00
Lex Berezhny 4bd6569c9c lbrynet.schema.schema 2018-11-04 10:50:15 -05:00
Lex Berezhny dbf39baeb1 fix lbrynet.schema import 2018-11-04 10:50:15 -05:00
Lex Berezhny 3adf3c26f2 add set_build.py back in 2018-11-04 10:50:15 -05:00
Lex Berezhny fa0d13e630 protobuf==3.2.0 2018-11-04 10:50:15 -05:00
Lex Berezhny 227a82d6d3 update protobuf version 2018-11-04 10:50:15 -05:00
Lex Berezhny 65cfc1f63c removed lbryschema references from scripts and copied lbryschema requirements into local setup.py 2018-11-04 10:50:15 -05:00
Victor Shyba e46f0e2444 fix namespace lbryschema->lbrynet.schema 2018-11-04 10:50:15 -05:00
Victor Shyba 00bc7995c4 move lbryschema/tests/ into tests/unit/schema/ 2018-11-04 10:50:15 -05:00
Victor Shyba 2dcebe48b7 move lbryschema/lbryschema into lbrynet/schema 2018-11-04 10:50:15 -05:00
Lex Berezhny d12b6cddb3 fix integration tests 2018-11-04 02:28:36 -05:00
Lex Berezhny 699c057f2d upgrading to new torba 2018-11-04 02:28:36 -05:00
Lex Berezhny 3ce9f31f4b recursive channel lookup 2018-11-04 02:28:36 -05:00
Lex Berezhny 485d5d55e8 pylint fix 2018-11-04 02:28:36 -05:00
Lex Berezhny 6ee9f2513b add extra debug logging when a channel is not a claim errors 2018-11-04 02:28:36 -05:00
Daniel Brereton 81a6e14a5c added alt text 2018-11-02 19:38:30 -04:00
czakarian 496ffe0096 Fix typos / grammar changes 2018-10-31 16:35:39 -07:00
Preetham Viswanathan 216c760179 Merge pull request #1 from preethamvishy/preethamvishy/pr-template
Update pull request template
2018-10-30 17:16:56 -04:00
Preetham Viswanathan 01eca3004f Update PULL_REQUEST_TEMPLATE.md 2018-10-30 16:57:41 -04:00
GwanYeong Kim 2f47a3758e Remove unused Python modules in tests/unit/test_cli.py 2018-10-30 10:32:34 -04:00
pushkyn 73bcc1796f Add GitHub pull request template 2018-10-30 10:27:47 -04:00
Jack Robison e03e41ad31 bump 2018-10-26 16:51:23 -04:00
Jack Robison 6cb30f8e05 Merge branch 'fix-update-stream'
[API] add `aioupnp_version` to `upnp` field in the response to `status`
[API] fix uncaught claim signature errors
2018-10-26 16:49:51 -04:00
Jack Robison d9b1be6214 add aioupnp version to upnp status
-set non-blank not found message if upnp fails
2018-10-26 16:48:52 -04:00
Jack Robison 5f2f06c269 fix loggly 2018-10-26 13:01:50 -04:00
Jack Robison e1dcdcf4db catch claim signature error 2018-10-26 13:01:50 -04:00
Jack Robison 416472c5d8 log stream update failure 2018-10-26 13:01:46 -04:00
Alyssa Callahan 196f3ba92d added Internal use section 2018-10-24 15:47:07 -04:00
Lex Berezhny 6feae93c58 Update lbrynet/wallet/claim_proofs.py 2018-10-23 12:02:10 -04:00
Lex Berezhny 203d30d4f7 Update lbrynet/wallet/claim_proofs.py 2018-10-23 12:02:10 -04:00
Hugo d0a7ca841b Upgrade unit tests to use more useful asserts 2018-10-23 12:02:10 -04:00
Hugo 2e6c848c39 Upgrade Python syntax with pyupgrade --py36-plus 2018-10-23 12:02:10 -04:00
267 changed files with 9253 additions and 2910 deletions

No files matched your search

+21
View File
@@ -51,3 +51,24 @@ Tell us what happens instead
## Screenshots
<!-- If a screenshot would help explain the bug, please include one or two here -->
## Internal Use
### Acceptance Criteria
1.
2.
3.
### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed
+38
View File
@@ -0,0 +1,38 @@
## PR Checklist
Please check all that apply to this PR using "x":
- [ ] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
- [ ] I have checked that this PR does not introduce a breaking change
- [ ] This PR introduces breaking changes and I have provided a detailed explanation below
## PR Type
What kind of change does this PR introduce?
Why is this change necessary?
<!-- Please check all that apply to this PR using "x". -->
- [ ] Bugfix
- [ ] Feature
- [ ] Breaking changes (bugfix or feature that introduces breaking changes)
- [ ] Code style update (formatting)
- [ ] Refactoring (no functional changes)
- [ ] Documentation changes
- [ ] Other - Please describe:
## Fixes
Issue Number: N/A
## What is the current behavior?
## What is the new behavior?
## Other information
<!-- If this PR contains a breaking change, please describe the impact and solution strategy for existing applications below. -->
+1 -1
View File
@@ -9,7 +9,7 @@
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=CVS
ignore=CVS,schema
# Add files or directories matching the regex patterns to the
# blacklist. The regex matches against base names, not paths.
+45 -32
View File
@@ -9,50 +9,34 @@ jobs:
- stage: code quality
name: "pylint lbrynet"
install:
- pip install astroid==2.0.4
# newer astroid fails in pylint so we pre-install older version
- pip install pylint
- pip install git+https://github.com/lbryio/torba.git
- pip install git+https://github.com/lbryio/lbryschema.git
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .
script: pylint lbrynet
- &tests
stage: test
name: "Unit Tests w/ Python 3.7"
name: "Unit Tests"
install:
- pip install coverage
- pip install git+https://github.com/lbryio/electrumx.git#lbryumx
- pip install git+https://github.com/lbryio/orchstr8.git
- pip install git+https://github.com/lbryio/lbryschema.git
- pip install git+https://github.com/lbryio/lbryumx.git
- pip install git+https://github.com/lbryio/torba.git
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .[test]
script:
- HOME=/tmp coverage run -p --source=lbrynet -m unittest discover -v tests.unit.wallet
- HOME=/tmp coverage run -p --source=lbrynet -m twisted.trial --reactor=asyncio tests.unit.analytics tests.unit.components tests.unit.core tests.unit.cryptstream tests.unit.database tests.unit.dht tests.unit.lbryfilemanager tests.unit.lbrynet_daemon tests.unit.test_cli tests.unit.test_customLogger
- HOME=/tmp coverage run -p --source=lbrynet -m twisted.trial --reactor=asyncio tests.unit.analytics tests.unit.components tests.unit.core tests.unit.cryptstream tests.unit.database tests.unit.dht tests.unit.lbryfilemanager tests.unit.lbrynet_daemon tests.unit.test_cli tests.unit.test_customLogger tests.unit.schema
after_success:
- coverage combine
- bash <(curl -s https://codecov.io/bash)
- <<: *tests
name: "Unit Tests w/ Python 3.6"
python: "3.6"
- <<: *tests
name: "DHT Tests w/ Python 3.7"
script: HOME=/tmp coverage run --source=lbrynet -m twisted.trial --reactor=asyncio tests.functional
- <<: *tests
name: "DHT Tests w/ Python 3.6"
python: "3.6"
name: "DHT Tests"
script: HOME=/tmp coverage run --source=lbrynet -m twisted.trial --reactor=asyncio tests.functional
- name: "Integration Tests"
install:
- pip install tox-travis coverage
- pushd .. && git clone https://github.com/lbryio/electrumx.git --branch lbryumx && popd
- pushd .. && git clone https://github.com/lbryio/orchstr8.git && popd
- pushd .. && git clone https://github.com/lbryio/lbryschema.git && popd
- pushd .. && git clone https://github.com/lbryio/lbryumx.git && cd lbryumx && git checkout afd34f323dd94c516108a65240f7d17aea8efe85 && cd .. && popd
- pushd .. && git clone https://github.com/lbryio/torba.git && popd
script: tox
after_success:
@@ -65,33 +49,50 @@ jobs:
services:
- docker
install:
- docker pull cdrx/pyinstaller-windows:python3-32bit
- docker pull lbry/pyinstaller34_32bits:py371
script:
- docker run -v "$(pwd):/src/lbry" cdrx/pyinstaller-windows:python3-32bit lbry/scripts/wine_build.sh
- python scripts/set_build.py
- docker run -v "$(pwd):/src/lbry" lbry/pyinstaller34_32bits:py371 lbry/scripts/wine_build.sh
- sudo zip -j dist/lbrynet-windows.zip dist/lbrynet.exe
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-windows.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-windows.zip
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})/win/
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- &build
name: "Linux"
python: "3.6"
env: OS=linux
install:
- pip3 install pyinstaller
- pip3 install git+https://github.com/lbryio/torba.git
- pip3 install git+https://github.com/lbryio/lbryschema.git
- python scripts/set_build.py
- pip3 install -e .
script:
- pyinstaller -F -n lbrynet lbrynet/cli.py
- pyinstaller -F -n lbrynet lbrynet/extras/cli.py
- chmod +x dist/lbrynet
- zip -j dist/lbrynet-${OS}.zip dist/lbrynet
- ./dist/lbrynet --version
env: OS=linux
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-${OS}.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
@@ -99,14 +100,26 @@ jobs:
- lbrynet-${OS}.zip
# artifact uploader thinks lbrynet is a directory, https://github.com/travis-ci/artifacts/issues/78
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})/${OS}/lbrynet
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- <<: *build
name: "Mac"
os: osx
osx_image: xcode9.4
osx_image: xcode8.3
language: generic
env: OS=mac
cache: false
before_install:
- brew upgrade python
install:
- python3 --version
- pip3 --version
- pip3 install pyinstaller
- git clone https://github.com/lbryio/torba.git --depth 1
- sed -i -e "s/'plyvel',//" torba/setup.py
- cd torba && pip3 install -e . && cd ..
- python3 scripts/set_build.py
- pip3 install -e .
cache:
directories:
+29 -29
View File
@@ -8,44 +8,44 @@ can and probably will change functionality and break backwards compatibility
at anytime.
## Unreleased
### Security
*
*
## [0.30.1] - 2018-12-09
### API
*
*
* add `aioupnp_version` to `upnp` field in the response to `status`
* fixed uncaught claim signature errors
* `--blocking` support for `abandon` command (https://github.com/lbryio/lbry/pull/1618)
* don't create SSL keys if SSL is disabled (https://github.com/lbryio/lbry/pull/1644)
* added `account_send` command (https://github.com/lbryio/lbry/pull/1654)
* fixes for claim send to address (https://github.com/lbryio/lbry/pull/1651)
* better handle errors on resolve (https://github.com/lbryio/lbry/pull/1662)
* fix `transaction_show` (https://github.com/lbryio/lbry/pull/1689)
### Schema
* merged `lbryschema` repository into `lbrynet` (https://github.com/lbryio/lbry/pull/1444)
* updated `protobuf` to `3.2.0` (https://github.com/lbryio/lbry/issues/1602)
* use `types` repo for generating python protobufs (https://github.com/lbryio/lbry/pull/1605)
### Wallet
*
*
### File Manager
*
*
* fixed resolving names with invalid channel signatures
* fixed wallet sync issues
* fixed wallet locking/encryption issues
### DHT
*
*
### P2P
*
*
### Reflector
*
*
* fixed retrying kademlia rpc calls (https://github.com/lbryio/lbry/pull/1495)
* fixed `KeyError` in `iterativeFindValue`, this is used by `peer_list` and when finding peers to download blobs from (https://github.com/lbryio/lbry/issues/1638)
* settings external ip address and ability to disable more DHT components (https://github.com/lbryio/lbry/pull/1604)
* improved routing table bucket initialization (https://github.com/lbryio/lbry/pull/1633)
### Database
*
*
### Known Issues
*
*
* resolved too many variables in sqlite issue (https://github.com/lbryio/lbry/pull/1612)
### Misc
* switch to Python 3.7
* brought back icon for windows binary
* using custom list of mime types with more types (https://github.com/lbryio/lbry/pull/1700)
* add balance to startup report (https://github.com/lbryio/lbry/pull/1613)
* drop `six` dependency (https://github.com/lbryio/lbry/pull/1660)
* drop `mock` dependency (https://github.com/lbryio/lbry/pull/1661)
## [0.30.0] - 2018-10-22
This release is the most ambitious upgrade to lbrynet yet; including a brand
+7 -7
View File
@@ -1,17 +1,17 @@
# Installing LBRY
If only the json-rpc API server is needed the recommended way to install LBRY is to use a pre-built binary. We provide binaries for all major operating systems. See the [README](README.md).
If only the JSON-RPC API server is needed, the recommended way to install LBRY is to use a pre-built binary. We provide binaries for all major operating systems. See the [README](README.md).
These instructions are for installing LBRY from source, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
Here's a video walkthrough of this setup which is itself hosted by the LBRY network and provided via [spee.ch](https://github.com/lbryio/spee.ch):
Here's a video walkthrough of this setup, which is itself hosted by the LBRY network and provided via [spee.ch](https://github.com/lbryio/spee.ch):
[![Setup for development](https://spee.ch/2018-10-04-17-13-54-017046806.png)](https://spee.ch/967f99344308f1e90f0620d91b6c93e4dfb240e0/lbrynet-dev-setup.mp4)
## Prerequisites
Running `lbrynet` from source requires Python 3.6 or higher (3.7 is preferred). Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/)
After installing python 3 you'll need to install some additional libraries depending on your operating system.
After installing python 3, you'll need to install some additional libraries depending on your operating system.
### Virtualenv
@@ -42,7 +42,7 @@ sudo apt-get install build-essential python3.7 python3.7-dev git python-virtuale
On Raspbian, you will also need to install `python-pyparsing`.
If you're running another Linux flavor, install the equivalent of the above packages for your system.
If you're running another Linux distro, install the equivalent of the above packages for your system.
## Installation
@@ -51,16 +51,16 @@ To install:
```
git clone https://github.com/lbryio/lbry.git
cd lbry
virtualenv lbry-venv --python=python3.7
source lbry-venv/bin/activate
python --version # Python 2 is not supported. Make sure you're on Python 3.7
pip install --editable .[test] # [test] installs extras needed for running the tests
pip install -e .
```
To verify your install, `which lbrynet` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
To verify your installation, `which lbrynet` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
## Run the tests
To run the unit tests from the repo directory:
+20 -7
View File
@@ -1,12 +1,19 @@
# <img src="https://raw.githubusercontent.com/lbryio/lbry/master/lbry.png" alt="LBRY" width="48" height="36" /> LBRY [![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry) [![Test Coverage](https://codecov.io/gh/lbryio/lbry/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry)
# <img src="https://raw.githubusercontent.com/lbryio/lbry/master/lbry.png" alt="LBRY" width="48" height="36" /> LBRY SDK [![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry) [![Test Coverage](https://codecov.io/gh/lbryio/lbry/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry)
LBRY is an open-source protocol providing distribution, discovery, and purchase of digital content (data) via a decentralized network. It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, and rights and access rules. LBRY also provides a data network that consists of peers uploading and downloading data from other peers, possibly in exchange for payments, and a distributed hash table, used by peers to discover other peers.
LBRY is a decentralized peer-to-peer network providing distribution, discovery, and purchase of digital content (data). It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, rights and access rules. LBRY also provides a data network that consists of peers (seeders) uploading and downloading data from other peers, possibly in exchange for payments, as well as a distributed hash table used by peers to discover other peers.
This project provides an SDK for building applications using the LBRY protocol. The SDK is written in Python 3.7+ using Twisted. For other languages or platforms, a JSONRPC API is provided.
LBRY SDK for Python is currently the most full featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components:
* Built on Python 3.7+ and `asyncio`.
* DHT (Distributed Hash Table) implementation for finding peers ([lbrynet.dht](https://github.com/lbryio/lbry/tree/master/lbrynet/dht)).
* Blob exchange protocol for downloading content and negotiating payments ([lbrynet.blob_exchange](https://github.com/lbryio/lbry/tree/master/lbrynet/blob_exchange)).
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([lbrynet.schema](https://github.com/lbryio/lbry/tree/master/lbrynet/schema)).
* Wallet implementation for the LBRY blockchain ([lbrynet.extras.wallet](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/wallet)).
* Daemon with a JSON-RPC API to ease building end user applications in any language and for automating various tasks ([lbrynet.extras.daemon](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/daemon)).
## Installation
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases, for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
## Usage
@@ -14,13 +21,15 @@ Run `lbrynet start` to launch the API server.
By default, `lbrynet` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
Our [quickstart guide](http://lbry.io/quickstart) provides a simple walkthrough and examples for learning.
Our [quickstart guide](https://lbry.tech/playground) provides a simple walkthrough and examples for learning.
With the daemon running, `lbrynet commands` will show you a list of commands.
The full API is documented [here](https://lbry.tech/api/sdk).
## Running from source
Installing from source is also relatively painless, full instructions are in [INSTALL.md](INSTALL.md)
Installing from source is also relatively painless. Full instructions are in [INSTALL.md](INSTALL.md)
## Contributing
@@ -36,8 +45,12 @@ We take security seriously. Please contact security@lbry.io regarding any securi
## Contact
The primary contact for this project is [@jackrobison](mailto:jack@lbry.io).
The primary contact for this project is [@eukreign](mailto:lex@lbry.io).
## Additional information and links
The documentation for the API can be found [here](https://lbry.tech/api/sdk).
Daemon defaults, ports, and other settings are documented [here](https://lbry.tech/resources/daemon-settings).
Settings can be configured using a daemon-settings.yml file. An example can be found [here](https://github.com/lbryio/lbry/blob/master/example_daemon_settings.yml).
+3 -3
View File
@@ -100,7 +100,7 @@
<div class="md-flex__cell md-flex__cell--shrink">
<a href="/" title="LBRY" class="md-header-nav__button md-logo">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" width="24" height="24">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" alt="LBRY logo" width="24" height="24">
</a>
</div>
@@ -196,7 +196,7 @@
<label class="md-nav__title md-nav__title--site" for="drawer">
<span class="md-nav__button md-logo">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" width="48" height="48">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" alt="LBRY logo" width="48" height="48">
</span>
LBRY
@@ -304,4 +304,4 @@
</body>
</html>
</html>
+147 -61
View File
@@ -43,19 +43,13 @@
{
"name": "account_id",
"type": "str",
"description": "If provided only the balance for this account will be given",
"description": "If provided only the balance for this account will be given. Otherwise default account.",
"is_required": false
},
{
"name": "address",
"type": "str",
"description": "If provided only the balance for this address will be given",
"is_required": false
},
{
"name": "include_unconfirmed",
"type": "bool",
"description": "Include unconfirmed",
"name": "confirmations",
"type": "int",
"description": "Only include transactions with this many confirmed blocks.",
"is_required": false
}
],
@@ -83,18 +77,25 @@
{
"name": "account_decrypt",
"description": "Decrypt an encrypted account, this will remove the wallet password",
"arguments": [],
"arguments": [
{
"name": "account_id",
"type": "str",
"description": "id for the account to decrypt",
"is_required": false
}
],
"returns": "(bool) true if wallet is decrypted, otherwise false"
},
{
"name": "account_encrypt",
"description": "Encrypt a wallet with a password, if the wallet is already encrypted this will update\nthe password",
"description": "Encrypt an unencrypted account with a password",
"arguments": [
{
"name": "new_password",
"name": "account_id",
"type": "str",
"description": "password string to be used for encrypting wallet",
"is_required": true
"description": "id for the account to encrypt",
"is_required": false
}
],
"returns": "(bool) true if wallet is decrypted, otherwise false"
@@ -107,13 +108,13 @@
"name": "to_account",
"type": "str",
"description": "send to this account",
"is_required": true
"is_required": false
},
{
"name": "from_account",
"type": "str",
"description": "spend from this account",
"is_required": true
"is_required": false
},
{
"name": "amount",
@@ -155,13 +156,7 @@
{
"name": "confirmations",
"type": "int",
"description": "required confirmations (default: 6)",
"is_required": false
},
{
"name": "include_reserved",
"type": "bool",
"description": "include reserved UTXOs (default: false)",
"description": "required confirmations (default: 0)",
"is_required": false
},
{
@@ -179,6 +174,19 @@
],
"returns": "(map) balance of account(s)"
},
{
"name": "account_lock",
"description": "Lock an unlocked account",
"arguments": [
{
"name": "account_id",
"type": "str",
"description": "id for the account to lock",
"is_required": false
}
],
"returns": "(bool) true if account is locked, otherwise false"
},
{
"name": "account_max_address_gap",
"description": "Finds ranges of consecutive addresses that are unused and returns the length\nof the longest such range: for change and receiving address chains. This is\nuseful to figure out ideal values to set for 'receiving_gap' and 'change_gap'\naccount settings.",
@@ -205,6 +213,25 @@
],
"returns": "(map) details of removed account"
},
{
"name": "account_send",
"description": "Send the same number of credits to multiple addresses.",
"arguments": [
{
"name": "account_id",
"type": "str",
"description": "account to fund the transaction",
"is_required": false
},
{
"name": "broadcast",
"type": "bool",
"description": "actually broadcast the transaction, default: false.",
"is_required": false
}
],
"returns": ""
},
{
"name": "account_set",
"description": "Change various settings on an account.",
@@ -259,10 +286,10 @@
"description": "Unlock an encrypted account",
"arguments": [
{
"name": "password",
"name": "account_id",
"type": "str",
"description": "password for unlocking wallet",
"is_required": true
"description": "id for the account to unlock",
"is_required": false
}
],
"returns": "(bool) true if account is unlocked, otherwise false"
@@ -295,23 +322,22 @@
"type": "str",
"description": "id of the account to use",
"is_required": false
},
{
"name": "page",
"type": "int",
"description": "page to return during paginating",
"is_required": false
},
{
"name": "page_size",
"type": "int",
"description": "number of items on page during pagination",
"is_required": false
}
],
"returns": "List of wallet addresses"
},
{
"name": "address_public_key",
"description": "Get public key from wallet address",
"arguments": [
{
"name": "address",
"type": "str",
"description": "address for which to get the public key",
"is_required": true
}
],
"returns": "(list) list of public keys associated with address.\n Could contain more than one public key if multisig."
},
{
"name": "address_unused",
"description": "Return an address containing no balance, will create\na new address if there is none.",
@@ -535,7 +561,26 @@
{
"name": "channel_list",
"description": "Get certificate claim infos for channels that can be published to",
"arguments": [],
"arguments": [
{
"name": "account_id",
"type": "str",
"description": "id of the account to use",
"is_required": false
},
{
"name": "page",
"type": "int",
"description": "page to return during paginating",
"is_required": false
},
{
"name": "page_size",
"type": "int",
"description": "number of items on page during pagination",
"is_required": false
}
],
"returns": "(list) ClaimDict, includes 'is_mine' field to indicate if the certificate claim\n is in the wallet."
},
{
@@ -553,6 +598,12 @@
"type": "decimal",
"description": "bid amount on the channel",
"is_required": true
},
{
"name": "account_id",
"type": "str",
"description": "id of the account to store channel",
"is_required": false
}
],
"returns": "(dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (float) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }"
@@ -584,6 +635,12 @@
"type": "str",
"description": "id of the account to use",
"is_required": false
},
{
"name": "blocking",
"type": "bool",
"description": "wait until abandon is in mempool",
"is_required": false
}
],
"returns": "(dict) Dictionary containing result of the claim\n {\n success: (bool) True if txn is successful\n txid : (str) txid of resulting transaction\n }"
@@ -641,6 +698,18 @@
"type": "str",
"description": "id of the account to query",
"is_required": false
},
{
"name": "page",
"type": "int",
"description": "page to return during paginating",
"is_required": false
},
{
"name": "page_size",
"type": "int",
"description": "number of items on page during pagination",
"is_required": false
}
],
"returns": "(list) List of name claims owned by user\n [\n {\n 'address': (str) address that owns the claim\n 'amount': (float) amount assigned to the claim\n 'blocks_to_expiration': (int) number of blocks until it expires\n 'category': (str) \"claim\", \"update\" , or \"support\"\n 'claim_id': (str) claim ID of the claim\n 'confirmations': (int) number of blocks of confirmations for the claim\n 'expiration_height': (int) the block height which the claim will expire\n 'expired': (bool) true if expired, false otherwise\n 'height': (int) height of the block containing the claim\n 'is_spent': (bool) true if claim is abandoned, false otherwise\n 'name': (str) name of the claim\n 'permanent_url': (str) permanent url of the claim,\n 'txid': (str) txid of the claim\n 'nout': (int) nout of the claim\n 'value': (str) value of the claim\n },\n ]"
@@ -676,25 +745,6 @@
],
"returns": "(dict) Dictionary containing the transaction information\n {\n \"hex\": (str) raw transaction,\n \"inputs\": (list) inputs(dict) used for the transaction,\n \"outputs\": (list) outputs(dict) for the transaction,\n \"total_fee\": (int) fee in dewies,\n \"total_input\": (int) total of inputs in dewies,\n \"total_output\": (int) total of outputs in dewies(input - fees),\n \"txid\": (str) txid of the transaction,\n }"
},
{
"name": "claim_renew",
"description": "Renew claim(s) or support(s)",
"arguments": [
{
"name": "outpoint",
"type": "str",
"description": "outpoint of the claim to renew",
"is_required": true
},
{
"name": "height",
"type": "str",
"description": "update claims expiring before or at this block height",
"is_required": true
}
],
"returns": "(dict) Dictionary where key is the the original claim's outpoint and\n value is the result of the renewal\n {\n outpoint:{\n\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (float) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n },\n }"
},
{
"name": "claim_send_to_address",
"description": "Send a name claim to an address",
@@ -1190,6 +1240,18 @@
"description": "claim id of the publisher channel, does not check for channel claim being in the wallet. This allows publishing to a channel where only the certificate private key is in the wallet.",
"is_required": false
},
{
"name": "channel_account_id",
"type": "str",
"description": "one or more account ids for accounts to look in for channel certificates, defaults to all accounts.",
"is_required": false
},
{
"name": "account_id",
"type": "str",
"description": "account to use for funding the transaction",
"is_required": false
},
{
"name": "claim_address",
"type": "str",
@@ -1363,7 +1425,7 @@
"name": "status",
"description": "Get daemon status",
"arguments": [],
"returns": "(dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'is_first_run': bool,\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'database': (bool),\n 'wallet': (bool),\n 'session': (bool),\n 'dht': (bool),\n 'hash_announcer': (bool),\n 'stream_identifier': (bool),\n 'file_manager': (bool),\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'peer_protocol_server': (bool),\n 'reflector': (bool),\n 'upnp': (bool),\n 'exchange_rate_manager': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the file manager,\n }\n }"
"returns": "(dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'is_first_run': bool,\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'database': (bool),\n 'wallet': (bool),\n 'session': (bool),\n 'dht': (bool),\n 'hash_announcer': (bool),\n 'stream_identifier': (bool),\n 'file_manager': (bool),\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'peer_protocol_server': (bool),\n 'reflector': (bool),\n 'upnp': (bool),\n 'exchange_rate_manager': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the file manager,\n },\n 'upnp': {\n 'aioupnp_version': (str),\n 'redirects': {\n <TCP | UDP>: (int) external_port,\n },\n 'gateway': (str) manufacturer and model,\n 'dht_redirect_set': (bool),\n 'peer_redirect_set': (bool),\n 'external_ip': (str) external ip address,\n }\n }"
},
{
"name": "stop",
@@ -1424,6 +1486,18 @@
"type": "str",
"description": "id of the account to query",
"is_required": false
},
{
"name": "page",
"type": "int",
"description": "page to return during paginating",
"is_required": false
},
{
"name": "page_size",
"type": "int",
"description": "number of items on page during pagination",
"is_required": false
}
],
"returns": "(list) List of transactions\n\n {\n \"claim_info\": (list) claim info if in txn [{\n \"address\": (str) address of claim,\n \"balance_delta\": (float) bid amount,\n \"amount\": (float) claim amount,\n \"claim_id\": (str) claim id,\n \"claim_name\": (str) claim name,\n \"nout\": (int) nout\n }],\n \"abandon_info\": (list) abandon info if in txn [{\n \"address\": (str) address of abandoned claim,\n \"balance_delta\": (float) returned amount,\n \"amount\": (float) claim amount,\n \"claim_id\": (str) claim id,\n \"claim_name\": (str) claim name,\n \"nout\": (int) nout\n }],\n \"confirmations\": (int) number of confirmations for the txn,\n \"date\": (str) date and time of txn,\n \"fee\": (float) txn fee,\n \"support_info\": (list) support info if in txn [{\n \"address\": (str) address of support,\n \"balance_delta\": (float) support amount,\n \"amount\": (float) support amount,\n \"claim_id\": (str) claim id,\n \"claim_name\": (str) claim name,\n \"is_tip\": (bool),\n \"nout\": (int) nout\n }],\n \"timestamp\": (int) timestamp,\n \"txid\": (str) txn id,\n \"update_info\": (list) update info if in txn [{\n \"address\": (str) address of claim,\n \"balance_delta\": (float) credited/debited\n \"amount\": (float) absolute amount,\n \"claim_id\": (str) claim id,\n \"claim_name\": (str) claim name,\n \"nout\": (int) nout\n }],\n \"value\": (float) value of txn\n }"
@@ -1450,6 +1524,18 @@
"type": "str",
"description": "id of the account to query",
"is_required": false
},
{
"name": "page",
"type": "int",
"description": "page to return during paginating",
"is_required": false
},
{
"name": "page_size",
"type": "int",
"description": "number of items on page during pagination",
"is_required": false
}
],
"returns": "(list) List of unspent transaction outputs (UTXOs)\n [\n {\n \"address\": (str) the output address\n \"amount\": (float) unspent amount\n \"height\": (int) block height\n \"is_claim\": (bool) is the tx a claim\n \"is_coinbase\": (bool) is the tx a coinbase tx\n \"is_support\": (bool) is the tx a support\n \"is_update\": (bool) is the tx an update\n \"nout\": (int) nout of the output\n \"txid\": (str) txid of the output\n },\n ...\n ]"
+26
View File
@@ -0,0 +1,26 @@
# This is an example daemon_settings.yml file.
# See https://lbry.tech/resources/daemon-settings for all configuration keys and values
share_usage_data: True
lbryum_servers:
- lbryumx1.lbry.io:50001
- lbryumx2.lbry.io:50001
blockchain_name: lbrycrd_main
data_dir: /home/lbry/.lbrynet
download_directory: /home/lbry/downloads
delete_blobs_on_remove: True
dht_node_port: 4444
peer_port: 3333
use_upnp: True
use_auth_http: True
use_https: True
#components_to_skip:
# - peer_protocol_server
# - hash_announcer
# - dht
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 7.4 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.6 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 6.1 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 97 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 361 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 5.3 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

+4 -1
View File
@@ -1,7 +1,10 @@
import logging
from lbrynet.custom_logger import install_logger
__name__ = "lbrynet"
__version__ = "0.30.0"
__version__ = "0.30.3"
version = tuple(__version__.split('.'))
install_logger()
logging.getLogger(__name__).addHandler(logging.NullHandler())
-1
View File
@@ -1 +0,0 @@
from . import paths
-26
View File
@@ -1,26 +0,0 @@
# App root files dir
# https://developer.android.com/reference/android/content/ContextWrapper.html#getFilesDir%28%29
def android_files_dir():
return None
# Base internal storage path
def android_internal_storage_dir():
return None
# Base external (SD card, if present) storage path
def android_external_storage_dir():
return None
# Internal device storage (private app folder)
# https://developer.android.com/reference/android/content/ContextWrapper.html#getExternalFilesDirs(java.lang.String)
def android_app_internal_storage_dir():
return None
# External (app folder on SD card, if present) storage
# https://developer.android.com/reference/android/content/ContextWrapper.html#getExternalFilesDirs(java.lang.String)
def android_app_external_storage_dir():
return None
@@ -7,7 +7,7 @@ from cryptography.hazmat.primitives.ciphers import Cipher, modes
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from cryptography.hazmat.primitives.padding import PKCS7
from cryptography.hazmat.backends import default_backend
from lbrynet.core.BlobInfo import BlobInfo
from lbrynet.p2p.BlobInfo import BlobInfo
from lbrynet.blob.blob_file import MAX_BLOB_SIZE
log = logging.getLogger(__name__)
@@ -75,7 +75,7 @@ class StreamBlobDecryptor:
def finish_decrypt():
bytes_left = len(self.buff) % (AES.block_size // 8)
if bytes_left != 0:
log.warning(self.buff[-1 * (AES.block_size // 8):].encode('hex'))
log.warning(binascii.hexlify(self.buff[-1 * (AES.block_size // 8):]).decode())
raise Exception("blob %s has incorrect padding: %i bytes left" %
(self.blob.blob_hash, bytes_left))
data_to_decrypt, self.buff = self.buff, b''
@@ -6,7 +6,7 @@ import logging
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from twisted.internet import defer
from lbrynet.cryptstream.CryptBlob import CryptStreamBlobMaker
from lbrynet.blob.CryptBlob import CryptStreamBlobMaker
log = logging.getLogger(__name__)
@@ -2,16 +2,16 @@
Utilities for turning plain files into LBRY Files.
"""
import logging
import os
import logging
from binascii import hexlify
from twisted.internet import defer
from twisted.protocols.basic import FileSender
from lbrynet.core.StreamDescriptor import BlobStreamDescriptorWriter, EncryptedFileStreamType
from lbrynet.core.StreamDescriptor import format_sd_info, get_stream_hash, validate_descriptor
from lbrynet.cryptstream.CryptStreamCreator import CryptStreamCreator
from lbrynet.p2p.StreamDescriptor import BlobStreamDescriptorWriter, EncryptedFileStreamType
from lbrynet.p2p.StreamDescriptor import format_sd_info, get_stream_hash, validate_descriptor
from lbrynet.blob.CryptStreamCreator import CryptStreamCreator
log = logging.getLogger(__name__)
@@ -6,13 +6,13 @@ from binascii import hexlify, unhexlify
from twisted.internet import defer
from lbrynet import conf
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.HTTPBlobDownloader import HTTPBlobDownloader
from lbrynet.core.utils import short_hash
from lbrynet.lbry_file.client.EncryptedFileDownloader import EncryptedFileSaver
from lbrynet.lbry_file.client.EncryptedFileDownloader import EncryptedFileDownloader
from lbrynet.file_manager.EncryptedFileStatusReport import EncryptedFileStatusReport
from lbrynet.core.StreamDescriptor import save_sd_info
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.p2p.HTTPBlobDownloader import HTTPBlobDownloader
from lbrynet.utils import short_hash
from lbrynet.blob.client.EncryptedFileDownloader import EncryptedFileSaver
from lbrynet.blob.client.EncryptedFileDownloader import EncryptedFileDownloader
from lbrynet.blob.EncryptedFileStatusReport import EncryptedFileStatusReport
from lbrynet.p2p.StreamDescriptor import save_sd_info
log = logging.getLogger(__name__)
@@ -93,7 +93,7 @@ class ManagedEncryptedFileDownloader(EncryptedFileSaver):
elif status == ManagedEncryptedFileDownloader.STATUS_FINISHED:
self.completed = True
else:
raise Exception("Unknown status for stream %s: %s" % (self.stream_hash, status))
raise Exception(f"Unknown status for stream {self.stream_hash}: {status}")
@defer.inlineCallbacks
def stop(self, err=None, change_status=True):
@@ -7,15 +7,14 @@ from binascii import hexlify, unhexlify
from twisted.internet import defer, task, reactor
from twisted.python.failure import Failure
from lbrynet.reflector.reupload import reflect_file
from lbrynet.file_manager.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet.file_manager.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet.core.StreamDescriptor import EncryptedFileStreamType, get_sd_info
from lbrynet.cryptstream.client.CryptStreamDownloader import AlreadyStoppedError
from lbrynet.cryptstream.client.CryptStreamDownloader import CurrentlyStoppingError
from lbrynet.core.utils import safe_start_looping_call, safe_stop_looping_call
from lbrynet import conf
from lbrynet.extras.reflector.reupload import reflect_file
from lbrynet.blob.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet.blob.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamType, get_sd_info
from lbrynet.blob.client.CryptStreamDownloader import AlreadyStoppedError
from lbrynet.blob.client.CryptStreamDownloader import CurrentlyStoppingError
from lbrynet.utils import safe_start_looping_call, safe_stop_looping_call
log = logging.getLogger(__name__)
-4
View File
@@ -1,4 +0,0 @@
from .blob_file import BlobFile
from .creator import BlobFileCreator
from .writer import HashBlobWriter
from .reader import HashBlobReader
+22 -4
View File
@@ -1,10 +1,10 @@
import logging
import os
import logging
from twisted.internet import defer, threads
from twisted.web.client import FileBodyProducer
from twisted.python.failure import Failure
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError, InvalidBlobHashError
from lbrynet.core.utils import is_valid_blobhash
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbrynet.p2p.Error import DownloadCanceledError, InvalidDataError, InvalidBlobHashError
from lbrynet.blob.writer import HashBlobWriter
from lbrynet.blob.reader import HashBlobReader
@@ -12,6 +12,24 @@ log = logging.getLogger(__name__)
MAX_BLOB_SIZE = 2 * 2 ** 20
# digest_size is in bytes, and blob hashes are hex encoded
blobhash_length = get_lbry_hash_obj().digest_size * 2
def is_valid_hashcharacter(char):
return char in "0123456789abcdef"
def is_valid_blobhash(blobhash):
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: True/False
"""
return len(blobhash) == blobhash_length and all(is_valid_hashcharacter(l) for l in blobhash)
class BlobFile:
"""
@@ -182,7 +200,7 @@ class BlobFile:
if writer.len_so_far != self.length:
err_string = "blob length is %i vs expected %i" % (writer.len_so_far, self.length)
else:
err_string = "blob hash is %s vs expected %s" % (writer.blob_hash, self.blob_hash)
err_string = f"blob hash is {writer.blob_hash} vs expected {self.blob_hash}"
errback_finished_deferred(Failure(InvalidDataError(err_string)))
d = defer.succeed(None)
else:
@@ -1,6 +1,6 @@
import binascii
from twisted.internet import defer
from lbrynet.cryptstream.CryptBlob import StreamBlobDecryptor
from lbrynet.blob.CryptBlob import StreamBlobDecryptor
class CryptBlobHandler:
@@ -1,13 +1,14 @@
from binascii import unhexlify
import logging
from lbrynet.core.client.BlobRequester import BlobRequester
from lbrynet.core.client.ConnectionManager import ConnectionManager
from lbrynet.core.client.DownloadManager import DownloadManager
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.cryptstream.client.CryptBlobHandler import CryptBlobHandler
from binascii import unhexlify
from twisted.internet import defer
from twisted.python.failure import Failure
from lbrynet.p2p.client.BlobRequester import BlobRequester
from lbrynet.p2p.client.ConnectionManager import ConnectionManager
from lbrynet.p2p.client.DownloadManager import DownloadManager
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.blob.client.CryptBlobHandler import CryptBlobHandler
log = logging.getLogger(__name__)
@@ -2,14 +2,14 @@ import os
import logging
import traceback
from binascii import hexlify, unhexlify
from lbrynet.core.StreamDescriptor import save_sd_info
from lbrynet.cryptstream.client.CryptStreamDownloader import CryptStreamDownloader
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.Error import FileOpenError
from lbrynet.lbry_file.client.EncryptedFileMetadataHandler import EncryptedFileMetadataHandler
from twisted.internet import defer, threads
from lbrynet.p2p.StreamDescriptor import save_sd_info
from lbrynet.blob.client.CryptStreamDownloader import CryptStreamDownloader
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.p2p.Error import FileOpenError
from lbrynet.blob.client.EncryptedFileMetadataHandler import EncryptedFileMetadataHandler
log = logging.getLogger(__name__)
@@ -44,8 +44,19 @@ class EncryptedFileDownloader(CryptStreamDownloader):
self.blob_manager, download_manager)
def _start(self):
def check_start_succeeded(success):
if success:
self.starting = False
self.stopped = False
self.completed = False
return True
else:
return self._start_failed()
self.download_manager = self._get_download_manager()
d = self._setup_output()
d.addCallback(lambda _: CryptStreamDownloader._start(self))
d.addCallback(lambda _: self.download_manager.start_downloading())
d.addCallbacks(check_start_succeeded)
return d
def _setup_output(self):
@@ -1,6 +1,6 @@
from lbrynet.core.StreamDescriptor import EncryptedFileStreamType
from lbrynet.core.StreamDescriptor import EncryptedFileStreamDescriptorValidator
from lbrynet.core.DownloadOption import DownloadOption, DownloadOptionChoice
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamType
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamDescriptorValidator
from lbrynet.p2p.DownloadOption import DownloadOption, DownloadOptionChoice
def add_lbry_file_to_sd_identifier(sd_identifier):
File renamed without changes.
+1 -1
View File
@@ -3,7 +3,7 @@ import logging
from io import BytesIO
from twisted.internet import defer
from twisted.web.client import FileBodyProducer
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
+2 -2
View File
@@ -1,8 +1,8 @@
import logging
from io import BytesIO
from twisted.python.failure import Failure
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.p2p.Error import DownloadCanceledError, InvalidDataError
from lbrynet.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
-171
View File
@@ -1,171 +0,0 @@
import sys
from twisted.internet import asyncioreactor
if 'twisted.internet.reactor' not in sys.modules:
asyncioreactor.install()
else:
from twisted.internet import reactor
if not isinstance(reactor, asyncioreactor.AsyncioSelectorReactor) and getattr(sys, 'frozen', False):
# pyinstaller hooks install the default reactor before
# any of our code runs, see kivy for similar problem:
# https://github.com/kivy/kivy/issues/4182
del sys.modules['twisted.internet.reactor']
asyncioreactor.install()
from twisted.internet import reactor
import json
import asyncio
from aiohttp.client_exceptions import ClientConnectorError
from requests.exceptions import ConnectionError
from docopt import docopt
from textwrap import dedent
from lbrynet import __name__ as lbrynet_name
from lbrynet.daemon.Daemon import Daemon
from lbrynet.daemon.DaemonControl import start as daemon_main
from lbrynet.daemon.DaemonConsole import main as daemon_console
from lbrynet.daemon.auth.client import LBRYAPIClient
from lbrynet.core.system_info import get_platform
async def execute_command(method, params, conf_path=None):
# this check if the daemon is running or not
try:
api = await LBRYAPIClient.get_client(conf_path)
await api.status()
except (ClientConnectorError, ConnectionError):
await api.session.close()
print("Could not connect to daemon. Are you sure it's running?")
return 1
# this actually executes the method
try:
resp = await api.call(method, params)
await api.session.close()
print(json.dumps(resp["result"], indent=2))
except KeyError:
if resp["error"]["code"] == -32500:
print(json.dumps(resp["error"], indent=2))
else:
print(json.dumps(resp["error"]["message"], indent=2))
def print_help():
print(dedent("""
NAME
lbrynet - LBRY command line client.
USAGE
lbrynet [--conf <config file>] <command> [<args>]
EXAMPLES
lbrynet commands # list available commands
lbrynet status # get daemon status
lbrynet --conf ~/l1.conf status # like above but using ~/l1.conf as config file
lbrynet resolve_name what # resolve a name
lbrynet help resolve_name # get help for a command
"""))
def print_help_for_command(command):
fn = Daemon.callable_methods.get(command)
if fn:
print(dedent(fn.__doc__))
else:
print("Invalid command name")
def normalize_value(x, key=None):
if not isinstance(x, str):
return x
if key in ('uri', 'channel_name', 'name', 'file_name', 'download_directory'):
return x
if x.lower() == 'true':
return True
if x.lower() == 'false':
return False
if x.isdigit():
return int(x)
return x
def remove_brackets(key):
if key.startswith("<") and key.endswith(">"):
return str(key[1:-1])
return key
def set_kwargs(parsed_args):
kwargs = {}
for key, arg in parsed_args.items():
k = None
if arg is None:
continue
elif key.startswith("--") and remove_brackets(key[2:]) not in kwargs:
k = remove_brackets(key[2:])
elif remove_brackets(key) not in kwargs:
k = remove_brackets(key)
kwargs[k] = normalize_value(arg, k)
return kwargs
def main(argv=None):
argv = argv or sys.argv[1:]
if not argv:
print_help()
return 1
conf_path = None
if len(argv) and argv[0] == "--conf":
if len(argv) < 2:
print("No config file specified for --conf option")
print_help()
return 1
conf_path = argv[1]
argv = argv[2:]
method, args = argv[0], argv[1:]
if method in ['help', '--help', '-h']:
if len(args) == 1:
print_help_for_command(args[0])
else:
print_help()
return 0
elif method in ['version', '--version', '-v']:
print("{lbrynet_name} {lbrynet_version}".format(
lbrynet_name=lbrynet_name, **get_platform(get_ip=False)
))
return 0
elif method == 'start':
sys.exit(daemon_main(args, conf_path))
elif method == 'console':
sys.exit(daemon_console())
elif method not in Daemon.callable_methods:
if method not in Daemon.deprecated_methods:
print('{} is not a valid command.'.format(method))
return 1
new_method = Daemon.deprecated_methods[method].new_command
if new_method is None:
print("{} is permanently deprecated and does not have a replacement command.".format(method))
return 0
print("{} is deprecated, using {}.".format(method, new_method))
method = new_method
fn = Daemon.callable_methods[method]
parsed = docopt(fn.__doc__, args)
params = set_kwargs(parsed)
loop = asyncio.get_event_loop()
loop.run_until_complete(execute_command(method, params, conf_path))
return 0
if __name__ == "__main__":
sys.exit(main())
+143 -126
View File
@@ -1,24 +1,15 @@
import base58
import json
import logging
import os
import re
import sys
import yaml
import typing
import json
import logging
import envparse
import base58
import yaml
from appdirs import user_data_dir, user_config_dir
from lbrynet.core import utils
from lbrynet.core.Error import InvalidCurrencyError, NoSuchDirectoryError
from lbrynet.androidhelpers.paths import (
android_internal_storage_dir,
android_app_internal_storage_dir
)
try:
from lbrynet.winhelpers.knownpaths import get_path, FOLDERID, UserHandle
except (ImportError, ValueError, NameError):
# Android platform: NameError: name 'c_wchar' is not defined
pass
from lbrynet import utils
from lbrynet.p2p.Error import InvalidCurrencyError
log = logging.getLogger(__name__)
@@ -49,8 +40,6 @@ DEFAULT_DHT_NODES = [
('lbrynet3.lbry.io', 4444)
]
DEFAULT_SETTINGS_FILENAME = 'daemon_settings.yml'
settings_decoders = {
'.json': json.loads,
'.yml': yaml.load
@@ -61,89 +50,68 @@ settings_encoders = {
'.yml': yaml.safe_dump
}
# set by CLI when the user specifies an alternate config file path
conf_file = None
def _get_old_directories(platform_type):
directories = {}
if platform_type == WINDOWS:
appdata = get_path(FOLDERID.RoamingAppData, UserHandle.current)
directories['data'] = os.path.join(appdata, 'lbrynet')
directories['lbryum'] = os.path.join(appdata, 'lbryum')
directories['download'] = get_path(FOLDERID.Downloads, UserHandle.current)
elif platform_type == DARWIN:
directories['data'] = user_data_dir('LBRY')
directories['lbryum'] = os.path.expanduser('~/.lbryum')
directories['download'] = os.path.expanduser('~/Downloads')
elif platform_type == LINUX:
directories['data'] = os.path.expanduser('~/.lbrynet')
directories['lbryum'] = os.path.expanduser('~/.lbryum')
directories['download'] = os.path.expanduser('~/Downloads')
else:
raise ValueError('unknown platform value')
return directories
def _get_new_directories(platform_type):
directories = {}
if platform_type == ANDROID:
directories['data'] = '%s/lbrynet' % android_app_internal_storage_dir()
directories['lbryum'] = '%s/lbryum' % android_app_internal_storage_dir()
directories['download'] = '%s/Download' % android_internal_storage_dir()
elif platform_type == WINDOWS:
directories['data'] = user_data_dir('lbrynet', 'lbry')
directories['lbryum'] = user_data_dir('lbryum', 'lbry')
directories['download'] = get_path(FOLDERID.Downloads, UserHandle.current)
elif platform_type == DARWIN:
directories = _get_old_directories(platform_type)
elif platform_type == LINUX:
directories['data'] = user_data_dir('lbry/lbrynet')
directories['lbryum'] = user_data_dir('lbry/lbryum')
try:
with open(os.path.join(user_config_dir(), 'user-dirs.dirs'), 'r') as xdg:
down_dir = re.search(r'XDG_DOWNLOAD_DIR=(.+)', xdg.read()).group(1)
down_dir = re.sub('\$HOME', os.getenv('HOME'), down_dir)
directories['download'] = re.sub('\"', '', down_dir)
except EnvironmentError:
directories['download'] = os.getenv('XDG_DOWNLOAD_DIR')
if not directories['download']:
directories['download'] = os.path.expanduser('~/Downloads')
else:
raise ValueError('unknown platform value')
return directories
if 'ANDROID_ARGUMENT' in os.environ:
# https://github.com/kivy/kivy/blob/master/kivy/utils.py#L417-L421
platform = ANDROID
dirs = _get_new_directories(ANDROID)
elif 'darwin' in sys.platform:
platform = DARWIN
dirs = _get_old_directories(DARWIN)
elif 'win' in sys.platform:
platform = WINDOWS
if os.path.isdir(_get_old_directories(WINDOWS)['data']) or \
os.path.isdir(_get_old_directories(WINDOWS)['lbryum']):
dirs = _get_old_directories(WINDOWS)
else:
dirs = _get_new_directories(WINDOWS)
else:
platform = LINUX
if os.path.isdir(_get_old_directories(LINUX)['data']) or \
os.path.isdir(_get_old_directories(LINUX)['lbryum']):
dirs = _get_old_directories(LINUX)
else:
dirs = _get_new_directories(LINUX)
default_data_dir = dirs['data']
default_lbryum_dir = dirs['lbryum']
default_download_dir = dirs['download']
ICON_PATH = 'icons' if platform is WINDOWS else 'app.icns'
def get_windows_directories() -> typing.Tuple[str, str, str]:
from lbrynet.winpaths import get_path, FOLDERID, UserHandle
download_dir = get_path(FOLDERID.Downloads, UserHandle.current)
# old
appdata = get_path(FOLDERID.RoamingAppData, UserHandle.current)
data_dir = os.path.join(appdata, 'lbrynet')
lbryum_dir = os.path.join(appdata, 'lbryum')
if os.path.isdir(data_dir) or os.path.isdir(lbryum_dir):
return data_dir, lbryum_dir, download_dir
# new
data_dir = user_data_dir('lbrynet', 'lbry')
lbryum_dir = user_data_dir('lbryum', 'lbry')
download_dir = get_path(FOLDERID.Downloads, UserHandle.current)
return data_dir, lbryum_dir, download_dir
def get_darwin_directories() -> typing.Tuple[str, str, str]:
data_dir = user_data_dir('LBRY')
lbryum_dir = os.path.expanduser('~/.lbryum')
download_dir = os.path.expanduser('~/Downloads')
return data_dir, lbryum_dir, download_dir
def get_linux_directories() -> typing.Tuple[str, str, str]:
download_dir = None
try:
with open(os.path.join(user_config_dir(), 'user-dirs.dirs'), 'r') as xdg:
down_dir = re.search(r'XDG_DOWNLOAD_DIR=(.+)', xdg.read()).group(1)
down_dir = re.sub('\$HOME', os.getenv('HOME'), down_dir)
download_dir = re.sub('\"', '', down_dir)
except EnvironmentError:
download_dir = os.getenv('XDG_DOWNLOAD_DIR')
if not download_dir:
download_dir = os.path.expanduser('~/Downloads')
# old
data_dir = os.path.expanduser('~/.lbrynet')
lbryum_dir = os.path.expanduser('~/.lbryum')
if os.path.isdir(data_dir) or os.path.isdir(lbryum_dir):
return data_dir, lbryum_dir, download_dir
# new
return user_data_dir('lbry/lbrynet'), user_data_dir('lbry/lbryum'), download_dir
def server_port(server_and_port):
server, port = server_and_port.split(':')
return server, int(port)
@@ -154,7 +122,7 @@ def server_list(servers):
def server_list_reverse(servers):
return ["%s:%s" % (server, port) for server, port in servers]
return [f"{server}:{port}" for server, port in servers]
class Env(envparse.Env):
@@ -228,6 +196,11 @@ FIXED_SETTINGS = {
}
ADJUSTABLE_SETTINGS = {
'data_dir': (str, ''), # these blank defaults will be updated to OS specific defaults
'wallet_dir': (str, ''),
'lbryum_wallet_dir': (str, ''), # to be deprecated
'download_directory': (str, ''),
# By default, daemon will block all cross origin requests
# but if this is set, this value will be used for the
# Access-Control-Allow-Origin. For example
@@ -246,18 +219,15 @@ ADJUSTABLE_SETTINGS = {
# will not be made automatically)
'auto_renew_claim_height_delta': (int, 0),
'cache_time': (int, 150),
'data_dir': (str, default_data_dir),
'data_rate': (float, .0001), # points/megabyte
'delete_blobs_on_remove': (bool, True),
'dht_node_port': (int, 4444),
'download_directory': (str, default_download_dir),
'download_timeout': (int, 180),
'download_mirrors': (list, ['blobs.lbry.io']),
'is_generous_host': (bool, True),
'announce_head_blobs_only': (bool, True),
'concurrent_announcers': (int, DEFAULT_CONCURRENT_ANNOUNCERS),
'known_dht_nodes': (list, DEFAULT_DHT_NODES, server_list, server_list_reverse),
'lbryum_wallet_dir': (str, default_lbryum_dir),
'max_connections_per_stream': (int, 5),
'seek_head_blob_first': (bool, True),
# TODO: writing json on the cmd line is a pain, come up with a nicer
@@ -292,16 +262,34 @@ ADJUSTABLE_SETTINGS = {
}
class Config:
def __init__(self, fixed_defaults, adjustable_defaults, persisted_settings=None,
environment=None, cli_settings=None):
optional_str = typing.Optional[str]
class Config:
def __init__(self, fixed_defaults, adjustable_defaults: typing.Dict, persisted_settings=None, environment=None,
cli_settings=None, data_dir: optional_str = None, wallet_dir: optional_str = None,
download_dir: optional_str = None, file_name: optional_str = None):
self._installation_id = None
self._session_id = base58.b58encode(utils.generate_id()).decode()
self._node_id = None
self._fixed_defaults = fixed_defaults
self._adjustable_defaults = adjustable_defaults
# copy the default adjustable settings
self._adjustable_defaults = {k: v for k, v in adjustable_defaults.items()}
# set the os specific default directories
if platform is WINDOWS:
self.default_data_dir, self.default_wallet_dir, self.default_download_dir = get_windows_directories()
elif platform is DARWIN:
self.default_data_dir, self.default_wallet_dir, self.default_download_dir = get_darwin_directories()
elif platform is LINUX:
self.default_data_dir, self.default_wallet_dir, self.default_download_dir = get_linux_directories()
else:
assert None not in [data_dir, wallet_dir, download_dir]
self.default_data_dir = data_dir
self.default_wallet_dir = wallet_dir
self.default_download_dir = download_dir
self._data = {
TYPE_DEFAULT: {}, # defaults
@@ -338,6 +326,31 @@ class Config:
cli_settings = {}
self._validate_settings(cli_settings)
self._data[TYPE_CLI].update(cli_settings)
self.file_name = file_name or 'daemon_settings.yml'
@property
def data_dir(self) -> optional_str:
data_dir = self.get('data_dir')
if not data_dir:
return
return os.path.expanduser(os.path.expandvars(data_dir))
@property
def download_dir(self) -> optional_str:
download_dir = self.get('download_directory')
if not download_dir:
return
return os.path.expanduser(os.path.expandvars(download_dir))
@property
def wallet_dir(self) -> optional_str:
if self.get('lbryum_wallet_dir') and not self.get('wallet_dir'):
log.warning("'lbryum_wallet_dir' setting will be deprecated, please update to 'wallet_dir'")
self['wallet_dir'] = self['lbryum_wallet_dir']
wallet_dir = self.get('wallet_dir')
if not wallet_dir:
return
return os.path.expanduser(os.path.expandvars(wallet_dir))
def __repr__(self):
return self.get_current_settings_dict().__repr__()
@@ -367,7 +380,7 @@ class Config:
def _assert_valid_data_type(self, data_type):
if data_type not in self._data:
raise KeyError('{} in is not a valid data type'.format(data_type))
raise KeyError(f'{data_type} in is not a valid data type')
def get_valid_setting_names(self):
return self._data[TYPE_DEFAULT].keys()
@@ -377,7 +390,7 @@ class Config:
def _assert_valid_setting(self, name):
if not self._is_valid_setting(name):
raise KeyError('{} is not a valid setting'.format(name))
raise KeyError(f'{name} is not a valid setting')
def _validate_settings(self, data):
invalid_settings = set(data.keys()) - set(self.get_valid_setting_names())
@@ -387,7 +400,7 @@ class Config:
def _assert_editable_setting(self, name):
self._assert_valid_setting(name)
if name in self._fixed_defaults:
raise ValueError('{} is not an editable setting'.format(name))
raise ValueError(f'{name} is not an editable setting')
def _assert_valid_setting_value(self, name, value):
if name == "max_key_fee":
@@ -397,7 +410,7 @@ class Config:
elif name == "download_directory":
directory = str(value)
if not os.path.exists(directory):
raise NoSuchDirectoryError(directory)
log.warning("download directory '%s' does not exist", directory)
def is_default(self, name):
"""Check if a config value is wasn't specified by the user
@@ -442,7 +455,7 @@ class Config:
for possible_data_type in self._search_order:
if name in self._data[possible_data_type]:
return self._data[possible_data_type][name]
raise KeyError('{} is not a valid setting'.format(name))
raise KeyError(f'{name} is not a valid setting')
def set(self, name, value, data_types=(TYPE_RUNTIME,)):
"""Set a config value
@@ -485,16 +498,15 @@ class Config:
}
def save_conf_file_settings(self):
path = conf_file or self.get_valid_settings_filename() or DEFAULT_SETTINGS_FILENAME
# reverse the conversions done after loading the settings from the conf
# file
rev = self._convert_conf_file_lists_reverse(self._data[TYPE_PERSISTED])
ext = os.path.splitext(path)[1]
ext = os.path.splitext(self.file_name)[1]
encoder = settings_encoders.get(ext, False)
if not encoder:
raise ValueError('Unknown settings format: {}. Available formats: {}'
.format(ext, list(settings_encoders.keys())))
with open(path, 'w') as settings_file:
with open(os.path.join(self.data_dir, self.file_name), 'w') as settings_file:
settings_file.write(encoder(rev))
@staticmethod
@@ -522,14 +534,18 @@ class Config:
settings.node_id = settings.get_node_id()
def load_conf_file_settings(self):
path = conf_file or self.get_valid_settings_filename()
self._read_conf_file(path)
path = os.path.join(self.data_dir or self.default_data_dir, self.file_name)
if os.path.isfile(path):
self._read_conf_file(path)
self['data_dir'] = self.data_dir or self.default_data_dir
self['download_directory'] = self.download_dir or self.default_download_dir
self['wallet_dir'] = self.wallet_dir or self.default_wallet_dir
# initialize members depending on config file
self.initialize_post_conf_load()
def _read_conf_file(self, path):
if not path:
return
if not path or not os.path.exists(path):
raise FileNotFoundError(path)
ext = os.path.splitext(path)[1]
decoder = settings_decoders.get(ext, False)
if not decoder:
@@ -564,13 +580,15 @@ class Config:
# although there is a risk of a race condition here we don't
# expect there to be multiple processes accessing this
# directory so the risk can be ignored
if not os.path.isdir(self['data_dir']):
os.makedirs(self['data_dir'])
return self['data_dir']
if not os.path.isdir(self.data_dir):
os.makedirs(self.data_dir)
if not os.path.isdir(os.path.join(self.data_dir, "blobfiles")):
os.makedirs(os.path.join(self.data_dir, "blobfiles"))
return self.data_dir
def ensure_wallet_dir(self):
if not os.path.isdir(self['lbryum_wallet_dir']):
os.makedirs(self['lbryum_wallet_dir'])
if not os.path.isdir(self.wallet_dir):
os.makedirs(self.wallet_dir)
def get_log_filename(self):
"""
@@ -582,7 +600,7 @@ class Config:
def get_api_connection_string(self, user: str = None, password: str = None) -> str:
return 'http%s://%s%s:%i/%s' % (
"" if not self['use_https'] else "s",
"" if not (user and password) else "%s:%s@" % (user, password),
"" if not (user and password) else f"{user}:{password}@",
self['api_host'],
self['api_port'],
self['API_ADDRESS']
@@ -591,15 +609,6 @@ class Config:
def get_db_revision_filename(self):
return os.path.join(self.ensure_data_dir(), self['DB_REVISION_FILE_NAME'])
def get_valid_settings_filename(self):
data_dir = self.ensure_data_dir()
yml_path = os.path.join(data_dir, 'daemon_settings.yml')
json_path = os.path.join(data_dir, 'daemon_settings.json')
if os.path.isfile(yml_path):
return yml_path
elif os.path.isfile(json_path):
return json_path
def get_installation_id(self):
install_id_filename = os.path.join(self.ensure_data_dir(), "install_id")
if not self._installation_id:
@@ -628,7 +637,7 @@ class Config:
return self._session_id
settings = None # type: Config
settings: Config = None
def get_default_env():
@@ -643,10 +652,18 @@ def get_default_env():
return Env(**env_defaults)
def initialize_settings(load_conf_file=True):
def initialize_settings(load_conf_file: typing.Optional[bool] = True,
data_dir: optional_str = None, wallet_dir: optional_str = None,
download_dir: optional_str = None):
global settings
if settings is None:
settings = Config(FIXED_SETTINGS, ADJUSTABLE_SETTINGS,
environment=get_default_env())
environment=get_default_env(), data_dir=data_dir, wallet_dir=wallet_dir,
download_dir=download_dir)
if load_conf_file:
settings.load_conf_file_settings()
settings['data_dir'] = settings.data_dir or settings.default_data_dir
settings['download_directory'] = settings.download_dir or settings.default_download_dir
settings['wallet_dir'] = settings.wallet_dir or settings.default_wallet_dir
settings.ensure_data_dir()
settings.ensure_wallet_dir()
-59
View File
@@ -1,59 +0,0 @@
import platform
import json
import subprocess
import os
from six.moves.urllib import request
from six.moves.urllib.error import URLError
from lbryschema import __version__ as lbryschema_version
from lbrynet import build_type, __version__ as lbrynet_version
from lbrynet.conf import ROOT_DIR
import logging.handlers
log = logging.getLogger(__name__)
def get_lbrynet_version() -> str:
if build_type.BUILD == "dev":
try:
with open(os.devnull, 'w') as devnull:
git_dir = ROOT_DIR + '/.git'
return subprocess.check_output(
['git', '--git-dir='+git_dir, 'describe', '--dirty', '--always'],
stderr=devnull
).decode().strip().lstrip('v')
except (subprocess.CalledProcessError, OSError):
log.debug("failed to get version from git")
return lbrynet_version
def get_platform(get_ip: bool = True) -> dict:
p = {
"processor": platform.processor(),
"python_version": platform.python_version(),
"platform": platform.platform(),
"os_release": platform.release(),
"os_system": platform.system(),
"lbrynet_version": get_lbrynet_version(),
"lbryschema_version": lbryschema_version,
"build": build_type.BUILD, # CI server sets this during build step
}
if p["os_system"] == "Linux":
try:
import distro
p["distro"] = distro.info()
p["desktop"] = os.environ.get('XDG_CURRENT_DESKTOP', 'Unknown')
except ModuleNotFoundError:
pass
# TODO: remove this from get_platform and add a get_external_ip function using treq
if get_ip:
try:
response = json.loads(request.urlopen("https://api.lbry.io/ip").read())
if not response['success']:
raise URLError("failed to get external ip")
p['ip'] = response['data']['ip']
except (URLError, AssertionError):
p['ip'] = "Could not determine IP"
return p
File renamed without changes.
-8
View File
@@ -1,8 +0,0 @@
"""
Classes and functions for dealing with Crypt Streams.
Crypt Streams are encrypted blobs and metadata tying those blobs together. At least some of the
metadata is generally stored in a Stream Descriptor File, for example containing a public key
used to bind blobs to the stream and a symmetric key used to encrypt the blobs. The list of blobs
may or may not be present.
"""
+6 -3
View File
@@ -13,7 +13,7 @@ TRACE = 5
# caller stack frame.
#
if hasattr(sys, 'frozen'): # support for py2exe
_srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
_srcfile = "logging{}__init__{}".format(os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
_srcfile = __file__[:-4] + '.py'
else:
@@ -102,5 +102,8 @@ class Logger(logging.Logger):
self._log(TRACE, msg, args, **kwargs)
logging.setLoggerClass(Logger)
logging.addLevelName(TRACE, 'TRACE')
def install_logger():
current = logging.getLoggerClass()
if current is not Logger:
logging.setLoggerClass(Logger)
logging.addLevelName(TRACE, 'TRACE')
-74
View File
@@ -1,74 +0,0 @@
import os
import sys
# Set SSL_CERT_FILE env variable for Twisted SSL verification on Windows
# This needs to happen before anything else
if 'win' in sys.platform:
import certifi
os.environ['SSL_CERT_FILE'] = certifi.where()
from lbrynet.core import log_support
import argparse
import logging.handlers
from twisted.internet import reactor
from lbrynet import conf
from lbrynet.core import utils, system_info
from lbrynet.daemon.Daemon import Daemon
log = logging.getLogger(__name__)
def test_internet_connection():
return utils.check_connection()
def start(argv=None, conf_path=None):
if conf_path is not None:
conf.conf_file = conf_path
conf.initialize_settings()
parser = argparse.ArgumentParser()
parser.add_argument(
"--http-auth", dest="useauth", action="store_true", default=conf.settings['use_auth_http']
)
parser.add_argument(
'--quiet', dest='quiet', action="store_true",
help='Disable all console output.'
)
parser.add_argument(
'--verbose', nargs="*",
help=('Enable debug output. Optionally specify loggers for which debug output '
'should selectively be applied.')
)
parser.add_argument(
'--version', action="store_true",
help='Show daemon version and quit'
)
args = parser.parse_args(argv)
if args.useauth:
conf.settings.update({'use_auth_http': args.useauth}, data_types=(conf.TYPE_CLI,))
if args.version:
version = system_info.get_platform(get_ip=False)
version['installation_id'] = conf.settings.installation_id
print(utils.json_dumps_pretty(version))
return
lbrynet_log = conf.settings.get_log_filename()
log_support.configure_logging(lbrynet_log, not args.quiet, args.verbose)
log_support.configure_loggly_handler()
log.debug('Final Settings: %s', conf.settings.get_current_settings_dict())
log.info("Starting lbrynet-daemon from command line")
if test_internet_connection():
daemon = Daemon()
daemon.start_listening()
reactor.run()
else:
log.info("Not connected to internet, unable to start")
-1
View File
@@ -1 +0,0 @@
from . import Components # register Component classes
File renamed without changes.
+2 -2
View File
@@ -1,5 +1,5 @@
from collections import UserDict
from . import constants
from lbrynet.dht import constants
class DictDataStore(UserDict):
@@ -44,7 +44,7 @@ class DictDataStore(UserDict):
del self[key]
def hasPeersForBlob(self, key):
return True if key in self and len(tuple(self.filter_bad_and_expired_peers(key))) else False
return bool(key in self and len(tuple(self.filter_bad_and_expired_peers(key))))
def addPeerToBlob(self, contact, key, compact_address, lastPublished, originallyPublished, originalPublisherID):
if key in self:
+1 -1
View File
@@ -1,4 +1,4 @@
from .error import DecodeError
from lbrynet.dht.error import DecodeError
def bencode(data):
-2
View File
@@ -13,14 +13,12 @@ class DecodeError(Exception):
Should be raised by an C{Encoding} implementation if decode operation
fails
"""
pass
class BucketFull(Exception):
"""
Raised when the bucket is full
"""
pass
class UnknownRemoteException(Exception):
-1
View File
@@ -10,7 +10,6 @@ class IDataStore(Interface):
def keys(self):
""" Return a list of the keys in this data store """
pass
def removeExpiredPeers(self):
pass
+7 -7
View File
@@ -1,8 +1,8 @@
import logging
from twisted.internet import defer
from .distance import Distance
from .error import TimeoutError
from . import constants
from lbrynet.dht.distance import Distance
from lbrynet.dht.error import TimeoutError
from lbrynet.dht import constants
log = logging.getLogger(__name__)
@@ -67,7 +67,7 @@ class _IterativeFind:
def getContactTriples(self, result):
if self.is_find_value_request:
contact_triples = result['contacts']
contact_triples = result[b'contacts']
else:
contact_triples = result
for contact_tup in contact_triples:
@@ -112,11 +112,11 @@ class _IterativeFind:
# We are looking for a value, and the remote node didn't have it
# - mark it as the closest "empty" node, if it is
# TODO: store to this peer after finding the value as per the kademlia spec
if 'closestNodeNoValue' in self.find_value_result:
if b'closestNodeNoValue' in self.find_value_result:
if self.is_closer(contact):
self.find_value_result['closestNodeNoValue'] = contact
self.find_value_result[b'closestNodeNoValue'] = contact
else:
self.find_value_result['closestNodeNoValue'] = contact
self.find_value_result[b'closestNodeNoValue'] = contact
contactTriples = self.getContactTriples(result)
for contactTriple in contactTriples:
if (contactTriple[1], contactTriple[2]) in ((c.address, c.port) for c in self.already_contacted):
+3 -3
View File
@@ -1,8 +1,8 @@
import logging
from . import constants
from .distance import Distance
from .error import BucketFull
from lbrynet.dht import constants
from lbrynet.dht.distance import Distance
from lbrynet.dht.error import BucketFull
log = logging.getLogger(__name__)
+1 -1
View File
@@ -7,7 +7,7 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
from . import msgtypes
from lbrynet.dht import msgtypes
class MessageTranslator:
+3 -3
View File
@@ -7,8 +7,8 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
from lbrynet.core.utils import generate_id
from . import constants
from lbrynet.utils import generate_id
from lbrynet.dht import constants
class Message:
@@ -48,5 +48,5 @@ class ErrorMessage(ResponseMessage):
def __init__(self, rpcID, nodeID, exceptionType, errorMessage):
super().__init__(rpcID, nodeID, errorMessage)
if isinstance(exceptionType, type):
exceptionType = ('%s.%s' % (exceptionType.__module__, exceptionType.__name__)).encode()
exceptionType = (f'{exceptionType.__module__}.{exceptionType.__name__}').encode()
self.exceptionType = exceptionType
+21 -30
View File
@@ -5,17 +5,12 @@ from functools import reduce
from twisted.internet import defer, error, task
from lbrynet.core.utils import generate_id, DeferredDict
from lbrynet.core.call_later_manager import CallLaterManager
from lbrynet.core.PeerManager import PeerManager
from .error import TimeoutError
from . import constants
from . import routingtable
from . import datastore
from . import protocol
from .peerfinder import DHTPeerFinder
from .contact import ContactManager
from .iterativefind import iterativeFind
from lbrynet.utils import generate_id, DeferredDict
from lbrynet.dht.call_later_manager import CallLaterManager
from lbrynet.dht.error import TimeoutError
from lbrynet.dht import constants, routingtable, datastore, protocol
from lbrynet.dht.contact import ContactManager
from lbrynet.dht.iterativefind import iterativeFind
log = logging.getLogger(__name__)
@@ -83,8 +78,8 @@ class Node(MockKademliaHelper):
def __init__(self, node_id=None, udpPort=4000, dataStore=None,
routingTableClass=None, networkProtocol=None,
externalIP=None, peerPort=3333, listenUDP=None,
callLater=None, resolve=None, clock=None, peer_finder=None,
peer_manager=None, interface='', externalUDPPort=None):
callLater=None, resolve=None, clock=None,
interface='', externalUDPPort=None):
"""
@param dataStore: The data store to use. This must be class inheriting
from the C{DataStore} interface (or providing the
@@ -124,20 +119,13 @@ class Node(MockKademliaHelper):
else:
self._routingTable = routingTableClass(self.node_id, self.clock.seconds)
# Initialize this node's network access mechanisms
if networkProtocol is None:
self._protocol = protocol.KademliaProtocol(self)
else:
self._protocol = networkProtocol
# Initialize the data storage mechanism used by this node
self._protocol = networkProtocol or protocol.KademliaProtocol(self)
self.token_secret = self._generateID()
self.old_token_secret = None
self.externalIP = externalIP
self.peerPort = peerPort
self.externalUDPPort = externalUDPPort or self.port
self._dataStore = dataStore or datastore.DictDataStore(self.clock.seconds)
self.peer_manager = peer_manager or PeerManager()
self.peer_finder = peer_finder or DHTPeerFinder(self, self.peer_manager)
self._join_deferred = None
#def __del__(self):
@@ -228,10 +216,10 @@ class Node(MockKademliaHelper):
# find the closest peers to us
closest = yield self._iterativeFind(self.node_id, shortlist if not self.contacts else None)
yield _ping_contacts(closest)
# # query random hashes in our bucket key ranges to fill or split them
# random_ids_in_range = self._routingTable.getRefreshList()
# while random_ids_in_range:
# yield self.iterativeFindNode(random_ids_in_range.pop())
# query random hashes in our bucket key ranges to fill or split them
random_ids_in_range = self._routingTable.getRefreshList()
while random_ids_in_range:
yield self.iterativeFindNode(random_ids_in_range.pop())
defer.returnValue(None)
@defer.inlineCallbacks
@@ -255,7 +243,7 @@ class Node(MockKademliaHelper):
yield _iterative_join()
@defer.inlineCallbacks
def start(self, known_node_addresses=None):
def start(self, known_node_addresses=None, block_on_join=False):
""" Causes the Node to attempt to join the DHT network by contacting the
known DHT nodes. This can be called multiple times if the previous attempt
has failed or if the Node has lost all the contacts.
@@ -270,8 +258,11 @@ class Node(MockKademliaHelper):
self.start_listening()
yield self._protocol._listening
# TODO: Refresh all k-buckets further away than this node's closest neighbour
yield self.joinNetwork(known_node_addresses or [])
self.start_looping_calls()
d = self.joinNetwork(known_node_addresses or [])
d.addCallback(lambda _: self.start_looping_calls())
d.addCallback(lambda _: log.info("Joined the dht"))
if block_on_join:
yield d
def start_looping_calls(self):
self.safe_start_looping_call(self._change_token_lc, constants.tokenSecretChangeInterval)
@@ -499,7 +490,7 @@ class Node(MockKademliaHelper):
if 0 <= port <= 65536:
compact_port = port.to_bytes(2, 'big')
else:
raise TypeError('Invalid port: {}'.format(port))
raise TypeError(f'Invalid port: {port}')
compact_address = compact_ip + compact_port + rpc_contact.id
now = int(self.clock.seconds())
originallyPublished = now - age
@@ -580,7 +571,7 @@ class Node(MockKademliaHelper):
"""
return generate_id()
# from lbrynet.core.utils import profile_deferred
# from lbrynet.p2p.utils import profile_deferred
# @profile_deferred()
@defer.inlineCallbacks
def _iterativeFind(self, key, startupShortlist=None, rpc='findNode', exclude=None):
+20 -10
View File
@@ -3,12 +3,8 @@ import errno
from binascii import hexlify
from twisted.internet import protocol, defer
from .error import BUILTIN_EXCEPTIONS, UnknownRemoteException, TimeoutError, TransportNotConnected
from . import constants
from . import encoding
from . import msgtypes
from . import msgformat
from lbrynet.dht import constants, encoding, msgformat, msgtypes
from lbrynet.dht.error import BUILTIN_EXCEPTIONS, UnknownRemoteException, TimeoutError, TransportNotConnected
log = logging.getLogger(__name__)
@@ -118,7 +114,20 @@ class KademliaProtocol(protocol.DatagramProtocol):
return args
return args + ({b'protocolVersion': self._protocolVersion},)
@defer.inlineCallbacks
def sendRPC(self, contact, method, args):
for _ in range(constants.rpcAttempts):
try:
response = yield self._sendRPC(contact, method, args)
return response
except TimeoutError:
if contact.contact_is_good:
log.debug("RETRY %s ON %s", method, contact)
continue
else:
raise
def _sendRPC(self, contact, method, args):
"""
Sends an RPC to the specified contact
@@ -153,11 +162,12 @@ class KademliaProtocol(protocol.DatagramProtocol):
df = defer.Deferred()
def _remove_contact(failure): # remove the contact from the routing table and track the failure
contact.update_last_failed()
try:
self._node.removeContact(contact)
if not contact.contact_is_good:
self._node.removeContact(contact)
except (ValueError, IndexError):
pass
contact.update_last_failed()
return failure
def _update_contact(result): # refresh the contact in the routing table
@@ -338,7 +348,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
while seqNumber < totalPackets:
packetData = data[startPos:startPos + self.msgSizeLimit]
encSeqNumber = chr(seqNumber >> 8) + chr(seqNumber & 0xff)
txData = '\x00%s%s%s\x00%s' % (encTotalPackets, encSeqNumber, rpcID, packetData)
txData = f'\x00{encTotalPackets}{encSeqNumber}{rpcID}\x00{packetData}'
self._scheduleSendNext(txData, address)
startPos += self.msgSizeLimit
@@ -422,7 +432,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
else:
result = func()
except Exception as e:
log.exception("error handling request for %s:%i %s", senderContact.address, senderContact.port, method)
log.error("error handling request for %s:%i %s", senderContact.address, senderContact.port, method)
df.errback(e)
else:
df.callback(result)
+20 -19
View File
@@ -6,14 +6,12 @@
# may be created by processing this file with epydoc: http://epydoc.sf.net
import random
from binascii import unhexlify
from twisted.internet import defer
from . import constants
from . import kbucket
from .error import TimeoutError
from .distance import Distance
import logging
from twisted.internet import defer
from lbrynet.dht import constants, kbucket
from lbrynet.dht.error import TimeoutError
from lbrynet.dht.distance import Distance
log = logging.getLogger(__name__)
@@ -215,7 +213,7 @@ class TreeRoutingTable:
now = int(self._getTime())
for bucket in self._buckets[startIndex:]:
if force or now - bucket.lastAccessed >= constants.refreshTimeout:
searchID = self._randomIDInBucketRange(bucketIndex)
searchID = self.midpoint_id_in_bucket_range(bucketIndex)
refreshIDs.append(searchID)
bucketIndex += 1
return refreshIDs
@@ -263,22 +261,25 @@ class TreeRoutingTable:
i += 1
return i
def _randomIDInBucketRange(self, bucketIndex):
def random_id_in_bucket_range(self, bucketIndex):
""" Returns a random ID in the specified k-bucket's range
@param bucketIndex: The index of the k-bucket to use
@type bucketIndex: int
"""
idValue = random.randrange(
self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax)
randomID = hex(idValue)[2:]
if randomID[-1] == 'L':
randomID = randomID[:-1]
if len(randomID) % 2 != 0:
randomID = '0' + randomID
randomID = unhexlify(randomID)
randomID = ((constants.key_bits // 8) - len(randomID)) * b'\x00' + randomID
return randomID
random_id = int(random.randrange(self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax))
return random_id.to_bytes(constants.key_bits // 8, 'big')
def midpoint_id_in_bucket_range(self, bucketIndex):
""" Returns the middle ID in the specified k-bucket's range
@param bucketIndex: The index of the k-bucket to use
@type bucketIndex: int
"""
half = int((self._buckets[bucketIndex].rangeMax - self._buckets[bucketIndex].rangeMin) // 2)
return int(self._buckets[bucketIndex].rangeMin + half).to_bytes(constants.key_bits // 8, 'big')
def _splitBucket(self, oldBucketIndex):
""" Splits the specified k-bucket into two new buckets which together
File renamed without changes.
+284
View File
@@ -0,0 +1,284 @@
import sys
import os
import json
import asyncio
import argparse
import typing
# Set SSL_CERT_FILE env variable for Twisted SSL verification on Windows
# This needs to happen before anything else
if 'win' in sys.platform:
import certifi
os.environ['SSL_CERT_FILE'] = certifi.where()
from twisted.internet import asyncioreactor
if 'twisted.internet.reactor' not in sys.modules:
asyncioreactor.install()
else:
from twisted.internet import reactor
if not isinstance(reactor, asyncioreactor.AsyncioSelectorReactor) and getattr(sys, 'frozen', False):
# pyinstaller hooks install the default reactor before
# any of our code runs, see kivy for similar problem:
# https://github.com/kivy/kivy/issues/4182
del sys.modules['twisted.internet.reactor']
asyncioreactor.install()
from twisted.internet import reactor
import logging
from aiohttp.client_exceptions import ClientConnectorError
from requests.exceptions import ConnectionError
from docopt import docopt
from textwrap import dedent
from lbrynet import conf, log_support, __name__ as lbrynet_name
from lbrynet.utils import check_connection, json_dumps_pretty
from lbrynet.extras.daemon.Daemon import Daemon
from lbrynet.extras.daemon.DaemonConsole import main as daemon_console
from lbrynet.extras.daemon.auth.client import LBRYAPIClient
from lbrynet.extras.system_info import get_platform
log = logging.getLogger(lbrynet_name)
optional_path_getter_type = typing.Optional[typing.Callable[[], str]]
def start_daemon(settings: typing.Optional[typing.Dict] = None,
console_output: typing.Optional[bool] = True, verbose: typing.Optional[typing.List[str]] = None,
data_dir: typing.Optional[str] = None, wallet_dir: typing.Optional[str] = None,
download_dir: typing.Optional[str] = None):
settings = settings or {}
conf.initialize_settings(data_dir=data_dir, wallet_dir=wallet_dir, download_dir=download_dir)
for k, v in settings.items():
conf.settings.update({k, v}, data_types=(conf.TYPE_CLI,))
log_support.configure_logging(conf.settings.get_log_filename(), console_output, verbose)
log_support.configure_loggly_handler()
log.debug('Final Settings: %s', conf.settings.get_current_settings_dict())
log.info("Starting lbrynet-daemon from command line")
if check_connection():
daemon = Daemon()
daemon.start_listening()
reactor.run()
else:
log.info("Not connected to internet, unable to start")
def start_daemon_with_cli_args(argv=None, data_dir: typing.Optional[str] = None,
wallet_dir: typing.Optional[str] = None, download_dir: typing.Optional[str] = None):
parser = argparse.ArgumentParser()
parser.add_argument(
"--http-auth", dest="useauth", action="store_true", default=False
)
parser.add_argument(
'--quiet', dest='quiet', action="store_true",
help='Disable all console output.'
)
parser.add_argument(
'--verbose', nargs="*",
help=('Enable debug output. Optionally specify loggers for which debug output '
'should selectively be applied.')
)
parser.add_argument(
'--version', action="store_true",
help='Show daemon version and quit'
)
args = parser.parse_args(argv)
settings = {}
if args.useauth:
settings['use_auth_http'] = True
verbose = None
if args.verbose:
verbose = args.verbose
console_output = not args.quiet
if args.version:
print(json_dumps_pretty(get_platform()))
return
return start_daemon(settings, console_output, verbose, data_dir, wallet_dir, download_dir)
async def execute_command(method, params, data_dir: typing.Optional[str] = None,
wallet_dir: typing.Optional[str] = None, download_dir: typing.Optional[str] = None):
# this check if the daemon is running or not
conf.initialize_settings(data_dir=data_dir, wallet_dir=wallet_dir, download_dir=download_dir)
api = None
try:
api = await LBRYAPIClient.get_client()
await api.status()
except (ClientConnectorError, ConnectionError):
if api:
await api.session.close()
print("Could not connect to daemon. Are you sure it's running?")
return 1
# this actually executes the method
resp = await api.call(method, params)
try:
await api.session.close()
print(json.dumps(resp["result"], indent=2))
except KeyError:
if resp["error"]["code"] == -32500:
print(json.dumps(resp["error"], indent=2))
else:
print(json.dumps(resp["error"]["message"], indent=2))
def print_help():
print(dedent("""
NAME
lbrynet - LBRY command line client.
USAGE
lbrynet [--data_dir=<blob and database directory>] [--wallet_dir=<wallet directory>]
[--download_dir=<downloads directory>] <command> [<args>]
EXAMPLES
lbrynet start # starts the daemon and listens for jsonrpc commands
lbrynet help # display this message
lbrynet help <command_name> # get help for a command(doesn't need the daemon to be running)
lbrynet commands # list available commands
lbrynet status # get the running status of the daemon
lbrynet resolve what # resolve a name
lbrynet --wallet_dir=~/wallet2 start # start the daemon using an alternative wallet directory
lbrynet --data_dir=~/lbry start # start the daemon using an alternative data directory
lbrynet --data_dir=~/lbry <command_name> # run a command on a daemon using an alternative data directory,
# which can contain a full daemon_settings.yml config file.
# Note: since the daemon is what runs the wallet and
# downloads files, only the --data_dir setting is needed when
# running commands. The wallet_dir and download_dir would only
# by used when starting the daemon.
"""))
def print_help_for_command(command):
fn = Daemon.callable_methods.get(command)
if fn:
print(dedent(fn.__doc__))
else:
print("Invalid command name")
def normalize_value(x, key=None):
if not isinstance(x, str):
return x
if key in ('uri', 'channel_name', 'name', 'file_name', 'download_directory'):
return x
if x.lower() == 'true':
return True
if x.lower() == 'false':
return False
if x.isdigit():
return int(x)
return x
def remove_brackets(key):
if key.startswith("<") and key.endswith(">"):
return str(key[1:-1])
return key
def set_kwargs(parsed_args):
kwargs = {}
for key, arg in parsed_args.items():
k = None
if arg is None:
continue
elif key.startswith("--") and remove_brackets(key[2:]) not in kwargs:
k = remove_brackets(key[2:])
elif remove_brackets(key) not in kwargs:
k = remove_brackets(key)
kwargs[k] = normalize_value(arg, k)
return kwargs
def main(argv=None):
argv = argv or sys.argv[1:]
if not argv:
print_help()
return 1
dir_args = {}
if len(argv) > 2:
dir_arg_keys = [
'data_dir',
'wallet_dir',
'download_directory'
]
for arg in argv:
found_dir_arg = False
for key in dir_arg_keys:
if arg.startswith(f'--{key}='):
if key in dir_args:
print(f"Multiple values provided for '{key}' argument")
print_help()
return 1
dir_args[key] = os.path.expanduser(os.path.expandvars(arg.lstrip(f'--{key}=')))
found_dir_arg = True
if not found_dir_arg:
break
argv = argv[len(dir_args):]
data_dir = dir_args.get('data_dir')
wallet_dir = dir_args.get('wallet_dir')
download_dir = dir_args.get('download_directory')
for k, v in dir_args.items():
if not os.path.isdir(v):
print(f"'{data_dir}' is not a directory, cannot use it for {k}")
return 1
method, args = argv[0], argv[1:]
if method in ['help', '--help', '-h']:
if len(args) == 1:
print_help_for_command(args[0])
else:
print_help()
return 0
elif method in ['version', '--version', '-v']:
print("{lbrynet_name} {lbrynet_version}".format(
lbrynet_name=lbrynet_name, **get_platform()
))
return 0
elif method == 'start':
sys.exit(start_daemon_with_cli_args(args, data_dir, wallet_dir, download_dir))
elif method == 'console':
sys.exit(daemon_console())
elif method not in Daemon.callable_methods:
if method not in Daemon.deprecated_methods:
print(f'{method} is not a valid command.')
return 1
new_method = Daemon.deprecated_methods[method].new_command
if new_method is None:
print(f"{method} is permanently deprecated and does not have a replacement command.")
return 0
print(f"{method} is deprecated, using {new_method}.")
method = new_method
fn = Daemon.callable_methods[method]
parsed = docopt(fn.__doc__, args)
params = set_kwargs(parsed)
loop = asyncio.get_event_loop()
loop.run_until_complete(execute_command(method, params, data_dir, wallet_dir, download_dir))
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -1,7 +1,7 @@
import logging
from twisted.internet import defer
from twisted._threads import AlreadyQuit
from .ComponentManager import ComponentManager
from lbrynet.extras.daemon.ComponentManager import ComponentManager
log = logging.getLogger(__name__)
@@ -1,7 +1,8 @@
import logging
from twisted.internet import defer
from lbrynet.core.Error import ComponentStartConditionNotMet
from lbrynet.p2p.Error import ComponentStartConditionNotMet
from lbrynet.extras.daemon.PeerManager import PeerManager
from lbrynet.extras.daemon.PeerFinder import DHTPeerFinder
log = logging.getLogger(__name__)
@@ -33,13 +34,15 @@ class RequiredCondition(metaclass=RequiredConditionType):
class ComponentManager:
default_component_classes = {}
def __init__(self, reactor=None, analytics_manager=None, skip_components=None, **override_components):
def __init__(self, reactor=None, analytics_manager=None, skip_components=None,
peer_manager=None, peer_finder=None, **override_components):
self.skip_components = skip_components or []
self.reactor = reactor
self.component_classes = {}
self.components = set()
self.analytics_manager = analytics_manager
self.peer_manager = peer_manager or PeerManager()
self.peer_finder = peer_finder or DHTPeerFinder(self)
for component_name, component_class in self.default_component_classes.items():
if component_name in override_components:
@@ -60,7 +63,8 @@ class ComponentManager:
try:
component = self.get_component(condition.component)
result = condition.evaluate(component)
except Exception as err:
except Exception:
log.exception('failed to evaluate condition:')
result = False
return result, "" if result else condition.message
@@ -113,10 +117,10 @@ class ComponentManager:
:return: (defer.Deferred)
"""
for component_name, cb in callbacks.items():
if component_name not in self.component_classes:
raise NameError("unknown component: %s" % component_name)
if component_name not in self.skip_components:
raise NameError("unknown component: %s" % component_name)
if not callable(cb):
raise ValueError("%s is not callable" % cb)
@@ -2,34 +2,39 @@ import os
import asyncio
import logging
import treq
import json
import math
import binascii
from hashlib import sha256
from types import SimpleNamespace
from twisted.internet import defer, threads, reactor, error, task
import lbryschema
from aioupnp import __version__ as aioupnp_version
from aioupnp.upnp import UPnP
from aioupnp.fault import UPnPError
from lbrynet import conf
from lbrynet.core.utils import DeferredDict
from lbrynet.core.PaymentRateManager import OnlyFreePaymentsManager
from lbrynet.core.RateLimiter import RateLimiter
from lbrynet.core.BlobManager import DiskBlobManager
from lbrynet.core.StreamDescriptor import StreamDescriptorIdentifier, EncryptedFileStreamType
from lbrynet.wallet.manager import LbryWalletManager
from lbrynet.wallet.network import Network
from lbrynet.core.server.BlobRequestHandler import BlobRequestHandlerFactory
from lbrynet.core.server.ServerProtocol import ServerProtocolFactory
from lbrynet.daemon.Component import Component
from lbrynet.daemon.ExchangeRateManager import ExchangeRateManager
from lbrynet.database.storage import SQLiteStorage
from lbrynet.dht import node, hashannouncer
from lbrynet.file_manager.EncryptedFileManager import EncryptedFileManager
from lbrynet.lbry_file.client.EncryptedFileDownloader import EncryptedFileSaverFactory
from lbrynet.lbry_file.client.EncryptedFileOptions import add_lbry_file_to_sd_identifier
from lbrynet.reflector import ServerFactory as reflector_server_factory
from lbrynet.core.utils import generate_id
import lbrynet.schema
from lbrynet import conf
from lbrynet.blob.EncryptedFileManager import EncryptedFileManager
from lbrynet.blob.client.EncryptedFileDownloader import EncryptedFileSaverFactory
from lbrynet.blob.client.EncryptedFileOptions import add_lbry_file_to_sd_identifier
from lbrynet.dht.node import Node
from lbrynet.extras.daemon.Component import Component
from lbrynet.extras.daemon.ExchangeRateManager import ExchangeRateManager
from lbrynet.extras.daemon.storage import SQLiteStorage
from lbrynet.extras.daemon.HashAnnouncer import DHTHashAnnouncer
from lbrynet.extras.reflector.server.server import ReflectorServerFactory
from lbrynet.extras.wallet import LbryWalletManager
from lbrynet.extras.wallet import Network
from lbrynet.utils import DeferredDict, generate_id
from lbrynet.p2p.PaymentRateManager import OnlyFreePaymentsManager
from lbrynet.p2p.RateLimiter import RateLimiter
from lbrynet.p2p.BlobManager import DiskBlobManager
from lbrynet.p2p.StreamDescriptor import StreamDescriptorIdentifier, EncryptedFileStreamType
from lbrynet.p2p.server.BlobRequestHandler import BlobRequestHandlerFactory
from lbrynet.p2p.server.ServerProtocol import ServerProtocolFactory
log = logging.getLogger(__name__)
@@ -41,7 +46,6 @@ HEADERS_COMPONENT = "blockchain_headers"
WALLET_COMPONENT = "wallet"
DHT_COMPONENT = "dht"
HASH_ANNOUNCER_COMPONENT = "hash_announcer"
STREAM_IDENTIFIER_COMPONENT = "stream_identifier"
FILE_MANAGER_COMPONENT = "file_manager"
PEER_PROTOCOL_SERVER_COMPONENT = "peer_protocol_server"
REFLECTOR_COMPONENT = "reflector"
@@ -55,53 +59,18 @@ def from_future(coroutine: asyncio.coroutine) -> defer.Deferred:
return defer.Deferred.fromFuture(asyncio.ensure_future(coroutine))
def get_wallet_config():
wallet_type = GCS('wallet')
if wallet_type == conf.LBRYCRD_WALLET:
raise ValueError('LBRYcrd Wallet is no longer supported')
elif wallet_type != conf.LBRYUM_WALLET:
raise ValueError('Wallet Type {} is not valid'.format(wallet_type))
lbryum_servers = {address: {'t': str(port)}
for address, port in GCS('lbryum_servers')}
config = {
'auto_connect': True,
'chain': GCS('blockchain_name'),
'default_servers': lbryum_servers
}
if 'use_keyring' in conf.settings:
config['use_keyring'] = GCS('use_keyring')
if conf.settings['lbryum_wallet_dir']:
config['lbryum_path'] = GCS('lbryum_wallet_dir')
return config
class ConfigSettings:
@staticmethod
def get_conf_setting(setting_name):
return conf.settings[setting_name]
@staticmethod
def get_blobfiles_dir():
if conf.settings['BLOBFILES_DIR'] == "blobfiles":
return os.path.join(GCS("data_dir"), "blobfiles")
else:
log.info("Using non-default blobfiles directory: %s", conf.settings['BLOBFILES_DIR'])
return conf.settings['BLOBFILES_DIR']
@staticmethod
def get_node_id():
return conf.settings.node_id
@staticmethod
def get_external_ip():
from lbrynet.core.system_info import get_platform
platform = get_platform(get_ip=True)
return platform['ip']
# Shorthand for common ConfigSettings methods
CS = ConfigSettings
GCS = ConfigSettings.get_conf_setting
@defer.inlineCallbacks
def get_external_ip(): # used if upnp is disabled or non-functioning
try:
buf = []
response = yield treq.get("https://api.lbry.io/ip")
yield treq.collect(response, buf.append)
parsed = json.loads(b"".join(buf).decode())
if parsed['success']:
return parsed['data']['ip']
return
except Exception as err:
return
class DatabaseComponent(Component):
@@ -117,7 +86,7 @@ class DatabaseComponent(Component):
@staticmethod
def get_current_db_revision():
return 9
return 10
@staticmethod
def get_revision_filename():
@@ -133,18 +102,6 @@ class DatabaseComponent(Component):
# check directories exist, create them if they don't
log.info("Loading databases")
if not os.path.exists(GCS('download_directory')):
os.mkdir(GCS('download_directory'))
if not os.path.exists(GCS('data_dir')):
os.mkdir(GCS('data_dir'))
self._write_db_revision_file(self.get_current_db_revision())
log.debug("Created the db revision file: %s", self.get_revision_filename())
if not os.path.exists(CS.get_blobfiles_dir()):
os.mkdir(CS.get_blobfiles_dir())
log.debug("Created the blobfile directory: %s", str(CS.get_blobfiles_dir()))
if not os.path.exists(self.get_revision_filename()):
log.warning("db_revision file not found. Creating it")
self._write_db_revision_file(self.get_current_db_revision())
@@ -158,16 +115,16 @@ class DatabaseComponent(Component):
'Your database is revision %i, expected %i' %
(old_revision, self.get_current_db_revision()))
if old_revision < self.get_current_db_revision():
from lbrynet.database.migrator import dbmigrator
from lbrynet.extras.daemon.migrator import dbmigrator
log.info("Upgrading your databases (revision %i to %i)", old_revision, self.get_current_db_revision())
yield threads.deferToThread(
dbmigrator.migrate_db, GCS('data_dir'), old_revision, self.get_current_db_revision()
dbmigrator.migrate_db, conf.settings.data_dir, old_revision, self.get_current_db_revision()
)
self._write_db_revision_file(self.get_current_db_revision())
log.info("Finished upgrading the databases.")
# start SQLiteStorage
self.storage = SQLiteStorage(GCS('data_dir'))
self.storage = SQLiteStorage(conf.settings.data_dir)
yield self.storage.setup()
@defer.inlineCallbacks
@@ -185,11 +142,11 @@ class HeadersComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self.headers_dir = os.path.join(conf.settings['lbryum_wallet_dir'], 'lbc_mainnet')
self.headers_dir = os.path.join(conf.settings.wallet_dir, 'lbc_mainnet')
self.headers_file = os.path.join(self.headers_dir, 'headers')
self.old_file = os.path.join(conf.settings['lbryum_wallet_dir'], 'blockchain_headers')
self.old_file = os.path.join(conf.settings.wallet_dir, 'blockchain_headers')
self._downloading_headers = None
self._headers_progress_percent = None
self._headers_progress_percent = 0
@property
def component(self):
@@ -210,7 +167,7 @@ class HeadersComponent(Component):
self._headers_progress_percent = math.ceil(local_size / final_size * 100)
local_header_size = self.local_header_file_size()
resume_header = {"Range": "bytes={}-".format(local_header_size)}
resume_header = {"Range": f"bytes={local_header_size}-"}
response = yield treq.get(HEADERS_URL, headers=resume_header)
got_406 = response.code == 406 # our file is bigger
final_size_after_download = response.length + local_header_size
@@ -247,7 +204,7 @@ class HeadersComponent(Component):
ledger = SimpleNamespace()
ledger.config = {
'default_servers': conf.settings['lbryum_servers'],
'data_path': conf.settings['lbryum_wallet_dir']
'data_path': conf.settings.wallet_dir
}
net = Network(ledger)
first_connection = net.on_connected.first
@@ -284,7 +241,7 @@ class HeadersComponent(Component):
hashsum.update(headers_file.read(checksum_length_in_bytes))
current_checksum = hashsum.hexdigest()
if current_checksum != checksum:
msg = "Expected checksum {}, got {}".format(checksum, current_checksum)
msg = f"Expected checksum {checksum}, got {current_checksum}"
log.warning("Wallet file corrupted, checksum mismatch. " + msg)
log.warning("Deleting header file so it can be downloaded again.")
os.unlink(self.headers_file)
@@ -335,7 +292,7 @@ class WalletComponent(Component):
return self.wallet_manager
def get_status(self):
if self.wallet_manager:
if self.wallet_manager and self.running:
local_height = self.wallet_manager.network.get_local_height()
remote_height = self.wallet_manager.network.get_server_height()
best_hash = self.wallet_manager.get_best_blockhash()
@@ -352,7 +309,7 @@ class WalletComponent(Component):
conf.settings.ensure_wallet_dir()
log.info("Starting torba wallet")
storage = self.component_manager.get_component(DATABASE_COMPONENT)
lbryschema.BLOCKCHAIN_NAME = conf.settings['blockchain_name']
lbrynet.schema.BLOCKCHAIN_NAME = conf.settings['blockchain_name']
self.wallet_manager = yield f2d(LbryWalletManager.from_lbrynet_config(conf.settings, storage))
self.wallet_manager.old_db = storage
yield f2d(self.wallet_manager.start())
@@ -365,7 +322,7 @@ class WalletComponent(Component):
class BlobComponent(Component):
component_name = BLOB_COMPONENT
depends_on = [DATABASE_COMPONENT, DHT_COMPONENT]
depends_on = [DATABASE_COMPONENT]
def __init__(self, component_manager):
super().__init__(component_manager)
@@ -377,8 +334,12 @@ class BlobComponent(Component):
def start(self):
storage = self.component_manager.get_component(DATABASE_COMPONENT)
dht_node = self.component_manager.get_component(DHT_COMPONENT)
self.blob_manager = DiskBlobManager(CS.get_blobfiles_dir(), storage, dht_node._dataStore)
datastore = None
if DHT_COMPONENT not in self.component_manager.skip_components:
dht_node = self.component_manager.get_component(DHT_COMPONENT)
if dht_node:
datastore = dht_node._dataStore
self.blob_manager = DiskBlobManager(os.path.join(conf.settings.data_dir, "blobfiles"), storage, datastore)
return self.blob_manager.setup()
def stop(self):
@@ -411,32 +372,34 @@ class DHTComponent(Component):
def get_status(self):
return {
'node_id': binascii.hexlify(CS.get_node_id()),
'node_id': binascii.hexlify(conf.settings.get_node_id()),
'peers_in_routing_table': 0 if not self.dht_node else len(self.dht_node.contacts)
}
@defer.inlineCallbacks
def start(self):
self.upnp_component = self.component_manager.get_component(UPNP_COMPONENT)
self.external_peer_port = self.upnp_component.upnp_redirects.get("TCP", GCS("peer_port"))
self.external_udp_port = self.upnp_component.upnp_redirects.get("UDP", GCS("dht_node_port"))
node_id = CS.get_node_id()
self.external_peer_port = self.upnp_component.upnp_redirects.get("TCP", conf.settings["peer_port"])
self.external_udp_port = self.upnp_component.upnp_redirects.get("UDP", conf.settings["dht_node_port"])
node_id = conf.settings.get_node_id()
if node_id is None:
node_id = generate_id()
external_ip = self.upnp_component.external_ip
if not external_ip:
log.warning("UPnP component failed to get external ip")
external_ip = yield get_external_ip()
if not external_ip:
log.warning("failed to get external ip")
self.dht_node = node.Node(
self.dht_node = Node(
node_id=node_id,
udpPort=GCS('dht_node_port'),
udpPort=conf.settings['dht_node_port'],
externalUDPPort=self.external_udp_port,
externalIP=self.upnp_component.external_ip,
externalIP=external_ip,
peerPort=self.external_peer_port
)
self.dht_node.start_listening()
yield self.dht_node._protocol._listening
d = self.dht_node.joinNetwork(GCS('known_dht_nodes'))
d.addCallback(lambda _: self.dht_node.start_looping_calls())
d.addCallback(lambda _: log.info("Joined the dht"))
yield self.dht_node.start(conf.settings['known_dht_nodes'], block_on_join=False)
log.info("Started the dht")
@defer.inlineCallbacks
@@ -460,7 +423,7 @@ class HashAnnouncerComponent(Component):
def start(self):
storage = self.component_manager.get_component(DATABASE_COMPONENT)
dht_node = self.component_manager.get_component(DHT_COMPONENT)
self.hash_announcer = hashannouncer.DHTHashAnnouncer(dht_node, storage)
self.hash_announcer = DHTHashAnnouncer(dht_node, storage)
yield self.hash_announcer.start()
@defer.inlineCallbacks
@@ -493,41 +456,6 @@ class RateLimiterComponent(Component):
return defer.succeed(None)
class StreamIdentifierComponent(Component):
component_name = STREAM_IDENTIFIER_COMPONENT
depends_on = [DHT_COMPONENT, RATE_LIMITER_COMPONENT, BLOB_COMPONENT, DATABASE_COMPONENT, WALLET_COMPONENT]
def __init__(self, component_manager):
super().__init__(component_manager)
self.sd_identifier = StreamDescriptorIdentifier()
@property
def component(self):
return self.sd_identifier
@defer.inlineCallbacks
def start(self):
dht_node = self.component_manager.get_component(DHT_COMPONENT)
rate_limiter = self.component_manager.get_component(RATE_LIMITER_COMPONENT)
blob_manager = self.component_manager.get_component(BLOB_COMPONENT)
storage = self.component_manager.get_component(DATABASE_COMPONENT)
wallet = self.component_manager.get_component(WALLET_COMPONENT)
add_lbry_file_to_sd_identifier(self.sd_identifier)
file_saver_factory = EncryptedFileSaverFactory(
dht_node.peer_finder,
rate_limiter,
blob_manager,
storage,
wallet,
GCS('download_directory')
)
yield self.sd_identifier.add_stream_downloader_factory(EncryptedFileStreamType, file_saver_factory)
def stop(self):
pass
class PaymentRateComponent(Component):
component_name = PAYMENT_RATE_COMPONENT
@@ -548,8 +476,8 @@ class PaymentRateComponent(Component):
class FileManagerComponent(Component):
component_name = FILE_MANAGER_COMPONENT
depends_on = [DHT_COMPONENT, RATE_LIMITER_COMPONENT, BLOB_COMPONENT, DATABASE_COMPONENT, WALLET_COMPONENT,
STREAM_IDENTIFIER_COMPONENT, PAYMENT_RATE_COMPONENT]
depends_on = [RATE_LIMITER_COMPONENT, BLOB_COMPONENT, DATABASE_COMPONENT, WALLET_COMPONENT,
PAYMENT_RATE_COMPONENT]
def __init__(self, component_manager):
super().__init__(component_manager)
@@ -568,15 +496,26 @@ class FileManagerComponent(Component):
@defer.inlineCallbacks
def start(self):
dht_node = self.component_manager.get_component(DHT_COMPONENT)
rate_limiter = self.component_manager.get_component(RATE_LIMITER_COMPONENT)
blob_manager = self.component_manager.get_component(BLOB_COMPONENT)
storage = self.component_manager.get_component(DATABASE_COMPONENT)
wallet = self.component_manager.get_component(WALLET_COMPONENT)
sd_identifier = self.component_manager.get_component(STREAM_IDENTIFIER_COMPONENT)
sd_identifier = StreamDescriptorIdentifier()
add_lbry_file_to_sd_identifier(sd_identifier)
file_saver_factory = EncryptedFileSaverFactory(
self.component_manager.peer_finder,
rate_limiter,
blob_manager,
storage,
wallet,
conf.settings.download_dir
)
yield sd_identifier.add_stream_downloader_factory(EncryptedFileStreamType, file_saver_factory)
payment_rate_manager = self.component_manager.get_component(PAYMENT_RATE_COMPONENT)
log.info('Starting the file manager')
self.file_manager = EncryptedFileManager(dht_node.peer_finder, rate_limiter, blob_manager, wallet,
self.file_manager = EncryptedFileManager(self.component_manager.peer_finder, rate_limiter, blob_manager, wallet,
payment_rate_manager, storage, sd_identifier)
yield self.file_manager.setup()
log.info('Done setting up file manager')
@@ -588,7 +527,7 @@ class FileManagerComponent(Component):
class PeerProtocolServerComponent(Component):
component_name = PEER_PROTOCOL_SERVER_COMPONENT
depends_on = [UPNP_COMPONENT, DHT_COMPONENT, RATE_LIMITER_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT,
depends_on = [UPNP_COMPONENT, RATE_LIMITER_COMPONENT, BLOB_COMPONENT, WALLET_COMPONENT,
PAYMENT_RATE_COMPONENT]
def __init__(self, component_manager):
@@ -603,7 +542,7 @@ class PeerProtocolServerComponent(Component):
def start(self):
wallet = self.component_manager.get_component(WALLET_COMPONENT)
upnp = self.component_manager.get_component(UPNP_COMPONENT)
peer_port = GCS('peer_port')
peer_port = conf.settings['peer_port']
query_handlers = {
handler.get_primary_query_identifier(): handler for handler in [
BlobRequestHandlerFactory(
@@ -617,7 +556,7 @@ class PeerProtocolServerComponent(Component):
}
server_factory = ServerProtocolFactory(
self.component_manager.get_component(RATE_LIMITER_COMPONENT), query_handlers,
self.component_manager.get_component(DHT_COMPONENT).peer_manager
self.component_manager.peer_manager
)
try:
@@ -641,11 +580,11 @@ class PeerProtocolServerComponent(Component):
class ReflectorComponent(Component):
component_name = REFLECTOR_COMPONENT
depends_on = [DHT_COMPONENT, BLOB_COMPONENT, FILE_MANAGER_COMPONENT]
depends_on = [BLOB_COMPONENT, FILE_MANAGER_COMPONENT]
def __init__(self, component_manager):
super().__init__(component_manager)
self.reflector_server_port = GCS('reflector_port')
self.reflector_server_port = conf.settings['reflector_port']
self.reflector_server = None
@property
@@ -655,16 +594,15 @@ class ReflectorComponent(Component):
@defer.inlineCallbacks
def start(self):
log.info("Starting reflector server")
dht_node = self.component_manager.get_component(DHT_COMPONENT)
blob_manager = self.component_manager.get_component(BLOB_COMPONENT)
file_manager = self.component_manager.get_component(FILE_MANAGER_COMPONENT)
reflector_factory = reflector_server_factory(dht_node.peer_manager, blob_manager, file_manager)
reflector_factory = ReflectorServerFactory(self.component_manager.peer_manager, blob_manager, file_manager)
try:
self.reflector_server = yield reactor.listenTCP(self.reflector_server_port, reflector_factory)
log.info('Started reflector on port %s', self.reflector_server_port)
except error.CannotListenError as e:
log.exception("Couldn't bind reflector to port %d", self.reflector_server_port)
raise ValueError("{} lbrynet may already be running on your computer.".format(e))
raise ValueError(f"{e} lbrynet may already be running on your computer.")
@defer.inlineCallbacks
def stop(self):
@@ -679,9 +617,9 @@ class UPnPComponent(Component):
def __init__(self, component_manager):
super().__init__(component_manager)
self._int_peer_port = GCS('peer_port')
self._int_dht_node_port = GCS('dht_node_port')
self.use_upnp = GCS('use_upnp')
self._int_peer_port = conf.settings['peer_port']
self._int_dht_node_port = conf.settings['dht_node_port']
self.use_upnp = conf.settings['use_upnp']
self.upnp = None
self.upnp_redirects = {}
self.external_ip = None
@@ -694,10 +632,12 @@ class UPnPComponent(Component):
@defer.inlineCallbacks
def _setup_redirects(self):
upnp_redirects = yield DeferredDict({
"UDP": from_future(self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port")),
"TCP": from_future(self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port"))
})
d = {}
if PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components:
d["TCP"] = from_future(self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port"))
if DHT_COMPONENT not in self.component_manager.skip_components:
d["UDP"] = from_future(self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port"))
upnp_redirects = yield DeferredDict(d)
self.upnp_redirects.update(upnp_redirects)
@defer.inlineCallbacks
@@ -709,33 +649,41 @@ class UPnPComponent(Component):
log.info("found upnp gateway: %s", self.upnp.gateway.manufacturer_string)
except Exception as err:
log.warning("upnp discovery failed: %s", err)
return
self.upnp = None
# update the external ip
try:
external_ip = yield from_future(self.upnp.get_external_ip())
if external_ip == "0.0.0.0":
log.warning("upnp doesn't know the external ip address (returned 0.0.0.0), using fallback")
external_ip = CS.get_external_ip()
if self.external_ip and self.external_ip != external_ip:
log.info("external ip changed from %s to %s", self.external_ip, external_ip)
elif not self.external_ip:
log.info("got external ip: %s", external_ip)
self.external_ip = external_ip
except (asyncio.TimeoutError, UPnPError):
pass
if not self.upnp_redirects: # setup missing redirects
external_ip = None
if self.upnp:
try:
upnp_redirects = yield DeferredDict({
"UDP": from_future(self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port")),
"TCP": from_future(self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port"))
})
external_ip = yield from_future(self.upnp.get_external_ip())
if external_ip != "0.0.0.0" and not self.external_ip:
log.info("got external ip from UPnP: %s", external_ip)
except (asyncio.TimeoutError, UPnPError):
pass
if external_ip == "0.0.0.0" or not external_ip:
log.warning("unable to get external ip from UPnP, checking lbry.io fallback")
external_ip = yield get_external_ip()
if self.external_ip and self.external_ip != external_ip:
log.info("external ip changed from %s to %s", self.external_ip, external_ip)
self.external_ip = external_ip
assert self.external_ip is not None # TODO: handle going/starting offline
if not self.upnp_redirects and self.upnp: # setup missing redirects
try:
log.info("add UPnP port mappings")
d = {}
if PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components:
d["TCP"] = from_future(self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port"))
if DHT_COMPONENT not in self.component_manager.skip_components:
d["UDP"] = from_future(self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port"))
upnp_redirects = yield DeferredDict(d)
log.info("set up redirects: %s", upnp_redirects)
self.upnp_redirects.update(upnp_redirects)
except (asyncio.TimeoutError, UPnPError):
self.upnp = None
return self._maintain_redirects()
else: # check existing redirects are still active
elif self.upnp: # check existing redirects are still active
found = set()
mappings = yield from_future(self.upnp.get_redirects())
for mapping in mappings:
@@ -743,7 +691,7 @@ class UPnPComponent(Component):
if proto in self.upnp_redirects and mapping['NewExternalPort'] == self.upnp_redirects[proto]:
if mapping['NewInternalClient'] == self.upnp.lan_address:
found.add(proto)
if 'UDP' not in found:
if 'UDP' not in found and DHT_COMPONENT not in self.component_manager.skip_components:
try:
udp_port = yield from_future(
self.upnp.get_next_mapping(self._int_dht_node_port, "UDP", "LBRY DHT port")
@@ -752,7 +700,7 @@ class UPnPComponent(Component):
log.info("refreshed upnp redirect for dht port: %i", udp_port)
except (asyncio.TimeoutError, UPnPError):
del self.upnp_redirects['UDP']
if 'TCP' not in found:
if 'TCP' not in found and PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components:
try:
tcp_port = yield from_future(
self.upnp.get_next_mapping(self._int_peer_port, "TCP", "LBRY peer port")
@@ -761,22 +709,28 @@ class UPnPComponent(Component):
log.info("refreshed upnp redirect for peer port: %i", tcp_port)
except (asyncio.TimeoutError, UPnPError):
del self.upnp_redirects['TCP']
if 'TCP' in self.upnp_redirects and 'UDP' in self.upnp_redirects:
log.debug("upnp redirects are still active")
if ('TCP' in self.upnp_redirects
and PEER_PROTOCOL_SERVER_COMPONENT not in self.component_manager.skip_components) and (
'UDP' in self.upnp_redirects and DHT_COMPONENT not in self.component_manager.skip_components):
if self.upnp_redirects:
log.debug("upnp redirects are still active")
@defer.inlineCallbacks
def start(self):
log.info("detecting external ip")
if not self.use_upnp:
self.external_ip = CS.get_external_ip()
self.external_ip = yield get_external_ip()
return
success = False
yield self._maintain_redirects()
if self.upnp:
if not self.upnp_redirects:
if not self.upnp_redirects and not all([x in self.component_manager.skip_components for x in
(DHT_COMPONENT, PEER_PROTOCOL_SERVER_COMPONENT)]):
log.error("failed to setup upnp, debugging infomation: %s", self.upnp.zipped_debugging_info)
else:
success = True
log.debug("set up upnp port redirects for gateway: %s", self.upnp.gateway.manufacturer_string)
if self.upnp_redirects:
log.debug("set up upnp port redirects for gateway: %s", self.upnp.gateway.manufacturer_string)
else:
log.error("failed to setup upnp")
self.component_manager.analytics_manager.send_upnp_setup_success_fail(success, self.get_status())
@@ -792,8 +746,9 @@ class UPnPComponent(Component):
def get_status(self):
return {
'aioupnp_version': aioupnp_version,
'redirects': self.upnp_redirects,
'gateway': '' if not self.upnp else self.upnp.gateway.manufacturer_string,
'gateway': 'No gateway found' if not self.upnp else self.upnp.gateway.manufacturer_string,
'dht_redirect_set': 'UDP' in self.upnp_redirects,
'peer_redirect_set': 'TCP' in self.upnp_redirects,
'external_ip': self.external_ip
@@ -1,5 +1,4 @@
import logging.handlers
import mimetypes
import os
import requests
import urllib
@@ -14,41 +13,37 @@ from twisted.internet import defer, reactor
from twisted.internet.task import LoopingCall
from twisted.python.failure import Failure
from torba.constants import COIN
from torba.baseaccount import SingleKey, HierarchicalDeterministic
from torba.client.baseaccount import SingleKey, HierarchicalDeterministic
from lbryschema.claim import ClaimDict
from lbryschema.uri import parse_lbry_uri
from lbryschema.error import URIParseError, DecodeError
from lbryschema.validator import validate_claim_id
from lbryschema.address import decode_address
from lbryschema.decode import smart_decode
# TODO: importing this when internet is disabled raises a socket.gaierror
from lbrynet.core.system_info import get_lbrynet_version
from lbrynet import conf
from lbrynet.reflector import reupload
from lbrynet.daemon.Components import d2f, f2d
from lbrynet.daemon.Components import WALLET_COMPONENT, DATABASE_COMPONENT, DHT_COMPONENT, BLOB_COMPONENT
from lbrynet.daemon.Components import STREAM_IDENTIFIER_COMPONENT, FILE_MANAGER_COMPONENT, RATE_LIMITER_COMPONENT
from lbrynet.daemon.Components import EXCHANGE_RATE_MANAGER_COMPONENT, PAYMENT_RATE_COMPONENT, UPNP_COMPONENT
from lbrynet.daemon.ComponentManager import RequiredCondition
from lbrynet.daemon.Downloader import GetStream
from lbrynet.daemon.Publisher import Publisher
from lbrynet.daemon.auth.server import AuthJSONRPCServer
from lbrynet.core import utils, system_info
from lbrynet.core.StreamDescriptor import download_sd_blob
from lbrynet.core.Error import InsufficientFundsError, UnknownNameError
from lbrynet.core.Error import DownloadDataTimeout, DownloadSDTimeout
from lbrynet.core.Error import NullFundsError, NegativeFundsError
from lbrynet.core.Error import ResolveError
from lbrynet import conf, utils, __version__
from lbrynet.dht.error import TimeoutError
from lbrynet.core.Peer import Peer
from lbrynet.core.SinglePeerDownloader import SinglePeerDownloader
from lbrynet.core.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.wallet.account import Account as LBCAccount
from lbrynet.wallet.manager import LbryWalletManager
from lbrynet.wallet.dewies import dewies_to_lbc, lbc_to_dewies
from lbrynet.blob.blob_file import is_valid_blobhash
from lbrynet.extras import system_info
from lbrynet.extras.reflector import reupload
from lbrynet.extras.daemon.Components import d2f, f2d
from lbrynet.extras.daemon.Components import WALLET_COMPONENT, DATABASE_COMPONENT, DHT_COMPONENT, BLOB_COMPONENT
from lbrynet.extras.daemon.Components import FILE_MANAGER_COMPONENT, RATE_LIMITER_COMPONENT
from lbrynet.extras.daemon.Components import EXCHANGE_RATE_MANAGER_COMPONENT, PAYMENT_RATE_COMPONENT, UPNP_COMPONENT
from lbrynet.extras.daemon.ComponentManager import RequiredCondition
from lbrynet.extras.daemon.Downloader import GetStream
from lbrynet.extras.daemon.Publisher import Publisher
from lbrynet.extras.daemon.auth.server import AuthJSONRPCServer
from lbrynet.extras.daemon.mime_types import guess_mime_type
from lbrynet.extras.wallet import LbryWalletManager
from lbrynet.extras.wallet.account import Account as LBCAccount
from lbrynet.extras.wallet.dewies import dewies_to_lbc, lbc_to_dewies
from lbrynet.p2p.StreamDescriptor import download_sd_blob
from lbrynet.p2p.Error import InsufficientFundsError, UnknownNameError, DownloadDataTimeout, DownloadSDTimeout
from lbrynet.p2p.Error import NullFundsError, NegativeFundsError, ResolveError
from lbrynet.p2p.Peer import Peer
from lbrynet.p2p.SinglePeerDownloader import SinglePeerDownloader
from lbrynet.p2p.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.schema.claim import ClaimDict
from lbrynet.schema.uri import parse_lbry_uri
from lbrynet.schema.error import URIParseError, DecodeError
from lbrynet.schema.validator import validate_claim_id
from lbrynet.schema.address import decode_address
from lbrynet.schema.decode import smart_decode
log = logging.getLogger(__name__)
requires = AuthJSONRPCServer.requires
@@ -171,9 +166,9 @@ def sort_claim_results(claims):
def is_first_run():
if os.path.isfile(conf.settings.get_db_revision_filename()):
return False
if os.path.isfile(os.path.join(conf.settings['data_dir'], 'lbrynet.sqlite')):
if os.path.isfile(os.path.join(conf.settings.data_dir, 'lbrynet.sqlite')):
return False
if os.path.isfile(os.path.join(conf.settings['lbryum_wallet_dir'], 'blockchain_headers')):
if os.path.isfile(os.path.join(conf.settings.wallet_dir, 'blockchain_headers')):
return False
return True
@@ -211,7 +206,6 @@ class Daemon(AuthJSONRPCServer):
DATABASE_COMPONENT: "storage",
DHT_COMPONENT: "dht_node",
WALLET_COMPONENT: "wallet_manager",
STREAM_IDENTIFIER_COMPONENT: "sd_identifier",
FILE_MANAGER_COMPONENT: "file_manager",
EXCHANGE_RATE_MANAGER_COMPONENT: "exchange_rate_manager",
PAYMENT_RATE_COMPONENT: "payment_rate_manager",
@@ -221,7 +215,7 @@ class Daemon(AuthJSONRPCServer):
}
def __init__(self, analytics_manager=None, component_manager=None):
to_skip = list(conf.settings['components_to_skip'])
to_skip = conf.settings['components_to_skip']
if 'reflector' not in to_skip and not conf.settings['run_reflector_server']:
to_skip.append('reflector')
looping_calls = {
@@ -242,7 +236,6 @@ class Daemon(AuthJSONRPCServer):
self.storage = None
self.dht_node = None
self.wallet_manager: LbryWalletManager = None
self.sd_identifier = None
self.file_manager = None
self.exchange_rate_manager = None
self.payment_rate_manager = None
@@ -278,7 +271,7 @@ class Daemon(AuthJSONRPCServer):
def setup(self):
log.info("Starting lbrynet-daemon")
log.info("Platform: %s", json.dumps(system_info.get_platform()))
yield super(Daemon, self).setup()
yield super().setup()
log.info("Started lbrynet-daemon")
def _stop_streams(self):
@@ -288,7 +281,7 @@ class Daemon(AuthJSONRPCServer):
def _shutdown(self):
self._stop_streams()
return super(Daemon, self)._shutdown()
return super()._shutdown()
def _download_blob(self, blob_hash, rate_manager=None, timeout=None):
"""
@@ -306,7 +299,7 @@ class Daemon(AuthJSONRPCServer):
rate_manager = rate_manager or self.payment_rate_manager
timeout = timeout or 30
downloader = StandaloneBlobDownloader(
blob_hash, self.blob_manager, self.dht_node.peer_finder, self.rate_limiter,
blob_hash, self.blob_manager, self.component_manager.peer_finder, self.rate_limiter,
rate_manager, self.wallet_manager, timeout
)
return downloader.download()
@@ -373,8 +366,8 @@ class Daemon(AuthJSONRPCServer):
self.analytics_manager.send_download_started(download_id, name, claim_dict)
self.analytics_manager.send_new_download_start(download_id, name, claim_dict)
self.streams[sd_hash] = GetStream(
self.sd_identifier, self.wallet_manager, self.exchange_rate_manager, self.blob_manager,
self.dht_node.peer_finder, self.rate_limiter, self.payment_rate_manager, self.storage,
self.file_manager.sd_identifier, self.wallet_manager, self.exchange_rate_manager, self.blob_manager,
self.component_manager.peer_finder, self.rate_limiter, self.payment_rate_manager, self.storage,
conf.settings['max_key_fee'], conf.settings['disable_max_key_fee'], conf.settings['data_rate'],
timeout
)
@@ -433,7 +426,7 @@ class Daemon(AuthJSONRPCServer):
if blob:
return self.blob_manager.get_blob(blob[0])
return download_sd_blob(
sd_hash.decode(), self.blob_manager, self.dht_node.peer_finder, self.rate_limiter,
sd_hash.decode(), self.blob_manager, self.component_manager.peer_finder, self.rate_limiter,
self.payment_rate_manager, self.wallet_manager, timeout=conf.settings['peer_search_timeout'],
download_mirrors=conf.settings['download_mirrors']
)
@@ -451,7 +444,7 @@ class Daemon(AuthJSONRPCServer):
Get total stream size in bytes from a sd blob
"""
d = self.sd_identifier.get_metadata_for_sd_blob(sd_blob)
d = self.file_manager.sd_identifier.get_metadata_for_sd_blob(sd_blob)
d.addCallback(lambda metadata: metadata.validator.info_to_show())
d.addCallback(lambda info: int(dict(info)['stream_size']))
return d
@@ -469,9 +462,7 @@ class Daemon(AuthJSONRPCServer):
"""
Calculate estimated LBC cost for a stream given its size in bytes
"""
cost = self._get_est_cost_from_stream_size(size)
resolved = await self.wallet_manager.resolve(uri)
if uri in resolved and 'claim' in resolved[uri]:
@@ -536,7 +527,6 @@ class Daemon(AuthJSONRPCServer):
sd blob will be downloaded to determine the stream size
"""
if size is not None:
return self.get_est_cost_using_known_size(uri, size)
return self.get_est_cost_from_uri(uri)
@@ -545,7 +535,7 @@ class Daemon(AuthJSONRPCServer):
def _get_lbry_file_dict(self, lbry_file):
key = hexlify(lbry_file.key) if lbry_file.key else None
full_path = os.path.join(lbry_file.download_directory, lbry_file.file_name)
mime_type = mimetypes.guess_type(full_path)[0]
mime_type = guess_mime_type(lbry_file.file_name)
if os.path.isfile(full_path):
with open(full_path) as written_file:
written_file.seek(0, os.SEEK_END)
@@ -597,7 +587,7 @@ class Daemon(AuthJSONRPCServer):
lbry_file = l_f
break
else:
raise NoValidSearch('{} is not a valid search operation'.format(search_by))
raise NoValidSearch(f'{search_by} is not a valid search operation')
if return_json and lbry_file:
lbry_file = yield self._get_lbry_file_dict(lbry_file)
defer.returnValue(lbry_file)
@@ -773,6 +763,7 @@ class Daemon(AuthJSONRPCServer):
'managed_files': (int) count of files in the file manager,
},
'upnp': {
'aioupnp_version': (str),
'redirects': {
<TCP | UDP>: (int) external_port,
},
@@ -851,7 +842,7 @@ class Daemon(AuthJSONRPCServer):
message,
conf.settings.installation_id,
platform_name,
get_lbrynet_version()
__version__
)
return self._render_response(True)
@@ -984,7 +975,7 @@ class Daemon(AuthJSONRPCServer):
fn = self.callable_methods.get(command)
if fn is None:
raise Exception(
"No help available for '{}'. It is not a valid command.".format(command)
f"No help available for '{command}'. It is not a valid command."
)
return self._render_response({
@@ -1224,7 +1215,7 @@ class Daemon(AuthJSONRPCServer):
)
if self.ledger.network.is_connected:
await self.ledger.update_account(account)
await self.ledger.subscribe_account(account)
self.default_wallet.save()
@@ -1259,7 +1250,7 @@ class Daemon(AuthJSONRPCServer):
)
if self.ledger.network.is_connected:
await self.ledger.update_account(account)
await self.ledger.subscribe_account(account)
self.default_wallet.save()
@@ -1448,7 +1439,7 @@ class Daemon(AuthJSONRPCServer):
return self.get_account_or_error(account_id).get_max_gap()
@requires("wallet")
def jsonrpc_account_fund(self, to_account, from_account, amount='0.0',
def jsonrpc_account_fund(self, to_account=None, from_account=None, amount='0.0',
everything=False, outputs=1, broadcast=False):
"""
Transfer some amount (or --everything) to an account from another
@@ -1457,8 +1448,8 @@ class Daemon(AuthJSONRPCServer):
be used together with --everything).
Usage:
account_fund (<to_account> | --to_account=<to_account>)
(<from_account> | --from_account=<from_account>)
account_fund [<to_account> | --to_account=<to_account>]
[<from_account> | --from_account=<from_account>]
(<amount> | --amount=<amount> | --everything)
[<outputs> | --outputs=<outputs>]
[--broadcast]
@@ -1475,8 +1466,8 @@ class Daemon(AuthJSONRPCServer):
(map) transaction performing requested action
"""
to_account = self.get_account_or_error(to_account, 'to_account')
from_account = self.get_account_or_error(from_account, 'from_account')
to_account = self.get_account_or_default(to_account, 'to_account')
from_account = self.get_account_or_default(from_account, 'from_account')
amount = self.get_dewies_or_error('amount', amount) if amount else None
if not isinstance(outputs, int):
raise ValueError("--outputs must be an integer.")
@@ -1487,6 +1478,35 @@ class Daemon(AuthJSONRPCServer):
outputs=outputs, broadcast=broadcast
)
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_account_send(self, amount, addresses, account_id=None, broadcast=False):
"""
Send the same number of credits to multiple addresses.
Usage:
account_send <amount> <addresses>... [--account_id=<account_id>] [--broadcast]
Options:
--account_id=<account_id> : (str) account to fund the transaction
--broadcast : (bool) actually broadcast the transaction, default: false.
Returns:
"""
amount = self.get_dewies_or_error("amount", amount)
if not amount:
raise NullFundsError
elif amount < 0:
raise NegativeFundsError()
for address in addresses:
decode_address(address)
account = self.get_account_or_default(account_id)
result = await account.send_to_addresses(amount, addresses, broadcast)
self.analytics_manager.send_credits_sent()
return result
@requires(WALLET_COMPONENT)
def jsonrpc_address_is_mine(self, address, account_id=None):
"""
@@ -1776,8 +1796,8 @@ class Daemon(AuthJSONRPCServer):
results[resolved_uri] = resolved[resolved_uri]
return results
@requires(STREAM_IDENTIFIER_COMPONENT, WALLET_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, BLOB_COMPONENT,
DHT_COMPONENT, RATE_LIMITER_COMPONENT, PAYMENT_RATE_COMPONENT, DATABASE_COMPONENT,
@requires(WALLET_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, BLOB_COMPONENT,
RATE_LIMITER_COMPONENT, PAYMENT_RATE_COMPONENT, DATABASE_COMPONENT,
conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_get(self, uri, file_name=None, timeout=None):
"""
@@ -1836,7 +1856,8 @@ class Daemon(AuthJSONRPCServer):
raise ResolveError(
"Failed to resolve stream at lbry://{}".format(uri.replace("lbry://", ""))
)
if 'error' in resolved:
raise ResolveError(f"error resolving stream: {resolved['error']}")
txid, nout, name = resolved['txid'], resolved['nout'], resolved['name']
claim_dict = ClaimDict.load_dict(resolved['value'])
sd_hash = claim_dict.source_hash.decode()
@@ -1888,7 +1909,7 @@ class Daemon(AuthJSONRPCServer):
search_type, value = get_lbry_file_search_value(kwargs)
lbry_file = yield self._get_lbry_file(search_type, value, return_json=False)
if not lbry_file:
raise Exception('Unable to find a file for {}:{}'.format(search_type, value))
raise Exception(f'Unable to find a file for {search_type}:{value}')
if status == 'start' and lbry_file.stopped or status == 'stop' and not lbry_file.stopped:
yield self.file_manager.toggle_lbry_file_running(lbry_file)
@@ -1962,7 +1983,7 @@ class Daemon(AuthJSONRPCServer):
response = yield self._render_response(result)
defer.returnValue(response)
@requires(STREAM_IDENTIFIER_COMPONENT, WALLET_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, BLOB_COMPONENT,
@requires(WALLET_COMPONENT, EXCHANGE_RATE_MANAGER_COMPONENT, BLOB_COMPONENT,
DHT_COMPONENT, RATE_LIMITER_COMPONENT, PAYMENT_RATE_COMPONENT, DATABASE_COMPONENT,
conditions=[WALLET_IS_UNLOCKED])
def jsonrpc_stream_cost_estimate(self, uri, size=None):
@@ -2206,19 +2227,14 @@ class Daemon(AuthJSONRPCServer):
available = await account.get_balance()
if amount >= available:
# TODO: add check for existing claim balance
#balance = yield self.wallet.get_max_usable_balance_for_claim(name)
#max_bid_amount = balance - MAX_UPDATE_FEE_ESTIMATE
#if balance <= MAX_UPDATE_FEE_ESTIMATE:
raise InsufficientFundsError(
"Insufficient funds, please deposit additional LBC. Minimum additional LBC needed {}"
.format(round((amount - available) / COIN + 0.01, 2))
)
# .format(MAX_UPDATE_FEE_ESTIMATE - balance))
#elif bid > max_bid_amount:
# raise InsufficientFundsError(
# "Please lower the bid value, the maximum amount you can specify for this claim is {}."
# .format(max_bid_amount))
existing_claims = await account.get_claims(claim_name=name)
if len(existing_claims) == 1:
available += existing_claims[0].get_estimator(self.ledger).effective_amount
if amount >= available:
raise InsufficientFundsError(
f"Please lower the bid value, the maximum amount "
f"you can specify for this claim is {dewies_to_lbc(available)}."
)
metadata = metadata or {}
if fee is not None:
@@ -2270,7 +2286,7 @@ class Daemon(AuthJSONRPCServer):
}
}
# this will be used to verify the format with lbryschema
# this will be used to verify the format with lbrynet.schema
claim_copy = deepcopy(claim_dict)
if sources is not None:
claim_dict['stream']['source'] = sources
@@ -2291,11 +2307,11 @@ class Daemon(AuthJSONRPCServer):
raise Exception("no source provided to publish")
try:
ClaimDict.load_dict(claim_copy)
# the metadata to use in the claim can be serialized by lbryschema
# the metadata to use in the claim can be serialized by lbrynet.schema
except DecodeError as err:
# there was a problem with a metadata field, raise an error here rather than
# waiting to find out when we go to publish the claim (after having made the stream)
raise Exception("invalid publish metadata: %s" % err.message)
raise Exception(f"invalid publish metadata: {err}")
certificate = None
if channel_id or channel_name:
@@ -2320,7 +2336,7 @@ class Daemon(AuthJSONRPCServer):
)
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_claim_abandon(self, claim_id=None, txid=None, nout=None, account_id=None):
async def jsonrpc_claim_abandon(self, claim_id=None, txid=None, nout=None, account_id=None, blocking=True):
"""
Abandon a name and reclaim credits from the claim
@@ -2328,12 +2344,14 @@ class Daemon(AuthJSONRPCServer):
claim_abandon [<claim_id> | --claim_id=<claim_id>]
[<txid> | --txid=<txid>] [<nout> | --nout=<nout>]
[--account_id=<account_id>]
[--blocking]
Options:
--claim_id=<claim_id> : (str) claim_id of the claim to abandon
--txid=<txid> : (str) txid of the claim to abandon
--nout=<nout> : (int) nout of the claim to abandon
--account_id=<account_id> : (str) id of the account to use
--blocking : (bool) wait until abandon is in mempool
Returns:
(dict) Dictionary containing result of the claim
@@ -2353,6 +2371,8 @@ class Daemon(AuthJSONRPCServer):
tx = await self.wallet_manager.abandon_claim(claim_id, txid, nout, account)
self.analytics_manager.send_claim_action('abandon')
if blocking:
await self.ledger.wait(tx)
return {"success": True, "tx": tx}
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
@@ -2613,6 +2633,8 @@ class Daemon(AuthJSONRPCServer):
results[chan_uri] = {"error": "%s is not a valid uri" % chan_uri}
resolved = await self.wallet_manager.resolve(*valid_uris, page=page, page_size=page_size)
if 'error' in resolved:
return {'error': resolved['error']}
for u in resolved:
if 'error' in resolved[u]:
results[u] = resolved[u]
@@ -2812,7 +2834,7 @@ class Daemon(AuthJSONRPCServer):
Delete a blob
Usage:
blob_delete (<blob_hash> | --blob_hash=<blob_hash)
blob_delete (<blob_hash> | --blob_hash=<blob_hash>)
Options:
--blob_hash=<blob_hash> : (str) blob hash of the blob to delete
@@ -2848,7 +2870,7 @@ class Daemon(AuthJSONRPCServer):
(list) List of contact dictionaries {'host': <peer ip>, 'port': <peer port>, 'node_id': <peer node id>}
"""
if not utils.is_valid_blobhash(blob_hash):
if not is_valid_blobhash(blob_hash):
raise Exception("invalid blob hash")
finished_deferred = self.dht_node.iterativeFindValue(unhexlify(blob_hash))
@@ -3124,6 +3146,7 @@ class Daemon(AuthJSONRPCServer):
result['buckets'] = {}
for i in range(len(self.dht_node._routingTable._buckets)):
result['buckets'][i] = []
for contact in self.dht_node._routingTable._buckets[i]._contacts:
blobs = list(hosts.pop(contact)) if contact in hosts else []
blob_hashes.update(blobs)
@@ -3133,7 +3156,7 @@ class Daemon(AuthJSONRPCServer):
"node_id": hexlify(contact.id).decode(),
"blobs": blobs,
}
result['buckets'].setdefault(i, []).append(host)
result['buckets'][i].append(host)
contact_set.add(hexlify(contact.id).decode())
result['contacts'] = list(contact_set)
@@ -3282,7 +3305,7 @@ class Daemon(AuthJSONRPCServer):
certificates = await self.wallet_manager.get_certificates(
private_key_accounts=accounts, claim_name=channel_name)
if not certificates:
raise ValueError("Couldn't find channel with name '{}'.".format(channel_name))
raise ValueError(f"Couldn't find channel with name '{channel_name}'.")
return certificates[0]
raise ValueError("Couldn't find channel because a channel name or channel_id was not provided.")
@@ -3307,7 +3330,7 @@ class Daemon(AuthJSONRPCServer):
.format(account_id, account.ledger.symbol, argument_name)
)
return account
raise ValueError("Couldn't find account: {}.".format(account_id))
raise ValueError(f"Couldn't find account: {account_id}.")
@staticmethod
def get_dewies_or_error(argument: str, lbc: str):
@@ -3356,7 +3379,7 @@ def get_lbry_file_search_value(search_fields):
value = search_fields.get(searchtype, None)
if value is not None:
return searchtype, value
raise NoValidSearch('{} is missing a valid search type'.format(search_fields))
raise NoValidSearch(f'{search_fields} is missing a valid search type')
def iter_lbry_file_search_values(search_fields):
@@ -6,12 +6,10 @@ import logging.handlers
from twisted.internet import defer, reactor, threads
from aiohttp import client_exceptions
from lbrynet import analytics
from lbrynet import conf
from lbrynet.core import utils
from lbrynet.core import log_support
from lbrynet.daemon.auth.client import LBRYAPIClient
from lbrynet.daemon.Daemon import Daemon
from lbrynet import utils, conf, log_support
from lbrynet.extras.daemon import analytics
from lbrynet.extras.daemon.auth.client import LBRYAPIClient
from lbrynet.extras.daemon.Daemon import Daemon
log = logging.getLogger(__name__)
@@ -105,7 +103,7 @@ To exit:
"""
welcome = "{:*^60}\n".format(" Welcome to the lbrynet interactive console! ")
welcome += "\n".join(["{:<60}".format(w) for w in welcometext.splitlines()])
welcome += "\n".join([f"{w:<60}" for w in welcometext.splitlines()])
welcome += "\n%s" % ("*" * 60)
welcome = color(welcome, "grey")
banner = color(logo, "green") + color(welcome, "grey")
@@ -156,7 +154,7 @@ def run_terminal(callable_methods, started_daemon, quiet=False):
try:
reactor.callLater(0, reactor.stop)
except Exception as err:
print("error stopping reactor: {}".format(err))
print(f"error stopping reactor: {err}")
return sys.exit(status)
locs.update({'exit': exit})
@@ -1,19 +1,17 @@
import logging
import os
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from lbrynet.daemon.Components import f2d
from lbryschema.fee import Fee
from lbrynet.core.Error import InsufficientFundsError, KeyFeeAboveMaxAllowed, InvalidStreamDescriptorError
from lbrynet.core.Error import DownloadDataTimeout, DownloadCanceledError, DownloadSDTimeout
from lbrynet.core.utils import safe_start_looping_call, safe_stop_looping_call
from lbrynet.core.StreamDescriptor import download_sd_blob
from lbrynet.file_manager.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet import conf
from torba.constants import COIN
from lbrynet.wallet.dewies import dewies_to_lbc
from lbrynet.schema.fee import Fee
from lbrynet.p2p.Error import InsufficientFundsError, KeyFeeAboveMaxAllowed, InvalidStreamDescriptorError
from lbrynet.p2p.Error import DownloadDataTimeout, DownloadCanceledError
from lbrynet.p2p.StreamDescriptor import download_sd_blob
from lbrynet.blob.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from torba.client.constants import COIN
from lbrynet.extras.wallet.dewies import dewies_to_lbc
from lbrynet.extras.daemon.Components import f2d
INITIALIZING_CODE = 'initializing'
DOWNLOAD_METADATA_CODE = 'downloading_metadata'
@@ -34,13 +32,16 @@ log = logging.getLogger(__name__)
class GetStream:
def __init__(self, sd_identifier, wallet, exchange_rate_manager, blob_manager, peer_finder, rate_limiter,
payment_rate_manager, storage, max_key_fee, disable_max_key_fee, data_rate=None, timeout=None):
payment_rate_manager, storage, max_key_fee, disable_max_key_fee, data_rate=None, timeout=None,
reactor=None):
if not reactor:
from twisted.internet import reactor
self.reactor = reactor
self.timeout = timeout or conf.settings['download_timeout']
self.data_rate = data_rate or conf.settings['data_rate']
self.max_key_fee = max_key_fee or conf.settings['max_key_fee'][1]
self.disable_max_key_fee = disable_max_key_fee or conf.settings['disable_max_key_fee']
self.download_directory = conf.settings['download_directory']
self.download_directory = conf.settings.download_dir
self.timeout_counter = 0
self.code = None
self.sd_hash = None
@@ -53,44 +54,17 @@ class GetStream:
self.sd_identifier = sd_identifier
self.storage = storage
self.downloader = None
self.checker = LoopingCall(self.check_status)
# fired when the download is complete
self.finished_deferred = None
# fired after the metadata and the first data blob have been downloaded
self.data_downloading_deferred = defer.Deferred(None)
self.wrote_data = False
@property
def download_path(self):
return os.path.join(self.download_directory, self.downloader.file_name)
def _check_status(self, status):
if status.num_completed > 0 and not self.data_downloading_deferred.called:
self.data_downloading_deferred.callback(True)
if self.data_downloading_deferred.called:
safe_stop_looping_call(self.checker)
else:
log.debug("Waiting for stream data (%i seconds)", self.timeout_counter)
def check_status(self):
"""
Check if we've got the first data blob in the stream yet
"""
self.timeout_counter += 1
if self.timeout_counter > self.timeout:
if not self.data_downloading_deferred.called:
if self.downloader:
err = DownloadDataTimeout(self.sd_hash)
else:
err = DownloadSDTimeout(self.sd_hash)
self.data_downloading_deferred.errback(err)
safe_stop_looping_call(self.checker)
elif self.downloader:
d = self.downloader.status()
d.addCallback(self._check_status)
else:
log.debug("Waiting for stream descriptor (%i seconds)", self.timeout_counter)
def convert_max_fee(self):
currency, amount = self.max_key_fee['currency'], self.max_key_fee['amount']
return self.exchange_rate_manager.convert_currency(currency, "LBC", amount)
@@ -107,8 +81,8 @@ class GetStream:
if converted_fee_amount > (yield f2d(self.wallet.default_account.get_balance())):
raise InsufficientFundsError('Unable to pay the key fee of %s' % converted_fee_amount)
if converted_fee_amount > max_key_fee_amount and not self.disable_max_key_fee:
raise KeyFeeAboveMaxAllowed('Key fee %s above max allowed %s' % (converted_fee_amount,
max_key_fee_amount))
raise KeyFeeAboveMaxAllowed('Key fee {} above max allowed {}'.format(converted_fee_amount,
max_key_fee_amount))
converted_fee = {
'currency': 'LBC',
'amount': converted_fee_amount,
@@ -120,7 +94,7 @@ class GetStream:
for factory in factories:
if isinstance(factory, ManagedEncryptedFileDownloaderFactory):
return factory
raise Exception('No suitable factory was found in {}'.format(factories))
raise Exception(f'No suitable factory was found in {factories}')
@defer.inlineCallbacks
def get_downloader(self, factory, stream_metadata, file_name=None):
@@ -140,7 +114,7 @@ class GetStream:
reserved_points = self.wallet.reserve_points(address, fee_lbc)
if reserved_points is None:
raise InsufficientFundsError(
'Unable to pay the key fee of %s for %s' % (dewies_to_lbc(fee_lbc), name)
'Unable to pay the key fee of {} for {}'.format(dewies_to_lbc(fee_lbc), name)
)
return f2d(self.wallet.send_points_to_address(reserved_points, fee_lbc))
@@ -151,18 +125,13 @@ class GetStream:
else:
defer.returnValue(None)
@defer.inlineCallbacks
def finish(self, results, name):
self.set_status(DOWNLOAD_STOPPED_CODE, name)
log.info("Finished downloading lbry://%s (%s) --> %s", name, self.sd_hash[:6],
self.download_path)
safe_stop_looping_call(self.checker)
status = yield self.downloader.status()
self._check_status(status)
defer.returnValue(self.download_path)
return defer.succeed(self.download_path)
def fail(self, err):
safe_stop_looping_call(self.checker)
raise err
@defer.inlineCallbacks
@@ -194,8 +163,10 @@ class GetStream:
self.downloader = yield self._create_downloader(sd_blob, file_name=file_name)
yield self.pay_key_fee(key_fee, name)
yield self.storage.save_content_claim(self.downloader.stream_hash, "%s:%i" % (txid, nout))
log.info("Downloading lbry://%s (%s) --> %s", name, self.sd_hash[:6], self.download_path)
self.finished_deferred = self.downloader.start()
self.downloader.download_manager.progress_manager.wrote_first_data.addCallback(
self.data_downloading_deferred.callback
)
self.finished_deferred.addCallbacks(lambda result: self.finish(result, name), self.fail)
@defer.inlineCallbacks
@@ -211,16 +182,19 @@ class GetStream:
"""
self.set_status(INITIALIZING_CODE, name)
key_fee = yield self._initialize(stream_info)
safe_start_looping_call(self.checker, 1)
self.set_status(DOWNLOAD_METADATA_CODE, name)
try:
sd_blob = yield self._download_sd_blob()
yield self._download(sd_blob, name, key_fee, txid, nout, file_name)
self.set_status(DOWNLOAD_RUNNING_CODE, name)
yield self.data_downloading_deferred
log.info("Downloading lbry://%s (%s) --> %s", name, self.sd_hash[:6], self.download_path)
self.data_downloading_deferred.addTimeout(self.timeout, self.reactor)
try:
yield self.data_downloading_deferred
self.wrote_data = True
except defer.TimeoutError:
raise DownloadDataTimeout("data download timed out")
except (DownloadDataTimeout, InvalidStreamDescriptorError) as err:
safe_stop_looping_call(self.checker)
raise err
defer.returnValue((self.downloader, self.finished_deferred))
@@ -6,7 +6,7 @@ import treq
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from lbrynet.core.Error import InvalidExchangeRateResponse, CurrencyConversionError
from lbrynet.p2p.Error import InvalidExchangeRateResponse, CurrencyConversionError
log = logging.getLogger(__name__)
@@ -234,7 +234,7 @@ class ExchangeRateManager:
return self.convert_currency(
market.rate.currency_pair[1], to_currency, amount * market.rate.spot)
raise CurrencyConversionError(
'Unable to convert {} from {} to {}'.format(amount, from_currency, to_currency))
f'Unable to convert {amount} from {from_currency} to {to_currency}')
def fee_dict(self):
return {market: market.rate.as_dict() for market in self.market_feeds}
@@ -2,8 +2,7 @@ import binascii
import logging
from twisted.internet import defer, task
from lbrynet.core import utils
from lbrynet import conf
from lbrynet import utils, conf
log = logging.getLogger(__name__)
@@ -4,7 +4,6 @@ import logging
from twisted.internet import defer
from lbrynet import conf
log = logging.getLogger(__name__)
@@ -19,13 +18,12 @@ class DHTPeerFinder(DummyPeerFinder):
"""This class finds peers which have announced to the DHT that they have certain blobs"""
#implements(IPeerFinder)
def __init__(self, dht_node, peer_manager):
def __init__(self, component_manager):
"""
dht_node - an instance of dht.Node class
peer_manager - an instance of PeerManager class
component_manager - an instance of ComponentManager
"""
self.dht_node = dht_node
self.peer_manager = peer_manager
self.component_manager = component_manager
self.peer_manager = component_manager.peer_manager
self.peers = {}
self._ongoing_searchs = {}
@@ -40,27 +38,34 @@ class DHTPeerFinder(DummyPeerFinder):
Returns:
list of peers for the blob
"""
self.peers.setdefault(blob_hash, {(self.dht_node.externalIP, self.dht_node.peerPort,)})
if "dht" in self.component_manager.skip_components:
return defer.succeed([])
if not self.component_manager.all_components_running("dht"):
return defer.succeed([])
dht_node = self.component_manager.get_component("dht")
self.peers.setdefault(blob_hash, {(dht_node.externalIP, dht_node.peerPort,)})
if not blob_hash in self._ongoing_searchs or self._ongoing_searchs[blob_hash].called:
self._ongoing_searchs[blob_hash] = self._execute_peer_search(blob_hash, timeout)
peers = set(self._filter_self(blob_hash) if filter_self else self.peers[blob_hash])
self._ongoing_searchs[blob_hash] = self._execute_peer_search(dht_node, blob_hash, timeout)
def _filter_self(blob_hash):
my_host, my_port = dht_node.externalIP, dht_node.peerPort
return {(host, port) for host, port in self.peers[blob_hash] if (host, port) != (my_host, my_port)}
peers = set(_filter_self(blob_hash) if filter_self else self.peers[blob_hash])
return defer.succeed([self.peer_manager.get_peer(*peer) for peer in peers])
@defer.inlineCallbacks
def _execute_peer_search(self, blob_hash, timeout):
def _execute_peer_search(self, dht_node, blob_hash, timeout):
bin_hash = binascii.unhexlify(blob_hash)
finished_deferred = self.dht_node.iterativeFindValue(bin_hash, exclude=self.peers[blob_hash])
finished_deferred = dht_node.iterativeFindValue(bin_hash, exclude=self.peers[blob_hash])
timeout = timeout or conf.settings['peer_search_timeout']
if timeout:
finished_deferred.addTimeout(timeout, self.dht_node.clock)
finished_deferred.addTimeout(timeout, dht_node.clock)
try:
peer_list = yield finished_deferred
self.peers[blob_hash].update(set((host, port) for _, host, port in peer_list))
self.peers[blob_hash].update({(host, port) for _, host, port in peer_list})
except defer.TimeoutError:
log.debug("DHT timed out while looking peers for blob %s after %s seconds", blob_hash, timeout)
finally:
del self._ongoing_searchs[blob_hash]
def _filter_self(self, blob_hash):
my_host, my_port = self.dht_node.externalIP, self.dht_node.peerPort
return set((host, port) for host, port in self.peers[blob_hash] if (host, port) != (my_host, my_port))
@@ -1,4 +1,4 @@
from lbrynet.core.Peer import Peer
from lbrynet.p2p.Peer import Peer
class PeerManager:
@@ -1,9 +1,9 @@
import asyncio
import logging
import mimetypes
import os
from lbrynet.file_manager.EncryptedFileCreator import create_lbry_file
from lbrynet.blob.EncryptedFileCreator import create_lbry_file
from lbrynet.extras.daemon.mime_types import guess_mime_type
log = logging.getLogger(__name__)
@@ -28,9 +28,9 @@ class Publisher:
"""Create lbry file and make claim"""
log.info('Starting publish for %s', name)
if not os.path.isfile(file_path):
raise Exception("File {} not found".format(file_path))
raise Exception(f"File {file_path} not found")
if os.path.getsize(file_path) == 0:
raise Exception("Cannot publish empty file {}".format(file_path))
raise Exception(f"Cannot publish empty file {file_path}")
file_name = os.path.basename(file_path)
with open(file_path, 'rb') as read_handle:
@@ -43,7 +43,7 @@ class Publisher:
claim_dict['stream']['source'] = {}
claim_dict['stream']['source']['source'] = self.lbry_file.sd_hash
claim_dict['stream']['source']['sourceType'] = 'lbry_sd_hash'
claim_dict['stream']['source']['contentType'] = get_content_type(file_path)
claim_dict['stream']['source']['contentType'] = guess_mime_type(file_path)
claim_dict['stream']['source']['version'] = "_0_0_1" # need current version here
tx = await self.wallet.claim_name(
self.account, name, bid, claim_dict, self.certificate, holding_address
@@ -75,7 +75,3 @@ class Publisher:
))
self.lbry_file = [f for f in self.lbry_file_manager.lbry_files if f.stream_hash == stream_hash][0]
return tx
def get_content_type(filename):
return mimetypes.guess_type(filename)[0] or 'application/octet-stream'
+1
View File
@@ -0,0 +1 @@
from lbrynet.extras.daemon import Components # register Component classes
@@ -4,8 +4,8 @@ import logging
import treq
from twisted.internet import defer, task
from lbrynet import conf
from lbrynet.core import looping_call_manager, utils, system_info
from lbrynet import conf, utils
from lbrynet.extras import looping_call_manager, system_info
# Things We Track
SERVER_STARTUP = 'Server Startup'
@@ -136,8 +136,8 @@ class Manager:
def _get_looping_calls(self):
return [
('send_heartbeat', self._send_heartbeat, 60),
('update_tracked_metrics', self._update_tracked_metrics, 300),
('send_heartbeat', self._send_heartbeat, 300),
('update_tracked_metrics', self._update_tracked_metrics, 600),
]
def _setup_looping_calls(self):
@@ -3,7 +3,7 @@ from zope.interface import implementer
from twisted.cred import portal, checkers, credentials, error as cred_error
from twisted.internet import defer
from twisted.web import resource
from .keyring import Keyring
from lbrynet.extras.daemon.auth.keyring import Keyring
log = logging.getLogger(__name__)
@@ -4,7 +4,7 @@ import logging
from urllib.parse import urlparse
from lbrynet import conf
from lbrynet.daemon.auth.keyring import Keyring, APIKey
from lbrynet.extras.daemon.auth.keyring import Keyring, APIKey
log = logging.getLogger(__name__)
USER_AGENT = "AuthServiceProxy/0.1"
@@ -125,9 +125,6 @@ class AuthAPIClient:
class LBRYAPIClient:
@staticmethod
def get_client(conf_path=None):
conf.conf_file = conf_path
if not conf.settings:
conf.initialize_settings()
def get_client():
return AuthAPIClient.get_client() if conf.settings['use_auth_http'] else \
UnAuthAPIClient.from_url(conf.settings.get_api_connection_string())
@@ -4,8 +4,8 @@ from twisted.web import server, guard, resource
from twisted.cred import portal
from lbrynet import conf
from .auth import PasswordChecker, HttpPasswordRealm
from ..auth.keyring import Keyring
from lbrynet.extras.daemon.auth.auth import PasswordChecker, HttpPasswordRealm
from lbrynet.extras.daemon.auth.keyring import Keyring
log = logging.getLogger(__name__)
@@ -42,7 +42,7 @@ class AuthJSONRPCResource(resource.Resource):
realm = HttpPasswordRealm(self)
portal_to_realm = portal.Portal(realm, [checker, ])
root = guard.HTTPAuthSessionWrapper(
portal_to_realm, [guard.BasicCredentialFactory('Login to lbrynet api'), ]
portal_to_realm, [guard.BasicCredentialFactory(b'Login to lbrynet api'), ]
)
else:
log.info("Using non-authenticated API")
@@ -3,6 +3,7 @@ import datetime
import hmac
import hashlib
import base58
import logging
from OpenSSL.crypto import FILETYPE_PEM
from ssl import create_default_context, SSLContext
from cryptography.hazmat.backends import default_backend
@@ -12,7 +13,7 @@ from cryptography.x509.name import NameOID, NameAttribute
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
from twisted.internet import ssl
import logging
from lbrynet import conf
log = logging.getLogger(__name__)
@@ -72,8 +73,8 @@ class Keyring:
@classmethod
def load_from_disk(cls):
api_key_path = os.path.join(conf.settings['data_dir'], 'auth_token')
api_ssl_cert_path = os.path.join(conf.settings['data_dir'], 'api_ssl_cert.pem')
api_key_path = os.path.join(conf.settings.data_dir, 'auth_token')
api_ssl_cert_path = os.path.join(conf.settings.data_dir, 'api_ssl_cert.pem')
if not os.path.isfile(api_key_path) or not os.path.isfile(api_ssl_cert_path):
return
with open(api_key_path, 'rb') as f:
@@ -121,10 +122,10 @@ class Keyring:
auth_token = APIKey.create(seed=None, name="api")
with open(os.path.join(conf.settings['data_dir'], 'auth_token'), 'wb') as f:
with open(os.path.join(conf.settings.data_dir, 'auth_token'), 'wb') as f:
f.write(auth_token.secret.encode())
with open(os.path.join(conf.settings['data_dir'], 'api_ssl_cert.pem'), 'wb') as f:
with open(os.path.join(conf.settings.data_dir, 'api_ssl_cert.pem'), 'wb') as f:
f.write(public_certificate.encode())
return cls(auth_token, public_certificate, private_certificate)
@@ -1,10 +1,9 @@
import asyncio
import logging
from six.moves.urllib import parse as urlparse
from urllib import parse as urlparse
import json
import inspect
import signal
from functools import wraps
from twisted.web import server
from twisted.internet import defer
@@ -14,16 +13,17 @@ from twisted.internet.error import ConnectionDone, ConnectionLost
from txjsonrpc import jsonrpclib
from traceback import format_exc
from lbrynet import conf, analytics
from lbrynet.core.Error import InvalidAuthenticationToken
from lbrynet.core import utils
from lbrynet.core.Error import ComponentsNotStarted, ComponentStartConditionNotMet
from lbrynet.core.looping_call_manager import LoopingCallManager
from lbrynet.daemon.ComponentManager import ComponentManager
from .keyring import APIKey, Keyring
from .undecorated import undecorated
from .factory import AuthJSONRPCResource
from lbrynet.daemon.json_response_encoder import JSONResponseEncoder
from lbrynet import conf, utils
from lbrynet.extras.daemon import analytics
from lbrynet.p2p.Error import InvalidAuthenticationToken
from lbrynet.p2p.Error import ComponentsNotStarted, ComponentStartConditionNotMet
from lbrynet.extras.looping_call_manager import LoopingCallManager
from lbrynet.extras.daemon.ComponentManager import ComponentManager
from lbrynet.extras.daemon.auth.keyring import APIKey, Keyring
from lbrynet.extras.daemon.auth.undecorated import undecorated
from lbrynet.extras.daemon.auth.factory import AuthJSONRPCResource
from lbrynet.extras.daemon.json_response_encoder import JSONResponseEncoder
log = logging.getLogger(__name__)
EMPTY_PARAMS = [{}]
@@ -206,7 +206,6 @@ class AuthJSONRPCServer(AuthorizedBase):
self.announced_startup = False
self.sessions = {}
self.server = None
self.keyring = Keyring.generate_and_save()
@defer.inlineCallbacks
def start_listening(self):
@@ -284,7 +283,10 @@ class AuthJSONRPCServer(AuthorizedBase):
return d
def get_server_factory(self):
return AuthJSONRPCResource(self).getServerFactory(self.keyring, self._use_authentication, self._use_https)
keyring = None
if self._use_authentication or self._use_https:
keyring = Keyring.generate_and_save()
return AuthJSONRPCResource(self).getServerFactory(keyring, self._use_authentication, self._use_https)
def _set_headers(self, request, data, update_secret=False):
if conf.settings['allowed_origin']:
File renamed without changes.
@@ -1,10 +1,15 @@
import logging
from decimal import Decimal
from binascii import hexlify
from datetime import datetime
from json import JSONEncoder
from lbrynet.wallet.transaction import Transaction, Output
from lbrynet.wallet.dewies import dewies_to_lbc
from lbrynet.wallet.ledger import MainNetLedger
from ecdsa import BadSignatureError
from lbrynet.extras.wallet import MainNetLedger
from lbrynet.extras.wallet.transaction import Transaction, Output
from lbrynet.extras.wallet.dewies import dewies_to_lbc
log = logging.getLogger(__name__)
class JSONResponseEncoder(JSONEncoder):
@@ -47,7 +52,7 @@ class JSONResponseEncoder(JSONEncoder):
'amount': dewies_to_lbc(txo.amount),
'address': txo.get_address(self.ledger),
'height': tx_height,
'confirmations': best_height - tx_height if tx_height > 0 else tx_height
'confirmations': (best_height+1) - tx_height if tx_height > 0 else tx_height
}
if txo.is_change is not None:
output['is_change'] = txo.is_change
@@ -68,9 +73,18 @@ class JSONResponseEncoder(JSONEncoder):
output['valid_signature'] = None
if txo.channel is not None:
output['channel_name'] = txo.channel.claim_name
output['valid_signature'] = claim.validate_signature(
txo.get_address(self.ledger), txo.channel.claim
)
try:
output['valid_signature'] = claim.validate_signature(
txo.get_address(self.ledger), txo.channel.claim
)
except BadSignatureError:
output['valid_signature'] = False
except ValueError:
log.exception(
'txo.id: %s, txo.channel.id:%s, output: %s',
txo.id, txo.channel.id, output
)
output['valid_signature'] = False
if txo.script.is_claim_name:
output['type'] = 'claim'
File renamed without changes.
@@ -7,21 +7,23 @@ def migrate_db(db_dir, start, end):
current = start
while current < end:
if current == 1:
from lbrynet.database.migrator.migrate1to2 import do_migration
from .migrate1to2 import do_migration
elif current == 2:
from lbrynet.database.migrator.migrate2to3 import do_migration
from .migrate2to3 import do_migration
elif current == 3:
from lbrynet.database.migrator.migrate3to4 import do_migration
from .migrate3to4 import do_migration
elif current == 4:
from lbrynet.database.migrator.migrate4to5 import do_migration
from .migrate4to5 import do_migration
elif current == 5:
from lbrynet.database.migrator.migrate5to6 import do_migration
from .migrate5to6 import do_migration
elif current == 6:
from lbrynet.database.migrator.migrate6to7 import do_migration
from .migrate6to7 import do_migration
elif current == 7:
from lbrynet.database.migrator.migrate7to8 import do_migration
from .migrate7to8 import do_migration
elif current == 8:
from lbrynet.database.migrator.migrate8to9 import do_migration
from .migrate8to9 import do_migration
elif current == 9:
from .migrate9to10 import do_migration
else:
raise Exception("DB migration of version {} to {} is not available".format(current,
current+1))
@@ -2,13 +2,11 @@ import sqlite3
import os
import json
import logging
from lbryschema.decode import smart_decode
from lbrynet import conf
from lbrynet.schema.decode import smart_decode
log = logging.getLogger(__name__)
download_directory = conf.settings['download_directory']
CREATE_TABLES_QUERY = """
pragma foreign_keys=on;
pragma journal_mode=WAL;
@@ -190,7 +188,7 @@ def do_migration(db_dir):
# insert the file
transaction.execute(
"insert or ignore into file values (?, ?, ?, ?, ?)",
(stream_hash, stream_name, download_directory.encode('hex'),
(stream_hash, stream_name, conf.settings.download_dir.encode('hex'),
data_rate, status)
)
@@ -2,9 +2,9 @@ import sqlite3
import logging
import os
from lbrynet.core.Error import InvalidStreamDescriptorError
from lbrynet.core.StreamDescriptor import EncryptedFileStreamType, format_sd_info, format_blobs, validate_descriptor
from lbrynet.cryptstream.CryptBlob import CryptBlobInfo
from lbrynet.p2p.Error import InvalidStreamDescriptorError
from lbrynet.p2p.StreamDescriptor import EncryptedFileStreamType, format_sd_info, format_blobs, validate_descriptor
from lbrynet.blob.CryptBlob import CryptBlobInfo
log = logging.getLogger(__name__)
@@ -0,0 +1,11 @@
import sqlite3
import os
def do_migration(db_dir):
db_path = os.path.join(db_dir, "lbrynet.sqlite")
connection = sqlite3.connect(db_path)
cursor = connection.cursor()
cursor.executescript("ALTER TABLE claim ADD resolved_at int DEFAULT 0 NOT NULL;")
connection.commit()
connection.close()
+159
View File
@@ -0,0 +1,159 @@
import os
types_map = {
# http://www.iana.org/assignments/media-types
'.a': 'application/octet-stream',
'.ai': 'application/postscript',
'.aif': 'audio/x-aiff',
'.aifc': 'audio/x-aiff',
'.aiff': 'audio/x-aiff',
'.au': 'audio/basic',
'.avi': 'video/x-msvideo',
'.bat': 'text/plain',
'.bcpio': 'application/x-bcpio',
'.bin': 'application/octet-stream',
'.bmp': 'image/bmp',
'.c': 'text/plain',
'.cdf': 'application/x-netcdf',
'.cpio': 'application/x-cpio',
'.csh': 'application/x-csh',
'.css': 'text/css',
'.csv': 'text/csv',
'.dll': 'application/octet-stream',
'.doc': 'application/msword',
'.dot': 'application/msword',
'.dvi': 'application/x-dvi',
'.eml': 'message/rfc822',
'.eps': 'application/postscript',
'.epub': 'application/epub+zip',
'.etx': 'text/x-setext',
'.exe': 'application/octet-stream',
'.gif': 'image/gif',
'.gtar': 'application/x-gtar',
'.h': 'text/plain',
'.hdf': 'application/x-hdf',
'.htm': 'text/html',
'.html': 'text/html',
'.ico': 'image/vnd.microsoft.icon',
'.ief': 'image/ief',
'.iges': 'model/iges',
'.jpe': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.jpg': 'image/jpeg',
'.js': 'application/javascript',
'.json': 'application/json',
'.ksh': 'text/plain',
'.latex': 'application/x-latex',
'.m1v': 'video/mpeg',
'.m3u': 'application/vnd.apple.mpegurl',
'.m3u8': 'application/vnd.apple.mpegurl',
'.man': 'application/x-troff-man',
'.markdown': 'text/markdown',
'.md': 'text/markdown',
'.me': 'application/x-troff-me',
'.mht': 'message/rfc822',
'.mhtml': 'message/rfc822',
'.mif': 'application/x-mif',
'.mov': 'video/quicktime',
'.movie': 'video/x-sgi-movie',
'.mp2': 'audio/mpeg',
'.mp3': 'audio/mpeg',
'.mp4': 'video/mp4',
'.mpa': 'video/mpeg',
'.mpe': 'video/mpeg',
'.mpeg': 'video/mpeg',
'.mpg': 'video/mpeg',
'.ms': 'application/x-troff-ms',
'.nc': 'application/x-netcdf',
'.nws': 'message/rfc822',
'.o': 'application/octet-stream',
'.obj': 'application/octet-stream',
'.oda': 'application/oda',
'.p12': 'application/x-pkcs12',
'.p7c': 'application/pkcs7-mime',
'.pbm': 'image/x-portable-bitmap',
'.pdf': 'application/pdf',
'.pfx': 'application/x-pkcs12',
'.pgm': 'image/x-portable-graymap',
'.pl': 'text/plain',
'.png': 'image/png',
'.pnm': 'image/x-portable-anymap',
'.pot': 'application/vnd.ms-powerpoint',
'.ppa': 'application/vnd.ms-powerpoint',
'.ppm': 'image/x-portable-pixmap',
'.pps': 'application/vnd.ms-powerpoint',
'.ppt': 'application/vnd.ms-powerpoint',
'.ps': 'application/postscript',
'.pwz': 'application/vnd.ms-powerpoint',
'.py': 'text/x-python',
'.pyc': 'application/x-python-code',
'.pyo': 'application/x-python-code',
'.qt': 'video/quicktime',
'.ra': 'audio/x-pn-realaudio',
'.ram': 'application/x-pn-realaudio',
'.ras': 'image/x-cmu-raster',
'.rdf': 'application/xml',
'.rgb': 'image/x-rgb',
'.roff': 'application/x-troff',
'.rtx': 'text/richtext',
'.sgm': 'text/x-sgml',
'.sgml': 'text/x-sgml',
'.sh': 'application/x-sh',
'.shar': 'application/x-shar',
'.snd': 'audio/basic',
'.so': 'application/octet-stream',
'.src': 'application/x-wais-source',
'.stl': 'model/stl',
'.sv4cpio': 'application/x-sv4cpio',
'.sv4crc': 'application/x-sv4crc',
'.svg': 'image/svg+xml',
'.swf': 'application/x-shockwave-flash',
'.t': 'application/x-troff',
'.tar': 'application/x-tar',
'.tcl': 'application/x-tcl',
'.tex': 'application/x-tex',
'.texi': 'application/x-texinfo',
'.texinfo': 'application/x-texinfo',
'.tif': 'image/tiff',
'.tiff': 'image/tiff',
'.tr': 'application/x-troff',
'.tsv': 'text/tab-separated-values',
'.txt': 'text/plain',
'.ustar': 'application/x-ustar',
'.vcf': 'text/x-vcard',
'.wav': 'audio/x-wav',
'.webm': 'video/webm',
'.wiz': 'application/msword',
'.wsdl': 'application/xml',
'.xbm': 'image/x-xbitmap',
'.xlb': 'application/vnd.ms-excel',
'.xls': 'application/vnd.ms-excel',
'.xml': 'text/xml',
'.xpdl': 'application/xml',
'.xpm': 'image/x-xpixmap',
'.xsl': 'application/xml',
'.xwd': 'image/x-xwindowdump',
'.zip': 'application/zip',
# These are non-standard types, commonly found in the wild.
'.mid': 'audio/midi',
'.midi': 'audio/midi',
'.pct': 'image/pict',
'.pic': 'image/pict',
'.pict': 'image/pict',
'.rtf': 'application/rtf',
'.xul': 'text/xul',
# microsoft is special and has its own 'standard'
# https://docs.microsoft.com/en-us/windows/desktop/wmp/file-name-extensions
'.wmv': 'video/x-ms-wmv'
}
fallback_mime_type = 'application/octet-stream'
def guess_mime_type(path):
_, ext = os.path.splitext(path)
return types_map.get(ext, fallback_mime_type)
@@ -2,21 +2,27 @@ import logging
import os
import sqlite3
import traceback
import typing
from binascii import hexlify, unhexlify
from decimal import Decimal
from twisted.internet import defer, task, threads
from twisted.enterprise import adbapi
from lbryschema.claim import ClaimDict
from lbryschema.decode import smart_decode
from lbrynet.extras.wallet.dewies import dewies_to_lbc, lbc_to_dewies
from lbrynet import conf
from lbrynet.cryptstream.CryptBlob import CryptBlobInfo
from lbrynet.schema.claim import ClaimDict
from lbrynet.schema.decode import smart_decode
from lbrynet.blob.CryptBlob import CryptBlobInfo
from lbrynet.dht.constants import dataExpireTimeout
from torba.constants import COIN
from lbrynet.schema.uri import parse_lbry_uri
log = logging.getLogger(__name__)
def calculate_effective_amount(amount: str, supports: typing.Optional[typing.List[typing.Dict]] = None) -> str:
return dewies_to_lbc(
lbc_to_dewies(amount) + sum([lbc_to_dewies(support['amount']) for support in supports])
)
def _get_next_available_file_name(download_directory, file_name):
base_name, ext = os.path.splitext(file_name)
i = 0
@@ -381,8 +387,8 @@ class SQLiteStorage:
def _store_stream(transaction):
transaction.execute("insert into stream values (?, ?, ?, ?, ?);",
(stream_hash, sd_hash, stream_key, stream_name,
suggested_file_name))
(stream_hash, sd_hash, stream_key, stream_name,
suggested_file_name))
for blob_info in stream_blob_infos:
transaction.execute("insert into stream_blob values (?, ?, ?, ?)",
@@ -418,7 +424,7 @@ class SQLiteStorage:
def check_if_stream_exists(self, stream_hash):
d = self.db.runQuery("select stream_hash from stream where stream_hash=?", (stream_hash, ))
d.addCallback(lambda r: True if len(r) else False)
d.addCallback(lambda r: bool(len(r)))
return d
def get_blob_num_by_hash(self, stream_hash, blob_hash):
@@ -557,32 +563,27 @@ class SQLiteStorage:
for support in supports:
transaction.execute(
"insert into support values (?, ?, ?, ?)",
("%s:%i" % (support['txid'], support['nout']), claim_id, int(support['amount'] * COIN),
("%s:%i" % (support['txid'], support['nout']), claim_id, lbc_to_dewies(support['amount']),
support.get('address', ""))
)
return self.db.runInteraction(_save_support)
def get_supports(self, *claim_ids):
def _format_support(outpoint, supported_id, amount, address):
def _format_support(outpoint, amount):
return {
"txid": outpoint.split(":")[0],
"nout": int(outpoint.split(":")[1]),
"claim_id": supported_id,
"amount": float(Decimal(amount) / Decimal(COIN)),
"address": address,
"amount": dewies_to_lbc(amount),
}
def _get_supports(transaction):
if len(claim_ids) == 1:
bind = "=?"
else:
bind = "in ({})".format(','.join('?' for _ in range(len(claim_ids))))
return [
_format_support(*support_info)
for support_info in transaction.execute(
"select * from support where claim_id {}".format(bind),
for support_info in _batched_select(
transaction,
"select support_outpoint, amount from support where claim_id in {}",
tuple(claim_ids)
).fetchall()
)
]
return self.db.runInteraction(_get_supports)
@@ -598,12 +599,13 @@ class SQLiteStorage:
outpoint = "%s:%i" % (claim_info['txid'], claim_info['nout'])
claim_id = claim_info['claim_id']
name = claim_info['name']
amount = int(COIN * claim_info['amount'])
amount = lbc_to_dewies(claim_info['amount'])
height = claim_info['height']
address = claim_info['address']
sequence = claim_info['claim_sequence']
resolved_at = claim_info['height'] + claim_info['depth']
try:
certificate_id = claim_info['value'].get('content_claims_to_update', {}).get('certificateId')
certificate_id = claim_info['value'].get('publisherSignature', {}).get('certificateId')
except AttributeError:
certificate_id = None
try:
@@ -615,8 +617,9 @@ class SQLiteStorage:
source_hash = None
serialized = claim_info.get('hex') or hexlify(smart_decode(claim_info['value']).serialized)
transaction.execute(
"insert or replace into claim values (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(outpoint, claim_id, name, amount, height, serialized, certificate_id, address, sequence)
"insert or replace into claim values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(outpoint, claim_id, name, amount, height, serialized, certificate_id, address, sequence,
resolved_at)
)
if 'supports' in claim_info: # if this response doesn't have support info don't overwrite the existing
# support info
@@ -709,7 +712,9 @@ class SQLiteStorage:
"select claim_id from claim where claim_outpoint=?", current_associated_content
).fetchone()[0]
if current_associated_claim_id != new_claim_id:
raise Exception("invalid stream update")
raise Exception(
f"mismatching claim ids when updating stream {current_associated_claim_id} vs {new_claim_id}"
)
# update the claim associated to the file
transaction.execute("insert or replace into content_claim values (?, ?)", (stream_hash, claim_outpoint))
@@ -745,37 +750,34 @@ class SQLiteStorage:
if result and include_supports:
supports = yield self.get_supports(result['claim_id'])
result['supports'] = supports
result['effective_amount'] = float(
sum([support['amount'] for support in supports]) + result['amount']
)
result['effective_amount'] = calculate_effective_amount(result['amount'], supports)
defer.returnValue(result)
@defer.inlineCallbacks
def get_claims_from_stream_hashes(self, stream_hashes, include_supports=True):
def _batch_get_claim(transaction):
results = {}
bind = "({})".format(','.join('?' for _ in range(len(stream_hashes))))
claim_infos = transaction.execute(
claim_infos = _batched_select(
transaction,
"select content_claim.stream_hash, c.* from content_claim "
"inner join claim c on c.claim_outpoint=content_claim.claim_outpoint "
"and content_claim.stream_hash in {} order by c.rowid desc".format(bind),
tuple(stream_hashes)
).fetchall()
"and content_claim.stream_hash in {} order by c.rowid desc",
stream_hashes)
channel_id_infos = {}
for claim_info in claim_infos:
if claim_info[7]:
streams = channel_id_infos.get(claim_info[7], [])
streams.append(claim_info[0])
channel_id_infos[claim_info[7]] = streams
for claim_info in claim_infos:
stream_hash = claim_info[0]
result = _format_claim_response(*claim_info[1:])
results[stream_hash] = result
bind = "({})".format(','.join('?' for _ in range(len(channel_id_infos))))
for claim_id, channel_name in transaction.execute(
"select claim_id, claim_name from claim where claim_id in {}".format(bind),
tuple(channel_id_infos.keys())
).fetchall():
channel_names = _batched_select(
transaction,
"select claim_id, claim_name from claim where claim_id in {}",
tuple(channel_id_infos.keys())
)
for claim_id, channel_name in channel_names:
for stream_hash in channel_id_infos[claim_id]:
results[stream_hash]['channel_name'] = channel_name
return results
@@ -789,12 +791,28 @@ class SQLiteStorage:
claim = claims[stream_hash]
supports = all_supports.get(claim['claim_id'], [])
claim['supports'] = supports
claim['effective_amount'] = float(
sum([support['amount'] for support in supports]) + claim['amount']
)
claim['effective_amount'] = calculate_effective_amount(claim['amount'], supports)
claims[stream_hash] = claim
defer.returnValue(claims)
def get_resolved_at_for_outpoint(self, outpoint):
return self.run_and_return_one_or_none(
"select resolved_at from claim "
"where claim_outpoint = ?",
outpoint
)
def get_outpoint_for_uri(self, uri):
uri = parse_lbry_uri(uri)
claim_id_param = "%" if not uri.claim_id else uri.claim_id
return self.run_and_return_one_or_none(
"select claim_outpoint from claim "
"where claim_name=? and claim_id like ?"
"order by height desc, amount desc "
"limit 1",
uri.name, claim_id_param
)
@defer.inlineCallbacks
def get_claim(self, claim_outpoint, include_supports=True):
def _get_claim(transaction):
@@ -813,9 +831,7 @@ class SQLiteStorage:
if include_supports:
supports = yield self.get_supports(result['claim_id'])
result['supports'] = supports
result['effective_amount'] = float(
sum([support['amount'] for support in supports]) + result['amount']
)
result['effective_amount'] = calculate_effective_amount(result['amount'], supports)
defer.returnValue(result)
def get_unknown_certificate_ids(self):
@@ -875,18 +891,26 @@ class SQLiteStorage:
# Helper functions
def _format_claim_response(outpoint, claim_id, name, amount, height, serialized, channel_id, address, claim_sequence):
def _format_claim_response(outpoint, claim_id, name, amount, height, serialized, _channel_id, address, claim_sequence,
_resolved_at):
r = {
"name": name,
"claim_id": claim_id,
"address": address,
"claim_sequence": claim_sequence,
"hex": serialized,
"value": ClaimDict.deserialize(unhexlify(serialized)).claim_dict,
"height": height,
"amount": float(Decimal(amount) / Decimal(COIN)),
"amount": dewies_to_lbc(amount),
"nout": int(outpoint.split(":")[1]),
"txid": outpoint.split(":")[0],
"channel_claim_id": channel_id,
"channel_name": None
}
return r
def _batched_select(transaction, query, parameters):
for start_index in range(0, len(parameters), 900):
current_batch = parameters[start_index:start_index+900]
bind = "({})".format(','.join(['?'] * len(current_batch)))
for result in transaction.execute(query.format(bind), current_batch):
yield result
@@ -3,7 +3,7 @@ class LoopingCallManager:
self.calls = calls or {}
def register_looping_call(self, name, call):
assert name not in self.calls, '{} is already registered'.format(name)
assert name not in self.calls, f'{name} is already registered'
self.calls[name] = call
def start(self, name, *args):
@@ -63,7 +63,3 @@ If the transfer was not successful (False), the blob is re-added to the needed_b
Blob requests continue for each of the blobs the client has queued to send, when completed
the client disconnects.
"""
from lbrynet.reflector.server.server import ReflectorServerFactory as ServerFactory
from lbrynet.reflector.client.client import EncryptedFileReflectorClientFactory as ClientFactory
from lbrynet.reflector.client.blob import BlobReflectorClientFactory as BlobClientFactory
File renamed without changes.
@@ -5,7 +5,7 @@ from twisted.protocols.basic import FileSender
from twisted.internet.protocol import Protocol, ClientFactory
from twisted.internet import defer, error
from lbrynet.reflector.common import IncompleteResponse, REFLECTOR_V2
from lbrynet.extras.reflector.common import IncompleteResponse, REFLECTOR_V2
log = logging.getLogger(__name__)
@@ -111,7 +111,7 @@ class BlobReflectorClient(Protocol):
raise ValueError("Need protocol version number!")
server_version = int(response_dict['version'])
if self.protocol_version != server_version:
raise ValueError("I can't handle protocol version {}!".format(self.protocol_version))
raise ValueError(f"I can't handle protocol version {self.protocol_version}!")
self.received_handshake_response = True
return defer.succeed(True)
@@ -141,7 +141,7 @@ class BlobReflectorClient(Protocol):
self.read_handle = read_handle
return None
raise ValueError(
"Couldn't open that blob for some reason. blob_hash: {}".format(blob.blob_hash))
f"Couldn't open that blob for some reason. blob_hash: {blob.blob_hash}")
def send_blob_info(self):
log.debug("Send blob info for %s", self.next_blob_to_send.blob_hash)
Loaded 100 of 267 files, more files were not shown because too many files have changed in this diff. Show more