Compare commits

...
1701 Commits
Author SHA1 Message Date
Jack Robison d3ce3ecf2c Bump version: 0.9.2rc14 → 0.9.2rc15 2017-04-10 14:30:38 -04:00
Jack Robison a87834bacc skip test_parallel_connections 2017-04-10 14:21:46 -04:00
Jack Robison 105fa0b947 pylint 2017-04-10 13:59:31 -04:00
Alex Grintsvayg 8db7c37fa7 exploring dht 2017-04-10 13:45:43 -04:00
Alex Grintsvayg 54c29d4a8d formatting 2017-04-10 13:45:43 -04:00
Alex Grintsvayg a2eb0cad33 delete live lbrylive livestreaming code 2017-04-10 13:45:43 -04:00
Jack Robison 4c122f4ab0 Bump version: 0.9.2rc13 → 0.9.2rc14 2017-04-10 11:04:28 -04:00
Jack Robison a43127fffc add channel_new logging, fix docstring 2017-04-10 11:01:28 -04:00
Jack Robison 66ab1386c5 Bump version: 0.9.2rc12 → 0.9.2rc13 2017-04-10 09:57:55 -04:00
Jack Robison 22381f9427 update lbryum requirement 2017-04-10 09:57:49 -04:00
Jack Robison 5efbafec39 Bump version: 0.9.2rc11 → 0.9.2rc12 2017-04-09 21:51:42 -04:00
Jack Robison 0ff308e3eb Merge pull request #602 from lbryio/fix-file-with-abandoned-claim
fix file_list for files with abandoned claims
2017-04-09 21:51:02 -04:00
Jack Robison 9531bdf396 fix file_list for files with abandoned claims 2017-04-09 21:44:58 -04:00
Jack Robison 978139a631 Bump version: 0.9.2rc10 → 0.9.2rc11 2017-04-09 17:53:55 -04:00
Jack Robison 86117c431b Merge pull request #601 from lbryio/channels-fixes
Channels fixes
2017-04-09 17:53:16 -04:00
Jack Robison 585a17c830 update changelog 2017-04-09 17:49:31 -04:00
Jack Robison 04c119543f fix resolution and file_list bugs 2017-04-09 17:49:31 -04:00
Jack Robison 33c4856f00 update required lbryschema and lbryum versions 2017-04-09 17:49:10 -04:00
Jack Robison ead32f32d8 Bump version: 0.9.2rc9 → 0.9.2rc10 2017-04-08 23:58:02 -04:00
Alex Grintsvayg e54dc9a01d Bump version: 0.9.2rc8 → 0.9.2rc9 2017-04-08 19:24:23 -04:00
Jack Robison c0ccfe2046 Merge pull request #594 from lbryio/use-lbryio-price-api
use lbryio api for price data instead of bittrex
2017-04-07 20:13:55 -04:00
Jack Robison 40374ec202 use lbryio api for price data instead of bittrex 2017-04-07 19:33:13 -04:00
Jack Robison 6ed2c5831e install lbryschema before lbryum 2017-04-07 16:30:36 -04:00
Jack Robison dc0699b929 Merge pull request #593 from lbryio/channels
Channels
2017-04-07 15:55:11 -04:00
Jack Robison eb1bdcc312 update changelog 2017-04-07 14:04:18 -04:00
Jack Robison e9cfbea75f update lbrynet api and tests 2017-04-07 14:04:13 -04:00
Jack Robison ccc94a0db9 update wallet for new lbryum functionality 2017-04-06 20:50:29 -04:00
Jack Robison bcf7a28fc8 update download, handle fee conversion error, use uri instead of name to get() 2017-04-06 20:50:28 -04:00
Jack Robison 1880f64da2 raise default download timeout 2017-04-06 20:50:28 -04:00
Jack Robison 45b3837638 use ClaimDict.source_hash in utils.get_sd_hash 2017-04-06 20:50:28 -04:00
Alex Grintsvayg ee7ca0978f Bump version: 0.9.2rc7 → 0.9.2rc8 2017-04-06 19:14:19 -04:00
Alex Grintsvayg fc3573f823 Bump version: 0.9.2rc6 → 0.9.2rc7 2017-04-06 18:55:07 -04:00
Alex Grintsvayg 8e15eddf9b Bump version: 0.9.2rc5 → 0.9.2rc6 2017-04-06 18:39:49 -04:00
Alex Grintsvayg 880b974b84 Bump version: 0.9.2rc4 → 0.9.2rc5 2017-04-06 18:16:46 -04:00
Alex Grintsvayg 5b78215063 Bump version: 0.9.2rc3 → 0.9.2rc4 2017-04-06 17:50:40 -04:00
Alex Grin bf7882d393 Merge pull request #583 from lbryio/keyerror_fix
fix KeyError bug in dht
2017-04-06 16:09:39 -04:00
Alex Grin 7a798616d6 Update INSTALL.md 2017-04-05 18:34:25 -04:00
Jack Robison 6f909f9654 Merge pull request #586 from lbryio/lbryschema_2
Lbryschema integration
2017-04-04 22:00:07 -04:00
Jack Robison 528ce42821 smart_decode metadata shown in file commands 2017-04-04 21:50:27 -04:00
Kay Kurokawa 931038d478 bug fixes for lbryschema integration 2017-04-04 20:30:51 -04:00
Kay Kurokawa 60d6666abf added changelog 2017-04-04 20:30:51 -04:00
Kay Kurokawa ca041b5dc4 lbryschema integration 2017-04-04 20:30:51 -04:00
Kay Kurokawa f43b38a5ea update INSTALL.md to include protobuf dependencies for lbryschema 2017-04-04 20:27:12 -04:00
Kay Kurokawa 769329c90e add lbryschema to requirements.txt and setup.py 2017-04-04 20:27:12 -04:00
Kay Kurokawa 23596b434a rename update_stream and prevent it from using old metadata 2017-04-04 20:27:12 -04:00
Alex Grintsvayg 22f57f6490 add test to reproduce keyerror 2017-04-04 15:10:46 -04:00
Alex Grintsvayg 99e4f9b00b we dont always want to encode key here. bucket.keyInRange() already does it when necessary 2017-04-04 15:01:59 -04:00
Alex Grintsvayg bee67117c8 changelog 2017-03-31 14:52:56 -04:00
Alex Grintsvayg 502e2227b5 fix KeyError bug in dht 2017-03-31 14:26:19 -04:00
Alex Grintsvayg d3173b157e oops 2017-03-31 14:26:09 -04:00
Alex Grintsvayg c0a40605ad formatting and minor fixes on scripts 2017-03-31 14:23:35 -04:00
Alex Grintsvayg c20ba29c7a move scripts into scripts dir 2017-03-31 14:23:09 -04:00
Alex Grintsvayg 850f51140e formatting 2017-03-31 13:32:43 -04:00
Alex Grin 5ec891c9ac Merge pull request #572 from lbryio/minor-wallet-updates
Minor wallet updates
2017-03-31 10:30:48 -04:00
Alex Grin 5e936ef3cc Merge pull request #582 from gaojiuli/patch-1
Update FAQ.md
2017-03-31 10:29:32 -04:00
JiuLi Gao d23e3cf066 Update FAQ.md 2017-03-31 11:13:27 +08:00
Alex Grintsvayg bacc3a0b3a log receive request 2017-03-29 16:56:34 -04:00
Alex Grin 32e2ee8bf5 Merge pull request #569 from lbryio/fix_update
Remove update_metadata() function
2017-03-29 13:10:15 -04:00
Alex Grintsvayg beeaec986a share_debug_info is settable 2017-03-29 11:20:38 -04:00
Alex Grintsvayg e32aa928ac Bump version: 0.9.2rc2 → 0.9.2rc3 2017-03-29 11:10:57 -04:00
Alex Grintsvayg 1a562d70aa config to share debug info with lbry devs 2017-03-29 11:06:07 -04:00
Jeremy Kauffman 8dbadbd75b Merge pull request #573 from lbryio/get-by-claim-id
add claim_id param to get()
2017-03-29 09:41:14 -04:00
Jack Robison 370a91fdaa add claim_id param to get(), remove stream_info param 2017-03-28 22:04:13 -04:00
Alex Grintsvayg 1916b990b7 typo 2017-03-28 17:14:13 -04:00
Alex Grintsvayg 84d9461deb Minor wallet updates 2017-03-28 17:13:59 -04:00
Alex Grintsvayg 9e5ac6f73e typo 2017-03-28 17:13:21 -04:00
Umpei Kay Kurokawa 5dbe2fdcec Merge pull request #571 from lbryio/clarify_pulish_docstring
clarify publish API command docstring
2017-03-28 14:25:10 -04:00
Kay Kurokawa eabf9f1c4a clarify publish API command docstring 2017-03-28 11:48:26 -04:00
Kay Kurokawa bb4bf620f4 remove update_metadata() function 2017-03-27 16:46:41 -04:00
Alex Grintsvayg 7abfbc4c5f Merge branch 'connection_manager_improve_manage'
* connection_manager_improve_manage:
  add changelog
  decrease connection timeout
  connect to multiple peers at once in manage
2017-03-27 15:11:23 -04:00
Kay Kurokawa b2d0c53bbe add changelog 2017-03-27 15:11:15 -04:00
Kay Kurokawa ddd8fd1a77 decrease connection timeout 2017-03-27 15:11:15 -04:00
Kay Kurokawa 3af5e9dc5c connect to multiple peers at once in manage 2017-03-27 15:11:15 -04:00
Alex Grin 23f1cf9c82 Merge pull request #568 from lbryio/get_unused_address
get_unused_address / get_new_address
2017-03-27 13:06:36 -04:00
Kay Kurokawa 26f40b62f9 run gen_api_docs.py script 2017-03-27 12:51:19 -04:00
Kay Kurokawa b15fba9c34 add changelog 2017-03-27 12:45:16 -04:00
Kay Kurokawa b0fd6e57df create and properly distinguish get_unused_address and get_new_address 2017-03-27 12:41:53 -04:00
Alex Grintsvayg b04455f9c7 Bump version: 0.9.2rc1 → 0.9.2rc2 2017-03-25 10:29:07 -04:00
Alex Grintsvayg b4f5f2068f Merge branch 'fix_file_seed'
* fix_file_seed:
  Ran gen_api_docs.py
  rename file_seed
2017-03-25 10:20:19 -04:00
Kay Kurokawa 948d606268 Ran gen_api_docs.py 2017-03-25 10:20:07 -04:00
Kay Kurokawa f5c533e2f1 rename file_seed 2017-03-25 10:20:07 -04:00
Alex Grintsvayg a0a122f0ef Merge branch 'txrequests'
* txrequests:
  switch to txrequests to marry async requests to twisted
2017-03-24 17:20:43 -04:00
Alex Grintsvayg a2a29fc02c switch to txrequests to marry async requests to twisted 2017-03-24 17:20:33 -04:00
Alex Grintsvayg c9c84784c9 restore http error codes, add some color 2017-03-24 17:19:55 -04:00
Alex Grintsvayg 8fa2524e54 properly handle true/false cli args 2017-03-23 17:44:12 -04:00
Alex Grin 41257c07e6 Merge pull request #561 from lbryio/remove_check_pending
Remove pending_claims logic from Daemon
2017-03-23 16:52:45 -04:00
Kay Kurokawa e7b90055ed remove pending_claims logic from Daemon 2017-03-23 16:44:18 -04:00
Alex Grintsvayg 3f08b5f711 pylint errors 2017-03-23 16:05:26 -04:00
Alex Grintsvayg f0e7f38519 changelog 2017-03-23 15:45:50 -04:00
Alex Grintsvayg 12b428f02d Merge branch 'jsonrpc-fix-errors'
* jsonrpc-fix-errors:
  add todo
  format response and error properly
  fixes, refactors
  changelog
  hack for lbrynet-cli for now
  fix error handling in jsonrpc
2017-03-23 15:44:43 -04:00
Alex Grintsvayg 7708c4a15c add todo 2017-03-23 15:44:23 -04:00
Alex Grintsvayg 9410cd9e77 format response and error properly 2017-03-23 15:37:28 -04:00
Alex Grintsvayg 0c42bc6382 fixes, refactors 2017-03-23 14:11:01 -04:00
Alex Grintsvayg b4364f661c changelog 2017-03-23 11:09:40 -04:00
Alex Grintsvayg 63c71760dd hack for lbrynet-cli for now 2017-03-23 11:09:40 -04:00
Alex Grintsvayg 25d9f008de fix error handling in jsonrpc 2017-03-23 11:09:40 -04:00
Alex Grintsvayg 41fbb1399c Merge branch 'mk_download_dir'
* mk_download_dir:
  make directories in one place when initializing
  create download directory if it doesn't exist
  Fix restart procedure in DaemonControl
  Revert "Bump version: 0.9.2rc1 → 0.9.3rc1"
  Bump version: 0.9.2rc1 → 0.9.3rc1
  Bump version: 0.9.1 → 0.9.2rc1
  update changelog
  handle not being able to decode claim cache file
2017-03-23 11:07:15 -04:00
Kay Kurokawa 2cc2505cfd make directories in one place when initializing 2017-03-23 11:07:03 -04:00
Kay Kurokawa 7fac63a37b create download directory if it doesn't exist 2017-03-23 11:07:03 -04:00
Alex Grin c18c1725c4 Merge pull request #554 from lbryio/fix_restart
Fix restart procedure in DaemonControl
2017-03-22 16:52:55 -04:00
Kay Kurokawa 5e6a742bd3 Fix restart procedure in DaemonControl 2017-03-22 11:48:31 -04:00
Alex Grintsvayg 790baea191 Revert "Bump version: 0.9.2rc1 → 0.9.3rc1"
This reverts commit e8e2a0d353.
2017-03-21 17:24:07 -04:00
Alex Grintsvayg e8e2a0d353 Bump version: 0.9.2rc1 → 0.9.3rc1 2017-03-21 17:12:29 -04:00
Alex Grintsvayg da3ed0756e Bump version: 0.9.1 → 0.9.2rc1 2017-03-21 17:10:51 -04:00
Alex Grin 05f7f5c73c Merge pull request #550 from lbryio/handle-bad-claim-cache-file
Handle failure to read claim cache file
2017-03-21 16:44:37 -04:00
Jack Robison b19e3adadd update changelog 2017-03-21 16:33:17 -04:00
Jack Robison 824f2f1d0f handle not being able to decode claim cache file 2017-03-21 16:32:32 -04:00
Alex Grintsvayg f515583ec4 fix _get_transaction 2017-03-21 12:11:47 -04:00
Jack Robison c1a5806bc0 Merge pull request #532 from lbryio/fix_descriptor_get
Fix broken descriptor_get command
2017-03-20 16:50:19 -04:00
Jack Robison c66225ccf8 Merge branch 'master' into fix_descriptor_get 2017-03-20 16:42:10 -04:00
Alex Grintsvayg 8f3b53197f dont add expected payment when payment rate is 0 2017-03-20 15:37:45 -04:00
Alex Grin 4027733f2c Merge pull request #542 from lbryio/wallet-list
add wallet_list
2017-03-20 15:31:45 -04:00
Alex Grin 3d606b95aa Merge pull request #546 from lbryio/fix_cli_error_msg
format help message properly in case of error in CLI
2017-03-20 15:29:00 -04:00
Kay Kurokawa 85845414b8 format help message properly in case of error in CLI 2017-03-20 13:41:32 -04:00
Jack Robison a95c49bd08 add wallet_list 2017-03-19 10:52:51 -04:00
Alex Grintsvayg c422b5252e Bump version: 0.9.1rc6 → 0.9.1 2017-03-17 18:24:20 -04:00
Alex Grin cb4c93bcd9 Merge pull request #540 from lbryio/fix_getpubkey
fix wallet_public_key API command
2017-03-17 17:49:29 -04:00
Kay Kurokawa 4edd13f4f4 fix wallet_public_key API command 2017-03-17 16:45:34 -04:00
Alex Grintsvayg a4de0056ab Bump version: 0.9.1rc5 → 0.9.1rc6 2017-03-17 13:11:38 -04:00
Alex Grintsvayg ee77bb0670 unused imports 2017-03-17 13:10:40 -04:00
Alex Grintsvayg 6482619fac stupid bug 2017-03-17 13:07:23 -04:00
Alex Grintsvayg b00c20180b Bump version: 0.9.1rc4 → 0.9.1rc5 2017-03-16 17:56:25 -04:00
Kay Kurokawa 001ebd02e3 fix broken descriptor_get command 2017-03-16 17:55:35 -04:00
Alex Grintsvayg 15517732ae add config to disable reflecting on publish 2017-03-16 17:49:24 -04:00
Jeremy Kauffman ebb6813df3 Merge pull request #531 from lbryio/better_jsonrpc_publish
Change publish to take metadata fields as arguments, docstring change
2017-03-16 16:02:49 -04:00
Alex Grintsvayg bfd822782d small fixes 2017-03-16 15:35:54 -04:00
Kay Kurokawa e66a1d3577 add changelog 2017-03-16 14:48:12 -04:00
Kay Kurokawa 61d27f010e Change publish to take metadata fields as arguments, docstring change 2017-03-16 14:38:27 -04:00
Alex Grintsvayg 52d471d660 Bump version: 0.9.1rc3 → 0.9.1rc4 2017-03-15 21:53:36 -04:00
Alex Grintsvayg 33c9a7a098 Bump version: 0.9.1rc2 → 0.9.1rc3 2017-03-15 21:26:08 -04:00
Alex Grintsvayg 8201fc186d Bump version: 0.9.1rc1 → 0.9.1rc2 2017-03-15 19:59:59 -04:00
Alex Grintsvayg 9d7dcf4af7 no more stop-daemon command 2017-03-15 19:58:52 -04:00
Alex Grintsvayg 49b18a1fca minor 2017-03-15 16:33:41 -04:00
Alex Grintsvayg 26d2347812 update docs 2017-03-15 16:31:58 -04:00
Alex Grintsvayg 29dea4901e add --version flag, remove simplejson dependency 2017-03-15 16:24:25 -04:00
Alex Grintsvayg 1c2290dc8e update template 2017-03-15 14:53:00 -04:00
Alex Grintsvayg 8c664a939b add issues template 2017-03-15 14:49:52 -04:00
Alex Grintsvayg 12bc804622 Bump version: 0.9.0 → 0.9.1rc1 2017-03-15 12:34:32 -04:00
Alex Grintsvayg 593e8a94ad no more rc0 2017-03-15 12:32:49 -04:00
Alex Grintsvayg 2c4e4d4526 Revert "Bump version: 0.9.0 → 0.9.1rc0"
This reverts commit 74844f18ad.
2017-03-15 12:31:40 -04:00
Alex Grintsvayg 74844f18ad Bump version: 0.9.0 → 0.9.1rc0 2017-03-15 12:30:52 -04:00
Alex Grintsvayg 93406fe07f Revert "Bump version: 0.9.0 → 0.9.001"
This reverts commit 820e3091bf.
2017-03-15 11:31:36 -04:00
Alex Grintsvayg 820e3091bf Bump version: 0.9.0 → 0.9.001 2017-03-15 11:19:07 -04:00
Alex Grintsvayg c4b65fddac newline after jsonrpc response 2017-03-14 11:11:29 -04:00
Alex Grintsvayg 25dcf592a0 Bump version: 0.9.0rc17 → 0.9.0 2017-03-13 23:16:01 -04:00
Umpei Kay Kurokawa 02bae15b02 Merge pull request #528 from lbryio/api_doc_consis
Make api docstrings consistent
2017-03-13 20:50:45 -04:00
Kay Kurokawa b32b7091ad make api docstrings consistent 2017-03-13 20:43:18 -04:00
Alex Grintsvayg 28e662a53e Revert "Bump version: 0.9.0rc17 → 0.9.1rc0"
This reverts commit a440656a33.
2017-03-13 18:45:14 -04:00
Alex Grintsvayg a440656a33 Bump version: 0.9.0rc17 → 0.9.1rc0 2017-03-13 18:15:37 -04:00
Jack Robison 4e765b265f fix reading description in setup.py 2017-03-13 16:05:54 -04:00
Umpei Kay Kurokawa ec35eb2397 Merge pull request #526 from lbryio/api_doc_string
Daemon API doc string update
2017-03-13 14:18:14 -04:00
Alex Grintsvayg be4e1c99d5 update setup.py 2017-03-13 14:10:33 -04:00
Jeremy Kauffman 031d9c28ed more readme cleanup 2017-03-13 12:14:02 -04:00
Jeremy Kauffman 6d53a582cf running and usage 2017-03-13 12:05:13 -04:00
Jeremy Kauffman 64668d0043 update readme to reflect new packaging 2017-03-13 11:59:12 -04:00
Alex Grintsvayg 92d423aefd fix jsonrpc http headers 2017-03-13 11:54:40 -04:00
Kay Kurokawa 0ec9b6545f edit doc strings and update doc file 2017-03-12 14:45:46 -04:00
Umpei Kay Kurokawa 003b3df378 Merge pull request #525 from lbryio/fix_update
Fix bug in update
2017-03-10 18:39:46 -05:00
Umpei Kay Kurokawa f409d43dcc Merge pull request #524 from lbryio/fix_fee_validator
Fix fee validation in publisher and clarify publish docstring
2017-03-10 18:38:07 -05:00
Kay Kurokawa b2c2edfb12 fixed bug of decimal not being subtractable from float 2017-03-10 18:33:21 -05:00
Kay Kurokawa e7dc841bb7 fix fee validation in publisher and clarify publish docstring 2017-03-10 18:24:32 -05:00
Alex Grintsvayg 2303f88bd4 Bump version: 0.9.0rc16 → 0.9.0rc17 2017-03-10 16:30:28 -05:00
Umpei Kay Kurokawa 7291efac83 Merge pull request #513 from lbryio/fix_api_consistency_lbryum
Integrate more consistent dictionary keys and values from lbryum
2017-03-10 14:12:04 -05:00
Kay Kurokawa 8c61185387 handle decimal encoded string output from lbryum 2017-03-10 12:56:58 -05:00
Kay Kurokawa 371591f96e add changelog 2017-03-10 11:52:12 -05:00
Kay Kurokawa 98d3a4cb46 fix api consistency issues from lbryum 2017-03-10 11:52:12 -05:00
Alex Grintsvayg ab1bdb7eaf Bump version: 0.9.0rc15 → 0.9.0rc16 2017-03-09 15:30:36 -05:00
Umpei Kay Kurokawa 8ef3f81f0c Merge pull request #523 from lbryio/remove_unused_wallet_func
remove unused functions in wallet
2017-03-09 13:15:26 -05:00
Umpei Kay Kurokawa cf56d8fa4a Merge pull request #522 from lbryio/do_not_expose_get_claims_from_tx
Do not expose get_claims_from_tx() from the API
2017-03-09 13:15:13 -05:00
Kay Kurokawa 881efabc0c remove unused _get_status_of_claim() and get_name_and_validity_for_sd_hash() function in Wallet 2017-03-09 12:02:36 -05:00
Kay Kurokawa aef274577f Do not expose get_claims_from_tx() from the API 2017-03-09 11:54:34 -05:00
Jack Robison 5d6ca0aae6 add get_sd_hash to utils 2017-03-09 10:39:25 -05:00
Alex Grintsvayg 9a4230dd96 Bump version: 0.9.0rc14 → 0.9.0rc15 2017-03-09 10:37:44 -05:00
Alex Grintsvayg 3f4a4728e6 fixed my silly mistakes 2017-03-09 07:58:36 -05:00
Alex Grintsvayg 63cb2cf571 raise exceptions, dont yield them 2017-03-08 17:14:31 -05:00
Alex Grintsvayg 54b4b6984e fix jsonrpc help 2017-03-08 17:10:03 -05:00
Kay Kurokawa 98f90a4ad6 fixing minor debugging bug 2017-03-08 15:32:00 -05:00
Jack Robison e29a91b69a fix pylint 2017-03-08 14:19:54 -05:00
Jack Robison fdcc5e6829 fix file delete keyword arg bug 2017-03-08 14:04:40 -05:00
Umpei Kay Kurokawa da99f3f22b Merge pull request #493 from lbryio/filemanager_related_tests
Filemanager related unit tests
2017-03-08 11:53:46 -05:00
Jack Robison 992be881af fix _get_lbry_file return bug 2017-03-08 11:53:24 -05:00
Alex Grintsvayg a0cbd5af33 Merge branch 'less_verbose_loggly'
* less_verbose_loggly:
  make loggly logs less verbose
2017-03-08 11:23:37 -05:00
Alex Grintsvayg 68f5588bd9 make loggly logs less verbose 2017-03-08 11:23:28 -05:00
Umpei Kay Kurokawa 531fa63590 Merge pull request #519 from lbryio/fix_publish_docstring
fix docstring for daemon publish command
2017-03-08 10:50:54 -05:00
Kay Kurokawa 9048244cdc fix docstring for daemon publish command 2017-03-08 10:39:31 -05:00
Jack Robison 74754c9870 Merge pull request #517 from lbryio/fix-filter-by-outpoint
fix filter files by outpoint
2017-03-08 09:40:14 -05:00
Jack Robison a7c2ad470c fix tests 2017-03-08 09:36:32 -05:00
Jack Robison a132496643 fix filter files by outpoint 2017-03-08 09:31:36 -05:00
Kay Kurokawa f600038712 work on unit tests for file management related classes 2017-03-08 09:15:06 -05:00
Kay Kurokawa eae8f8a00e add random_lbry_hash() function to tests.util 2017-03-08 09:15:06 -05:00
Umpei Kay Kurokawa ac33663444 Merge pull request #510 from lbryio/remove_jsonrpc_coms
Removing unuseful/unused commands from daemon API
2017-03-07 21:23:33 -05:00
Kay Kurokawa ba8b7ca090 add changelog 2017-03-07 21:17:06 -05:00
Kay Kurokawa 1aa715c927 remove file_utils import from Daemon 2017-03-07 21:17:06 -05:00
Kay Kurokawa 931925be15 remove start and reveal from file_utils 2017-03-07 21:17:06 -05:00
Kay Kurokawa e2ed9f21d2 delete reveal and open command from api, not needed for electron 2017-03-07 21:17:06 -05:00
Kay Kurokawa e6cd4c9877 removing unuseful/unused commands from daemon API 2017-03-07 21:17:06 -05:00
Jack Robison e2cc212eb8 Merge pull request #512 from lbryio/file-list-refactor
File list refactor
2017-03-07 20:38:04 -05:00
Jack Robison 74cf14df48 Merge pull request #514 from lbryio/fix-compare-outpoint
Add string comparison to ClaimOutpoint
2017-03-07 20:36:44 -05:00
Alex Liebowitz 18c690d801 Add string comparison to ClaimOutpoint 2017-03-07 18:31:05 -05:00
Jack Robison 9e09d1fc38 handle multiple files to delete in file_delete 2017-03-07 18:30:18 -05:00
Jack Robison 454755169a explain get_count_for_stream in file_delete 2017-03-07 18:29:44 -05:00
Jack Robison bc9b990728 return file json from get 2017-03-07 18:29:44 -05:00
Jack Robison 7ad46e70be remove deprecated delete_lbry_file, move file deletion to lbry_file_manager 2017-03-07 18:29:44 -05:00
Jack Robison 4f9b5c4624 refactor file_list, remove deprecated file retrieval functions 2017-03-07 18:29:44 -05:00
Jack Robison 9744287370 change attribute "uri" to "name" 2017-03-07 18:29:44 -05:00
Alex Grintsvayg 59a5a3164f Bump version: 0.9.0rc13 → 0.9.0rc14 2017-03-07 16:03:19 -05:00
Alex Grintsvayg 6fb3313eaf Bump version: 0.9.0rc12 → 0.9.0rc13 2017-03-06 18:15:45 -05:00
Alex Grintsvayg 2bfbae9d4e Bump version: 0.9.0rc11 → 0.9.0rc12 2017-03-06 17:47:50 -05:00
Alex Grin daa0ff036d Merge pull request #507 from lbryio/fix_download_timeout
Fix downloading not timing out properly when unable to download sd blob
2017-03-03 17:14:00 -05:00
Kay Kurokawa d642ba039d Fix downloading not timing out properly when unable to download sd blob 2017-03-03 16:45:25 -05:00
Umpei Kay Kurokawa 9350b9e127 Merge pull request #501 from lbryio/fix_exchange_rate_manager
Improvements to exchange rate manager
2017-03-03 12:54:57 -05:00
Kay Kurokawa 9755e9b11f Improvements to exchange rate manager 2017-03-03 12:50:47 -05:00
Alex Grin a64424f28c Merge pull request #503 from lbryio/exclude_tests_setup
Exclude tests folders in setup.py
2017-03-02 09:04:18 -05:00
Kay Kurokawa eee1acb24a Exclude tests folders in setup.py 2017-02-28 15:03:00 -05:00
Alex Grintsvayg 0e76ca0767 update docs 2017-02-28 12:31:55 -05:00
Alex Liebowitz e8afad7af8 Bump version: 0.9.0rc10 → 0.9.0rc11 2017-02-27 17:49:07 -05:00
Jeremy Kauffman f9642018ae documentation config changes 2017-02-27 17:36:29 -05:00
Umpei Kay Kurokawa 41d96fec00 Merge pull request #500 from lbryio/fix_height_error
Add check in case local height is less than zero
2017-02-27 13:39:49 -05:00
Kay Kurokawa 88f270180c add check in case local height is less than zero 2017-02-27 13:28:58 -05:00
Umpei Kay Kurokawa 76df0e3940 Merge pull request #497 from lbryio/client_protocol_timeout
Adding timeout to ClientProtocol
2017-02-27 13:17:41 -05:00
Kay Kurokawa 7128b6401e adding missing __init__.py file in test folder 2017-02-27 13:10:23 -05:00
Kay Kurokawa 824fbcc25a adding timeout to ClientProtocol 2017-02-27 13:02:35 -05:00
Job Evers‐Meltzer fba91139e3 Merge pull request #499 from lbryio/requirements-vs-setup
cleaner separation between requirements and setup
2017-02-24 06:59:48 -06:00
jobevers 090840ee25 cleaner separation between requirements and setup 2017-02-23 14:18:52 -06:00
jobevers d3d8426430 Bump version: 0.9.0rc9 → 0.9.0rc10 2017-02-22 16:14:40 -06:00
jobevers 40d4fb5641 fix section headers on changelog 2017-02-22 13:13:14 -06:00
jobevers d986fc27f3 update changelog 2017-02-22 12:58:39 -06:00
jobevers 9af1cae249 Bump version: 0.9.0rc8 → 0.9.0rc9 2017-02-22 12:51:10 -06:00
jobevers 056ee2d1ce Stop using bumpversion to bump the changelog
It wasn't working very well so its being replaced by a stand-alone
script
2017-02-21 22:19:07 -06:00
jobevers 301e4ae0fa update version to 0.9.0rc8 2017-02-21 21:03:58 -06:00
jobevers d359949a5c check network exists on jsonrpc_status call 2017-02-21 15:30:36 -06:00
jobevers d4900fd607 add emacs temp files to gitignore 2017-02-21 15:15:52 -06:00
jobevers f596d42ca2 restore needed gmpy binary to build lbrynet 2017-02-21 14:08:34 -06:00
Job Evers‐Meltzer 04b36e2252 Merge pull request #486 from lbryio/electron
Electron
2017-02-21 13:19:55 -06:00
Alex Grintsvayg 2add545ab6 unbreak tests 2017-02-21 13:48:05 -05:00
jobevers 70cf179139 Merge remote-tracking branch 'origin/master' into electron 2017-02-21 12:07:36 -06:00
jobevers b6086a7d6d cleanup changelog 2017-02-21 12:05:50 -06:00
Alex Grintsvayg 62ab4c769d move blocks_behind into blockchain_status 2017-02-21 12:51:15 -05:00
jobevers c920663444 Bump version: 0.8.6 → 0.8.7 2017-02-21 11:36:44 -06:00
jobevers 1d30ae447a Merge branch 'master' into electron 2017-02-21 10:47:28 -06:00
jobevers ff5980dfca fix broken version check 2017-02-21 09:36:08 -06:00
jobevers e4b77a4a0a clean up lint errors 2017-02-21 08:45:12 -06:00
jobevers 666b568eea update version info 2017-02-21 08:39:32 -06:00
Job Evers‐Meltzer 998892ef0b Merge pull request #446 from lbryio/reseed
Script to Reseed a File
2017-02-21 06:11:32 -06:00
jobevers 847a79f526 use union merge on CHANGELOG.md
http://krlmlr.github.io/using-gitattributes-to-avoid-merge-conflicts/
2017-02-20 21:37:50 -06:00
Umpei Kay Kurokawa 07841a7803 Merge pull request #475 from lbryio/call_later
Adding reactor.callLater as a utils function
2017-02-20 19:15:12 -05:00
jobevers 4e55d674da WIP: update bumpversion 2017-02-20 16:29:12 -06:00
Kay Kurokawa 3522f9af7d use utils.call_later in DHTHash Announcer 2017-02-20 16:48:29 -05:00
Kay Kurokawa c27fe7af31 use utils.call_later in ConnectionManager 2017-02-20 16:48:29 -05:00
Kay Kurokawa 30c7559d3a adding reactor.callLater as a utils function 2017-02-20 16:48:23 -05:00
Kay Kurokawa 7e2fad2397 dummy hash announcer requires hash_queue_size() function 2017-02-20 16:48:22 -05:00
jobevers 06466db1b0 update readme 2017-02-20 15:12:47 -06:00
Umpei Kay Kurokawa 213a7ab66c Merge pull request #489 from lbryio/fix_dummy_hash_announcer
dummy hash announcer requires hash_queue_size() function
2017-02-20 14:48:28 -05:00
jobevers a1650d7f03 remove update url 2017-02-20 12:55:21 -06:00
Kay Kurokawa 5737d26ffe dummy hash announcer requires hash_queue_size() function 2017-02-20 12:05:12 -05:00
Alex Grintsvayg c85226d5a6 stop showing deprecated commands in cli help 2017-02-20 00:21:16 -05:00
Jack Robison 37b2297fb7 Bump version: 0.8.6rc0 → 0.8.6 2017-02-19 19:50:41 -05:00
Jack Robison 450ac7aa57 Bump version: 0.8.5 → 0.8.6rc0 2017-02-19 19:50:30 -05:00
Jack Robison 68e7c66e3a Merge pull request #495 from lbryio/fix-get
bug fixes in get and get_availability
2017-02-19 19:48:41 -05:00
Jack Robison 55f4e2939a update changelog 2017-02-19 19:37:50 -05:00
Jack Robison b18679af53 debug log 2017-02-19 19:37:50 -05:00
Jack Robison d0b64ef255 don't return blob timeout error from get_availability 2017-02-19 19:37:50 -05:00
Jack Robison e2f97ba0b4 get file by stream hash 2017-02-19 19:37:44 -05:00
Jack Robison c6e614ebec fix get()
-get_file_by_sd_hash raises NoSuchSDHash if there is no file, which
otherwise doesn’t get trapped

-remove unnecessary try/except around _DownloadNameHelper.__init__

-don’t delete self.waiting_on[name]
2017-02-19 19:17:25 -05:00
Jack Robison 76b4b1e939 Update CHANGELOG.md 2017-02-18 14:59:40 -05:00
Jack Robison 29ff39fdc9 Bump version: 0.8.5rc0 → 0.8.5 2017-02-18 14:50:39 -05:00
Jack Robison d2e1eeb61e Bump version: 0.8.4 → 0.8.5rc0 2017-02-18 14:26:57 -05:00
Jack Robison e86bada307 Merge pull request #494 from lbryio/fix-download-button
fix missing download button
2017-02-18 14:23:29 -05:00
Jack Robison 59750843d4 update changelog 2017-02-18 14:16:48 -05:00
Jack Robison ab46a33913 fix missing download button 2017-02-18 14:07:07 -05:00
jobevers 5bf5a7d0b1 Bump version: 0.8.4rc0 → 0.8.4 2017-02-17 14:39:37 -06:00
jobevers 23f2a42372 Bump version: 0.8.3 → 0.8.4rc0 2017-02-17 14:37:21 -06:00
jobevers 14b4c06969 Merge branch 'master' into electron 2017-02-17 09:46:35 -06:00
jobevers 6d4c606abd remove version check also supplies a download url 2017-02-16 21:25:09 -06:00
Job Evers‐Meltzer eb039201e8 Merge pull request #484 from lbryio/add-traceback-to-reported-errors
record traceback in error event
2017-02-16 15:50:26 -06:00
jobevers 723105157f check lbry-electron for releases 2017-02-16 14:18:32 -06:00
jobevers 17d32ca696 remove lbryum version check 2017-02-16 14:18:23 -06:00
jobevers 0eb8e1609f add changelog 2017-02-16 14:13:43 -06:00
jobevers 85390302e2 still more fixes 2017-02-16 14:13:01 -06:00
jobevers 993f2e54f9 more fixes for build 2017-02-16 14:13:01 -06:00
jobevers 87b7e5c742 more fixes for build 2017-02-16 14:13:01 -06:00
jobevers 8b9fef4ed5 forgot to install lbryum 2017-02-16 14:13:01 -06:00
jobevers d4cd8308f1 use build_script for appveyor 2017-02-16 14:13:01 -06:00
jobevers 73eebdb0e3 move to sudoless; install pre-reqs 2017-02-16 14:13:00 -06:00
jobevers 0c30655605 make appveyor build do nothing 2017-02-16 14:13:00 -06:00
jobevers e1455c3a0d add danger; build in venv 2017-02-16 14:13:00 -06:00
jobevers 205fd66a18 don't build lbry package on travis 2017-02-16 14:13:00 -06:00
jobevers 705b65fa3c Update twisted version
This fixes https://github.com/lbryio/lbry/issues/487
2017-02-16 14:13:00 -06:00
Job Evers‐Meltzer fb2823f59e don't need this anymore 2017-02-16 14:13:00 -06:00
jobevers ddfaf51393 simplify setup.py 2017-02-16 14:12:56 -06:00
jobevers f1231bd8e5 remove UI_Manager 2017-02-16 14:11:54 -06:00
jobevers 19aa1eebd5 use lbryum submodule instead 2017-02-16 14:11:36 -06:00
jobevers 24e7b7797c remove packaging 2017-02-16 14:09:59 -06:00
Job Evers-Meltzer caa8f73f7c remove header check 2017-02-16 14:09:43 -06:00
jobevers 0674b48223 record traceback in error event 2017-02-16 13:31:59 -06:00
Job Evers‐Meltzer 5bf75ef139 Merge pull request #404 from lbryio/match-allow-origin
Allow requests that match allowed-origin
2017-02-16 12:02:20 -06:00
Jack Robison 6e310517e0 Merge pull request #469 from lbryio/blob-commands
Blob commands
2017-02-16 12:18:23 -05:00
Job Evers-Meltzer e2db99f7ab allow requests that match allowed-origin 2017-02-16 11:44:23 -05:00
Jack Robison 3151c10556 update changelog 2017-02-16 11:20:11 -05:00
Jack Robison 3ddf8680fb blob commands
refactor blob_get
add descriptor_get
add blob_delete
refactor blob_list
2017-02-16 11:20:02 -05:00
Jack Robison 25ec8fde23 add timeout to get_availability and peer_list
-add optional timeout to DHTPeerFinder.find_peers_for_blob
-add peer_search_timeout setting
2017-02-16 11:17:37 -05:00
Jack Robison bcd026a1b6 add only-free payment rate manager for testing 2017-02-16 11:17:37 -05:00
Jack Robison 1a6290eacc log blob count at startup 2017-02-16 11:17:37 -05:00
Jack Robison f4217d5593 Merge pull request #485 from lbryio/return-pending-claim-lbry-file
file_get by name for file with pending claim
2017-02-16 11:16:46 -05:00
Jack Robison 8bba5c04db update tests and changelog 2017-02-16 09:12:57 -05:00
Jack Robison 345301a9ee return file with pending claim with get_lbry_file
if a lbry file from the winning claim is in the file manager, return
it. otherwise return a lbry file with a pending claim if one exists.
2017-02-16 09:12:57 -05:00
Jack Robison 0f907a209b add EncryptedFileDownloader.load_file_attributes 2017-02-16 09:12:47 -05:00
Jack Robison 03c0ff8529 Merge pull request #359 from lbryio/no-more-upload-allowed
No more upload_allowed
2017-02-15 22:52:07 -05:00
Jack Robison 80dd02483e Merge branch 'master' into no-more-upload-allowed 2017-02-15 22:39:26 -05:00
jobevers 8452620f99 Bump version: 0.8.3rc3 → 0.8.3 2017-02-15 16:29:40 -06:00
jobevers 287e62b71b bump lbryum version 2017-02-15 12:54:38 -06:00
Job Evers‐Meltzer 934a46a8b6 Merge pull request #452 from lbryio/retry-startup
Retry startup
2017-02-15 08:34:26 -06:00
Job Evers‐Meltzer 23e06ba872 Merge pull request #482 from lbryio/catch-unknown-name-error
Catch unknown name error
2017-02-15 07:54:04 -06:00
jobevers 1fd87b7caa add changelog 2017-02-15 07:09:43 -06:00
jobevers f7c105fd80 fixes 2017-02-15 06:48:59 -06:00
jobevers 9e2cda0cdc remove more upload_allowed arguments 2017-02-15 06:48:58 -06:00
jobevers 4fb5e932e6 fix indentation 2017-02-15 06:48:58 -06:00
jobevers fe4dac8236 change downloader options to data_rate 2017-02-15 06:48:58 -06:00
Job Evers-Meltzer 2a45c91175 fix get_blob calls with upload_allowed arguments 2017-02-15 06:48:58 -06:00
Job Evers-Meltzer 439a840700 Better log in tests on failure 2017-02-15 06:48:58 -06:00
Job Evers-Meltzer a38e5c855e Remove upload_allowed attribute
This is unused and largely pulluting the argument list
of too many functions.
2017-02-15 06:48:50 -06:00
Job Evers-Meltzer 90d3a0ef9c Add script to reseed a file 2017-02-15 06:43:41 -06:00
jobevers f989814e69 catch Unknownnameerror for resolve_name
Its expected behavior that a name won't resolve so this more
gracefully handles that case
2017-02-14 16:16:22 -06:00
Job Evers‐Meltzer 9f4ccfe7c3 Merge pull request #483 from lbryio/add-traceback
Add traceback to error in EncryptedFileManager
2017-02-14 16:02:14 -06:00
jobevers 520ac9b50a change filemanger setup to inlinecallbacks 2017-02-14 10:27:16 -06:00
jobevers dbb6c6f3a4 remove unused parts of file manager 2017-02-14 10:26:45 -06:00
jobevers 993d606bc6 add logging on downloader start 2017-02-14 10:26:03 -06:00
jobevers ffb48ef287 bugfix: on restore, dont wait for download to finish 2017-02-14 10:25:34 -06:00
jobevers 62fcbc23e3 bugfix: exception needs string interpolation 2017-02-14 10:25:03 -06:00
jobevers a43b726f32 update changelog 2017-02-14 09:29:42 -06:00
jobevers 3bdd33567b whitespace 2017-02-14 09:14:18 -06:00
jobevers 6bdb0280be better log exception in EncryptedFileManager
The error message in #478 is too generic. This wraps the source of the
error in a try/catch and logs the traceback.
2017-02-14 09:11:39 -06:00
jobevers 77746cea5f Bump version: 0.8.3rc2 → 0.8.3rc3 2017-02-14 07:55:54 -06:00
jobevers 25cede0dc3 remove branches restriction on travis builds 2017-02-14 07:55:45 -06:00
jobevers acfe034990 Bump version: 0.8.3rc1 → 0.8.3rc2 2017-02-13 23:34:36 -06:00
jobevers e2021deae1 Bump version: 0.8.3rc0 → 0.8.3rc1 2017-02-13 22:06:57 -06:00
Umpei Kay Kurokawa 37cc257555 Merge pull request #465 from lbryio/dht_hash_announcer_tests
Adjust hash reannounce based on queue size , test for DHTHashAnnouncer, immediately announce new blobs
2017-02-13 21:08:48 -05:00
jobevers a22758d5c7 try building tags too 2017-02-13 16:56:10 -06:00
Jack Robison aa8375673b Merge pull request #477 from lbryio/inlinecallbacks-refactor
Refactor publish, file manager, and file downloader
2017-02-13 17:42:54 -05:00
Kay Kurokawa 5831e253cd adding change log 2017-02-13 17:27:52 -05:00
Kay Kurokawa 34f5bc93ae adding immediate announce option to hash announce 2017-02-13 17:27:10 -05:00
Kay Kurokawa aa45b0e121 Fixes for other tests now that BlobManager relies on DHTHashAnnouncer.get_next_announce_time() 2017-02-13 17:27:10 -05:00
Kay Kurokawa 8e0f71ac74 Make hash reannounce time adjusted based on the queue of hashes yet to be announced, add test for DHTHashAnnouncer 2017-02-13 17:27:10 -05:00
Jack Robison 55445b1bb7 update changelog 2017-02-13 15:27:39 -05:00
Jack Robison 3cc8a4b168 add local and remote heights to blockchain status 2017-02-13 15:14:54 -05:00
Jack Robison 8859c1b0c1 fix claim_abandon 2017-02-13 15:14:54 -05:00
Jack Robison 867275461c feedback from job 2017-02-13 15:14:54 -05:00
Jack Robison 2190f4ac85 refactor GetStream
-convert to inline callbacks

-return more errors than just timeouts

-delete on timeout

-have a more understandable return condition (first data blob having
downloaded)
2017-02-13 15:14:54 -05:00
Jack Robison b51d3a28e3 update changelog 2017-02-13 15:14:54 -05:00
Jack Robison 7720724ec0 reflect lbry_files in looping call in EncryptedFileManager
-remove ManagedEncryptedFileDownloader._reupload

-clean up reflector functions in Daemon, move to reflector.reupload

-check ConnectionLost in reflector client

-close sd blob file handle when it wont be sent (otherwise read handle stays open)

-log reflector sd info

-give reflector client factory a lbry file
2017-02-13 15:13:37 -05:00
Umpei Kay Kurokawa dbd71c93e7 Merge pull request #458 from lbryio/connection_manager_tests
Connection Manager tests and upgrades
2017-02-13 13:55:33 -05:00
Jack Robison d137528f67 convert Daemon._delete_lbry_file to inlineCallbacks 2017-02-13 13:07:27 -05:00
Jack Robison e292abceee convert EncryptedFileManager to use inlineCallbacks 2017-02-13 13:07:27 -05:00
Jack Robison 8de6bd7c7a convert EncryptedFileDownloader to inlineCallbacks 2017-02-13 13:07:27 -05:00
jobevers bb9b221cf6 convert publish_sd_blob to inlineCallback 2017-02-13 13:07:27 -05:00
Jack Robison 5a1bab2eaa refactor Publisher and jsonrpc_publish
-add get_read_handle to file_utils

-don’t leave read handle hanging after creating lbry file

-get rid of threaded weirdness

-remove reflector functionality from Publisher

-fix updating with an existing stream

-reflect new stream in thread after broadcasting name claim
2017-02-13 13:07:27 -05:00
Kay Kurokawa 4f87e87d3e Adding connection manager tests, and modifying connection manager to be testeable, fix bug where failed connection was not marked as down 2017-02-13 12:17:27 -05:00
Alex Grintsvayg c7106dfc97 reduce loudness of reflector message 2017-02-13 12:12:32 -05:00
Alex Grin 11b1f8c7af Merge pull request #382 from lbryio/silence_socket_error
silence "[Errno 11] Resource temporarily unavailable" error
2017-02-12 20:46:59 -05:00
jobevers f5e6ff0c32 add changelog 2017-02-11 13:05:10 -06:00
jobevers 71123c0de1 add warning log on EWOULDBLOCK 2017-02-11 13:04:36 -06:00
Alex Grintsvayg 35b2e56bd4 silence "[Errno 11] Resource temporarily unavailable" error 2017-02-11 13:04:35 -06:00
jobevers 41abd0a171 Bump version: 0.8.2 → 0.8.3rc0 2017-02-10 14:16:29 -06:00
Jack Robison cea4eca775 update example script 2017-02-09 10:44:29 -05:00
Jack Robison 6b22debb79 Merge pull request #474 from lbryio/fix-update-claim
fix Wallet.claim_name, convert to inlineCallbacks
2017-02-09 10:23:48 -05:00
Jack Robison 630d0bf5be convert Wallet.claim_name to use inlineCallbacks 2017-02-09 10:13:41 -05:00
Jack Robison 19fe101d73 fix claim update
dict.update does not return the dict, so this can’t be lambda’d
2017-02-09 10:11:46 -05:00
Jack Robison 75cc0dcce9 Merge pull request #472 from lbryio/fix-bug-report-slack-key
Fix how the bug report code accesses Slack API key setting
2017-02-09 00:07:52 -05:00
Alex Liebowitz 0e8e489ad6 Fix how the bug report code accesses Slack API key setting
It was trying to access this setting as an attribute, which doesn't
work anymore. (Not sure if this is an API change in conf.settings or
just a bug.)
2017-02-08 23:05:32 -05:00
Jack Robison b03e7e14f3 Merge pull request #471 from lbryio/fix-delete-lbry-file
fix file_delete
2017-02-08 21:04:54 -05:00
Jack Robison e2050b4fc1 fix get_count_for_stream returning [(count, )] instead an int 2017-02-08 20:58:45 -05:00
Jack Robison d1aa69bfa3 Merge pull request #466 from lbryio/fix-untracked-lbry-streams
fix streams in stream_info_manager not being loaded by lbry_file_manager
2017-02-08 19:47:03 -05:00
Jack Robison 7e2456fa69 convert _setup, _check_stream_is_managed, _check_stream_info_manager, and _start_lbry_files to inlineCallbacks
remove unused MissingLBRYFile exception
2017-02-08 19:34:54 -05:00
Jack Robison ecac2ae2ca update changelog 2017-02-08 17:03:53 -05:00
Jack Robison bfc02dd3e5 fix streams in stream_info_manager not being loaded by lbry_file_manager 2017-02-08 17:02:02 -05:00
Jack Robison c527c32e5f Merge pull request #467 from lbryio/fix-reflector-bugs
Fix reflector bugs
2017-02-08 16:40:57 -05:00
Jack Robison 777419b4a9 log traceback on failed blob upload
-move status message on connectionDone to client, indicate if blobs
were sent or not (and how many blobs reflector still needs, if any)

-only try uploading failed blob once after first failure, to prevent
indefinite retries
2017-02-08 16:18:34 -05:00
Jack Robison 832a32474f update changelog 2017-02-08 14:46:18 -05:00
Jack Robison f2ddc9bd98 fix recursion depth bug upon failed blob 2017-02-08 14:28:30 -05:00
Jack Robison d053db8dfd add uri to stream reflector for better logging 2017-02-08 14:28:16 -05:00
Jack Robison a58a81a841 stopProducing in reflector client file_sender when uploading is done
this fixes exceptions.ValueError: I/O operation on closed file in
twisted.protocols.basic.ResumeProducing
2017-02-08 14:09:03 -05:00
jobevers df40c7fb76 bumpversion updates the changelog 2017-02-08 01:30:37 -06:00
jobevers ef65d8dd1c fix because the danger.systems code example was wrong 2017-02-07 23:08:35 -06:00
Job Evers‐Meltzer a8780f03a8 Add changelog (#463)
Add changelog and danger checks
2017-02-07 21:34:15 -06:00
Jeremy Kauffman 61e78c1b69 pip install must be run with upgrade flag 2017-02-07 20:51:10 -05:00
Jack Robison a167beaad1 Merge pull request #457 from lbryio/stream-availability
Reflector v2
2017-02-06 16:05:14 -05:00
Jack Robison 4c76e1b98f add blob reflector and v1 tests 2017-02-06 15:56:00 -05:00
Jack Robison f614a13db4 update documentation 2017-02-06 15:56:00 -05:00
Jack Robison 6fae07d29e reflector v2
-add {‘sd_blob_hash’: …, ‘sd_blob_size ‘: …} query type with
{‘send_sd_blob’: True/False, ‘needed_blobs’: []} response

this allows the reflector client to know how much of a stream reflector
already has covered, as to minimize the number of subsequent requests
and prevent streams from being partially reflected

-remove empty {} request
2017-02-06 15:55:46 -05:00
Jack Robison 2126f69c93 use only DBEncryptedFileMetadataManager in Daemon
-fix Daemon.stream_info_manager and Daemon.lbry_file_metadata_manager
being different
2017-02-06 13:15:50 -05:00
Jack Robison 6f0581104a Merge pull request #408 from lbryio/non-blocking-update-check
non-blocking version check
2017-02-05 15:44:29 -05:00
Alex Grintsvayg 47c92f04d2 urihandler takes you to show, not watch 2017-02-03 13:34:16 -05:00
Jack Robison 06be60e3a3 Merge pull request #406 from lbryio/installation-id
add installation id, re-initialize lbryid on each start
2017-02-02 11:37:31 -05:00
Jack Robison 45a6228b58 fix threaded lbryum check 2017-02-02 11:31:31 -05:00
jobevers 773cbe3ace Bump version: 0.8.1 → 0.8.2 2017-02-02 07:43:28 -08:00
Jack Robison c6648ff6bc add installation id, re-initialize lbryid on each start 2017-02-02 10:23:17 -05:00
Jack Robison f737b3aaf7 non-blocking version check
this fixes a bug where if github is down the app will fail to start.

-check for new version every 30 min instead of every 12 hours

-check connection problems every 30 seconds instead of every second
2017-02-02 10:18:09 -05:00
Job Evers‐Meltzer 41d937d1b4 Merge pull request #461 from lbryio/named-parameters
Fixes for named parameter changes on api
2017-02-01 20:56:01 -08:00
jobevers 8e349f17c1 fixes for named parameter changes on api 2017-02-01 20:47:53 -08:00
jobevers 4ab5ace668 Bump version: 0.8.1rc0 → 0.8.1 2017-02-01 15:57:05 -08:00
Job Evers‐Meltzer f46ae75f73 Merge pull request #459 from lbryio/lbryum-version
require minimum version for lbryum
2017-02-01 15:56:32 -08:00
jobevers 61c5a5b2a4 require minimum version for lbryum 2017-02-01 15:08:31 -08:00
Job Evers‐Meltzer 2c137a99d1 Merge pull request #453 from lbryio/add-details-to-error-event
Add details to error event
2017-02-01 14:23:26 -08:00
Job Evers‐Meltzer d1ed8a769a Merge pull request #450 from lbryio/block-too-deep
Fix #449: Block to deep error
2017-02-01 13:45:12 -08:00
jobevers 4bb3e2a154 remove is_lagging wallet attribute 2017-02-01 12:13:43 -08:00
Kay Kurokawa a5e8c561e4 Revert "adding connection manager tests, and modifying connection manager to be testeable"
This reverts commit 7b6e233d22, as it was an accidental push to master
2017-02-01 11:59:36 -05:00
Kay Kurokawa 7b6e233d22 adding connection manager tests, and modifying connection manager to be testeable 2017-02-01 11:52:16 -05:00
Job Evers‐Meltzer ac8a63bc33 Merge pull request #455 from lbryio/better-defaults
Change default behavior with daemon cmdline args
2017-01-31 10:13:18 -08:00
Alex Grintsvayg 15ebc39f95 Merge branch 'nicer_error_output'
* nicer_error_output:
  nicer error output
2017-01-31 10:38:19 -05:00
Alex Grintsvayg 46cb6b204a nicer error output 2017-01-31 10:38:07 -05:00
Alex Grin b219806959 Merge pull request #448 from lbryio/jsonrpc_params_in_function_signature
explicitly list jsonrpc params in function signature. add jsonrpc tests
2017-01-30 16:34:50 -05:00
jobevers 8fc6feb316 log to console by default 2017-01-30 12:08:32 -08:00
jobevers 7377158631 change no-launch to launchui 2017-01-30 12:04:07 -08:00
Alex Grintsvayg 86ba7ffd5c explicitly list jsonrpc params in function signature. add jsonrpc tests 2017-01-30 09:58:28 -08:00
jobevers ed227a239e Fixes #449
What was happening was the wallet claimed to be caught up before it
actually was and so the wallet’s local_height was still the value from
when lbry was last run, frequently more than 20 or 50 blocks
behind. _get_value_for_name uses the block at local_height as the
basis for the proof.  If _get_value_for_name is called during that
time between when the wallet claims to be caught up and it actually
is, the “Block too deep error” happens.  And since the discover page
of the UI does name resolution right away, the error basically happens
anytime somebody starts the app after not using it for a few hours.

This changes the startup behaviour of the wallet to
- use the `update` callback provided by lbryum
- check that local_height and network_height match before declaring
  that the wallet has caught up

For reference, the error is raised here:
https://github.com/lbryio/lbrycrd/blob/1b896ae75b0fc7081ab9e4a1b7d45c94dcd73cbd/src/rpc/claimtrie.cpp#L688
2017-01-30 09:32:04 -08:00
jobevers 7360015fa4 small refactor 2017-01-30 09:32:04 -08:00
jobevers 04ee2f786d remove backslash line continuation 2017-01-30 09:32:04 -08:00
jobevers 43451c485f change alerts to logs 2017-01-30 09:32:04 -08:00
jobevers 565f26bdad switch error event to use log record and add more details 2017-01-30 09:30:46 -08:00
jobevers b7478ba817 replace send_error with log.error 2017-01-30 09:30:46 -08:00
jobevers 1c92b5d2ea retry startup 2017-01-30 09:25:44 -08:00
Job Evers‐Meltzer b744a54340 Merge pull request #451 from lbryio/move-uri-handler
move urihandler out of packaging dir
2017-01-30 09:22:03 -08:00
Jeremy Kauffman 8805d1d0cf a joke I couldn't resist 2017-01-29 15:45:11 -05:00
jobevers b674e5c57a move urihandler out of packaging dir 2017-01-28 22:09:40 -08:00
Job Evers‐Meltzer 139af85b52 Merge pull request #445 from lbryio/retry-get
Retry the get call three times before failing
2017-01-27 10:08:25 -06:00
Alex Grintsvayg ee0f7375d6 fix install bug 2017-01-26 16:48:34 -05:00
Umpei Kay Kurokawa 893fe8823e Merge pull request #409 from lbryio/fix_wallet_race_condition
Fix wallet balance interfaces
2017-01-26 15:12:54 -05:00
Umpei Kay Kurokawa f54d42b8fa Merge pull request #441 from lbryio/connection_manager_logging
Better Connection manager and Client Protocol logging
2017-01-26 15:11:56 -05:00
jobevers 3dc1a523cf retry the get call three times before failing 2017-01-26 14:09:54 -06:00
Kay Kurokawa 90438ddf9f removing str() from logging functions 2017-01-26 12:14:33 -05:00
Kay Kurokawa 08e336e476 adding better debug logs for ConnectionManager and ClientProtocol 2017-01-26 12:09:55 -05:00
Kay Kurokawa eebf32362a add unit tests for wallet interface changes to balances 2017-01-26 11:41:39 -05:00
Kay Kurokawa 0e864f5df1 Update interface for wallet balance 2017-01-26 11:41:27 -05:00
Jack Robison 854dea797e Bump version: 0.8.0 → 0.8.1rc0 2017-01-25 17:49:52 -05:00
Alex Grintsvayg c88631d910 fix long lines 2017-01-25 17:00:21 -05:00
Alex Grintsvayg 897fba35df fix formatting, reenable pylint 2017-01-25 15:33:46 -05:00
Jack Robison 74b4e4508a Merge pull request #440 from lbryio/reflect-missing-blobs
Reflect missing blobs
2017-01-25 15:33:14 -05:00
Jack Robison 74a872d763 fix pylint errors 2017-01-25 13:10:38 -05:00
Jack Robison 3b65f5962c disable verbose twisted logs 2017-01-25 13:04:02 -05:00
Jack Robison 5fa2dfeca7 try reflecting all the blobs in a stream
previously only the sd blob was reflected, if the server indicated it
needed the blob then the rest of the stream would follow. this allowed
for many streams to be partially reflected, where for whatever reason
the connection was broken before the full upload was completed. this
meant that on a subsequent run, the client would falsely believe
reflector had the whole stream when it actually only had some portion
of it.

this solution isn’t ideal, I’m most of the way done with a better one,
but this can be deployed now.
2017-01-25 13:03:48 -05:00
Job Evers-Meltzer 7cac8dad7a remove header check 2017-01-25 01:12:56 -05:00
Job Evers‐Meltzer b80a1f66da Merge pull request #439 from lbryio/readme-dmg
Update readme to refer to lbry.dmg instead of lbry.pkg
2017-01-24 17:30:10 -05:00
alexliebowitz 20da97de6a Update readme to refer to lbry.dmg instead of lbry.pkg 2017-01-24 17:29:08 -05:00
Job Evers-Meltzer bb9d3bff74 Bump version: 0.8.0rc2 → 0.8.0 2017-01-24 11:56:36 -05:00
Job Evers‐Meltzer ec873129d5 Merge pull request #434 from lbryio/ensure-tmp-blobs-deleted
Bug fix: delete empty blobs
2017-01-24 11:55:13 -05:00
Job Evers‐Meltzer a90f413f94 Merge pull request #438 from lbryio/warn_large_wallet
Add warning for large wallet sizes
2017-01-24 11:54:53 -05:00
Alex Grin db1abd8cca Merge pull request #433 from lbryio/kauffj-patch-1
Clarify how to install, verify install, and make a change
2017-01-24 11:52:23 -05:00
Alex Grin 18a0b07181 Update INSTALL.md 2017-01-24 11:52:02 -05:00
Alex Grin 8fd7489d99 Update INSTALL.md 2017-01-24 11:51:12 -05:00
Kay Kurokawa ee84253b90 add warning for large wallet sizes 2017-01-23 18:55:20 -05:00
Job Evers‐Meltzer 06e45ee32a Merge pull request #410 from lbryio/reveal-open-refactor
Add open() API method and refactor/update reveal() API method (WIP)
2017-01-23 16:44:31 -05:00
Job Evers-Meltzer 7696f30009 add debug log to hash announcing 2017-01-21 22:48:16 -06:00
Job Evers-Meltzer c211d87164 Bug fix: delete empty blobs
When creating a CryptStream, the last blob is always empty. Previously, this
blob wouldn't be deleted and would instead just stick around in the blobfiles
directory.
2017-01-21 17:27:19 -06:00
Job Evers-Meltzer dc57b62157 add script to run tests 2017-01-21 16:04:38 -06:00
Jeremy Kauffman c71ac5b535 Ignore the folder we recommend to use for virtual-env in INSTALL.md 2017-01-21 11:19:44 -05:00
Jeremy Kauffman 5a467f55ba easier to read 2017-01-21 11:17:05 -05:00
Jeremy Kauffman 9a24bddead amend/clarify install instructions 2017-01-21 11:13:55 -05:00
Alex Liebowitz d3a63c38fc Fix pylint errors 2017-01-20 22:36:31 -05:00
Alex Liebowitz 08aa024499 Use inlineCallbacks for jsonrpc_open() and jsonrpc_reveal() 2017-01-20 22:29:27 -05:00
Alex Liebowitz 5db1a4d328 Refactor and update open() and reveal() RPC methods
- Both methods now take an SD hash instead of a path (more logical API
   and avoids potential security problems)
 - Moves the core logic into functions on a new module,
   lbry.core.file_utils
 - Adds reveal support for Windows
2017-01-20 22:29:27 -05:00
Alex Liebowitz e95adecd41 Rename jsonrpc_open_file to jsonrpc_open 2017-01-20 22:29:27 -05:00
Alex Liebowitz 8b68a752ed Add file_utils module 2017-01-20 22:29:27 -05:00
Alex Liebowitz 51a5510b38 Add open_file() RPC method 2017-01-20 22:29:27 -05:00
Job Evers-Meltzer 2c9c0da501 get_blob bug fix 2017-01-20 17:41:08 -06:00
Umpei Kay Kurokawa 60195ed6d4 Merge pull request #427 from lbryio/fix_new_address
Prevent excessive generation of new addresses
2017-01-20 17:14:21 -05:00
Job Evers‐Meltzer e37fbc4fdb Merge pull request #430 from lbryio/more-verified-time
missed another verified time check
2017-01-20 13:34:43 -06:00
Job Evers-Meltzer 3e774fc158 missed another verified time check 2017-01-20 18:47:53 +00:00
Alex Grintsvayg c62ee6bb0c fix loading settings from env 2017-01-20 13:38:49 -05:00
Alex Grintsvayg 3aea476014 ignore invalid config file keys 2017-01-20 12:39:59 -05:00
Job Evers-Meltzer 8c5f7a9849 fix missing arg in get_blob 2017-01-20 11:35:34 -06:00
Jeremy Kauffman 6bc3e8a3f4 fix typo in readme link 2017-01-20 12:27:19 -05:00
Job Evers‐Meltzer 6c571b5227 Blob verification fixes (#428)
* Move the blob verification to the actual Blob object
 * remove the check on verification time
 * remove get_blob_length from BlobManager

Removed because I'm not sure what checking verification time against ctime gets us, except some protection against an accidental modification of the blob.
2017-01-20 10:54:36 -06:00
Kay Kurokawa 8628b127f7 do not create new addresses in lbryum, get unused ones 2017-01-18 20:48:23 -05:00
Job Evers‐Meltzer 91d673a539 Merge pull request #423 from lbryio/check-wallet-on-status
Safegaurd against there not being a wallet on status call
2017-01-18 10:10:52 -06:00
Job Evers-Meltzer 851ab7ba28 Safegaurd against there not being a wallet on status call
Also switch status call to an inlineCallback
2017-01-18 09:36:01 -06:00
Job Evers-Meltzer 7bc1bc487f Bump version: 0.8.0rc1 → 0.8.0rc2 2017-01-17 22:08:34 -06:00
Job Evers‐Meltzer 05d50c1821 Merge pull request #420 from lbryio/fix-watch-resource
fix resource needed for watch
2017-01-17 22:07:06 -06:00
Job Evers‐Meltzer 40b6476cf7 Merge pull request #421 from lbryio/add-back-id
Add back id
2017-01-17 22:06:56 -06:00
Job Evers-Meltzer 13bfcb3561 fix missing id_ parameter to _render_error 2017-01-17 20:01:28 -06:00
Job Evers-Meltzer 1a850b0256 log exception on callback failure 2017-01-17 20:01:04 -06:00
Job Evers-Meltzer 78b0f923f4 fix resource needed for watch 2017-01-17 19:26:09 -06:00
Job Evers-Meltzer 1aa7981ab1 download blob script update 2017-01-17 17:11:14 -06:00
Alex Grin 17d9c0ab97 Merge pull request #417 from lbryio/refactor_settings
refactor conf.settings
2017-01-17 14:53:01 -05:00
Umpei Kay Kurokawa 8d1404bd1e Merge pull request #413 from lbryio/fix_fee_out
fix log message in publisher
2017-01-17 14:05:57 -05:00
Alex Grintsvayg 4634179a7e more long lines 2017-01-17 12:51:33 -05:00
Alex Grintsvayg 23872c77e9 fix long lines 2017-01-17 12:31:48 -05:00
Alex Grintsvayg cac8267e85 updates, more refactoring for greater clarity 2017-01-17 12:29:09 -05:00
Alex Grintsvayg 267c6cbaca refactor conf.settings 2017-01-16 22:32:54 -05:00
Kay Kurokawa 03b214a9d2 convert claim_out['fee'] to float 2017-01-16 15:36:37 -05:00
Job Evers-Meltzer 8a026071eb Bump version: 0.8.0rc0 → 0.8.0rc1 2017-01-16 12:01:35 -05:00
Job Evers‐Meltzer 9fb1cd0944 Merge pull request #412 from lbryio/revert-405-no-more-osx-on-travis
Revert "Remove osx from travis build matrix"
2017-01-16 10:53:56 -06:00
Job Evers‐Meltzer a338d1f0e1 Revert "Remove osx from travis build matrix" 2017-01-16 11:52:19 -05:00
Job Evers-Meltzer 8a338b37b2 Bump version: 0.7.12 → 0.8.0rc0 2017-01-13 14:15:01 -06:00
Job Evers‐Meltzer 2b2fa29db5 Merge pull request #395 from lbryio/fix_publish_output
Fix Daemon RPC command publish output
2017-01-13 14:02:23 -06:00
Job Evers‐Meltzer ddcfa7f97c Merge pull request #383 from lbryio/logging-improvements
Logging Improvements
2017-01-13 13:42:35 -06:00
Job Evers‐Meltzer ddf24de16d Merge pull request #407 from lbryio/handle-closed-api-requests-rebase
handle dropped api requests
2017-01-13 13:33:42 -06:00
Alex Grintsvayg 8277df1034 minor nits 2017-01-13 12:26:37 -05:00
Job Evers‐Meltzer 3bed768d10 Merge pull request #402 from lbryio/fix-needed-blobs
Bug Fix for ConnectionManager
2017-01-13 10:42:50 -06:00
Jack Robison e03bba0b62 handle dropped api requests 2017-01-13 10:41:59 -06:00
Job Evers-Meltzer e61260a263 merge with change stop to be an inlineCallback 2017-01-13 08:24:07 -06:00
Job Evers-Meltzer a9261b2032 change stop to be an inlineCallback 2017-01-13 08:24:07 -06:00
Job Evers-Meltzer 9366a6e561 bug fix: reset blob_requester between start/stop 2017-01-13 08:24:06 -06:00
Job Evers-Meltzer 488c04543d add TODO comment 2017-01-13 08:24:06 -06:00
Job Evers-Meltzer cb2bb6ee6b bug fix: properly stop ConnectionManager
It is possible (likely) that a manage call is in progress when
`stop` is called. When that happens, _manage will continue to
run, and schedule another call - and the manager won't actually stop,
and will likely cause an error as other components have been torn down.

This fix adds a deferred that gets created when a manage call starts
and is fired when its done.  At this points its safe to start the
stopping process.  Also add a check to not schedule another manage
call if we're stopped

This fixes https://app.asana.com/0/142330900434470/239832897034382
2017-01-13 08:24:06 -06:00
Job Evers-Meltzer 0bb62515a8 reorder functions to match call order 2017-01-13 08:24:06 -06:00
Job Evers-Meltzer f535d96929 switch stop to inlineCallback 2017-01-13 08:24:06 -06:00
Job Evers-Meltzer 15c5075d38 refactor closing peers 2017-01-13 08:24:06 -06:00
Job Evers-Meltzer 2449604844 fixup: bug fix for stopping downloads 2017-01-13 08:24:06 -06:00
Job Evers-Meltzer 6035a84675 bug fix for stopping downloads
deferredlist returns a tuple of (success, result) and the previous
code was looking at the entire tuple not the success values.
2017-01-13 08:24:05 -06:00
Job Evers-Meltzer 1151019186 Improve downloader error handling
Instead of having an error cause a timeout, send
the error directly to the callback
2017-01-13 08:24:05 -06:00
Job Evers-Meltzer 8075ced1af bug fix for file_seed
cannot return a deferred in inlineCallbacks, it must be a value
2017-01-13 08:23:52 -06:00
Job Evers-Meltzer 1b2c46e224 make _download_manager private 2017-01-13 08:19:23 -06:00
Alex Grin 7f6e9fff30 Merge pull request #403 from lbryio/fix-result
bug fix: jsonrpc methods no longer have a 'result' key
2017-01-12 12:19:27 -05:00
Job Evers‐Meltzer bb1b7b66f7 Merge pull request #405 from lbryio/no-more-osx-on-travis
Remove osx from travis build matrix
2017-01-12 11:14:54 -06:00
Job Evers-Meltzer deae84a563 remove osx from travis build matrix 2017-01-12 10:45:31 -06:00
Job Evers-Meltzer 20f113d3da bug fix: jsonrpc methods no longer have a 'result' key 2017-01-12 09:55:38 -06:00
Job Evers‐Meltzer 61702ba1e4 Merge pull request #301 from lbryio/query-available-blobs
Query available blobs and Upload to Lighthouse Scripts
2017-01-12 08:42:08 -06:00
Alex Grintsvayg d100a24a29 Merge branch 'api_docs'
* api_docs:
  api doc generator
2017-01-11 17:30:26 -05:00
Alex Grintsvayg d06211015d api doc generator 2017-01-11 17:30:16 -05:00
Alex Grintsvayg 93ff4f63b5 Merge branch 'rename_api_methods'
* rename_api_methods:
  updates based on job's comments
  appveyor is a line-length nazi
  detect authentication error and warn accordingly
  remove authjsonrpc subhandlers. we dont use them
  rpc works with authentication
  restore one deprecated method (for now)
  incorporate job's changes
  remove return codes that are not being used
  make sure deprecated methods still return results
  better cli help
  remove extraneous ellipses in messages
  rename api endpoints
2017-01-11 16:36:59 -05:00
Alex Grintsvayg 86aa8e1de0 updates based on job's comments 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 495a48add1 appveyor is a line-length nazi 2017-01-11 15:39:28 -05:00
Alex Grintsvayg a6457d2c0a detect authentication error and warn accordingly 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 5c429e15bd remove authjsonrpc subhandlers. we dont use them 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 05725e0dc9 rpc works with authentication 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 53acb2d9cd restore one deprecated method (for now) 2017-01-11 15:39:28 -05:00
Alex Grintsvayg f050337dbc incorporate job's changes 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 6d7b09a3aa remove return codes that are not being used 2017-01-11 15:39:28 -05:00
Alex Grintsvayg dd87aa3ce6 make sure deprecated methods still return results 2017-01-11 15:39:28 -05:00
Alex Grintsvayg b602b09ca3 better cli help 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 628defa891 remove extraneous ellipses in messages 2017-01-11 15:39:28 -05:00
Alex Grintsvayg 962edb8e4c rename api endpoints 2017-01-11 15:39:28 -05:00
Job Evers-Meltzer 5dbf9034ec Add scripts related to querying / downloading sd blobs 2017-01-11 12:05:34 -06:00
Umpei Kay Kurokawa a67aab0797 Merge pull request #393 from lbryio/fix_github_version_request
Fix obtaining version from github
2017-01-10 20:42:08 -05:00
Kay Kurokawa 52500bfbf5 use latest release for getting lbryum version, add a timeout when requesting lbrynet/lbryum version 2017-01-10 12:55:47 -05:00
Alex Grin f4790a46ee Update README.md 2017-01-09 18:07:39 -05:00
Jeremy Kauffman fd5b4f403a improvements to readme and install instructions (#396)
* improvements to readme and install instructions

* Update README.md

* Update INSTALL.md
2017-01-09 18:01:50 -05:00
Job Evers‐Meltzer c393bd97e8 Merge pull request #388 from lbryio/hash-announce
ensure hash_announcer exists before announcing
2017-01-09 15:23:53 -06:00
Job Evers‐Meltzer d32bb9270a Merge pull request #376 from lbryio/add-id
add id field to jsonrpc response
2017-01-09 15:23:24 -06:00
Kay Kurokawa 9e7b69f4a0 fix publish output 2017-01-09 14:35:29 -05:00
Jeremy Kauffman 3c60b35e2e Merge pull request #384 from lbryio/ubuntu_install_paths
fix linux packaging to work with new install paths
2017-01-06 14:58:29 -05:00
Job Evers‐Meltzer 69c0da695b Merge pull request #387 from lbryio/wallet-whitespace
cleanup whitespace for increased readability
2017-01-06 08:20:06 -06:00
Job Evers-Meltzer 68887d145f cleanup whitespace for increased readability 2017-01-06 08:04:18 -06:00
Job Evers-Meltzer f37808859f ensure hash_announcer exists before announcing 2017-01-06 08:00:22 -06:00
Jeremy Kauffman 14c835b41e fix linux packaging to work with new install paths 2017-01-04 09:51:15 -05:00
Job Evers abc122f77b Don't send log messages to loggly in dev mode 2017-01-03 22:17:44 -06:00
Job Evers 0ababb7f3b Route twisted logging messages to the python stdlib logger 2017-01-03 22:16:57 -06:00
Job Evers-Meltzer 30af8a0e20 Bump version: 0.7.12rc4 → 0.7.12 2017-01-03 21:09:25 -06:00
Job Evers‐Meltzer 389fc67d51 Merge pull request #381 from lbryio/fix-watch
Fix watch
2017-01-03 21:08:33 -06:00
Job Evers c2fc15160f Fix watch
I changed the _download_name return type without realizing that
it was also used by the watch endpoint.  This switches the endpoint
to go through get so that watches can be tracked just like downloads
2017-01-03 18:08:31 -06:00
Job Evers-Meltzer 40d4f5a292 Bump version: 0.7.12rc3 → 0.7.12rc4 2017-01-03 15:29:20 -06:00
Job Evers‐Meltzer 38e6b4e6c6 Merge pull request #379 from lbryio/open-ui-asap
open the ui as soon as possible
2017-01-03 15:28:50 -06:00
Jack 3541625825 move launch into own function 2017-01-03 15:51:42 -05:00
Job Evers‐Meltzer f8f83fccf1 Merge pull request #375 from lbryio/fix-stop-errors
Fix stop errors
2017-01-03 14:13:24 -06:00
Job Evers‐Meltzer b3175af16f Merge pull request #374 from lbryio/allow-jsonrpc-v2
upgrade txJson-RCP to 0.5
2017-01-03 14:11:48 -06:00
Jack 5f8a06443c open the ui as soon as UIManager finishes setting itself up 2017-01-03 11:44:29 -05:00
Job Evers 4e0194d576 add id field to jsonrpc response 2017-01-02 20:12:37 -06:00
Job Evers 16fc416d88 more informative name error 2017-01-02 19:47:34 -06:00
Job Evers 6faee910de Raise exception if lbry_file can't be found 2017-01-02 19:46:40 -06:00
Job Evers f7308beee9 upgrade txJson-RCP to 0.5
The older version we were running didn't support jsonrpc v2 for
error codes, which both the command line client and ui client
were expecting.
2017-01-02 19:15:25 -06:00
Job Evers-Meltzer 5233dc303e Bump version: 0.7.12rc2 → 0.7.12rc3 2017-01-02 17:18:46 -06:00
Job Evers‐Meltzer 3d0913ca94 Merge pull request #373 from lbryio/bundled-version
Set ui_version for Bundled UI
2017-01-02 17:17:31 -06:00
Job Evers‐Meltzer c37bb0d1ed Merge pull request #372 from lbryio/fix-stop
fix the stop script
2017-01-02 16:50:34 -06:00
Job Evers-Meltzer a4a9a2b0b0 fix the stop script 2017-01-02 16:40:50 -06:00
Job Evers-Meltzer 9e320921a9 Bump version: 0.7.12rc1 → 0.7.12rc2 2017-01-02 16:12:35 -06:00
Jack Robison 21602dc5a1 Merge pull request #368 from lbryio/bug-report
bug reports sent to slack
2017-01-02 16:24:57 -05:00
Jack 2cf08fda2d use urllib for loggly url formatting 2017-01-02 16:09:28 -05:00
Jack c2c7d2b327 send bug reports to slack 2017-01-02 14:52:24 -05:00
Job Evers‐Meltzer 585f802a91 Merge pull request #370 from lbryio/fail-wallet
Stop eating wallet connection error
2017-01-02 12:02:35 -06:00
Job Evers-Meltzer 26cf04e0c1 refactor requirements parsing 2017-01-02 12:01:55 -06:00
Job Evers-Meltzer 3bb8f4ac21 set ui_version for bundled ui 2017-01-02 12:01:55 -06:00
Job Evers‐Meltzer 2afc4c25a2 Merge pull request #371 from lbryio/fix-settings
Fix settings
2017-01-01 11:24:25 -06:00
Job Evers-Meltzer a5c25ff887 Stop eating wallet connection error 2016-12-30 22:34:08 -06:00
Job Evers-Meltzer e823bb2012 Bug fix in auth.server
Fixes a mistake in fa8190356a,
where an usage of _log_and_render_error was overlooked
2016-12-30 16:48:44 -06:00
Job Evers-Meltzer eec25791a5 fix lbryid in settings dict
The lbryid is stored as raw bytes, need to hexlify
it before sending it over to the client
2016-12-30 16:47:50 -06:00
Job Evers-Meltzer 18a47a048a Bump version: 0.7.12rc0 → 0.7.12rc1 2016-12-30 15:42:13 -06:00
Job Evers‐Meltzer 6936dbd588 Merge pull request #369 from lbryio/track-errors
track errors in analytics
2016-12-30 15:39:41 -06:00
Job Evers‐Meltzer 621530b3b8 Merge pull request #367 from lbryio/analytics-on-get
Analytics on get
2016-12-30 15:39:06 -06:00
Job Evers-Meltzer 123fda1844 track errors in analytics 2016-12-30 15:09:34 -06:00
Job Evers-Meltzer 1299c23bcc fix cli 2016-12-30 15:07:24 -06:00
Job Evers-Meltzer 1747f53276 Remove _set_events
Function was removed in 101985ca11,
but wasn't removed from setup()
2016-12-30 14:42:03 -06:00
Job Evers-Meltzer 669a567545 add id to link download events 2016-12-30 13:35:16 -06:00
Job Evers-Meltzer 6bdc9069c9 on get, first check if we already have the name 2016-12-30 13:35:16 -06:00
Job Evers-Meltzer ad1e2650e3 add events for downloading stages 2016-12-30 13:31:20 -06:00
Job Evers-Meltzer 1bf1b7624c more inlineCallback cleanup 2016-12-30 13:28:38 -06:00
Job Evers-Meltzer 0b53fde352 refactor connectionmanager to use inlineCallbacks 2016-12-30 13:28:34 -06:00
Job Evers-Meltzer 4eb10b56c1 small changes 2016-12-30 13:27:17 -06:00
Job Evers-Meltzer 227323b7a4 convert more to inlineCallbacks 2016-12-30 13:27:16 -06:00
Job Evers-Meltzer 1256beea96 remove blob_requester from DownloadManager 2016-12-30 13:27:16 -06:00
Job Evers-Meltzer 01cc4f28e0 Refactor jsonrpc_get
Convert some of the calls to inlineCallbacks, which
allowed the code to be cleaned up and made more clear
2016-12-30 13:27:12 -06:00
Job Evers-Meltzer 689ac431d3 bug fix in log.fail()
The value of the callback needs to be returned.  In particular,
this allows the error to be passed along.
2016-12-30 13:22:06 -06:00
Job Evers-Meltzer ccfb1f3921 refactor: name private/public variables 2016-12-30 13:22:06 -06:00
Job Evers‐Meltzer cc008766ec Merge pull request #361 from lbryio/analytics-on-startup
Analytics on startup
2016-12-30 13:21:14 -06:00
Job Evers-Meltzer 4b1530b554 fix typo in osx build script 2016-12-30 13:05:19 -06:00
Job Evers-Meltzer 5eecae8889 record server startup event as part of start_server 2016-12-30 13:05:19 -06:00
Job Evers-Meltzer 5aa10031bc Move log_support import earlier
Since log_support configures the logging class, it needs
to be early in the initialization process.
2016-12-30 13:05:19 -06:00
Job Evers-Meltzer 52010ade41 add docstring 2016-12-30 13:05:19 -06:00
Job Evers-Meltzer 101985ca11 Remove analytics setup code from daemon
Now that lbryid and session_id are available at startup
we don't need to reconfigure the analytics manager
inside the daemon
2016-12-30 13:05:19 -06:00
Job Evers-Meltzer 8f61fb3fc4 Initialize settings, along with lbryid and session_id on startup 2016-12-30 13:05:19 -06:00
Job Evers-Meltzer 39b2e44492 Replace settings with conf.settings
In the next commit I change conf.settings to be initialized
at runtime instead of load time and so any import of
`from lbrynet.conf import settings` will be and stay None if
it happens before the initialization.
2016-12-30 13:05:18 -06:00
Job Evers-Meltzer 26bc6d05a0 Consistently use lbryid over lbry_id variable name 2016-12-30 12:40:36 -06:00
Job Evers‐Meltzer 47f71fd363 Merge pull request #360 from lbryio/store-offer
Only store the offer explicitly when its sent
2016-12-30 12:15:06 -06:00
Job Evers‐Meltzer 4574e3317c Merge pull request #363 from lbryio/wallet-storage
Wallet storage
2016-12-30 11:41:38 -06:00
Job Evers‐Meltzer b59c098d58 Merge pull request #364 from lbryio/fix_serverprotocol
Fix bug in server protocol
2016-12-30 11:28:08 -06:00
Job Evers-Meltzer 22ff87b885 fix tests for new query_handler_factory format 2016-12-30 09:07:24 -06:00
Job Evers‐Meltzer 42edf49468 Merge pull request #365 from lbryio/fix-set-settings-typo
Fix typo in set_settings API method
2016-12-30 08:22:14 -06:00
Alex Liebowitz da0b4e9a1a Fix typo in set_settings API method 2016-12-30 05:00:28 -05:00
Kay Kurokawa 7d5dd1ff51 fix bug in ServerProtocol 2016-12-28 14:34:38 -05:00
Umpei Kay Kurokawa d1ccac1b04 Merge pull request #362 from lbryio/fix_getvalueforname_bug
Fixing get_value_for_name bug
2016-12-28 13:35:48 -05:00
Job Evers‐Meltzer a289886365 Merge pull request #350 from lbryio/reflector-test
scripts to test reflector
2016-12-28 11:29:14 -06:00
Job Evers-Meltzer 2c7fc3fb46 switch startup sequence to use inlineCallbacks
This cleans up the process, makes it more readable,
and better surfaces errors.
2016-12-28 11:21:10 -06:00
Job Evers-Meltzer fc24d9b5aa pull out metadata storage from the wallet 2016-12-28 11:18:55 -06:00
Kay Kurokawa da366495b0 fixing get_value_for_name bug 2016-12-28 10:37:53 -05:00
Umpei Kay Kurokawa 552ed77882 Merge pull request #358 from lbryio/fix_wallet_init_3
Fix wallet initialization
2016-12-22 12:12:27 -05:00
Jack Robison c53e0baca6 Merge pull request #357 from lbryio/no-brew-error
Fix error on brew when package is already installed
2016-12-21 22:43:39 -05:00
Jack Robison aa3353ae49 Merge pull request #345 from lbryio/better-dht-shutdown
Better dht shutdown.
2016-12-21 17:15:17 -05:00
Job Evers‐Meltzer 521ad57b61 Merge pull request #353 from lbryio/update-six
allow more recent versions of six
2016-12-21 12:20:11 -08:00
Job Evers-Meltzer f62456bb68 allow more recent versions of six 2016-12-20 18:06:59 -08:00
Job Evers-Meltzer f7864e8cb7 Only store the offer explicitly when its sent
Ran into a situation where I was recieving an offer response without
something in protocol_offers. I think this was because
get_and_set_rate doesn't always populate that dictionary.

Worried that there still might be issues if the connection is
unreliable: if a response comes in twice it will error or
if a second offer is made without a response to the first
there will be an error.
2016-12-20 10:37:42 -08:00
Kay Kurokawa 1690e71ad1 setup Commands class on run, instead of on initialization 2016-12-19 19:15:59 -05:00
Kay Kurokawa 7d02e7d370 changing wallet initialization order 2016-12-19 19:15:50 -05:00
Umpei Kay Kurokawa 7404b127bf Merge pull request #354 from lbryio/fix_publisher_bug
Fixing bug in publisher
2016-12-19 19:12:00 -05:00
Kay Kurokawa 10810226ff Fixing bug in publisher 2016-12-19 18:41:35 -05:00
Job Evers-Meltzer a2497e0854 Fix error on brew when package is already installed 2016-12-19 15:39:06 -08:00
Jack Robison c0f22decd7 Merge pull request #347 from lbryio/log-error-on-lbry-files
Log failure when attempting to get_lbry_files
2016-12-19 16:22:37 -05:00
Jack Robison 3da82ce895 Merge pull request #348 from lbryio/remove-old-settings
remove unqlite requirement and old Settings.py
2016-12-19 14:38:31 -05:00
Jack ba90fb3349 fix os x build 2016-12-19 14:10:37 -05:00
Jack 618d0ebd5b newline 2016-12-19 13:40:33 -05:00
Jack f301741807 re-use lbryid 2016-12-19 13:27:45 -05:00
Jack 0044d6cf75 BlobAvailabilityTracker.sample bug, remove whitespace 2016-12-17 20:05:27 -05:00
Jack 5a76a03cb6 remove unqlite requirement
-conditionally import PTCWallet (needed for testing, requires unqlite)

-install unqlite on travis and appveyor
2016-12-17 20:03:35 -05:00
Jack 09846413bd remove Settings.py
remove old settings class, which was only used for the lbryid and
required unqlite.

this allows the daemon startup to be simplified, it previously relied
on Settings in a few places. lbryid had been stored in the old
settings, this initializes it at startup.
2016-12-17 20:01:06 -05:00
Job Evers-Meltzer a8cb255d24 Bump version: 0.7.11rc0 → 0.7.12rc0 2016-12-16 21:45:02 -06:00
Job Evers-Meltzer 6172924021 Bump version: 0.7.8 → 0.7.11rc0 2016-12-16 20:58:24 -06:00
Job Evers‐Meltzer 7bb570c051 Merge pull request #352 from lbryio/fixup-logging
change error log on resolve-name to info
2016-12-16 20:00:37 -06:00
Job Evers-Meltzer d5d0d0e31c change error log on resolve-name to info 2016-12-16 19:51:10 -06:00
Job Evers‐Meltzer 83e797d7f6 Merge pull request #351 from lbryio/ssl-handshake
Fix SSL handshake issue
2016-12-16 19:25:00 -06:00
Job Evers-Meltzer 6c2b3d9ed6 fix ssl handshake error 2016-12-16 19:13:23 -06:00
Job Evers-Meltzer c6f4c17597 upgrade to py2app v0.10 2016-12-16 19:13:23 -06:00
Job Evers-Meltzer d19afd856b allow skipping of some time consuming parts of the build 2016-12-16 19:00:25 -06:00
Job Evers-Meltzer 29e59e783a scripts to test reflector 2016-12-16 15:58:20 -06:00
Job Evers‐Meltzer 107860ea19 Merge pull request #349 from lbryio/force-pyobjc-3.1.1
Pin pyobjc to version 3.1.1
2016-12-16 13:37:08 -06:00
Job Evers-Meltzer a9bfd8d6ae Pin pyobjc to version 3.1.1 2016-12-16 13:02:52 -06:00
Job Evers‐Meltzer 9ee95d57fc Merge pull request #342 from lbryio/update-https
Update https
2016-12-16 11:28:42 -06:00
Job Evers-Meltzer 4a8a8b8d33 sync setup.py with requirements.txt 2016-12-16 11:04:47 -06:00
Job Evers-Meltzer c825d88dbf OSX is having https issues, hopefully this fixes it
When I build locally, the resulting app doesn't have https issues. Looking
over the build script output, I notice that its copying

copying /usr/local/opt/openssl/lib/libssl.1.0.0.dylib -> /Users/jobevers/tmp/lbry/packaging/osx/lbry-osx-app/dist/LBRY.app/Contents/Frameworks

such a line isn't present when travis builds.  This version of libssl
is installed by homebrew. Fingers crossed.
2016-12-16 10:35:13 -06:00
Job Evers-Meltzer 15ade508aa add ndg-httpsclient as a requirement 2016-12-16 10:35:13 -06:00
Job Evers-Meltzer 53df361d23 Log failure when attempting to get_lbry_files
I had an old version of lbryum installed and so no
names were coming back. But the daemon was failing
silently. This at least logs the error.
2016-12-16 00:32:06 -06:00
Job Evers-Meltzer 8fe15f507b Better dht shutdown.
The old code relied on the timing of DelayedCalls
to see what had been called or not, but unfortunately
we don't have a real-time OS so the timing on DelayedCalls
can only be approximate. Changed to explicitly keep
track of which calls had been made.

This simplifies the shutdown logic drastically, and I believe
we can take out the reactor.iterate() now
2016-12-15 23:44:35 -06:00
Job Evers‐Meltzer 3dde7af576 Merge pull request #339 from lbryio/all-prs
All The PRs
2016-12-15 19:42:35 -06:00
Job Evers-Meltzer 7af6e9e0dd Improve BlobAvailabilityTracker performance
For daemons with a lot of blobs, getting mean availabity
will be slow. Samples the blobs in an attempt at getting
better performance.
2016-12-15 19:03:37 -06:00
Job Evers-Meltzer 56d394fb5f Add timing to hash announcements
This could potentially be a performance issue
on reflector or any daemon with a large number of blobs.
2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 3585d861ff Add TRACE level logging 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer f181af8547 Cleanup code in BlobManager 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 84baa5e065 dht: refactor _msgTimeout 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 781b915717 remove unused check_consistency function 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer fa29c92760 Add setting to disable reuploading to reflector 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer fec917b9c1 rename platform to system_info 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 4f3b5cd802 Better logging on DHT errors
Timeout errors are common on the dht so log those
at debug, but other errors need to (potentially) recieve
more attention
2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 250831a86a remove unused distance function from routingtable 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 323bccb0ae another distance optimization 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 0084d4684f Add distance optimization 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 740fad5cbe Refactor dht.node.iterativeFind
Move nested functions into a helper class.
Add new, smaller functions to increase readability
2016-12-15 18:58:15 -06:00
Job Evers-Meltzer fa8190356a replace log_support.failure calls 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 4ca33505b5 logging: add kwargs to fail() call 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 20b6b22334 Add callback functionality to log.fail
This enables the common pattern of being able to log a failure and
then do something.
2016-12-15 18:58:15 -06:00
Job Evers-Meltzer b8c91c61bb Make a new logger with a fail method that is useful for errbacks.
Extracting useful tracebacks and line numbers from failures
withing twisted's deferred can be a pain. Hopefully this is a step
in the right direction.
2016-12-15 18:58:15 -06:00
Job Evers-Meltzer c30ea04959 Remove commented code
This is the result of running eradicate (https://github.com/myint/eradicate)
on the code and double-checking the changes.
2016-12-15 18:58:06 -06:00
Umpei Kay Kurokawa 27a09d4a4f Merge pull request #331 from lbryio/fix_wallet_threading_issue
Fix wallet concurrency issue
2016-12-15 10:57:11 -05:00
Kay Kurokawa 23bf43f40c fixing bug where broadcast_transaction function would not return a deferred 2016-12-15 10:28:49 -05:00
Kay Kurokawa 6f60b8b827 work on resolving threading issues with lbryum in Wallet.py 2016-12-15 10:28:49 -05:00
Kay Kurokawa 965f8f1238 removing unused get_transaction function from daemon and wallet 2016-12-15 10:28:49 -05:00
Jack Robison ab78b7f5fa Merge pull request #338 from lbryio/zipsafe-or-bust
Fix the Build
2016-12-14 22:04:58 -05:00
Job Evers-Meltzer 083092ad5e pylint: add check for mixed-indentation 2016-12-14 19:22:00 -06:00
Job Evers-Meltzer 7bce37a720 pylint: add check for missing-final-newline 2016-12-14 19:21:59 -06:00
Job Evers-Meltzer 14ac2bac39 pylint: fix trailing-whitespace 2016-12-14 19:21:59 -06:00
Jeremy Kauffman 32fa2460ff sort the help functions 2016-12-14 19:21:54 -06:00
Job Evers-Meltzer f796f701f2 Immediately announce completed blobs
It doesn't seem necessary to wait up to 60 seconds for a new blob to
be announced to the dht. Immediately announce it and schedule the next
announce time as usual.
2016-12-14 19:05:24 -06:00
Kay Kurokawa fe4ea9b33a use assertFailure, move around assertEqual arguments to standard locations 2016-12-14 19:05:24 -06:00
Kay Kurokawa c278972f26 adding unit tests for Wallet.py 2016-12-14 19:05:23 -06:00
Kay Kurokawa bce41ddab4 Do not return 'success' and 'reason' as outputs in claim commands
We throw an Exception instead
2016-12-14 19:05:05 -06:00
Job Evers-Meltzer 666a62ae3d try setting zip_safe=False 2016-12-14 16:28:38 -06:00
Jack 5bf20096b2 fix wallet_balance when checking fee amount 2016-12-12 14:32:33 -05:00
Alex Grintsvayg 52099c7da7 update gitignore 2016-12-08 18:35:57 -05:00
Job Evers‐Meltzer 9619b8dfca Merge pull request #299 from lbryio/long-lines
Wrap Long lines
2016-12-07 11:41:21 -05:00
Job Evers-Meltzer e8faadc324 fix more long lines 2016-12-07 10:10:46 -05:00
Job Evers-Meltzer 667d5ac622 add emacs tmp files to pylint blacklist 2016-12-07 09:38:38 -05:00
Job Evers-Meltzer afba045c29 add line-to-long check 2016-12-07 09:38:37 -05:00
Job Evers-Meltzer b64fa51567 fixup long lines 2016-12-07 09:38:34 -05:00
Job Evers-Meltzer fe3e25982c don't lint unused files 2016-12-07 09:35:16 -05:00
Jack Robison 9ad55e93b9 Merge pull request #315 from kaykurokawa/fix_claim_bug
Fix bug to exclude support in get_my_claim
2016-12-07 09:25:44 -05:00
Jack Robison 9e7e168175 Merge pull request #319 from lbryio/reenable-startup-status-codes
Reenable startup status
2016-12-06 17:00:03 -05:00
Jack Robison f84df50d7f Merge pull request #317 from lbryio/remove-lbrynet-console
Remove lbrynet-console
2016-12-06 16:59:52 -05:00
Jack f0ed4e5513 move listen to DaemonServer startup 2016-12-06 16:34:50 -05:00
Jack Robison 0af445cbbc Merge pull request #318 from lbryio/build-osx
put back accidental deleted build
2016-12-06 16:21:05 -05:00
Job Evers-Meltzer 99fe28366f put back accidental deleted build 2016-12-06 15:16:41 -05:00
alexliebowitz 39005ffc14 Merge pull request #310 from lbryio/allow-cors-config
allow Access-Control-Allow-Origin to be configurable
2016-12-06 15:11:38 -05:00
Jack 239e5c2126 remove unused imports in LBRYApp.py 2016-12-06 13:01:36 -05:00
Jack Robison 28e88d89a5 Merge pull request #313 from lbryio/config-bittrex
Add bittrex to conf
2016-12-06 12:41:45 -05:00
Jack Robison c85b795fc6 Merge pull request #316 from lbryio/rm-old-install
remove old installation script
2016-12-06 12:40:00 -05:00
Jack 979f8efa79 remove unused lbrynet-console 2016-12-06 12:35:27 -05:00
Jack Robison 4c4bccfe73 Merge pull request #312 from lbryio/local-ui-conf
local_ui_path needs to be a str
2016-12-06 12:23:45 -05:00
Jack 8cb0f99e30 fix pylint errors 2016-12-06 12:22:43 -05:00
Job Evers-Meltzer 44554b63f4 remove old installation script 2016-12-06 12:15:56 -05:00
Umpei Kay Kurokawa 6076861cd0 Merge pull request #304 from kaykurokawa/remove_lbrycrd
Remove lbrycrdd
2016-12-06 12:06:09 -05:00
Job Evers‐Meltzer 8b427a92ac Merge pull request #311 from lbryio/fix-server-port
port needs to be an int
2016-12-06 09:48:08 -05:00
Job Evers‐Meltzer 6c4736d1bd Merge pull request #314 from lbryio/proper-getattr
__getattr__ should call __getattribute__
2016-12-06 09:47:04 -05:00
Kay Kurokawa 9c50cfdcdb fix bug in get_my_claim 2016-12-05 17:17:17 -05:00
Job Evers-Meltzer 8f80e8a3da add bittrex to conf 2016-12-05 17:14:25 -05:00
Job Evers-Meltzer cd44ae02a3 __getattr__ should call __getattribute__ 2016-12-05 17:06:45 -05:00
Job Evers-Meltzer 848f1f025c local_ui_path needs to be a str 2016-12-05 16:29:18 -05:00
Job Evers-Meltzer d2cc8355dc port needs to be an int 2016-12-05 16:20:16 -05:00
Job Evers ee00d1984a allow Access-Control-Allow-Origin to be configurable 2016-12-05 15:34:48 -05:00
Jack Robison 2fb71610a4 Merge pull request #294 from lbryio/approximate-stream-price
Add 'size' parameter to get_est_cost
2016-12-05 12:22:26 -05:00
Jack 948be0f040 clean up tests 2016-12-05 11:48:44 -05:00
Jack b9bb337a07 add tests for daemon.get_est_cost 2016-12-05 11:48:44 -05:00
Jack d93f86030c clean up get_est_cost functions 2016-12-05 11:48:44 -05:00
Jack d9e1e88c05 refactor get_est_cost
-add size parameter, if provided the sd blob won’t be downloaded (can
be used with size information from lighthouse)

-check if blob was already downloaded before trying to download it again
2016-12-05 11:48:43 -05:00
Job Evers-Meltzer f579c8dbb8 Bump version: 0.7.7 → 0.7.8 2016-12-02 20:05:08 -06:00
Job Evers-Meltzer 0005ba0715 pin dmgbuild version
When run with v1.2.0 the build fails with:

dmgbuild -s dmg_settings.py LBRY lbrynet.0.7.7.dmg
Traceback (most recent call last):
  File "[..]/2.7/bin/dmgbuild", line 36, in <module>
    dmgbuild.build_dmg(args.filename, args.volume_name, args.settings, defines=defines, lookForHiDPI=args.lookForHiDPI)
  File "[..]/2.7/lib/python2.7/site-packages/dmgbuild/core.py", line 440, in build_dmg
    d['.']['pBBk'] = background_bmk
  File "[..]/2.7/lib/python2.7/site-packages/ds_store/store.py", line 1183, in __setitem__
    entry_type = value[0]
  File "[..]/2.7/lib/python2.7/site-packages/mac_alias/bookmark.py", line 365, in __getitem__
    raise KeyError('Key not found')
KeyError: u'Key not found'
2016-12-02 16:31:45 -06:00
Jack Robison 13d74d1f76 Merge pull request #297 from lbryio/better-name-error
Provide better feedback on why a name is invalid
2016-12-02 12:08:58 -05:00
Job Evers-Meltzer ddf678af8a Bump version: 0.7.7rc0 → 0.7.7 2016-12-02 09:47:27 -06:00
Job Evers‐Meltzer 2c6d6d1602 Merge pull request #303 from lbryio/settings-api-fixes
Fixes to get_settings() and set_settings()
2016-12-02 09:41:22 -06:00
Alex Liebowitz 7d0e9f6ab9 Tweak logic in ApplicationSettings
Use self.environ instead of ENVIRONMENT to avoid global reference
2016-12-01 23:51:58 -05:00
Kay Kurokawa 200a22c93b Remove support for lbrycrdd wallet and related code 2016-12-01 16:45:48 -05:00
Alex Grin 31a90e7f60 Merge pull request #302 from lbryio/fix-analytics-error
Attempt to fix analytics badstatusline error
2016-12-01 16:25:05 -05:00
Job Evers-Meltzer 2bf098b5a3 add tests for settings change 2016-12-01 15:22:53 -05:00
Alex Liebowitz 458561fe6f In AdjustableSettings, initialize all keys on init
Settings.update() method expects keys to already be present, so load
them all up front
2016-12-01 15:22:52 -05:00
Alex Liebowitz 157d6dca93 Save settings in conf.py 2016-12-01 15:22:52 -05:00
Alex Liebowitz 7b1b7fbf13 Call conf.settings.get_dict() in jsonrpc_get_daemon_settings() 2016-12-01 15:22:51 -05:00
Alex Liebowitz 876cdce51f Add Config.get_dict()
We don't want the version from DefaultSettings, so use super() to get
the standard functionality from conf.Settings.
2016-12-01 15:22:51 -05:00
Alex Liebowitz 205a10bc8e Refactor how adjustable settings are pulled out of config.settings
- Factor out this functionality into separate method
 - Change the set_settings() JSON-RPC method to use this (before, it was
   just returning all fields, which doesn't work anymore after the
   settings refactor)
2016-12-01 15:22:51 -05:00
Alex Liebowitz ef8d1cfdc2 Rename var in Settings.update() 2016-12-01 15:22:51 -05:00
Alex Liebowitz 879c01ba00 Add separate list of excluded fields to AdjustableSettings
Adds "environ" field. Also renames __excluded to _excluded so it can
be overridden by child classes.
2016-12-01 15:22:44 -05:00
Alex Liebowitz 8bb7517f32 Merge pull request #300 from kaykurokawa/fix_claim_defer
Missing the return of a defer.fail when claim fails, changing to raise Exception
2016-12-01 13:49:29 -05:00
Job Evers-Meltzer 70d9b3ed74 Attempt to fix analytics badstatusline error
We've started seeing a lot of:
ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Hopefully this fixes the problem. Its hard to test, so I'll
have to check the logs after releasing.
2016-12-01 12:13:12 -06:00
Job Evers-Meltzer f39a21cf7b Provide better feedback on why a name is invalid 2016-12-01 11:02:37 -06:00
Umpei Kay Kurokawa 67b169c5a1 Merge pull request #300 from kaykurokawa/fix_claim_defer
Missing the return of a defer.fail when claim fails, changing to raise Exception
2016-11-30 23:08:48 -05:00
Kay Kurokawa ff10546152 Missing the return of a defer.fail when claim fails, changing to raise Exception 2016-11-30 22:57:23 -05:00
Umpei Kay Kurokawa bd6fe4f986 Merge pull request #290 from kaykurokawa/fix_logging2
Converting some log.info to log.debug in Wallet
2016-11-30 19:22:16 -05:00
Kay Kurokawa 360f4e50a0 converting some log.info to log.debug in Wallet 2016-11-30 16:37:28 -05:00
Job Evers‐Meltzer f11f486f5a Merge pull request #292 from lbryio/allow-rc
allow for rc in version values
2016-11-30 14:16:17 -06:00
Jack Robison fe53cc97d9 Merge pull request #285 from lbryio/more-flexible-origin-check
Allow 0.0.0.0 for api interface
2016-11-30 14:04:52 -05:00
Jack Robison baf6ff3f50 Merge pull request #296 from lbryio/dont-allow-zero-bid
don't allow 0 bid in publish
2016-11-30 13:59:31 -05:00
Jack 051e63d76b don't allow 0 bid 2016-11-29 20:17:06 -05:00
Job Evers-Meltzer 8cdda7d845 Bump version: 0.7.6 → 0.7.7rc0 2016-11-28 23:33:57 -06:00
Job Evers-Meltzer 6a8789050c Allow 0.0.0.0 for api interface
For a host to be able to access the daemon running inside a docker
container the damon needs to be listening to 0.0.0.0

- move the API_INTERFACE setting to the adjustablesettings
- check the port matches as well as the interface
2016-11-28 19:19:05 -06:00
Job Evers‐Meltzer 9140c95a0c Merge pull request #291 from lbryio/fix-pylint
fix pylint errors
2016-11-28 15:32:16 -06:00
Job Evers-Meltzer 4b52c88d45 fix functional tests 2016-11-28 13:55:38 -06:00
Jeremy Kauffman b7dd9c176a remove broken link and clarify INSTALL link 2016-11-28 14:47:21 -05:00
Job Evers-Meltzer 6f2e2e727e fix pylint errors 2016-11-28 13:33:34 -06:00
Jack Robison 48b137fe4c Merge pull request #287 from lbryio/version-add-lbry-id
get_lbry_session_info
2016-11-28 14:26:43 -05:00
Jack a6326ee7a1 get_lbry_session_info 2016-11-28 14:16:21 -05:00
Job Evers‐Meltzer c91034ee4c Update INSTALL.md
Change instructions to prefer to use pip for installation
2016-11-28 12:57:52 -06:00
Umpei Kay Kurokawa 7c505f2e23 Merge pull request #245 from kaykurokawa/add_nout_final
use nOut in addition with txid to identify claims
2016-11-25 17:15:04 -05:00
Jack bfcd84a568 use nOut in addition with txid to identify claims 2016-11-25 16:59:23 -05:00
Jeremy Kauffman a314c530bc remove outdate doc url 2016-11-25 15:20:39 -05:00
Alex Liebowitz 4cb2a8f2aa Include LBRY ID in version() API method result 2016-11-23 20:29:45 -05:00
Alex Liebowitz 9d272befcd Convert length of shortened plaintext LBRY ID to constant 2016-11-23 20:28:16 -05:00
Job Evers‐Meltzer 1695b26416 Merge pull request #276 from lbryio/configuration
Better parse environment variables for configuration
2016-11-22 23:24:13 -06:00
Job Evers 91fb78b077 Update TODO with more useful info 2016-11-22 18:14:41 -06:00
Job Evers-Meltzer 0bbc8f5b7e add envparse to requirements 2016-11-22 18:14:41 -06:00
Job Evers-Meltzer 464bd11c3a Improve parsing of environment variables
Switch adjustable settings to use the envparse library. This
provides more flexible and robust parsing.
2016-11-22 18:14:36 -06:00
Job Evers-Meltzer b88e771eb2 bug fix: use proper settings dictionary 2016-11-22 16:03:37 -06:00
Job Evers-Meltzer 7a76763610 rename Setting to Settings 2016-11-22 16:03:37 -06:00
Job Evers-Meltzer 02d2bad03f allow json to be used to set settings 2016-11-22 16:03:37 -06:00
Jack Robison f65a50a6a3 Merge pull request #260 from lbryio/error-messages
meaningful error messages, improve cli, remove unused search from daemon
2016-11-22 15:54:21 -05:00
Jack 0a9ef07787 Merge branch 'master' into error-messages
# Conflicts:
#	lbrynet/lbrynet_daemon/Daemon.py
#	lbrynet/lbrynet_daemon/auth/server.py
2016-11-22 15:44:47 -05:00
Jack Robison 0e9ad16a2e Merge pull request #278 from lbryio/fix-uncaught-analytics-exception
fix uncaught analytics exception
2016-11-22 15:25:07 -05:00
Jack cd0a9ba9e8 feedback from job 2016-11-22 15:18:01 -05:00
Jack e08d0eb8f9 _log_and_render_error 2016-11-22 15:11:25 -05:00
Jack Robison 82f9cc57fd Merge pull request #283 from lbryio/fail-blank-uri
fail verify_name_characters for blank uri
2016-11-22 13:55:12 -05:00
Umpei Kay Kurokawa ae53129b25 Merge pull request #282 from kaykurokawa/fix_autoconnect2
fix spelling mistake for auto connect
2016-11-21 22:13:13 -05:00
Jack 543f65fc24 fail verify_name_characters for blank name 2016-11-21 21:58:04 -05:00
Kay Kurokawa 896a16c15d fix spelling mistake for auto connect 2016-11-19 19:29:58 -05:00
Job Evers-Meltzer 3d7dee1208 allow for rc in version values 2016-11-19 16:58:40 -06:00
Alex Liebowitz e03db79f79 Always return array from jsonrpc_get_lbry_files()
Before, would return false when there were no files in the DB.
2016-11-16 23:57:02 -05:00
Jack 0b0150ad73 fix uncaught analytics exception 2016-11-16 19:10:49 -05:00
Jack Robison 0490250da4 Merge pull request #272 from lbryio/setup-ui-defaults
Give priority to arguments over settings in UIManager.setup()
2016-11-16 18:53:26 -05:00
Job Evers-Meltzer dd952ddbe3 Bug fix: pkg_resources doesn't like cx_freeze.
Getting the location of bundling resources doesn't work
with cx_freeze so putting in some temporary error handling
while we figure out a better way to bundle the UI.
2016-11-16 15:15:46 -06:00
Alex Liebowitz 776fe41e03 Switch back to "or" style for default params in UIManager.setup() 2016-11-16 16:09:42 -05:00
Alex Liebowitz 8b1ea0a6a7 Give priority to arguments over settings in UIManager.setup() 2016-11-16 15:58:11 -05:00
Job Evers-Meltzer 183617c85b try patched version of cx_freeze 4.3.3 2016-11-16 14:53:27 -06:00
Job Evers-Meltzer 386c67fa70 revert cx-freeze back to 4.3.3 2016-11-16 14:23:48 -06:00
Jack ba7f326e72 use log_support 2016-11-16 15:16:15 -05:00
Job Evers-Meltzer b9caf2c4a9 Bump version: 0.7.6rc0 → 0.7.6 2016-11-16 11:06:53 -06:00
Job Evers‐Meltzer 3188165514 Merge pull request #274 from lbryio/fix-rc-regex
remove - from regex for rc tag
2016-11-15 23:33:57 -06:00
Job Evers-Meltzer 1c036cce78 remove - from regex for rc tag 2016-11-15 22:01:20 -06:00
Alex Grin a2e648fcfc Merge pull request #273 from lbryio/test-build-type
set build type in travis
2016-11-15 23:01:15 -05:00
Alex Grintsvayg b75a89e876 set build type in travis 2016-11-15 22:30:10 -05:00
Alex Grintsvayg 82a3b5e9c5 Merge branch 'better_metrics'
* better_metrics:
  improve error analytics
  use constants for wallet types
2016-11-15 10:32:14 -05:00
Alex Grintsvayg 7ecc8103d8 improve error analytics
- move analytics creation earlier in the startup process
- add several events to track startup, startup failure, other errors
- abstract platform detection into separate file
- add "build" type to platform so we can differentiate devs vs live users
2016-11-15 10:31:35 -05:00
Alex Grintsvayg 6b72b4de11 use constants for wallet types 2016-11-15 10:31:35 -05:00
Job Evers‐Meltzer 8e1ad0aa8b Merge pull request #270 from lbryio/survive-lbry-down
Bug fix: continue even if log fails to upload
2016-11-15 08:48:05 -06:00
Job Evers‐Meltzer 5b7f80487f Merge pull request #271 from lbryio/log-lbryum-on-verbose
enable lbryum info logging when verbose
2016-11-15 08:47:51 -06:00
Job Evers‐Meltzer ebc400804f Merge pull request #262 from lbryio/dry-reflector
Remove repeated code for reflector
2016-11-14 20:54:53 -06:00
Job Evers-Meltzer 9940327b9a enable lbryum info logging when verbose 2016-11-14 20:34:56 -06:00
Job Evers-Meltzer b300afb504 Bug fix: continue even if log fails to upload
lbry.io being down caused lbrynet to die when uploading a log.
2016-11-14 20:16:34 -06:00
Job Evers-Meltzer 0941618048 move reflector blob client to its own module 2016-11-14 19:33:36 -06:00
Job Evers-Meltzer abc7b11e26 remove repeated code for reflector 2016-11-14 19:33:36 -06:00
Job Evers-Meltzer 85dee27c92 whitespace 2016-11-14 19:33:36 -06:00
Jack 5ae3485411 Merge branch 'master' into error-messages
# Conflicts:
#	lbrynet/dht/protocol.py
2016-11-14 14:11:02 -05:00
Jack 8f98469969 unused imports 2016-11-14 14:10:19 -05:00
Jack e1cdbdc91d fix coveralls error 2016-11-14 14:02:19 -05:00
Jack 908c9a3bfa feedback from job 2016-11-14 13:53:11 -05:00
Jack f45a2cba32 fix insufficient funds and max key fee bug 2016-11-14 13:41:05 -05:00
Jack Robison 763fefb33f Merge pull request #256 from kaykurokawa/fix_get_my_claim
Update right after making claim will fail due to get_my_claim()
2016-11-14 13:37:54 -05:00
Job Evers‐Meltzer ce7463f684 Merge pull request #269 from lbryio/normalize-version
Normalize python version
2016-11-11 14:48:33 -06:00
Job Evers-Meltzer cd65b6fd13 Normalize python version
setuptools will force a version to be 'normalized'
which means that it removes the dash.

This will probably cause some weirdness on the version
numbers in the windows build; might have to re-work
my patches on cx_freeze: https://bitbucket.org/jobevers/cx_freeze/commits/all
2016-11-11 14:34:01 -06:00
Jack 472bb6af1b remove unused lighthouse search from daemon 2016-11-11 13:43:58 -05:00
Jack 7f96e35b0e clean up daemon imports and settings var name 2016-11-11 13:42:51 -05:00
Jack f9fef3d586 reactor.iterate() in dht shutdown
-fixes enigmatic  `’Port' object has no attribute 'socket’` error
2016-11-11 13:41:36 -05:00
Jack ec4f9011b9 meaningful API error messages 2016-11-11 13:40:19 -05:00
Job Evers-Meltzer d634c009d3 Bump version: 0.7.5 → 0.7.6-rc0 2016-11-11 12:04:21 -06:00
Job Evers‐Meltzer 1872256d7a Merge pull request #268 from lbryio/bundle-rc
add ui bundle to RC too
2016-11-11 12:02:43 -06:00
Job Evers-Meltzer 81d107d99c better regex for tag match on appveyor 2016-11-11 11:44:35 -06:00
Job Evers‐Meltzer a7589914d0 Merge pull request #267 from lbryio/fix-dht-ports
Restore reactor.iterate in dht shutdown
2016-11-11 11:42:43 -06:00
Job Evers‐Meltzer 0736fa506b Merge pull request #266 from lbryio/startup-logging
more logging on startup
2016-11-11 11:11:02 -06:00
Job Evers‐Meltzer dfec3c1a15 Merge pull request #264 from lbryio/bumpversion
Add release candidates to bumpversion
2016-11-11 11:10:37 -06:00
Job Evers‐Meltzer edc89be794 Merge pull request #257 from lbryio/ensure-ui-loads
bug fix: ensure the bundled ui is loaded
2016-11-11 10:26:54 -06:00
Job Evers-Meltzer d41881596d Restore reactor.iterate in dht shutdown
This reverts the change made in 693fef1964

Not sure why this is needed, but taking this out sometimes causes
`exceptions.AttributeError: 'Port' object has no attribute 'socket'`
to happen on shutdown.
2016-11-11 10:21:50 -06:00
Job Evers-Meltzer 3b34bddf0a more logging on startup 2016-11-11 10:14:41 -06:00
Job Evers-Meltzer 23c012f417 add ui bundle to RC too 2016-11-11 09:49:27 -06:00
Job Evers-Meltzer 2c8d643f4b Add release candidates to bumpversion
The release process now looks like:
starting version: 0.7.5
bumpversion patch -> 0.7.6-rc0
bumpversion candidate -> 0.7.6-rc1
bumpversion release -> 0.7.6
2016-11-11 07:19:39 -06:00
Jack Robison 0abc081edc Merge pull request #259 from lbryio/use-whole-message-in-auth
use whole message to make hmac
2016-11-10 17:07:39 -05:00
Kay Kurokawa 11b36ff4c8 fixing to remove supports from get_my_claim 2016-11-10 12:52:35 -05:00
alexliebowitz c407d32f5d Merge pull request #258 from lbryio/fix-user-specified-ui-bug
Fix deferred for loading user-specified UIs
2016-11-10 03:51:58 -05:00
Alex Liebowitz 87cc487051 Fix deferred for loading user-specified UIs
In UIManager, pass self._load_ui into deferred as function object
instead of calling immediately
2016-11-10 03:47:32 -05:00
Jack Robison 5ea945ca0b Merge pull request #249 from lbryio/settings-bugs
fix --wallet argument bug, use .json conf file if not given a .yml file
2016-11-09 16:53:46 -05:00
Kay Kurokawa 1f06339d5a fix get_my_claim command 2016-11-09 15:19:58 -05:00
Job Evers-Meltzer 09d248b3cc bug fix: ensure the bundled ui is loaded 2016-11-09 13:34:04 -06:00
Umpei Kay Kurokawa 28bf5215e4 Merge pull request #251 from kaykurokawa/fix_unknonwname_exc
fixing UnknownNameError exception initialization
2016-11-09 12:37:03 -05:00
Job Evers-Meltzer f3a71245e8 Add missing files
Somehow these files got lost in a rebase. I'm sort of at a loss
for how, but well, adding them back in.
2016-11-09 11:35:03 -06:00
Job Evers‐Meltzer 37fee71898 Merge pull request #247 from lbryio/bad-whitespace
add bad-whitespace check to pylint
2016-11-09 11:27:45 -06:00
Job Evers-Meltzer 61a354fa60 add bad-whitespace check to pylint 2016-11-09 11:27:07 -06:00
Job Evers‐Meltzer 1d4b2c3527 Merge pull request #253 from lbryio/bundled-ui-squash
Bundle the UI into non-tagged builds.
2016-11-09 11:03:23 -06:00
Job Evers-Meltzer 694b5b9ee5 switch branch to master 2016-11-09 11:02:00 -06:00
Job Evers-Meltzer 838508f342 Bundle the UI into non-tagged builds.
Updates the UIManager to check if a bundled UI was included
in the package and preferentially use that.
2016-11-09 09:29:39 -06:00
Job Evers-Meltzer 694db88ac1 bug fix: fail build if tests fail 2016-11-09 09:28:47 -06:00
Job Evers-Meltzer cb7c849273 bug fix: actually read the environment variable 2016-11-09 09:28:18 -06:00
Alex Grin 9d82be6664 that server's not up anymore 2016-11-09 10:13:36 -05:00
Job Evers‐Meltzer 66745b0820 Merge pull request #242 from lbryio/shutdown-issues
Shutdown issues
2016-11-09 08:17:13 -06:00
Job Evers‐Meltzer f845729adb Merge pull request #246 from lbryio/refactor-upload
Refactor upload
2016-11-09 08:16:34 -06:00
Job Evers-Meltzer 6a587d03cf refactor upload endpoint 2016-11-09 08:16:08 -06:00
Job Evers-Meltzer affb7d837a better logging, cleanup whitespace 2016-11-09 08:16:08 -06:00
Kay Kurokawa 4ad6405f35 fixing UnknownNameError exception initialization 2016-11-08 14:28:17 -05:00
Job Evers‐Meltzer 906e6b238e Merge pull request #243 from lbryio/unicode-publish
Unicode publish
2016-11-08 08:45:26 -06:00
Jack 297865902b use json conf file if not given yml 2016-11-05 14:23:48 -04:00
Jack ab15074ae4 fix --wallet=lbrycrd argument 2016-11-05 14:10:26 -04:00
Job Evers-Meltzer b68a1838f2 eliminate unused return value 2016-11-04 11:44:50 -05:00
Job Evers-Meltzer 43b2cd1433 cleanup whitespace and long-lines 2016-11-04 11:44:50 -05:00
Job Evers-Meltzer 0036685ece use more readable KB, MB 2016-11-04 11:44:50 -05:00
Job Evers-Meltzer d06c670a12 Allow publishing with unicode filenames
encodes unicode into a string before hashing.  Also adds tests
to verify that the fix works and that normal files
can be made into EncryptedFiles
2016-11-04 11:44:50 -05:00
Job Evers-Meltzer 6d6cfbc7cc whitespace 2016-11-03 16:57:42 -05:00
Job Evers-Meltzer 3c977f961f improve osx app shutdown 2016-11-03 16:57:29 -05:00
Job Evers-Meltzer 59c4593bf4 whitespace 2016-11-03 14:42:57 -05:00
Job Evers-Meltzer 5bccfdb244 add logging to shutdown process 2016-11-03 14:42:45 -05:00
Job Evers-Meltzer 10cb20d08f fix whitespace and long-lines 2016-11-03 14:22:16 -05:00
Job Evers-Meltzer 693fef1964 remove reactor.iterate from dht shutdown 2016-11-03 10:43:24 -05:00
Job Evers-Meltzer fe7700d726 Bump version: 0.7.4 → 0.7.5 2016-11-02 10:50:42 -05:00
Job Evers‐Meltzer fee6de1c1f Merge pull request #238 from lbryio/missing-conf
fix missing conf file on first run
2016-11-02 10:49:49 -05:00
Jack 7d04ae2237 use whole message to make hmac 2016-11-02 11:38:09 -04:00
Jack c748572bf6 feedback from job 2016-11-02 11:32:46 -04:00
Jack 660bf74d42 fix missing conf file on first run 2016-11-02 10:41:47 -04:00
Job Evers-Meltzer 4db3be345b Bump version: 0.7.3 → 0.7.4 2016-11-02 07:43:11 -05:00
Job Evers‐Meltzer dcea8a8c78 Merge pull request #236 from lbryio/better-log-and-kill
More robustly stop the reactor
2016-11-02 07:36:21 -05:00
Job Evers-Meltzer 815696485f More robustly stop the reactor
Was getting a "cannot stop a reactor that isn't running error"
when playing around with error handling on startup.

Following the suggestion from http://stackoverflow.com/a/36738480
it seems to work better
2016-11-02 07:23:37 -05:00
Job Evers‐Meltzer b316b56885 Merge pull request #234 from lbryio/unicode-settings
download directory and ui bug fixes for windows
2016-11-02 07:19:51 -05:00
Job Evers‐Meltzer 269dc47b69 Merge pull request #230 from kaykurokawa/better_stream_info_error
Better error detection in Wallet for invalid streams
2016-11-02 07:13:48 -05:00
Job Evers‐Meltzer a7e3246a82 Merge pull request #233 from lbryio/add-more-pylint
lint the osx and windows apps
2016-11-01 18:33:44 -05:00
Jack 96ef5089fc fix data_dir problem in UIManager 2016-11-01 16:40:50 -04:00
Job Evers-Meltzer c0d96a5733 fix bug in jsonrpc_get_settings 2016-11-01 15:39:51 -05:00
Job Evers-Meltzer 7de4340c38 lint the osx and windows apps
This also fixes a trivial bug in the osx release.
2016-11-01 15:39:50 -05:00
kkurokawa 7dd5e7ceeb spacing after comma fix, use Exception init in InvalidStreamInfoError 2016-11-01 16:35:44 -04:00
Jack 9de7354ea0 drop connection on failed read blob 2016-11-01 16:32:34 -04:00
Jack cfbdda411a whoops 2016-11-01 13:23:02 -04:00
Jack 96c4837cb8 clean up errback 2016-11-01 13:19:13 -04:00
Jack 2294e6bf36 Merge branch 'master' into unicode-settings 2016-11-01 13:06:20 -04:00
Jack 4cffc32328 catch availability check error 2016-11-01 13:05:19 -04:00
Jack d1b23e25d9 allow unicode settings 2016-11-01 12:53:04 -04:00
Job Evers-Meltzer e5ba70c3ba Bump version: 0.7.2 → 0.7.3 2016-11-01 11:38:45 -05:00
Job Evers‐Meltzer c88e4aecf5 Merge pull request #232 from lbryio/read-config
Read config
2016-11-01 11:33:17 -05:00
Job Evers‐Meltzer 469df84990 Merge pull request #229 from kaykurokawa/fix_updates
Fix updates in name_claim function
2016-11-01 11:16:03 -05:00
Job Evers-Meltzer a4ec706b06 enable console logging for apps 2016-11-01 11:14:04 -05:00
Job Evers-Meltzer 30533a7986 Read config file in windows and osx
Also moves the config loading to BEFORE the log configuration
2016-11-01 11:14:04 -05:00
kkurokawa 39b47bc569 better error detection in _get_stream_info_from_value 2016-11-01 10:34:08 -04:00
Job Evers‐Meltzer 96104e65ed Merge pull request #226 from lbryio/start-tcp-earlier
Start listening before launching browser
2016-11-01 07:33:33 -05:00
kkurokawa e9bab33e2a fixing claim updates 2016-10-31 19:17:37 -04:00
Job Evers-Meltzer 3b6006d759 Start listening before launching browser 2016-10-31 14:00:07 -05:00
Job Evers‐Meltzer 9535a24706 Merge pull request #221 from lbryio/less-lbryum-logs
set lbryum to log at warning level
2016-10-30 17:04:37 -05:00
Job Evers‐Meltzer 8f2be46536 Add coveralls badge 2016-10-30 16:55:12 -05:00
Job Evers-Meltzer 9ca80a7cf7 set lbryum to log at warning level
lbryum is very chatty at the info level. As a rough estimate a bit
over half of our logs in loggly are a result of lbrum.

The better, long-term, solution would be to modify lbryum to move more
logs to the debug level.
2016-10-30 16:47:37 -05:00
Job Evers‐Meltzer 72078f4af3 Merge pull request #220 from lbryio/add-coverage
Add coverage reporting
2016-10-30 16:05:46 -05:00
Job Evers-Meltzer ff9ea37e0c add coverage reporting 2016-10-30 15:44:45 -05:00
Job Evers-Meltzer 9baf261cb3 Bump version: 0.7.1 → 0.7.2 2016-10-30 14:47:21 -05:00
Job Evers-Meltzer f03a15d503 remove unneeded 'after_first_run' check 2016-10-30 14:46:16 -05:00
Job Evers-Meltzer 1a70a20c8b Bump version: 0.7.0 → 0.7.1 2016-10-30 10:57:41 -05:00
Job Evers-Meltzer 359cb83fab change windows log file location 2016-10-30 10:46:58 -05:00
Job Evers-Meltzer 0893858385 Bump version: 0.6.10 → 0.7.0 2016-10-28 18:03:40 -05:00
Job Evers-Meltzer e1997a6852 dumb bug fix 2016-10-28 17:55:34 -05:00
Job Evers‐Meltzer 0f4fee475f Merge pull request #201 from lbryio/reflect-my-unavailable-streams
Upload unavailable streams to reflector on startup
2016-10-28 17:29:28 -05:00
Job Evers-Meltzer 864e963dea Merge remote-tracking branch 'origin/master' into reflect-my-unavailable-streams
Conflicts:
	lbrynet/core/log_support.py
2016-10-28 17:16:39 -05:00
Job Evers‐Meltzer 3d0debd775 Merge pull request #218 from lbryio/misc-fixes
various fixes
2016-10-28 17:12:07 -05:00
Job Evers‐Meltzer 6cc2b48a06 Merge pull request #217 from lbryio/safer-conversion
For settings: On conversion failure, fallback to default values
2016-10-28 16:57:40 -05:00
Job Evers-Meltzer dee0917642 various fixes 2016-10-28 16:57:25 -05:00
Job Evers-Meltzer 395a68b261 For settings: On conversion failure, fallback to default values 2016-10-28 16:29:16 -05:00
Job Evers‐Meltzer 0d066b2d11 Merge pull request #213 from lbryio/log-errors-on-resolve-name
Log error when resolve name fails
2016-10-28 16:25:19 -05:00
Job Evers-Meltzer 353ede26b8 Log error when resolve name fails 2016-10-28 16:03:58 -05:00
Job Evers‐Meltzer 46cfa20f65 Merge pull request #204 from lbryio/cleanup-logging
Cleanup logging
2016-10-28 15:56:54 -05:00
Job Evers‐Meltzer 30b1dd3ce6 Merge pull request #210 from lbryio/app-startup
make app startup consistent across versions
2016-10-28 15:55:09 -05:00
Jack da601818a3 return from restore_status properly 2016-10-28 16:47:45 -04:00
Job Evers-Meltzer 6fa624ec8a switch LogUploader to use an already encoded lbryid 2016-10-28 15:44:24 -05:00
Job Evers-Meltzer 779ba8e214 Fix missing conf imports 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 80f548fdfd use same logging for apps 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 6f9ec2fa66 remove repeated log code 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 40437e4a82 log cleanup 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 2ee6fac014 redo logging configuration for lbrynet-daemon script 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 8fe5eb514a make app startup consistent across versions 2016-10-28 15:41:48 -05:00
Jack c4312cc387 Merge branch 'master' into reflect-my-unavailable-streams 2016-10-28 16:05:16 -04:00
Jack 110a9b1e0f whoops 2016-10-28 15:56:22 -04:00
Jack 035a1cf758 feedback from job 2016-10-28 15:48:51 -04:00
Jack d4785849e1 remove unused imports 2016-10-28 14:22:37 -04:00
Job Evers-Meltzer b41756e38a Bump version: 0.6.9 → 0.6.10 2016-10-28 12:29:15 -05:00
Job Evers‐Meltzer 60ee278519 Merge pull request #215 from lbryio/delete_running
delete RUNNING.md
2016-10-28 11:10:54 -05:00
Alex Grintsvayg b49e44926a this is very out of date 2016-10-28 12:01:10 -04:00
Job Evers‐Meltzer 81187da5b7 Merge pull request #211 from lbryio/refactor-settings
Refactor settings
2016-10-27 15:36:11 -05:00
Job Evers-Meltzer e70ecc8ac9 change settings object for packages too 2016-10-27 14:57:02 -05:00
Job Evers-Meltzer 862055a6de move settings into conf file 2016-10-27 14:31:27 -05:00
Job Evers-Meltzer 7076c0c870 Require env settings are in LBRY namespace 2016-10-27 11:05:21 -05:00
Job Evers-Meltzer 2cd6c644a0 Merge remote-tracking branch 'origin/master' into refactor-settings
Conflicts:
	lbrynet/lbrynet_daemon/Daemon.py
	lbrynet/lbrynet_daemon/DaemonControl.py
	lbrynet/lbrynet_daemon/DaemonServer.py
2016-10-27 10:34:48 -05:00
Job Evers-Meltzer 0edacbe4c8 Merge remote-tracking branch 'origin/master' into refactor-settings
Conflicts:
	lbrynet/conf.py
	lbrynet/lbrynet_daemon/Daemon.py
	lbrynet/lbrynet_daemon/DaemonCLI.py
	lbrynet/lbrynet_daemon/UIManager.py
	tests/functional/test_misc.py
2016-10-27 10:18:56 -05:00
Job Evers‐Meltzer 4b330d0e9f Update Installation instructions 2016-10-27 09:25:06 -05:00
Job Evers‐Meltzer 8ce08f7da1 Merge pull request #206 from lbryio/shutdown-during-setup
Shutdown during setup
2016-10-26 19:09:33 -05:00
Jack 127987f3ba upload log if lbry id is not yet set, clean up crash log 2016-10-26 17:39:47 -05:00
Job Evers-Meltzer 3c7eec9456 raise error if database is newer 2016-10-26 17:39:46 -05:00
Job Evers-Meltzer 5d45345398 shutdown if there are errors during setup 2016-10-26 17:39:43 -05:00
Jack Robison b360eb6181 Merge pull request #207 from lbryio/better-json
Use better json formatting
2016-10-26 16:32:19 -04:00
Jack 369cd516c0 move availability check and fix from ManagedEncryptedFileDownloader to ReflectorAvailabilityHelper 2016-10-26 16:17:01 -04:00
Jack 24320d6444 remove debug line 2016-10-26 03:22:45 -04:00
Jack 1951ea09cd updates from master, more refactoring
-lbrynet.lbrynet_daemon.auth.client.LBRYAPIClient.config will detect if
it needs to return the auth/non-auth version
2016-10-26 03:16:33 -04:00
Job Evers-Meltzer 4d391e7429 Use better json formatting
Apply json.dumps at the end of the formatting process instead of the
middle.  This allows for proper escaping of quotes and allows for
actual json to be emitted.
2016-10-25 17:49:35 -05:00
Job Evers‐Meltzer fb88cdb445 Merge pull request #203 from lbryio/github-bug
catch exception if github is down
2016-10-24 11:02:12 -05:00
Job Evers‐Meltzer 572f1d1f6e Merge pull request #200 from lbryio/cleanup-tests
Start to cleanup functional tests to improve readability
2016-10-24 11:01:20 -05:00
Job Evers‐Meltzer 27a8114b6f Merge pull request #197 from lbryio/add-dht-tests
Rename DHT tests such that trial automatically runs them
2016-10-24 11:00:40 -05:00
Job Evers‐Meltzer 6f3873d99b Merge pull request #178 from lbryio/track-data-upload-rebase
Track Data Uploaded and other stuff
2016-10-24 08:20:29 -07:00
Job Evers-Meltzer e62e2cc440 catch exception if github is down 2016-10-21 16:12:56 -07:00
Job Evers-Meltzer a5797cdeb3 Better handling of lbry file search
- replace lbry file search strings with constants
- refactor repeated code for deciding what search type to use
2016-10-20 16:01:54 -07:00
Jack 3dd99fdc92 upload unavailable streams to reflector on startup 2016-10-20 15:40:35 -04:00
Job Evers-Meltzer 0c2265a8da replace some strings 2016-10-20 11:30:38 -07:00
Job Evers-Meltzer 01811621a6 shorten is_valid_blobhash logic 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 229ed0d7dc rename summarize to summarize_and_reset 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 7f800ce5bb fix rebase error 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer aa291ca79a fix pylint 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 813267ae18 fix tests 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer fbf7928412 rename events to be seperate for each metric 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 7167d47631 bug fixes 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 8b1bb673c1 report available blobs 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 417a8f719f Add looping call manager to analytics manager 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer ddb88eb1c9 move LoopingCallManager to own module 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer d5f0001950 small cleanups 2016-10-20 11:29:15 -07:00
Job Evers-Meltzer 4a07b56e8a refactor analytics manager to use dependency injection 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 9da1b3c5b7 analytics manager bug fix 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer f9f07c47e2 move analytics manager 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer f5213fb77a GetFileHelper bug fixes 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 48a61605de bug fix: looping calls 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 76fb7697ff misc 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer b71a3fee43 add tracked metrics reporting to analytics 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 3a91896d8a track the amount of data uploaded 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 36ae0d5f20 remove utils file - function moved to core 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 3f727d892c refactor events; add tests 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer c104ed3f8b better time handling in tests 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 9416376a34 refactor: add GetFileHelper
Move the code related to getting a lbry file into
its own class
2016-10-20 11:26:10 -07:00
Job Evers-Meltzer 66e139c566 set development version based on environment variable 2016-10-20 11:26:10 -07:00
Job Evers-Meltzer d0a82c68df move analytics related calls into their own class 2016-10-20 11:25:17 -07:00
Job Evers-Meltzer 93e2b3e20a Extract some looping call code out of the daemon 2016-10-20 11:25:17 -07:00
Job Evers-Meltzer 2e59e5e3b8 extract individual functions 2016-10-20 11:16:36 -07:00
Jack 727ea12dbd Bump version: 0.6.8 → 0.6.9 2016-10-20 12:41:49 -04:00
Jack Robison ea3225c897 Merge pull request #198 from lbryio/fix-blobrequester
fix problems in blobrequester
2016-10-20 12:41:26 -04:00
Job Evers-Meltzer bfcfa9dc3c cleanup tests to improve readability 2016-10-20 09:30:08 -07:00
Jack ea21821558 fix issues with is_generous
-stop requesting blobs if price limit is reached and the offer still
has not been accepted

-drop max rate to old flat rate (0.005lbc/mb)
2016-10-20 12:29:55 -04:00
Jack a9ae1a1016 fix bug where support claim is returned when it shouldn't be 2016-10-20 11:15:35 -04:00
Jack 3f5efb1fa3 fix problems in blobrequester
-also move is_generous_host into a conf setting
2016-10-20 01:35:18 -04:00
Job Evers-Meltzer 9ba38c84cc Bump version: 0.6.7 → 0.6.8 2016-10-19 15:56:58 -07:00
Job Evers-Meltzer a5d3c0bbe1 rename tests such that trial automatically runs them 2016-10-19 16:38:59 -05:00
Jack Robison d4aa109b73 Merge pull request #193 from lbryio/check-remote-conn
Use lbry.io instead of google to check for internet connection
2016-10-19 15:06:33 -04:00
Jack Robison bdeccb49b8 Merge pull request #191 from lbryio/verify-published-file-exists
make publish fail when given an invalid file
2016-10-19 15:05:33 -04:00
Jack Robison 35ce7bbd61 Merge pull request #195 from lbryio/fail-appveyor
Appveyor build needs to fail when tests fail
2016-10-19 15:04:20 -04:00
Jack 293933163f move USE_AUTH_HTTP to adjustable settings 2016-10-19 14:59:10 -04:00
Jack 22fc26797d pylint fixes 2016-10-19 14:40:01 -04:00
Jack 8a95c8b7af whoops 2016-10-19 14:26:19 -04:00
Jack 33674f8cc3 remove logging in conf, rearrange platform 2016-10-19 00:39:19 -04:00
Jack 6799de645b Merge branch 'master' into refactor-settings
# Conflicts:
#	packaging/osx/lbry-osx-app/lbrygui/LBRYApp.py
#	packaging/windows/lbry-win32-app/LBRYWin32App.py
2016-10-19 00:19:45 -04:00
Jack 06a88599df --http-auth flag to use authentication, refactor settings
settings will prefer, in order:
-defaults
-settings in config file
-settings given as environmental variables
-settings given as command line args
2016-10-19 00:12:44 -04:00
Jack 03877f7ebd remove str 2016-10-18 17:11:45 -04:00
Job Evers-Meltzer 9f2dbd4291 Appveyor build needs to fail when tests fail
According to
https://www.appveyor.com/docs/build-configuration/#script-blocks-in-build-configuration
a build only fails if an exception is thrown. This is causing builds to pass even if
the tests fail. That same link suggests adding the code that is in this commit.
2016-10-18 09:44:21 -05:00
Job Evers 6b7d575e95 use lbry.io to check for an internet connection 2016-10-17 20:11:08 -05:00
Job Evers 640db41555 only need lbrycrdd_path when using lbrycrdd 2016-10-17 19:58:04 -05:00
Jack d860c2a2b4 make publish fail when given an invalid file
otherwise a less obvious error is returned in
lbrynet_daemon.Publisher.Publisher._show_publish_error
2016-10-17 14:25:36 -04:00
Jack 733bb3bccf Bump version: 0.6.6 → 0.6.7 2016-10-16 02:55:19 -04:00
Jack Robison 307751277d Merge pull request #189 from lbryio/include-jsonschema
fix jsonschema includes
2016-10-16 02:54:55 -04:00
Jack 62c2539475 include jsonschema.schemas files 2016-10-16 02:49:19 -04:00
Jack 5b15da2c55 add jsonschema to windows build 2016-10-16 00:29:32 -04:00
Jack 830f5ff6e3 Bump version: 0.6.5 → 0.6.6 2016-10-15 23:35:17 -04:00
Jack ea1f260ed9 Bump version: 0.6.4 → 0.6.5 2016-10-14 23:06:03 -04:00
Jack Robison 3e86254324 Merge pull request #188 from lbryio/fix-DaemonRequest-import
fix DaemonRequest import in apps
2016-10-14 23:04:52 -04:00
Jack b5311dd1c3 fix DaemonRequest import in apps 2016-10-14 23:01:15 -04:00
Jack a8a581d35c Merge branch 'master' into auth-jsonrpc
# Conflicts:
#	lbrynet/conf.py
#	lbrynet/core/Error.py
#	lbrynet/lbrynet_daemon/Daemon.py
#	lbrynet/lbrynet_daemon/DaemonControl.py
#	lbrynet/lbrynet_daemon/ExchangeRateManager.py
2016-10-14 18:25:37 -04:00
Job Evers-Meltzer 5f129faf32 Bump version: 0.6.3 → 0.6.4 2016-10-14 16:06:12 -05:00
Job Evers‐Meltzer 018d78be6f Merge pull request #187 from lbryio/new-metadata-system
Rewrite of migration and validation system using JSON Schema
2016-10-14 15:51:49 -05:00
Alex Liebowitz b5a3929bb5 Fix imports 2016-10-14 16:41:02 -04:00
Alex Liebowitz b67b4b2ef6 Refactor metadata upgrade version range calculation to use generator 2016-10-14 15:58:18 -04:00
Alex Liebowitz bed3f551fd Light refactor of Metadata class 2016-10-14 15:58:04 -04:00
Alex Liebowitz 5b0b8a19ff Add requirements.txt for jsonschema 2016-10-14 15:57:09 -04:00
Alex Liebowitz 988c9595c7 Add default MIME type during publication
Needed because the new JSON schema enforces that the content type is a
string (not null).
2016-10-14 10:37:19 -04:00
Alex Liebowitz 92700808eb Add back current_version attribute to Metadata 2016-10-14 10:37:19 -04:00
Alex Liebowitz 0dd9aa0d67 Update error that daemon catches for malformed metadata
Now jsonschema.ValidationError instead of AssertionError
2016-10-14 10:37:19 -04:00
Alex Liebowitz 9cf1d6b710 Tests for new JSON Schema system 2016-10-14 10:37:18 -04:00
Alex Liebowitz d8d462f43c Convert Metadata and Fee validators to use new JSON Schema-based system 2016-10-14 10:37:18 -04:00
Alex Liebowitz 3f22f39ce1 Refactor Validator to new StructuredDict class that uses JSON Schema
- Uses JSON schema for all validation (so far no custom code needed)
 - Can migrate up and down with any versioning scheme
 - Does migrations with regular dictionary operations instead of a DSL
2016-10-14 10:35:17 -04:00
Job Evers-Meltzer e647663c34 Fix error when version is None
Logs were reporting:

Failure instance: Traceback: <type 'exceptions.AttributeError'>: StrictVersion instance has no attribute 'version'
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/DaemonServer.py:121:requestReceived
twisted/web/server.pyc:183:process
twisted/web/server.pyc:234:render
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:450:render
--- <exception caught here> ---
twisted/internet/defer.pyc:150:maybeDeferred
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:1561:jsonrpc_version
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/core/utils.py:39:version_is_greater_than
distutils/version.pyc:140:__cmp__
2016-10-14 10:35:17 -04:00
Jack Robison b5987c75da Merge pull request #184 from lbryio/missing-version
Fix error when version is None
2016-10-13 15:02:49 -04:00
Jack Robison 32a973943f Merge pull request #173 from lbryio/blob-prices
Blob prices
2016-10-13 14:59:26 -04:00
Jack 99c060719a clean up test_strategy 2016-10-13 14:00:46 -04:00
Jack d9cd215a60 set seed for strategy tests 2016-10-13 13:48:18 -04:00
Jack 2d3b1fd4eb clean up test_misc 2016-10-13 13:36:22 -04:00
Jack 19c2264429 clean up 2016-10-13 13:35:55 -04:00
Job Evers-Meltzer ae4b984f0b Fix error when version is None
Logs were reporting:

Failure instance: Traceback: <type 'exceptions.AttributeError'>: StrictVersion instance has no attribute 'version'
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/DaemonServer.py:121:requestReceived
twisted/web/server.pyc:183:process
twisted/web/server.pyc:234:render
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:450:render
--- <exception caught here> ---
twisted/internet/defer.pyc:150:maybeDeferred
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:1561:jsonrpc_version
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/core/utils.py:39:version_is_greater_than
distutils/version.pyc:140:__cmp__
2016-10-11 20:48:29 -05:00
Job Evers‐Meltzer 897c113db1 Merge pull request #180 from lbryio/configure-lbryum
Configure lbryum
2016-10-11 16:46:39 -05:00
Jack 50e5a59420 remove unused imports 2016-10-07 17:29:20 -04:00
alexliebowitz 8735112f70 Merge pull request #182 from lbryio/no-cache-static-file
NoCacheStaticFile
2016-10-07 17:15:35 -04:00
Jack 5a0b5dc680 Merge branch 'master' into auth-jsonrpc
# Conflicts:
#	lbrynet/lbrynet_daemon/Daemon.py
#	lbrynet/lbrynet_daemon/DaemonControl.py
2016-10-07 17:11:43 -04:00
Job Evers‐Meltzer d91e9aa910 Fix Instructions 2016-10-07 16:17:47 -04:00
Jack be1803d558 remove unused imports 2016-10-07 14:52:54 -04:00
Jack 84bbf2f76e move set_no_cache into own function 2016-10-07 14:28:38 -04:00
Jack fe7ea7c679 refactor resources and file streamer into their own files, use NoCacheStaticFile instead of static.File 2016-10-07 14:01:59 -04:00
Jack 782e197e66 Merge branch 'master' into blob-prices 2016-10-05 23:25:39 -04:00
Jack 88be425ab2 fix pylint errors 2016-10-05 23:24:22 -04:00
Jack 7025c5102a skip live transfer 2016-10-05 23:17:45 -04:00
Jack 1ee9b3c7a6 add reply_to_offer to livepaymentratemanager 2016-10-05 23:03:16 -04:00
Jack 4f60a98eb2 fix functional tests, add a few unit tests,
-add ‘generous’ parameter for NegotiatedPaymentRateManager to turn free
hosting on/off, by default set to true.
2016-10-05 22:58:34 -04:00
Job Evers-Meltzer a71e87a398 throw error when invalid wallet is specified 2016-10-04 13:59:04 -05:00
Job Evers-Meltzer 0ecfad0ae9 add configuration to LBRYumWallet 2016-10-04 13:58:44 -05:00
Job Evers-Meltzer e909ae566b add TODO 2016-10-03 16:19:58 -05:00
Jack dc2f0adb3e update LiveStreamPaymentRateManager 2016-10-03 16:54:46 -04:00
Jack 180d981ed8 fix more functional tests 2016-10-03 16:37:27 -04:00
Jack fd9bc6214a fix functional tests 2016-10-03 15:40:27 -04:00
Jack b3f3ed8026 start fixing functional tests 2016-10-03 02:59:40 -04:00
Jack 967700dc28 BlobRequestHandler and Strategy tests 2016-10-03 02:44:58 -04:00
Jack 3ad4ad50ec Bump version: 0.6.2 → 0.6.3 2016-10-02 03:33:12 -04:00
Jack Robison 7f980446db Merge pull request #137 from lbryio/html5
More reliable stream producer
2016-10-02 03:31:54 -04:00
Alex Liebowitz 04ced2e975 Fix import issues in DaemonServer.py 2016-10-02 02:50:17 -04:00
Alex Liebowitz 2253943eba Shorten time between sending chunks of video stream 2016-10-02 02:03:47 -04:00
Alex Liebowitz a6fcd5d1f2 More refactoring in LBRYStreamProducer 2016-10-01 23:35:07 -04:00
Alex Liebowitz f245822814 Refactor of LBRYStreamProducer 2016-10-01 23:35:07 -04:00
Alex Liebowitz aa3aff91d0 Refactor LBRYStreamProducer and add Content-Length header
Also fixes producer pause/unpause behavior and adds slight delay
between sending chunks
2016-10-01 23:33:51 -04:00
Alex Liebowitz 5537dd878f Add basic support for streaming partially downloaded files 2016-10-01 23:29:08 -04:00
Jack 5e4e619708 squelch loud log 2016-10-01 15:20:19 -04:00
Jack d741c705cd add some tests, don't use lighthouse for cost ests 2016-10-01 01:09:27 -04:00
Jack d83abd81f8 job's comments 2016-09-30 14:26:13 -04:00
Jack 640e000b9f fix timeouts in get_availability 2016-09-30 13:35:13 -04:00
Jack 8d961d6ce6 add get_availability 2016-09-30 13:28:01 -04:00
Jack 0a612ba36c Merge branch 'master' into blob-prices 2016-09-30 12:36:10 -04:00
Jack eb7cc8463b Bump version: 0.6.1 → 0.6.2 2016-09-30 11:26:55 -04:00
Jack 50f0430d04 fix broken windows import 2016-09-30 11:26:37 -04:00
Jack 6e5ea935f6 Bump version: 0.6.0 → 0.6.1 2016-09-30 11:18:07 -04:00
Jack c4eb3de5b2 update app imports 2016-09-30 11:17:47 -04:00
Jack f79b651b05 Bump version: 0.5.0 → 0.6.0 2016-09-30 02:16:06 -04:00
Jack c130879ef7 frontload bug
divide availability adjusted price by frontload factor, which will be
close to 1 while index is low and close to 2 when index is high - the
reduction approaches 50% as index goes up.
2016-09-30 00:36:23 -04:00
Jack 5c391f4bb4 remove unnecessary class 2016-09-30 00:12:17 -04:00
Jack b853656aa6 Merge branch 'master' into blob-prices
# Conflicts:
#	lbrynet/core/server/BlobRequestHandler.py
2016-09-30 00:11:53 -04:00
Jack Robison 8222f940b4 Merge pull request #177 from lbryio/blob-request-handler-refactor
refactor BlobRequestHandler
2016-09-29 23:52:14 -04:00
Jack 71b989b29b update from master
resolve merge conflicts:
lbrynet/core/client/BlobRequester.py
lbrynet/core/Peer.py
lbrynet/lbryfilemanager/EncryptedFileManager.py
lbrynet/lbrynet_daemon/Daemon.py
lbrynet/lbrynet_daemon/Publisher.py
2016-09-29 23:34:59 -04:00
Jack Robison 7e2f7e6f68 Merge pull request #170 from lbryio/appveyor-deploy
appveyor updates
2016-09-29 12:51:16 -04:00
Jack Robison aefb6c5dc8 Merge pull request #175 from lbryio/skip-tests
Tests requiring epoll should only run on linux
2016-09-29 12:50:44 -04:00
Jack Robison b892b81dad Merge pull request #172 from lbryio/add-venv-instr
encourage use of virtualenv
2016-09-29 12:50:05 -04:00
Jack Robison 4d63629919 Merge pull request #164 from lbryio/refactor-blob-requester
Refactor blob requester
2016-09-29 12:49:28 -04:00
Jack 4727ee0403 Merge branch 'master' into refactor-blob-requester 2016-09-29 12:48:34 -04:00
Jack Robison fc47e8074f Merge pull request #174 from lbryio/rename-LBRY
Remove LBRY Prefix
2016-09-29 12:47:50 -04:00
Jack ff7829a078 Merge branch 'master' into rename-LBRY 2016-09-29 12:47:22 -04:00
Job Evers-Meltzer d05bbea131 refactor BlobRequestHandler 2016-09-28 17:00:34 -07:00
Job Evers‐Meltzer e08a681c8f Merge pull request #176 from lbryio/start-stream-event
Start stream event
2016-09-28 12:15:56 -07:00
Job Evers-Meltzer 6a72b3683e send Download Started event 2016-09-28 09:07:25 -07:00
Job Evers-Meltzer b201fc331c simplify callback when downloading 2016-09-28 09:06:51 -07:00
Job Evers-Meltzer 27337a9243 appveyor updates
- move init and build steps into dedicated scripts
- deploy msi artifact on tag
2016-09-28 07:38:12 -07:00
Jack b671a3c2ba Merge branch 'master' into blob-prices 2016-09-27 23:59:14 -04:00
Jack 1cc6b7658c get uploads working
-add error catching in exchange rate manager
-add free data on first request with default negotiation strategy
2016-09-27 23:56:08 -04:00
Job Evers-Meltzer 12e2827286 tests requiring epoll should only run on linux 2016-09-27 16:01:47 -07:00
Job Evers-Meltzer 548d37929e more fixes 2016-09-27 12:08:32 -07:00
Job Evers-Meltzer eb783b6b6f rename File to EncryptedFile 2016-09-27 11:26:01 -07:00
Job Evers-Meltzer d1b06c3688 more fixes 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer d16b3429cd more work to fix tests 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer 1e4b5cb450 rename test files 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer a9da6e4bdf run sed on tests too 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer a099e2db6a fix errors 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer 7dbeb1f2cf Remove LBRY prefix from class and module names
This is the result of running:
    grep -RlI LBRY . | xargs sed -i 's/LBRY\([A-Z]\)/\1/g'
2016-09-27 11:18:35 -07:00
Job Evers-Meltzer 550fd185a3 rename LBRYFile to EncryptedFile 2016-09-27 11:18:16 -07:00
Jack 1720cce3b0 negotiated downloads
-uploads are still underway
2016-09-27 13:52:44 -04:00
Job Evers-Meltzer 078b254d1d rename files to remove LBRY prefix 2016-09-25 18:11:23 -07:00
Job Evers-Meltzer 6057dc0804 whitespace 2016-09-25 17:18:29 -07:00
Job Evers-Meltzer 5ea7c4f016 fix typo in string format 2016-09-25 17:17:48 -07:00
Job Evers-Meltzer 4334cdace1 Refactor blob requester to better seperate out the three requests
Each blob requests has three parts
- A request to see what blobs are available
- A request to download and pay for one of those blobs
- A request to negotiate data costs

Before the refactoring all of the code to accomplish this was mixed into
one class.  They are now three seperate classes.

There is still a lot of room for improvement, but it is at least better than
it was.
2016-09-25 16:55:55 -07:00
Alex Grintsvayg 6803bfae99 add more bins to path 2016-09-24 15:48:07 -04:00
Jack c49b0fdd49 Bump version: 0.4.9 → 0.5.0 2016-09-24 01:50:14 -04:00
Jack bf3c56f0ec only run unit tests on windows 2016-09-24 01:47:37 -04:00
Jack Robison bbcecea28d Merge pull request #165 from lbryio/get-my-claim
add get_my_claim, remove is_mine from get_claim_info response
2016-09-24 01:01:51 -04:00
Jack a936782ddc fix amount formatting between get_claim_info and get_my_claim 2016-09-24 00:58:54 -04:00
Jack 7ac0fc5a24 convert amount 2016-09-24 00:49:03 -04:00
Jack e115158f86 docstring 2016-09-23 03:04:59 -04:00
Jack bf34fa39b4 unused import 2016-09-23 03:03:15 -04:00
Jack 5dd29da84f blob upload/download history and BlobPriceAndAvailabilityTracker 2016-09-23 03:02:17 -04:00
Job Evers-Meltzer 2107ffdd18 encourage use of virtualenv 2016-09-22 10:19:28 -07:00
Jack 492fc510fa move _get_my_unspent_claim to function 2016-09-21 22:10:19 -04:00
Jack Robison c75d240a65 Merge pull request #169 from lbryio/functional-tests
Enable functional tests
2016-09-21 22:05:16 -04:00
Jack dbeb6bc3bc update uri handler 2016-09-21 21:55:10 -04:00
Jack 4d79a7828c more cleaning up 2016-09-21 21:36:06 -04:00
Jack aecafbbebd clean up 2016-09-21 03:49:52 -04:00
Jack 130f9cfc4d api sessions
-user starts a httpauthsession with an api key and name
-user initializes jsonrpc hmac secret to sha256 of session id
-server sends new random hmac secret after each api call

-a user without an authenticated session will get a authorization error
2016-09-20 16:58:30 -04:00
Job Evers-Meltzer 98d02b3fce [ci skip] add newlines 2016-09-19 12:07:47 -07:00
Jack 4d52a33ee5 Bump version: 0.4.8 → 0.4.9 2016-09-16 04:12:55 -04:00
Jack 7ad916f116 fix api being blocked for /? pages 2016-09-16 04:12:13 -04:00
Jack 05b47a9180 Bump version: 0.4.7 → 0.4.8 2016-09-16 03:56:14 -04:00
Jack Robison 9a412bbc76 Merge pull request #168 from lbryio/fewer-desktop-shortcuts
no extra desktop shortcuts in windows
2016-09-16 03:50:04 -04:00
Jack Robison b3a16f95f7 Merge pull request #167 from lbryio/block-api-requests-by-referer
block api requests by referer
2016-09-16 03:49:55 -04:00
Jack 871b3be09a no extra desktop shortcuts 2016-09-16 03:42:25 -04:00
Jack 7085b1e3c3 block api requests by referer 2016-09-16 03:40:06 -04:00
Jack Robison a9e4343775 Merge pull request #166 from lbryio/daemon-settings-yml
.yml daemon settings file
2016-09-16 01:22:20 -04:00
Jack 3d4d895d9b fix windows requirement 2016-09-16 01:14:56 -04:00
Jack 3d89e3fd93 add to appveyor.yml 2016-09-16 01:08:43 -04:00
Jack a8e2dd3f2a add pyyaml to requirements.txt 2016-09-16 00:57:30 -04:00
Jack 3ae0664fa9 use prettier file format for daemon conf file 2016-09-16 00:14:25 -04:00
Jack 98b9f2cf73 Bump version: 0.4.6 → 0.4.7 2016-09-15 23:06:56 -04:00
Jack Robison 6fa65ebb9f Merge pull request #163 from lbryio/windows-cli
fix windows cli
2016-09-15 22:57:44 -04:00
Jack add163a092 move get_params to own function, add some tests 2016-09-15 22:20:00 -04:00
Jack fcfed3f441 add get_my_claim and get rid of is_mine in get_claim_info 2016-09-15 20:15:20 -04:00
Jack 7b84d9b10a handle no params 2016-09-15 15:04:42 -04:00
Jack e9a1dbca4b fix windows cli
make cli easier to use on windows by converting keyword args to json

cli now accepts commands like:

lbrynet-cli resolve_name name=one

as well as:

lbrynet-cli resolve_name ‘{“name”: “one”}’
2016-09-15 14:47:32 -04:00
Jack 145a2654d2 Bump version: 0.4.5 → 0.4.6 2016-09-15 12:36:37 -04:00
Jack Robison 9f14cdb1d7 Merge pull request #162 from lbryio/fix-appveyor
fix appveyor error
2016-09-15 12:36:02 -04:00
Jack 5cefa2e995 set default logger in __init__ 2016-09-15 12:19:49 -04:00
Jack 13cbe0fd42 fix appveyor error 2016-09-14 23:03:18 -04:00
Jack Robison c8bb0c0bc8 Merge pull request #161 from lbryio/delete-lbry-file-without-claim
Delete lbry file with abandoned claim
2016-09-14 22:41:43 -04:00
Jack 4cc941fbec make abandon_claim compatible with old function name 2016-09-14 22:36:18 -04:00
Jack b010192f8e improve logging 2016-09-14 22:28:59 -04:00
Jack 2b1af8eea2 delete lbry file with an abandoned claim 2016-09-14 22:27:57 -04:00
Jack 6ea1f6f78c block api calls from bad origins 2016-09-14 17:39:19 -04:00
Alex Grintsvayg 1215470cc1 MIT license 2016-09-13 22:10:12 -04:00
Jack Robison 94195b7018 Merge pull request #160 from lbryio/allow-retry-after-timeout
Allow download attempts after having timed out and lower the name cache time
2016-09-13 15:17:13 -04:00
Jack 3c9c178bdc fix conf update 2016-09-12 15:44:56 -04:00
Jack cc64547920 Bump version: 0.4.4 → 0.4.5 2016-09-12 15:42:48 -04:00
Jack a23f2ca956 allow more download attempts after having timed out
-also lower the name cache time
2016-09-12 15:39:19 -04:00
Jack Robison 4dacc52ecb Merge pull request #159 from lbryio/rename-abandon-name
Include unconfirmed balance and rename abandon_name
2016-09-12 14:50:45 -04:00
Jack ccaeecfd46 bug fix 2016-09-12 13:17:21 -04:00
Jack 3e45449e8c return unconfirmed balance 2016-09-12 13:16:56 -04:00
Jack f352a3c597 rename abandon_name to abandon_claim 2016-09-12 13:16:03 -04:00
Job Evers-Meltzer d9a08f0c03 import unittest 2016-09-09 23:44:07 -07:00
Job Evers-Meltzer c77f24dde3 skip instead of fix tests 2016-09-09 23:39:35 -07:00
Job Evers-Meltzer 48c31a2119 remove redundant reflector test 2016-09-09 23:37:51 -07:00
Job Evers-Meltzer 5a578d6d79 enable functional tests 2016-09-09 23:29:46 -07:00
Jack 8cdbc847d1 add get_search_servers to allowed_during_startup 2016-09-09 05:11:51 -04:00
Jack 54dbf66cb7 update secure vars for appveyor 2016-09-09 04:14:56 -04:00
Jack 177d384126 Bump version: 0.4.3 → 0.4.4 2016-09-09 04:12:42 -04:00
Jack Robison 561fe757f3 Merge pull request #152 from DaveA50/master
Fix shortcut icons
2016-09-09 04:09:48 -04:00
Jack Robison c670ab638f Merge pull request #158 from lbryio/get-search-servers
add get_search_servers
2016-09-09 04:04:51 -04:00
Jack eea5cbbafc add get_search_servers 2016-09-09 04:04:35 -04:00
Jack Robison e4c61a4a13 Merge pull request #155 from lbryio/get-block-by-height
add 'height' parameter to get_block
2016-09-09 00:57:51 -04:00
Jack 0f0afadc94 add 'height' parameter to get_block 2016-09-09 00:48:54 -04:00
Alex Grintsvayg 659563f27a fix indent 2016-09-08 13:41:11 -04:00
Alex Grintsvayg cbd8a68d0f print actual json 2016-09-08 13:40:43 -04:00
Jack Robison 54229282a4 Merge pull request #149 from lbryio/address-is-mine
Add address_is_mine() API method
2016-09-06 13:09:08 -04:00
David Amrhein 514575d0f1 Fix Shortcuts icons 2016-09-03 14:39:39 -04:00
David Amrhein 756427560c Attempt to fix CannotListenError 2016-09-02 16:04:27 -04:00
Alex Liebowitz 900eeb177a Add address_is_mine() API method 2016-09-02 01:27:30 -04:00
David Amrhein e67fa3cadc Merge remote-tracking branch 'origin/master' 2016-09-01 10:31:20 -04:00
Dave-A c7c32fd4dc Merge pull request #12 from lbryio/master
Update
2016-09-01 10:30:23 -04:00
Jack 9471748f4c Bump version: 0.4.2 → 0.4.3 2016-09-01 01:35:44 -04:00
Jack Robison 920b1868bb Merge pull request #148 from lbryio/windows-setup
sign msi on appveyor, add command line to msi, remove leveldb requirement
2016-09-01 01:25:02 -04:00
Jack Robison eed7c54aaf Merge pull request #146 from kaykurokawa/fix_lbryum_getbalance
Use new exclude_claimtrietx option in lbryum getbalance command, fix maturity use
2016-09-01 01:24:08 -04:00
Jack 36e58a33ad delete old windows setup.py 2016-09-01 01:09:38 -04:00
Jack 93e8ff749c another try 2016-09-01 00:47:38 -04:00
Jack 6cb22444ac try fixing signtool again 2016-09-01 00:40:06 -04:00
Jack 78df101473 add quotes 2016-09-01 00:29:57 -04:00
Jack 840508b8ba add signtool to the env path 2016-09-01 00:19:02 -04:00
Jack 2761254790 fix pfx decryption 2016-08-31 23:54:00 -04:00
Jack f47d174f0b fix file 2016-08-31 23:09:01 -04:00
Jack c50fbc7935 fix path 2016-08-31 23:02:07 -04:00
Jack 75f23a1723 sign msi 2016-08-31 23:00:16 -04:00
Jack f0aa31f9e8 unused import 2016-08-31 20:45:49 -04:00
Jack aaf59218da comment out migrate0to1 import 2016-08-31 20:41:29 -04:00
Jack 0203ef66ea build windows in setup.py, drop leveldb requirement 2016-08-31 20:20:07 -04:00
Jack Robison 9c66108dc5 Merge pull request #142 from lbryio/skip-dust
skip dust transactions
2016-08-30 20:29:56 -04:00
kkurokawa 558da0a027 Using new exclude_claimtrietx option in lbryum's getbalance command. Fixing incorrect use of maturity. 2016-08-29 22:33:10 -04:00
Jack 1d39ec0317 Bump version: 0.4.1 → 0.4.2 2016-08-29 16:41:48 -04:00
Jack Robison 281b6468a0 Merge pull request #145 from lbryio/appveyor
build msi in appveyor
2016-08-29 16:39:46 -04:00
Jack Robison 9ba6da1f4d set development version to false 2016-08-29 16:28:49 -04:00
Jack bde7a7fed6 use normal lighthouse servers 2016-08-29 16:27:45 -04:00
David Amrhein 7e021746e4 Merge remote-tracking branch 'origin/master' 2016-08-29 16:22:16 -04:00
Dave-A a61bb5f528 Merge pull request #11 from lbryio/master
Update
2016-08-29 16:21:55 -04:00
Jack 6a5289b2a2 add master to branches 2016-08-29 15:50:54 -04:00
Jack 081a5df82c notifications 2016-08-29 15:24:55 -04:00
Jack 3e77ec8008 add slack notifications 2016-08-29 15:21:06 -04:00
Jack b0b222afa8 add errback 2016-08-29 14:55:26 -04:00
Jack d85d6f627d Merge branch 'master' into appveyor 2016-08-29 14:48:05 -04:00
Jack 9505f93da6 skip reflector test on windows 2016-08-29 14:47:54 -04:00
Jack 1671034330 fix json bug 2016-08-29 14:33:56 -04:00
Jack 3f0ca563e4 json encode metadata just before sending it 2016-08-29 14:26:20 -04:00
Jack ffa2a5e8fd skip pylint for now 2016-08-29 13:48:45 -04:00
Jack c3633895d0 Merge branch 'master' into appveyor 2016-08-29 13:46:08 -04:00
Jack c15af0a63a Bump version: 0.4.0 → 0.4.1 2016-08-29 13:44:46 -04:00
Jack 56e2d77039 fix _send_name_claim json 2016-08-29 13:44:18 -04:00
Jack 2cf6fa14a1 pylint path 2016-08-29 00:34:24 -04:00
Jack c374725255 test_script 2016-08-28 23:11:24 -04:00
Jack babfb6b8d6 add . to appveyor file, try tests 2016-08-28 23:05:40 -04:00
Jack 2ffa8ec7d0 use yml generated from appveyor 2016-08-28 19:26:37 -04:00
Jack b516119224 build appveyor branch only 2016-08-28 19:23:16 -04:00
Jack c0e6bb97d8 remove lbrynet_gui include 2016-08-28 17:33:30 -04:00
Jack cc30223ff1 don't cd to lbry in init 2016-08-28 16:42:07 -04:00
Jack 53c8748942 try again 2016-08-28 16:20:28 -04:00
Jack b3a8966e71 fix path 2016-08-28 15:42:14 -04:00
Jack 74800a0767 download lbrynet tar.gz 2016-08-28 15:34:04 -04:00
Jack cadd1f993d clone lbryio/lbry repo in appveyor build script 2016-08-28 15:10:29 -04:00
David Amrhein d3807cb86c Python script for win registry 2016-08-28 14:33:46 -04:00
Dave-A 735e9195b5 Merge pull request #10 from lbryio/master
Update
2016-08-28 14:14:08 -04:00
Jack a6619a628e flag to use development lighthouse servers 2016-08-28 05:25:48 -04:00
Jack 456572f87c Bump version: 0.3.22 → 0.4.0 2016-08-28 05:01:49 -04:00
Jack f024b96934 allow int fee amount 2016-08-28 03:09:36 -04:00
Jack Robison 25c71bcd98 Merge pull request #140 from lbryio/reflect-blobs
reflect_all_blobs daemon command
2016-08-28 01:33:13 -04:00
Jack Robison 319e7ecc61 Merge pull request #131 from lbryio/content_type
change content-type to content_type, refactor metadata validation
2016-08-28 01:29:51 -04:00
Jack Robison d9ece5e96a Merge pull request #143 from lbryio/sandbox-displayed-files
prevent lbry files opened through the ui from making api calls
2016-08-28 01:28:56 -04:00
Jack Robison 4bfe4450b1 Merge pull request #144 from lbryio/pr/141
Add windows URI Handling
2016-08-28 01:28:12 -04:00
Jack f9f12ffb61 move uri handler to packaging/uri_handler 2016-08-28 01:10:28 -04:00
Jack 63f81d1033 prevent a lbry files opened through the ui from making api calls 2016-08-27 23:16:48 -04:00
Jack 8895d21abf skip dust transactions 2016-08-27 02:42:20 -04:00
David Amrhein 2ac1ebeaa0 Temp solution for distributing the regedit for windows URI handling 2016-08-27 01:37:25 -04:00
David Amrhein 25437602d7 Add URI handling as an arg 2016-08-27 01:32:28 -04:00
Jack Robison 8159c71950 update os x instructions 2016-08-26 22:13:04 -04:00
Jack aa66af7ba6 fix tests 2016-08-26 20:22:11 -04:00
Jack be3633622c whoops 2016-08-26 20:13:10 -04:00
Jack 31655a641c fix setup.py 2016-08-26 19:59:17 -04:00
Jack b235f6fc6b send all blobs to reflector 2016-08-26 19:58:53 -04:00
Jack Robison 76b297b173 Merge pull request #139 from lbryio/get_all_verified_blobs
Get all verified blobs
2016-08-26 19:45:49 -04:00
Jack Robison 1e5cff1d8b Merge pull request #130 from lbryio/osx-command-line-install
add os x install script to replace out of date and non existent one in lbry-setup
2016-08-26 16:08:08 -04:00
Jack Robison 4e2296d5f6 Merge pull request #115 from lbryio/get-stream-timeout
raise Exception on GetStream timeout
2016-08-26 16:07:42 -04:00
Dave-A 18310632eb Merge pull request #9 from lbryio/master
Update
2016-08-26 10:16:46 -04:00
Jack 80ed8d9869 Bump version: 0.3.21 → 0.3.22 2016-08-26 00:56:06 -04:00
Jack 2a2658d440 fix Access-Control-Allow-Origin problem 2016-08-26 00:50:31 -04:00
Jack 710e549c03 add get_all_verified_blobs 2016-08-26 00:32:33 -04:00
Jack 47fae977dd fix Access-Control-Allow-Origin problem 2016-08-25 23:39:05 -04:00
Alex Grintsvayg 0111919975 jack fixed the unicode bug here 2016-08-25 19:09:55 -04:00
Jack 80837cb040 Merge branch 'master' into content_type
fix conflicts:
	lbrynet/lbrynet_daemon/LBRYPublisher.py
2016-08-25 17:59:20 -04:00
Jack 46153a8991 second shot at pylint error 2016-08-23 02:25:32 -04:00
Jack a9fccfd4ce fix pylint error 2016-08-23 02:17:32 -04:00
Dave-A dfb57e9438 Merge pull request #8 from lbryio/master
Update version
2016-08-23 02:17:29 -04:00
Jack eda2731702 Bump version: 0.3.20 -> 0.3.21 2016-08-23 01:59:50 -04:00
Jack Robison f47eac5935 Merge pull request #136 from DaveA50/master
Add Windows support
2016-08-23 01:53:50 -04:00
David Amrhein 3bb4dece5a Use main lbry repo for gmpy wheel url 2016-08-23 01:26:35 -04:00
David Amrhein 0504e231a1 Use main lbryum repo 2016-08-23 01:16:04 -04:00
David Amrhein 6097e14617 Make systray thread a daemon to close on api.stop() 2016-08-23 01:05:07 -04:00
David Amrhein 0b149ce5f1 Remove old osx imports 2016-08-22 21:21:56 -04:00
Dave-A 79384e7bf8 Merge pull request #7 from lbryio/master
Update
2016-08-22 20:02:35 -04:00
Jack Robison 01eb79cac2 Merge pull request #135 from lbryio/lbrycrdwallet-updateclaim-fix
LBRYcrdWallet update claim fix
2016-08-22 19:58:28 -04:00
Jack 128b32c62c even less noisy logs 2016-08-22 19:56:42 -04:00
David Amrhein c1233bd26b Make 'rb' file mode windows specific 2016-08-22 19:55:08 -04:00
David Amrhein e23f67def6 File handle for publishing on windows needs to handle binary data as 'rb' 2016-08-22 19:24:57 -04:00
Jack 95f4b29be5 noisy log 2016-08-22 18:59:17 -04:00
Jack 5727c708cf fix get_transaction
previously get_tx_json was only in LBRYumWallet
2016-08-22 18:43:52 -04:00
Jack e49f0f99a1 LBRYcrdWallet update claim fix
-fix log line that could raise an exception
-json encode value sent to lbrycrd-cli updateclaim
2016-08-22 16:57:22 -04:00
David Amrhein 84d340382b Fix for icon path when running shortcut 2016-08-21 18:58:50 -04:00
David Amrhein 5bfb067d0f Rename to just Open 2016-08-21 18:49:41 -04:00
David Amrhein d8cb62a92c Run win32 as systray app 2016-08-21 18:44:16 -04:00
Dave-A aab8cc40f9 Merge pull request #6 from lbryio/master
Update from Origin
2016-08-21 14:13:59 -04:00
Jack 9c82689f65 Bump version: 0.3.19 -> 0.3.20 2016-08-21 04:04:11 -04:00
Jack Robison ce2755467d Merge pull request #133 from sonatagreen/patch-1
spelling fix
2016-08-21 03:44:10 -04:00
Jack Robison 02b19402ae Merge pull request #116 from lbryio/reflector
Backend for pushing newly created files to a designated server
2016-08-21 03:43:03 -04:00
Jack 80af66e852 Merge branch 'master' into reflector
# Conflicts:
#	lbrynet/core/log_support.py
2016-08-21 01:12:29 -04:00
Jack 15cfa45646 remove duplicate function 2016-08-21 00:58:25 -04:00
Jack Robison ac63849b49 Merge pull request #132 from lbryio/fix-get-history
fix get_history in lbrycrdwallet
2016-08-21 00:07:41 -04:00
Jack Robison 02142e352b Merge pull request #127 from lbryio/disable-more-noisy-loggers
reduce logging to sub fire hazard levels
2016-08-21 00:07:19 -04:00
Jack 3d92413372 debug logging 2016-08-20 23:47:41 -04:00
Jack 5022ed2acc blob_completed before moving to next one
otherwise blob files download to the blobfiles directory, but they are
never recorded in blobs.db
2016-08-20 23:47:12 -04:00
Sonata Green 31b9d22649 spelling fix 2016-08-19 21:12:02 -05:00
Jack 7a54894015 debugging stuff 2016-08-19 02:41:23 -04:00
Jack 7100345a09 Merge branch 'master' into reflector 2016-08-18 22:42:27 -04:00
Jack 929db15214 fix get_history in lbrycrdwallet 2016-08-18 22:15:49 -04:00
Jack d9455926c5 Merge branch 'master' into content_type 2016-08-18 21:56:55 -04:00
David Amrhein ba5cd741e1 Testing copying instead of moving upload file to avoid access error 2016-08-18 10:45:03 -04:00
Dave-A 8c94b6243e Merge pull request #5 from lbryio/master
Update to 0.3.19
2016-08-18 10:38:07 -04:00
Jack f6473e3383 Bump version: 0.3.18 -> 0.3.19 2016-08-18 06:41:17 -04:00
Jack Robison 3cd9d6a604 Merge pull request #129 from kimihiro64/patch-2
Proper no-cache for web-ui index.html
2016-08-18 06:39:11 -04:00
Jack 142281777e stop heartbeat looping call on shutdown 2016-08-18 06:37:57 -04:00
Jack 8e7b8af180 call blob_completed when is_validated is true 2016-08-18 06:23:27 -04:00
Jack c046cd02d5 call blob_completed after receiving blob 2016-08-18 05:58:13 -04:00
Jack f5508fcdf1 typo 2016-08-18 05:36:17 -04:00
Jack adc2eab6da log request dict 2016-08-18 05:36:01 -04:00
Jack 702698d8aa stop heartbeat looping call on shutdown 2016-08-18 05:25:23 -04:00
David Amrhein 73dbe1de37 Fix relative pathing for cacert for win builds 2016-08-18 01:25:48 -04:00
Jack c5706fa28b fix imports in tests 2016-08-18 00:38:38 -04:00
Jack c906a9571e add service identity 2016-08-18 00:31:05 -04:00
Jack 038712f431 install script 2016-08-18 00:30:57 -04:00
David Amrhein ed2d2c6f86 Ignore msi_data shortcuts, which didn't work anyway 2016-08-18 00:13:31 -04:00
kimihiro64 e7158e5a4b Fix editor spacing again 2016-08-17 23:02:38 -05:00
kimihiro64 c66df259a6 fix editor spacing 2016-08-17 23:02:00 -05:00
kimihiro64 0b8437fa12 Proper no-cache for web-ui index.html 2016-08-17 22:34:20 -05:00
Jack a32b6ebf82 add reflect jsonrpc command 2016-08-17 21:33:41 -04:00
Jack edbda1696e logging fire hazard 2016-08-17 20:24:48 -04:00
Jack ece218785c reduce logging to sub fire hazard levels 2016-08-17 20:20:03 -04:00
Jack cc1b3609a1 Merge branch 'master' into reflector
# Conflicts:
#	lbrynet/__init__.py
2016-08-17 19:01:38 -04:00
Alex Grintsvayg 0841aad07a fix double-tab bug on linux 2016-08-17 14:30:56 -04:00
David Amrhein 30b04ed0c2 Change naming conventions 2016-08-17 14:25:02 -04:00
David Amrhein 90b28b3dbd Pass WindowsError when moving uploaded file 2016-08-17 14:15:55 -04:00
Jack 0f7ca8f24e move metadata and validator classes into their own directory 2016-08-17 12:28:15 -04:00
David Amrhein 726d85ebd0 Remove blindrepeater.yapsy-plugin from build 2016-08-17 12:13:47 -04:00
David Amrhein c524de9198 Update protobuf out of beta 2016-08-17 12:12:28 -04:00
Dave-A 2fbf03d8ca Merge pull request #4 from lbryio/master
Update from lbry
2016-08-17 12:03:08 -04:00
Jack 68b8112af0 Merge branch 'master' into content_type 2016-08-17 03:04:14 -04:00
Jack 975e83a015 Bump version: 0.3.17 -> 0.3.18 2016-08-17 02:39:48 -04:00
Jack 1ddb02dbb7 speed up search
get results for get_est_cost from lighthouse instead of manually trying
unless force=True
2016-08-17 02:38:59 -04:00
Jack Robison 1159204ed8 Merge pull request #126 from lbryio/pretty-dmg
Make a pretty dmg
2016-08-17 01:36:44 -04:00
Jack Robison a5212e7741 Merge pull request #121 from lbryio/save-settings
Save settings
2016-08-17 01:34:50 -04:00
Jack Robison 1932fd72e3 Merge pull request #123 from lbryio/add-heartbeat
Add basic analytics api and heartbeat event
2016-08-17 01:34:31 -04:00
Jack Robison c061fe091b Merge pull request #122 from lbryio/add-debug-logging
Add debug logging
2016-08-17 01:33:07 -04:00
Jack Robison 3621983a4d Merge pull request #119 from lbryio/update-protobuf
Update protobuf
2016-08-17 01:31:48 -04:00
Dave-A b6656690ca Merge pull request #3 from lbryio/master
Update from lbry
2016-08-16 21:13:24 -04:00
David Amrhein 0c9e82ee6b Update appveyor to DaveA50 git repo links 2016-08-16 20:04:11 -04:00
David Amrhein b8ebb1faf1 Convert to lbryum github repo 2016-08-16 19:58:47 -04:00
David Amrhein 690575193f Add gmpy windows wheel 2016-08-16 18:01:18 -04:00
David Amrhein 9e9cf63c90 Add appveyor.yml build script 2016-08-16 17:54:36 -04:00
Job Evers-Meltzer f5dc213e53 need to not be in build dir 2016-08-16 15:16:55 -05:00
Job Evers-Meltzer 138e9c1486 throw more things at wall 2016-08-16 15:03:41 -05:00
Job Evers-Meltzer 7053c0dc93 argh. so much hate 2016-08-16 14:52:08 -05:00
Job Evers-Meltzer ee8a653b13 argh. so much hate 2016-08-16 14:43:49 -05:00
Job Evers-Meltzer 71fad7e97f also need wheel 2016-08-16 13:41:04 -05:00
Job Evers-Meltzer 04740eb928 need to detect if ON_TRAVIS earlier 2016-08-16 13:24:38 -05:00
Job Evers-Meltzer 8c63d58c96 tmp: debug travis 2016-08-16 13:21:38 -05:00
Job Evers-Meltzer 991672d423 move pyobj building earlier 2016-08-16 13:16:25 -05:00
Job Evers-Meltzer 55ce38cc87 temp: debug travis 2016-08-16 12:52:55 -05:00
Job Evers‐Meltzer 96f3390407 Merge pull request #120 from lbryio/defer-later
replace callLater with deferLater
2016-08-16 12:36:43 -05:00
Job Evers-Meltzer aa2426d4e4 ignore DS_Store files 2016-08-16 12:26:19 -05:00
Job Evers-Meltzer d0e3e13852 ignore osx build artifacts 2016-08-16 12:25:35 -05:00
Job Evers-Meltzer a2a1e30d68 remove unnecessary libgmp 2016-08-16 12:25:18 -05:00
Job Evers-Meltzer a6f08b9e0c Prettifies the osx dmg
Adds a new background image, along with precisely located icon files.
Also updates the icon files to have white backgrounds. The icon files were
created using the 256px icon used in the ubuntu build.
2016-08-16 12:22:14 -05:00
David Amrhein e70842a38c Better fix for finding protobuf namespace 2016-08-15 23:09:46 -04:00
David Amrhein 2af98d7d42 Add loggly and request futures to includes 2016-08-15 20:48:27 -04:00
David Amrhein f28eee34e8 Merge remote-tracking branch 'origin/master' 2016-08-15 20:31:59 -04:00
Dave-A 208610b645 Merge pull request #2 from lbryio/master
Update from base
2016-08-15 20:31:31 -04:00
David Amrhein e38f8076bc Create site-package protobuf __init__ if needed 2016-08-15 19:40:32 -04:00
David Amrhein 6180e1a48b Update to build and distribute >v0.3.12 2016-08-15 19:19:26 -04:00
Jack e8fe1cd14d more test fixes 2016-08-15 18:39:09 -04:00
Jack 75d1fa585c fix tests 2016-08-15 18:30:14 -04:00
Jack 072ff4b02e fix tests 2016-08-15 18:13:36 -04:00
Jack d8c4e2a72e whoops 2016-08-15 17:54:43 -04:00
Jack 66396de6a3 remove as_json and serialize 2016-08-15 17:41:38 -04:00
Jack 1ca9f575ef decorator prettiness 2016-08-15 17:07:50 -04:00
Jack 272e074a15 change content-type to content_type, make changing metadata easier 2016-08-14 01:00:49 -04:00
Job Evers 9a738c3783 Add basic analytics api and heartbeat event 2016-08-12 13:37:26 -05:00
Jack 6b4519e959 squelch very verbose dht error log 2016-08-12 10:46:05 -05:00
Job Evers-Meltzer 9b1d5213a6 improve logging 2016-08-12 10:44:37 -05:00
Job Evers-Meltzer f9ed9a4505 fixup logging some more 2016-08-12 10:42:31 -05:00
Job Evers-Meltzer 3ed7ce5630 Ensure settings are actually committed
Without committing or closing the db, settings were not being saved.

Also, adds a run_in_thread decorator to simplify the code.
2016-08-12 10:08:21 -05:00
Job Evers 63cd8343c4 replace callLater with deferLater 2016-08-11 13:41:42 -05:00
Jack 2d5f8aed24 whitespace 2016-08-11 12:38:10 -04:00
Jack b8d23d0965 add announce_all_blobs_to_dht 2016-08-11 12:36:13 -04:00
Job Evers-Meltzer f4ef92e653 fixup logging some more 2016-08-11 05:37:45 +00:00
Job Evers-Meltzer c7430f4ae9 better more logging on the server side 2016-08-11 05:25:45 +00:00
Job Evers-Meltzer 623fda3087 improve logging 2016-08-11 05:16:50 +00:00
Job Evers-Meltzer 80f27f2b29 Revert "more logging on the server side"
This reverts commit f83daa5a22.
2016-08-11 05:11:18 +00:00
Jack defbd73b4f squelch very verbose dht error log 2016-08-11 01:07:51 -04:00
Jack 5285982515 resolve reflector ip 2016-08-11 01:07:51 -04:00
Job Evers-Meltzer f83daa5a22 more logging on the server side 2016-08-11 04:45:25 +00:00
Job Evers-Meltzer 0a07c8f131 log on closed/lost connection 2016-08-10 19:39:28 -05:00
Job Evers-Meltzer 35481a92f5 actually cleanup patched time.time 2016-08-10 19:36:52 -05:00
Job Evers-Meltzer 0995d864e8 connectTCP wants an ip address, not a url 2016-08-10 19:09:20 -05:00
Job Evers-Meltzer 1a42e43573 log all the things 2016-08-10 19:06:26 -05:00
Jack 132aa569fe move run_reflector_server into settings file and add debug lines 2016-08-10 16:51:46 -04:00
Job Evers-Meltzer e93a787bd6 update protobuf out of beta 2016-08-10 09:28:13 -05:00
Job Evers-Meltzer d6f902653f put import in alphabetical order 2016-08-10 08:35:15 -05:00
Job Evers-Meltzer d172d43ddf add reflector functional test to travis 2016-08-10 08:29:44 -05:00
Job Evers-Meltzer 67909724e7 fix up import paths 2016-08-10 07:44:41 -05:00
Jack 1c88cbc5f6 Merge branch 'master' into reflector 2016-08-10 02:31:28 -04:00
Jack 3fb4831804 raise exception when curr_pos > 100 and json fails to decode 2016-08-10 02:30:41 -04:00
Jack b65570fb16 drop connection on response longer than 100 bytes that can't' be decoded 2016-08-09 19:53:13 -04:00
Jack 871e6e6f64 remove files that didn't fix test 2016-08-09 17:53:34 -04:00
Jack 0cd92a96c0 add reflector files to tests directory 2016-08-09 17:46:25 -04:00
Jack ff31ddd2d5 Bump version: 0.3.16 -> 0.3.17 2016-08-09 14:40:18 -04:00
Jack e9f5b43785 fix exceptions 2016-08-09 14:29:59 -04:00
Jack 36b41081ab Bump version: 0.3.15 -> 0.3.16 2016-08-09 12:41:11 -04:00
Jack Robison 1092313892 Merge pull request #118 from lbryio/add-LBRYDaemonRequest
add LBRYDaemonRequest to os x app
2016-08-09 12:35:08 -04:00
Jack Robison ce605cb8f3 Merge pull request #117 from lbryio/update-lbrycrdd
use most recent lbrycrdd release on github
2016-08-09 12:32:40 -04:00
Jack 0733d885fb start reflector upload in publish 2016-08-09 12:26:34 -04:00
Jack ef6fe3d1d9 add variable to enable running reflector server 2016-08-09 12:18:46 -04:00
Jack 82a4fea81a fix log statement 2016-08-09 12:12:40 -04:00
Jack 14a0252819 hooking up reflector plumbing 2016-08-09 12:07:26 -04:00
Job Evers-Meltzer 849aeb65f0 use most recent lbrycrdd release on github 2016-08-09 11:03:40 -05:00
Jack a99dc6e93d add LBRYDaemonRequest to os x app 2016-08-09 11:19:01 -04:00
Alex Liebowitz 1a5fcc1348 Add requests_futures dependency 2016-08-09 07:59:27 -04:00
Jack c9d88f5628 Bump version: 0.3.14 -> 0.3.15 2016-08-09 03:29:51 -04:00
Jack Robison 0910711aca Merge pull request #111 from lbryio/update-claim
claim updates
2016-08-09 03:29:12 -04:00
Jack 6a1a9ef6bc encode claim id to be shown in log 2016-08-09 03:25:20 -04:00
Jack ef1c059079 clean up log line 2016-08-09 03:18:44 -04:00
Jack a05ab37373 Merge branch 'master' into update-claim
# Conflicts:
#	lbrynet/lbrynet_daemon/LBRYDaemon.py
2016-08-09 03:09:06 -04:00
Jack 7ea88cfd22 crudely fix loggly import problem 2016-08-09 03:01:33 -04:00
Jack Robison 8d14fe6a90 Merge pull request #113 from lbryio/log-to-loggly
Log to loggly
2016-08-09 02:30:55 -04:00
Jack Robison 49a05b42f9 Merge pull request #114 from lbryio/no-unused-imports
Remove all unused imports; add check to pylint
2016-08-09 02:30:49 -04:00
Jack b6da1d7769 decode and reverse claim id for updateclaim in lbryum 2016-08-09 01:26:55 -04:00
Jimmy Kiselak 7e2ad58edd get reflector client and server working; add func test to prove it 2016-08-09 00:59:50 -04:00
Jack 7ad0d98404 Merge branch 'master' into update-claim 2016-08-08 21:39:41 -04:00
Jack b7052bba95 rename variable 2016-08-08 21:34:05 -04:00
Jack 4a5618bcc4 get update claim working with lbryumwallet 2016-08-08 21:32:39 -04:00
Job Evers-Meltzer c1850fe604 raise Exception on GetStream timeout 2016-08-08 16:26:33 -05:00
Job Evers-Meltzer fccdf32c9f Remove all unused imports; add check to pylint 2016-08-08 15:42:35 -05:00
Jack c474411e1b delete bad records that got into name_metadata 2016-08-08 16:35:55 -04:00
Job Evers-Meltzer 9033eac5b1 Remove commented code 2016-08-08 15:07:25 -05:00
Job Evers-Meltzer 0ba0426591 Refactor lbrycrdd path function for osx 2016-08-08 15:06:48 -05:00
Job Evers-Meltzer 9a9b540701 Move some ServerRequestHandling logging to debug level 2016-08-08 15:04:50 -05:00
Job Evers-Meltzer b8ddbb9a26 Add loggly logger
It configured twice, once right at startup without the lbry id and then
after the settings database is available it gets reconfigured with the
lbry id and a session id.
2016-08-08 15:02:36 -05:00
Job Evers-Meltzer bc3c2b96ed disable noisy loggers 2016-08-08 14:59:30 -05:00
Job Evers-Meltzer 622dc87d84 Remove log handlers with the same name
Also renames functions to follow underscore convention
2016-08-08 14:57:49 -05:00
Jack 18edbdf404 squelch resolve_name logging 2016-08-08 14:58:31 -04:00
Job Evers-Meltzer 02a3eab693 add loggly dependency 2016-08-08 11:39:11 -05:00
Job Evers-Meltzer dded868a80 remove leveldb requirement 2016-08-08 11:37:01 -05:00
Jack 6fe878bdb9 make load_from_hex a class method 2016-08-08 03:18:32 -04:00
Jack 505ece9e36 add txid param to get_claim_info and update todo note 2016-08-08 02:37:08 -04:00
Jack 72acfa1e0c add support_claim 2016-08-08 02:32:56 -04:00
Jack 02cd8b9a44 fix callback 2016-08-08 02:08:09 -04:00
Jack f32250f431 whoops 2016-08-08 01:54:41 -04:00
Jack 1523cad044 refactor get_claim_info 2016-08-08 01:46:19 -04:00
Jack ed55353700 bug fix 2016-08-08 01:43:28 -04:00
Jack 8d3bd45102 get_claims_for_name daemon function 2016-08-08 01:43:10 -04:00
Jack f69753f6a0 add get_claims_for_name 2016-08-08 01:42:37 -04:00
Jimmy Kiselak b7e2e87ac1 fill in much of the skeleton for the reflector protocol 2016-08-07 22:33:40 -04:00
Jack 3b98508986 clean up claim_name function 2016-08-07 16:13:47 -04:00
Jack 36ebfc42d9 update from master 2016-08-06 21:44:14 -04:00
Jack ae5f55ff65 Bump version: 0.3.13 -> 0.3.14 2016-08-06 20:07:28 -04:00
Jack Robison 01131da5b7 Merge pull request #99 from lbryio/refactor-refresh
Add DownloadNameHelper and ResolveNameHelper
2016-08-06 20:06:43 -04:00
Jack 66ae865de0 don't make new lbry file if publish is given a full metadata dict, remove update_claim() 2016-08-06 20:02:51 -04:00
Job Evers-Meltzer 8c50f08fec refactor _start_download 2016-08-05 11:37:24 -05:00
Job Evers-Meltzer e69de47daf refactor wait_on_write stopping condition 2016-08-05 11:37:24 -05:00
Job Evers-Meltzer acef10e848 fix merge oversight 2016-08-05 11:37:24 -05:00
Job Evers edb9d50d9e fix typo in docstring 2016-08-05 11:37:24 -05:00
Job Evers b989d4f6be Bug fix: correct the check on wait_or_get_stream
I reversed the check on lbry_file. Compare to the original code:
https://github.com/lbryio/lbry/blob/e67f55bd7447cbee730d895b7f35cf53cf69dba5/lbrynet/lbrynet_daemon/LBRYDaemon.py#L1119
2016-08-05 11:37:24 -05:00
Job Evers 84dd5c0eca Revert "revert get() refactor"
This reverts commit 5eb7c64ec3.
2016-08-05 11:37:24 -05:00
Job Evers-Meltzer db8daf20b7 quiet noisy log statement 2016-08-05 05:50:24 -05:00
Jack cb985b5a03 log conditions of lbry_file failing to start 2016-08-04 23:27:44 -04:00
Jack 28c4e38272 make line neater 2016-08-04 23:27:25 -04:00
Jack b29377dc2b keep claim id history 2016-08-04 23:27:10 -04:00
Jack ca09592a72 make logs easier to scroll through 2016-08-04 23:10:15 -04:00
Jack b0e14fae43 update over existing claim in publish() 2016-08-04 23:08:54 -04:00
Jack dfa68f126b get update_name in lbrycrdwallet working 2016-08-04 20:43:39 -04:00
Jack 4570f68727 add claim_id to lbry_file and to get_claim_info 2016-08-04 18:44:12 -04:00
Jack 4705acba55 use lbryum updateclaim 2016-08-04 02:22:10 -04:00
Jimmy Kiselak 1dea20a358 include sd blob hashes in list of blob hashes to send via reflector 2016-08-03 23:11:29 -04:00
Jimmy Kiselak 191f661b35 skeleton for reflector protocol 2016-08-03 23:03:14 -04:00
Jack f3ab38ba8f Bump version: 0.3.12 -> 0.3.13 2016-08-03 18:01:18 -04:00
Jack Robison 2d0a892e70 Merge pull request #110 from lbryio/lighthouse-client
Lighthouse client
2016-08-03 18:00:04 -04:00
Jack Robison 00eefe87d7 Merge pull request #109 from lbryio/download-descriptor
add download_descriptor
2016-08-03 17:59:52 -04:00
Jack f68fd96555 break out sd timeout into a variable 2016-08-03 17:57:26 -04:00
Jack fc50a3242f add note about request.notifyFinish 2016-08-03 17:49:13 -04:00
Jack 64e9ce2bfa comments about the upnp redirects 2016-08-03 17:44:41 -04:00
Jack e8d04b1921 specify a lighthouse server 2016-08-03 17:44:32 -04:00
Jack 9fedf14e55 request.notifyFinish 2016-08-03 12:46:46 -04:00
David Amrhein 7d535ef7d1 Make ca_path checks for frozen windows only 2016-08-03 10:16:14 -04:00
Jack 85d610bcdf add get_peers_for_hash 2016-08-03 03:16:06 -04:00
Jack 3af8b7bd0c use upnp redirects if they're already set 2016-08-03 01:27:40 -04:00
Jack 6215d85581 add LighthouseClient, announce sd hash after publishing 2016-08-02 18:34:48 -04:00
Jack 308d9f5abd add todo 2016-08-02 17:33:57 -04:00
Jack bcfabba038 fast search
use cost from lighthouse
2016-08-02 16:45:06 -04:00
Jack 69139b1987 already called error 2016-08-02 16:03:47 -04:00
Jack 3f8f9d9d73 add timeout to download_sd_blob 2016-08-02 02:54:04 -04:00
Jack 2283579109 reorganize download_descriptor 2016-08-01 23:15:32 -04:00
Jack 1e0af94ba2 add download_descriptor 2016-08-01 22:31:49 -04:00
David Amrhein dd631328e0 Update windows distribution setup 2016-07-30 22:54:51 -04:00
David Amrhein cca2a57c69 Set cacert environ path for windows distribution 2016-07-30 22:52:49 -04:00
David Amrhein bbfbcacc6a Add windows icons for packaging 2016-07-30 22:10:30 -04:00
David Amrhein 3f95c1a9d5 Skip db migration in windows 2016-07-30 17:09:38 -04:00
David Amrhein 7bd05e9b16 Handle creation of settings directory in windows 2016-07-30 17:09:15 -04:00
Dave-A 10742d7c51 Merge pull request #1 from lbryio/master
Update from original
2016-07-30 16:48:25 -04:00
Jack bbf7940e45 Bump version: 0.3.11 -> 0.3.12 2016-07-29 23:44:04 -04:00
Jack Robison f5af71f4e6 Merge pull request #107 from lbryio/metadata-version
Metadata versioning, publishing things, and USD denominated key fees
2016-07-29 23:43:11 -04:00
Jack 1ffcd67c9d specify exception 2016-07-29 23:05:01 -04:00
Jack cf8592ba12 docstring and todo 2016-07-29 22:49:50 -04:00
Jack 1e1d036590 clean log line 2016-07-29 22:44:38 -04:00
Jack 64afb6d305 clean up get_est_costs in search 2016-07-29 22:43:50 -04:00
Jack 20a9e018f7 revert updates to git 2016-07-29 22:36:30 -04:00
Jack a574c33ac6 multiple search servers 2016-07-29 21:37:44 -04:00
Jack d21e458fe1 remove debug line 2016-07-29 02:42:58 -04:00
Jack bb3ba53c6e fix search results 2016-07-28 22:39:34 -04:00
Jack ba2e4df76b remove debug stuff 2016-07-28 21:31:19 -04:00
Jack 33e7e4bee3 fancy search 2016-07-28 21:25:37 -04:00
Jack b83dce2cb3 no tricky characters 2016-07-28 18:48:29 -04:00
Jack 6821676853 handle decimals in jsonrpc return 2016-07-28 18:12:54 -04:00
Jack 52024986ee derp 2016-07-28 16:32:59 -04:00
Jack 2e3342d20d fix test 2016-07-28 16:27:16 -04:00
Jack 419aa1d78f fix est data cost / search 2016-07-28 16:12:20 -04:00
Jack de4ddd578f improve readability of metadata and fee requirements 2016-07-28 15:24:20 -04:00
Jack 5dd68a7ed4 add get_claim_info, clean up _get_stream_info_from_value 2016-07-28 14:55:17 -04:00
Jack 881168dc7a fix too many args 2016-07-28 13:16:33 -04:00
Job Evers-Meltzer 295e427abe bug fix: check if fee is in metadata before loading 2016-07-28 12:04:25 -05:00
Job Evers 67c5d15f51 stupid typo 2016-07-28 11:46:33 -05:00
Job Evers 39a7f37f35 muck with pending_claims... still not sure this is correct 2016-07-28 11:43:47 -05:00
Job Evers 15d276b0bc fix case where to and from currencies are the same 2016-07-28 11:43:20 -05:00
Jack 4387025c48 another still 2016-07-28 06:07:33 -04:00
Jack 52c20f02c7 one more 2016-07-28 06:03:47 -04:00
Jack 413dab8094 whoops 2016-07-28 05:59:10 -04:00
Jack 6b1d11a04d fix rate in test 2016-07-28 05:53:31 -04:00
Jack 5441379f38 fix LBRYExchangeRateManager test 2016-07-28 05:49:31 -04:00
Jack 52d954c64e don't spam the market apis 2016-07-28 05:34:20 -04:00
Jack 415495fc16 LBRYExchangeRateManager 2016-07-28 05:30:13 -04:00
Jack 4223298634 test wrong metadata version 2016-07-28 02:49:35 -04:00
Jack e058c71019 test data 2016-07-28 02:46:46 -04:00
Jack 12b2bbfa26 make sure claimed ver is true 2016-07-28 02:36:07 -04:00
Jack 7d22bfffa5 fee tests 2016-07-28 02:30:37 -04:00
Jack 4ade80c8a9 clean up metadata and fee 2016-07-28 02:19:45 -04:00
Jack 9674381fc2 Merge remote-tracking branch 'origin/metadata-version-avec-tests' into metadata-version 2016-07-27 20:18:00 -04:00
Jack f6d91280d7 remove debug print line 2016-07-27 15:03:03 -04:00
Jack e4ebbd8f69 neater force param in resolve_name 2016-07-27 15:02:54 -04:00
Jack 7f5560de6d comment spacing and removing setLevel 2016-07-27 15:02:19 -04:00
Job Evers-Meltzer 0925171e07 bug fix: use txid attribute, not key 2016-07-27 13:59:26 -05:00
Job Evers-Meltzer 900d3ad01d Fixes errback not being callable
Small code refactoring to make it more obvious what
both the callback and errback are doing
2016-07-27 13:12:20 -05:00
Job Evers-Meltzer 6a1efedc1f some example tests. please expand on these 2016-07-27 12:05:29 -05:00
Job Evers-Meltzer 0f602e0e62 allow bittrex fee to be changed 2016-07-27 11:59:57 -05:00
Jack 3a425fc47d Merge branch 'master' into metadata-version 2016-07-27 05:07:23 -04:00
Jack 36722cbbd8 Bump version: 0.3.10 -> 0.3.11 2016-07-27 03:58:25 -04:00
Jack 9572336e96 Merge branch 'master' into metadata-version
# Conflicts:
#	lbrynet/core/utils.py
#	lbrynet/lbrynet_daemon/LBRYDaemon.py
2016-07-27 03:54:10 -04:00
Jack 4895cde474 fix conflicts 2016-07-27 03:48:32 -04:00
Jack Robison 21ca1a73cf Merge pull request #100 from lbryio/change-ui-location
web-ui files are on S3 now
2016-07-27 03:44:28 -04:00
Jack Robison 457c1c6ae6 Merge pull request #101 from lbryio/version-cmp
Properly Compare Version Strings
2016-07-27 03:43:13 -04:00
Jack Robison c1ff127101 Merge pull request #102 from lbryio/log-configure
Fix: Log configuration should not happen when a module is imported
2016-07-27 03:42:51 -04:00
Jack 5e815a5832 merge master 2016-07-27 03:22:03 -04:00
Jack 4604e9cd54 check pending claims after one is made 2016-07-27 03:00:44 -04:00
Jack ef580a7680 errback for pending of date lbry_file 2016-07-27 01:51:09 -04:00
Jack 2102776be8 another errback 2016-07-27 01:44:01 -04:00
Jack f3af7a6b77 add errback for unknown name 2016-07-27 01:39:23 -04:00
Jack 5abeceac1f abandon previous claim in update 2016-07-27 01:13:41 -04:00
Jack 22f73a0817 add force parameter to resolve_name 2016-07-26 21:46:04 -04:00
Jack 29b5aef73a call restore() after claiming name 2016-07-26 21:24:58 -04:00
Jack 3e602bce52 fix publish callback 2016-07-26 20:58:37 -04:00
Jack fe39901885 update claim when publishing to an already claimed name
-also save name_metadata for each unique claim rather than for each name
2016-07-26 20:52:05 -04:00
Jack ee9e8b365c add debug log line 2016-07-26 18:42:51 -04:00
Jack bd6fa35d4a disable verbose dht error 2016-07-26 18:19:40 -04:00
Jack 5e1fdf66b0 fix download problem 2016-07-26 18:18:34 -04:00
Jack d86e709b90 reduce min blob prices 2016-07-26 17:24:59 -04:00
Jack f82cffe882 verbose logging 2016-07-26 17:21:41 -04:00
Jack 3d88eb92c1 change default max fee to $25 2016-07-26 17:21:03 -04:00
Jack 0811ebb52d fix key payment problems 2016-07-26 17:20:51 -04:00
Jack 491d431ea5 remove price updater from daemon 2016-07-26 17:19:26 -04:00
Job Evers‐Meltzer ceeebebcad Merge pull request #106 from lbryio/no-blobs-in-logs
Have TimeoutError hexlify blobs
2016-07-26 14:57:30 -05:00
Job Evers‐Meltzer 04d766aba7 Merge pull request #105 from lbryio/bug-fixes
misc bug fixes and code cleanup
2016-07-26 14:56:50 -05:00
Job Evers‐Meltzer 3b431e180e Merge pull request #98 from lbryio/log-cleanup
Remove datetime.now() from log statements
2016-07-26 14:54:36 -05:00
Job Evers a2eab1577d Have TimeoutError hexlify blobs 2016-07-26 13:54:46 -05:00
Job Evers-Meltzer f8dd3d05dc misc bug fixes and code cleanup 2016-07-26 12:24:25 -05:00
Job Evers-Meltzer 93993e62d6 Fix: Log configuration should not happen when a module is imported
Instead, move the responsibility to the main program. Also,
each module had the same, redundant setup.
2016-07-26 12:01:03 -05:00
Job Evers‐Meltzer 483c88b958 Merge pull request #104 from lbryio/no-print
Remove print line
2016-07-26 11:36:00 -05:00
Job Evers 4a1e846958 Remove print line
This line is responsible for flooding the console with messages like
<_io.BytesIO object at 0x7f9e286e52f0>
2016-07-26 11:34:13 -05:00
Job Evers-Meltzer 927b9afe7c forgot to add the test file 2016-07-26 09:34:00 -05:00
Jack 9e8827f015 resolve name bug 2016-07-25 23:46:18 -04:00
Jack ca45f461ae spelling 2016-07-25 23:42:03 -04:00
Jack 7e5af11c2d fix key fee payment problem 2016-07-25 23:41:26 -04:00
Jack 102436274f USD denominated max_key_fee in settings 2016-07-25 22:46:04 -04:00
Jack 49c26a76b4 remove fee checking from autofetcher script 2016-07-25 22:28:01 -04:00
Jack d469ef7bf2 add googlefinance to requirements 2016-07-25 22:21:38 -04:00
Jack a5374cb876 fix currencies and LBRYFeeFormat 2016-07-25 22:17:31 -04:00
Jack a53e911a0c stop looping call when stop() is called 2016-07-25 21:48:39 -04:00
Jack c4a78a149b LBRYFee things
-move exchange rate updates to wallet, do them every half hour
-add convention checker for fees
-pay BTC or USD denominated key fees in converted LBC amount
2016-07-25 21:45:42 -04:00
Job Evers-Meltzer 3814912adf Move to core.utils and expand to check lbryum too 2016-07-25 16:09:13 -05:00
Jack 75ef652cb3 add googlefinance to setup.py 2016-07-25 17:08:56 -04:00
Job Evers-Meltzer 1eae3fe5dc pylint doesn't like distutils 2016-07-25 14:53:07 -05:00
Job Evers-Meltzer d0bc383a79 bug fix: version comparison is not lexographic.
Use distutils.version to compare version strings.
2016-07-25 12:25:10 -05:00
Job Evers-Meltzer ea5d31b530 OCD: alphabetize and group imports
I'm not the only one:
https://google.github.io/styleguide/pyguide.html?showone=Imports_formatting#Imports_formatting
2016-07-25 12:19:19 -05:00
Job Evers‐Meltzer e56ad19e81 Merge pull request #95 from lbryio/deployment-update
Updates the deployment process and the version check associated with it
2016-07-25 12:10:29 -05:00
Job Evers ed393eb75c Updates the deployment process and the version check associated with it
This commit supports steps 1 and 2 in the new workflow:
1. Change the logic in the daemon to check the github api for the latest release that is not a pre release
2. Change travis to mark all releases as pre release
3. When we are ready to stage a release we push a tag to master. Travis builds the packages and releases them
4. We manually check them
5. Remove the pre release mark when we are happy
2016-07-25 11:13:47 -05:00
Jack 2e2d309d38 BTC/USD denominated fees, metadata revisions 2016-07-25 01:40:26 -04:00
Jack 6bfb33adb2 fix version comparison 2016-07-24 22:03:32 -04:00
Job Evers-Meltzer 05138e1db1 web-ui files are on S3 now 2016-07-22 10:58:37 -05:00
Jack b65402db41 debug line 2016-07-21 16:47:47 -04:00
Jack fbef187400 Merge branch 'master' into metadata-version
# Conflicts:
#	lbrynet/conf.py
#	lbrynet/lbrynet_daemon/LBRYDaemon.py
2016-07-21 16:45:41 -04:00
Jack 25d1b8b9e5 move towards better fees and metadata 2016-07-21 16:11:14 -04:00
Job Evers ec91594083 Remove datetime.now() from log statements 2016-07-21 10:41:26 -05:00
Jack e67f55bd74 disable debug logging 2016-07-20 22:40:55 -04:00
Jack Robison 276a7f9d2a Merge pull request #92 from lbryio/file-uploads
Add support for file uploads (for Publish page)
2016-07-20 22:37:08 -04:00
Jack 4888d7a567 Bump version: 0.3.9 -> 0.3.10 2016-07-20 22:34:16 -04:00
Jack Robison 767f78315b Merge pull request #97 from lbryio/lbryum-catchup
lbryum fixes
2016-07-20 22:33:16 -04:00
Jack 4311b218fb fix auto_connect 2016-07-20 22:13:22 -04:00
Jack 89892042dd remove unused line 2016-07-20 20:57:05 -04:00
Jack a22dfb3864 cleaner logging
from https://github.com/lbryio/lbry/pull/94
2016-07-20 20:56:56 -04:00
Jack f40c42dcf0 Merge branch 'master' into lbryum-catchup 2016-07-20 20:38:42 -04:00
Jack 1d31aa9da3 Bump version: 0.3.8 -> 0.3.9 2016-07-20 20:34:41 -04:00
Jack f999073fb4 revert download refactor
to be re-merged after fixes
2016-07-20 20:34:02 -04:00
Jack 5eb7c64ec3 revert get() refactor 2016-07-20 20:30:57 -04:00
Jack 7e79818555 Revert "Merge branch 'master' into lbryum-catchup"
This reverts commit a51576b250, reversing
changes made to 0bee2689d1.
2016-07-20 20:24:49 -04:00
Jack a51576b250 Merge branch 'master' into lbryum-catchup 2016-07-20 20:17:08 -04:00
Jack 0bee2689d1 download fixes 2016-07-20 20:17:01 -04:00
Jack c232743963 fix LBRYumWallet._do_send_many 2016-07-20 20:13:44 -04:00
Job Evers‐Meltzer c2d5f092b3 Merge pull request #94 from lbryio/logging-reformat-2
use asctime in logging format
2016-07-20 18:41:54 -05:00
Jack 8e8e186de9 Merge branch 'master' into lbryum-catchup 2016-07-20 19:29:11 -04:00
Jack 55f47a2b1b lbryum catchup 2016-07-20 17:32:15 -04:00
Job Evers-Meltzer 5c27b9eb90 use asctime in logging format
Instead of manually adding datetime.now() use
asctime formatter to do it for us.
2016-07-20 07:21:12 -05:00
Job Evers‐Meltzer 1d018533d1 Merge pull request #81 from lbryio/deprecate-lbry-console
deprecate lbry-console script
2016-07-20 07:07:16 -05:00
Job Evers‐Meltzer 34b8841f5c Merge pull request #73 from lbryio/refactor-dl-name
Refactor jsonrpc_get, _download_name, _resolve_name
2016-07-20 07:07:02 -05:00
Alex Liebowitz a54166a27c Add support for file uploads (for Publish page) 2016-07-20 02:36:55 -04:00
Job Evers 4adec39dce Bump version: 0.3.7 -> 0.3.8 2016-07-18 16:00:52 -05:00
Job Evers‐Meltzer e63ae08638 Merge pull request #86 from lbryio/back-to-lbryum
Revert default wallet back to lbryum
2016-07-18 15:33:03 -05:00
Job Evers‐Meltzer f1b4c6a1ec Cache pyobjc (#87)
* cache pyobjc modules as wheels
2016-07-18 14:50:05 -05:00
Job Evers-Meltzer 78b4b3ffa8 revert default wallet back to lbryum 2016-07-18 11:30:21 -05:00
Job Evers‐Meltzer f51fd48cdb Merge pull request #85 from lbryio/update-protobuf
Update protobuf library
2016-07-18 11:26:47 -05:00
Job Evers-Meltzer 9127a816e3 Update protobuf library
Updating to protobuf-v3.0.0b3 because it
fixes a bug here https://github.com/google/protobuf/blob/e841bac4fcf47f809e089a70d5f84ac37b3883df/python/google/protobuf/internal/python_message.py#L59
where six.moves is not always available.
In particular, the system python on OSX has an old version of six
which is without six.moves.
2016-07-18 10:12:49 -05:00
Job Evers-Meltzer 1a5cd9b062 Forget about trying to set a label.
Setting a label also sets the account, but the LBRY app
only looks at the default "" account.  Marking the imported
keys would have been nice, but it doesn't work.
2016-07-18 09:14:50 -05:00
Job Evers-Meltzer a86b4c4e58 bug fix: use sys.exit() not exit() 2016-07-18 08:52:56 -05:00
Job Evers-Meltzer a384960a7b deprecate lbry-console script 2016-07-18 08:45:28 -05:00
Job Evers‐Meltzer 5817b80c94 script to migrate private keys to lbrycrd (#79)
* script to migrate private keys to lbrycrd
* handle compressed keys too
* validate private key actually gets added
* add lbrycrdd and lbrycrd-cli checks
* exit if no wallet path
* move each imported address into the default account
2016-07-17 20:17:42 -05:00
Job Evers-Meltzer 97efd63c99 deprecate lbry-console script 2016-07-17 19:19:00 -05:00
Job Evers-Meltzer 5ebcc01bcc Bump version: 0.3.6 -> 0.3.7 2016-07-17 14:10:11 -05:00
Job Evers‐Meltzer ddcc008ea6 Merge pull request #78 from lbryio/force-wallet
emergency fix: force lbrycrd to be the default wallet
2016-07-17 14:08:41 -05:00
Job Evers-Meltzer 8ec10e2304 emergency fix: force lbrycrd to be the default wallet
Ignore save settings for wallet type and use the default wallet (lbrycrd)
instead.  lbryum can be used but only if specified on the command line.
2016-07-17 13:45:26 -05:00
Job Evers-Meltzer 172f275bc7 Refactor _resolve_name.
Continue using the delegation/helper pattern for the daemon.
2016-07-16 09:24:27 -05:00
Job Evers‐Meltzer 0dac5943bb Merge pull request #74 from llllIHARWINDERIllll/patch-3
Need to have git installed
2016-07-15 13:42:21 -05:00
Jack 2dcd2b13be add LBRYMetadata.py 2016-07-15 14:09:43 -04:00
Harwinder 8743decd17 Need to have git installed.
You cant execute the code that follows after if you havent installed git already.
2016-07-15 20:01:25 +02:00
Job Evers-Meltzer a15d7ca543 Refactor _download_name
Nested functions are the devil, especially ones that
use variables from the outer scope. Refactoring
_download_name to use a helper class helps make
the scoping more explicit and will undoubtably
prevent bugs in the future.

I think this makes _download_name drastically more readable.

Also cleaned up some duplicated code and
made download_directory respect the passed in parameter
instead of being the default.
2016-07-15 12:42:26 -05:00
Job Evers-Meltzer a90029ec50 update doc string 2016-07-15 11:37:04 -05:00
Job Evers-Meltzer 563896b126 fix bug in reveal code 2016-07-15 11:33:38 -05:00
Job Evers-Meltzer 5aecd02668 Refactor jsonrpc_get.
Move parameter handling into its own function and
better use the `.get()` function for dictionaries.

Early return on the failed checks is more readable.

The lambda function in the callback was long and
hard to read so moved it out.
2016-07-15 11:32:46 -05:00
Jeremy Kauffman eef6312a17 Merge pull request #71 from llllIHARWINDERIllll/patch-2
Cant use git if its not installed
2016-07-14 16:26:23 -04:00
Harwinder 1e48f5c8f4 Cant use git if its not installed
First install needed packages then execute the task. Cant run the whole code if git isnt installed :)
2016-07-14 22:25:32 +02:00
Jeremy Kauffman 7908fa2814 Merge pull request #70 from llllIHARWINDERIllll/patch-1
Adding git to linux install command
2016-07-14 16:16:51 -04:00
Harwinder 7ffb3ea042 Added git to linux install command
Some people dont have git installed so adding it to it solves confusion for people new to linux.
2016-07-14 18:41:23 +02:00
Job Evers‐Meltzer 842dd90780 Merge pull request #66 from lbryio/add-tests-blob-handler
add tests for BlobRequestHandler and run on travis
2016-07-14 09:33:06 -05:00
kimihiro64 ee71a03f96 Revert to 2 params as isinstance cannot support 3+ (#69)
* Revert to 2 params as isinstance cannot support 3+

* Use basestring for maximum efficiency
2016-07-13 22:44:02 -05:00
Jack 5221631b40 Bump version: 0.3.5 -> 0.3.6 2016-07-12 17:30:58 -04:00
Job Evers‐Meltzer ade2b4ac0d Merge pull request #68 from lbryio/switch-slack
switch slack notification token
2016-07-12 15:25:21 -05:00
Job Evers-Meltzer 643b95c0e4 switch slack notification token 2016-07-12 14:48:13 -05:00
Jack Robison ed0f9d4fb5 Merge pull request #67 from lbryio/real-lbryum
Real lbryum
2016-07-12 15:35:54 -04:00
Jack 6f353c31cb Merge branch 'master' into real-lbryum 2016-07-11 22:04:32 -04:00
Jack 404412a950 return publish error message 2016-07-11 21:57:34 -04:00
Job Evers‐Meltzer 0e1778d8da Merge pull request #65 from lbryio/add-slack-notifications
add slack notifications to travis
2016-07-11 13:28:56 -05:00
Job Evers-Meltzer dfc0ca9525 change slack key and encrypt it 2016-07-11 12:05:27 -05:00
Job Evers-Meltzer 53bf1fe4fd add send_blob tests 2016-07-10 21:56:00 -05:00
Jack 5c05daa007 update import to new file name 2016-07-10 22:21:50 -04:00
Job Evers-Meltzer f74f075b4e add tests for BlobRequestHandler and run on travis 2016-07-10 18:48:14 -05:00
Jack 1b53e8e98c re-enable lbryum 2016-07-09 13:31:07 -04:00
Job Evers-Meltzer 4caee284bc add slack notifications to travis 2016-07-07 12:06:07 -05:00
Alex Grintsvayg 5368c91399 lbry replaces old lbrynet 2016-07-07 00:18:29 -04:00
Alex Grintsvayg 00b1881f20 dont forget to run bumpversion instead of manually bumping 2016-07-06 22:56:23 -04:00
Jack b727bd20c8 version bump 2016-07-06 22:48:38 -04:00
Jack 5a703c116f linux lbrycrdd path 2016-07-06 22:46:59 -04:00
Alex Grintsvayg 57f728f40e desc update 2016-07-06 22:29:12 -04:00
Alex Grintsvayg 74bbde0c34 fix "launch as root" bug, update package name and maintainer 2016-07-06 21:54:55 -04:00
Alex Liebowitz c58fc0781c Call xdg-open properly in jsonrpc_reveal() 2016-07-06 15:57:00 -04:00
Alex Grintsvayg 7a266d49c1 fix 2016-07-06 15:33:06 -04:00
Alex Grintsvayg 344eeeb8c0 typo 2016-07-06 15:04:41 -04:00
Alex Grintsvayg 08d50ddfc5 bundle lbrycrdd bins instead of downloading them 2016-07-06 14:58:06 -04:00
Alex Grintsvayg f83c892b13 fix lbry symlink, fix icon path, add .desktop file to bumpversion 2016-07-06 14:13:11 -04:00
Alex Grintsvayg 7dbec9d395 typo 2016-07-06 13:41:47 -04:00
Alex Grintsvayg 5171c801d9 add lbry icons, symlink into user path 2016-07-06 13:34:17 -04:00
Alex Grintsvayg 72b4a778a6 dont forget bumpversion 2016-07-06 10:09:40 -04:00
Jack Robison 0254df63f9 Merge pull request #64 from lbryio/check-for-new-ui
Check for new ui periodically
2016-07-06 04:35:06 -04:00
Jack 03c02ff74f lbrycrdd path wierdness 2016-07-06 03:37:31 -04:00
Jack 9277c3c674 add help for when things go wrong 2016-07-06 03:02:55 -04:00
Jack 34b1259dc6 move check for if daemon is running to inside of function 2016-07-06 02:20:18 -04:00
Jack 257e38de12 add command line wrapper for lbrynet-daemon calls 2016-07-06 02:17:38 -04:00
Jack 1984132eae add update_available keys to version()
as to not require making the version string comparison in whatever is
making the daemon call
2016-07-06 01:46:49 -04:00
Jack 8b1d75ea98 check every 30 minutes for a new ui version
-previously the check for a new ui version was only at startup, which
resulted in bugs persisting after they’d been fixed

-add option to skip requirements check

-try to load an existing ui if it can’t be obtained from github
2016-07-06 01:27:25 -04:00
Job Evers‐Meltzer a79af11371 Merge pull request #60 from lbryio/change-make-deb
Use patched version of make-deb
2016-07-05 15:08:42 -05:00
Job Evers 087a9b70b4 Use patched version of make-deb
Ran into a bug in make-deb that fails when there is utf-8 characters
in the git commit message. https://github.com/nylas/make-deb/issues/13.

I wrote a patch to fix the bug and am installing that version until the
make-deb maintainers incorporate it
2016-07-05 14:57:48 -05:00
Alex Grintsvayg 8b37952128 make-deb doesn't like utf8 arrow in commit message 2016-07-05 15:53:51 -04:00
Alex Grintsvayg 8336930091 Bump version: 0.3.2 → 0.3.3 2016-07-05 15:20:35 -04:00
Alex Grintsvayg d875325cee better non-root fix, restore version var 2016-07-05 15:20:30 -04:00
Alex Grintsvayg f1c5d1cb23 Bump version: 0.3.1 → 0.3.2 2016-07-05 14:30:35 -04:00
Alex Grintsvayg b55752e715 add bumpversion script 2016-07-05 14:29:24 -04:00
Alex Grintsvayg 36a99e6ec8 run lbry as non-root user 2016-07-05 14:06:41 -04:00
Jack df3c5b9a77 open conf for write in app 2016-07-04 20:47:43 -04:00
Jack 4c2ab23c1f write to the file that's supposed to be written to 2016-07-04 20:19:04 -04:00
Jack 0a3f81e364 fix download directory not existing problem 2016-07-04 20:02:38 -04:00
Jack 69c5631b25 make lbrycrddpath.conf if it doesnt exist, point to binary in app 2016-07-04 19:53:01 -04:00
Jack cca4331ac4 create lbrycrddpath.conf if it doesnt exist 2016-07-04 19:34:51 -04:00
Alex Grintsvayg 5972dbb6a7 fix lbry init script 2016-07-04 19:18:22 -04:00
Job Evers-Meltzer 794f2d8576 version bump: 0.3.0.2 2016-07-04 16:51:29 -05:00
Alex Liebowitz e35dce530a Set default download dir to ~/Downloads on Linux 2016-07-04 17:49:48 -04:00
Job Evers-Meltzer 6034bd3cb2 version bump: 0.3.0.1 2016-07-04 15:47:12 -05:00
Job Evers‐Meltzer 42c77bbb67 Merge pull request #51 from lbryio/restart-lbrycrd
add a more useful exception if auth fails on lbrycrd
2016-07-04 15:45:44 -05:00
Job Evers‐Meltzer d504c8f4a3 Merge pull request #50 from lbryio/lbrycrd-in-bundle
Add lbrycrd to app bundle on macos
2016-07-04 15:42:24 -05:00
Job Evers-Meltzer 976c8b3ce0 add a more useful exception if auth fails on lbrycrd 2016-07-04 15:40:52 -05:00
Alex Liebowitz 905e7e41a6 Merge branch 'master' of https://github.com/lbryio/lbry 2016-07-04 16:26:35 -04:00
Job Evers-Meltzer 16f48d0400 Add lbrycrd to app bundle on macos 2016-07-04 15:18:08 -05:00
Alex Liebowitz 6d7608d851 Create lbrycrd.conf if needed 2016-07-04 16:16:34 -04:00
Jack 7cd30efbe9 download lbrycrd binaries and set up lbrycrddpath.conf 2016-07-04 08:41:44 -04:00
Jack Robison 8c3652e1ac Merge pull request #49 from lbryio/real-lbrycrd2
real blockchain
2016-07-04 05:48:49 -04:00
Jack 7b9e3aace9 use lbrycrd 2016-07-04 05:43:26 -04:00
Jack 568737e123 change lbryum version constant name 2016-07-04 05:15:53 -04:00
Jack 808e3ea6e3 rename LBRYcrdWallet.py to LBRYWallet.py and change default wallet to lbrycrd 2016-07-04 04:57:30 -04:00
Jack e4f14d871f update setup.py 2016-07-01 03:36:51 -04:00
Jack 1cedd8ee5b update name 2016-07-01 03:16:08 -04:00
Jack b80fe24179 tanstaafl 2016-07-01 02:55:20 -04:00
Jack 3cea41a854 remove lbrynet_gui 2016-07-01 02:42:34 -04:00
Jack 62b80c13cc version bump 2016-06-30 01:26:33 -04:00
Jack 08b612575c update_claim fix and clean up publish keys
-replace ‘value’ key with ‘metadata’

-update claim metadata in db when an update is published
2016-06-28 23:20:28 -04:00
Jack b3b804bf2d add update_name 2016-06-28 14:28:59 -04:00
Jack 09d8a86e30 rename variable 2016-06-28 02:06:04 -04:00
Jack cc5d917c72 download fix 2016-06-28 01:51:05 -04:00
Jack 7cc3e9d5ef publish updates
-adds a base set of metadata fields required for results to be rendered
in conf.py, including language and content-type

-dont support old style claims on the new blockchain
2016-06-27 17:07:59 -04:00
Job Evers‐Meltzer 87aa9fc6c4 Merge pull request #48 from lbryio/add-cache
add caching for travis
2016-06-27 11:50:23 -05:00
Job Evers 8d5c315817 add caching for travis 2016-06-27 11:21:30 -05:00
Jack f89486f5fe port and import name 2016-06-24 13:12:27 -04:00
292 changed files with 18914 additions and 19238 deletions

No files matched your search

+13
View File
@@ -0,0 +1,13 @@
version: 1.0.{build}
notifications:
- provider: Slack
incoming_webhook:
secure: LuxwG5OZnnA//gmSXzCKu8/FRqYjCgGfVFqajSsGHeQ1HQNp7rYNhQpsO8/3PK63xKJj3wzt86DJekf9q9Q5OcHa9AHXUQbEveX0psd7elw=
clone_folder: c:\projects\lbry
build_script:
- echo "Not currently building on appveyor"
+20
View File
@@ -0,0 +1,20 @@
[bumpversion]
current_version = 0.9.2rc15
commit = True
tag = True
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>[a-z]+)(?P<candidate>\d+))?
serialize =
{major}.{minor}.{patch}{release}{candidate}
{major}.{minor}.{patch}
[bumpversion:part:candidate]
first_value = 1
[bumpversion:part:release]
optional_value = production
values =
rc
production
[bumpversion:file:lbrynet/__init__.py]
+1
View File
@@ -0,0 +1 @@
/CHANGELOG.md merge=union
+48
View File
@@ -0,0 +1,48 @@
<!--
Thanks for reporting an issue to LBRY and helping us improve!
To make it possible for us to help you, please fill out below information carefully.
Before reporting any issues, please make sure that you're using the latest version.
- App releases: https://github.com/lbryio/lbry-app/releases
- Standalone daemon: https://github.com/lbryio/lbry/releases
We are also available on Slack at https://slack.lbry.io
-->
## The Issue
### Steps to reproduce
1.
2.
3.
### Expected behaviour
Tell us what should happen
### Actual behaviour
Tell us what happens instead
## System Configuration
<!-- For the app, this info is in the About section at the bottom of the Help page.
You can include a screenshot instead of typing it out -->
<!-- For the daemon, run:
curl 'http://localhost:5279/lbryapi' --data '{"method":"version"}'
and include the full output -->
- LBRY Daemon version:
- LBRY App version:
- LBRY Installation ID:
- Operating system:
## Anything Else
<!-- Include anything else that does not fit into the above sections -->
## Screenshots
<!-- If a screenshot would help explain the bug, please include one or two here -->
+16 -20
View File
@@ -1,27 +1,23 @@
*.pyc
*.xml
.idea/.name
*.iml
*.log
*.txt
lbrynet.egg-info/PKG-INFO
*.egg
*.so
*.xml
*.iml
*.log
*.pem
*.decTest
*.prof
.#*
/build
/dist
/lbrynet.egg-info
/docs_build
/lbry-venv
*.so
*.pem
*.decTest
.idea/.name
.coverage
.DS_Store
# temporary files from the twisted.trial test runner
_trial_temp/
+10 -3
View File
@@ -11,6 +11,11 @@
# paths.
ignore=CVS
# Add files or directories matching the regex patterns to the
# blacklist. The regex matches against base names, not paths.
# `\.#.*` - add emacs tmp files to the blacklist
ignore-patterns=\.#.*
# Pickle collected data for later comparisons.
persistent=yes
@@ -121,7 +126,8 @@ logging-modules=logging
bad-functions=map,filter,input
# Good variable names which should always be accepted, separated by a comma
good-names=i,j,k,ex,Run,_
# allow `d` as its used frequently for deferred callback chains
good-names=i,j,k,ex,Run,_,d
# Bad variable names which should always be refused, separated by a comma
bad-names=foo,bar,baz,toto,tutu,tata
@@ -293,12 +299,13 @@ ignored-modules=twisted.internet.reactor,leveldb
# List of classes names for which member attributes should not be checked
# (useful for classes with attributes dynamically set). This supports can work
# with qualified names.
ignored-classes=twisted.internet,RequestMessage
ignored-classes=twisted.internet.reactor,RequestMessage
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
generated-members=lbrynet.lbrynet_daemon.LBRYDaemon.Parameters
[IMPORTS]
+35 -54
View File
@@ -1,59 +1,40 @@
matrix:
include:
- os: linux
sudo: required
dist: trusty
# dh-virtualenv requires that we use the same python interpreter
# that comes with the system, so we don't want to use anything that
# travis would try to set-up for us in python
language: generic
- os: osx
# Use generic language for osx
# python 2.7 is broken on osx on travis, so follow we have to specify the installation ourselves
# https://github.com/travis-ci/travis-ci/issues/2312#issuecomment-195620855
language: generic
osx_image: xcode7.3
os: linux
dist: trusty
language: python
python: 2.7
before_install:
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ./packaging/travis/setup_osx.sh; fi
notifications:
slack:
secure: "Am13HPtpgCMljh0MDVuoFHvQXB8yhf4Kvf/qAeSp5N0vsHGL70CSF9Ahccw8dVPE6mbuak1OGtSUb6/UaErLHkpz3ztaRLkDa9x7CmBB3Kynnh8oO2VbB7b/2ROULqkhF4VZmAnNfwrQrbC3gs8Sybp261Nyc7y4ww15xDYBrk2fyq4ds2DCaJdRxfJUJFonrZ6KXr3fVaXosO6cjuyS8eRodcmrqsT4cCtinjNTD1hGWoH107E4ObSmpVelxQO193KhNJMRiLlEcVkvYUOqIWBtwdGHbNE/6Yeuq1TXgKJ0KeJWAmW3wTfUYNngGXNAsyCnrhul5TKNevNzfIAQZHvRsczYiWPJV6LtohHT0CcUiCXJtvEPOyahEBfwK3etY/xxFqny7N9OEmpdW2sgsEPNPX2LJynJti2rQA9SuAD1ogR3ZpDy/NXoaAZf8PTdPcuNUMULV9PGG7tVrLBecO/W1qO6hdFxwlLdgqGLxAENZgGp++v/DhPk/WvtmHj7iTbRq0nxaTWyX5uKOn2ADH+k/yfutjv6BsQU9xNyPeZEEtuEpc6X6waiYn/8G9vl9PecvKC5H0MgsZ6asAxmg7mZ3VSMFG7mo8ENeOhSZ0Oz6ZTBILL3wFccZA9uJIq7NWmqC9dRiGiuKXBB62No7sINoHg3114e2xYa9qvNmGg="
cache:
directories:
- $HOME/.cache/pip
- $HOME/Library/Caches/pip
- $TRAVIS_BUILD_DIR/cache/wheel
addons:
apt:
packages:
- libgmp3-dev
- build-essential
- git
- libssl-dev
- libffi-dev
install:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then ./packaging/travis/install_dependencies_and_run_tests.sh; fi
before_install:
- virtualenv venv
- source venv/bin/activate
before_script:
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then openssl aes-256-cbc -k "$ENCRYPTION_SECRET" -in packaging/osx/certs/dist.cer.enc -d -a -out packaging/osx/certs/dist.cer; fi
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then openssl aes-256-cbc -k "$ENCRYPTION_SECRET" -in packaging/osx/certs/dist.p12.enc -d -a -out packaging/osx/certs/dist.p12; fi
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ./packaging/osx/add-key.sh; fi
install:
- pip install -U pip
- pip install -r requirements.txt
- pip install git+https://github.com/lbryio/lbryum.git
- pip install .
script:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then bash packaging/ubuntu/ubuntu_package_setup.sh -t; fi
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then cd packaging/osx/lbry-osx-app && ./setup_app.sh && cd $TRAVIS_BUILD_DIR; fi
# fail the build if this is a build for a tag and we don't have the versions matching; allow tags that start with 'test' to pass
- if [[ -n "${TRAVIS_TAG}" ]]; then if [[ "${TRAVIS_TAG}" == test* ]] || [[ "v`python setup.py -V`" = "${TRAVIS_TAG}" ]]; then true; else false; fi; fi
deploy:
- provider: releases
file: "${TRAVIS_BUILD_DIR}/`python setup.py --name`_`python setup.py -V`_amd64.deb"
skip_cleanup: true
on:
tags: true
condition: "$TRAVIS_OS_NAME = linux"
# this is the oauth token for the lbry-ci user
api_key:
secure: nKdWGROnLNodx9k9nWvq2wezkPvSVL8zF63qjPXjhdOe9kCUbcLp7WnFDYpn9EJj4Pofq/ejeCHwjA+5x7JUP3Szk7SlJV61B4c/5hl64rl7oSKOoKskjdE2jaOG3CJuOUrh0yQ59U3vMMABcsnw/wJaCIuu/erdPIm8g8R+stu1YHOGtl5Y9WiW+zLJn2vc3GooV1TWtki9EnrmFfw0Vrqc4RMVMFB1ojE7ggrK1LIwcmGSbLIYzker1ZRz8SCy+84sGk4//V+2i2NNiz5AkPuG7BBGrU2twE9nD23IlruJAdVdi71P3ytAmi0kKyvxIU4VeNaqyTk9zeL5IB9J5IIgvekHgKcsKhFUZ6QcXT1Xfxl4ELftvWCTHWiewnXFdqLcG9GZiUaE6+7wdalwDAP3tqS2emiibetlBZERHR+RMR00ej+1MBYWGMlTse/0Tglndv0a2qqgAJYLKPRT02hTRYGxZ1MrJe+WGnChRmzwgLVTIgZuiDciFOahN0TYGSORk6OpnZBsxvpzSqDw5UDJx0BmbJ1xMNDFbOs8ubZ9yIpB9yNMGw66FPacOF61XNYnmA68ILC28UtOFKuuHLrUPbM5JmQkDVhtTfFbBnyHefyCLAL4MHvJJKGi1oaOXjYaJ/J095h636/kQ0cHHuVMgoWUQZOQ44xRAz7tMuc=
- provider: releases
file: "${TRAVIS_BUILD_DIR}/packaging/osx/lbry-osx-app/`python setup.py --name`.`python setup.py -V`.dmg"
skip_cleanup: true
on:
tags: true
condition: "$TRAVIS_OS_NAME = osx"
# this is the oauth token for the lbry-ci user
api_key:
secure: nKdWGROnLNodx9k9nWvq2wezkPvSVL8zF63qjPXjhdOe9kCUbcLp7WnFDYpn9EJj4Pofq/ejeCHwjA+5x7JUP3Szk7SlJV61B4c/5hl64rl7oSKOoKskjdE2jaOG3CJuOUrh0yQ59U3vMMABcsnw/wJaCIuu/erdPIm8g8R+stu1YHOGtl5Y9WiW+zLJn2vc3GooV1TWtki9EnrmFfw0Vrqc4RMVMFB1ojE7ggrK1LIwcmGSbLIYzker1ZRz8SCy+84sGk4//V+2i2NNiz5AkPuG7BBGrU2twE9nD23IlruJAdVdi71P3ytAmi0kKyvxIU4VeNaqyTk9zeL5IB9J5IIgvekHgKcsKhFUZ6QcXT1Xfxl4ELftvWCTHWiewnXFdqLcG9GZiUaE6+7wdalwDAP3tqS2emiibetlBZERHR+RMR00ej+1MBYWGMlTse/0Tglndv0a2qqgAJYLKPRT02hTRYGxZ1MrJe+WGnChRmzwgLVTIgZuiDciFOahN0TYGSORk6OpnZBsxvpzSqDw5UDJx0BmbJ1xMNDFbOs8ubZ9yIpB9yNMGw66FPacOF61XNYnmA68ILC28UtOFKuuHLrUPbM5JmQkDVhtTfFbBnyHefyCLAL4MHvJJKGi1oaOXjYaJ/J095h636/kQ0cHHuVMgoWUQZOQ44xRAz7tMuc=
env:
global:
# needed to unlock the identity file when its on the keychain
- secure: "aqgVNHfeh6JUqTTWG5+W+tTsIJkePS6HyLkcZlq6ODYrfdGKa90EeV4q4wdGfP5IkrBbf+WBFACDGVp574E7vfMLMNKUDuUtgnGVPwqOIhjG82ud8Xa7qF7lsw50QOnRYYVd2GLlCIzk8sffT5ncjSPN2ClNVM5iTwCkC8TNSAVnEJxu6bG6hcaT4uCWWjs0m+39O1xJwxJ4vTjpE/gy+j2FTSaUR5cNavOVCyJqpeKlga9aoBVsQHWvxlYurdWbwRLwVIV7bDE+sYPnwn1nMFQpx5RZ1AX1Z2UxNFKcYzLJgcWe85OjxLyT4udX+XZ9SLsdOjm1n201OLEKsmTxmHS8yqpbu6+pKQ1rLMEVpgPGfS8DdtqZyb3z0u6q4jztpm+uBe8hnFgwQFGXO63nOQsI0n1PMR3evAnVx7jYt7y/UALs2A9yjosMwDqgql1FMhyd0OJGo8Ky8YpDsr2J5zGVvIqt2/N+lP+SOe2D1J/5EhGlY0o4tqIAFskh3q8/GR6UGm70KT2l4LJqrU1WGrGxPJ+HoOEmvG6eqLesk03fAX3v5+DgXZWErnzXMIOGDPaFVpmx+G9VZTIlmf+3Wbu9TnZE5PRwFTdP1rqjGpjUhHeF0VTc2qgNq1OfL98CBO1wLgA195+em58cELalnwDMWUTmY1Jt1kUuAtCc11U="
# used to decypt the aes decrypt the certificate and identity files
- secure: "FzftiEBFMngQIci5KZ+tpKs/BubalhWXJ9f8yogGNVa3XplLheUXYgcsNU8sYT2MJaKAHN9fSjHWb67UtKT7yXddXxeOPnW6wJ3ua/FXnpynsmeGKTfh3stvgjpzdXous67uHmCWMMklfb6z7UuDohjUMAe5n5HKw5tq1RzTKpc2kJacOC6qUT5laUOvULyCaO9E9HmbHeR5ZeXAC6pnzX2ccsSrcXvPozHzBIZ9RyothKs+CZw8PEuJo07RRL8meboegqYOUrOYuz6A2gS2mZJoy59ivZKOFxS5shEuv2Jt80RyfyxBoUpKFq8OG3Am5nAEzDiTIGzmIoKDEGKeTagk/sEtSZXiMDkzDT4GX/j1rUNLCBU87bXEFS2zfRsrfg8c1XZPIzDYBT1PY2QtLBdddF5zzDoKdPLJ3sjN+fZFE5RlnwfwnMHriVRZZlzjcdk0Z06gKTBCxUg5BZamnOOK+K8qunMJXVS+Vmi5u4RoTZZiCosUlYKnSKJ8suO9C0+znxoViusPqP4ONprNHgDoZw+UKio84QW3PrZv9h4zH/D+msDgJRZ0ceqDD+6Wz1J8Mm5ptW1GOLh/IU12TPXjxteqh0Um2vv5eIPmjK9uEK666kK7PqtPDkYhAfWvF+nmOOyPMJfbP4MW/i9WHNF4ghsIMbPKfqNhgSmfrYw="
- pip install cython
- pip install mock pylint unqlite
- ./run_pylint.sh
- ./run_tests.sh
- rvm use 2.3.1 && gem install danger --version '~> 4.0' && danger
+247
View File
@@ -0,0 +1,247 @@
# Change Log
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/) with
regard to the json-rpc api. As we're currently pre-1.0 release, we
can and probably will change functionality and break backwards compatability
at anytime.
## [Unreleased]
### Added
* Add `claim_id` parameter to `claim_show`
* Add `hex` field to claim responses for the raw claim value
* Add an `error` field to to file responses if an error occurs
### Changed
*
*
*
### Fixed
* `file_list` for files with bad signatures
* return None from resolve commands when nothing is found
* return lbry files with claims that are abandoned
## [0.9.2rc9] - 2017-04-08
### Added
* Use `claim_id` instead of outpoint for `claim_abandon`
* Add `channel_name` parameter to `publish`
* Add `delete_all` parameter to `file_delete` to allow deleting multiple files
* Add `channel_list_mine`
* Add `channel_new`
* Add `resolve` to resolve lbry uris
### Changed
* Use `uri` instead of `name` for `get`, remove explicit `claim_id` parameter
* Increase default download timeout
* Use lbry.io api for exchange rate data
## [0.9.2rc4] - 2017-04-06
### Changed
* Use lbryschema library for metadata
### Fixed
* Removed update_metadata function that could cause update problems
* Fix DHT contact bug
## [0.9.2rc3] - 2017-03-29
### Added
* Create wallet_unused_address API command
* Add `claim_id` parameter to `get`, used to specify non-default claim for `name`
### Changed
* wallet_new_address API command always returns new address
* Improved ConnectionManager speed
* Remove unused `stream_info` parameter in `get`
## [0.9.2rc2] - 2017-03-25
### Added
* Add `wallet_list` command
* Add checks for missing/extraneous params when calling jsonrpc commands
* Added colors to cli error messages
### Changed
* Removed check_pending logic from Daemon
* Switched to txrequests so requests can use twisted event loop
* Renamed API command file_seed to file_set_status
### Fixed
* Fix restart procedure in DaemonControl
* Create download directory if it doesn't exist
* Fixed descriptor_get
* Fixed jsonrpc_reflect()
* Fixed api help return
* Fixed API command descriptor_get
* Fixed API command transaction_show
* Fixed error handling for jsonrpc commands
## [0.9.2rc1] - 2017-03-21
### Added
* Add `wallet_list` command
### Changed
* Dont add expected payment to wallet when payment rate is 0
### Fixed
* Fixed descriptor_get
* Fixed jsonrpc_reflect()
* Fixed api help return
* Fixed API command descriptor_get
* Fixed API command transaction_show
* Handle failure to decode claim cache file
## [0.9.1] - 2017-03-17
### Fixed
* Fix wallet_public_key API command
## [0.9.1rc5] - 2017-03-16
### Added
* publish API command can take metadata fields as arguments
* Added `reflect_uploads` config to disable reflecting on upload
### Fixed
* Fixed jsonrpc_reflect()
* Fixed api help return
## [0.9.1rc2] - 2017-03-15
### Added
* Added `--version` flag
### Changed
* Removed `simplejson` dependency in favor of bulitin `json`
## [0.9.0rc17] - 2017-03-10
### Fixed
* Added string comparison to ClaimOutpoint (needed to look things up by outpoint)
* Remove unused API commands from daemon
* Fix file filter `outpoint`
* Made dictionary key names in API commmand outputs to be more consistent
## [0.9.0rc15] - 2017-03-09
### Added
* Add file filters: `claim_id`, `outpoint`, and `rowid`
* Make loggly logs less verbose
### Changed
* Change file filter `uri` to `name` and return field `lbry_uri` to `name`
* Refactor file_list, add `full_status` argument to populate resource intensive fields
* Remove deprecated file commands: `get_lbry_files`, `get_lbry_file`, and `file_get`
* Remove deprecated `delete_lbry_file` command
* Return standard file json from `get`
### Fixed
* Added string comparison to ClaimOutpoint (needed to look things up by outpoint)
* Remove unused API commands from daemon
* Fix file filter `outpoint`
## [0.9.0rc12] - 2017-03-06
### Fixed
* Fixed ExchangeRateManager freezing the app
* Fixed download not timing out properly when downloading sd blob
* Fixed ExchangeRateManager freezing the app
* Fixed download not timing out properly when downloading sd blob
* Fixed get not reassembling an already downloaded file that was deleted from download directory
## [0.9.0rc11] - 2017-02-27
### Fixed
* Added timeout to ClientProtocol
* Add check for when local height of wallet is less than zero
## [0.9.0rc9] - 2017-02-22
### Changed
* Add blockchain status to jsonrpc_status
## [0.8.7] - 2017-02-21
## [0.8.6] - 2017-02-19
## [0.8.6rc0] - 2017-02-19
### Changed
* Add `file_get` by stream hash
* Add utils.call_later to replace reactor.callLater
### Fixed
* Fix unhandled error in `get`
* Fix sd blob timeout handling in `get_availability`, return 0.0
## [0.8.5] - 2017-02-18
## [0.8.5rc0] - 2017-02-18
### Fixed
* Fix result expected by ui from file_get for missing files
## [0.8.4] - 2017-02-17
## [0.8.4rc0] - 2017-02-17
### Changed
* Remove unused upload_allowed option
* Remove code related to packaging as that step is now done in the electron client
* Remove lbryum version check; use lbry-electron as version source
* Include download url in version check
### Fixed
* add misssing traceback to logging
## [0.8.3] - 2017-02-15
### Fixed
* Get lbry files with pending claims
* Add better logging to help track down [#478](https://github.com/lbryio/lbry/issues/478)
* Catch UnknownNameErrors when resolving a name. [#479](https://github.com/lbryio/lbry/issues/479)
### Changed
* Add blob_get, descriptor_get, and blob_delete
* Add filter keyword args to blob_list
* Refactor get_availability
* Add optional peer search timeout, add peer_search_timeout setting
## [0.8.3rc3] - 2017-02-14
## [0.8.3rc2] - 2017-02-13
## [0.8.3rc1] - 2017-02-13
### Changed
* make connection manager unit testeable
### Fixed
* Change EWOULDBLOCK error in DHT to warning. #481
* mark peers as down if it fails download protocol
* Made hash reannounce time to be adjustable to fix [#432](https://github.com/lbryio/lbry/issues/432)
## [0.8.3rc0] - 2017-02-10
### Changed
* Convert EncryptedFileDownloader to inlineCallbacks
* Convert EncryptedFileManager to use inlineCallbacks
* Convert Daemon._delete_lbry_file to inlineCallbacks
* Add uri to stream reflector to de-obfuscate reflector logs
* Simplify lbrynet.lbrynet_daemon.Publisher
* Reflect streams in file manager looping call rather than in each file
* Convert GetStream to inclineCallbacks
* Change callback condition in GetStream to the first data blob completing
* Add local and remote heights to blockchain status
### Fixed
* Fix recursion depth error upon failed blob
* Call stopProducing in reflector client file_sender when uploading is done
* Ensure streams in stream_info_manager are saved in lbry_file_manager
* Fixed file_delete not deleting data from stream_info_manager [#470](https://github.com/lbryio/lbry/issues/470)
* Fixed upload of bug reports to Slack ([#472](https://github.com/lbryio/lbry/issues/472))
* Fixed claim updates [#473](https://github.com/lbryio/lbry/issues/473)
* Handle ConnectionLost error in reflector client
* Fix updating a claim where the stream doesn't change
* Fix claim_abandon
## [0.8.1] - 2017-02-01
### Changed
* reflect all the blobs in a stream
* change command line flags so that the more common usage is the default
* change daemon function signatures to include names arguments
### Fixed
* disable verbose twisted logs
* improved wallet balance calculations
* fix block too deep error
## [0.8.0] - 2017-01-24
### Changed
* renamed api endpoints
* improved command line user experience
* integrate twisted logging with python logging
* Updated READMEs
### Fixed
* Fixed bug where ConnectionManager wasn't being stopped
* Fixed: #343
* Stop hanging if github is down
* paths for debian package have been updated to be correct
* improved output of the publish command
+6
View File
@@ -0,0 +1,6 @@
# Add a CHANGELOG entry for app changes
has_app_changes = !git.modified_files.grep(/lbrynet/).empty?
if !git.modified_files.include?("CHANGELOG.md") && has_app_changes
fail("Please include a CHANGELOG entry.")
message "See http://keepachangelog.com/en/0.3.0/ for details on good changelog guidelines"
end
+8 -7
View File
@@ -4,15 +4,15 @@ Q: How do I get lbry for command line?
A: In order to run lbry from command line, you need more than the packaged app/deb.
######On OS X
###### On OS X
You can install LBRY command line by running `curl -sL https://rawgit.com/lbryio/lbry-setup/master/lbry_setup_osx.sh | sudo bash` in a terminal. This script will install lbrynet and its dependancies, as well as the app.
######On Linux
###### On Linux
On Ubuntu or Mint you can install the prerequisites and lbrynet by running
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip git
git clone https://github.com/lbryio/lbry.git
cd lbry
sudo python setup.py install
@@ -77,9 +77,10 @@ Note: the lbry api can only be used while either the app or lbrynet-daemon comma
sys.exit(0)
api = JSONRPCProxy.from_url(API_CONNECTION_STRING)
if not api.is_running():
print api.daemon_status()
status = api.status()
if not status['is_running']:
print status
else:
for func in api.help():
print "%s:\n%s" % (func, api.help({'function': func}))
for cmd in api.commands():
print "%s:\n%s" % (cmd, api.help({'command': cmd}))
+61 -30
View File
@@ -1,49 +1,49 @@
#### Installing the LBRY app
--------------------------
# Installing LBRY
Installing LBRY is simple. You can get a dmg installer for OS X (Mavericks and up) or a .deb for linux [here](https://lbry.io/get).
The recommended way to install LBRY is to use an installer. We provide installers for all major operating systems. See the [README](README.md).
##### OS X
Just drag and drop LBRY.app into your applications folder (replacing any older versions). When it's running you'll have a LBRY icon in your status bar.
These instructions are for installing from source code, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
##### Linux
Double click the .deb file and follow the prompts. The app can be started by searching "LBRY", and it can be turned off by clicking the red 'x' in the browser interface.
## Prerequisites
On both systems you can also open the UI while the app is running by going to lbry://lbry in Firefox or Safari, or localhost:5279 in Chrome.
Before starting, you'll need to install some additional libraries depending on your operating system.
### OS X Prerequisites
Install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
#### Installing LBRY command line
--------------------------
Remaining dependencies can then be installed by running:
##### OS X
You can install LBRY command line by running `curl -sL https://rawgit.com/lbryio/lbry-setup/master/lbry_setup_osx.sh | sudo bash` in a terminal. This script will install lbrynet and its dependancies, as well as the app. You can start LBRY by either starting the app or by running `lbrynet-daemon` from a terminal.
```
brew install mpfr libmpc protobuf
sudo easy_install pip
sudo pip install virtualenv
```
##### Linux
On Ubuntu or Mint you can install the prerequisites and lbrynet by running
```
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip
git clone https://github.com/lbryio/lbry.git
cd lbry
sudo python setup.py install
```
### Linux Prerequisites
To start LBRY, run `lbrynet-daemon` in a terminal.
On Ubuntu (we recommend 16.04), install the following:
#### On windows:
```
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip git python-virtualenv libssl-dev libffi-dev python-protobuf
```
If you're running another Linux flavor, install the equivalent of the above packages for your system.
### Windows Prerequisites
Install [mingw32](http://www.mingw.org/) base and c++ compiler.
Add C:\MinGW\bin to the windows PATH.
Enable distutils to compile with mingw32 by creating a distutils.cfg file in *PYTHONPATH\Lib\distutils* containing
Enable distutils to compile with mingw32 by creating a distutils.cfg file in *PYTHONPATH\Lib\distutils* containing:
```
[build]
compiler = mingw32
```
If using virtualenv, copy the *PYTHONPATH\Lib\distutils* directory to the virtualenv.
If using virtualenv, which is recommended, copy the *PYTHONPATH\Lib\distutils* directory to the virtualenv.
It's recommended to use [Unofficial Windows Binaries for Python Extension Packages](http://www.lfd.uci.edu/~gohlke/pythonlibs/) for as many of the required packages as possible.
Currently, available binaries include:
@@ -56,12 +56,43 @@ Currently, available binaries include:
- requests
- gmpy
Install the each of the preceding binaries with `pip install *.whl`
Install the above binaries with `pip install *.whl`
Install pywin32 system files by run `python.exe Scripts\pywin32_postinstall.py -install` from an elevated command prompt.
Install the rest of the required modules with the standard `pip install module` command
Finally, you'll need [miniupnpc](https://pypi.python.org/pypi/miniupnpc/1.9). Download the source and compile with MinGW by running `mingw32make.bat`. Then install the module by running `python setupmingw32.py install`.
The one module which requires additional work is [miniupnpc](https://pypi.python.org/pypi/miniupnpc/1.9).
Download the source and compile with MinGW by running `mingw32make.bat`
Then install the module by running `python setupmingw32.py install`
## Installation
To install:
```
git clone https://github.com/lbryio/lbry.git
cd lbry
virtualenv lbry-venv
source lbry-venv/bin/activate
pip install -U -r requirements.txt
pip install --editable .
```
To start LBRY, run `lbrynet-daemon` in a terminal.
To verify your install, `which lbrynet-daemon` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
## Making a Change
If you are not already inside the virtual environment, run:
```
source lbry-venv/bin/activate
```
Then run:
```
pip install --editable .
```
This will update `lbrynet-daemon` and other executables.
+12 -19
View File
@@ -1,22 +1,15 @@
Copyright (c) 2015, LBRY, Inc.
All rights reserved.
The MIT License (MIT)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
Copyright (c) 2015-2016 LBRY Inc
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the
following conditions:
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-2
View File
@@ -1,2 +0,0 @@
include lbrynet/lbrynet_console/plugins/blindrepeater.yapsy-plugin
recursive-include lbrynet/lbrynet_gui *.gif *.ico *.xbm *.conf
+27 -47
View File
@@ -1,71 +1,51 @@
[![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry)
[![Coverage Status](https://coveralls.io/repos/github/lbryio/lbry/badge.svg)](https://coveralls.io/github/lbryio/lbry)
# LBRYnet
# LBRY
LBRYnet is a fully decentralized network for distributing data. It consists of peers uploading
and downloading data from other peers, possibly in exchange for payments, and a distributed hash
table, used by peers to discover other peers.
LBRY is an open-source protocol providing distribution, discovery, and purchase of digital content (data) via a decentralized network.
## Installation
This repo is a reference implementation of the [LBRY API](https://lbry.io/api).
Download the [latest release](https://github.com/lbryio/lbry/releases/latest) or see [INSTALL.md](INSTALL.md) for manual installation.
It provides a daemon that can interact with the network via a json-rpc interface over HTTP.
## Overview
## Installing
On LBRYnet, data is broken into chunks, and each chunk is specified by its sha384 hash sum. This
guarantees that peers can verify the correctness of each chunk without having to know anything
about its contents, and can confidently re-transmit the chunk to other peers. Peers wishing to
transmit chunks to other peers announce to the distributed hash table that they are associated
with the sha384 hash sum in question. When a peer wants to download that chunk from the network,
it asks the distributed hash table which peers are associated with that sha384 hash sum. The
distributed hash table can also be used more generally. It simply stores IP addresses and
ports which are associated with 384-bit numbers, and can be used by any type of application to
help peers find each other. For example, an application for which clients don't know all of the
necessary chunks may use some identifier, chosen by the application, to find clients which do
know all of the necessary chunks.
**Note**: This project no longer directly bundles a graphic interface (browser). If you want to use LBRY via a browser, [use the LBRY App](https://github.com/lbryio/lbry-app).
## For Developers
Our [releases page](https://github.com/lbryio/lbry-app/releases/latest) contains pre-built binaries of the latest release, pre-releases, and past releases, for macOS, Debian-based Linux, and Windows.
The bundled LBRY application uses the lbrynet JSONRPC api found in `lbrynet.lbrynet_daemon.LBRYDaemon`. This api allows for applications and web services like the lbry browser UI to interact with lbrynet. If you've installed lbrynet, you can run `lbrynet-daemon` without running the app. While the app or `lbrynet-daemon` is running, you can use the following to show the help for all the available commands:
Installing from source is also relatively painless, full instructions are in [INSTALL.md](INSTALL.md)
```
from jsonrpc.proxy import JSONRPCProxy
## Running
try:
from lbrynet.conf import API_CONNECTION_STRING
except:
print "You don't have lbrynet installed!"
API_CONNECTION_STRING = "http://localhost:5279/lbryapi"
api = JSONRPCProxy.from_url(API_CONNECTION_STRING)
if not api.is_running():
print api.daemon_status()
else:
for func in api.help():
print "%s:\n%s" % (func, api.help({'function': func}))
```
Run `lbrynet-daemon` to launch the daemon.
If you've installed lbrynet, it comes with a file sharing application, called `lbrynet-daemon`, which breaks
files into chunks, encrypts them with a symmetric key, computes their sha384 hash sum, generates
a special file called a 'stream descriptor' containing the hash sums and some other file metadata,
and makes the chunks available for download by other peers. A peer wishing to download the file
must first obtain the 'stream descriptor' and then may open it with his `lbrynet-daemon` client,
download all of the chunks by locating peers with the chunks via the DHT, and then combine the
chunks into the original file, according to the metadata included in the 'stream descriptor'.
## Using
For detailed instructions, see [INSTALL.md](INSTALL.md) and [RUNNING.md](RUNNING.md).
By default, `lbrynet-daemon` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
Documentation: doc.lbry.io (may be out of date)
Our [quickstart guide](http://lbry.io/quickstart) provides clear sample usages and free credits for learning.
Source code: https://github.com/lbryio/lbry
The full API is documented [here](https://lbry.io/api).
To contribute, [join us on Slack](https://lbry-slackin.herokuapp.com/) or contact josh@lbry.io. Pull requests are also welcome.
## What is LBRY?
LBRY is a protocol providing fully decentralized network for the discovery, distribution, and payment of data.
It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, and rights and access rules.
LBRY also provides a data network consists of peers uploading and downloading data from other peers, possibly in exchange for payments, and a distributed hash table, used by peers to discover other peers.
## Contributions
To contribute, [join us on Slack](https://slack.lbry.io/) or contact jeremy@lbry.io. Pull requests are also welcome.
## Support
Please open an issue and describe your situation in detail. We will respond as soon as we can.
For private issues, contact josh@lbry.io.
For private issues, contact jeremy@lbry.io.
## License
-146
View File
@@ -1,146 +0,0 @@
How to watch It's a Wonderful Life via LBRY
## Quickest quick guide
Create a directory called lbry, and go into that directory
Download the file https://raw.githubusercontent.com/lbryio/lbry-setup/master/lbry_setup.sh and run it in that directory
Once it's done building, type:
```
lbrynet-console
```
A console application will load, and after a moment you will be presented with a ">" signifying
that the console is ready for commands.
If it's your first time running lbrynet-console, you should be given some credits to test the
network. If they don't show up within a minute or two, let us know, and we'll send you some.
After your credits have shown up, type
```
get wonderfullife
```
into the prompt and hit enter.
You will be asked if you want to cancel, change options, save, and perhaps stream the file.
Enter 's' for save and then hit enter.
The file should start downloading. Enter the command 'status' to check on the status of files
that are in the process of being downloaded.
To stop lbrynet-console, enter the command 'exit'.
## Slightly longer install guide
### Installing lbrycrd, the full blockchain client
Note: this process takes upwards of an hour and is not necessary to use lbrynet.
```
git clone --depth=1 -b alpha https://github.com/lbryio/lbrycrd.git
cd lbrycrd
sudo apt-get install build-essential libtool autotools-dev autoconf pkg-config libssl-dev libboost-all-dev libdb-dev libdb++-dev libqt4-dev libprotobuf-dev protobuf-compiler
./autogen.sh
./configure --with-incompatible-bdb --without-gui
make
```
When make has completed, create the directory where LBRYcrd data will be stored. ~/.lbrycrd is where LBRYcrd will look by default and so is recommended.
```
mkdir ~/.lbrycrd
echo 'rpcuser=rpcuser
rpcpassword=rpcpassword' > ~/.lbrycrd/lbrycrd.conf
# (use a long random password if your computer is on a network anyone else has access to. e.g, pwgen -s 20)
cd ..
```
### Installing lbrynet from source
Acquire the LBRYnet source code from https://github.com/lbryio/lbry
```
cd lbry
sudo apt-get install libgmp3-dev build-essential python-dev python-pip
```
(with virtualenv)
```
python-virtualenv
virtualenv .
source bin/activate
python setup.py install
```
to deactivate the virtualenv later:
```
deactivate
```
to reactivate it, go to the directory in which you created it and:
```
source bin/activate
```
(without virtualenv)
```
python setup.py build bdist_egg
sudo python setup.py install
```
## Slightly longer running guide
### lbrynet-console can be set to use lbrycrdd instead of the built in lightweight client.
To run lbrynet-console with lbrycrdd:
```
lbrynet-console </path/to/lbrycrdd>
```
If lbrycrdd is not already running, lbrynet will launch it at that path, and will shut it down
when lbrynet exits. If lbrycrdd is already running, lbrynet will not launch it and will not
shut it down, but it will connect to it and use it.
### Running lbrycrdd manually
From the lbrycrd directory, run:
```
./src/lbrycrdd -server -daemon
```
If you want to mine LBC, also use the flag '-gen', so:
```
./src/lbrycrdd -server -daemon -gen
```
Warning: This will put a heavy load on your CPU
It will take a few minutes for your client to download the whole block chain.
To shut lbrycrdd down: from the lbrycrd directory, run:
```
./src/lbrycrd-cli stop
```
Any questions or problems, email jimmy@lbry.io
+79
View File
@@ -0,0 +1,79 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="80.207558mm"
height="58.081333mm"
viewBox="0 0 284.20001 205.8"
id="svg3479"
version="1.1"
inkscape:version="0.91 r13725"
sodipodi:docname="lbry-white-logo-only.svg">
<defs
id="defs3481" />
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="2"
inkscape:zoom="0.35"
inkscape:cx="-470.0429"
inkscape:cy="-5.6714247"
inkscape:document-units="px"
inkscape:current-layer="layer1"
showgrid="false"
inkscape:window-width="1920"
inkscape:window-height="1056"
inkscape:window-x="0"
inkscape:window-y="24"
inkscape:window-maximized="1"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0" />
<metadata
id="metadata3484">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(-363.61433,-320.89078)">
<g
id="g3396"
transform="translate(363.61433,320.89078)">
<g
id="g3398">
<polygon
style="fill:#ffffff"
points="138.8,155.2 271,74 271,68.2 146.2,8 7,94.1 7,132.6 138.8,197.8 276.4,113.4 280.3,119.4 139.2,205.8 0,137 0,90.2 145.8,0 278,63.8 278,77.9 139.2,163.2 34.6,111.9 34.8,104 "
id="polygon3400" />
</g>
<g
id="g3402">
<polygon
style="fill:#ffffff"
points="276.5,128.5 278.5,115.9 266.3,113.8 267.1,108.9 284.2,111.8 281.4,129.3 "
id="polygon3404" />
</g>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.3 KiB

+651
View File
@@ -0,0 +1,651 @@
# LBRY JSON-RPC API Documentation
## blob_announce_all
```text
Announce all blobs to the DHT
Args:
None
Returns:
(str) Success/fail message
```
## blob_delete
```text
Delete a blob
Args:
'blob_hash': (str) hash of blob to get
Returns:
(str) Success/fail message
```
## blob_get
```text
Download and return a blob
Args:
'blob_hash': (str) blob hash of blob to get
'timeout'(optional): (int) timeout in number of seconds
'encoding'(optional): (str) by default no attempt at decoding is made,
can be set to one of the following decoders:
'json'
'payment_rate_manager'(optional): if not given the default payment rate manager
will be used. supported alternative rate managers:
'only-free'
Returns
(str) Success/Fail message or (dict) decoded data
```
## blob_list
```text
Returns blob hashes. If not given filters, returns all blobs known by the blob manager
Args:
'uri' (optional): (str) filter by blobs in stream for winning claim
'stream_hash' (optional): (str) filter by blobs in given stream hash
'sd_hash' (optional): (str) filter by blobs in given sd hash
'needed' (optional): (bool) only return needed blobs
'finished' (optional): (bool) only return finished blobs
'page_size' (optional): (int) limit number of results returned
'page' (optional): (int) filter to page x of [page_size] results
Returns:
(list) List of blob hashes
```
## blob_reflect_all
```text
Reflects all saved blobs
Args:
None
Returns:
(bool) true if successful
```
## block_show
```text
Get contents of a block
Args:
'blockhash': (str) hash of the block to look up
Returns:
(dict) Requested block
```
## claim_abandon
```text
Abandon a name and reclaim credits from the claim
Args:
'txid': (str) txid of claim
'nout': (int) nout of claim
Return:
(dict) Dictionary containing result of the claim
{
txid : (str) txid of resulting transaction
fee : (float) fee paid for the transaction
}
```
## claim_list
```text
Get claims for a name
Args:
'name': (str) search for claims on this name
Returns
(dict) State of claims assigned for the name
{
'claims': (list) list of claims for the name
[
{
'amount': (float) amount assigned to the claim
'effective_amount': (float) total amount assigned to the claim,
including supports
'claim_id': (str) claim ID of the claim
'height': (int) height of block containing the claim
'txid': (str) txid of the claim
'nout': (int) nout of the claim
'supports': (list) a list of supports attached to the claim
'value': (str) the value of the claim
},
]
'supports_without_claims': (list) supports without any claims attached to them
'last_takeover_height': (int) the height of last takeover for the name
}
```
## claim_list_mine
```text
List my name claims
Args:
None
Returns
(list) List of name claims owned by user
[
{
'address': (str) address that owns the claim
'amount': (float) amount assigned to the claim
'blocks_to_expiration': (int) number of blocks until it expires
'category': (str) "claim", "update" , or "support"
'claim_id': (str) claim ID of the claim
'confirmations': (int) number of blocks of confirmations for the claim
'expiration_height': (int) the block height which the claim will expire
'expired': (bool) true if expired, false otherwise
'height': (int) height of the block containing the claim
'is_spent': (bool) true if claim is abandoned, false otherwise
'name': (str) name of the claim
'txid': (str) txid of the cliam
'nout': (int) nout of the claim
'value': (str) value of the claim
},
]
```
## claim_new_support
```text
Support a name claim
Args:
'name': (str) name
'claim_id': (str) claim ID of claim to support
'amount': (float) amount to support by
Return:
(dict) Dictionary containing result of the claim
{
txid : (str) txid of resulting support claim
nout : (int) nout of the resulting support claim
fee : (float) fee paid for the transaction
}
```
## claim_show
```text
Resolve claim info from a LBRY name
Args:
'name': (str) name to look up, do not include lbry:// prefix
'txid'(optional): (str) if specified, look for claim with this txid
'nout'(optional): (int) if specified, look for claim with this nout
Returns:
(dict) Dictionary contaning claim info, (bool) false if claim is not
resolvable
{
'txid': (str) txid of claim
'nout': (int) nout of claim
'amount': (float) amount of claim
'value': (str) value of claim
'height' : (int) height of claim takeover
'claim_id': (str) claim ID of claim
'supports': (list) list of supports associated with claim
}
```
## commands
```text
Return a list of available commands
Returns:
(list) list of available commands
```
## daemon_stop
```text
Stop lbrynet-daemon
Returns:
(string) Shutdown message
```
## descriptor_get
```text
Download and return a sd blob
Args:
'sd_hash': (str) hash of sd blob
'timeout'(optional): (int) timeout in number of seconds
'payment_rate_manager'(optional): (str) if not given the default payment rate manager
will be used. supported alternative rate managers:
only-free
Returns
(str) Success/Fail message or (dict) decoded data
```
## file_delete
```text
Delete a lbry file
Args:
'name' (optional): (str) delete file by lbry name,
'sd_hash' (optional): (str) delete file by sd hash,
'file_name' (optional): (str) delete file by the name in the downloads folder,
'stream_hash' (optional): (str) delete file by stream hash,
'claim_id' (optional): (str) delete file by claim ID,
'outpoint' (optional): (str) delete file by claim outpoint,
'rowid': (optional): (int) delete file by rowid in the file manager
'delete_target_file' (optional): (bool) delete file from downloads folder,
defaults to true if false only the blobs and
db entries will be deleted
Returns:
(bool) true if deletion was successful
```
## file_list
```text
List files limited by optional filters
Args:
'name' (optional): (str) filter files by lbry name,
'sd_hash' (optional): (str) filter files by sd hash,
'file_name' (optional): (str) filter files by the name in the downloads folder,
'stream_hash' (optional): (str) filter files by stream hash,
'claim_id' (optional): (str) filter files by claim id,
'outpoint' (optional): (str) filter files by claim outpoint,
'rowid' (optional): (int) filter files by internal row id,
'full_status': (optional): (bool) if true populate the 'message' and 'size' fields
Returns:
(list) List of files
[
{
'completed': (bool) true if download is completed,
'file_name': (str) name of file,
'download_directory': (str) download directory,
'points_paid': (float) credit paid to download file,
'stopped': (bool) true if download is stopped,
'stream_hash': (str) stream hash of file,
'stream_name': (str) stream name ,
'suggested_file_name': (str) suggested file name,
'sd_hash': (str) sd hash of file,
'name': (str) name claim attached to file
'outpoint': (str) claim outpoint attached to file
'claim_id': (str) claim ID attached to file,
'download_path': (str) download path of file,
'mime_type': (str) mime type of file,
'key': (str) key attached to file,
'total_bytes': (int) file size in bytes, None if full_status is false
'written_bytes': (int) written size in bytes
'message': (str), None if full_status is false
'metadata': (dict) Metadata dictionary
},
]
```
## file_set_status
```text
Start or stop downloading a file
Args:
'status': (str) "start" or "stop"
'name' (optional): (str) start file by lbry name,
'sd_hash' (optional): (str) start file by the hash in the name claim,
'file_name' (optional): (str) start file by its name in the downloads folder,
Returns:
(str) Confirmation message
```
## get
```text
Download stream from a LBRY name.
Args:
'name': (str) name to download
'file_name'(optional): (str) a user specified name for the downloaded file
'stream_info'(optional): (str) specified stream info overrides name
'timeout'(optional): (int) download timeout in number of seconds
'download_directory'(optional): (str) path to directory where file will be saved
'wait_for_write'(optional): (bool) defaults to True. When set, waits for the file to
only start to be written before returning any results.
Returns:
(dict) Dictionary contaning information about the stream
{
'completed': (bool) true if download is completed,
'file_name': (str) name of file,
'download_directory': (str) download directory,
'points_paid': (float) credit paid to download file,
'stopped': (bool) true if download is stopped,
'stream_hash': (str) stream hash of file,
'stream_name': (str) stream name ,
'suggested_file_name': (str) suggested file name,
'sd_hash': (str) sd hash of file,
'name': (str) name claim attached to file
'outpoint': (str) claim outpoint attached to file
'claim_id': (str) claim ID attached to file,
'download_path': (str) download path of file,
'mime_type': (str) mime type of file,
'key': (str) key attached to file,
'total_bytes': (int) file size in bytes, None if full_status is false
'written_bytes': (int) written size in bytes
'message': (str), None if full_status is false
'metadata': (dict) Metadata dictionary
}
```
## get_availability
```text
Get stream availability for a winning claim
Args:
'name' : (str) lbry name
'sd_timeout' (optional): (int) sd blob download timeout
'peer_timeout' (optional): (int) how long to look for peers
Returns:
(float) Peers per blob / total blobs
```
## help
```text
Return a useful message for an API command
Args:
'command'(optional): (str) command to retrieve documentation for
Returns:
(str) if given a command, returns documentation about that command
otherwise returns general help message
```
## peer_list
```text
Get peers for blob hash
Args:
'blob_hash': (str) blob hash
'timeout'(optional): (int) peer search timeout in seconds
Returns:
(list) List of contacts
```
## publish
```text
Make a new name claim and publish associated data to lbrynet,
update over existing claim if user already has a claim for name.
Fields required in the final Metadata are:
'title'
'description'
'author'
'language'
'license',
'nsfw'
Metadata can be set by either using the metadata argument or by setting individual arguments
fee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,
or sources. Individual arguments will overwrite the fields specified in metadata argument.
Args:
'name': (str) name to be claimed
'bid': (float) amount of credits to commit in this claim,
'metadata'(optional): (dict) Metadata to associate with the claim.
'file_path'(optional): (str) path to file to be associated with name. If provided,
a lbry stream of this file will be used in 'sources'.
If no path is given but a metadata dict is provided, the source
from the given metadata will be used.
'fee'(optional): (dict) Dictionary representing key fee to download content:
{currency_symbol: {'amount': float, 'address': str, optional}}
supported currencies: LBC, USD, BTC
If an address is not provided a new one will be automatically
generated. Default fee is zero.
'title'(optional): (str) title of the file
'description'(optional): (str) description of the file
'author'(optional): (str) author of the file
'language'(optional): (str), language code
'license'(optional): (str) license for the file
'license_url'(optional): (str) URL to license
'thumbnail'(optional): (str) thumbnail URL for the file
'preview'(optional): (str) preview URL for the file
'nsfw'(optional): (bool) True if not safe for work
'sources'(optional): (dict){'lbry_sd_hash':sd_hash} specifies sd hash of file
Returns:
(dict) Dictionary containing result of the claim
{
'tx' : (str) hex encoded transaction
'txid' : (str) txid of resulting claim
'nout' : (int) nout of the resulting claim
'fee' : (float) fee paid for the claim transaction
'claim_id' : (str) claim ID of the resulting claim
}
```
## reflect
```text
Reflect a stream
Args:
'sd_hash': (str) sd_hash of lbry file
Returns:
(bool) true if successful
```
## report_bug
```text
Report a bug to slack
Args:
'message': (str) message to send
Returns:
(bool) true if successful
```
## resolve_name
```text
Resolve stream info from a LBRY name
Args:
'name': (str) name to look up, do not include lbry:// prefix
Returns:
(dict) Metadata dictionary from name claim, None if the name is not
resolvable
```
## send_amount_to_address
```text
Send credits to an address
Args:
'amount': (float) the amount to send
'address': (str) the address of the recipient in base58
Returns:
(bool) true if payment successfully scheduled
```
## settings_get
```text
Get daemon settings
Returns:
(dict) Dictionary of daemon settings
See ADJUSTABLE_SETTINGS in lbrynet/conf.py for full list of settings
```
## settings_set
```text
Set daemon settings
Args:
'run_on_startup': (bool) currently not supported
'data_rate': (float) data rate,
'max_key_fee': (float) maximum key fee,
'download_directory': (str) path of where files are downloaded,
'max_upload': (float), currently not supported
'max_download': (float), currently not supported
'download_timeout': (int) download timeout in seconds
'search_timeout': (float) search timeout in seconds
'cache_time': (int) cache timeout in seconds
Returns:
(dict) Updated dictionary of daemon settings
```
## status
```text
Return daemon status
Args:
'session_status' (optional): (bool) true to return session status,
default is false
Returns:
(dict) Daemon status dictionary
```
## stream_cost_estimate
```text
Get estimated cost for a lbry stream
Args:
'name': (str) lbry name
'size' (optional): (int) stream size, in bytes. if provided an sd blob
won't be downloaded.
Returns:
(float) Estimated cost in lbry credits
```
## transaction_list
```text
List transactions belonging to wallet
Args:
None
Returns:
(list) List of transactions
```
## transaction_show
```text
Get a decoded transaction from a txid
Args:
'txid': (str) txid of transaction
Returns:
(dict) JSON formatted transaction
```
## version
```text
Get lbry version information
Args:
None
Returns:
(dict) Dictionary of lbry version information
{
'build': (str) build type (e.g. "dev", "rc", "release"),
'ip': (str) remote ip, if available,
'lbrynet_update_available': (bool) whether there's an update available,
'lbrynet_version': (str) lbrynet_version,
'lbryum_version': (str) lbryum_version,
'os_release': (str) os release string
'os_system': (str) os name
'platform': (str) platform string
'processor': (str) processor type,
'python_version': (str) python version,
'remote_lbrynet': (str) most recent lbrynet version available from github
}
```
## wallet_balance
```text
Return the balance of the wallet
Returns:
(float) amount of lbry credits in wallet
```
## wallet_is_address_mine
```text
Checks if an address is associated with the current wallet.
Args:
'address': (str) address to check in base58
Returns:
(bool) true, if address is associated with current wallet
```
## wallet_list
```text
List wallet addresses
Args:
None
Returns:
List of wallet addresses
```
## wallet_new_address
```text
Generate a new wallet address
Args:
None
Returns:
(str) New wallet address in base58
```
## wallet_public_key
```text
Get public key from wallet address
Args:
'address': (str) wallet address in base58
Returns:
(list) list of public keys associated with address.
Could contain more than one public key if multisig.
```
## wallet_unused_address
```text
Return an address containing no balance, will create
a new address if there is none.
Args:
None
Returns:
(str) Unused wallet address in base58
```
+3 -5
View File
@@ -1,8 +1,6 @@
import logging
log = logging.getLogger(__name__)
logging.getLogger(__name__).addHandler(logging.NullHandler())
log.setLevel(logging.ERROR)
__version__ = "0.9.2rc15"
version = tuple(__version__.split('.'))
version = (0, 2, 5)
__version__ = ".".join([str(x) for x in version])
logging.getLogger(__name__).addHandler(logging.NullHandler())
+6
View File
@@ -0,0 +1,6 @@
from constants import *
from events import *
from api import Api
from track import Track
from manager import Manager
from logging_handler import Handler
+81
View File
@@ -0,0 +1,81 @@
import functools
import json
import logging
from requests import auth
from txrequests import Session
from lbrynet import conf
from lbrynet.analytics import utils
log = logging.getLogger(__name__)
def log_response(fn):
def _log_error(failure):
log.warning('Failed to send an analytics event. %s', failure.getTraceback())
@functools.wraps(fn)
def wrapper(*args, **kwargs):
d = fn(*args, **kwargs)
d.addErrback(_log_error)
return d
return wrapper
class Api(object):
def __init__(self, session, url, write_key):
self.session = session
self.url = url
self.write_key = write_key
def post(self, endpoint, data):
# there is an issue with a timing condition with keep-alive
# that is best explained here: https://github.com/mikem23/keepalive-race
#
# If you make a request, wait just the right amount of time,
# then make another request, the requests module may opt to
# reuse the connection, but by the time the server gets it the
# timeout will have expired.
#
# by forcing the connection to close, we will disable the keep-alive.
assert endpoint[0] == '/'
headers = {"Connection": "close"}
return self.session.post(
self.url + endpoint, json=data, auth=self.auth, headers=headers)
@property
def auth(self):
return auth.HTTPBasicAuth(self.write_key, '')
@log_response
def batch(self, events):
"""Send multiple events in one request.
Each event needs to have its type specified.
"""
data = json.dumps({
'batch': events,
'sentAt': utils.now(),
})
log.debug('sending %s events', len(events))
log.debug('Data: %s', data)
return self.post('/batch', data)
@log_response
def track(self, event):
"""Send a single tracking event"""
log.debug('Sending track event: %s', event)
return self.post('/track', event)
@classmethod
def new_instance(cls, session=None):
"""Initialize an instance using values from the configuration"""
if not session:
session = Session()
return cls(
session,
conf.settings['ANALYTICS_ENDPOINT'],
utils.deobfuscate(conf.settings['ANALYTICS_TOKEN'])
)
+4
View File
@@ -0,0 +1,4 @@
"""Constants for metrics"""
BLOB_BYTES_UPLOADED = 'Blob Bytes Uploaded'
BLOB_BYTES_AVAILABLE = 'Blob Bytes Available'
+128
View File
@@ -0,0 +1,128 @@
import logging
from lbrynet.core import utils
from lbrynet.conf import LBRYUM_WALLET
log = logging.getLogger(__name__)
def get_sd_hash(stream_info):
if not stream_info:
return None
try:
return stream_info['stream']['source']['source']
except (KeyError, TypeError, ValueError):
log.debug('Failed to get sd_hash from %s', stream_info, exc_info=True)
return None
class Events(object):
def __init__(self, context, installation_id, session_id):
"""Contains all of the analytics events that can be sent
Attributes:
context: usually the output of `make_context`
installation_id: id unique to this installation. Can be anything, but
generally should be base58 encoded.
session_id: id for tracking events during this session. Can be
anything, but generally should be base58 encoded.
"""
self.context = context
self.installation_id = installation_id
self.session_id = session_id
def update_context(self, context):
self.context = context
def server_startup(self):
return self._event('Server Startup')
def server_startup_success(self):
return self._event('Server Startup Success')
def server_startup_error(self, message):
return self._event('Server Startup Error', {
'message': message,
})
def heartbeat(self):
return self._event('Heartbeat')
def download_started(self, *args, **kwargs):
properties = download_properties(*args, **kwargs)
return self._event('Download Started', properties)
def download_errored(self, *args, **kwargs):
properties = download_properties(*args, **kwargs)
return self._event('Download Errored', properties)
def download_finished(self, *args, **kwargs):
properties = download_properties(*args, **kwargs)
return self._event('Download Finished', properties)
def error(self, log_record):
"""Record when a log message of ERROR or higher was emitted"""
properties = {
'message': log_record.message,
'module': log_record.module,
'lineno': log_record.lineno,
'name': log_record.name,
'traceback': log_record.exc_text,
}
return self._event('Error', properties)
def metric_observed(self, metric_name, value):
properties = {
'value': value,
}
return self._event(metric_name, properties)
def _event(self, event, event_properties=None):
return {
'userId': 'lbry',
'event': event,
'properties': self._properties(event_properties),
'context': self.context,
'timestamp': utils.isonow()
}
def _properties(self, event_properties=None):
event_properties = event_properties or {}
properties = {
'lbry_id': self.installation_id,
'session_id': self.session_id,
}
properties.update(event_properties)
return properties
def make_context(platform, wallet):
return {
'app': {
'name': 'lbrynet',
'version': platform['lbrynet_version'],
'python_version': platform['python_version'],
'build': platform['build'],
'wallet': {
'name': wallet,
'version': platform['lbryum_version'] if wallet == LBRYUM_WALLET else None
},
},
# TODO: expand os info to give linux/osx specific info
'os': {
'name': platform['os_system'],
'version': platform['os_release']
},
'library': {
'name': 'lbrynet-analytics',
'version': '1.0.0'
},
}
def download_properties(id_, name, stream_info=None):
return {
'download_id': id_,
'name': name,
'stream_info': get_sd_hash(stream_info)
}
+14
View File
@@ -0,0 +1,14 @@
import logging
class Handler(logging.Handler):
"""A logging handler that reports errors to the analytics manager"""
def __init__(self, manager, level=logging.ERROR):
self.manager = manager
logging.Handler.__init__(self, level)
def emit(self, record):
# We need to call format to ensure that record.message and
# record.exc_text attributes are populated
self.format(record)
self.manager.send_error(record)
+116
View File
@@ -0,0 +1,116 @@
from twisted.internet import defer
from twisted.internet import task
from lbrynet import conf
from lbrynet.core import looping_call_manager
from lbrynet.core.system_info import get_platform
import constants
from api import Api
from events import Events, make_context
from track import Track
class Manager(object):
def __init__(self, analytics_api, events_generator, track):
self.analytics_api = analytics_api
self.events_generator = events_generator
self.track = track
self.looping_call_manager = self.setup_looping_calls()
self.is_started = False
@classmethod
def new_instance(cls, api=None, events=None):
if api is None:
api = Api.new_instance()
if events is None:
events = Events(
make_context(get_platform(), conf.settings['wallet']),
conf.settings.installation_id,
conf.settings.get_session_id(),
)
return cls(api, events, Track())
def update_events_generator(self, events_generator):
self.events_generator = events_generator
def _get_looping_calls(self):
return [
('send_heartbeat', self._send_heartbeat, 60),
('update_tracked_metrics', self._update_tracked_metrics, 300),
]
def setup_looping_calls(self):
call_manager = looping_call_manager.LoopingCallManager()
for name, fn, _ in self._get_looping_calls():
call_manager.register_looping_call(name, task.LoopingCall(fn))
return call_manager
def start(self):
if not self.is_started:
for name, _, interval in self._get_looping_calls():
self.looping_call_manager.start(name, interval)
self.is_started = True
def shutdown(self):
self.looping_call_manager.shutdown()
def send_server_startup(self):
event = self.events_generator.server_startup()
self.analytics_api.track(event)
def send_server_startup_success(self):
event = self.events_generator.server_startup_success()
self.analytics_api.track(event)
def send_server_startup_error(self, message):
event = self.events_generator.server_startup_error(message)
self.analytics_api.track(event)
def send_download_started(self, id_, name, stream_info=None):
event = self.events_generator.download_started(id_, name, stream_info)
self.analytics_api.track(event)
def send_download_errored(self, id_, name, stream_info=None):
event = self.events_generator.download_errored(id_, name, stream_info)
self.analytics_api.track(event)
def send_download_finished(self, id_, name, stream_info=None):
event = self.events_generator.download_finished(id_, name, stream_info)
self.analytics_api.track(event)
def send_error(self, message):
event = self.events_generator.error(message)
self.analytics_api.track(event)
def register_repeating_metric(self, event_name, value_generator, frequency=300):
lcall = task.LoopingCall(self._send_repeating_metric, event_name, value_generator)
self.looping_call_manager.register_looping_call(event_name, lcall)
lcall.start(frequency)
def _send_heartbeat(self):
heartbeat = self.events_generator.heartbeat()
self.analytics_api.track(heartbeat)
def _update_tracked_metrics(self):
should_send, value = self.track.summarize_and_reset(constants.BLOB_BYTES_UPLOADED)
if should_send:
event = self.events_generator.metric_observed(constants.BLOB_BYTES_UPLOADED, value)
self.analytics_api.track(event)
def _send_repeating_metric(self, event_name, value_generator):
result = value_generator()
if_deferred(result, self._send_repeating_metric_value, event_name)
def _send_repeating_metric_value(self, result, event_name):
should_send, value = result
if should_send:
event = self.events_generator.metric_observed(event_name, value)
self.analytics_api.track(event)
def if_deferred(maybe_deferred, callback, *args, **kwargs):
if isinstance(maybe_deferred, defer.Deferred):
maybe_deferred.addCallback(callback, *args, **kwargs)
else:
callback(maybe_deferred, *args, **kwargs)
+24
View File
@@ -0,0 +1,24 @@
import collections
class Track(object):
"""Track and summarize observations of metrics."""
def __init__(self):
self.data = collections.defaultdict(list)
def add_observation(self, metric, value):
self.data[metric].append(value)
def summarize_and_reset(self, metric, op=sum):
"""Apply `op` on the current values for `metric`.
This operation also resets the metric.
Returns:
a tuple (should_send, value)
"""
try:
values = self.data.pop(metric)
return True, op(values)
except KeyError:
return False, None
+2
View File
@@ -0,0 +1,2 @@
# dont touch this. CI server changes this during build/deployment
BUILD = "dev"
+465 -40
View File
@@ -1,48 +1,473 @@
"""
Some network wide and also application specific parameters
"""
import base58
import json
import logging
import os
import sys
import yaml
import envparse
from appdirs import user_data_dir
from lbrynet.core import utils
log = logging.getLogger(__name__)
ENV_NAMESPACE = 'LBRY_'
LBRYCRD_WALLET = 'lbrycrd'
LBRYUM_WALLET = 'lbryum'
PTC_WALLET = 'ptc'
PROTOCOL_PREFIX = 'lbry'
APP_NAME = 'LBRY'
LINUX = 1
DARWIN = 2
WINDOWS = 3
KB = 2 ** 10
MB = 2 ** 20
DEFAULT_DHT_NODES = [
('lbrynet1.lbry.io', 4444),
('lbrynet2.lbry.io', 4444),
('lbrynet3.lbry.io', 4444)
]
settings_decoders = {
'.json': json.loads,
'.yml': yaml.load
}
settings_encoders = {
'.json': json.dumps,
'.yml': yaml.safe_dump
}
if sys.platform.startswith('darwin'):
platform = DARWIN
default_download_directory = os.path.join(os.path.expanduser('~'), 'Downloads')
default_data_dir = user_data_dir('LBRY')
default_lbryum_dir = os.path.join(os.path.expanduser('~'), '.lbryum')
elif sys.platform.startswith('win'):
platform = WINDOWS
from lbrynet.winhelpers.knownpaths import get_path, FOLDERID, UserHandle
default_download_directory = get_path(FOLDERID.Downloads, UserHandle.current)
default_data_dir = os.path.join(
get_path(FOLDERID.RoamingAppData, UserHandle.current), 'lbrynet')
default_lbryum_dir = os.path.join(
get_path(FOLDERID.RoamingAppData, UserHandle.current), 'lbryum')
else:
platform = LINUX
default_download_directory = os.path.join(os.path.expanduser('~'), 'Downloads')
default_data_dir = os.path.join(os.path.expanduser('~'), '.lbrynet')
default_lbryum_dir = os.path.join(os.path.expanduser('~'), '.lbryum')
ICON_PATH = 'icons' if platform is WINDOWS else 'app.icns'
MAX_HANDSHAKE_SIZE = 2**16
MAX_REQUEST_SIZE = 2**16
MAX_BLOB_REQUEST_SIZE = 2**16
MAX_RESPONSE_INFO_SIZE = 2**16
MAX_BLOB_INFOS_TO_REQUEST = 20
BLOBFILES_DIR = ".blobfiles"
BLOB_SIZE = 2**21
MIN_BLOB_DATA_PAYMENT_RATE = .005 # points/megabyte
MIN_BLOB_INFO_PAYMENT_RATE = .02 # points/1000 infos
MIN_VALUABLE_BLOB_INFO_PAYMENT_RATE = .05 # points/1000 infos
MIN_VALUABLE_BLOB_HASH_PAYMENT_RATE = .05 # points/1000 infos
MAX_CONNECTIONS_PER_STREAM = 5
def server_port(server_and_port):
server, port = server_and_port.split(':')
return server, int(port)
KNOWN_DHT_NODES = [('104.236.42.182', 4000),
('lbryum1.lbry.io', 4444),
('lbryum2.lbry.io', 4444),
('lbryum3.lbry.io', 4444)]
POINTTRADER_SERVER = 'http://ec2-54-187-192-68.us-west-2.compute.amazonaws.com:2424'
#POINTTRADER_SERVER = 'http://127.0.0.1:2424'
class Env(envparse.Env):
"""An Env parser that automatically namespaces the variables with LBRY"""
LOG_FILE_NAME = "lbrynet.log"
LOG_POST_URL = "https://lbry.io/log-upload"
def __init__(self, **schema):
self.original_schema = schema
my_schema = {
self._convert_key(key): self._convert_value(value)
for key, value in schema.items()
}
envparse.Env.__init__(self, **my_schema)
CRYPTSD_FILE_EXTENSION = ".cryptsd"
def __call__(self, key, *args, **kwargs):
my_key = self._convert_key(key)
return super(Env, self).__call__(my_key, *args, **kwargs)
API_INTERFACE = "localhost"
API_ADDRESS = "lbryapi"
API_PORT = 5279
ICON_PATH = "app.icns"
APP_NAME = "LBRY"
API_CONNECTION_STRING = "http://%s:%i/%s" % (API_INTERFACE, API_PORT, API_ADDRESS)
UI_ADDRESS = "http://" + API_INTERFACE + ":" + str(API_PORT)
PROTOCOL_PREFIX = "lbry"
@staticmethod
def _convert_key(key):
return ENV_NAMESPACE + key.upper()
@staticmethod
def _convert_value(value):
""" Allow value to be specified as a tuple or list.
If you do this, the tuple/list must be of the
form (cast, default) or (cast, default, subcast)
"""
if isinstance(value, (tuple, list)):
new_value = {'cast': value[0], 'default': value[1]}
if len(value) == 3:
new_value['subcast'] = value[2]
return new_value
return value
TYPE_DEFAULT = 'default'
TYPE_PERSISTED = 'persisted'
TYPE_ENV = 'env'
TYPE_CLI = 'cli'
TYPE_RUNTIME = 'runtime'
FIXED_SETTINGS = {
'ANALYTICS_ENDPOINT': 'https://api.segment.io/v1',
'ANALYTICS_TOKEN': 'Ax5LZzR1o3q3Z3WjATASDwR5rKyHH0qOIRIbLmMXn2H=',
'API_ADDRESS': 'lbryapi',
'APP_NAME': APP_NAME,
'BLOBFILES_DIR': 'blobfiles',
'BLOB_SIZE': 2 * MB,
'CRYPTSD_FILE_EXTENSION': '.cryptsd',
'CURRENCIES': {
'BTC': {'type': 'crypto'},
'LBC': {'type': 'crypto'},
'USD': {'type': 'fiat'},
},
'DB_REVISION_FILE_NAME': 'db_revision',
'ICON_PATH': ICON_PATH,
'LOGGLY_TOKEN': 'LJEzATH4AzRgAwxjAP00LwZ2YGx3MwVgZTMuBQZ3MQuxLmOv',
'LOG_FILE_NAME': 'lbrynet.log',
'LOG_POST_URL': 'https://lbry.io/log-upload',
'MAX_BLOB_REQUEST_SIZE': 64 * KB,
'MAX_HANDSHAKE_SIZE': 64 * KB,
'MAX_REQUEST_SIZE': 64 * KB,
'MAX_RESPONSE_INFO_SIZE': 64 * KB,
'MAX_BLOB_INFOS_TO_REQUEST': 20,
'PROTOCOL_PREFIX': PROTOCOL_PREFIX,
'SLACK_WEBHOOK': ('nUE0pUZ6Yl9bo29epl5moTSwnl5wo20ip2IlqzywMKZiIQSFZR5'
'AHx4mY0VmF0WQZ1ESEP9kMHZlp1WzJwWOoKN3ImR1M2yUAaMyqGZ='),
'SOURCE_TYPES': ['lbry_sd_hash', 'url', 'btih'],
'WALLET_TYPES': [LBRYUM_WALLET, LBRYCRD_WALLET],
}
ADJUSTABLE_SETTINGS = {
# By default, daemon will block all cross origin requests
# but if this is set, this value will be used for the
# Access-Control-Allow-Origin. For example
# set to '*' to allow all requests, or set to 'http://localhost:8080'
# if you're running a test UI on that port
'allowed_origin': (str, ''),
# Changing this value is not-advised as it could potentially
# expose the lbrynet daemon to the outside world which would
# give an attacker access to your wallet and you could lose
# all of your credits.
'api_host': (str, 'localhost'),
'api_port': (int, 5279),
'cache_time': (int, 150),
'check_ui_requirements': (bool, True),
'data_dir': (str, default_data_dir),
'data_rate': (float, .0001), # points/megabyte
'default_ui_branch': (str, 'master'),
'delete_blobs_on_remove': (bool, True),
'dht_node_port': (int, 4444),
'download_directory': (str, default_download_directory),
'download_timeout': (int, 180),
'host_ui': (bool, True),
'is_generous_host': (bool, True),
'known_dht_nodes': (list, DEFAULT_DHT_NODES, server_port),
# TODO: this should not be configured; move it elsewhere
'last_version': (dict, {'lbrynet': '0.0.1', 'lbryum': '0.0.1'}),
'lbryum_wallet_dir': (str, default_lbryum_dir),
'local_ui_path': (str, ''),
'max_connections_per_stream': (int, 5),
'max_download': (float, 0.0),
# TODO: this field is more complicated than it needs to be because
# it goes through a Fee validator when loaded by the exchange rate
# manager. Look into refactoring the exchange rate conversion to
# take in a simpler form.
#
# TODO: writing json on the cmd line is a pain, come up with a nicer
# parser for this data structure. (maybe MAX_KEY_FEE': USD:25
'max_key_fee': (json.loads, {'currency':'USD', 'amount': 25.0, 'address':''}),
'max_search_results': (int, 25),
'max_upload': (float, 0.0),
'min_info_rate': (float, .02), # points/1000 infos
'min_valuable_hash_rate': (float, .05), # points/1000 infos
'min_valuable_info_rate': (float, .05), # points/1000 infos
'peer_port': (int, 3333),
'pointtrader_server': (str, 'http://127.0.0.1:2424'),
'reflector_port': (int, 5566),
'reflect_uploads': (bool, True),
'reflector_servers': (list, [('reflector.lbry.io', 5566)], server_port),
'run_on_startup': (bool, False),
'run_reflector_server': (bool, False),
'sd_download_timeout': (int, 3),
'share_debug_info': (bool, True), # whether to share diagnostic info with LBRY
'peer_search_timeout': (int, 3),
'search_servers': (list, ['lighthouse1.lbry.io:50005']),
'search_timeout': (float, 5.0),
'startup_scripts': (list, []),
'ui_branch': (str, 'master'),
'use_auth_http': (bool, False),
'use_upnp': (bool, True),
'wallet': (str, LBRYUM_WALLET),
}
class Config(object):
def __init__(self, fixed_defaults, adjustable_defaults, persisted_settings=None,
environment=None, cli_settings=None):
self._installation_id = None
self._session_id = base58.b58encode(utils.generate_id())
self._fixed_defaults = fixed_defaults
self._adjustable_defaults = adjustable_defaults
self._data = {
TYPE_DEFAULT: {}, # defaults
TYPE_PERSISTED: {}, # stored settings from daemon_settings.yml (or from a db, etc)
TYPE_ENV: {}, # settings from environment variables
TYPE_CLI: {}, # command-line arguments
TYPE_RUNTIME: {}, # set during runtime (using self.set(), etc)
}
# the order in which a piece of data is searched for. earlier types override later types
self._search_order = (
TYPE_RUNTIME, TYPE_CLI, TYPE_ENV, TYPE_PERSISTED, TYPE_DEFAULT
)
self._data[TYPE_DEFAULT].update(self._fixed_defaults)
self._data[TYPE_DEFAULT].update(
{k: v[1] for (k, v) in self._adjustable_defaults.iteritems()})
if persisted_settings is None:
persisted_settings = {}
self._validate_settings(persisted_settings)
self._data[TYPE_PERSISTED].update(persisted_settings)
env_settings = self._parse_environment(environment)
self._validate_settings(env_settings)
self._data[TYPE_ENV].update(env_settings)
if cli_settings is None:
cli_settings = {}
self._validate_settings(cli_settings)
self._data[TYPE_CLI].update(cli_settings)
def __repr__(self):
return self.get_current_settings_dict().__repr__()
def __iter__(self):
for k in self._data[TYPE_DEFAULT].iterkeys():
yield k
def __getitem__(self, name):
return self.get(name)
def __setitem__(self, name, value):
return self.set(name, value)
def __contains__(self, name):
return name in self._data[TYPE_DEFAULT]
@staticmethod
def _parse_environment(environment):
env_settings = {}
if environment is not None:
assert isinstance(environment, Env)
for opt in environment.original_schema:
if environment(opt) is not None:
env_settings[opt] = environment(opt)
return env_settings
def _assert_valid_data_type(self, data_type):
assert data_type in self._data, KeyError('{} in is not a valid data type'.format(data_type))
def get_valid_setting_names(self):
return self._data[TYPE_DEFAULT].keys()
def _is_valid_setting(self, name):
return name in self.get_valid_setting_names()
def _assert_valid_setting(self, name):
assert self._is_valid_setting(name), \
KeyError('{} is not a valid setting'.format(name))
def _validate_settings(self, data):
invalid_settings = set(data.keys()) - set(self.get_valid_setting_names())
if len(invalid_settings) > 0:
raise KeyError('invalid settings: {}'.format(', '.join(invalid_settings)))
def _assert_editable_setting(self, name):
self._assert_valid_setting(name)
assert name not in self._fixed_defaults, \
ValueError('{} is not an editable setting'.format(name))
def get(self, name, data_type=None):
"""Get a config value
Args:
name: the name of the value to get
data_type: if given, get the value from a specific data set (see below)
Returns: the config value for the given name
If data_type is None, get() will search for the given name in each data set, in
order of precedence. It will return the first value it finds. This is the "effective"
value of a config name. For example, ENV values take precedence over DEFAULT values,
so if a value is present in ENV and in DEFAULT, the ENV value will be returned
"""
self._assert_valid_setting(name)
if data_type is not None:
self._assert_valid_data_type(data_type)
return self._data[data_type][name]
for data_type in self._search_order:
if name in self._data[data_type]:
return self._data[data_type][name]
raise KeyError('{} is not a valid setting'.format(name))
def set(self, name, value, data_types=(TYPE_RUNTIME,)):
"""Set a config value
Args:
name: the name of the value to set
value: the value
data_types: what type(s) of data this is
Returns: None
By default, this sets the RUNTIME value of a config. If you wish to set other
data types (e.g. PERSISTED values to save to a file, CLI values from parsed
command-line options, etc), you can specify that with the data_types param
"""
self._assert_editable_setting(name)
for data_type in data_types:
self._assert_valid_data_type(data_type)
self._data[data_type][name] = value
def update(self, updated_settings, data_types=(TYPE_RUNTIME,)):
for k, v in updated_settings.iteritems():
try:
self.set(k, v, data_types=data_types)
except (KeyError, AssertionError):
pass
def get_current_settings_dict(self):
current_settings = {}
for key in self.get_valid_setting_names():
current_settings[key] = self.get(key)
return current_settings
def get_adjustable_settings_dict(self):
return {
key: val for key, val in self.get_current_settings_dict().iteritems()
if key in self._adjustable_defaults
}
def save_conf_file_settings(self):
path = self.get_conf_filename()
ext = os.path.splitext(path)[1]
encoder = settings_encoders.get(ext, False)
assert encoder is not False, 'Unknown settings format %s' % ext
with open(path, 'w') as settings_file:
settings_file.write(encoder(self._data[TYPE_PERSISTED]))
def load_conf_file_settings(self):
path = self.get_conf_filename()
ext = os.path.splitext(path)[1]
decoder = settings_decoders.get(ext, False)
assert decoder is not False, 'Unknown settings format %s' % ext
try:
with open(path, 'r') as settings_file:
data = settings_file.read()
decoded = self._fix_old_conf_file_settings(decoder(data))
log.info('Loaded settings file: %s', path)
self._validate_settings(decoded)
self._data[TYPE_PERSISTED].update(decoded)
except (IOError, OSError) as err:
log.info('%s: Failed to update settings from %s', err, path)
def _fix_old_conf_file_settings(self, settings_dict):
if 'API_INTERFACE' in settings_dict:
settings_dict['api_host'] = settings_dict['API_INTERFACE']
del settings_dict['API_INTERFACE']
if 'startup_scripts' in settings_dict:
del settings_dict['startup_scripts']
if 'upload_log' in settings_dict:
settings_dict['share_debug_info'] = settings_dict['upload_log']
del settings_dict['upload_log']
for key in settings_dict.keys():
if not self._is_valid_setting(key):
log.warning('Ignoring invalid conf file setting: %s', key)
del settings_dict[key]
return settings_dict
def ensure_data_dir(self):
# although there is a risk of a race condition here we don't
# expect there to be multiple processes accessing this
# directory so the risk can be ignored
if not os.path.isdir(self['data_dir']):
os.makedirs(self['data_dir'])
return self['data_dir']
def get_log_filename(self):
"""
Return the log file for this platform.
Also ensure the containing directory exists.
"""
return os.path.join(self.ensure_data_dir(), self['LOG_FILE_NAME'])
def get_api_connection_string(self):
return 'http://%s:%i/%s' % (self['api_host'], self['api_port'], self['API_ADDRESS'])
def get_ui_address(self):
return 'http://%s:%i' % (self['api_host'], self['api_port'])
def get_db_revision_filename(self):
return os.path.join(self.ensure_data_dir(), self['DB_REVISION_FILE_NAME'])
def get_conf_filename(self):
data_dir = self.ensure_data_dir()
yml_path = os.path.join(data_dir, 'daemon_settings.yml')
json_path = os.path.join(data_dir, 'daemon_settings.json')
if os.path.isfile(yml_path):
return yml_path
elif os.path.isfile(json_path):
return json_path
else:
return yml_path
def get_installation_id(self):
install_id_filename = os.path.join(self.ensure_data_dir(), "install_id")
if not self._installation_id:
if os.path.isfile(install_id_filename):
with open(install_id_filename, "r") as install_id_file:
self._installation_id = install_id_file.read()
if not self._installation_id:
self._installation_id = base58.b58encode(utils.generate_id())
with open(install_id_filename, "w") as install_id_file:
install_id_file.write(self._installation_id)
return self._installation_id
def get_session_id(self):
return self._session_id
# type: Config
settings = None
def get_default_env():
env_defaults = {}
for k, v in ADJUSTABLE_SETTINGS.iteritems():
if len(v) == 3:
env_defaults[k] = (v[0], None, v[2])
else:
env_defaults[k] = (v[0], None)
return Env(**env_defaults)
def initialize_settings(load_conf_file=True):
global settings
if settings is None:
settings = Config(FIXED_SETTINGS, ADJUSTABLE_SETTINGS,
environment=get_default_env())
settings.installation_id = settings.get_installation_id()
if load_conf_file:
settings.load_conf_file_settings()
DEFAULT_WALLET = "lbryum"
WALLET_TYPES = ["lbryum", "lbrycrd"]
DEFAULT_TIMEOUT = 30
DEFAULT_MAX_SEARCH_RESULTS = 25
DEFAULT_MAX_KEY_FEE = 100.0
DEFAULT_SEARCH_TIMEOUT = 3.0
DEFAULT_CACHE_TIME = 3600
DEFAULT_UI_BRANCH = "master"
+111
View File
@@ -0,0 +1,111 @@
import logging
import random
import time
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from decimal import Decimal
log = logging.getLogger(__name__)
class BlobAvailabilityTracker(object):
"""
Class to track peer counts for known blobs, and to discover new popular blobs
Attributes:
availability (dict): dictionary of peers for known blobs
"""
def __init__(self, blob_manager, peer_finder, dht_node):
self.availability = {}
self._last_mean_availability = Decimal(0.0)
self._blob_manager = blob_manager
self._peer_finder = peer_finder
self._dht_node = dht_node
self._check_popular = LoopingCall(self._update_most_popular)
self._check_mine = LoopingCall(self._update_mine)
def start(self):
log.info("Starting %s", self)
self._check_popular.start(30)
self._check_mine.start(600)
def stop(self):
log.info("Stopping %s", self)
if self._check_popular.running:
self._check_popular.stop()
if self._check_mine.running:
self._check_mine.stop()
def get_blob_availability(self, blob, timeout=None):
def _get_peer_count(peers):
have_blob = sum(1 for peer in peers if peer.is_available())
return {blob: have_blob}
d = self._peer_finder.find_peers_for_blob(blob, timeout)
d.addCallback(_get_peer_count)
return d
def get_availability_for_blobs(self, blobs, timeout=None):
dl = [self.get_blob_availability(blob, timeout) for blob in blobs if blob]
d = defer.DeferredList(dl)
d.addCallback(lambda results: [val for success, val in results if success])
return d
@property
def last_mean_availability(self):
return max(Decimal(0.01), self._last_mean_availability)
def _update_peers_for_blob(self, blob):
def _save_peer_info(blob_hash, peers):
v = {blob_hash: peers}
self.availability.update(v)
return v
d = self._peer_finder.find_peers_for_blob(blob)
d.addCallback(lambda r: [[c.host, c.port, c.is_available()] for c in r])
d.addCallback(lambda peers: _save_peer_info(blob, peers))
return d
def _get_most_popular(self):
dl = []
for (hash, _) in self._dht_node.get_most_popular_hashes(100):
encoded = hash.encode('hex')
dl.append(self._update_peers_for_blob(encoded))
return defer.DeferredList(dl)
def _update_most_popular(self):
d = self._get_most_popular()
d.addCallback(lambda _: self._set_mean_peers())
def _update_mine(self):
def _get_peers(blobs):
dl = []
for hash in blobs:
dl.append(self._update_peers_for_blob(hash))
return defer.DeferredList(dl)
def sample(blobs):
return random.sample(blobs, min(len(blobs), 100))
start = time.time()
log.debug('==> Updating the peers for my blobs')
d = self._blob_manager.get_all_verified_blobs()
# as far as I can tell, this only is used to set _last_mean_availability
# which... seems like a very expensive operation for such little payoff.
# so taking a sample should get about the same effect as querying the entire
# list of blobs
d.addCallback(sample)
d.addCallback(_get_peers)
d.addCallback(lambda _: self._set_mean_peers())
d.addCallback(lambda _: log.debug('<== Done updating peers for my blobs. Took %s seconds',
time.time() - start))
# although unused, need to return or else the looping call
# could overrun on a previous call
return d
def _set_mean_peers(self):
num_peers = [len(self.availability[blob]) for blob in self.availability]
mean = Decimal(sum(num_peers)) / Decimal(max(1, len(num_peers)))
self._last_mean_availability = mean
+1 -1
View File
@@ -15,4 +15,4 @@ class BlobInfo(object):
def __init__(self, blob_hash, blob_num, length):
self.blob_hash = blob_hash
self.blob_num = blob_num
self.length = length
self.length = length
+154 -227
View File
@@ -2,13 +2,12 @@ import logging
import os
import time
import sqlite3
from twisted.internet import threads, defer
from twisted.python.failure import Failure
from twisted.enterprise import adbapi
from lbrynet.core.HashBlob import BlobFile, TempBlob, BlobFileCreator, TempBlobCreator
from lbrynet.core.server.DHTHashAnnouncer import DHTHashSupplier
from lbrynet.core.utils import is_valid_blobhash
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.core.Error import NoSuchBlobError
from lbrynet.core.sqlite_helpers import rerun_if_locked
@@ -25,19 +24,19 @@ class BlobManager(DHTHashSupplier):
def setup(self):
pass
def get_blob(self, blob_hash, upload_allowed, length):
def get_blob(self, blob_hash, length=None):
pass
def get_blob_creator(self):
pass
def _make_new_blob(self, blob_hash, upload_allowed, length):
def _make_new_blob(self, blob_hash, length):
pass
def blob_completed(self, blob, next_announce_time=None):
pass
def completed_blobs(self, blobs_to_check):
def completed_blobs(self, blobhashes_to_check):
pass
def hashes_to_announce(self):
@@ -49,12 +48,6 @@ class BlobManager(DHTHashSupplier):
def delete_blob(self, blob_hash):
pass
def get_blob_length(self, blob_hash):
pass
def check_consistency(self):
pass
def blob_requested(self, blob_hash):
pass
@@ -67,7 +60,23 @@ class BlobManager(DHTHashSupplier):
def blob_paid_for(self, blob_hash, amount):
pass
def get_all_verified_blobs(self):
pass
def add_blob_to_download_history(self, blob_hash, host, rate):
pass
def add_blob_to_upload_history(self, blob_hash, host, rate):
pass
def _immediate_announce(self, blob_hashes):
if self.hash_announcer:
return self.hash_announcer.immediate_announce(blob_hashes)
# TODO: Having different managers for different blobs breaks the
# abstraction of a HashBlob. Why should the management of blobs
# care what kind of Blob it has?
class DiskBlobManager(BlobManager):
"""This class stores blobs on the hard disk"""
def __init__(self, hash_announcer, blob_dir, db_dir):
@@ -77,8 +86,10 @@ class DiskBlobManager(BlobManager):
self.db_conn = None
self.blob_type = BlobFile
self.blob_creator_type = BlobFileCreator
# TODO: consider using an LRU for blobs as there could potentially
# be thousands of blobs loaded up, many stale
self.blobs = {}
self.blob_hashes_to_delete = {} # {blob_hash: being_deleted (True/False)}
self.blob_hashes_to_delete = {} # {blob_hash: being_deleted (True/False)}
self._next_manage_call = None
def setup(self):
@@ -93,69 +104,50 @@ class DiskBlobManager(BlobManager):
if self._next_manage_call is not None and self._next_manage_call.active():
self._next_manage_call.cancel()
self._next_manage_call = None
#d = self.db_conn.close()
self.db_conn = None
return defer.succeed(True)
def get_blob(self, blob_hash, upload_allowed, length=None):
"""Return a blob identified by blob_hash, which may be a new blob or a blob that is already on the hard disk"""
# TODO: if blob.upload_allowed and upload_allowed is False, change upload_allowed in blob and on disk
def get_blob(self, blob_hash, length=None):
"""Return a blob identified by blob_hash, which may be a new blob or a
blob that is already on the hard disk
"""
assert length is None or isinstance(length, int)
if blob_hash in self.blobs:
return defer.succeed(self.blobs[blob_hash])
return self._make_new_blob(blob_hash, upload_allowed, length)
return self._make_new_blob(blob_hash, length)
def get_blob_creator(self):
return self.blob_creator_type(self, self.blob_dir)
def _make_new_blob(self, blob_hash, upload_allowed, length=None):
blob = self.blob_type(self.blob_dir, blob_hash, upload_allowed, length)
def _make_new_blob(self, blob_hash, length=None):
log.debug('Making a new blob for %s', blob_hash)
blob = self.blob_type(self.blob_dir, blob_hash, length)
self.blobs[blob_hash] = blob
d = self._completed_blobs([blob_hash])
def check_completed(completed_blobs):
def set_length(length):
blob.length = length
if len(completed_blobs) == 1 and completed_blobs[0] == blob_hash:
blob.verified = True
inner_d = self._get_blob_length(blob_hash)
inner_d.addCallback(set_length)
inner_d.addCallback(lambda _: blob)
else:
inner_d = defer.succeed(blob)
return inner_d
d.addCallback(check_completed)
return d
return defer.succeed(blob)
def blob_completed(self, blob, next_announce_time=None):
if next_announce_time is None:
next_announce_time = time.time()
return self._add_completed_blob(blob.blob_hash, blob.length,
time.time(), next_announce_time)
next_announce_time = self.get_next_announce_time()
d = self._add_completed_blob(blob.blob_hash, blob.length, next_announce_time)
d.addCallback(lambda _: self._immediate_announce([blob.blob_hash]))
return d
def completed_blobs(self, blobs_to_check):
return self._completed_blobs(blobs_to_check)
def completed_blobs(self, blobhashes_to_check):
return self._completed_blobs(blobhashes_to_check)
def hashes_to_announce(self):
next_announce_time = time.time() + self.hash_reannounce_time
return self._get_blobs_to_announce(next_announce_time)
return self._get_blobs_to_announce()
def creator_finished(self, blob_creator):
log.debug("blob_creator.blob_hash: %s", blob_creator.blob_hash)
assert blob_creator.blob_hash is not None
assert blob_creator.blob_hash not in self.blobs
assert blob_creator.length is not None
new_blob = self.blob_type(self.blob_dir, blob_creator.blob_hash, True, blob_creator.length)
new_blob.verified = True
new_blob = self.blob_type(self.blob_dir, blob_creator.blob_hash, blob_creator.length)
self.blobs[blob_creator.blob_hash] = new_blob
if self.hash_announcer is not None:
self.hash_announcer.immediate_announce([blob_creator.blob_hash])
next_announce_time = time.time() + self.hash_reannounce_time
d = self.blob_completed(new_blob, next_announce_time)
else:
d = self.blob_completed(new_blob)
self._immediate_announce([blob_creator.blob_hash])
next_announce_time = self.get_next_announce_time()
d = self.blob_completed(new_blob, next_announce_time)
return d
def delete_blobs(self, blob_hashes):
@@ -163,19 +155,23 @@ class DiskBlobManager(BlobManager):
if not blob_hash in self.blob_hashes_to_delete:
self.blob_hashes_to_delete[blob_hash] = False
def update_all_last_verified_dates(self, timestamp):
return self._update_all_last_verified_dates(timestamp)
def immediate_announce_all_blobs(self):
d = self._get_all_verified_blob_hashes()
d.addCallback(self.hash_announcer.immediate_announce)
d.addCallback(self._immediate_announce)
return d
def get_blob_length(self, blob_hash):
return self._get_blob_length(blob_hash)
def get_all_verified_blobs(self):
d = self._get_all_verified_blob_hashes()
d.addCallback(self.completed_blobs)
return d
def check_consistency(self):
return self._check_consistency()
def add_blob_to_download_history(self, blob_hash, host, rate):
d = self._add_blob_to_download_history(blob_hash, host, rate)
return d
def add_blob_to_upload_history(self, blob_hash, host, rate):
d = self._add_blob_to_upload_history(blob_hash, host, rate)
return d
def _manage(self):
from twisted.internet import reactor
@@ -206,7 +202,8 @@ class DiskBlobManager(BlobManager):
d = defer.succeed(True)
def log_error(err):
log.warning("Failed to delete completed blobs from the db: %s", err.getErrorMessage())
log.warning(
"Failed to delete completed blobs from the db: %s", err.getErrorMessage())
d.addErrback(log_error)
return d
@@ -220,8 +217,10 @@ class DiskBlobManager(BlobManager):
for blob_hash, being_deleted in self.blob_hashes_to_delete.items():
if being_deleted is False:
self.blob_hashes_to_delete[blob_hash] = True
d = self.get_blob(blob_hash, True)
d.addCallbacks(delete, set_not_deleting, callbackArgs=(blob_hash,), errbackArgs=(blob_hash,))
d = self.get_blob(blob_hash)
d.addCallbacks(
delete, set_not_deleting,
callbackArgs=(blob_hash,), errbackArgs=(blob_hash,))
ds.append(d)
dl = defer.DeferredList(ds, consumeErrors=True)
dl.addCallback(delete_from_db)
@@ -235,72 +234,46 @@ class DiskBlobManager(BlobManager):
# one that opened it. The individual connections in the pool are not used in multiple
# threads.
self.db_conn = adbapi.ConnectionPool('sqlite3', self.db_file, check_same_thread=False)
return self.db_conn.runQuery("create table if not exists blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" last_verified_time real, " +
" next_announce_time real"
")")
def create_tables(transaction):
transaction.execute("create table if not exists blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" last_verified_time real, " +
" next_announce_time real)")
transaction.execute("create table if not exists download (" +
" id integer primary key autoincrement, " +
" blob text, " +
" host text, " +
" rate float, " +
" ts integer)")
transaction.execute("create table if not exists upload (" +
" id integer primary key autoincrement, " +
" blob text, " +
" host text, " +
" rate float, " +
" ts integer)")
return self.db_conn.runInteraction(create_tables)
@rerun_if_locked
def _add_completed_blob(self, blob_hash, length, timestamp, next_announce_time=None):
def _add_completed_blob(self, blob_hash, length, next_announce_time):
log.debug("Adding a completed blob. blob_hash=%s, length=%s", blob_hash, str(length))
if next_announce_time is None:
next_announce_time = timestamp
d = self.db_conn.runQuery("insert into blobs values (?, ?, ?, ?)",
(blob_hash, length, timestamp, next_announce_time))
d = self.db_conn.runQuery(
"insert into blobs (blob_hash, blob_length, next_announce_time) values (?, ?, ?)",
(blob_hash, length, next_announce_time)
)
d.addErrback(lambda err: err.trap(sqlite3.IntegrityError))
return d
@rerun_if_locked
def _completed_blobs(self, blobs_to_check):
blobs_to_check = filter(is_valid_blobhash, blobs_to_check)
def get_blobs_in_db(db_transaction):
blobs_in_db = [] # [(blob_hash, last_verified_time)]
for b in blobs_to_check:
result = db_transaction.execute("select last_verified_time from blobs where blob_hash = ?",
(b,))
row = result.fetchone()
if row is not None:
blobs_in_db.append((b, row[0]))
return blobs_in_db
def get_valid_blobs(blobs_in_db):
def check_blob_verified_date(b, verified_time):
file_path = os.path.join(self.blob_dir, b)
if os.path.isfile(file_path):
if verified_time > os.path.getctime(file_path):
return True
return False
def return_valid_blobs(results):
valid_blobs = []
for (b, verified_date), (success, result) in zip(blobs_in_db, results):
if success is True and result is True:
valid_blobs.append(b)
return valid_blobs
ds = []
for b, verified_date in blobs_in_db:
ds.append(threads.deferToThread(check_blob_verified_date, b, verified_date))
dl = defer.DeferredList(ds)
dl.addCallback(return_valid_blobs)
return dl
d = self.db_conn.runInteraction(get_blobs_in_db)
d.addCallback(get_valid_blobs)
return d
@rerun_if_locked
def _get_blob_length(self, blob):
d = self.db_conn.runQuery("select blob_length from blobs where blob_hash = ?", (blob,))
d.addCallback(lambda r: r[0][0] if len(r) else Failure(NoSuchBlobError(blob)))
return d
#length, verified_time, next_announce_time = json.loads(self.db.Get(blob))
#return length
@defer.inlineCallbacks
def _completed_blobs(self, blobhashes_to_check):
"""Returns of the blobhashes_to_check, which are valid"""
blobs = yield defer.DeferredList([self.get_blob(b) for b in blobhashes_to_check])
blob_hashes = [b.blob_hash for success, b in blobs if success and b.verified]
defer.returnValue(blob_hashes)
@rerun_if_locked
def _update_blob_verified_timestamp(self, blob, timestamp):
@@ -308,7 +281,7 @@ class DiskBlobManager(BlobManager):
(blob, timestamp))
@rerun_if_locked
def _get_blobs_to_announce(self, next_announce_time):
def _get_blobs_to_announce(self):
def get_and_update(transaction):
timestamp = time.time()
@@ -316,16 +289,16 @@ class DiskBlobManager(BlobManager):
"where next_announce_time < ? and blob_hash is not null",
(timestamp,))
blobs = [b for b, in r.fetchall()]
transaction.execute("update blobs set next_announce_time = ? where next_announce_time < ?",
(next_announce_time, timestamp))
next_announce_time = self.get_next_announce_time(len(blobs))
transaction.execute(
"update blobs set next_announce_time = ? where next_announce_time < ?",
(next_announce_time, timestamp))
log.debug("Got %s blobs to announce, next announce time is in %s seconds",
len(blobs), next_announce_time-time.time())
return blobs
return self.db_conn.runInteraction(get_and_update)
@rerun_if_locked
def _update_all_last_verified_dates(self, timestamp):
return self.db_conn.runQuery("update blobs set last_verified_date = ?", (timestamp,))
@rerun_if_locked
def _delete_blobs_from_db(self, blob_hashes):
@@ -335,90 +308,41 @@ class DiskBlobManager(BlobManager):
return self.db_conn.runInteraction(delete_blobs)
@rerun_if_locked
def _check_consistency(self):
ALREADY_VERIFIED = 1
NEWLY_VERIFIED = 2
INVALID = 3
current_time = time.time()
d = self.db_conn.runQuery("select blob_hash, blob_length, last_verified_time from blobs")
def check_blob(blob_hash, blob_length, verified_time):
file_path = os.path.join(self.blob_dir, blob_hash)
if os.path.isfile(file_path):
if verified_time >= os.path.getctime(file_path):
return ALREADY_VERIFIED
else:
h = get_lbry_hash_obj()
len_so_far = 0
f = open(file_path)
while True:
data = f.read(2**12)
if not data:
break
h.update(data)
len_so_far += len(data)
if len_so_far == blob_length and h.hexdigest() == blob_hash:
return NEWLY_VERIFIED
return INVALID
def do_check(blobs):
already_verified = []
newly_verified = []
invalid = []
for blob_hash, blob_length, verified_time in blobs:
status = check_blob(blob_hash, blob_length, verified_time)
if status == ALREADY_VERIFIED:
already_verified.append(blob_hash)
elif status == NEWLY_VERIFIED:
newly_verified.append(blob_hash)
else:
invalid.append(blob_hash)
return already_verified, newly_verified, invalid
def update_newly_verified(transaction, blobs):
for b in blobs:
transaction.execute("update blobs set last_verified_time = ? where blob_hash = ?",
(current_time, b))
def check_blobs(blobs):
@rerun_if_locked
def update_and_return(status_lists):
already_verified, newly_verified, invalid = status_lists
d = self.db_conn.runInteraction(update_newly_verified, newly_verified)
d.addCallback(lambda _: status_lists)
return d
d = threads.deferToThread(do_check, blobs)
d.addCallback(update_and_return)
return d
d.addCallback(check_blobs)
return d
@rerun_if_locked
def _get_all_verified_blob_hashes(self):
d = self.db_conn.runQuery("select blob_hash, last_verified_time from blobs")
d = self.db_conn.runQuery("select blob_hash from blobs")
def get_verified_blobs(blobs):
verified_blobs = []
for blob_hash, verified_time in blobs:
for blob_hash, in blobs:
file_path = os.path.join(self.blob_dir, blob_hash)
if os.path.isfile(file_path):
if verified_time > os.path.getctime(file_path):
verified_blobs.append(blob_hash)
verified_blobs.append(blob_hash)
return verified_blobs
d.addCallback(lambda blobs: threads.deferToThread(get_verified_blobs, blobs))
return d
@rerun_if_locked
def _add_blob_to_download_history(self, blob_hash, host, rate):
ts = int(time.time())
d = self.db_conn.runQuery(
"insert into download values (null, ?, ?, ?, ?) ",
(blob_hash, str(host), float(rate), ts))
return d
@rerun_if_locked
def _add_blob_to_upload_history(self, blob_hash, host, rate):
ts = int(time.time())
d = self.db_conn.runQuery(
"insert into upload values (null, ?, ?, ?, ?) ",
(blob_hash, str(host), float(rate), ts))
return d
# TODO: Having different managers for different blobs breaks the
# abstraction of a HashBlob. Why should the management of blobs
# care what kind of Blob it has?
class TempBlobManager(BlobManager):
"""This class stores blobs in memory"""
def __init__(self, hash_announcer):
@@ -439,16 +363,16 @@ class TempBlobManager(BlobManager):
self._next_manage_call.cancel()
self._next_manage_call = None
def get_blob(self, blob_hash, upload_allowed, length=None):
def get_blob(self, blob_hash, length=None):
if blob_hash in self.blobs:
return defer.succeed(self.blobs[blob_hash])
return self._make_new_blob(blob_hash, upload_allowed, length)
return self._make_new_blob(blob_hash, length)
def get_blob_creator(self):
return self.blob_creator_type(self)
def _make_new_blob(self, blob_hash, upload_allowed, length=None):
blob = self.blob_type(blob_hash, upload_allowed, length)
def _make_new_blob(self, blob_hash, length=None):
blob = self.blob_type(blob_hash, length)
self.blobs[blob_hash] = blob
return defer.succeed(blob)
@@ -458,14 +382,24 @@ class TempBlobManager(BlobManager):
self.blob_next_announces[blob.blob_hash] = next_announce_time
return defer.succeed(True)
def completed_blobs(self, blobs_to_check):
blobs = [b.blob_hash for b in self.blobs.itervalues() if b.blob_hash in blobs_to_check and b.is_validated()]
def completed_blobs(self, blobhashes_to_check):
blobs = [
b.blob_hash for b in self.blobs.itervalues()
if b.blob_hash in blobhashes_to_check and b.is_validated()
]
return defer.succeed(blobs)
def get_all_verified_blobs(self):
d = self.completed_blobs(self.blobs)
return d
def hashes_to_announce(self):
now = time.time()
blobs = [blob_hash for blob_hash, announce_time in self.blob_next_announces.iteritems() if announce_time < now]
next_announce_time = now + self.hash_reannounce_time
blobs = [
blob_hash for blob_hash, announce_time in self.blob_next_announces.iteritems()
if announce_time < now
]
next_announce_time = self.get_next_announce_time(len(blobs))
for b in blobs:
self.blob_next_announces[b] = next_announce_time
return defer.succeed(blobs)
@@ -474,17 +408,16 @@ class TempBlobManager(BlobManager):
assert blob_creator.blob_hash is not None
assert blob_creator.blob_hash not in self.blobs
assert blob_creator.length is not None
new_blob = self.blob_type(blob_creator.blob_hash, True, blob_creator.length)
new_blob.verified = True
new_blob = self.blob_type(blob_creator.blob_hash, blob_creator.length)
# TODO: change this; its breaks the encapsulation of the
# blob. Maybe better would be to have the blob_creator
# produce a blob.
new_blob.data_buffer = blob_creator.data_buffer
new_blob.length = blob_creator.length
new_blob._verified = True
self.blobs[blob_creator.blob_hash] = new_blob
if self.hash_announcer is not None:
self.hash_announcer.immediate_announce([blob_creator.blob_hash])
next_announce_time = time.time() + self.hash_reannounce_time
d = self.blob_completed(new_blob, next_announce_time)
else:
d = self.blob_completed(new_blob)
self._immediate_announce([blob_creator.blob_hash])
next_announce_time = self.get_next_announce_time()
d = self.blob_completed(new_blob, next_announce_time)
d.addCallback(lambda _: new_blob)
return d
@@ -493,14 +426,9 @@ class TempBlobManager(BlobManager):
if not blob_hash in self.blob_hashes_to_delete:
self.blob_hashes_to_delete[blob_hash] = False
def get_blob_length(self, blob_hash):
if blob_hash in self.blobs:
if self.blobs[blob_hash].length is not None:
return defer.succeed(self.blobs[blob_hash].length)
return defer.fail(NoSuchBlobError(blob_hash))
def immediate_announce_all_blobs(self):
return self.hash_announcer.immediate_announce(self.blobs.iterkeys())
if self.hash_announcer:
return self.hash_announcer.immediate_announce(self.blobs.iterkeys())
def _manage(self):
from twisted.internet import reactor
@@ -514,7 +442,6 @@ class TempBlobManager(BlobManager):
d.addCallback(lambda _: set_next_manage_call())
def _delete_blobs_marked_for_deletion(self):
def remove_from_list(b_h):
del self.blob_hashes_to_delete[b_h]
log.info("Deleted blob %s", blob_hash)
@@ -543,4 +470,4 @@ class TempBlobManager(BlobManager):
d = defer.fail(Failure(NoSuchBlobError(blob_hash)))
log.warning("Blob %s cannot be deleted because it is unknown")
ds.append(d)
return defer.DeferredList(ds)
return defer.DeferredList(ds)
+1 -1
View File
@@ -18,4 +18,4 @@ class DownloadOption(object):
self.long_description = long_description
self.short_description = short_description
self.default_value = default_value
self.default_value_description = default_value_description
self.default_value_description = default_value_description
+43 -8
View File
@@ -22,12 +22,29 @@ class ConnectionClosedBeforeResponseError(Exception):
pass
class KeyFeeAboveMaxAllowed(Exception):
pass
class InvalidExchangeRateResponse(Exception):
def __init__(self, source, reason):
Exception.__init__(self, 'Failed to get exchange rate from {}:{}'.format(source, reason))
self.source = source
self.reason = reason
class UnknownNameError(Exception):
def __init__(self, name):
Exception.__init__(self, 'Name {} is unknown'.format(name))
self.name = name
def __str__(self):
return repr(self.name)
class InvalidName(Exception):
def __init__(self, name, invalid_characters):
self.name = name
self.invalid_characters = invalid_characters
Exception.__init__(
self, 'URI contains invalid characters: {}'.format(','.join(invalid_characters)))
class UnknownStreamTypeError(Exception):
@@ -43,11 +60,11 @@ class InvalidStreamDescriptorError(Exception):
class InvalidStreamInfoError(Exception):
def __init__(self, name):
def __init__(self, name, stream_info):
msg = '{} has claim with invalid stream info: {}'.format(name, stream_info)
Exception.__init__(self, msg)
self.name = name
def __str__(self):
return repr(self.name)
self.stream_info = stream_info
class MisbehavingPeerError(Exception):
@@ -70,9 +87,27 @@ class NoSuchBlobError(Exception):
pass
class NoSuchStreamHashError(Exception):
class NoSuchStreamHash(Exception):
pass
class NoSuchSDHash(Exception):
"""
Raised if sd hash is not known
"""
class InvalidBlobHashError(Exception):
pass
pass
class InvalidHeaderError(Exception):
pass
class InvalidAuthenticationToken(Exception):
pass
class NegotiationError(Exception):
pass
+4 -1
View File
@@ -11,5 +11,8 @@ class DummyHashAnnouncer(object):
def add_supplier(self, *args):
pass
def hash_queue_size(self):
return 0
def immediate_announce(self, *args):
pass
pass
+60 -42
View File
@@ -8,9 +8,10 @@ from twisted.internet import interfaces, defer, threads
from twisted.protocols.basic import FileSender
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.conf import BLOB_SIZE
from lbrynet import conf
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.core.utils import is_valid_blobhash
log = logging.getLogger(__name__)
@@ -48,16 +49,22 @@ class HashBlobWriter(object):
self.write_handle = write_handle
self.length_getter = length_getter
self.finished_cb = finished_cb
self.hashsum = get_lbry_hash_obj()
self._hashsum = get_lbry_hash_obj()
self.len_so_far = 0
@property
def blob_hash(self):
return self._hashsum.hexdigest()
def write(self, data):
self.hashsum.update(data)
self._hashsum.update(data)
self.len_so_far += len(data)
if self.len_so_far > self.length_getter():
self.finished_cb(self, Failure(InvalidDataError("Length so far is greater than the expected length."
" %s to %s" % (str(self.len_so_far),
str(self.length_getter())))))
self.finished_cb(
self,
Failure(InvalidDataError("Length so far is greater than the expected length."
" %s to %s" % (self.len_so_far,
self.length_getter()))))
else:
if self.write_handle is None:
log.debug("Tried to write to a write_handle that was None.")
@@ -75,32 +82,35 @@ class HashBlobWriter(object):
class HashBlob(object):
"""A chunk of data available on the network which is specified by a hashsum"""
def __init__(self, blob_hash, upload_allowed, length=None):
def __init__(self, blob_hash, length=None):
assert is_valid_blobhash(blob_hash)
self.blob_hash = blob_hash
self.length = length
self.writers = {} # {Peer: writer, finished_deferred}
self.finished_deferred = None
self.verified = False
self.upload_allowed = upload_allowed
self._verified = False
self.readers = 0
@property
def verified(self):
# protect verified from being modified by other classes
return self._verified
def set_length(self, length):
if self.length is not None and length == self.length:
return True
if self.length is None and 0 <= length <= BLOB_SIZE:
if self.length is None and 0 <= length <= conf.settings['BLOB_SIZE']:
self.length = length
return True
log.warning("Got an invalid length. Previous length: %s, Invalid length: %s", str(self.length), str(length))
log.warning("Got an invalid length. Previous length: %s, Invalid length: %s",
self.length, length)
return False
def get_length(self):
return self.length
def is_validated(self):
if self.verified:
return True
else:
return False
return bool(self._verified)
def is_downloading(self):
if self.writers:
@@ -126,14 +136,15 @@ class HashBlob(object):
def writer_finished(self, writer, err=None):
def fire_finished_deferred():
self.verified = True
self._verified = True
for p, (w, finished_deferred) in self.writers.items():
if w == writer:
finished_deferred.callback(self)
del self.writers[p]
return True
log.warning("Somehow, the writer that was accepted as being valid was already removed. writer: %s",
str(writer))
log.warning(
"Somehow, the writer that was accepted as being valid was already removed: %s",
writer)
return False
def errback_finished_deferred(err):
@@ -147,8 +158,8 @@ class HashBlob(object):
w.cancel()
if err is None:
if writer.len_so_far == self.length and writer.hashsum.hexdigest() == self.blob_hash:
if self.verified is False:
if writer.len_so_far == self.length and writer.blob_hash == self.blob_hash:
if self._verified is False:
d = self._save_verified_blob(writer)
d.addCallbacks(lambda _: fire_finished_deferred(), errback_finished_deferred)
d.addCallback(lambda _: cancel_other_downloads())
@@ -158,7 +169,7 @@ class HashBlob(object):
else:
err_string = "length vs expected: {0}, {1}, hash vs expected: {2}, {3}"
err_string = err_string.format(self.length, writer.len_so_far, self.blob_hash,
writer.hashsum.hexdigest())
writer.blob_hash)
errback_finished_deferred(Failure(InvalidDataError(err_string)))
d = defer.succeed(True)
else:
@@ -190,7 +201,7 @@ class HashBlob(object):
return self.blob_hash[:16]
def __repr__(self):
return str(self)
return '<{}({})>'.format(self.__class__.__name__, str(self))
class BlobFile(HashBlob):
@@ -202,6 +213,14 @@ class BlobFile(HashBlob):
self.file_path = os.path.join(blob_dir, self.blob_hash)
self.setting_verified_blob_lock = threading.Lock()
self.moved_verified_blob = False
if os.path.isfile(self.file_path):
self.set_length(os.path.getsize(self.file_path))
# This assumes that the hash of the blob has already been
# checked as part of the blob creation process. It might
# be worth having a function that checks the actual hash;
# its probably too expensive to have that check be part of
# this call.
self._verified = True
def open_for_writing(self, peer):
if not peer in self.writers:
@@ -216,19 +235,20 @@ class BlobFile(HashBlob):
return None, None, None
def open_for_reading(self):
if self.verified is True:
if self._verified is True:
file_handle = None
try:
file_handle = open(self.file_path, 'rb')
self.readers += 1
return file_handle
except IOError:
log.exception('Failed to open %s', self.file_path)
self.close_read_handle(file_handle)
return None
def delete(self):
if not self.writers and not self.readers:
self.verified = False
self._verified = False
self.moved_verified_blob = False
def delete_from_file_system():
@@ -238,13 +258,15 @@ class BlobFile(HashBlob):
d = threads.deferToThread(delete_from_file_system)
def log_error(err):
log.warning("An error occurred deleting %s: %s", str(self.file_path), err.getErrorMessage())
log.warning("An error occurred deleting %s: %s",
str(self.file_path), err.getErrorMessage())
return err
d.addErrback(log_error)
return d
else:
return defer.fail(Failure(ValueError("File is currently being read or written and cannot be deleted")))
return defer.fail(Failure(
ValueError("File is currently being read or written and cannot be deleted")))
def close_read_handle(self, file_handle):
if file_handle is not None:
@@ -293,17 +315,18 @@ class TempBlob(HashBlob):
return None, None, None
def open_for_reading(self):
if self.verified is True:
if self._verified is True:
return StringIO(self.data_buffer)
return None
def delete(self):
if not self.writers and not self.readers:
self.verified = False
self._verified = False
self.data_buffer = ''
return defer.succeed(True)
else:
return defer.fail(Failure(ValueError("Blob is currently being read or written and cannot be deleted")))
return defer.fail(Failure(
ValueError("Blob is currently being read or written and cannot be deleted")))
def close_read_handle(self, file_handle):
file_handle.close()
@@ -326,7 +349,7 @@ class TempBlob(HashBlob):
class HashBlobCreator(object):
def __init__(self, blob_manager):
self.blob_manager = blob_manager
self.hashsum = get_lbry_hash_obj()
self._hashsum = get_lbry_hash_obj()
self.len_so_far = 0
self.blob_hash = None
self.length = None
@@ -339,9 +362,8 @@ class HashBlobCreator(object):
if self.length == 0:
self.blob_hash = None
else:
self.blob_hash = self.hashsum.hexdigest()
self.blob_hash = self._hashsum.hexdigest()
d = self._close()
if self.blob_hash is not None:
d.addCallback(lambda _: self.blob_manager.creator_finished(self))
d.addCallback(lambda _: self.blob_hash)
@@ -350,7 +372,7 @@ class HashBlobCreator(object):
return d
def write(self, data):
self.hashsum.update(data)
self._hashsum.update(data)
self.len_so_far += len(data)
self._write(data)
@@ -370,16 +392,11 @@ class BlobFileCreator(HashBlobCreator):
def _close(self):
temp_file_name = self.out_file.name
self.out_file.close()
def change_file_name():
shutil.move(temp_file_name, os.path.join(self.blob_dir, self.blob_hash))
return True
if self.blob_hash is not None:
d = threads.deferToThread(change_file_name)
shutil.move(temp_file_name, os.path.join(self.blob_dir, self.blob_hash))
else:
d = defer.succeed(True)
return d
os.remove(temp_file_name)
return defer.succeed(True)
def _write(self, data):
self.out_file.write(data)
@@ -388,10 +405,11 @@ class BlobFileCreator(HashBlobCreator):
class TempBlobCreator(HashBlobCreator):
def __init__(self, blob_manager):
HashBlobCreator.__init__(self, blob_manager)
# TODO: use StringIO
self.data_buffer = ''
def _close(self):
return defer.succeed(True)
def _write(self, data):
self.data_buffer += data
self.data_buffer += data
File diff suppressed because it is too large. Load diff
+62
View File
@@ -0,0 +1,62 @@
from decimal import Decimal
class Offer(object):
"""A rate offer to download blobs from a host."""
RATE_ACCEPTED = "RATE_ACCEPTED"
RATE_TOO_LOW = "RATE_TOO_LOW"
RATE_UNSET = "RATE_UNSET"
def __init__(self, offer):
self._state = None
self.rate = None
if isinstance(offer, Decimal):
self.rate = round(offer, 5)
elif isinstance(offer, float):
self.rate = round(Decimal(offer), 5)
if self.rate is None or self.rate < Decimal(0.0):
self.unset()
@property
def is_accepted(self):
return self._state is Offer.RATE_ACCEPTED
@property
def is_too_low(self):
return self._state is Offer.RATE_TOO_LOW
@property
def is_unset(self):
return self._state is Offer.RATE_UNSET
@property
def message(self):
if self.is_accepted:
return Offer.RATE_ACCEPTED
elif self.is_too_low:
return Offer.RATE_TOO_LOW
elif self.is_unset:
return Offer.RATE_UNSET
return None
def accept(self):
if self.is_unset or self._state is None:
self._state = Offer.RATE_ACCEPTED
def reject(self):
if self.is_unset or self._state is None:
self._state = Offer.RATE_TOO_LOW
def unset(self):
self._state = Offer.RATE_UNSET
def handle(self, reply_message):
if reply_message == Offer.RATE_TOO_LOW:
self.reject()
elif reply_message == Offer.RATE_ACCEPTED:
self.accept()
elif reply_message == Offer.RATE_UNSET:
self.unset()
else:
raise Exception("Unknown offer reply %s" % str(reply_message))
+19 -12
View File
@@ -7,12 +7,12 @@ from Crypto.Hash import SHA512
from Crypto.PublicKey import RSA
from lbrynet.core.client.ClientRequest import ClientRequest
from lbrynet.core.Error import RequestCanceledError
from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, ILBRYWallet
from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, IWallet
from lbrynet.pointtraderclient import pointtraderclient
from twisted.internet import defer, threads
from zope.interface import implements
from twisted.python.failure import Failure
from lbrynet.core.LBRYcrdWallet import ReservedPoints
from lbrynet.core.Wallet import ReservedPoints
log = logging.getLogger(__name__)
@@ -21,7 +21,7 @@ log = logging.getLogger(__name__)
class PTCWallet(object):
"""This class sends payments to peers and also ensures that expected payments are received.
This class is only intended to be used for testing."""
implements(ILBRYWallet)
implements(IWallet)
def __init__(self, db_dir):
self.db_dir = db_dir
@@ -111,14 +111,16 @@ class PTCWallet(object):
return PointTraderKeyQueryHandlerFactory(self)
def reserve_points(self, peer, amount):
"""
Ensure a certain amount of points are available to be sent as payment, before the service is rendered
"""Ensure a certain amount of points are available to be sent as
payment, before the service is rendered
@param peer: The peer to which the payment will ultimately be sent
@param amount: The amount of points to reserve
@return: A ReservedPoints object which is given to send_points once the service has been rendered
@return: A ReservedPoints object which is given to send_points
once the service has been rendered
"""
if self.wallet_balance >= self.total_reserved_points + amount:
self.total_reserved_points += amount
@@ -157,7 +159,8 @@ class PTCWallet(object):
ds = []
for peer, points in self.queued_payments.items():
if peer in self.peer_pub_keys:
d = pointtraderclient.send_points(self.private_key, self.peer_pub_keys[peer], points)
d = pointtraderclient.send_points(
self.private_key, self.peer_pub_keys[peer], points)
self.wallet_balance -= points
self.total_reserved_points -= points
ds.append(d)
@@ -208,10 +211,13 @@ class PTCWallet(object):
min_expected_balance = sum([a[0] for a in expected_payments if a[1] < expected_cutoff])
received_balance = 0
if self.peer_pub_keys[peer] in self.received_payments:
received_balance = sum([a[0] for a in self.received_payments[self.peer_pub_keys[peer]]])
received_balance = sum([
a[0] for a in self.received_payments[self.peer_pub_keys[peer]]])
if min_expected_balance > received_balance:
log.warning("Account in bad standing: %s (pub_key: %s), expected amount = %s, received_amount = %s",
str(peer), self.peer_pub_keys[peer], str(min_expected_balance), str(received_balance))
log.warning(
"Account in bad standing: %s (pub_key: %s), expected amount = %s, "
"received_amount = %s",
peer, self.peer_pub_keys[peer], min_expected_balance, received_balance)
def _open_db(self):
def open_db():
@@ -285,7 +291,8 @@ class PointTraderKeyQueryHandlerFactory(object):
return 'public_key'
def get_description(self):
return "Point Trader Address - an address for receiving payments on the point trader testing network"
return ("Point Trader Address - an address for receiving payments on the "
"point trader testing network")
class PointTraderKeyQueryHandler(object):
@@ -320,4 +327,4 @@ class PointTraderKeyQueryHandler(object):
log.warning("Expected a public key, but did not receive one")
return defer.fail(Failure(ValueError("Expected but did not receive a public key")))
else:
return defer.succeed({})
return defer.succeed({})
+100 -3
View File
@@ -1,6 +1,13 @@
from lbrynet.core.Strategy import get_default_strategy, OnlyFreeStrategy
from lbrynet import conf
from decimal import Decimal
class BasePaymentRateManager(object):
def __init__(self, rate):
self.min_blob_data_payment_rate = rate
def __init__(self, rate=None, info_rate=None):
self.min_blob_data_payment_rate = rate if rate is not None else conf.settings['data_rate']
self.min_blob_info_payment_rate = (
info_rate if info_rate is not None else conf.settings['min_info_rate'])
class PaymentRateManager(object):
@@ -26,4 +33,94 @@ class PaymentRateManager(object):
return self.min_blob_data_payment_rate
def record_points_paid(self, amount):
self.points_paid += amount
self.points_paid += amount
class NegotiatedPaymentRateManager(object):
def __init__(self, base, availability_tracker, generous=None):
"""
@param base: a BasePaymentRateManager
@param availability_tracker: a BlobAvailabilityTracker
@param rate: the min blob data payment rate
"""
self.base = base
self.points_paid = 0.0
self.blob_tracker = availability_tracker
self.generous = generous if generous is not None else conf.settings['is_generous_host']
self.strategy = get_default_strategy(self.blob_tracker,
base_price=self.base.min_blob_data_payment_rate,
is_generous=generous)
def get_rate_blob_data(self, peer, blobs):
response = self.strategy.make_offer(peer, blobs)
return response.rate
def accept_rate_blob_data(self, peer, blobs, offer):
offer = self.strategy.respond_to_offer(offer, peer, blobs)
self.strategy.update_accepted_offers(peer, offer)
return offer.is_accepted
def reply_to_offer(self, peer, blobs, offer):
reply = self.strategy.respond_to_offer(offer, peer, blobs)
self.strategy.update_accepted_offers(peer, reply)
return reply
def get_rate_for_peer(self, peer):
return self.strategy.accepted_offers.get(peer, False)
def record_points_paid(self, amount):
self.points_paid += amount
def record_offer_reply(self, peer, offer):
self.strategy.update_accepted_offers(peer, offer)
def price_limit_reached(self, peer):
if peer in self.strategy.pending_sent_offers:
offer = self.strategy.pending_sent_offers[peer]
return (offer.is_too_low and
round(Decimal.from_float(offer.rate), 5) >= round(self.strategy.max_rate, 5))
return False
class OnlyFreePaymentsManager(object):
def __init__(self, **kwargs):
"""
A payment rate manager that will only ever accept and offer a rate of 0.0,
Used for testing
"""
self.base = BasePaymentRateManager(0.0, 0.0)
self.points_paid = 0.0
self.generous = True
self.strategy = OnlyFreeStrategy()
def get_rate_blob_data(self, peer, blobs):
response = self.strategy.make_offer(peer, blobs)
return response.rate
def accept_rate_blob_data(self, peer, blobs, offer):
offer = self.strategy.respond_to_offer(offer, peer, blobs)
self.strategy.update_accepted_offers(peer, offer)
return offer.is_accepted
def reply_to_offer(self, peer, blobs, offer):
reply = self.strategy.respond_to_offer(offer, peer, blobs)
self.strategy.update_accepted_offers(peer, reply)
return reply
def get_rate_for_peer(self, peer):
return self.strategy.accepted_offers.get(peer, False)
def record_points_paid(self, amount):
self.points_paid += amount
def record_offer_reply(self, peer, offer):
self.strategy.update_accepted_offers(peer, offer)
def price_limit_reached(self, peer):
if peer in self.strategy.pending_sent_offers:
offer = self.strategy.pending_sent_offers[peer]
if offer.rate > 0.0:
return True
return False
+16 -6
View File
@@ -1,19 +1,23 @@
from collections import defaultdict
import datetime
from collections import defaultdict
from lbrynet.core import utils
# Do not create this object except through PeerManager
class Peer(object):
def __init__(self, host, port):
self.host = host
self.port = port
# If a peer is reported down, we wait till this time till reattempting connection
self.attempt_connection_at = None
# Number of times this Peer has been reported to be down, resets to 0 when it is up
self.down_count = 0
# Number of succesful connections (with full protocol completion) to this peer
self.success_count = 0
self.score = 0
self.stats = defaultdict(float) # {string stat_type, float count}
def is_available(self):
if (self.attempt_connection_at is None or
datetime.datetime.today() > self.attempt_connection_at):
if self.attempt_connection_at is None or utils.today() > self.attempt_connection_at:
return True
return False
@@ -21,10 +25,13 @@ class Peer(object):
self.down_count = 0
self.attempt_connection_at = None
def report_success(self):
self.success_count += 1
def report_down(self):
self.down_count += 1
timeout_time = datetime.timedelta(seconds=60 * self.down_count)
self.attempt_connection_at = datetime.datetime.today() + timeout_time
self.attempt_connection_at = utils.today() + timeout_time
def update_score(self, score_change):
self.score += score_change
@@ -33,4 +40,7 @@ class Peer(object):
self.stats[stat_type] += count
def __str__(self):
return self.host + ":" + str(self.port)
return '{}:{}'.format(self.host, self.port)
def __repr__(self):
return 'Peer({!r}, {!r})'.format(self.host, self.port)
+1 -1
View File
@@ -16,4 +16,4 @@ class DummyPeerFinder(object):
return defer.succeed([])
def get_most_popular_hashes(self, num_to_return):
return []
return []
+1 -1
View File
@@ -11,4 +11,4 @@ class PeerManager(object):
return peer
peer = Peer(host, port)
self.peers.append(peer)
return peer
return peer
+62
View File
@@ -0,0 +1,62 @@
from zope.interface import implementer
from decimal import Decimal
from lbrynet.interfaces import IBlobPriceModel
from lbrynet import conf
def get_default_price_model(blob_tracker, base_price, **kwargs):
return MeanAvailabilityWeightedPrice(blob_tracker, base_price, **kwargs)
class ZeroPrice(object):
def __init__(self):
self.base_price = 0.0
def calculate_price(self, blob):
return 0.0
class MeanAvailabilityWeightedPrice(object):
"""Calculate mean-blob-availability and stream-position weighted price for a blob
Attributes:
base_price (float): base price
alpha (float): constant, > 0.0 and <= 1.0, used to more highly
value blobs at the beginning of a stream.
alpha defaults to 1.0, which has a null effect
blob_tracker (BlobAvailabilityTracker): blob availability tracker
"""
implementer(IBlobPriceModel)
def __init__(self, tracker, base_price=None, alpha=1.0):
self.blob_tracker = tracker
base_price = base_price if base_price is not None else conf.settings['data_rate']
self.base_price = Decimal(base_price)
self.alpha = Decimal(alpha)
def calculate_price(self, blob):
mean_availability = self.blob_tracker.last_mean_availability
availability = self.blob_tracker.availability.get(blob, [])
index = 0 # blob.index
availability_mult = self._get_availability_multiplier(mean_availability, availability)
price = self.base_price * availability_mult / self._frontload(index)
return round(price, 5)
def _get_availability_multiplier(self, mean_availability, availability):
return Decimal(max(1, mean_availability) / Decimal(max(1, len(availability))))
def _frontload(self, index):
"""Get front-load multiplier, used to weight prices of blobs in a
stream towards the front of the stream.
At index 0, returns 1.0
As index increases, return value approaches 2.0
@param index: blob position in stream
@return: front-load multiplier
"""
return Decimal(2.0) - (self.alpha ** index)
+14 -5
View File
@@ -1,8 +1,13 @@
import logging
from zope.interface import implements
from lbrynet.interfaces import IRateLimiter
from twisted.internet import task
log = logging.getLogger(__name__)
class DummyRateLimiter(object):
def __init__(self):
self.dl_bytes_this_second = 0
@@ -64,6 +69,7 @@ class RateLimiter(object):
self.protocols = []
def start(self):
log.info("Starting %s", self)
self.tick_call = task.LoopingCall(self.tick)
self.tick_call.start(self.tick_interval)
@@ -74,6 +80,7 @@ class RateLimiter(object):
self.unthrottle_ul()
def stop(self):
log.info("Stopping %s", self)
if self.tick_call is not None:
self.tick_call.stop()
self.tick_call = None
@@ -87,14 +94,16 @@ class RateLimiter(object):
#throttling
def check_dl(self):
if self.max_dl_bytes is not None and self.dl_bytes_this_interval > self.max_dl_bytes * self.tick_interval:
need_throttle = (self.max_dl_bytes is not None and
self.dl_bytes_this_interval > self.max_dl_bytes * self.tick_interval)
if need_throttle:
from twisted.internet import reactor
reactor.callLater(0, self.throttle_dl)
def check_ul(self):
if self.max_ul_bytes is not None and self.ul_bytes_this_interval > self.max_ul_bytes * self.tick_interval:
need_throttle = (self.max_ul_bytes is not None and
self.ul_bytes_this_interval > self.max_ul_bytes * self.tick_interval)
if need_throttle:
from twisted.internet import reactor
reactor.callLater(0, self.throttle_ul)
@@ -144,4 +153,4 @@ class RateLimiter(object):
def unregister_protocol(self, protocol):
if protocol in self.protocols:
self.protocols.remove(protocol)
self.protocols.remove(protocol)
+120 -54
View File
@@ -1,6 +1,5 @@
import logging
import miniupnpc
from lbrynet.core.PTCWallet import PTCWallet
from lbrynet.core.BlobManager import DiskBlobManager, TempBlobManager
from lbrynet.dht import node
from lbrynet.core.PeerManager import PeerManager
@@ -9,68 +8,96 @@ from lbrynet.core.client.DHTPeerFinder import DHTPeerFinder
from lbrynet.core.HashAnnouncer import DummyHashAnnouncer
from lbrynet.core.server.DHTHashAnnouncer import DHTHashAnnouncer
from lbrynet.core.utils import generate_id
from lbrynet.core.PaymentRateManager import BasePaymentRateManager
from lbrynet.core.PaymentRateManager import BasePaymentRateManager, NegotiatedPaymentRateManager
from lbrynet.core.BlobAvailability import BlobAvailabilityTracker
from twisted.internet import threads, defer
log = logging.getLogger(__name__)
class LBRYSession(object):
"""This class manages all important services common to any application that uses the network:
the hash announcer, which informs other peers that this peer is associated with some hash. Usually,
this means this peer has a blob identified by the hash in question, but it can be used for other
purposes.
the peer finder, which finds peers that are associated with some hash.
the blob manager, which keeps track of which blobs have been downloaded and provides access to them,
the rate limiter, which attempts to ensure download and upload rates stay below a set maximum,
and upnp, which opens holes in compatible firewalls so that remote peers can connect to this peer."""
def __init__(self, blob_data_payment_rate, db_dir=None, lbryid=None, peer_manager=None, dht_node_port=None,
known_dht_nodes=None, peer_finder=None, hash_announcer=None,
blob_dir=None, blob_manager=None, peer_port=None, use_upnp=True,
rate_limiter=None, wallet=None, dht_node_class=node.Node):
"""
@param blob_data_payment_rate: The default payment rate for blob data
class Session(object):
"""This class manages all important services common to any application that uses the network.
the hash announcer, which informs other peers that this peer is
associated with some hash. Usually, this means this peer has a
blob identified by the hash in question, but it can be used for
other purposes.
the peer finder, which finds peers that are associated with some
hash.
the blob manager, which keeps track of which blobs have been
downloaded and provides access to them,
the rate limiter, which attempts to ensure download and upload
rates stay below a set maximum
upnp, which opens holes in compatible firewalls so that remote
peers can connect to this peer.
"""
def __init__(self, blob_data_payment_rate, db_dir=None,
lbryid=None, peer_manager=None, dht_node_port=None,
known_dht_nodes=None, peer_finder=None,
hash_announcer=None, blob_dir=None,
blob_manager=None, peer_port=None, use_upnp=True,
rate_limiter=None, wallet=None,
dht_node_class=node.Node, blob_tracker_class=None,
payment_rate_manager_class=None, is_generous=True):
"""@param blob_data_payment_rate: The default payment rate for blob data
@param db_dir: The directory in which levelDB files should be stored
@param lbryid: The unique ID of this node
@param peer_manager: An object which keeps track of all known peers. If None, a PeerManager will be created
@param peer_manager: An object which keeps track of all known
peers. If None, a PeerManager will be created
@param dht_node_port: The port on which the dht node should listen for incoming connections
@param dht_node_port: The port on which the dht node should
listen for incoming connections
@param known_dht_nodes: A list of nodes which the dht node should use to bootstrap into the dht
@param known_dht_nodes: A list of nodes which the dht node
should use to bootstrap into the dht
@param peer_finder: An object which is used to look up peers that are associated with some hash. If None,
a DHTPeerFinder will be used, which looks for peers in the distributed hash table.
@param peer_finder: An object which is used to look up peers
that are associated with some hash. If None, a
DHTPeerFinder will be used, which looks for peers in the
distributed hash table.
@param hash_announcer: An object which announces to other peers that this peer is associated with some hash.
If None, and peer_port is not None, a DHTHashAnnouncer will be used. If None and
peer_port is None, a DummyHashAnnouncer will be used, which will not actually announce
anything.
@param hash_announcer: An object which announces to other
peers that this peer is associated with some hash. If
None, and peer_port is not None, a DHTHashAnnouncer will
be used. If None and peer_port is None, a
DummyHashAnnouncer will be used, which will not actually
announce anything.
@param blob_dir: The directory in which blobs will be stored. If None and blob_manager is None, blobs will
be stored in memory only.
@param blob_dir: The directory in which blobs will be
stored. If None and blob_manager is None, blobs will be
stored in memory only.
@param blob_manager: An object which keeps track of downloaded blobs and provides access to them. If None,
and blob_dir is not None, a DiskBlobManager will be used, with the given blob_dir.
If None and blob_dir is None, a TempBlobManager will be used, which stores blobs in
memory only.
@param blob_manager: An object which keeps track of downloaded
blobs and provides access to them. If None, and blob_dir
is not None, a DiskBlobManager will be used, with the
given blob_dir. If None and blob_dir is None, a
TempBlobManager will be used, which stores blobs in memory
only.
@param peer_port: The port on which other peers should connect to this peer
@param peer_port: The port on which other peers should connect
to this peer
@param use_upnp: Whether or not to try to open a hole in the firewall so that outside peers can connect to
this peer's peer_port and dht_node_port
@param use_upnp: Whether or not to try to open a hole in the
firewall so that outside peers can connect to this peer's
peer_port and dht_node_port
@param rate_limiter: An object which keeps track of the amount of data transferred to and from this peer,
and can limit that rate if desired
@param rate_limiter: An object which keeps track of the amount
of data transferred to and from this peer, and can limit
that rate if desired
@param wallet: An object which will be used to keep track of expected payments and which will pay peers.
If None, a wallet which uses the Point Trader system will be used, which is meant for testing
only
@param wallet: An object which will be used to keep track of
expected payments and which will pay peers. If None, a
wallet which uses the Point Trader system will be used,
which is meant for testing only
@return:
"""
self.db_dir = db_dir
@@ -88,6 +115,9 @@ class LBRYSession(object):
self.blob_dir = blob_dir
self.blob_manager = blob_manager
self.blob_tracker = None
self.blob_tracker_class = blob_tracker_class or BlobAvailabilityTracker
self.peer_port = peer_port
self.use_upnp = use_upnp
@@ -103,6 +133,9 @@ class LBRYSession(object):
self.dht_node = None
self.base_payment_rate_manager = BasePaymentRateManager(blob_data_payment_rate)
self.payment_rate_manager = None
self.payment_rate_manager_class = payment_rate_manager_class or NegotiatedPaymentRateManager
self.is_generous = is_generous
def setup(self):
"""Create the blob directory and database if necessary, start all desired services"""
@@ -113,6 +146,7 @@ class LBRYSession(object):
self.lbryid = generate_id()
if self.wallet is None:
from lbrynet.core.PTCWallet import PTCWallet
self.wallet = PTCWallet(self.db_dir)
if self.peer_manager is None:
@@ -135,7 +169,10 @@ class LBRYSession(object):
def shut_down(self):
"""Stop all services"""
log.info('Shutting down %s', self)
ds = []
if self.blob_manager is not None:
ds.append(defer.maybeDeferred(self.blob_tracker.stop))
if self.dht_node is not None:
ds.append(defer.maybeDeferred(self.dht_node.stop))
if self.rate_limiter is not None:
@@ -169,18 +206,36 @@ class LBRYSession(object):
self.external_ip = external_ip
if self.peer_port is not None:
if u.getspecificportmapping(self.peer_port, 'TCP') is None:
u.addportmapping(self.peer_port, 'TCP', u.lanaddr, self.peer_port, 'LBRY peer port', '')
u.addportmapping(
self.peer_port, 'TCP', u.lanaddr, self.peer_port,
'LBRY peer port', '')
self.upnp_redirects.append((self.peer_port, 'TCP'))
log.info("Set UPnP redirect for TCP port %d", self.peer_port)
else:
# see comment below
log.warning("UPnP redirect already set for TCP port %d", self.peer_port)
self.upnp_redirects.append((self.peer_port, 'TCP'))
if self.dht_node_port is not None:
if u.getspecificportmapping(self.dht_node_port, 'UDP') is None:
u.addportmapping(self.dht_node_port, 'UDP', u.lanaddr, self.dht_node_port, 'LBRY DHT port', '')
u.addportmapping(
self.dht_node_port, 'UDP', u.lanaddr, self.dht_node_port,
'LBRY DHT port', '')
self.upnp_redirects.append((self.dht_node_port, 'UDP'))
log.info("Set UPnP redirect for UPD port %d", self.dht_node_port)
log.info("Set UPnP redirect for UDP port %d", self.dht_node_port)
else:
# TODO: check that the existing redirect was
# put up by an old lbrynet session before
# grabbing it if such a disconnected redirect
# exists, then upnp won't work unless the
# redirect is appended or is torn down and set
# back up. a bad shutdown of lbrynet could
# leave such a redirect up and cause problems
# on the next start. this could be
# problematic if a previous lbrynet session
# didn't make the redirect, and it was made by
# another application
log.warning("UPnP redirect already set for UDP port %d", self.dht_node_port)
self.upnp_redirects.append((self.dht_node_port, 'UDP'))
return True
return False
@@ -196,10 +251,7 @@ class LBRYSession(object):
from twisted.internet import reactor
log.debug("Starting the dht")
def match_port(h, p):
return h, p
log.info("Starting DHT")
def join_resolved_addresses(result):
addresses = []
@@ -217,7 +269,7 @@ class LBRYSession(object):
ds = []
for host, port in self.known_dht_nodes:
d = reactor.resolve(host)
d.addCallback(match_port, port)
d.addCallback(lambda h: (h, port)) # match host to port
ds.append(d)
self.dht_node = self.dht_node_class(
@@ -244,18 +296,30 @@ class LBRYSession(object):
if self.blob_dir is None:
self.blob_manager = TempBlobManager(self.hash_announcer)
else:
self.blob_manager = DiskBlobManager(self.hash_announcer, self.blob_dir, self.db_dir)
self.blob_manager = DiskBlobManager(self.hash_announcer,
self.blob_dir,
self.db_dir)
if self.blob_tracker is None:
self.blob_tracker = self.blob_tracker_class(self.blob_manager,
self.peer_finder,
self.dht_node)
if self.payment_rate_manager is None:
self.payment_rate_manager = self.payment_rate_manager_class(
self.base_payment_rate_manager,
self.blob_tracker,
self.is_generous)
self.rate_limiter.start()
d1 = self.blob_manager.setup()
d2 = self.wallet.start()
dl = defer.DeferredList([d1, d2], fireOnOneErrback=True, consumeErrors=True)
dl.addErrback(lambda err: err.value.subFailure)
dl.addCallback(lambda _: self.blob_tracker.start())
return dl
def _unset_upnp(self):
log.info("Unsetting upnp for %s", self)
def threaded_unset_upnp():
u = miniupnpc.UPnP()
@@ -264,7 +328,9 @@ class LBRYSession(object):
u.selectigd()
for port, protocol in self.upnp_redirects:
if u.getspecificportmapping(port, protocol) is None:
log.warning("UPnP redirect for %s %d was removed by something else.", protocol, port)
log.warning(
"UPnP redirect for %s %d was removed by something else.",
protocol, port)
else:
u.deleteportmapping(port, protocol)
log.info("Removed UPnP redirect for %s %d.", protocol, port)
+152
View File
@@ -0,0 +1,152 @@
from zope.interface import implementer
from decimal import Decimal
from lbrynet import conf
from lbrynet.interfaces import INegotiationStrategy
from lbrynet.core.Offer import Offer
from lbrynet.core.PriceModel import MeanAvailabilityWeightedPrice, ZeroPrice
def get_default_strategy(blob_tracker, **kwargs):
return BasicAvailabilityWeightedStrategy(blob_tracker, **kwargs)
class Strategy(object):
"""
Base for negotiation strategies
"""
implementer(INegotiationStrategy)
def __init__(self, price_model, max_rate, min_rate, is_generous=None):
self.price_model = price_model
self.is_generous = (
is_generous if is_generous is not None else conf.settings['is_generous_host'])
self.accepted_offers = {}
self.pending_sent_offers = {}
self.offers_sent = {}
self.offers_received = {}
self.max_rate = max_rate or Decimal(self.price_model.base_price * 50)
self.min_rate = Decimal(min_rate)
def _make_rate_offer(self, rates, offer_count):
return NotImplementedError()
def _get_response_rate(self, rates, offer_count):
return NotImplementedError()
def make_offer(self, peer, blobs):
offer_count = self.offers_sent.get(peer, 0)
self._add_offer_sent(peer)
if peer in self.accepted_offers:
# if there was a previous accepted offer, use that
offer = self.accepted_offers[peer]
if peer in self.pending_sent_offers:
del self.pending_sent_offers[peer]
elif offer_count == 0 and self.is_generous:
# Try asking for it for free
offer = Offer(Decimal(0.0))
self.pending_sent_offers.update({peer: offer})
else:
rates = [self.price_model.calculate_price(blob) for blob in blobs]
price = self._make_rate_offer(rates, offer_count)
offer = Offer(price)
self.pending_sent_offers.update({peer: offer})
return offer
def respond_to_offer(self, offer, peer, blobs):
offer_count = self.offers_received.get(peer, 0)
self._add_offer_received(peer)
rates = [self.price_model.calculate_price(blob) for blob in blobs]
price = self._get_response_rate(rates, offer_count)
if peer in self.accepted_offers:
offer = self.accepted_offers[peer]
elif offer.rate == 0.0 and offer_count == 0 and self.is_generous:
# give blobs away for free by default on the first request
offer.accept()
self.accepted_offers.update({peer: offer})
elif offer.rate >= price:
offer.accept()
self.accepted_offers.update({peer: offer})
else:
offer.reject()
if peer in self.accepted_offers:
del self.accepted_offers[peer]
return offer
def update_accepted_offers(self, peer, offer):
if not offer.is_accepted and peer in self.accepted_offers:
del self.accepted_offers[peer]
if offer.is_accepted:
self.accepted_offers.update({peer: offer})
self.pending_sent_offers.update({peer: offer})
def _add_offer_sent(self, peer):
turn = self.offers_sent.get(peer, 0) + 1
self.offers_sent.update({peer: turn})
def _add_offer_received(self, peer):
turn = self.offers_received.get(peer, 0) + 1
self.offers_received.update({peer: turn})
def _bounded_price(self, price):
price_for_return = Decimal(min(self.max_rate, max(price, self.min_rate)))
return price_for_return
class BasicAvailabilityWeightedStrategy(Strategy):
"""Basic strategy to target blob prices based on supply relative to mean supply
Discount price target with each incoming request, and raise it
with each outgoing from the modeled price until the rate is
accepted or a threshold is reached
"""
implementer(INegotiationStrategy)
def __init__(self, blob_tracker, acceleration=1.25,
deceleration=0.9, max_rate=None,
min_rate=0.0,
is_generous=None,
base_price=0.0001, alpha=1.0):
price_model = MeanAvailabilityWeightedPrice(
blob_tracker, base_price=base_price, alpha=alpha)
Strategy.__init__(self, price_model, max_rate, min_rate, is_generous)
self._acceleration = Decimal(acceleration) # rate of how quickly to ramp offer
self._deceleration = Decimal(deceleration)
def _get_mean_rate(self, rates):
mean_rate = Decimal(sum(rates)) / Decimal(max(len(rates), 1))
return mean_rate
def _premium(self, rate, turn):
return rate * (self._acceleration ** Decimal(turn))
def _discount(self, rate, turn):
return rate * (self._deceleration ** Decimal(turn))
def _get_response_rate(self, rates, offer_count):
rate = self._get_mean_rate(rates)
discounted = self._discount(rate, offer_count)
rounded_price = round(discounted, 5)
return self._bounded_price(rounded_price)
def _make_rate_offer(self, rates, offer_count):
rate = self._get_mean_rate(rates)
with_premium = self._premium(rate, offer_count)
rounded_price = round(with_premium, 5)
return self._bounded_price(rounded_price)
class OnlyFreeStrategy(Strategy):
implementer(INegotiationStrategy)
def __init__(self, *args, **kwargs):
price_model = ZeroPrice()
Strategy.__init__(self, price_model, 0.0, 0.0, True)
def _get_mean_rate(self, rates):
return 0.0
def _get_response_rate(self, rates, offer_count):
return 0.0
def _make_rate_offer(self, rates, offer_count):
return 0.0
+3 -3
View File
@@ -64,13 +64,13 @@ class StreamCreator(object):
def _finished(self):
pass
# TODO: move the stream creation process to its own thread and
# remove the reactor from this process.
def write(self, data):
from twisted.internet import reactor
self._write(data)
if self.stopped is False and self.streaming is False:
reactor.callLater(0, self.producer.resumeProducing)
def _write(self, data):
pass
pass
+40 -24
View File
@@ -73,7 +73,9 @@ class StreamDescriptorWriter(object):
return self._write_stream_descriptor(json.dumps(sd_info))
def _write_stream_descriptor(self, raw_data):
"""This method must be overridden by subclasses to write raw data to the stream descriptor"""
"""This method must be overridden by subclasses to write raw data to
the stream descriptor
"""
pass
@@ -126,45 +128,55 @@ class StreamDescriptorIdentifier(object):
and returns the appropriate ones based on the type of the stream descriptor given
"""
def __init__(self):
self._sd_info_validators = {} # {stream_type: IStreamDescriptorValidator}
self._stream_options = {} # {stream_type: IStreamOptions}
self._stream_downloader_factories = defaultdict(list) # {stream_type: [IStreamDownloaderFactory]}
# {stream_type: IStreamDescriptorValidator}
self._sd_info_validators = {}
# {stream_type: IStreamOptions
self._stream_options = {}
# {stream_type: [IStreamDownloaderFactory]}
self._stream_downloader_factories = defaultdict(list)
def add_stream_type(self, stream_type, sd_info_validator, stream_options):
"""
This is how the StreamDescriptorIdentifier learns about new types of stream descriptors.
"""This is how the StreamDescriptorIdentifier learns about new types of stream descriptors.
There can only be one StreamDescriptorValidator for each type of stream.
@param stream_type: A string representing the type of stream descriptor. This must be unique to
this stream descriptor.
@param stream_type: A string representing the type of stream
descriptor. This must be unique to this stream descriptor.
@param sd_info_validator: A class implementing the IStreamDescriptorValidator interface. This class's
constructor will be passed the raw metadata in the stream descriptor file and its 'validate' method
will then be called. If the validation step fails, an exception will be thrown, preventing the stream
descriptor from being further processed.
@param sd_info_validator: A class implementing the
IStreamDescriptorValidator interface. This class's
constructor will be passed the raw metadata in the stream
descriptor file and its 'validate' method will then be
called. If the validation step fails, an exception will be
thrown, preventing the stream descriptor from being
further processed.
@param stream_options: A class implementing the IStreamOptions interface. This class's constructor will be
passed the sd_info_validator object containing the raw metadata from the stream descriptor file.
@param stream_options: A class implementing the IStreamOptions
interface. This class's constructor will be passed the
sd_info_validator object containing the raw metadata from
the stream descriptor file.
@return: None
"""
self._sd_info_validators[stream_type] = sd_info_validator
self._stream_options[stream_type] = stream_options
def add_stream_downloader_factory(self, stream_type, factory):
"""
Register a stream downloader factory with the StreamDescriptorIdentifier.
"""Register a stream downloader factory with the StreamDescriptorIdentifier.
This is how the StreamDescriptorIdentifier determines what factories may be used to process different stream
descriptor files. There must be at least one factory for each type of stream added via
"add_stream_info_validator".
This is how the StreamDescriptorIdentifier determines what
factories may be used to process different stream descriptor
files. There must be at least one factory for each type of
stream added via "add_stream_info_validator".
@param stream_type: A string representing the type of stream descriptor which the factory knows how to process.
@param stream_type: A string representing the type of stream
descriptor which the factory knows how to process.
@param factory: An object implementing the IStreamDownloaderFactory interface.
@return: None
"""
self._stream_downloader_factories[stream_type].append(factory)
@@ -198,7 +210,7 @@ class StreamDescriptorIdentifier(object):
return self._stream_downloader_factories[stream_type]
def _get_validator(self, stream_type):
if not stream_type in self._stream_downloader_factories:
if not stream_type in self._sd_info_validators:
raise UnknownStreamTypeError(stream_type)
return self._sd_info_validators[stream_type]
@@ -236,6 +248,10 @@ def download_sd_blob(session, blob_hash, payment_rate_manager):
@return: An object of type HashBlob
"""
downloader = StandaloneBlobDownloader(blob_hash, session.blob_manager, session.peer_finder,
session.rate_limiter, payment_rate_manager, session.wallet)
return downloader.download()
downloader = StandaloneBlobDownloader(blob_hash,
session.blob_manager,
session.peer_finder,
session.rate_limiter,
payment_rate_manager,
session.wallet)
return downloader.download()
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -4,4 +4,4 @@ Classes and functions which can be used by any application wishing to make use o
This includes classes for connecting to other peers and downloading blobs from them, listening for
connections from peers and responding to their requests, managing locally stored blobs, sending
and receiving payments, and locating peers in the DHT.
"""
"""
+451 -201
View File
@@ -1,18 +1,43 @@
from collections import defaultdict
import logging
from collections import defaultdict
from decimal import Decimal
from twisted.internet import defer
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.core.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
from lbrynet.core.Error import InvalidResponseError, RequestCanceledError, NoResponseError
from lbrynet.core.Error import ConnectionClosedBeforeResponseError
from lbrynet.core.Error import InvalidResponseError, RequestCanceledError, NoResponseError
from lbrynet.core.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
from lbrynet.core.client.ClientRequest import ClientRequest, ClientBlobRequest
from lbrynet.interfaces import IRequestCreator
from lbrynet.core.Offer import Offer
log = logging.getLogger(__name__)
def get_points(num_bytes, rate):
if isinstance(rate, float):
return 1.0 * num_bytes * rate / 2**20
elif isinstance(rate, Decimal):
return 1.0 * num_bytes * float(rate) / 2**20
else:
raise Exception("Unknown rate type")
def cache(fn):
"""Caches the function call for each instance"""
attr = '__{}_value'.format(fn.__name__)
def helper(self):
if not hasattr(self, attr):
value = fn(self)
setattr(self, attr, value)
return getattr(self, attr)
return helper
class BlobRequester(object):
implements(IRequestCreator)
@@ -21,65 +46,23 @@ class BlobRequester(object):
self.peer_finder = peer_finder
self.payment_rate_manager = payment_rate_manager
self.wallet = wallet
self.download_manager = download_manager
self._download_manager = download_manager
self._peers = defaultdict(int) # {Peer: score}
self._available_blobs = defaultdict(list) # {Peer: [blob_hash]}
self._unavailable_blobs = defaultdict(list) # {Peer: [blob_hash]}}
self._protocol_prices = {} # {ClientProtocol: price}
self._protocol_offers = {}
self._price_disagreements = [] # [Peer]
self._protocol_tries = {}
self._maxed_out_peers = []
self._incompatible_peers = []
######## IRequestCreator #########
def send_next_request(self, peer, protocol):
sent_request = False
if self._blobs_to_download() and self._should_send_request_to(peer):
a_r = self._get_availability_request(peer)
d_r = self._get_download_request(peer)
p_r = None
if a_r is not None or d_r is not None:
p_r = self._get_price_request(peer, protocol)
if a_r is not None:
d1 = protocol.add_request(a_r)
d1.addCallback(self._handle_availability, peer, a_r)
d1.addErrback(self._request_failed, "availability request", peer)
sent_request = True
if d_r is not None:
reserved_points = self._reserve_points(peer, protocol, d_r.max_pay_units)
if reserved_points is not None:
# Note: The following three callbacks will be called when the blob has been
# fully downloaded or canceled
d_r.finished_deferred.addCallbacks(self._download_succeeded, self._download_failed,
callbackArgs=(peer, d_r.blob),
errbackArgs=(peer,))
d_r.finished_deferred.addBoth(self._pay_or_cancel_payment, protocol, reserved_points, d_r.blob)
d_r.finished_deferred.addErrback(self._handle_download_error, peer, d_r.blob)
d2 = protocol.add_blob_request(d_r)
# Note: The following two callbacks will be called as soon as the peer sends its
# response, which will be before the blob has finished downloading, but may be
# after the blob has been canceled. For example,
# 1) client sends request to Peer A
# 2) the blob is finished downloading from peer B, and therefore this one is canceled
# 3) client receives response from Peer A
# Therefore, these callbacks shouldn't rely on there being a blob about to be
# downloaded.
d2.addCallback(self._handle_incoming_blob, peer, d_r)
d2.addErrback(self._request_failed, "download request", peer)
sent_request = True
else:
d_r.cancel(InsufficientFundsError())
d_r.finished_deferred.addErrback(lambda _: True)
return defer.fail(InsufficientFundsError())
if sent_request is True:
if p_r is not None:
d3 = protocol.add_request(p_r)
d3.addCallback(self._handle_price_response, peer, p_r, protocol)
d3.addErrback(self._request_failed, "price request", peer)
return defer.succeed(sent_request)
"""Makes an availability request, download request and price request"""
if not self.should_send_next_request(peer):
return defer.succeed(False)
return self._send_next_request(peer, protocol)
def get_new_peers(self):
d = self._get_hash_for_peer_search()
@@ -87,36 +70,35 @@ class BlobRequester(object):
return d
######### internal calls #########
def should_send_next_request(self, peer):
return (
self._blobs_to_download() and
self._should_send_request_to(peer)
)
def _download_succeeded(self, arg, peer, blob):
log.info("Blob %s has been successfully downloaded from %s", str(blob), str(peer))
self._update_local_score(peer, 5.0)
peer.update_stats('blobs_downloaded', 1)
peer.update_score(5.0)
self.blob_manager.blob_completed(blob)
return arg
def _send_next_request(self, peer, protocol):
log.debug('Sending a blob request for %s and %s', peer, protocol)
availability = AvailabilityRequest(self, peer, protocol, self.payment_rate_manager)
download = DownloadRequest(self, peer, protocol, self.payment_rate_manager, self.wallet)
price = PriceRequest(self, peer, protocol, self.payment_rate_manager)
def _download_failed(self, reason, peer):
if not reason.check(DownloadCanceledError, PriceDisagreementError):
self._update_local_score(peer, -10.0)
return reason
sent_request = False
if availability.can_make_request():
availability.make_request_and_handle_response()
sent_request = True
if price.can_make_request():
# TODO: document why a PriceRequest is only made if an
# Availability or Download request was made
price.make_request_and_handle_response()
sent_request = True
if download.can_make_request():
try:
download.make_request_and_handle_response()
sent_request = True
except InsufficientFundsError as err:
return defer.fail(err)
def _pay_or_cancel_payment(self, arg, protocol, reserved_points, blob):
if blob.length != 0 and (not isinstance(arg, Failure) or arg.check(DownloadCanceledError)):
self._pay_peer(protocol, blob.length, reserved_points)
else:
self._cancel_points(reserved_points)
return arg
def _handle_download_error(self, err, peer, blob_to_download):
if not err.check(DownloadCanceledError, PriceDisagreementError, RequestCanceledError):
log.warning("An error occurred while downloading %s from %s. Error: %s",
blob_to_download.blob_hash, str(peer), err.getTraceback())
if err.check(PriceDisagreementError):
# Don't kill the whole connection just because a price couldn't be agreed upon.
# Other information might be desired by other request creators at a better rate.
return True
return err
return defer.succeed(sent_request)
def _get_hash_for_peer_search(self):
r = None
@@ -139,7 +121,10 @@ class BlobRequester(object):
def choose_best_peers(peers):
bad_peers = self._get_bad_peers()
return [p for p in peers if not p in bad_peers]
without_bad_peers = [p for p in peers if not p in bad_peers]
without_maxed_out_peers = [
p for p in without_bad_peers if p not in self._maxed_out_peers]
return without_maxed_out_peers
d.addCallback(choose_best_peers)
@@ -174,146 +159,411 @@ class BlobRequester(object):
return False
def _blobs_to_download(self):
needed_blobs = self.download_manager.needed_blobs()
needed_blobs = self._download_manager.needed_blobs()
return sorted(needed_blobs, key=lambda b: b.is_downloading())
def _blobs_without_sources(self):
return [b for b in self.download_manager.needed_blobs() if not self._hash_available(b.blob_hash)]
def _get_availability_request(self, peer):
all_needed = [b.blob_hash for b in self._blobs_to_download() if not b.blob_hash in self._available_blobs[peer]]
# sort them so that the peer will be asked first for blobs it hasn't said it doesn't have
to_request = sorted(all_needed, key=lambda b: b in self._unavailable_blobs[peer])[:20]
if to_request:
r_dict = {'requested_blobs': to_request}
response_identifier = 'available_blobs'
request = ClientRequest(r_dict, response_identifier)
return request
return None
def _get_download_request(self, peer):
request = None
to_download = [b for b in self._blobs_to_download() if self._hash_available_on(b.blob_hash, peer)]
while to_download and request is None:
blob_to_download = to_download[0]
to_download = to_download[1:]
if not blob_to_download.is_validated():
d, write_func, cancel_func = blob_to_download.open_for_writing(peer)
def counting_write_func(data):
peer.update_stats('blob_bytes_downloaded', len(data))
return write_func(data)
if d is not None:
request_dict = {'requested_blob': blob_to_download.blob_hash}
response_identifier = 'incoming_blob'
request = ClientBlobRequest(request_dict, response_identifier, counting_write_func, d,
cancel_func, blob_to_download)
log.info("Requesting blob %s from %s", str(blob_to_download), str(peer))
return request
return [
b for b in self._download_manager.needed_blobs()
if not self._hash_available(b.blob_hash)
]
def _price_settled(self, protocol):
if protocol in self._protocol_prices:
return True
return False
def _get_price_request(self, peer, protocol):
request = None
if not protocol in self._protocol_prices:
self._protocol_prices[protocol] = self.payment_rate_manager.get_rate_blob_data(peer)
request_dict = {'blob_data_payment_rate': self._protocol_prices[protocol]}
request = ClientRequest(request_dict, 'blob_data_payment_rate')
def _update_local_score(self, peer, amount):
self._peers[peer] += amount
class RequestHelper(object):
def __init__(self, requestor, peer, protocol, payment_rate_manager):
self.requestor = requestor
self.peer = peer
self.protocol = protocol
self.payment_rate_manager = payment_rate_manager
@property
def protocol_prices(self):
return self.requestor._protocol_prices
@property
def protocol_offers(self):
return self.requestor._protocol_offers
@property
def available_blobs(self):
return self.requestor._available_blobs[self.peer]
@property
def unavailable_blobs(self):
return self.requestor._unavailable_blobs[self.peer]
@property
def maxed_out_peers(self):
return self.requestor._maxed_out_peers
def update_local_score(self, score):
self.requestor._update_local_score(self.peer, score)
def _request_failed(self, reason, request_type):
if reason.check(RequestCanceledError):
return
if reason.check(NoResponseError):
self.requestor._incompatible_peers.append(self.peer)
log.warning("A request of type '%s' failed. Reason: %s, Error type: %s",
request_type, reason.getErrorMessage(), reason.type)
self.update_local_score(-10.0)
if isinstance(reason, InvalidResponseError) or isinstance(reason, NoResponseError):
self.peer.update_score(-10.0)
else:
self.peer.update_score(-2.0)
if reason.check(ConnectionClosedBeforeResponseError):
return
return reason
def get_rate(self):
if self.payment_rate_manager.price_limit_reached(self.peer):
if self.peer not in self.maxed_out_peers:
self.maxed_out_peers.append(self.peer)
return None
rate = self.protocol_prices.get(self.protocol)
if rate is None:
if self.peer in self.payment_rate_manager.strategy.pending_sent_offers:
pending = self.payment_rate_manager.strategy.pending_sent_offers[self.peer]
if not pending.is_too_low and not pending.is_accepted:
return pending.rate
rate = self.payment_rate_manager.get_rate_blob_data(self.peer, self.available_blobs)
return rate
def _handle_incoming_blob(response_dict, peer, request):
if not request.response_identifier in response_dict:
return InvalidResponseError("response identifier not in response")
if not type(response_dict[request.response_identifier]) == dict:
return InvalidResponseError("response not a dict. got %s" %
type(response_dict[request.response_identifier]))
response = response_dict[request.response_identifier]
if 'error' in response:
# This means we're not getting our blob for some reason
if response['error'] == "RATE_UNSET":
# Stop the download with an error that won't penalize the peer
request.cancel(PriceDisagreementError())
else:
# The peer has done something bad so we should get out of here
return InvalidResponseError("Got an unknown error from the peer: %s" %
(response['error'],))
else:
if not 'blob_hash' in response:
return InvalidResponseError("Missing the required field 'blob_hash'")
if not response['blob_hash'] == request.request_dict['requested_blob']:
return InvalidResponseError(
"Incoming blob does not match expected. Incoming: %s. Expected: %s" %
(response['blob_hash'], request.request_dict['requested_blob'])
)
if not 'length' in response:
return InvalidResponseError("Missing the required field 'length'")
if not request.blob.set_length(response['length']):
return InvalidResponseError("Could not set the length of the blob")
return True
def _handle_download_error(err, peer, blob_to_download):
if not err.check(DownloadCanceledError, PriceDisagreementError, RequestCanceledError):
log.warning("An error occurred while downloading %s from %s. Error: %s",
blob_to_download.blob_hash, str(peer), err.getTraceback())
if err.check(PriceDisagreementError):
# Don't kill the whole connection just because a price couldn't be agreed upon.
# Other information might be desired by other request creators at a better rate.
return True
return err
class AvailabilityRequest(RequestHelper):
"""Ask a peer what blobs it has available.
Results are saved in `_available_blobs` and `_unavailable_blobs`
on the parent BlobRequester.
"""
def can_make_request(self):
return self.get_top_needed_blobs()
def make_request_and_handle_response(self):
request = self._get_request()
self._handle_request(request)
def _get_request(self):
to_request = self.get_top_needed_blobs()
if not to_request:
raise Exception('Unable to make a request without available blobs')
return self._make_request(to_request)
@cache
def get_top_needed_blobs(self, limit=20):
all_needed = [
b.blob_hash for b in self.requestor._blobs_to_download()
if not self.is_available(b)
]
# sort them so that the peer will be asked first for blobs it
# hasn't said it doesn't have
sorted_needed = sorted(
all_needed,
key=lambda b: b in self.unavailable_blobs
)
return sorted_needed[:limit]
def is_available(self, blob):
return blob.blob_hash in self.available_blobs
def _make_request(self, to_request):
log.debug('Requesting blobs: %s', to_request)
r_dict = {'requested_blobs': to_request}
response_identifier = 'available_blobs'
request = ClientRequest(r_dict, response_identifier)
return request
def _update_local_score(self, peer, amount):
self._peers[peer] += amount
def _handle_request(self, a_r):
log.debug('making an availability request')
d1 = self.protocol.add_request(a_r)
d1.addCallback(self._handle_availability, a_r)
d1.addErrback(self._request_failed, "availability request")
def _reserve_points(self, peer, protocol, max_bytes):
assert protocol in self._protocol_prices
points_to_reserve = 1.0 * max_bytes * self._protocol_prices[protocol] / 2**20
return self.wallet.reserve_points(peer, points_to_reserve)
def _handle_availability(self, response_dict, request):
assert request.response_identifier == 'available_blobs'
if 'available_blobs' not in response_dict:
raise InvalidResponseError("response identifier not in response")
log.debug("Received a response to the availability request")
# save available blobs
blob_hashes = response_dict['available_blobs']
for blob_hash in blob_hashes:
if blob_hash in request.request_dict['requested_blobs']:
self.process_available_blob_hash(blob_hash, request)
# everything left in the request is missing
for blob_hash in request.request_dict['requested_blobs']:
self.unavailable_blobs.append(blob_hash)
return True
def _pay_peer(self, protocol, num_bytes, reserved_points):
def process_available_blob_hash(self, blob_hash, request):
log.debug("The server has indicated it has the following blob available: %s", blob_hash)
self.available_blobs.append(blob_hash)
self.remove_from_unavailable_blobs(blob_hash)
request.request_dict['requested_blobs'].remove(blob_hash)
def remove_from_unavailable_blobs(self, blob_hash):
if blob_hash in self.unavailable_blobs:
self.unavailable_blobs.remove(blob_hash)
class PriceRequest(RequestHelper):
"""Ask a peer if a certain price is acceptable"""
def can_make_request(self):
if len(self.available_blobs) and not self.protocol in self.protocol_prices:
return self.get_rate() is not None
return False
def make_request_and_handle_response(self):
request = self._get_price_request()
self._handle_price_request(request)
def _get_price_request(self):
rate = self.get_rate()
if rate is None:
log.debug("No blobs to request from %s", self.peer)
raise Exception('Cannot make a price request without a payment rate')
log.debug("Offer rate %s to %s for %i blobs", rate, self.peer, len(self.available_blobs))
request_dict = {'blob_data_payment_rate': rate}
assert self.protocol not in self.protocol_offers
self.protocol_offers[self.protocol] = rate
return ClientRequest(request_dict, 'blob_data_payment_rate')
def _handle_price_request(self, price_request):
d = self.protocol.add_request(price_request)
d.addCallback(self._handle_price_response, price_request)
d.addErrback(self._request_failed, "price request")
def _handle_price_response(self, response_dict, request):
assert request.response_identifier == 'blob_data_payment_rate'
if 'blob_data_payment_rate' not in response_dict:
return InvalidResponseError("response identifier not in response")
offer_value = self.protocol_offers.pop(self.protocol)
offer = Offer(offer_value)
offer.handle(response_dict['blob_data_payment_rate'])
self.payment_rate_manager.record_offer_reply(self.peer, offer)
if offer.is_accepted:
log.info("Offered rate %f/mb accepted by %s", offer.rate, self.peer.host)
self.protocol_prices[self.protocol] = offer.rate
return True
elif offer.is_too_low:
log.debug("Offered rate %f/mb rejected by %s", offer.rate, self.peer.host)
return not self.payment_rate_manager.price_limit_reached(self.peer)
else:
log.warning("Price disagreement")
self.requestor._price_disagreements.append(self.peer)
return False
class DownloadRequest(RequestHelper):
"""Choose a blob and download it from a peer and also pay the peer for the data."""
def __init__(self, requester, peer, protocol, payment_rate_manager, wallet):
RequestHelper.__init__(self, requester, peer, protocol, payment_rate_manager)
self.wallet = wallet
def can_make_request(self):
if self.protocol in self.protocol_prices:
return self.get_blob_details()
return False
def make_request_and_handle_response(self):
request = self._get_request()
self._handle_download_request(request)
def _get_request(self):
blob_details = self.get_blob_details()
if not blob_details:
raise Exception('No blobs available to download')
return self._make_request(blob_details)
@cache
def get_blob_details(self):
"""Open a blob for writing and return the details.
If no blob can be opened, returns None.
"""
to_download = self.get_available_blobs()
return self.find_blob(to_download)
def get_available_blobs(self):
available_blobs = [
b for b in self.requestor._blobs_to_download()
if self.requestor._hash_available_on(b.blob_hash, self.peer)
]
log.debug('available blobs: %s', available_blobs)
return available_blobs
def find_blob(self, to_download):
"""Return the first blob in `to_download` that is successfully opened for write."""
for blob in to_download:
if blob.is_validated():
log.debug('Skipping blob %s as its already validated', blob)
continue
d, write_func, cancel_func = blob.open_for_writing(self.peer)
if d is not None:
return BlobDownloadDetails(blob, d, write_func, cancel_func, self.peer)
log.debug('Skipping blob %s as there was an issue opening it for writing', blob)
return None
def _make_request(self, blob_details):
blob = blob_details.blob
request = ClientBlobRequest(
{'requested_blob': blob.blob_hash},
'incoming_blob',
blob_details.counting_write_func,
blob_details.deferred,
blob_details.cancel_func,
blob
)
log.debug("Requesting blob %s from %s", blob.blob_hash, self.peer)
return request
def _handle_download_request(self, client_blob_request):
reserved_points = self.reserve_funds_or_cancel(client_blob_request)
self.add_callbacks_to_download_request(client_blob_request, reserved_points)
self.create_add_blob_request(client_blob_request)
def reserve_funds_or_cancel(self, client_blob_request):
reserved_points = self._reserve_points(client_blob_request.max_pay_units)
if reserved_points is not None:
return reserved_points
client_blob_request.cancel(InsufficientFundsError())
client_blob_request.finished_deferred.addErrback(lambda _: True)
raise InsufficientFundsError()
def add_callbacks_to_download_request(self, client_blob_request, reserved_points):
# Note: The following three callbacks will be called when the blob has been
# fully downloaded or canceled
client_blob_request.finished_deferred.addCallbacks(
self._download_succeeded,
self._download_failed,
callbackArgs=(client_blob_request.blob,),
)
client_blob_request.finished_deferred.addBoth(
self._pay_or_cancel_payment, reserved_points, client_blob_request.blob)
client_blob_request.finished_deferred.addErrback(
_handle_download_error, self.peer, client_blob_request.blob)
def _pay_or_cancel_payment(self, arg, reserved_points, blob):
if self._can_pay_peer(blob, arg):
self._pay_peer(blob.length, reserved_points)
d = self.requestor.blob_manager.add_blob_to_download_history(
str(blob), str(self.peer.host), float(self.protocol_prices[self.protocol]))
else:
self._cancel_points(reserved_points)
return arg
def _can_pay_peer(self, blob, arg):
return (
blob.length != 0 and
(not isinstance(arg, Failure) or arg.check(DownloadCanceledError))
)
def _pay_peer(self, num_bytes, reserved_points):
assert num_bytes != 0
assert protocol in self._protocol_prices
point_amount = 1.0 * num_bytes * self._protocol_prices[protocol] / 2**20
rate = self.get_rate()
point_amount = get_points(num_bytes, rate)
self.wallet.send_points(reserved_points, point_amount)
self.payment_rate_manager.record_points_paid(point_amount)
def _cancel_points(self, reserved_points):
self.wallet.cancel_point_reservation(reserved_points)
def _handle_availability(self, response_dict, peer, request):
if not request.response_identifier in response_dict:
raise InvalidResponseError("response identifier not in response")
log.debug("Received a response to the availability request")
blob_hashes = response_dict[request.response_identifier]
for blob_hash in blob_hashes:
if blob_hash in request.request_dict['requested_blobs']:
log.debug("The server has indicated it has the following blob available: %s", blob_hash)
self._available_blobs[peer].append(blob_hash)
if blob_hash in self._unavailable_blobs[peer]:
self._unavailable_blobs[peer].remove(blob_hash)
request.request_dict['requested_blobs'].remove(blob_hash)
for blob_hash in request.request_dict['requested_blobs']:
self._unavailable_blobs[peer].append(blob_hash)
return True
def create_add_blob_request(self, client_blob_request):
d = self.protocol.add_blob_request(client_blob_request)
# Note: The following two callbacks will be called as soon as the peer sends its
# response, which will be before the blob has finished downloading, but may be
# after the blob has been canceled. For example,
# 1) client sends request to Peer A
# 2) the blob is finished downloading from peer B, and therefore this one is canceled
# 3) client receives response from Peer A
# Therefore, these callbacks shouldn't rely on there being a blob about to be
# downloaded.
d.addCallback(_handle_incoming_blob, self.peer, client_blob_request)
d.addErrback(self._request_failed, "download request")
def _handle_incoming_blob(self, response_dict, peer, request):
if not request.response_identifier in response_dict:
return InvalidResponseError("response identifier not in response")
if not type(response_dict[request.response_identifier]) == dict:
return InvalidResponseError("response not a dict. got %s" %
(type(response_dict[request.response_identifier]),))
response = response_dict[request.response_identifier]
if 'error' in response:
# This means we're not getting our blob for some reason
if response['error'] == "RATE_UNSET":
# Stop the download with an error that won't penalize the peer
request.cancel(PriceDisagreementError())
else:
# The peer has done something bad so we should get out of here
return InvalidResponseError("Got an unknown error from the peer: %s" %
(response['error'],))
else:
if not 'blob_hash' in response:
return InvalidResponseError("Missing the required field 'blob_hash'")
if not response['blob_hash'] == request.request_dict['requested_blob']:
return InvalidResponseError("Incoming blob does not match expected. Incoming: %s. Expected: %s" %
(response['blob_hash'], request.request_dict['requested_blob']))
if not 'length' in response:
return InvalidResponseError("Missing the required field 'length'")
if not request.blob.set_length(response['length']):
return InvalidResponseError("Could not set the length of the blob")
return True
def _reserve_points(self, num_bytes):
# jobevers: there was an assertion here, but I don't think it
# was a valid assertion to make. It is possible for a rate to
# not yet been set for this protocol or for it to have been
# removed so instead I switched it to check if a rate has been set
# and calculate it if it has not
rate = self.get_rate()
points_to_reserve = get_points(num_bytes, rate)
return self.wallet.reserve_points(self.peer, points_to_reserve)
def _handle_price_response(self, response_dict, peer, request, protocol):
if not request.response_identifier in response_dict:
return InvalidResponseError("response identifier not in response")
assert protocol in self._protocol_prices
response = response_dict[request.response_identifier]
if response == "RATE_ACCEPTED":
return True
else:
del self._protocol_prices[protocol]
self._price_disagreements.append(peer)
return True
def _download_succeeded(self, arg, blob):
log.info("Blob %s has been successfully downloaded from %s", blob, self.peer)
self.update_local_score(5.0)
self.peer.update_stats('blobs_downloaded', 1)
self.peer.update_score(5.0)
self.requestor.blob_manager.blob_completed(blob)
return arg
def _request_failed(self, reason, request_type, peer):
if reason.check(RequestCanceledError):
return
if reason.check(NoResponseError):
self._incompatible_peers.append(peer)
log.warning("Blob requester: a request of type '%s' failed. Reason: %s, Error type: %s",
str(request_type), reason.getErrorMessage(), reason.type)
self._update_local_score(peer, -10.0)
if isinstance(reason, InvalidResponseError) or isinstance(reason, NoResponseError):
peer.update_score(-10.0)
else:
peer.update_score(-2.0)
if reason.check(ConnectionClosedBeforeResponseError):
return
return reason
def _download_failed(self, reason):
if not reason.check(DownloadCanceledError, PriceDisagreementError):
self.update_local_score(-10.0)
return reason
class BlobDownloadDetails(object):
"""Contains the information needed to make a ClientBlobRequest from an open blob"""
def __init__(self, blob, deferred, write_func, cancel_func, peer):
self.blob = blob
self.deferred = deferred
self.write_func = write_func
self.cancel_func = cancel_func
self.peer = peer
def counting_write_func(self, data):
self.peer.update_stats('blob_bytes_downloaded', len(data))
return self.write_func(data)
+72 -42
View File
@@ -1,9 +1,12 @@
import json
import logging
from decimal import Decimal
from twisted.internet import error, defer
from twisted.internet.protocol import Protocol, ClientFactory
from twisted.protocols.policies import TimeoutMixin
from twisted.python import failure
from lbrynet.conf import MAX_RESPONSE_INFO_SIZE as MAX_RESPONSE_SIZE
from lbrynet import conf
from lbrynet.core import utils
from lbrynet.core.Error import ConnectionClosedBeforeResponseError, NoResponseError
from lbrynet.core.Error import DownloadCanceledError, MisbehavingPeerError
from lbrynet.core.Error import RequestCanceledError
@@ -14,12 +17,19 @@ from zope.interface import implements
log = logging.getLogger(__name__)
class ClientProtocol(Protocol):
implements(IRequestSender, IRateLimited)
def encode_decimal(obj):
if isinstance(obj, Decimal):
return float(obj)
raise TypeError(repr(obj) + " is not JSON serializable")
class ClientProtocol(Protocol, TimeoutMixin):
implements(IRequestSender, IRateLimited)
######### Protocol #########
PROTOCOL_TIMEOUT = 30
def connectionMade(self):
log.debug("Connection made to %s", self.factory.peer)
self._connection_manager = self.factory.connection_manager
self._rate_limiter = self.factory.rate_limiter
self.peer = self.factory.peer
@@ -30,19 +40,23 @@ class ClientProtocol(Protocol):
self._next_request = {}
self.connection_closed = False
self.connection_closing = False
# This needs to be set for TimeoutMixin
self.callLater = utils.call_later
self.peer.report_up()
self._ask_for_request()
def dataReceived(self, data):
log.debug("Data receieved from %s", self.peer)
self.setTimeout(None)
self._rate_limiter.report_dl_bytes(len(data))
if self._downloading_blob is True:
self._blob_download_request.write(data)
else:
self._response_buff += data
if len(self._response_buff) > MAX_RESPONSE_SIZE:
log.warning("Response is too large. Size %s", len(self._response_buff))
if len(self._response_buff) > conf.settings['MAX_RESPONSE_INFO_SIZE']:
log.warning("Response is too large from %s. Size %s",
self.peer, len(self._response_buff))
self.transport.loseConnection()
response, extra_data = self._get_valid_response(self._response_buff)
if response is not None:
@@ -51,29 +65,34 @@ class ClientProtocol(Protocol):
if self._downloading_blob is True and len(extra_data) != 0:
self._blob_download_request.write(extra_data)
def timeoutConnection(self):
log.info("Connection timed out to %s", self.peer)
self.peer.report_down()
self.transport.abortConnection()
def connectionLost(self, reason):
log.debug("Connection lost to %s: %s", self.peer, reason)
self.setTimeout(None)
self.connection_closed = True
if reason.check(error.ConnectionDone):
err = failure.Failure(ConnectionClosedBeforeResponseError())
else:
err = reason
#if self._response_deferreds:
# log.warning("Lost connection with active response deferreds. %s", str(self._response_deferreds))
for key, d in self._response_deferreds.items():
del self._response_deferreds[key]
d.errback(err)
if self._blob_download_request is not None:
self._blob_download_request.cancel(err)
self._connection_manager.protocol_disconnected(self.peer, self)
self.factory.connection_was_made_deferred.callback(True)
######### IRequestSender #########
def add_request(self, request):
if request.response_identifier in self._response_deferreds:
return defer.fail(failure.Failure(ValueError("There is already a request for that response active")))
raise ValueError("There is already a request for that response active")
self._next_request.update(request.request_dict)
d = defer.Deferred()
log.debug("Adding a request. Request: %s", str(request))
log.debug("Adding a request for %s. Request: %s", self.peer, request.request_dict)
self._response_deferreds[request.response_identifier] = d
return d
@@ -86,7 +105,7 @@ class ClientProtocol(Protocol):
blob_request.finished_deferred.addErrback(self._handle_response_error)
return d
else:
return defer.fail(failure.Failure(ValueError("There is already a blob download request active")))
raise ValueError("There is already a blob download request active")
def cancel_requests(self):
self.connection_closing = True
@@ -105,14 +124,12 @@ class ClientProtocol(Protocol):
######### Internal request handling #########
def _handle_request_error(self, err):
log.error("An unexpected error occurred creating or sending a request to %s. Error message: %s",
str(self.peer), err.getTraceback())
log.error(
"An unexpected error occurred creating or sending a request to %s. Error message: %s",
self.peer, err.getTraceback())
self.transport.loseConnection()
def _ask_for_request(self):
log.debug("In _ask_for_request")
if self.connection_closed is True or self.connection_closing is True:
return
@@ -122,17 +139,20 @@ class ClientProtocol(Protocol):
self._send_request_message(request_msg)
else:
# The connection manager has indicated that this connection should be terminated
log.info("Closing the connection to %s due to having no further requests to send", str(self.peer))
log.info(
"Closing the connection to %s due to having no further requests to send",
self.peer)
self.peer.report_success()
self.transport.loseConnection()
d = self._connection_manager.get_next_request(self.peer, self)
d.addCallback(send_request_or_close)
d.addErrback(self._handle_request_error)
def _send_request_message(self, request_msg):
self.setTimeout(self.PROTOCOL_TIMEOUT)
# TODO: compare this message to the last one. If they're the same,
# TODO: incrementally delay this message.
m = json.dumps(request_msg)
m = json.dumps(request_msg, default=encode_decimal)
self.transport.write(m)
def _get_valid_response(self, response_msg):
@@ -156,21 +176,24 @@ class ClientProtocol(Protocol):
def _handle_response_error(self, err):
# If an error gets to this point, log it and kill the connection.
if not err.check(MisbehavingPeerError, ConnectionClosedBeforeResponseError, DownloadCanceledError,
RequestCanceledError):
log.error("The connection to %s is closing due to an unexpected error: %s", str(self.peer),
err.getErrorMessage())
if not err.check(RequestCanceledError): # The connection manager is closing the connection, so
# there's no need to do it here.
expected_errors = (MisbehavingPeerError, ConnectionClosedBeforeResponseError,
DownloadCanceledError, RequestCanceledError)
if not err.check(expected_errors):
log.error("The connection to %s is closing due to an unexpected error: %s",
self.peer, err.getErrorMessage())
if not err.check(RequestCanceledError):
# The connection manager is closing the connection, so
# there's no need to do it here.
return err
def _handle_response(self, response):
ds = []
log.debug("Handling a response. Current expected responses: %s", str(self._response_deferreds))
log.debug(
"Handling a response from %s. Expected responses: %s. Actual responses: %s",
self.peer, self._response_deferreds.keys(), response.keys())
for key, val in response.items():
if key in self._response_deferreds:
d = self._response_deferreds[key]
del self._response_deferreds[key]
d = self._response_deferreds.pop(key)
d.callback({key: val})
ds.append(d)
for k, d in self._response_deferreds.items():
@@ -184,6 +207,7 @@ class ClientProtocol(Protocol):
d.addErrback(self._handle_response_error)
ds.append(d)
# TODO: are we sure we want to consume errors here
dl = defer.DeferredList(ds, consumeErrors=True)
def get_next_request(results):
@@ -191,20 +215,23 @@ class ClientProtocol(Protocol):
for success, result in results:
if success is False:
failed = True
log.info("The connection is closing due to an error: %s", str(result.getTraceback()))
if not isinstance(result.value, DownloadCanceledError):
log.info(result.value)
log.info("The connection to %s is closing due to an error: %s",
self.peer, result.getTraceback())
self.peer.report_down()
if failed is False:
log.debug("Asking for another request.")
from twisted.internet import reactor
reactor.callLater(0, self._ask_for_request)
#self._ask_for_request()
log.debug("Asking for another request from %s", self.peer)
self._ask_for_request()
else:
log.debug("Not asking for another request.")
log.debug("Not asking for another request from %s", self.peer)
self.transport.loseConnection()
dl.addCallback(get_next_request)
def _downloading_finished(self, arg):
log.debug("The blob has finished downloading")
log.debug("The blob has finished downloading from %s", self.peer)
self._blob_download_request = None
self._downloading_blob = False
return arg
@@ -215,10 +242,8 @@ class ClientProtocol(Protocol):
# TODO: always be this way. it's done this way now because the client has no other way
# TODO: of telling the server it wants the download to stop. It would be great if the
# TODO: protocol had such a mechanism.
log.info("Closing the connection to %s because the download of blob %s was canceled",
str(self.peer), str(self._blob_download_request.blob))
#self.transport.loseConnection()
#return True
log.debug("Closing the connection to %s because the download of blob %s was canceled",
self.peer, self._blob_download_request.blob)
return err
######### IRateLimited #########
@@ -244,13 +269,18 @@ class ClientProtocolFactory(ClientFactory):
self.rate_limiter = rate_limiter
self.connection_manager = connection_manager
self.p = None
# This defer fires and returns True when connection was
# made and completed, or fires and returns False if
# connection failed
self.connection_was_made_deferred = defer.Deferred()
def clientConnectionFailed(self, connector, reason):
log.debug("Connection failed to %s: %s", self.peer, reason)
self.peer.report_down()
self.connection_manager.protocol_disconnected(self.peer, connector)
self.connection_was_made_deferred.callback(False)
def buildProtocol(self, addr):
p = self.protocol()
p.factory = self
self.p = p
return p
return p
+3 -3
View File
@@ -1,4 +1,4 @@
from lbrynet.conf import BLOB_SIZE
from lbrynet import conf
class ClientRequest(object):
@@ -17,11 +17,11 @@ class ClientBlobRequest(ClientPaidRequest):
def __init__(self, request_dict, response_identifier, write_func, finished_deferred,
cancel_func, blob):
if blob.length is None:
max_pay_units = BLOB_SIZE
max_pay_units = conf.settings['BLOB_SIZE']
else:
max_pay_units = blob.length
ClientPaidRequest.__init__(self, request_dict, response_identifier, max_pay_units)
self.write = write_func
self.finished_deferred = finished_deferred
self.cancel = cancel_func
self.blob = blob
self.blob = blob
+150 -114
View File
@@ -1,11 +1,12 @@
import random
import logging
from twisted.internet import defer
from twisted.internet import defer, reactor
from zope.interface import implements
from lbrynet import interfaces
from lbrynet.conf import MAX_CONNECTIONS_PER_STREAM
from lbrynet import conf
from lbrynet.core.client.ClientProtocol import ClientProtocolFactory
from lbrynet.core.Error import InsufficientFundsError
from lbrynet.core import utils
log = logging.getLogger(__name__)
@@ -19,8 +20,11 @@ class PeerConnectionHandler(object):
class ConnectionManager(object):
implements(interfaces.IConnectionManager)
MANAGE_CALL_INTERVAL_SEC = 5
TCP_CONNECT_TIMEOUT = 15
def __init__(self, downloader, rate_limiter, primary_request_creators, secondary_request_creators):
def __init__(self, downloader, rate_limiter,
primary_request_creators, secondary_request_creators):
self.downloader = downloader
self.rate_limiter = rate_limiter
self._primary_request_creators = primary_request_creators
@@ -28,57 +32,85 @@ class ConnectionManager(object):
self._peer_connections = {} # {Peer: PeerConnectionHandler}
self._connections_closing = {} # {Peer: deferred (fired when the connection is closed)}
self._next_manage_call = None
# a deferred that gets fired when a _manage call is set
self._manage_deferred = None
self.stopped = True
log.info("%s initialized", self._get_log_name())
# this identifies what the connection manager is for,
# used for logging purposes only
def _get_log_name(self):
out = 'Connection Manager Unknown'
if hasattr(self.downloader, 'stream_name'):
out = 'Connection Manager '+self.downloader.stream_name
elif hasattr(self.downloader, 'blob_hash'):
out = 'Connection Manager '+self.downloader.blob_hash
return out
def _start(self):
self.stopped = False
if self._next_manage_call is not None and self._next_manage_call.active() is True:
self._next_manage_call.cancel()
def start(self):
from twisted.internet import reactor
self.stopped = False
if self._next_manage_call is not None and self._next_manage_call.active() is True:
self._next_manage_call.cancel()
self._next_manage_call = reactor.callLater(0, self._manage)
log.debug("%s starting", self._get_log_name())
self._start()
self._next_manage_call = utils.call_later(0, self.manage)
return defer.succeed(True)
@defer.inlineCallbacks
def stop(self):
log.debug("%s stopping", self._get_log_name())
self.stopped = True
if self._next_manage_call is not None and self._next_manage_call.active() is True:
# wait for the current manage call to finish
if self._manage_deferred:
yield self._manage_deferred
# in case we stopped between manage calls, cancel the next one
if self._next_manage_call and self._next_manage_call.active():
self._next_manage_call.cancel()
self._next_manage_call = None
closing_deferreds = []
for peer in self._peer_connections.keys():
yield self._close_peers()
def close_connection(p):
log.info("Abruptly closing a connection to %s due to downloading being paused",
str(p))
def num_peer_connections(self):
return len(self._peer_connections)
if self._peer_connections[p].factory.p is not None:
d = self._peer_connections[p].factory.p.cancel_requests()
else:
d = defer.succeed(True)
def _close_peers(self):
def disconnect_peer(p):
d = defer.Deferred()
self._connections_closing[p] = d
self._peer_connections[p].connection.disconnect()
if p in self._peer_connections:
del self._peer_connections[p]
return d
def disconnect_peer():
d = defer.Deferred()
self._connections_closing[p] = d
self._peer_connections[p].connection.disconnect()
if p in self._peer_connections:
del self._peer_connections[p]
return d
def close_connection(p):
log.debug("%s Abruptly closing a connection to %s due to downloading being paused",
self._get_log_name(), p)
if self._peer_connections[p].factory.p is not None:
d = self._peer_connections[p].factory.p.cancel_requests()
else:
d = defer.succeed(True)
d.addBoth(lambda _: disconnect_peer(p))
return d
d.addBoth(lambda _: disconnect_peer())
return d
closing_deferreds.append(close_connection(peer))
closing_deferreds = [close_connection(peer) for peer in self._peer_connections.keys()]
return defer.DeferredList(closing_deferreds)
@defer.inlineCallbacks
def get_next_request(self, peer, protocol):
log.debug("Trying to get the next request for peer %s", str(peer))
log.debug("%s Trying to get the next request for peer %s", self._get_log_name(), peer)
if not peer in self._peer_connections or self.stopped is True:
log.debug("The peer has already been told to shut down.")
return defer.succeed(False)
log.debug("%s The peer %s has already been told to shut down.",
self._get_log_name(), peer)
defer.returnValue(False)
requests = yield self._send_primary_requests(peer, protocol)
have_request = any(r[1] for r in requests if r[0] is True)
if have_request:
yield self._send_secondary_requests(peer, protocol)
defer.returnValue(have_request)
def _send_primary_requests(self, peer, protocol):
def handle_error(err):
err.trap(InsufficientFundsError)
self.downloader.insufficient_funds(err)
@@ -96,96 +128,100 @@ class ConnectionManager(object):
self._peer_connections[peer].request_creators.append(request_creator)
return request_sent
def check_requests(requests):
have_request = True in [r[1] for r in requests if r[0] is True]
return have_request
def get_secondary_requests_if_necessary(have_request):
if have_request is True:
ds = []
for s_r_c in self._secondary_request_creators:
d = s_r_c.send_next_request(peer, protocol)
ds.append(d)
dl = defer.DeferredList(ds)
else:
dl = defer.succeed(None)
dl.addCallback(lambda _: have_request)
return dl
ds = []
for p_r_c in self._primary_request_creators:
d = p_r_c.send_next_request(peer, protocol)
d.addErrback(handle_error)
d.addCallback(check_if_request_sent, p_r_c)
ds.append(d)
return defer.DeferredList(ds, fireOnOneErrback=True)
dl = defer.DeferredList(ds, fireOnOneErrback=True)
dl.addCallback(check_requests)
dl.addCallback(get_secondary_requests_if_necessary)
return dl
def _send_secondary_requests(self, peer, protocol):
ds = [
s_r_c.send_next_request(peer, protocol)
for s_r_c in self._secondary_request_creators
]
return defer.DeferredList(ds)
def protocol_disconnected(self, peer, protocol):
@defer.inlineCallbacks
def manage(self, schedule_next_call=True):
self._manage_deferred = defer.Deferred()
if len(self._peer_connections) < conf.settings['max_connections_per_stream']:
log.debug("%s have %d connections, looking for %d",
self._get_log_name(), len(self._peer_connections),
conf.settings['max_connections_per_stream'])
ordered_request_creators = self._rank_request_creator_connections()
peers = yield self._get_new_peers(ordered_request_creators)
new_conns = conf.settings['max_connections_per_stream'] - len(self._peer_connections)
peers = self._pick_best_peers(peers, new_conns)
for peer in peers:
self._connect_to_peer(peer)
self._manage_deferred.callback(None)
self._manage_deferred = None
if not self.stopped and schedule_next_call:
self._next_manage_call = utils.call_later(self.MANAGE_CALL_INTERVAL_SEC, self.manage)
def _rank_request_creator_connections(self):
"""Returns an ordered list of our request creators, ranked according
to which has the least number of connections open that it
likes
"""
def count_peers(request_creator):
return len([
p for p in self._peer_connections.itervalues()
if request_creator in p.request_creators])
return sorted(self._primary_request_creators, key=count_peers)
@defer.inlineCallbacks
def _get_new_peers(self, request_creators):
log.debug("%s Trying to get a new peer to connect to", self._get_log_name())
if not request_creators:
defer.returnValue(None)
new_peers = yield request_creators[0].get_new_peers()
if not new_peers:
new_peers = yield self._get_new_peers(request_creators[1:])
defer.returnValue(new_peers)
def _pick_best_peers(self, peers, num_peers_to_pick):
# TODO: Eventually rank them based on past performance/reputation. For now
# TODO: just pick the first to which we don't have an open connection
log.debug("%s Got a list of peers to choose from: %s",
self._get_log_name(), peers)
log.debug("%s Current connections: %s",
self._get_log_name(), self._peer_connections.keys())
log.debug("%s List of connection states: %s", self._get_log_name(),
[p_c_h.connection.state for p_c_h in self._peer_connections.values()])
if peers is None:
return []
out = [peer for peer in peers if peer not in self._peer_connections]
random.shuffle(out)
return out[0:num_peers_to_pick]
def _connect_to_peer(self, peer):
if peer is None or self.stopped:
return
log.debug("%s Trying to connect to %s", self._get_log_name(), peer)
factory = ClientProtocolFactory(peer, self.rate_limiter, self)
factory.connection_was_made_deferred.addCallback(
lambda c_was_made: self._peer_disconnected(c_was_made, peer))
self._peer_connections[peer] = PeerConnectionHandler(self._primary_request_creators[:],
factory)
connection = reactor.connectTCP(peer.host, peer.port, factory,
timeout=self.TCP_CONNECT_TIMEOUT)
self._peer_connections[peer].connection = connection
def _peer_disconnected(self, connection_was_made, peer):
log.debug("%s protocol disconnected for %s",
self._get_log_name(), peer)
if peer in self._peer_connections:
del self._peer_connections[peer]
if peer in self._connections_closing:
d = self._connections_closing[peer]
del self._connections_closing[peer]
d.callback(True)
return connection_was_made
def _rank_request_creator_connections(self):
"""
@return: an ordered list of our request creators, ranked according to which has the least number of
connections open that it likes
"""
def count_peers(request_creator):
return len([p for p in self._peer_connections.itervalues() if request_creator in p.request_creators])
return sorted(self._primary_request_creators, key=count_peers)
def _connect_to_peer(self, peer):
from twisted.internet import reactor
if peer is not None and self.stopped is False:
log.debug("Trying to connect to %s", str(peer))
factory = ClientProtocolFactory(peer, self.rate_limiter, self)
self._peer_connections[peer] = PeerConnectionHandler(self._primary_request_creators[:],
factory)
connection = reactor.connectTCP(peer.host, peer.port, factory)
self._peer_connections[peer].connection = connection
def _manage(self):
from twisted.internet import reactor
def get_new_peers(request_creators):
log.debug("Trying to get a new peer to connect to")
if len(request_creators) > 0:
log.debug("Got a creator to check: %s", str(request_creators[0]))
d = request_creators[0].get_new_peers()
d.addCallback(lambda h: h if h is not None else get_new_peers(request_creators[1:]))
return d
else:
return defer.succeed(None)
def pick_best_peer(peers):
# TODO: Eventually rank them based on past performance/reputation. For now
# TODO: just pick the first to which we don't have an open connection
log.debug("Got a list of peers to choose from: %s", str(peers))
if peers is None:
return None
for peer in peers:
if not peer in self._peer_connections:
log.debug("Got a good peer. Returning peer %s", str(peer))
return peer
log.debug("Couldn't find a good peer to connect to")
return None
if len(self._peer_connections) < MAX_CONNECTIONS_PER_STREAM:
ordered_request_creators = self._rank_request_creator_connections()
d = get_new_peers(ordered_request_creators)
d.addCallback(pick_best_peer)
d.addCallback(self._connect_to_peer)
self._next_manage_call = reactor.callLater(1, self._manage)
+29 -13
View File
@@ -1,6 +1,13 @@
import binascii
import logging
from zope.interface import implements
from twisted.internet import defer, reactor
from lbrynet.interfaces import IPeerFinder
from lbrynet.core.utils import short_hash
log = logging.getLogger(__name__)
class DHTPeerFinder(object):
@@ -21,6 +28,7 @@ class DHTPeerFinder(object):
self.next_manage_call = reactor.callLater(60, self.run_manage_loop)
def stop(self):
log.info("Stopping %s", self)
if self.next_manage_call is not None and self.next_manage_call.active():
self.next_manage_call.cancel()
self.next_manage_call = None
@@ -28,21 +36,29 @@ class DHTPeerFinder(object):
def _manage_peers(self):
pass
def find_peers_for_blob(self, blob_hash):
@defer.inlineCallbacks
def find_peers_for_blob(self, blob_hash, timeout=None):
def _trigger_timeout():
if not finished_deferred.called:
log.debug("Peer search for %s timed out", short_hash(blob_hash))
finished_deferred.cancel()
bin_hash = binascii.unhexlify(blob_hash)
finished_deferred = self.dht_node.getPeersForBlob(bin_hash)
def filter_peers(peer_list):
peers = set(peer_list)
good_peers = []
for host, port in peers:
peer = self.peer_manager.get_peer(host, port)
if peer.is_available() is True:
good_peers.append(peer)
return good_peers
if timeout is not None:
reactor.callLater(timeout, _trigger_timeout)
d = self.dht_node.getPeersForBlob(bin_hash)
d.addCallback(filter_peers)
return d
peer_list = yield finished_deferred
peers = set(peer_list)
good_peers = []
for host, port in peers:
peer = self.peer_manager.get_peer(host, port)
if peer.is_available() is True:
good_peers.append(peer)
defer.returnValue(good_peers)
def get_most_popular_hashes(self, num_to_return):
return self.dht_node.get_most_popular_hashes(num_to_return)
return self.dht_node.get_most_popular_hashes(num_to_return)
+15 -12
View File
@@ -11,15 +11,12 @@ log = logging.getLogger(__name__)
class DownloadManager(object):
implements(interfaces.IDownloadManager)
def __init__(self, blob_manager, upload_allowed):
def __init__(self, blob_manager):
self.blob_manager = blob_manager
self.upload_allowed = upload_allowed
self.blob_requester = None
self.blob_info_finder = None
self.progress_manager = None
self.blob_handler = None
self.connection_manager = None
self.blobs = {}
self.blob_infos = {}
@@ -60,28 +57,31 @@ class DownloadManager(object):
d1.addBoth(check_stop, "progress manager")
d2 = self.connection_manager.stop()
d2.addBoth(check_stop, "connection manager")
dl = defer.DeferredList([d1, d2])
dl.addCallback(lambda xs: False not in xs)
dl = defer.DeferredList([d1, d2], consumeErrors=True)
dl.addCallback(lambda results: all([success for success, val in results]))
return dl
def add_blobs_to_download(self, blob_infos):
log.debug("Adding %s to blobs", str(blob_infos))
log.debug("Adding %s blobs to blobs", len(blob_infos))
def add_blob_to_list(blob, blob_num):
self.blobs[blob_num] = blob
log.info("Added blob (hash: %s, number %s) to the list", str(blob.blob_hash), str(blob_num))
log.debug(
"Added blob (hash: %s, number %s) to the list", blob.blob_hash, blob_num)
def error_during_add(err):
log.warning("An error occurred adding the blob to blobs. Error:%s", err.getErrorMessage())
log.warning(
"An error occurred adding the blob to blobs. Error:%s", err.getErrorMessage())
return err
ds = []
for blob_info in blob_infos:
if not blob_info.blob_num in self.blobs:
self.blob_infos[blob_info.blob_num] = blob_info
log.debug("Trying to get the blob associated with blob hash %s", str(blob_info.blob_hash))
d = self.blob_manager.get_blob(blob_info.blob_hash, self.upload_allowed, blob_info.length)
log.debug(
"Trying to get the blob associated with blob hash %s", blob_info.blob_hash)
d = self.blob_manager.get_blob(blob_info.blob_hash, blob_info.length)
d.addCallback(add_blob_to_list, blob_info.blob_num)
d.addErrback(error_during_add)
ds.append(d)
@@ -108,7 +108,10 @@ class DownloadManager(object):
if not self.blobs:
return self.calculate_total_bytes()
else:
to_be_outputted = [b for n, b in self.blobs.iteritems() if n >= self.progress_manager.last_blob_outputted]
to_be_outputted = [
b for n, b in self.blobs.iteritems()
if n >= self.progress_manager.last_blob_outputted
]
return sum([b.length for b in to_be_outputted if b.length is not None])
def calculate_bytes_left_to_download(self):
@@ -93,8 +93,8 @@ class DummyBlobHandler(object):
class StandaloneBlobDownloader(object):
def __init__(self, blob_hash, blob_manager, peer_finder, rate_limiter, payment_rate_manager, wallet):
def __init__(self, blob_hash, blob_manager, peer_finder,
rate_limiter, payment_rate_manager, wallet):
self.blob_hash = blob_hash
self.blob_manager = blob_manager
self.peer_finder = peer_finder
@@ -112,7 +112,7 @@ class StandaloneBlobDownloader(object):
self.stop()
self.finished_deferred = defer.Deferred(canceller=cancel_download)
self.download_manager = DownloadManager(self.blob_manager, True)
self.download_manager = DownloadManager(self.blob_manager)
self.download_manager.blob_requester = BlobRequester(self.blob_manager, self.peer_finder,
self.payment_rate_manager, self.wallet,
self.download_manager)
@@ -142,4 +142,4 @@ class StandaloneBlobDownloader(object):
def insufficient_funds(self, err):
self.stop()
if not self.finished_deferred.called:
self.finished_deferred.errback(err)
self.finished_deferred.errback(err)
+23 -8
View File
@@ -10,7 +10,8 @@ log = logging.getLogger(__name__)
class StreamProgressManager(object):
implements(IProgressManager)
def __init__(self, finished_callback, blob_manager, download_manager, delete_blob_after_finished=False):
def __init__(self, finished_callback, blob_manager,
download_manager, delete_blob_after_finished=False):
self.finished_callback = finished_callback
self.blob_manager = blob_manager
self.delete_blob_after_finished = delete_blob_after_finished
@@ -69,8 +70,8 @@ class StreamProgressManager(object):
log.debug("delete_blob_after_finished is True")
blobs = self.download_manager.blobs
if blob_num in blobs:
log.debug("Telling the blob manager, %s, to delete blob %s", str(self.blob_manager),
blobs[blob_num].blob_hash)
log.debug("Telling the blob manager, %s, to delete blob %s",
self.blob_manager, blobs[blob_num].blob_hash)
self.blob_manager.delete_blobs([blobs[blob_num].blob_hash])
else:
log.debug("Blob number %s was not in blobs", str(blob_num))
@@ -79,26 +80,40 @@ class StreamProgressManager(object):
class FullStreamProgressManager(StreamProgressManager):
def __init__(self, finished_callback, blob_manager, download_manager, delete_blob_after_finished=False):
def __init__(self, finished_callback, blob_manager,
download_manager, delete_blob_after_finished=False):
StreamProgressManager.__init__(self, finished_callback, blob_manager, download_manager,
delete_blob_after_finished)
self.outputting_d = None
######### IProgressManager #########
def _done(self, i, blobs):
"""Return true if `i` is a blob number we don't have"""
return (
i not in blobs or
(
not blobs[i].is_validated() and
i not in self.provided_blob_nums
)
)
def stream_position(self):
blobs = self.download_manager.blobs
if not blobs:
return 0
else:
for i in xrange(max(blobs.iterkeys())):
if not i in blobs or (not blobs[i].is_validated() and not i in self.provided_blob_nums):
if self._done(i, blobs):
return i
return max(blobs.iterkeys()) + 1
def needed_blobs(self):
blobs = self.download_manager.blobs
return [b for n, b in blobs.iteritems() if not b.is_validated() and not n in self.provided_blob_nums]
return [
b for n, b in blobs.iteritems()
if not b.is_validated() and not n in self.provided_blob_nums
]
######### internal #########
@@ -131,7 +146,7 @@ class FullStreamProgressManager(StreamProgressManager):
current_blob_num = self.last_blob_outputted + 1
if current_blob_num in blobs and blobs[current_blob_num].is_validated():
log.info("Outputting blob %s", str(self.last_blob_outputted + 1))
log.debug("Outputting blob %s", str(self.last_blob_outputted + 1))
self.provided_blob_nums.append(self.last_blob_outputted + 1)
d = self.download_manager.handle_blob(self.last_blob_outputted + 1)
d.addCallback(lambda _: finished_outputting_blob())
@@ -147,4 +162,4 @@ class FullStreamProgressManager(StreamProgressManager):
d.addErrback(log_error)
else:
self.outputting_d.callback(True)
self.outputting_d = None
self.outputting_d = None
+1 -1
View File
@@ -15,4 +15,4 @@ def sign_with_pass_phrase(m, pass_phrase):
def verify_signature(m, signature, pub_key):
return seccure.verify(m, signature, pub_key, curve="brainpoolp384r1")
return seccure.verify(m, signature, pub_key, curve="brainpoolp384r1")
+17
View File
@@ -0,0 +1,17 @@
import os
from contextlib import contextmanager
@contextmanager
def get_read_handle(path):
"""
Get os independent read handle for a file
"""
if os.name == "nt":
file_mode = 'rb'
else:
file_mode = 'r'
read_handle = open(path, file_mode)
yield read_handle
read_handle.close()
+382
View File
@@ -0,0 +1,382 @@
import inspect
import json
import logging
import logging.handlers
import os
import sys
import traceback
from txrequests import Session
import twisted.python.log
from lbrynet import __version__ as lbrynet_version, analytics, build_type, conf
from lbrynet.core import utils
####
# This code is copied from logging/__init__.py in the python source code
####
#
# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
if hasattr(sys, 'frozen'): # support for py2exe
_srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
_srcfile = __file__[:-4] + '.py'
else:
_srcfile = __file__
_srcfile = os.path.normcase(_srcfile)
#####
TRACE = 5
def bg_cb(sess, resp):
""" Don't do anything with the response """
pass
class HTTPSHandler(logging.Handler):
def __init__(self, url, fqdn=False, localname=None, facility=None, session=None):
logging.Handler.__init__(self)
self.url = url
self.fqdn = fqdn
self.localname = localname
self.facility = facility
self.session = session if session is not None else Session()
def get_full_message(self, record):
if record.exc_info:
return '\n'.join(traceback.format_exception(*record.exc_info))
else:
return record.getMessage()
def emit(self, record):
try:
payload = self.format(record)
self.session.post(self.url, data=payload, background_callback=bg_cb)
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
DEFAULT_FORMAT = "%(asctime)s %(levelname)-8s %(name)s:%(lineno)d: %(message)s"
DEFAULT_FORMATTER = logging.Formatter(DEFAULT_FORMAT)
LOGGLY_URL = "https://logs-01.loggly.com/inputs/{token}/tag/{tag}"
def remove_handlers(log, handler_name):
for handler in log.handlers:
if handler.name == handler_name:
log.removeHandler(handler)
def _log_decorator(fn):
"""Configure a logging handler.
`fn` is a function that returns a logging handler. The returned
handler has its log-level set and is attached to the specified
logger or the root logger.
"""
def helper(*args, **kwargs):
log = kwargs.pop('log', logging.getLogger())
level = kwargs.pop('level', logging.INFO)
if not isinstance(level, int):
# despite the name, getLevelName returns
# the numeric level when passed a text level
level = logging.getLevelName(level)
handler = fn(*args, **kwargs)
configure_handler(handler, log, level)
return handler
return helper
def configure_handler(handler, log, level):
if handler.name:
remove_handlers(log, handler.name)
handler.setLevel(level)
log.addHandler(handler)
# need to reduce the logger's level down to the
# handler's level or else the handler won't
# get those messages
if log.level > level:
log.setLevel(level)
return handler
def disable_third_party_loggers():
logging.getLogger('requests').setLevel(logging.WARNING)
logging.getLogger('urllib3').setLevel(logging.WARNING)
logging.getLogger('BitcoinRPC').setLevel(logging.INFO)
logging.getLogger('lbryum').setLevel(logging.WARNING)
logging.getLogger('twisted').setLevel(logging.WARNING)
@_log_decorator
def configure_console(**kwargs):
"""Convenience function to configure a log-handler that outputs to stdout"""
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(DEFAULT_FORMATTER)
handler.name = 'console'
return handler
@_log_decorator
def configure_file_handler(file_name, **kwargs):
"""Convenience function to configure a log-handler that writes to `file_name`"""
handler = logging.handlers.RotatingFileHandler(file_name, maxBytes=2097152, backupCount=5)
handler.setFormatter(DEFAULT_FORMATTER)
handler.name = 'file'
return handler
def configure_analytics_handler(analytics_manager):
handler = analytics.Handler(analytics_manager)
handler.name = 'analytics'
return configure_handler(handler, logging.getLogger(), logging.ERROR)
def get_loggly_url(token=None, version=None):
token = token or utils.deobfuscate(conf.settings['LOGGLY_TOKEN'])
version = version or lbrynet_version
return LOGGLY_URL.format(token=token, tag='lbrynet-' + version)
def configure_loggly_handler(*args, **kwargs):
if build_type.BUILD == 'dev':
return
level = kwargs.pop('level', logging.WARNING)
_configure_loggly_handler(*args, level=level, **kwargs)
@_log_decorator
def _configure_loggly_handler(url=None, **kwargs):
url = url or get_loggly_url()
formatter = JsonFormatter(**kwargs)
handler = HTTPSHandler(url)
handler.setFormatter(formatter)
handler.name = 'loggly'
return handler
class JsonFormatter(logging.Formatter):
"""Format log records using json serialization"""
def __init__(self, **kwargs):
self.attributes = kwargs
def format(self, record):
data = {
'loggerName': record.name,
'asciTime': self.formatTime(record),
'fileName': record.filename,
'functionName': record.funcName,
'levelNo': record.levelno,
'lineNo': record.lineno,
'levelName': record.levelname,
'message': record.getMessage(),
}
data.update(self.attributes)
if record.exc_info:
data['exc_info'] = self.formatException(record.exc_info)
return json.dumps(data)
####
# This code is copied from logging/__init__.py in the python source code
####
def findCaller(srcfile=None):
"""Returns the filename, line number and function name of the caller"""
srcfile = srcfile or _srcfile
f = inspect.currentframe()
# On some versions of IronPython, currentframe() returns None if
# IronPython isn't run with -X:Frames.
if f is not None:
f = f.f_back
rv = "(unknown file)", 0, "(unknown function)"
while hasattr(f, "f_code"):
co = f.f_code
filename = os.path.normcase(co.co_filename)
# ignore any function calls that are in this file
if filename == srcfile:
f = f.f_back
continue
rv = (filename, f.f_lineno, co.co_name)
break
return rv
###
def failure(failure, log, msg, *args):
"""Log a failure message from a deferred.
Args:
failure: twisted.python.failure.Failure
log: a python logger instance
msg: the message to log. Can use normal logging string interpolation.
the last argument will be set to the error message from the failure.
args: values to substitute into `msg`
"""
args += (failure.getErrorMessage(),)
exc_info = (failure.type, failure.value, failure.getTracebackObject())
log.error(msg, *args, exc_info=exc_info)
def convert_verbose(verbose):
"""Convert the results of the --verbose flag into a list of logger names
if --verbose is not provided, args.verbose will be None and logging
should be at the info level.
if --verbose is provided, but not followed by any arguments, then
args.verbose = [] and debug logging should be enabled for all of lbrynet
along with info logging on lbryum.
if --verbose is provided and followed by arguments, those arguments
will be in a list
"""
if verbose is None:
return []
if verbose == []:
return ['lbrynet', 'lbryum']
return verbose
def configure_logging(file_name, console, verbose=None):
"""Apply the default logging configuration.
Enables two log-handlers at the INFO level: a file logger and a loggly logger.
Optionally turns on a console logger that defaults to the INFO level, with
specified loggers being set to the DEBUG level.
Args:
file_name: the file to which logs should be saved
console: If true, enable a console logger
verbose: a list of loggers to set to debug level.
See `convert_verbose` for more details.
"""
verbose = convert_verbose(verbose)
configure_twisted()
configure_file_handler(file_name)
configure_loggly_handler()
disable_third_party_loggers()
if console:
# if there are some loggers at the debug level, we need
# to enable the console to allow debug. Otherwise, only
# allow info.
level = 'DEBUG' if verbose else 'INFO'
handler = configure_console(level=level)
if 'lbryum' in verbose:
# TODO: this enables lbryum logging on the other handlers
# too which isn't consistent with how verbose logging
# happens with other loggers. Should change the configuration
# so that its only logging at the INFO level for the console.
logging.getLogger('lbryum').setLevel(logging.INFO)
verbose.remove('lbryum')
if verbose:
handler.addFilter(LoggerNameFilter(verbose))
def configure_twisted():
"""Setup twisted logging to output events to the python stdlib logger"""
# I tried using the new logging api
# https://twistedmatrix.com/documents/current/core/howto/logger.html#compatibility-with-standard-library-logging
# and it simply didn't work
observer = twisted.python.log.PythonLoggingObserver()
observer.start()
class LoggerNameFilter(object):
"""Filter a log record based on its name.
Allows all info level and higher records to pass thru.
Debug records pass if the log record name (or a parent) match
the input list of logger names.
"""
def __init__(self, logger_names):
self.logger_names = logger_names
def filter(self, record):
if record.levelno >= logging.INFO:
return True
name = record.name
while name:
if name in self.logger_names:
return True
name = get_parent(name)
return False
def get_parent(logger_name):
names = logger_name.split('.')
if len(names) == 1:
return ''
names = names[:-1]
return '.'.join(names)
class Logger(logging.Logger):
"""A logger that has an extra `fail` method useful for handling twisted failures."""
def fail(self, callback=None, *args, **kwargs):
"""Returns a function to log a failure from an errback.
The returned function appends the error message and extracts
the traceback from `err`.
Example usage:
d.addErrback(log.fail(), 'This is an error message')
Although odd, making the method call is necessary to extract
out useful filename and line number information; otherwise the
reported values are from inside twisted's deferred handling
code.
Args:
callback: callable to call after making the log. The first argument
will be the `err` from the deferred
args: extra arguments to pass into `callback`
Returns: a function that takes the following arguments:
err: twisted.python.failure.Failure
msg: the message to log, using normal logging string iterpolation.
msg_args: the values to subtitute into `msg`
msg_kwargs: set `level` to change from the default ERROR severity. Other
keywoards are treated as normal log kwargs.
"""
fn, lno, func = findCaller()
def _fail(err, msg, *msg_args, **msg_kwargs):
level = msg_kwargs.pop('level', logging.ERROR)
msg += ": %s"
msg_args += (err.getErrorMessage(),)
exc_info = (err.type, err.value, err.getTracebackObject())
record = self.makeRecord(
self.name, level, fn, lno, msg, msg_args, exc_info, func, msg_kwargs)
self.handle(record)
if callback:
try:
return callback(err, *args, **kwargs)
except Exception:
# log.fail is almost always called within an
# errback. If callback fails and we didn't catch
# the exception we would need to attach a second
# errback to deal with that, which we will almost
# never do and then we end up with an unhandled
# error that will get swallowed by twisted
self.exception('Failed to run callback')
return _fail
def trace(self, msg, *args, **kwargs):
if self.isEnabledFor(TRACE):
self._log(TRACE, msg, args, **kwargs)
logging.setLoggerClass(Logger)
logging.addLevelName(TRACE, 'TRACE')
+20
View File
@@ -0,0 +1,20 @@
class LoopingCallManager(object):
def __init__(self, calls=None):
self.calls = calls or {}
def register_looping_call(self, name, call):
assert name not in self.calls, '{} is already registered'.format(name)
self.calls[name] = call
def start(self, name, *args):
lcall = self.calls[name]
if not lcall.running:
lcall.start(*args)
def stop(self, name):
self.calls[name].stop()
def shutdown(self):
for lcall in self.calls.itervalues():
if lcall.running:
lcall.stop()
@@ -40,10 +40,11 @@ class BlobAvailabilityHandler(object):
def handle_queries(self, queries):
if self.query_identifiers[0] in queries:
log.debug("Received the client's list of requested blobs")
log.info("Received the client's list of requested blobs")
d = self._get_available_blobs(queries[self.query_identifiers[0]])
def set_field(available_blobs):
log.debug("available blobs: %s", str(available_blobs))
return {'available_blobs': available_blobs}
d.addCallback(set_field)
@@ -55,4 +56,4 @@ class BlobAvailabilityHandler(object):
def _get_available_blobs(self, requested_blobs):
d = self.blob_manager.completed_blobs(requested_blobs)
return d
return d
+131 -56
View File
@@ -1,10 +1,13 @@
import logging
from twisted.internet import defer
from twisted.protocols.basic import FileSender
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler, IBlobSender
from lbrynet.core.Offer import Offer
from lbrynet import analytics
from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler, IBlobSender
log = logging.getLogger(__name__)
@@ -12,15 +15,17 @@ log = logging.getLogger(__name__)
class BlobRequestHandlerFactory(object):
implements(IQueryHandlerFactory)
def __init__(self, blob_manager, wallet, payment_rate_manager):
def __init__(self, blob_manager, wallet, payment_rate_manager, track):
self.blob_manager = blob_manager
self.wallet = wallet
self.payment_rate_manager = payment_rate_manager
self.track = track
######### IQueryHandlerFactory #########
def build_query_handler(self):
q_h = BlobRequestHandler(self.blob_manager, self.wallet, self.payment_rate_manager)
q_h = BlobRequestHandler(
self.blob_manager, self.wallet, self.payment_rate_manager, self.track)
return q_h
def get_primary_query_identifier(self):
@@ -32,18 +37,23 @@ class BlobRequestHandlerFactory(object):
class BlobRequestHandler(object):
implements(IQueryHandler, IBlobSender)
PAYMENT_RATE_QUERY = 'blob_data_payment_rate'
BLOB_QUERY = 'requested_blob'
AVAILABILITY_QUERY = 'requested_blobs'
def __init__(self, blob_manager, wallet, payment_rate_manager):
def __init__(self, blob_manager, wallet, payment_rate_manager, track):
self.blob_manager = blob_manager
self.payment_rate_manager = payment_rate_manager
self.wallet = wallet
self.query_identifiers = ['blob_data_payment_rate', 'requested_blob']
self.query_identifiers = [self.PAYMENT_RATE_QUERY, self.BLOB_QUERY, self.AVAILABILITY_QUERY]
self.track = track
self.peer = None
self.blob_data_payment_rate = None
self.read_handle = None
self.currently_uploading = None
self.file_sender = None
self.blob_bytes_uploaded = 0
self._blobs_requested = []
######### IQueryHandler #########
@@ -53,44 +63,22 @@ class BlobRequestHandler(object):
request_handler.register_blob_sender(self)
def handle_queries(self, queries):
response = {}
if self.query_identifiers[0] in queries:
if not self.handle_blob_data_payment_rate(queries[self.query_identifiers[0]]):
response['blob_data_payment_rate'] = "RATE_TOO_LOW"
else:
response['blob_data_payment_rate'] = 'RATE_ACCEPTED'
response = defer.succeed({})
log.debug("Handle query: %s", str(queries))
if self.query_identifiers[1] in queries:
log.debug("Received the client's request to send a blob")
response_fields = {}
response['incoming_blob'] = response_fields
if self.blob_data_payment_rate is None:
response_fields['error'] = "RATE_UNSET"
return defer.succeed(response)
else:
d = self.blob_manager.get_blob(queries[self.query_identifiers[1]], True)
def open_blob_for_reading(blob):
if blob.is_validated():
read_handle = blob.open_for_reading()
if read_handle is not None:
self.currently_uploading = blob
self.read_handle = read_handle
log.debug("Sending %s to client", str(blob))
response_fields['blob_hash'] = blob.blob_hash
response_fields['length'] = blob.length
return response
log.debug("We can not send %s", str(blob))
response_fields['error'] = "BLOB_UNAVAILABLE"
return response
d.addCallback(open_blob_for_reading)
return d
else:
return defer.succeed(response)
if self.AVAILABILITY_QUERY in queries:
self._blobs_requested = queries[self.AVAILABILITY_QUERY]
response.addCallback(lambda r: self._reply_to_availability(r, self._blobs_requested))
if self.PAYMENT_RATE_QUERY in queries:
offered_rate = queries[self.PAYMENT_RATE_QUERY]
offer = Offer(offered_rate)
if offer.rate is None:
log.warning("Empty rate offer")
response.addCallback(lambda r: self._handle_payment_rate_query(offer, r))
if self.BLOB_QUERY in queries:
incoming = queries[self.BLOB_QUERY]
response.addCallback(lambda r: self._reply_to_send_request(r, incoming))
return response
######### IBlobSender #########
@@ -108,12 +96,92 @@ class BlobRequestHandler(object):
######### internal #########
def handle_blob_data_payment_rate(self, requested_payment_rate):
if not self.payment_rate_manager.accept_rate_blob_data(self.peer, requested_payment_rate):
return False
def _reply_to_availability(self, request, blobs):
d = self._get_available_blobs(blobs)
def set_available(available_blobs):
log.debug("available blobs: %s", str(available_blobs))
request.update({'available_blobs': available_blobs})
return request
d.addCallback(set_available)
return d
def _handle_payment_rate_query(self, offer, request):
blobs = self._blobs_requested
log.debug("Offered rate %f LBC/mb for %i blobs", offer.rate, len(blobs))
reply = self.payment_rate_manager.reply_to_offer(self.peer, blobs, offer)
if reply.is_accepted:
self.blob_data_payment_rate = offer.rate
request[self.PAYMENT_RATE_QUERY] = "RATE_ACCEPTED"
log.debug("Accepted rate: %f", offer.rate)
elif reply.is_too_low:
request[self.PAYMENT_RATE_QUERY] = "RATE_TOO_LOW"
log.debug("Reject rate: %f", offer.rate)
elif reply.is_unset:
log.warning("Rate unset")
request['incoming_blob'] = {'error': 'RATE_UNSET'}
log.debug("Returning rate query result: %s", str(request))
return request
def _handle_blob_query(self, response, query):
log.debug("Received the client's request to send a blob")
response['incoming_blob'] = {}
if self.blob_data_payment_rate is None:
response['incoming_blob'] = {'error': "RATE_UNSET"}
return response
else:
self.blob_data_payment_rate = requested_payment_rate
return True
return self._send_blob(response, query)
def _send_blob(self, response, query):
d = self.blob_manager.get_blob(query)
d.addCallback(self.open_blob_for_reading, response)
return d
def open_blob_for_reading(self, blob, response):
response_fields = {}
d = defer.succeed(None)
if blob.is_validated():
read_handle = blob.open_for_reading()
if read_handle is not None:
self.currently_uploading = blob
self.read_handle = read_handle
log.info("Sending %s to %s", str(blob), self.peer)
response_fields['blob_hash'] = blob.blob_hash
response_fields['length'] = blob.length
response['incoming_blob'] = response_fields
d.addCallback(lambda _: self.record_transaction(blob))
d.addCallback(lambda _: response)
return d
log.debug("We can not send %s", str(blob))
response['incoming_blob'] = {'error': 'BLOB_UNAVAILABLE'}
d.addCallback(lambda _: response)
return d
def record_transaction(self, blob):
d = self.blob_manager.add_blob_to_upload_history(
str(blob), self.peer.host, self.blob_data_payment_rate)
return d
def _reply_to_send_request(self, response, incoming):
response_fields = {}
response['incoming_blob'] = response_fields
if self.blob_data_payment_rate is None:
log.debug("Rate not set yet")
response['incoming_blob'] = {'error': 'RATE_UNSET'}
return defer.succeed(response)
else:
log.debug("Requested blob: %s", str(incoming))
d = self.blob_manager.get_blob(incoming)
d.addCallback(lambda blob: self.open_blob_for_reading(blob, response))
return d
def _get_available_blobs(self, requested_blobs):
d = self.blob_manager.completed_blobs(requested_blobs)
return d
def send_file(self, consumer):
@@ -126,23 +194,30 @@ class BlobRequestHandler(object):
return inner_d
def count_bytes(data):
self.blob_bytes_uploaded += len(data)
self.peer.update_stats('blob_bytes_uploaded', len(data))
uploaded = len(data)
self.blob_bytes_uploaded += uploaded
self.peer.update_stats('blob_bytes_uploaded', uploaded)
self.track.add_observation(analytics.BLOB_BYTES_UPLOADED, uploaded)
return data
def start_transfer():
self.file_sender = FileSender()
log.info("Starting the file upload")
assert self.read_handle is not None, "self.read_handle was None when trying to start the transfer"
log.debug("Starting the file upload")
assert self.read_handle is not None, \
"self.read_handle was None when trying to start the transfer"
d = self.file_sender.beginFileTransfer(self.read_handle, consumer, count_bytes)
return d
def set_expected_payment():
log.info("Setting expected payment")
if self.blob_bytes_uploaded != 0 and self.blob_data_payment_rate is not None:
log.debug("Setting expected payment")
if (
self.blob_bytes_uploaded != 0 and self.blob_data_payment_rate is not None
and self.blob_data_payment_rate > 0
):
# TODO: explain why 2**20
self.wallet.add_expected_payment(self.peer,
self.currently_uploading.length * 1.0 *
self.blob_data_payment_rate / 2**20)
self.blob_data_payment_rate / 2 ** 20)
self.blob_bytes_uploaded = 0
self.peer.update_stats('blobs_uploaded', 1)
return None
@@ -154,6 +229,6 @@ class BlobRequestHandler(object):
self.currently_uploading = None
self.file_sender = None
if reason is not None and isinstance(reason, Failure):
log.info("Upload has failed. Reason: %s", reason.getErrorMessage())
log.warning("Upload has failed. Reason: %s", reason.getErrorMessage())
return _send_file()
return _send_file()
+63 -15
View File
@@ -1,9 +1,18 @@
import binascii
from twisted.internet import defer, task, reactor
import collections
import logging
import time
from twisted.internet import defer
from lbrynet.core import utils
log = logging.getLogger(__name__)
class DHTHashAnnouncer(object):
ANNOUNCE_CHECK_INTERVAL = 60
CONCURRENT_ANNOUNCERS = 5
"""This class announces to the DHT that this peer has certain blobs"""
def __init__(self, dht_node, peer_port):
self.dht_node = dht_node
@@ -14,14 +23,12 @@ class DHTHashAnnouncer(object):
self._concurrent_announcers = 0
def run_manage_loop(self):
from twisted.internet import reactor
if self.peer_port is not None:
self._announce_available_hashes()
self.next_manage_call = reactor.callLater(60, self.run_manage_loop)
self.next_manage_call = utils.call_later(self.ANNOUNCE_CHECK_INTERVAL, self.run_manage_loop)
def stop(self):
log.info("Stopping %s", self)
if self.next_manage_call is not None:
self.next_manage_call.cancel()
self.next_manage_call = None
@@ -31,11 +38,15 @@ class DHTHashAnnouncer(object):
def immediate_announce(self, blob_hashes):
if self.peer_port is not None:
return self._announce_hashes(blob_hashes)
return self._announce_hashes(blob_hashes, immediate=True)
else:
return defer.succeed(False)
def hash_queue_size(self):
return len(self.hash_queue)
def _announce_available_hashes(self):
log.debug('Announcing available hashes')
ds = []
for supplier in self.suppliers:
d = supplier.hashes_to_announce()
@@ -44,38 +55,75 @@ class DHTHashAnnouncer(object):
dl = defer.DeferredList(ds)
return dl
def _announce_hashes(self, hashes):
def _announce_hashes(self, hashes, immediate=False):
if not hashes:
return
log.debug('Announcing %s hashes', len(hashes))
# TODO: add a timeit decorator
start = time.time()
ds = []
for h in hashes:
announce_deferred = defer.Deferred()
ds.append(announce_deferred)
self.hash_queue.append((h, announce_deferred))
if immediate:
self.hash_queue.appendleft((h, announce_deferred))
else:
self.hash_queue.append((h, announce_deferred))
log.debug('There are now %s hashes remaining to be announced', self.hash_queue_size())
def announce():
if len(self.hash_queue):
h, announce_deferred = self.hash_queue.popleft()
log.debug('Announcing blob %s to dht', h)
d = self.dht_node.announceHaveBlob(binascii.unhexlify(h), self.peer_port)
d.chainDeferred(announce_deferred)
d.addBoth(lambda _: reactor.callLater(0, announce))
d.addBoth(lambda _: utils.call_later(0, announce))
else:
self._concurrent_announcers -= 1
for i in range(self._concurrent_announcers, 5):
# TODO: maybe make the 5 configurable
for i in range(self._concurrent_announcers, self.CONCURRENT_ANNOUNCERS):
self._concurrent_announcers += 1
announce()
return defer.DeferredList(ds)
d = defer.DeferredList(ds)
d.addCallback(lambda _: log.debug('Took %s seconds to announce %s hashes',
time.time() - start, len(hashes)))
return d
class DHTHashSupplier(object):
# 1 hour is the min time hash will be reannounced
MIN_HASH_REANNOUNCE_TIME = 60*60
# conservative assumption of the time it takes to announce
# a single hash
SINGLE_HASH_ANNOUNCE_DURATION = 1
"""Classes derived from this class give hashes to a hash announcer"""
def __init__(self, announcer):
if announcer is not None:
announcer.add_supplier(self)
self.hash_announcer = announcer
self.hash_reannounce_time = 60 * 60 # 1 hour
def hashes_to_announce(self):
pass
pass
def get_next_announce_time(self, num_hashes_to_announce=1):
"""
Hash reannounce time is set to current time + MIN_HASH_REANNOUNCE_TIME,
unless we are announcing a lot of hashes at once which could cause the
the announce queue to pile up. To prevent pile up, reannounce
only after a conservative estimate of when it will finish
to announce all the hashes.
Args:
num_hashes_to_announce: number of hashes that will be added to the queue
Returns:
timestamp for next announce time
"""
queue_size = self.hash_announcer.hash_queue_size()+num_hashes_to_announce
reannounce = max(self.MIN_HASH_REANNOUNCE_TIME,
queue_size*self.SINGLE_HASH_ANNOUNCE_DURATION)
return time.time() + reannounce
+5 -6
View File
@@ -33,10 +33,9 @@ class ServerProtocol(Protocol):
peer_info = self.transport.getPeer()
self.peer = self.factory.peer_manager.get_peer(peer_info.host, peer_info.port)
self.request_handler = ServerRequestHandler(self)
for query_handler_factory, enabled in self.factory.query_handler_factories.iteritems():
if enabled is True:
query_handler = query_handler_factory.build_query_handler()
query_handler.register_with_request_handler(self.request_handler, self.peer)
for query_handler_factory in self.factory.query_handler_factories.values():
query_handler = query_handler_factory.build_query_handler()
query_handler.register_with_request_handler(self.request_handler, self.peer)
log.debug("Setting the request handler")
self.factory.rate_limiter.register_protocol(self)
@@ -64,7 +63,7 @@ class ServerProtocol(Protocol):
self.transport.loseConnection()
def write(self, data):
log.debug("Writing %s bytes of data to the transport", str(len(data)))
log.trace("Writing %s bytes of data to the transport", len(data))
self.transport.write(data)
self.factory.rate_limiter.report_ul_bytes(len(data))
@@ -91,4 +90,4 @@ class ServerProtocolFactory(ServerFactory):
def __init__(self, rate_limiter, query_handler_factories, peer_manager):
self.rate_limiter = rate_limiter
self.query_handler_factories = query_handler_factories
self.peer_manager = peer_manager
self.peer_manager = peer_manager
+39 -27
View File
@@ -9,9 +9,10 @@ log = logging.getLogger(__name__)
class ServerRequestHandler(object):
"""This class handles requests from clients. It can upload blobs and return request for information about
more blobs that are associated with streams"""
"""This class handles requests from clients. It can upload blobs and
return request for information about more blobs that are
associated with streams.
"""
implements(interfaces.IPushProducer, interfaces.IConsumer, IRequestHandler)
def __init__(self, consumer):
@@ -51,18 +52,19 @@ class ServerRequestHandler(object):
from twisted.internet import reactor
if self.production_paused is False:
chunk = self.response_buff[:self.CHUNK_SIZE]
self.response_buff = self.response_buff[self.CHUNK_SIZE:]
if chunk != '':
log.debug("writing %s bytes to the client", str(len(chunk)))
self.consumer.write(chunk)
reactor.callLater(0, self._produce_more)
if self.production_paused:
return
chunk = self.response_buff[:self.CHUNK_SIZE]
self.response_buff = self.response_buff[self.CHUNK_SIZE:]
if chunk == '':
return
log.trace("writing %s bytes to the client", len(chunk))
self.consumer.write(chunk)
reactor.callLater(0, self._produce_more)
#IConsumer stuff
def registerProducer(self, producer, streaming):
#assert self.file_sender == producer
self.producer = producer
assert streaming is False
producer.resumeProducing()
@@ -79,7 +81,7 @@ class ServerRequestHandler(object):
def get_more_data():
if self.producer is not None:
log.debug("Requesting more data from the producer")
log.trace("Requesting more data from the producer")
self.producer.resumeProducing()
reactor.callLater(0, get_more_data)
@@ -90,19 +92,27 @@ class ServerRequestHandler(object):
log.debug("Received data")
log.debug("%s", str(data))
if self.request_received is False:
self.request_buff = self.request_buff + data
msg = self.try_to_parse_request(self.request_buff)
if msg is not None:
self.request_buff = ''
d = self.handle_request(msg)
if self.blob_sender is not None:
d.addCallback(lambda _: self.blob_sender.send_blob_if_requested(self))
d.addCallbacks(lambda _: self.finished_response(), self.request_failure_handler)
else:
log.info("Request buff not a valid json message")
log.info("Request buff: %s", str(self.request_buff))
return self._parse_data_and_maybe_send_blob(data)
else:
log.warning("The client sent data when we were uploading a file. This should not happen")
log.warning(
"The client sent data when we were uploading a file. This should not happen")
def _parse_data_and_maybe_send_blob(self, data):
self.request_buff = self.request_buff + data
msg = self.try_to_parse_request(self.request_buff)
if msg:
self.request_buff = ''
self._process_msg(msg)
else:
log.debug("Request buff not a valid json message")
log.debug("Request buff: %s", self.request_buff)
def _process_msg(self, msg):
d = self.handle_request(msg)
if self.blob_sender:
d.addCallback(lambda _: self.blob_sender.send_blob_if_requested(self))
d.addCallbacks(lambda _: self.finished_response(), self.request_failure_handler)
######### IRequestHandler #########
@@ -125,7 +135,7 @@ class ServerRequestHandler(object):
def send_response(self, msg):
m = json.dumps(msg)
log.info("Sending a response of length %s", str(len(m)))
log.debug("Sending a response of length %s", str(len(m)))
log.debug("Response: %s", str(m))
self.response_buff = self.response_buff + m
self._produce_more()
@@ -147,7 +157,9 @@ class ServerRequestHandler(object):
return response
def log_errors(err):
log.warning("An error occurred handling a client request. Error message: %s", err.getErrorMessage())
log.warning(
"An error occurred handling a client request. Error message: %s",
err.getErrorMessage())
return err
def send_response(response):
@@ -171,4 +183,4 @@ class ServerRequestHandler(object):
msg = json.loads(request_buff)
return msg
except ValueError:
return None
return None
+1 -1
View File
@@ -20,4 +20,4 @@ def rerun_if_locked(f):
d.addErrback(rerun, *args, **kwargs)
return d
return wrapper
return wrapper
+29
View File
@@ -0,0 +1,29 @@
import platform
import json
from urllib2 import urlopen
from lbrynet import __version__ as lbrynet_version
from lbrynet import build_type
from lbryum.version import LBRYUM_VERSION as lbryum_version
def get_platform(get_ip=True):
p = {
"processor": platform.processor(),
"python_version": platform.python_version(),
"platform": platform.platform(),
"os_release": platform.release(),
"os_system": platform.system(),
"lbrynet_version": lbrynet_version,
"lbryum_version": lbryum_version,
"build": build_type.BUILD, # CI server sets this during build step
}
if get_ip:
try:
p['ip'] = json.load(urlopen('http://jsonip.com'))['ip']
except:
p['ip'] = "Could not determine IP"
return p
+103 -9
View File
@@ -1,8 +1,53 @@
from lbrynet.core.cryptoutils import get_lbry_hash_obj
import base64
import datetime
import logging
import random
import socket
import string
import json
import pkg_resources
from lbryschema.claim import ClaimDict
from lbrynet.core.cryptoutils import get_lbry_hash_obj
# digest_size is in bytes, and blob hashes are hex encoded
blobhash_length = get_lbry_hash_obj().digest_size * 2
log = logging.getLogger(__name__)
blobhash_length = get_lbry_hash_obj().digest_size * 2 # digest_size is in bytes, and blob hashes are hex encoded
# defining these time functions here allows for easier overriding in testing
def now():
return datetime.datetime.now()
def utcnow():
return datetime.datetime.utcnow()
def isonow():
"""Return utc now in isoformat with timezone"""
return utcnow().isoformat() + 'Z'
def today():
return datetime.datetime.today()
def timedelta(**kwargs):
return datetime.timedelta(**kwargs)
def datetime_obj(*args, **kwargs):
return datetime.datetime(*args, **kwargs)
def call_later(delay, func, *args, **kwargs):
# Import here to ensure that it gets called after installing a reactor
# see: http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html
from twisted.internet import reactor
return reactor.callLater(delay, func, *args, **kwargs)
def generate_id(num=None):
@@ -14,15 +59,64 @@ def generate_id(num=None):
return h.digest()
def is_valid_hashcharacter(char):
return char in "0123456789abcdef"
def is_valid_blobhash(blobhash):
"""
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: Whether the blobhash is the correct length and contains only valid characters (0-9, a-f)
@return: True/False
"""
if len(blobhash) != blobhash_length:
return len(blobhash) == blobhash_length and all(is_valid_hashcharacter(l) for l in blobhash)
def version_is_greater_than(a, b):
"""Returns True if version a is more recent than version b"""
return pkg_resources.parse_version(a) > pkg_resources.parse_version(b)
def deobfuscate(obfustacated):
return base64.b64decode(obfustacated.decode('rot13'))
def obfuscate(plain):
return base64.b64encode(plain).encode('rot13')
def check_connection(server="lbry.io", port=80):
"""Attempts to open a socket to server:port and returns True if successful."""
try:
log.debug('Checking connection to %s:%s', server, port)
host = socket.gethostbyname(server)
s = socket.create_connection((host, port), 2)
log.debug('Connection successful')
return True
except Exception as ex:
log.info(
"Failed to connect to %s:%s. Maybe the internet connection is not working",
server, port, exc_info=True)
return False
for l in blobhash:
if l not in "0123456789abcdef":
return False
return True
def random_string(length=10, chars=string.ascii_lowercase):
return ''.join([random.choice(chars) for _ in range(length)])
def short_hash(hash_str):
return hash_str[:6]
def get_sd_hash(stream_info):
if not stream_info:
return None
if isinstance(stream_info, ClaimDict):
return stream_info.source_hash
return stream_info['stream']['source']['source']
def json_dumps_pretty(obj, **kwargs):
return json.dumps(obj, sort_keys=True, indent=2, separators=(',', ': '), **kwargs)
+15 -7
View File
@@ -1,7 +1,7 @@
import binascii
import logging
from Crypto.Cipher import AES
from lbrynet.conf import BLOB_SIZE
from lbrynet import conf
from lbrynet.core.BlobInfo import BlobInfo
@@ -35,8 +35,8 @@ class StreamBlobDecryptor(object):
def write_bytes():
if self.len_read < self.length:
num_bytes_to_decrypt = (len(self.buff) // self.cipher.block_size) * self.cipher.block_size
data_to_decrypt, self.buff = self.buff[:num_bytes_to_decrypt], self.buff[num_bytes_to_decrypt:]
num_bytes_to_decrypt = greatest_multiple(len(self.buff), self.cipher.block_size)
data_to_decrypt, self.buff = split(self.buff, num_bytes_to_decrypt)
write_func(self.cipher.decrypt(data_to_decrypt))
def finish_decrypt():
@@ -48,7 +48,6 @@ class StreamBlobDecryptor(object):
self.buff += data
self.len_read += len(data)
write_bytes()
#write_func(remove_padding(self.cipher.decrypt(self.buff)))
d = self.blob.read(decrypt_bytes)
d.addCallback(lambda _: finish_decrypt())
@@ -67,7 +66,7 @@ class CryptStreamBlobMaker(object):
self.length = 0
def write(self, data):
max_bytes_to_write = BLOB_SIZE - self.length - 1
max_bytes_to_write = conf.settings['BLOB_SIZE'] - self.length - 1
done = False
if max_bytes_to_write <= len(data):
num_bytes_to_write = max_bytes_to_write
@@ -91,7 +90,7 @@ class CryptStreamBlobMaker(object):
def _write_buffer(self):
num_bytes_to_encrypt = (len(self.buff) // AES.block_size) * AES.block_size
data_to_encrypt, self.buff = self.buff[:num_bytes_to_encrypt], self.buff[num_bytes_to_encrypt:]
data_to_encrypt, self.buff = split(self.buff, num_bytes_to_encrypt)
encrypted_data = self.cipher.encrypt(data_to_encrypt)
self.blob.write(encrypted_data)
@@ -106,4 +105,13 @@ class CryptStreamBlobMaker(object):
self.blob.write(encrypted_data)
def _return_info(self, blob_hash):
return CryptBlobInfo(blob_hash, self.blob_num, self.length, binascii.hexlify(self.iv))
return CryptBlobInfo(blob_hash, self.blob_num, self.length, binascii.hexlify(self.iv))
def greatest_multiple(a, b):
"""return the largest value `c`, that is a multiple of `b` and is <= `a`"""
return (a // b) * b
def split(buff, cutoff):
return buff[:cutoff], buff[cutoff:]
+10 -11
View File
@@ -18,22 +18,23 @@ log = logging.getLogger(__name__)
class CryptStreamCreator(StreamCreator):
"""Create a new stream with blobs encrypted by a symmetric cipher.
Each blob is encrypted with the same key, but each blob has its own initialization vector
which is associated with the blob when the blob is associated with the stream."""
Each blob is encrypted with the same key, but each blob has its
own initialization vector which is associated with the blob when
the blob is associated with the stream.
"""
def __init__(self, blob_manager, name=None, key=None, iv_generator=None):
"""
@param blob_manager: Object that stores and provides access to blobs.
"""@param blob_manager: Object that stores and provides access to blobs.
@type blob_manager: BlobManager
@param name: the name of the stream, which will be presented to the user
@type name: string
@param key: the raw AES key which will be used to encrypt the blobs. If None, a random key will
be generated.
@param key: the raw AES key which will be used to encrypt the
blobs. If None, a random key will be generated.
@type key: string
@param iv_generator: a generator which yields initialization vectors for the blobs. Will be called
once for each blob.
@param iv_generator: a generator which yields initialization
vectors for the blobs. Will be called once for each blob.
@type iv_generator: a generator function which yields strings
@return: None
@@ -72,10 +73,8 @@ class CryptStreamCreator(StreamCreator):
d.addCallback(self._blob_finished)
self.finished_deferreds.append(d)
log.debug("called close on final blob, returning from make_final_blob")
return d
def _write(self, data):
def close_blob(blob):
d = blob.close()
d.addCallback(self._blob_finished)
@@ -94,4 +93,4 @@ class CryptStreamCreator(StreamCreator):
self.current_blob = None
def _get_blob_maker(self, iv, blob_creator):
return CryptStreamBlobMaker(self.key, iv, self.blob_count, blob_creator)
return CryptStreamBlobMaker(self.key, iv, self.blob_count, blob_creator)
+1 -1
View File
@@ -5,4 +5,4 @@ Crypt Streams are encrypted blobs and metadata tying those blobs together. At le
metadata is generally stored in a Stream Descriptor File, for example containing a public key
used to bind blobs to the stream and a symmetric key used to encrypt the blobs. The list of blobs
may or may not be present.
"""
"""
@@ -14,6 +14,7 @@ class CryptBlobHandler(object):
######## IBlobHandler #########
def handle_blob(self, blob, blob_info):
blob_decryptor = StreamBlobDecryptor(blob, self.key, binascii.unhexlify(blob_info.iv), blob_info.length)
blob_decryptor = StreamBlobDecryptor(
blob, self.key, binascii.unhexlify(blob_info.iv), blob_info.length)
d = blob_decryptor.decrypt(self.write_func)
return d
return d
@@ -38,21 +38,22 @@ class CryptStreamDownloader(object):
implements(IStreamDownloader)
def __init__(self, peer_finder, rate_limiter, blob_manager,
payment_rate_manager, wallet, upload_allowed):
"""
Initialize a CryptStreamDownloader
payment_rate_manager, wallet):
"""Initialize a CryptStreamDownloader
@param peer_finder: An object which implements the IPeerFinder interface. Used to look up peers by a hashsum.
@param peer_finder: An object which implements the IPeerFinder
interface. Used to look up peers by a hashsum.
@param rate_limiter: An object which implements the IRateLimiter interface
@param blob_manager: A BlobManager object
@param payment_rate_manager: A PaymentRateManager object
@param payment_rate_manager: A NegotiatedPaymentRateManager object
@param wallet: An object which implements the ILBRYWallet interface
@param wallet: An object which implements the IWallet interface
@return:
"""
self.peer_finder = peer_finder
@@ -60,20 +61,16 @@ class CryptStreamDownloader(object):
self.blob_manager = blob_manager
self.payment_rate_manager = payment_rate_manager
self.wallet = wallet
self.upload_allowed = upload_allowed
self.key = None
self.stream_name = None
self.completed = False
self.stopped = True
self.stopping = False
self.starting = False
self.download_manager = None
self.finished_deferred = None
self.points_paid = 0.0
self.blob_requester = None
def __str__(self):
return str(self.stream_name)
@@ -85,7 +82,6 @@ class CryptStreamDownloader(object):
return self.stop()
def start(self):
if self.starting is True:
raise CurrentlyStartingError()
if self.stopping is True:
@@ -96,30 +92,24 @@ class CryptStreamDownloader(object):
self.starting = True
self.completed = False
self.finished_deferred = defer.Deferred()
fd = self.finished_deferred
d = self._start()
d.addCallback(lambda _: fd)
d.addCallback(lambda _: self.finished_deferred)
return d
@defer.inlineCallbacks
def stop(self, err=None):
def check_if_stop_succeeded(success):
self.stopping = False
if success is True:
self.stopped = True
self._remove_download_manager()
return success
if self.stopped is True:
raise AlreadyStoppedError()
if self.stopping is True:
raise CurrentlyStoppingError()
assert self.download_manager is not None
self.stopping = True
d = self.download_manager.stop_downloading()
d.addCallback(check_if_stop_succeeded)
d.addCallback(lambda _: self._fire_completed_deferred(err))
return d
success = yield self.download_manager.stop_downloading()
self.stopping = False
if success is True:
self.stopped = True
self._remove_download_manager()
yield self._fire_completed_deferred(err)
def _start_failed(self):
@@ -154,28 +144,32 @@ class CryptStreamDownloader(object):
return d
def _get_download_manager(self):
download_manager = DownloadManager(self.blob_manager, self.upload_allowed)
assert self.blob_requester is None
download_manager = DownloadManager(self.blob_manager)
# TODO: can we get rid of these circular references. I'm not
# smart enough to handle thinking about the interactions
# between them and have hope that there is a simpler way
# to accomplish what we want
download_manager.blob_info_finder = self._get_metadata_handler(download_manager)
download_manager.blob_requester = self._get_blob_requester(download_manager)
download_manager.progress_manager = self._get_progress_manager(download_manager)
download_manager.blob_handler = self._get_blob_handler(download_manager)
download_manager.wallet_info_exchanger = self.wallet.get_info_exchanger()
# blob_requester needs to be set before the connection manager is setup
self.blob_requester = self._get_blob_requester(download_manager)
download_manager.connection_manager = self._get_connection_manager(download_manager)
#return DownloadManager(self.blob_manager, self.blob_requester, self.metadata_handler,
# self.progress_manager, self.blob_handler, self.connection_manager)
return download_manager
def _remove_download_manager(self):
self.download_manager.blob_info_finder = None
self.download_manager.blob_requester = None
self.download_manager.progress_manager = None
self.download_manager.blob_handler = None
self.download_manager.wallet_info_exchanger = None
self.blob_requester = None
self.download_manager.connection_manager = None
self.download_manager = None
def _get_primary_request_creators(self, download_manager):
return [download_manager.blob_requester]
return [self.blob_requester]
def _get_secondary_request_creators(self, download_manager):
return [download_manager.wallet_info_exchanger]
@@ -184,11 +178,13 @@ class CryptStreamDownloader(object):
pass
def _get_blob_requester(self, download_manager):
return BlobRequester(self.blob_manager, self.peer_finder, self.payment_rate_manager, self.wallet,
return BlobRequester(self.blob_manager, self.peer_finder,
self.payment_rate_manager, self.wallet,
download_manager)
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager)
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager, download_manager)
def _get_write_func(self):
pass
@@ -207,7 +203,8 @@ class CryptStreamDownloader(object):
if err is not None:
d.errback(err)
else:
d.callback(self._get_finished_deferred_callback_value())
value = self._get_finished_deferred_callback_value()
d.callback(value)
else:
log.debug("Not firing the completed deferred because d is None")
@@ -220,4 +217,4 @@ class CryptStreamDownloader(object):
return self.stop()
def insufficient_funds(self, err):
return self.stop(err=err)
return self.stop(err=err)
+11 -9
View File
@@ -3,13 +3,15 @@ import logging
def migrate_db(db_dir, start, end):
current = start
old_dirs = []
while current < end:
if current == 0:
from lbrynet.db_migrator.migrate0to1 import do_migration
old_dirs.append(do_migration(db_dir))
current += 1
return old_dirs
if current == 1:
from lbrynet.db_migrator.migrate1to2 import do_migration
do_migration(db_dir)
else:
raise Exception(
"DB migration of version {} to {} is not available".format(current, current+1))
current += 1
return None
def run_migration_script():
@@ -19,7 +21,7 @@ def run_migration_script():
sys.stdout = open("migrator.out.log", 'w')
sys.stderr = open("migrator.err.log", 'w')
migrate_db(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]))
if __name__ == "__main__":
run_migration_script()
run_migration_script()
-307
View File
@@ -1,307 +0,0 @@
import sqlite3
import unqlite
import leveldb
import shutil
import os
import logging
import json
log = logging.getLogger(__name__)
known_dbs = ['lbryfile_desc.db', 'lbryfiles.db', 'valuable_blobs.db', 'blobs.db',
'lbryfile_blob.db', 'lbryfile_info.db', 'settings.db', 'blind_settings.db',
'blind_peers.db', 'blind_info.db', 'lbryfile_info.db', 'lbryfile_manager.db',
'live_stream.db', 'stream_info.db', 'stream_blob.db', 'stream_desc.db']
def do_move(from_dir, to_dir):
for known_db in known_dbs:
known_db_path = os.path.join(from_dir, known_db)
if os.path.exists(known_db_path):
log.debug("Moving %s to %s",
os.path.abspath(known_db_path),
os.path.abspath(os.path.join(to_dir, known_db)))
shutil.move(known_db_path, os.path.join(to_dir, known_db))
else:
log.debug("Did not find %s", os.path.abspath(known_db_path))
def do_migration(db_dir):
old_dir = os.path.join(db_dir, "_0_to_1_old")
new_dir = os.path.join(db_dir, "_0_to_1_new")
try:
log.info("Moving dbs from the real directory to %s", os.path.abspath(old_dir))
os.makedirs(old_dir)
do_move(db_dir, old_dir)
except:
log.error("An error occurred moving the old db files.")
raise
try:
log.info("Creating the new directory in %s", os.path.abspath(new_dir))
os.makedirs(new_dir)
except:
log.error("An error occurred creating the new directory.")
raise
try:
log.info("Doing the migration")
migrate_blob_db(old_dir, new_dir)
migrate_lbryfile_db(old_dir, new_dir)
migrate_livestream_db(old_dir, new_dir)
migrate_ptc_db(old_dir, new_dir)
migrate_lbryfile_manager_db(old_dir, new_dir)
migrate_settings_db(old_dir, new_dir)
migrate_repeater_db(old_dir, new_dir)
log.info("Migration succeeded")
except:
log.error("An error occurred during the migration. Restoring.")
do_move(old_dir, db_dir)
raise
try:
log.info("Moving dbs in the new directory to the real directory")
do_move(new_dir, db_dir)
db_revision = open(os.path.join(db_dir, 'db_revision'), mode='w+')
db_revision.write("1")
db_revision.close()
os.rmdir(new_dir)
except:
log.error("An error occurred moving the new db files.")
raise
return old_dir
def migrate_blob_db(old_db_dir, new_db_dir):
old_blob_db_path = os.path.join(old_db_dir, "blobs.db")
if not os.path.exists(old_blob_db_path):
return True
old_db = leveldb.LevelDB(old_blob_db_path)
new_db_conn = sqlite3.connect(os.path.join(new_db_dir, "blobs.db"))
c = new_db_conn.cursor()
c.execute("create table if not exists blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" last_verified_time real, " +
" next_announce_time real"
")")
new_db_conn.commit()
c = new_db_conn.cursor()
for blob_hash, blob_info in old_db.RangeIter():
blob_length, verified_time, announce_time = json.loads(blob_info)
c.execute("insert into blobs values (?, ?, ?, ?)",
(blob_hash, blob_length, verified_time, announce_time))
new_db_conn.commit()
new_db_conn.close()
def migrate_lbryfile_db(old_db_dir, new_db_dir):
old_lbryfile_db_path = os.path.join(old_db_dir, "lbryfiles.db")
if not os.path.exists(old_lbryfile_db_path):
return True
stream_info_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_info.db"))
stream_blob_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_blob.db"))
stream_desc_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_desc.db"))
db_conn = sqlite3.connect(os.path.join(new_db_dir, "lbryfile_info.db"))
c = db_conn.cursor()
c.execute("create table if not exists lbry_files (" +
" stream_hash text primary key, " +
" key text, " +
" stream_name text, " +
" suggested_file_name text" +
")")
c.execute("create table if not exists lbry_file_blobs (" +
" blob_hash text, " +
" stream_hash text, " +
" position integer, " +
" iv text, " +
" length integer, " +
" foreign key(stream_hash) references lbry_files(stream_hash)" +
")")
c.execute("create table if not exists lbry_file_descriptors (" +
" sd_blob_hash TEXT PRIMARY KEY, " +
" stream_hash TEXT, " +
" foreign key(stream_hash) references lbry_files(stream_hash)" +
")")
db_conn.commit()
c = db_conn.cursor()
for stream_hash, stream_info in stream_info_db.RangeIter():
key, name, suggested_file_name = json.loads(stream_info)
c.execute("insert into lbry_files values (?, ?, ?, ?)",
(stream_hash, key, name, suggested_file_name))
db_conn.commit()
c = db_conn.cursor()
for blob_hash_stream_hash, blob_info in stream_blob_db.RangeIter():
b_h, s_h = json.loads(blob_hash_stream_hash)
position, iv, length = json.loads(blob_info)
c.execute("insert into lbry_file_blobs values (?, ?, ?, ?, ?)",
(b_h, s_h, position, iv, length))
db_conn.commit()
c = db_conn.cursor()
for sd_blob_hash, stream_hash in stream_desc_db.RangeIter():
c.execute("insert into lbry_file_descriptors values (?, ?)",
(sd_blob_hash, stream_hash))
db_conn.commit()
db_conn.close()
def migrate_livestream_db(old_db_dir, new_db_dir):
old_db_path = os.path.join(old_db_dir, "stream_info.db")
if not os.path.exists(old_db_path):
return True
stream_info_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_info.db"))
stream_blob_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_blob.db"))
stream_desc_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_desc.db"))
db_conn = sqlite3.connect(os.path.join(new_db_dir, "live_stream.db"))
c = db_conn.cursor()
c.execute("create table if not exists live_streams (" +
" stream_hash text primary key, " +
" public_key text, " +
" key text, " +
" stream_name text, " +
" next_announce_time real" +
")")
c.execute("create table if not exists live_stream_blobs (" +
" blob_hash text, " +
" stream_hash text, " +
" position integer, " +
" revision integer, " +
" iv text, " +
" length integer, " +
" signature text, " +
" foreign key(stream_hash) references live_streams(stream_hash)" +
")")
c.execute("create table if not exists live_stream_descriptors (" +
" sd_blob_hash TEXT PRIMARY KEY, " +
" stream_hash TEXT, " +
" foreign key(stream_hash) references live_streams(stream_hash)" +
")")
db_conn.commit()
c = db_conn.cursor()
for stream_hash, stream_info in stream_info_db.RangeIter():
public_key, key, name, next_announce_time = json.loads(stream_info)
c.execute("insert into live_streams values (?, ?, ?, ?, ?)",
(stream_hash, public_key, key, name, next_announce_time))
db_conn.commit()
c = db_conn.cursor()
for blob_hash_stream_hash, blob_info in stream_blob_db.RangeIter():
b_h, s_h = json.loads(blob_hash_stream_hash)
position, revision, iv, length, signature = json.loads(blob_info)
c.execute("insert into live_stream_blobs values (?, ?, ?, ?, ?, ?, ?)",
(b_h, s_h, position, revision, iv, length, signature))
db_conn.commit()
c = db_conn.cursor()
for sd_blob_hash, stream_hash in stream_desc_db.RangeIter():
c.execute("insert into live_stream_descriptors values (?, ?)",
(sd_blob_hash, stream_hash))
db_conn.commit()
db_conn.close()
def migrate_ptc_db(old_db_dir, new_db_dir):
old_db_path = os.path.join(old_db_dir, "ptcwallet.db")
if not os.path.exists(old_db_path):
return True
old_db = leveldb.LevelDB(old_db_path)
try:
p_key = old_db.Get("private_key")
new_db = unqlite.UnQLite(os.path.join(new_db_dir, "ptcwallet.db"))
new_db['private_key'] = p_key
except KeyError:
pass
def migrate_lbryfile_manager_db(old_db_dir, new_db_dir):
old_db_path = os.path.join(old_db_dir, "lbryfiles.db")
if not os.path.exists(old_db_path):
return True
old_db = leveldb.LevelDB(old_db_path)
new_db = sqlite3.connect(os.path.join(new_db_dir, "lbryfile_info.db"))
c = new_db.cursor()
c.execute("create table if not exists lbry_file_options (" +
" blob_data_rate real, " +
" status text," +
" stream_hash text,"
" foreign key(stream_hash) references lbry_files(stream_hash)" +
")")
new_db.commit()
LBRYFILE_STATUS = "t"
LBRYFILE_OPTIONS = "o"
c = new_db.cursor()
for k, v in old_db.RangeIter():
key_type, stream_hash = json.loads(k)
if key_type == LBRYFILE_STATUS:
try:
rate = json.loads(old_db.Get(json.dumps((LBRYFILE_OPTIONS, stream_hash))))[0]
except KeyError:
rate = None
c.execute("insert into lbry_file_options values (?, ?, ?)",
(rate, v, stream_hash))
new_db.commit()
new_db.close()
def migrate_settings_db(old_db_dir, new_db_dir):
old_settings_db_path = os.path.join(old_db_dir, "settings.db")
if not os.path.exists(old_settings_db_path):
return True
old_db = leveldb.LevelDB(old_settings_db_path)
new_db = unqlite.UnQLite(os.path.join(new_db_dir, "settings.db"))
for k, v in old_db.RangeIter():
new_db[k] = v
def migrate_repeater_db(old_db_dir, new_db_dir):
old_repeater_db_path = os.path.join(old_db_dir, "valuable_blobs.db")
if not os.path.exists(old_repeater_db_path):
return True
old_db = leveldb.LevelDB(old_repeater_db_path)
info_db = sqlite3.connect(os.path.join(new_db_dir, "blind_info.db"))
peer_db = sqlite3.connect(os.path.join(new_db_dir, "blind_peers.db"))
unql_db = unqlite.UnQLite(os.path.join(new_db_dir, "blind_settings.db"))
BLOB_INFO_TYPE = 'b'
SETTING_TYPE = 's'
PEER_TYPE = 'p'
info_c = info_db.cursor()
info_c.execute("create table if not exists valuable_blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" reference text, " +
" peer_host text, " +
" peer_port integer, " +
" peer_score text" +
")")
info_db.commit()
peer_c = peer_db.cursor()
peer_c.execute("create table if not exists approved_peers (" +
" ip_address text, " +
" port integer" +
")")
peer_db.commit()
info_c = info_db.cursor()
peer_c = peer_db.cursor()
for k, v in old_db.RangeIter():
key_type, key_rest = json.loads(k)
if key_type == PEER_TYPE:
host, port = key_rest
peer_c.execute("insert into approved_peers values (?, ?)",
(host, port))
elif key_type == SETTING_TYPE:
unql_db[key_rest] = v
elif key_type == BLOB_INFO_TYPE:
blob_hash = key_rest
length, reference, peer_host, peer_port, peer_score = json.loads(v)
info_c.execute("insert into valuable_blobs values (?, ?, ?, ?, ?, ?)",
(blob_hash, length, reference, peer_host, peer_port, peer_score))
info_db.commit()
peer_db.commit()
info_db.close()
peer_db.close()
+77
View File
@@ -0,0 +1,77 @@
import sqlite3
import os
import logging
log = logging.getLogger(__name__)
UNSET_NOUT = -1
def do_migration(db_dir):
log.info("Doing the migration")
migrate_blockchainname_db(db_dir)
log.info("Migration succeeded")
def migrate_blockchainname_db(db_dir):
blockchainname_db = os.path.join(db_dir, "blockchainname.db")
# skip migration on fresh installs
if not os.path.isfile(blockchainname_db):
return
temp_db = sqlite3.connect(":memory:")
db_file = sqlite3.connect(blockchainname_db)
file_cursor = db_file.cursor()
mem_cursor = temp_db.cursor()
mem_cursor.execute("create table if not exists name_metadata ("
" name text, "
" txid text, "
" n integer, "
" sd_hash text)")
mem_cursor.execute("create table if not exists claim_ids ("
" claimId text, "
" name text, "
" txid text, "
" n integer)")
temp_db.commit()
name_metadata = file_cursor.execute("select * from name_metadata").fetchall()
claim_metadata = file_cursor.execute("select * from claim_ids").fetchall()
# fill n as V1_UNSET_NOUT, Wallet.py will be responsible for filling in correct n
for name, txid, sd_hash in name_metadata:
mem_cursor.execute(
"insert into name_metadata values (?, ?, ?, ?) ",
(name, txid, UNSET_NOUT, sd_hash))
for claim_id, name, txid in claim_metadata:
mem_cursor.execute(
"insert into claim_ids values (?, ?, ?, ?)",
(claim_id, name, txid, UNSET_NOUT))
temp_db.commit()
new_name_metadata = mem_cursor.execute("select * from name_metadata").fetchall()
new_claim_metadata = mem_cursor.execute("select * from claim_ids").fetchall()
file_cursor.execute("drop table name_metadata")
file_cursor.execute("create table name_metadata ("
" name text, "
" txid text, "
" n integer, "
" sd_hash text)")
for name, txid, n, sd_hash in new_name_metadata:
file_cursor.execute(
"insert into name_metadata values (?, ?, ?, ?) ", (name, txid, n, sd_hash))
file_cursor.execute("drop table claim_ids")
file_cursor.execute("create table claim_ids ("
" claimId text, "
" name text, "
" txid text, "
" n integer)")
for claim_id, name, txid, n in new_claim_metadata:
file_cursor.execute("insert into claim_ids values (?, ?, ?, ?)", (claim_id, name, txid, n))
db_file.commit()
db_file.close()
temp_db.close()
+2 -2
View File
@@ -10,7 +10,7 @@
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
@@ -111,7 +111,7 @@ the following:
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
+11 -8
View File
@@ -10,7 +10,7 @@
""" This module defines the charaterizing constants of the Kademlia network
C{checkRefreshInterval} and C{udpDatagramMaxSize} are implementation-specific
constants, and do not affect general Kademlia operation.
constants, and do not affect general Kademlia operation.
"""
######### KADEMLIA CONSTANTS ###########
@@ -28,14 +28,14 @@ rpcTimeout = 5
iterativeLookupDelay = rpcTimeout / 2
#: If a k-bucket has not been used for this amount of time, refresh it (in seconds)
refreshTimeout = 3600 # 1 hour
refreshTimeout = 3600 # 1 hour
#: The interval at which nodes replicate (republish/refresh) data they are holding
replicateInterval = refreshTimeout
# The time it takes for data to expire in the network; the original publisher of the data
# will also republish the data at this time if it is still valid
dataExpireTimeout = 86400 # 24 hours
dataExpireTimeout = 86400 # 24 hours
tokenSecretChangeInterval = 300 # 5 minutes
tokenSecretChangeInterval = 300 # 5 minutes
peer_request_timeout = 10
@@ -43,10 +43,13 @@ peer_request_timeout = 10
#: The interval in which the node should check its whether any buckets need refreshing,
#: or whether any data needs to be republished (in seconds)
checkRefreshInterval = refreshTimeout/5
checkRefreshInterval = refreshTimeout / 5
#: Max size of a single UDP datagram, in bytes. If a message is larger than this, it will
#: be spread accross several UDP packets.
udpDatagramMaxSize = 8192 # 8 KB
#: be spread across several UDP packets.
udpDatagramMaxSize = 8192 # 8 KB
key_bits = 384
from lbrynet.core.cryptoutils import get_lbry_hash_obj
h = get_lbry_hash_obj()
key_bits = h.digest_size * 8 # 384 bits
+14 -9
View File
@@ -10,17 +10,18 @@
class Contact(object):
""" Encapsulation for remote contact
This class contains information on a single remote contact, and also
provides a direct RPC API to the remote node which it represents
"""
def __init__(self, id, ipAddress, udpPort, networkProtocol, firstComm=0):
self.id = id
self.address = ipAddress
self.port = udpPort
self._networkProtocol = networkProtocol
self.commTime = firstComm
def __eq__(self, other):
if isinstance(other, Contact):
return self.id == other.id
@@ -28,7 +29,7 @@ class Contact(object):
return self.id == other
else:
return False
def __ne__(self, other):
if isinstance(other, Contact):
return self.id != other.id
@@ -38,26 +39,30 @@ class Contact(object):
return True
def compact_ip(self):
compact_ip = reduce(lambda buff, x: buff + bytearray([int(x)]), self.address.split('.'), bytearray())
compact_ip = reduce(
lambda buff, x: buff + bytearray([int(x)]), self.address.split('.'), bytearray())
return str(compact_ip)
def __str__(self):
return '<%s.%s object; IP address: %s, UDP port: %d>' % (self.__module__, self.__class__.__name__, self.address, self.port)
return '<%s.%s object; IP address: %s, UDP port: %d>' % (
self.__module__, self.__class__.__name__, self.address, self.port)
def __getattr__(self, name):
""" This override allows the host node to call a method of the remote
node (i.e. this contact) as if it was a local function.
For instance, if C{remoteNode} is a instance of C{Contact}, the
following will result in C{remoteNode}'s C{test()} method to be
called with argument C{123}::
remoteNode.test(123)
Such a RPC method call will return a Deferred, which will callback
when the contact responds with the result (or an error occurs).
This happens via this contact's C{_networkProtocol} object (i.e. the
host Node's C{_protocol} object).
"""
def _sendRPC(*args, **kwargs):
return self._networkProtocol.sendRPC(self, name, args, **kwargs)
return _sendRPC
+5 -152
View File
@@ -8,63 +8,27 @@
# may be created by processing this file with epydoc: http://epydoc.sf.net
import UserDict
#import sqlite3
import cPickle as pickle
import time
import os
import constants
class DataStore(UserDict.DictMixin):
""" Interface for classes implementing physical storage (for data
published via the "STORE" RPC) for the Kademlia DHT
@note: This provides an interface for a dict-like object
"""
def keys(self):
""" Return a list of the keys in this data store """
# def lastPublished(self, key):
# """ Get the time the C{(key, value)} pair identified by C{key}
# was last published """
# def originalPublisherID(self, key):
# """ Get the original publisher of the data's node ID
#
# @param key: The key that identifies the stored data
# @type key: str
#
# @return: Return the node ID of the original publisher of the
# C{(key, value)} pair identified by C{key}.
# """
# def originalPublishTime(self, key):
# """ Get the time the C{(key, value)} pair identified by C{key}
# was originally published """
# def setItem(self, key, value, lastPublished, originallyPublished, originalPublisherID):
# """ Set the value of the (key, value) pair identified by C{key};
# this should set the "last published" value for the (key, value)
# pair to the current time
# """
def addPeerToBlob(self, key, value, lastPublished, originallyPublished, originalPublisherID):
pass
# def __getitem__(self, key):
# """ Get the value identified by C{key} """
# def __setitem__(self, key, value):
# """ Convenience wrapper to C{setItem}; this accepts a tuple in the
# format: (value, lastPublished, originallyPublished, originalPublisherID) """
# self.setItem(key, *value)
# def __delitem__(self, key):
# """ Delete the specified key (and its value) """
class DictDataStore(DataStore):
""" A datastore using an in-memory Python dictionary """
def __init__(self):
# Dictionary format:
# { <key>: (<value>, <lastPublished>, <originallyPublished> <originalPublisherID>) }
@@ -74,33 +38,14 @@ class DictDataStore(DataStore):
""" Return a list of the keys in this data store """
return self._dict.keys()
# def lastPublished(self, key):
# """ Get the time the C{(key, value)} pair identified by C{key}
# was last published """
# return self._dict[key][1]
# def originalPublisherID(self, key):
# """ Get the original publisher of the data's node ID
#
# @param key: The key that identifies the stored data
# @type key: str
#
# @return: Return the node ID of the original publisher of the
# C{(key, value)} pair identified by C{key}.
# """
# return self._dict[key][3]
# def originalPublishTime(self, key):
# """ Get the time the C{(key, value)} pair identified by C{key}
# was originally published """
# return self._dict[key][2]
def removeExpiredPeers(self):
now = int(time.time())
def notExpired(peer):
if (now - peer[2]) > constants.dataExpireTimeout:
return False
return True
for key in self._dict.keys():
unexpired_peers = filter(notExpired, self._dict[key])
self._dict[key] = unexpired_peers
@@ -119,95 +64,3 @@ class DictDataStore(DataStore):
def getPeersForBlob(self, key):
if key in self._dict:
return [val[0] for val in self._dict[key]]
# def setItem(self, key, value, lastPublished, originallyPublished, originalPublisherID):
# """ Set the value of the (key, value) pair identified by C{key};
# this should set the "last published" value for the (key, value)
# pair to the current time
# """
# self._dict[key] = (value, lastPublished, originallyPublished, originalPublisherID)
# def __getitem__(self, key):
# """ Get the value identified by C{key} """
# return self._dict[key][0]
# def __delitem__(self, key):
# """ Delete the specified key (and its value) """
# del self._dict[key]
#class SQLiteDataStore(DataStore):
# """ Example of a SQLite database-based datastore
# """
# def __init__(self, dbFile=':memory:'):
# """
# @param dbFile: The name of the file containing the SQLite database; if
# unspecified, an in-memory database is used.
# @type dbFile: str
# """
# createDB = not os.path.exists(dbFile)
# self._db = sqlite3.connect(dbFile)
# self._db.isolation_level = None
# self._db.text_factory = str
# if createDB:
# self._db.execute('CREATE TABLE data(key, value, lastPublished, originallyPublished, originalPublisherID)')
# self._cursor = self._db.cursor()
# def keys(self):
# """ Return a list of the keys in this data store """
# keys = []
# try:
# self._cursor.execute("SELECT key FROM data")
# for row in self._cursor:
# keys.append(row[0].decode('hex'))
# finally:
# return keys
# def lastPublished(self, key):
# """ Get the time the C{(key, value)} pair identified by C{key}
# was last published """
# return int(self._dbQuery(key, 'lastPublished'))
# def originalPublisherID(self, key):
# """ Get the original publisher of the data's node ID
# @param key: The key that identifies the stored data
# @type key: str
# @return: Return the node ID of the original publisher of the
# C{(key, value)} pair identified by C{key}.
# """
# return self._dbQuery(key, 'originalPublisherID')
# def originalPublishTime(self, key):
# """ Get the time the C{(key, value)} pair identified by C{key}
# was originally published """
# return int(self._dbQuery(key, 'originallyPublished'))
# def setItem(self, key, value, lastPublished, originallyPublished, originalPublisherID):
# # Encode the key so that it doesn't corrupt the database
# encodedKey = key.encode('hex')
# self._cursor.execute("select key from data where key=:reqKey", {'reqKey': encodedKey})
# if self._cursor.fetchone() == None:
# self._cursor.execute('INSERT INTO data(key, value, lastPublished, originallyPublished, originalPublisherID) VALUES (?, ?, ?, ?, ?)', (encodedKey, buffer(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)), lastPublished, originallyPublished, originalPublisherID))
# else:
# self._cursor.execute('UPDATE data SET value=?, lastPublished=?, originallyPublished=?, originalPublisherID=? WHERE key=?', (buffer(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)), lastPublished, originallyPublished, originalPublisherID, encodedKey))
# def _dbQuery(self, key, columnName, unpickle=False):
# try:
# self._cursor.execute("SELECT %s FROM data WHERE key=:reqKey" % columnName, {'reqKey': key.encode('hex')})
# row = self._cursor.fetchone()
# value = str(row[0])
# except TypeError:
# raise KeyError, key
# else:
# if unpickle:
# return pickle.loads(value)
# else:
# return value
# def __getitem__(self, key):
# return self._dbQuery(key, 'value', unpickle=True)
# def __delitem__(self, key):
# self._cursor.execute("DELETE FROM data WHERE key=:reqKey", {'reqKey': key.encode('hex')})
+35 -28
View File
@@ -7,54 +7,59 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
class DecodeError(Exception):
""" Should be raised by an C{Encoding} implementation if decode operation
fails
"""
class Encoding(object):
""" Interface for RPC message encoders/decoders
All encoding implementations used with this library should inherit and
implement this.
"""
def encode(self, data):
""" Encode the specified data
@param data: The data to encode
This method has to support encoding of the following
types: C{str}, C{int} and C{long}
Any additional data types may be supported as long as the
implementing class's C{decode()} method can successfully
decode them.
@return: The encoded data
@rtype: str
"""
def decode(self, data):
""" Decode the specified data string
@param data: The data (byte string) to decode.
@type data: str
@return: The decoded data (in its correct type)
"""
class Bencode(Encoding):
""" Implementation of a Bencode-based algorithm (Bencode is the encoding
algorithm used by Bittorrent).
@note: This algorithm differs from the "official" Bencode algorithm in
that it can encode/decode floating point values in addition to
integers.
"""
def encode(self, data):
""" Encoder implementation of the Bencode algorithm
@param data: The data to encode
@type data: int, long, tuple, list, dict or str
@return: The encoded data
@rtype: str
"""
@@ -76,24 +81,25 @@ class Bencode(Encoding):
encodedDictItems += self.encode(data[key])
return 'd%se' % encodedDictItems
elif type(data) == float:
# This (float data type) is a non-standard extension to the original Bencode algorithm
# This (float data type) is a non-standard extension to the original Bencode algorithm
return 'f%fe' % data
elif data == None:
# This (None/NULL data type) is a non-standard extension to the original Bencode algorithm
# This (None/NULL data type) is a non-standard extension
# to the original Bencode algorithm
return 'n'
else:
print data
raise TypeError, "Cannot bencode '%s' object" % type(data)
def decode(self, data):
""" Decoder implementation of the Bencode algorithm
""" Decoder implementation of the Bencode algorithm
@param data: The encoded data
@type data: str
@note: This is a convenience wrapper for the recursive decoding
algorithm, C{_decodeRecursive}
@return: The decoded data, as a native Python type
@rtype: int, list, dict or str
"""
@@ -103,23 +109,23 @@ class Bencode(Encoding):
return self._decodeRecursive(data)[0]
except ValueError as e:
raise DecodeError, e.message
@staticmethod
def _decodeRecursive(data, startIndex=0):
""" Actual implementation of the recursive Bencode algorithm
Do not call this; use C{decode()} instead
"""
if data[startIndex] == 'i':
endPos = data[startIndex:].find('e')+startIndex
return (int(data[startIndex+1:endPos]), endPos+1)
endPos = data[startIndex:].find('e') + startIndex
return (int(data[startIndex + 1:endPos]), endPos + 1)
elif data[startIndex] == 'l':
startIndex += 1
decodedList = []
while data[startIndex] != 'e':
listData, startIndex = Bencode._decodeRecursive(data, startIndex)
decodedList.append(listData)
return (decodedList, startIndex+1)
return (decodedList, startIndex + 1)
elif data[startIndex] == 'd':
startIndex += 1
decodedDict = {}
@@ -130,18 +136,19 @@ class Bencode(Encoding):
return (decodedDict, startIndex)
elif data[startIndex] == 'f':
# This (float data type) is a non-standard extension to the original Bencode algorithm
endPos = data[startIndex:].find('e')+startIndex
return (float(data[startIndex+1:endPos]), endPos+1)
endPos = data[startIndex:].find('e') + startIndex
return (float(data[startIndex + 1:endPos]), endPos + 1)
elif data[startIndex] == 'n':
# This (None/NULL data type) is a non-standard extension to the original Bencode algorithm
return (None, startIndex+1)
# This (None/NULL data type) is a non-standard extension
# to the original Bencode algorithm
return (None, startIndex + 1)
else:
splitPos = data[startIndex:].find(':')+startIndex
splitPos = data[startIndex:].find(':') + startIndex
try:
length = int(data[startIndex:splitPos])
except ValueError, e:
raise DecodeError, e
startIndex = splitPos+1
endPos = startIndex+length
startIndex = splitPos + 1
endPos = startIndex + length
bytes = data[startIndex:endPos]
return (bytes, endPos)
+3 -4
View File
@@ -1,10 +1,9 @@
from collections import Counter
import datetime
class HashWatcher():
def __init__(self, ttl=600):
class HashWatcher(object):
def __init__(self):
self.ttl = 600
self.hashes = []
self.next_tick = None
@@ -32,4 +31,4 @@ class HashWatcher():
def _remove_old_hashes(self):
remove_time = datetime.datetime.now() - datetime.timedelta(minutes=10)
self.hashes = [h for h in self.hashes if h[1] < remove_time]
self.hashes = [h for h in self.hashes if h[1] < remove_time]
+17 -14
View File
@@ -9,6 +9,7 @@
import constants
class BucketFull(Exception):
""" Raised when the bucket is full """
@@ -16,6 +17,7 @@ class BucketFull(Exception):
class KBucket(object):
""" Description - later
"""
def __init__(self, rangeMin, rangeMax):
"""
@param rangeMin: The lower boundary for the range in the n-bit ID
@@ -31,17 +33,18 @@ class KBucket(object):
def addContact(self, contact):
""" Add contact to _contact list in the right order. This will move the
contact to the end of the k-bucket if it is already present.
@raise kademlia.kbucket.BucketFull: Raised when the bucket is full and
the contact isn't in the bucket
already
@param contact: The contact to add
@type contact: kademlia.contact.Contact
"""
if contact in self._contacts:
# Move the existing contact to the end of the list
# - using the new contact to allow add-on data (e.g. optimization-specific stuff) to pe updated as well
# - using the new contact to allow add-on data
# (e.g. optimization-specific stuff) to pe updated as well
self._contacts.remove(contact)
self._contacts.append(contact)
elif len(self._contacts) < constants.k:
@@ -56,7 +59,7 @@ class KBucket(object):
def getContacts(self, count=-1, excludeContact=None):
""" Returns a list containing up to the first count number of contacts
@param count: The amount of contacts to return (if 0 or less, return
all contacts)
@type count: int
@@ -64,12 +67,12 @@ class KBucket(object):
the list of returned values, it will be
discarded before returning. If a C{str} is
passed as this argument, it must be the
contact's ID.
contact's ID.
@type excludeContact: kademlia.contact.Contact or str
@raise IndexError: If the number of requested contacts is too large
@return: Return up to the first count number of contacts in a list
If no contacts are present an empty is returned
@rtype: list
@@ -96,7 +99,7 @@ class KBucket(object):
# enough contacts in list
else:
contactList = self._contacts[0:count]
if excludeContact in contactList:
contactList.remove(excludeContact)
@@ -104,24 +107,24 @@ class KBucket(object):
def removeContact(self, contact):
""" Remove given contact from list
@param contact: The contact to remove, or a string containing the
contact's node ID
@type contact: kademlia.contact.Contact or str
@raise ValueError: The specified contact is not in this bucket
"""
self._contacts.remove(contact)
def keyInRange(self, key):
""" Tests whether the specified key (i.e. node ID) is in the range
of the n-bit ID space covered by this k-bucket (in otherwords, it
returns whether or not the specified key should be placed in this
k-bucket)
@param key: The key to test
@type key: str or int
@return: C{True} if the key is in this k-bucket's range, or C{False}
if not.
@rtype: bool
+25 -14
View File
@@ -9,54 +9,65 @@
import msgtypes
class MessageTranslator(object):
""" Interface for RPC message translators/formatters
Classes inheriting from this should provide a translation services between
the classes used internally by this Kademlia implementation and the actual
data that is transmitted between nodes.
"""
def fromPrimitive(self, msgPrimitive):
""" Create an RPC Message from a message's string representation
@param msgPrimitive: The unencoded primitive representation of a message
@type msgPrimitive: str, int, list or dict
@return: The translated message object
@rtype: entangled.kademlia.msgtypes.Message
"""
def toPrimitive(self, message):
""" Create a string representation of a message
@param message: The message object
@type message: msgtypes.Message
@return: The message's primitive representation in a particular
messaging format
@rtype: str, int, list or dict
"""
class DefaultFormat(MessageTranslator):
""" The default on-the-wire message format for this library """
typeRequest, typeResponse, typeError = range(3)
headerType, headerMsgID, headerNodeID, headerPayload, headerArgs = range(5)
def fromPrimitive(self, msgPrimitive):
msgType = msgPrimitive[self.headerType]
if msgType == self.typeRequest:
msg = msgtypes.RequestMessage(msgPrimitive[self.headerNodeID], msgPrimitive[self.headerPayload], msgPrimitive[self.headerArgs], msgPrimitive[self.headerMsgID])
msg = msgtypes.RequestMessage(msgPrimitive[self.headerNodeID],
msgPrimitive[self.headerPayload],
msgPrimitive[self.headerArgs],
msgPrimitive[self.headerMsgID])
elif msgType == self.typeResponse:
msg = msgtypes.ResponseMessage(msgPrimitive[self.headerMsgID], msgPrimitive[self.headerNodeID], msgPrimitive[self.headerPayload])
msg = msgtypes.ResponseMessage(msgPrimitive[self.headerMsgID],
msgPrimitive[self.headerNodeID],
msgPrimitive[self.headerPayload])
elif msgType == self.typeError:
msg = msgtypes.ErrorMessage(msgPrimitive[self.headerMsgID], msgPrimitive[self.headerNodeID], msgPrimitive[self.headerPayload], msgPrimitive[self.headerArgs])
msg = msgtypes.ErrorMessage(msgPrimitive[self.headerMsgID],
msgPrimitive[self.headerNodeID],
msgPrimitive[self.headerPayload],
msgPrimitive[self.headerArgs])
else:
# Unknown message, no payload
msg = msgtypes.Message(msgPrimitive[self.headerMsgID], msgPrimitive[self.headerNodeID])
return msg
def toPrimitive(self, message):
msg = {self.headerMsgID: message.id,
def toPrimitive(self, message):
msg = {self.headerMsgID: message.id,
self.headerNodeID: message.nodeID}
if isinstance(message, msgtypes.RequestMessage):
msg[self.headerType] = self.typeRequest
+7 -5
View File
@@ -7,11 +7,12 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import hashlib
import random
from lbrynet.core.utils import generate_id
class Message(object):
""" Base class for messages - all "unknown" messages use this class """
def __init__(self, rpcID, nodeID):
self.id = rpcID
self.nodeID = nodeID
@@ -19,11 +20,10 @@ class Message(object):
class RequestMessage(Message):
""" Message containing an RPC request """
def __init__(self, nodeID, method, methodArgs, rpcID=None):
if rpcID == None:
hash = hashlib.sha384()
hash.update(str(random.getrandbits(255)))
rpcID = hash.digest()
rpcID = generate_id()
Message.__init__(self, rpcID, nodeID)
self.request = method
self.args = methodArgs
@@ -31,6 +31,7 @@ class RequestMessage(Message):
class ResponseMessage(Message):
""" Message containing the result from a successful RPC request """
def __init__(self, rpcID, nodeID, response):
Message.__init__(self, rpcID, nodeID)
self.response = response
@@ -38,6 +39,7 @@ class ResponseMessage(Message):
class ErrorMessage(ResponseMessage):
""" Message containing the error from an unsuccessful RPC request """
def __init__(self, rpcID, nodeID, exceptionType, errorMessage):
ResponseMessage.__init__(self, rpcID, nodeID, errorMessage)
if isinstance(exceptionType, type):
+349 -468
View File
File diff suppressed because it is too large. Load diff
+140 -76
View File
@@ -7,10 +7,15 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import logging
import binascii
import time
import socket
import errno
from twisted.internet import protocol, defer
from twisted.python import failure
from twisted.internet import error
import twisted.internet.reactor
import constants
@@ -20,25 +25,57 @@ import msgformat
from contact import Contact
reactor = twisted.internet.reactor
log = logging.getLogger(__name__)
class TimeoutError(Exception):
""" Raised when a RPC times out """
def __init__(self, remote_contact_id):
# remote_contact_id is a binary blob so we need to convert it
# into something more readable
msg = 'Timeout connecting to {}'.format(binascii.hexlify(remote_contact_id))
Exception.__init__(self, msg)
self.remote_contact_id = remote_contact_id
class Delay(object):
maxToSendDelay = 10 ** -3 # 0.05
minToSendDelay = 10 ** -5 # 0.01
def __init__(self, start=0):
self._next = start
# TODO: explain why this logic is like it is. And add tests that
# show that it actually does what it needs to do.
def __call__(self):
ts = time.time()
delay = 0
if ts >= self._next:
delay = self.minToSendDelay
self._next = ts + self.minToSendDelay
else:
delay = (self._next - ts) + self.maxToSendDelay
self._next += self.maxToSendDelay
return delay
class KademliaProtocol(protocol.DatagramProtocol):
""" Implements all low-level network-related functions of a Kademlia node """
msgSizeLimit = constants.udpDatagramMaxSize-26
maxToSendDelay = 10**-3#0.05
minToSendDelay = 10**-5#0.01
msgSizeLimit = constants.udpDatagramMaxSize - 26
def __init__(self, node, msgEncoder=encoding.Bencode(), msgTranslator=msgformat.DefaultFormat()):
def __init__(self, node, msgEncoder=encoding.Bencode(),
msgTranslator=msgformat.DefaultFormat()):
self._node = node
self._encoder = msgEncoder
self._translator = msgTranslator
self._sentMessages = {}
self._partialMessages = {}
self._partialMessagesProgress = {}
self._next = 0
self._callLaterList = {}
self._delay = Delay()
# keep track of outstanding writes so that they
# can be cancelled on shutdown
self._call_later_list = {}
def sendRPC(self, contact, method, args, rawResponse=False):
""" Sends an RPC to the specified contact
@@ -71,12 +108,15 @@ class KademliaProtocol(protocol.DatagramProtocol):
msgPrimitive = self._translator.toPrimitive(msg)
encodedMsg = self._encoder.encode(msgPrimitive)
log.debug("DHT SEND: %s(%s)", method, args)
df = defer.Deferred()
if rawResponse:
df._rpcRawResponse = True
# Set the RPC timeout timer
timeoutCall = reactor.callLater(constants.rpcTimeout, self._msgTimeout, msg.id) #IGNORE:E1101
timeoutCall = reactor.callLater(
constants.rpcTimeout, self._msgTimeout, msg.id) # IGNORE:E1101
# Transmit the data
self._send(encodedMsg, msg.id, (contact.address, contact.port))
self._sentMessages[msg.id] = (contact.id, df, timeoutCall)
@@ -110,10 +150,13 @@ class KademliaProtocol(protocol.DatagramProtocol):
except encoding.DecodeError:
# We received some rubbish here
return
except IndexError:
log.warning("Couldn't decode dht datagram from %s", address)
return
message = self._translator.fromPrimitive(msgPrimitive)
remoteContact = Contact(message.nodeID, address[0], address[1], self)
# Refresh the remote node's details in the local node's k-buckets
self._node.addContact(remoteContact)
@@ -122,14 +165,16 @@ class KademliaProtocol(protocol.DatagramProtocol):
self._handleRPC(remoteContact, message.id, message.request, message.args)
elif isinstance(message, msgtypes.ResponseMessage):
# Find the message that triggered this response
if self._sentMessages.has_key(message.id):
if message.id in self._sentMessages:
# Cancel timeout timer for this RPC
df, timeoutCall = self._sentMessages[message.id][1:3]
timeoutCall.cancel()
del self._sentMessages[message.id]
if hasattr(df, '_rpcRawResponse'):
# The RPC requested that the raw response message and originating address be returned; do not interpret it
# The RPC requested that the raw response message
# and originating address be returned; do not
# interpret it
df.callback((message, address))
elif isinstance(message, msgtypes.ErrorMessage):
# The RPC request raised a remote exception; raise it locally
@@ -138,7 +183,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
else:
localModuleHierarchy = self.__module__.split('.')
remoteHierarchy = message.exceptionType.split('.')
#strip the remote hierarchy
# strip the remote hierarchy
while remoteHierarchy[0] == localModuleHierarchy[0]:
remoteHierarchy.pop(0)
localModuleHierarchy.pop(0)
@@ -155,13 +200,13 @@ class KademliaProtocol(protocol.DatagramProtocol):
df.callback(message.response)
else:
# If the original message isn't found, it must have timed out
#TODO: we should probably do something with this...
# TODO: we should probably do something with this...
pass
def _send(self, data, rpcID, address):
""" Transmit the specified data over UDP, breaking it up into several
packets if necessary
If the data is spread over multiple UDP datagrams, the packets have the
following structure::
| | | | | |||||||||||| 0x00 |
@@ -169,15 +214,19 @@ class KademliaProtocol(protocol.DatagramProtocol):
| type ID | of packets |of this packet | | indicator|
| (1 byte) | (2 bytes) | (2 bytes) |(20 bytes)| (1 byte) |
| | | | | |||||||||||| |
@note: The header used for breaking up large data segments will
possibly be moved out of the KademliaProtocol class in the
future, into something similar to a message translator/encoder
class (see C{kademlia.msgformat} and C{kademlia.encoding}).
class (see C{kademlia.msgformat} and C{kademlia.encoding}).
"""
if len(data) > self.msgSizeLimit:
# We have to spread the data over multiple UDP datagrams, and provide sequencing information
# 1st byte is transmission type id, bytes 2 & 3 are the total number of packets in this transmission, bytes 4 & 5 are the sequence number for this specific packet
# We have to spread the data over multiple UDP datagrams,
# and provide sequencing information
#
# 1st byte is transmission type id, bytes 2 & 3 are the
# total number of packets in this transmission, bytes 4 &
# 5 are the sequence number for this specific packet
totalPackets = len(data) / self.msgSizeLimit
if len(data) % self.msgSizeLimit > 0:
totalPackets += 1
@@ -185,33 +234,35 @@ class KademliaProtocol(protocol.DatagramProtocol):
seqNumber = 0
startPos = 0
while seqNumber < totalPackets:
#reactor.iterate() #IGNORE:E1101
packetData = data[startPos:startPos+self.msgSizeLimit]
packetData = data[startPos:startPos + self.msgSizeLimit]
encSeqNumber = chr(seqNumber >> 8) + chr(seqNumber & 0xff)
txData = '\x00%s%s%s\x00%s' % (encTotalPackets, encSeqNumber, rpcID, packetData)
self._sendNext(txData, address)
self._scheduleSendNext(txData, address)
startPos += self.msgSizeLimit
seqNumber += 1
else:
self._sendNext(data, address)
self._scheduleSendNext(data, address)
def _sendNext(self, txData, address):
""" Send the next UDP packet """
ts = time.time()
delay = 0
if ts >= self._next:
delay = self.minToSendDelay
self._next = ts + self.minToSendDelay
else:
delay = (self._next-ts) + self.maxToSendDelay
self._next += self.maxToSendDelay
def _scheduleSendNext(self, txData, address):
"""Schedule the sending of the next UDP packet """
delay = self._delay()
key = object()
delayed_call = reactor.callLater(delay, self._write_and_remove, key, txData, address)
self._call_later_list[key] = delayed_call
def _write_and_remove(self, key, txData, address):
del self._call_later_list[key]
if self.transport:
laterCall = reactor.callLater(delay, self.transport.write, txData, address)
for key in self._callLaterList.keys():
if key <= ts:
del self._callLaterList[key]
self._callLaterList[self._next] = laterCall
try:
self.transport.write(txData, address)
except socket.error as err:
if err.errno == errno.EWOULDBLOCK:
# i'm scared this may swallow important errors, but i get a million of these
# on Linux and it doesnt seem to affect anything -grin
log.warning("Can't send data to dht: EWOULDBLOCK")
else:
raise err
def _sendResponse(self, contact, rpcID, response):
""" Send a RPC response to the specified contact
@@ -231,6 +282,7 @@ class KademliaProtocol(protocol.DatagramProtocol):
def _handleRPC(self, senderContact, rpcID, method, args):
""" Executes a local function in response to an RPC request """
# Set up the deferred callchain
def handleError(f):
self._sendError(senderContact, rpcID, f.type, f.getErrorMessage())
@@ -246,60 +298,72 @@ class KademliaProtocol(protocol.DatagramProtocol):
func = getattr(self._node, method, None)
if callable(func) and hasattr(func, 'rpcmethod'):
# Call the exposed Node method and return the result to the deferred callback chain
log.debug("DHT RECV CALL %s with args %s", method, args)
try:
##try:
## # Try to pass the sender's node id to the function...
result = func(*args, **{'_rpcNodeID': senderContact.id, '_rpcNodeContact': senderContact})
##except TypeError:
## # ...or simply call it if that fails
## result = func(*args)
kwargs = {'_rpcNodeID': senderContact.id, '_rpcNodeContact': senderContact}
result = func(*args, **kwargs)
except Exception, e:
df.errback(failure.Failure(e))
else:
df.callback(result)
else:
# No such exposed method
df.errback( failure.Failure( AttributeError('Invalid method: %s' % method) ) )
df.errback(failure.Failure(AttributeError('Invalid method: %s' % method)))
def _msgTimeout(self, messageID):
""" Called when an RPC request message times out """
# Find the message that timed out
if self._sentMessages.has_key(messageID):
remoteContactID, df = self._sentMessages[messageID][0:2]
if self._partialMessages.has_key(messageID):
# We are still receiving this message
# See if any progress has been made; if not, kill the message
if self._partialMessagesProgress.has_key(messageID):
if len(self._partialMessagesProgress[messageID]) == len(self._partialMessages[messageID]):
# No progress has been made
del self._partialMessagesProgress[messageID]
del self._partialMessages[messageID]
df.errback(failure.Failure(TimeoutError(remoteContactID)))
return
# Reset the RPC timeout timer
timeoutCall = reactor.callLater(constants.rpcTimeout, self._msgTimeout, messageID) #IGNORE:E1101
self._sentMessages[messageID] = (remoteContactID, df, timeoutCall)
return
del self._sentMessages[messageID]
# The message's destination node is now considered to be dead;
# raise an (asynchronous) TimeoutError exception and update the host node
self._node.removeContact(remoteContactID)
df.errback(failure.Failure(TimeoutError(remoteContactID)))
else:
if not self._sentMessages.has_key(messageID):
# This should never be reached
print "ERROR: deferred timed out, but is not present in sent messages list!"
log.error("deferred timed out, but is not present in sent messages list!")
return
remoteContactID, df = self._sentMessages[messageID][0:2]
if self._partialMessages.has_key(messageID):
# We are still receiving this message
self._msgTimeoutInProgress(messageID, remoteContactID, df)
return
del self._sentMessages[messageID]
# The message's destination node is now considered to be dead;
# raise an (asynchronous) TimeoutError exception and update the host node
self._node.removeContact(remoteContactID)
df.errback(failure.Failure(TimeoutError(remoteContactID)))
def _msgTimeoutInProgress(self, messageID, remoteContactID, df):
# See if any progress has been made; if not, kill the message
if self._hasProgressBeenMade(messageID):
# Reset the RPC timeout timer
timeoutCall = reactor.callLater(constants.rpcTimeout, self._msgTimeout, messageID)
self._sentMessages[messageID] = (remoteContactID, df, timeoutCall)
else:
# No progress has been made
del self._partialMessagesProgress[messageID]
del self._partialMessages[messageID]
df.errback(failure.Failure(TimeoutError(remoteContactID)))
def _hasProgressBeenMade(self, messageID):
return (
self._partialMessagesProgress.has_key(messageID) and
(
len(self._partialMessagesProgress[messageID]) !=
len(self._partialMessages[messageID])
)
)
def stopProtocol(self):
""" Called when the transport is disconnected.
Will only be called once, after all ports are disconnected.
"""
for key in self._callLaterList.keys():
log.info('Stopping DHT')
for delayed_call in self._call_later_list.values():
try:
if key > time.time():
self._callLaterList[key].cancel()
except Exception, e:
print e
del self._callLaterList[key]
#TODO: test: do we really need the reactor.iterate() call?
reactor.iterate()
delayed_call.cancel()
except (error.AlreadyCalled, error.AlreadyCancelled):
log.debug('Attempted to cancel a DelayedCall that was not active')
except Exception:
log.exception('Failed to cancel a DelayedCall')
# not sure why this is needed, but taking this out sometimes causes
# exceptions.AttributeError: 'Port' object has no attribute 'socket'
# to happen on shutdown
# reactor.iterate()
log.info('DHT stopped')
+94 -83
View File
@@ -5,24 +5,28 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import time, random
import time
import random
import constants
import kbucket
from protocol import TimeoutError
class RoutingTable(object):
""" Interface for RPC message translators/formatters
Classes inheriting from this should provide a suitable routing table for
a parent Node object (i.e. the local entity in the Kademlia network)
"""
def __init__(self, parentNodeID):
"""
@param parentNodeID: The n-bit node ID of the node to which this
routing table belongs
@type parentNodeID: str
"""
def addContact(self, contact):
""" Add the given contact to the correct k-bucket; if it already
exists, its status will be updated
@@ -30,21 +34,11 @@ class RoutingTable(object):
@param contact: The contact to add to this node's k-buckets
@type contact: kademlia.contact.Contact
"""
def distance(self, keyOne, keyTwo):
""" Calculate the XOR result between two string variables
@return: XOR result of two long variables
@rtype: long
"""
valKeyOne = long(keyOne.encode('hex'), 16)
valKeyTwo = long(keyTwo.encode('hex'), 16)
return valKeyOne ^ valKeyTwo
def findCloseNodes(self, key, count, _rpcNodeID=None):
""" Finds a number of known nodes closest to the node/value with the
specified key.
@param key: the n-bit key (i.e. the node or value ID) to search for
@type key: str
@param count: the amount of contacts to return
@@ -53,20 +47,22 @@ class RoutingTable(object):
Whatever ID is passed in the paramater will get
excluded from the list of returned contacts.
@type _rpcNodeID: str
@return: A list of node contacts (C{kademlia.contact.Contact instances})
closest to the specified key.
closest to the specified key.
This method will return C{k} (or C{count}, if specified)
contacts if at all possible; it will only return fewer if the
node is returning all of the contacts that it knows of.
@rtype: list
"""
def getContact(self, contactID):
""" Returns the (known) contact with the specified node ID
@raise ValueError: No contact with the specified contact ID is known
by this node
"""
def getRefreshList(self, startIndex=0, force=False):
""" Finds all k-buckets that need refreshing, starting at the
k-bucket with the specified index, and returns IDs to be searched for
@@ -83,22 +79,24 @@ class RoutingTable(object):
will be refreshed, regardless of the time they were last
accessed.
@type force: bool
@return: A list of node ID's that the parent node should search for
in order to refresh the routing Table
@rtype: list
"""
def removeContact(self, contactID):
""" Remove the contact with the specified node ID from the routing
table
@param contactID: The node ID of the contact to remove
@type contactID: str
"""
def touchKBucket(self, key):
""" Update the "last accessed" timestamp of the k-bucket which covers
the range containing the specified key in the key/ID space
@param key: A key in the range of the target k-bucket
@type key: str
"""
@@ -106,19 +104,20 @@ class RoutingTable(object):
class TreeRoutingTable(RoutingTable):
""" This class implements a routing table used by a Node class.
The Kademlia routing table is a binary tree whose leaves are k-buckets,
where each k-bucket contains nodes with some common prefix of their IDs.
This prefix is the k-bucket's position in the binary tree; it therefore
covers some range of ID values, and together all of the k-buckets cover
the entire n-bit ID (or key) space (with no overlap).
@note: In this implementation, nodes in the tree (the k-buckets) are
added dynamically, as needed; this technique is described in the 13-page
version of the Kademlia paper, in section 2.4. It does, however, use the
C{PING} RPC-based k-bucket eviction algorithm described in section 2.2 of
that paper.
"""
def __init__(self, parentNodeID):
"""
@param parentNodeID: The n-bit node ID of the node to which this
@@ -126,7 +125,7 @@ class TreeRoutingTable(RoutingTable):
@type parentNodeID: str
"""
# Create the initial (single) k-bucket covering the range of the entire n-bit ID space
self._buckets = [kbucket.KBucket(rangeMin=0, rangeMax=2**constants.key_bits)]
self._buckets = [kbucket.KBucket(rangeMin=0, rangeMax=2 ** constants.key_bits)]
self._parentNodeID = parentNodeID
def addContact(self, contact):
@@ -143,7 +142,8 @@ class TreeRoutingTable(RoutingTable):
try:
self._buckets[bucketIndex].addContact(contact)
except kbucket.BucketFull:
# The bucket is full; see if it can be split (by checking if its range includes the host node's id)
# The bucket is full; see if it can be split (by checking
# if its range includes the host node's id)
if self._buckets[bucketIndex].keyInRange(self._parentNodeID):
self._splitBucket(bucketIndex)
# Retry the insertion attempt
@@ -151,18 +151,20 @@ class TreeRoutingTable(RoutingTable):
else:
# We can't split the k-bucket
# NOTE:
# In section 2.4 of the 13-page version of the Kademlia paper, it is specified that
# in this case, the new contact should simply be dropped. However, in section 2.2,
# it states that the head contact in the k-bucket (i.e. the least-recently seen node)
# should be pinged - if it does not reply, it should be dropped, and the new contact
# added to the tail of the k-bucket. This implementation follows section 2.2 regarding
# this point.
headContact = self._buckets[bucketIndex]._contacts[0]
# In section 2.4 of the 13-page version of the
# Kademlia paper, it is specified that in this case,
# the new contact should simply be dropped. However,
# in section 2.2, it states that the head contact in
# the k-bucket (i.e. the least-recently seen node)
# should be pinged - if it does not reply, it should
# be dropped, and the new contact added to the tail of
# the k-bucket. This implementation follows section
# 2.2 regarding this point.
def replaceContact(failure):
""" Callback for the deferred PING RPC to see if the head
node in the k-bucket is still responding
@type failure: twisted.python.failure.Failure
"""
failure.trap(TimeoutError)
@@ -176,17 +178,18 @@ class TreeRoutingTable(RoutingTable):
pass
# ...and add the new one at the tail of the bucket
self.addContact(contact)
# Ping the least-recently seen contact in this k-bucket
headContact = self._buckets[bucketIndex]._contacts[0]
df = headContact.ping()
# If there's an error (i.e. timeout), remove the head contact, and append the new one
head_contact = self._buckets[bucketIndex]._contacts[0]
df = head_contact.ping()
# If there's an error (i.e. timeout), remove the head
# contact, and append the new one
df.addErrback(replaceContact)
def findCloseNodes(self, key, count, _rpcNodeID=None):
""" Finds a number of known nodes closest to the node/value with the
specified key.
@param key: the n-bit key (i.e. the node or value ID) to search for
@type key: str
@param count: the amount of contacts to return
@@ -195,39 +198,41 @@ class TreeRoutingTable(RoutingTable):
Whatever ID is passed in the paramater will get
excluded from the list of returned contacts.
@type _rpcNodeID: str
@return: A list of node contacts (C{kademlia.contact.Contact instances})
closest to the specified key.
closest to the specified key.
This method will return C{k} (or C{count}, if specified)
contacts if at all possible; it will only return fewer if the
node is returning all of the contacts that it knows of.
@rtype: list
"""
#if key == self.id:
# bucketIndex = 0 #TODO: maybe not allow this to continue?
#else:
bucketIndex = self._kbucketIndex(key)
closestNodes = self._buckets[bucketIndex].getContacts(constants.k, _rpcNodeID)
# This method must return k contacts (even if we have the node with the specified key as node ID),
# unless there is less than k remote nodes in the routing table
# This method must return k contacts (even if we have the node
# with the specified key as node ID), unless there is less
# than k remote nodes in the routing table
i = 1
canGoLower = bucketIndex-i >= 0
canGoHigher = bucketIndex+i < len(self._buckets)
# Fill up the node list to k nodes, starting with the closest neighbouring nodes known
canGoLower = bucketIndex - i >= 0
canGoHigher = bucketIndex + i < len(self._buckets)
# Fill up the node list to k nodes, starting with the closest neighbouring nodes known
while len(closestNodes) < constants.k and (canGoLower or canGoHigher):
#TODO: this may need to be optimized
# TODO: this may need to be optimized
if canGoLower:
closestNodes.extend(self._buckets[bucketIndex-i].getContacts(constants.k - len(closestNodes), _rpcNodeID))
canGoLower = bucketIndex-(i+1) >= 0
closestNodes.extend(
self._buckets[bucketIndex - i].getContacts(
constants.k - len(closestNodes), _rpcNodeID))
canGoLower = bucketIndex - (i + 1) >= 0
if canGoHigher:
closestNodes.extend(self._buckets[bucketIndex+i].getContacts(constants.k - len(closestNodes), _rpcNodeID))
canGoHigher = bucketIndex+(i+1) < len(self._buckets)
closestNodes.extend(
self._buckets[bucketIndex + i].getContacts(
constants.k - len(closestNodes), _rpcNodeID))
canGoHigher = bucketIndex + (i + 1) < len(self._buckets)
i += 1
return closestNodes
def getContact(self, contactID):
""" Returns the (known) contact with the specified node ID
@raise ValueError: No contact with the specified contact ID is known
by this node
"""
@@ -255,7 +260,7 @@ class TreeRoutingTable(RoutingTable):
will be refreshed, regardless of the time they were last
accessed.
@type force: bool
@return: A list of node ID's that the parent node should search for
in order to refresh the routing Table
@rtype: list
@@ -272,7 +277,7 @@ class TreeRoutingTable(RoutingTable):
def removeContact(self, contactID):
""" Remove the contact with the specified node ID from the routing
table
@param contactID: The node ID of the contact to remove
@type contactID: str
"""
@@ -280,13 +285,12 @@ class TreeRoutingTable(RoutingTable):
try:
self._buckets[bucketIndex].removeContact(contactID)
except ValueError:
#print 'removeContact(): Contact not in routing table'
return
def touchKBucket(self, key):
""" Update the "last accessed" timestamp of the k-bucket which covers
the range containing the specified key in the key/ID space
@param key: A key in the range of the target k-bucket
@type key: str
"""
@@ -296,17 +300,16 @@ class TreeRoutingTable(RoutingTable):
def _kbucketIndex(self, key):
""" Calculate the index of the k-bucket which is responsible for the
specified key (or ID)
@param key: The key for which to find the appropriate k-bucket index
@type key: str
@return: The index of the k-bucket responsible for the specified key
@rtype: int
"""
valKey = long(key.encode('hex'), 16)
i = 0
for bucket in self._buckets:
if bucket.keyInRange(valKey):
if bucket.keyInRange(key):
return i
else:
i += 1
@@ -314,31 +317,32 @@ class TreeRoutingTable(RoutingTable):
def _randomIDInBucketRange(self, bucketIndex):
""" Returns a random ID in the specified k-bucket's range
@param bucketIndex: The index of the k-bucket to use
@type bucketIndex: int
"""
idValue = random.randrange(self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax)
idValue = random.randrange(
self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax)
randomID = hex(idValue)[2:]
if randomID[-1] == 'L':
randomID = randomID[:-1]
if len(randomID) % 2 != 0:
randomID = '0' + randomID
randomID = randomID.decode('hex')
randomID = (constants.key_bits/8 - len(randomID))*'\x00' + randomID
randomID = (constants.key_bits / 8 - len(randomID)) * '\x00' + randomID
return randomID
def _splitBucket(self, oldBucketIndex):
""" Splits the specified k-bucket into two new buckets which together
cover the same range in the key/ID space
@param oldBucketIndex: The index of k-bucket to split (in this table's
list of k-buckets)
@type oldBucketIndex: int
"""
# Resize the range of the current (old) k-bucket
oldBucket = self._buckets[oldBucketIndex]
splitPoint = oldBucket.rangeMax - (oldBucket.rangeMax - oldBucket.rangeMin)/2
splitPoint = oldBucket.rangeMax - (oldBucket.rangeMax - oldBucket.rangeMin) / 2
# Create a new k-bucket to cover the range split off from the old bucket
newBucket = kbucket.KBucket(splitPoint, oldBucket.rangeMax)
oldBucket.rangeMax = splitPoint
@@ -352,16 +356,18 @@ class TreeRoutingTable(RoutingTable):
for contact in newBucket._contacts:
oldBucket.removeContact(contact)
class OptimizedTreeRoutingTable(TreeRoutingTable):
""" A version of the "tree"-type routing table specified by Kademlia,
along with contact accounting optimizations specified in section 4.1 of
of the 13-page version of the Kademlia paper.
"""
def __init__(self, parentNodeID):
TreeRoutingTable.__init__(self, parentNodeID)
# Cache containing nodes eligible to replace stale k-bucket entries
self._replacementCache = {}
def addContact(self, contact):
""" Add the given contact to the correct k-bucket; if it already
exists, its status will be updated
@@ -379,7 +385,8 @@ class OptimizedTreeRoutingTable(TreeRoutingTable):
try:
self._buckets[bucketIndex].addContact(contact)
except kbucket.BucketFull:
# The bucket is full; see if it can be split (by checking if its range includes the host node's id)
# The bucket is full; see if it can be split (by checking
# if its range includes the host node's id)
if self._buckets[bucketIndex].keyInRange(self._parentNodeID):
self._splitBucket(bucketIndex)
# Retry the insertion attempt
@@ -388,22 +395,26 @@ class OptimizedTreeRoutingTable(TreeRoutingTable):
# We can't split the k-bucket
# NOTE: This implementation follows section 4.1 of the 13 page version
# of the Kademlia paper (optimized contact accounting without PINGs
#- results in much less network traffic, at the expense of some memory)
# - results in much less network traffic, at the expense of some memory)
# Put the new contact in our replacement cache for the corresponding k-bucket (or update it's position if it exists already)
if not self._replacementCache.has_key(bucketIndex):
# Put the new contact in our replacement cache for the
# corresponding k-bucket (or update it's position if
# it exists already)
if bucketIndex not in self._replacementCache:
self._replacementCache[bucketIndex] = []
if contact in self._replacementCache[bucketIndex]:
self._replacementCache[bucketIndex].remove(contact)
#TODO: Using k to limit the size of the contact replacement cache - maybe define a seperate value for this in constants.py?
elif len(self._replacementCache) >= constants.k:
self._replacementCache.pop(0)
# TODO: Using k to limit the size of the contact
# replacement cache - maybe define a separate value for
# this in constants.py?
elif len(self._replacementCache[bucketIndex]) >= constants.k:
self._replacementCache[bucketIndex].pop(0)
self._replacementCache[bucketIndex].append(contact)
def removeContact(self, contactID):
""" Remove the contact with the specified node ID from the routing
table
@param contactID: The node ID of the contact to remove
@type contactID: str
"""
@@ -411,12 +422,12 @@ class OptimizedTreeRoutingTable(TreeRoutingTable):
try:
contact = self._buckets[bucketIndex].getContact(contactID)
except ValueError:
#print 'removeContact(): Contact not in routing table'
return
contact.failedRPCs += 1
if contact.failedRPCs >= 5:
if contact.failedRPCs >= 5:
self._buckets[bucketIndex].removeContact(contactID)
# Replace this stale contact with one from our replacemnent cache, if we have any
if self._replacementCache.has_key(bucketIndex):
# Replace this stale contact with one from our replacement cache, if we have any
if bucketIndex in self._replacementCache:
if len(self._replacementCache[bucketIndex]) > 0:
self._buckets[bucketIndex].addContact( self._replacementCache[bucketIndex].pop() )
self._buckets[bucketIndex].addContact(
self._replacementCache[bucketIndex].pop())
-103
View File
@@ -1,103 +0,0 @@
from lbrynet.dht.node import Node
import binascii
from twisted.internet import reactor, task
import logging
import sys
from lbrynet.core.utils import generate_id
log = logging.getLogger(__name__)
def print_usage():
print "Usage:\n%s UDP_PORT KNOWN_NODE_IP KNOWN_NODE_PORT HASH"
def join_network(udp_port, known_nodes):
lbryid = generate_id()
log.info('Creating Node...')
node = Node(udpPort=udp_port, lbryid=lbryid)
log.info('Joining network...')
d = node.joinNetwork(known_nodes)
def log_network_size():
log.info("Approximate number of nodes in DHT: %s", str(node.getApproximateTotalDHTNodes()))
log.info("Approximate number of blobs in DHT: %s", str(node.getApproximateTotalHashes()))
d.addCallback(lambda _: log_network_size())
d.addCallback(lambda _: node)
return d
def get_hosts(node, h):
def print_hosts(hosts):
print "Hosts returned from the DHT: "
print hosts
log.info("Looking up %s", h)
d = node.getPeersForBlob(h)
d.addCallback(print_hosts)
return d
def announce_hash(node, h):
d = node.announceHaveBlob(h, 34567)
def log_results(results):
for success, result in results:
if success:
log.info("Succeeded: %s", str(result))
else:
log.info("Failed: %s", str(result.getErrorMessage()))
d.addCallback(log_results)
return d
def get_args():
if len(sys.argv) < 5:
print_usage()
sys.exit(1)
udp_port = int(sys.argv[1])
known_nodes = [(sys.argv[2], int(sys.argv[3]))]
h = binascii.unhexlify(sys.argv[4])
return udp_port, known_nodes, h
def run_dht_script(dht_func):
log_format = "(%(asctime)s)[%(filename)s:%(lineno)s] %(funcName)s(): %(message)s"
logging.basicConfig(level=logging.DEBUG, format=log_format)
udp_port, known_nodes, h = get_args()
d = task.deferLater(reactor, 0, join_network, udp_port, known_nodes)
def run_dht_func(node):
return dht_func(node, h)
d.addCallback(run_dht_func)
def log_err(err):
log.error("An error occurred: %s", err.getTraceback())
return err
def shut_down():
log.info("Shutting down")
reactor.stop()
d.addErrback(log_err)
d.addBoth(lambda _: shut_down())
reactor.run()
def get_hosts_for_hash_in_dht():
run_dht_script(get_hosts)
def announce_hash_to_dht():
run_dht_script(announce_hash)
+130 -58
View File
@@ -36,11 +36,11 @@ class IRequestSender(Interface):
"""
def add_blob_request(self, blob_request):
"""
Add a request for a blob to the next message that will be sent to the peer.
"""Add a request for a blob to the next message that will be sent to the peer.
This will cause the protocol to call blob_request.write(data) for all incoming
data, after the response message has been parsed out, until blob_request.finished_deferred fires.
This will cause the protocol to call blob_request.write(data)
for all incoming data, after the response message has been
parsed out, until blob_request.finished_deferred fires.
@param blob_request: the request for the blob
@type blob_request: ClientBlobRequest
@@ -56,8 +56,7 @@ class IRequestCreator(Interface):
"""
def send_next_request(self, peer, protocol):
"""
Create a Request object for the peer and then give the protocol that request.
"""Create a Request object for the peer and then give the protocol that request.
@param peer: the Peer object which the request will be sent to.
@type peer: Peer
@@ -65,7 +64,8 @@ class IRequestCreator(Interface):
@param protocol: the protocol to pass the request to.
@type protocol: object which implements IRequestSender
@return: Deferred object which will callback with True or False depending on whether a Request was sent
@return: Deferred object which will callback with True or
False depending on whether a Request was sent
@rtype: Deferred which fires with boolean
"""
@@ -83,12 +83,12 @@ class IMetadataHandler(Interface):
Get metadata for the IDownloadManager.
"""
def get_initial_blobs(self):
"""
Return metadata about blobs that are known to be associated with the stream at the time that the
stream is set up.
"""Return metadata about blobs that are known to be associated with
the stream at the time that the stream is set up.
@return: Deferred object which will call back with a list of BlobInfo objects
@rtype: Deferred which fires with [BlobInfo]
"""
def final_blob_num(self):
@@ -156,11 +156,12 @@ class IDownloadManager(Interface):
"""
def needed_blobs(self):
"""
Returns a list of BlobInfos representing all of the blobs that the stream still needs to download.
"""Returns a list of BlobInfos representing all of the blobs that the
stream still needs to download.
@return: the list of BlobInfos representing blobs that the stream still needs to download.
@rtype: [BlobInfo]
"""
def final_blob_num(self):
@@ -172,14 +173,15 @@ class IDownloadManager(Interface):
"""
def handle_blob(self, blob_num):
"""
This function is called when the next blob in the stream is ready to be handled, whatever that may mean.
"""This function is called when the next blob in the stream is ready
to be handled, whatever that may mean.
@param blob_num: The blob_num of the blob that is ready to be handled.
@type blob_num: integer
@return: A Deferred which fires when the blob has been 'handled'
@rtype: Deferred which can fire with anything
"""
@@ -188,8 +190,8 @@ class IConnectionManager(Interface):
Connects to peers so that IRequestCreators can send their requests.
"""
def get_next_request(self, peer, protocol):
"""
Ask all IRequestCreators belonging to this object to create a Request for peer and give it to protocol
"""Ask all IRequestCreators belonging to this object to create a
Request for peer and give it to protocol
@param peer: the peer which the request will be sent to.
@type peer: Peer
@@ -197,9 +199,11 @@ class IConnectionManager(Interface):
@param protocol: the protocol which the request should be sent to by the IRequestCreator.
@type protocol: IRequestSender
@return: Deferred object which will callback with True or False depending on whether the IRequestSender
should send the request or hang up
@return: Deferred object which will callback with True or
False depending on whether the IRequestSender should send
the request or hang up
@rtype: Deferred which fires with boolean
"""
def protocol_disconnected(self, peer, protocol):
@@ -217,11 +221,12 @@ class IConnectionManager(Interface):
class IProgressManager(Interface):
"""
Responsible for keeping track of the progress of the download.
"""Responsible for keeping track of the progress of the download.
Specifically, it is their responsibility to decide which blobs
need to be downloaded and keep track of the progress of the
download
Specifically, it is their responsibility to decide which blobs need to be downloaded and keep track of
the progress of the download
"""
def stream_position(self):
"""
@@ -235,11 +240,12 @@ class IProgressManager(Interface):
"""
def needed_blobs(self):
"""
Returns a list of BlobInfos representing all of the blobs that the stream still needs to download.
"""Returns a list of BlobInfos representing all of the blobs that the
stream still needs to download.
@return: the list of BlobInfos representing blobs that the stream still needs to download.
@rtype: [BlobInfo]
"""
def blob_downloaded(self, blob, blob_info):
@@ -334,24 +340,26 @@ class IRateLimiter(Interface):
"""
def register_protocol(self, protocol):
"""
Register an IRateLimited object with the IRateLimiter so that the IRateLimiter can throttle it
"""Register an IRateLimited object with the IRateLimiter so that the
IRateLimiter can throttle it
@param protocol: An object implementing the interface IRateLimited
@type protocol: Object implementing IRateLimited
@return: None
"""
def unregister_protocol(self, protocol):
"""
Unregister an IRateLimited object so that it won't be throttled any more.
"""Unregister an IRateLimited object so that it won't be throttled any more.
@param protocol: An object implementing the interface IRateLimited, which was previously registered with this
@param protocol: An object implementing the interface
IRateLimited, which was previously registered with this
IRateLimiter via "register_protocol"
@type protocol: Object implementing IRateLimited
@return: None
"""
@@ -360,11 +368,11 @@ class IRequestHandler(Interface):
Pass client queries on to IQueryHandlers
"""
def register_query_handler(self, query_handler, query_identifiers):
"""
Register a query handler, which will be passed any queries that
"""Register a query handler, which will be passed any queries that
match any of the identifiers in query_identifiers
@param query_handler: the object which will handle queries matching the given query_identifiers
@param query_handler: the object which will handle queries
matching the given query_identifiers
@type query_handler: Object implementing IQueryHandler
@param query_identifiers: A list of strings representing the query identifiers
@@ -372,6 +380,7 @@ class IRequestHandler(Interface):
@type query_identifiers: [string]
@return: None
"""
def register_blob_sender(self, blob_sender):
@@ -462,13 +471,14 @@ class IStreamDownloaderOptions(Interface):
class IStreamDownloaderFactory(Interface):
"""
Construct IStreamDownloaders and provide options that will be passed to those IStreamDownloaders.
"""Construct IStreamDownloaders and provide options that will be
passed to those IStreamDownloaders.
"""
def can_download(self, sd_validator, payment_rate_manager):
"""
Decide whether the downloaders created by this factory can download the stream described by sd_validator
"""Decide whether the downloaders created by this factory can
download the stream described by sd_validator
@param sd_validator: object containing stream metadata
@type sd_validator: object which implements IStreamDescriptorValidator interface
@@ -478,13 +488,14 @@ class IStreamDownloaderFactory(Interface):
@return: True if the downloaders can download the stream, False otherwise
@rtype: bool
"""
def make_downloader(self, sd_validator, options, payment_rate_manager):
"""
Create an object that implements the IStreamDownloader interface
"""Create an object that implements the IStreamDownloader interface
@param sd_validator: object containing stream metadata which will be given to the IStreamDownloader
@param sd_validator: object containing stream metadata which
will be given to the IStreamDownloader
@type sd_validator: object which implements IStreamDescriptorValidator interface
@param options: a list of values that will be used by the IStreamDownloaderFactory to
@@ -497,6 +508,7 @@ class IStreamDownloaderFactory(Interface):
@return: a Deferred which fires with the downloader object
@rtype: Deferred which fires with IStreamDownloader
"""
def get_description(self):
@@ -513,12 +525,12 @@ class IStreamDownloader(Interface):
Use metadata and data from the network for some useful purpose.
"""
def start(self):
"""
start downloading the stream
"""start downloading the stream
@return: a Deferred which fires when the stream is finished downloading, or errbacks when the stream is
cancelled.
@return: a Deferred which fires when the stream is finished
downloading, or errbacks when the stream is cancelled.
@rtype: Deferred which fires with anything
"""
def insufficient_funds(self, err):
@@ -542,26 +554,28 @@ class IStreamDescriptorValidator(Interface):
def info_to_show(self):
"""
@return: A list of tuples representing metadata that should be presented to the user before starting the
download
@return: A list of tuples representing metadata that should be
presented to the user before starting the download
@rtype: [(string, string)]
"""
class ILBRYWallet(Interface):
"""
Send and receive payments.
class IWallet(Interface):
"""Send and receive payments.
To send a payment, a payment reservation must be obtained first. This guarantees that a payment
isn't promised if it can't be paid. When the service in question is rendered, the payment
reservation must be given to the ILBRYWallet along with the final price. The reservation can also
be canceled.
To send a payment, a payment reservation must be obtained
first. This guarantees that a payment isn't promised if it can't
be paid. When the service in question is rendered, the payment
reservation must be given to the IWallet along with the final
price. The reservation can also be canceled.
"""
def stop(self):
"""
Send out any unsent payments, close any connections, and stop checking for incoming payments.
"""Send out any unsent payments, close any connections, and stop
checking for incoming payments.
@return: None
"""
def start(self):
@@ -590,8 +604,8 @@ class ILBRYWallet(Interface):
"""
def reserve_points(self, peer, amount):
"""
Ensure a certain amount of points are available to be sent as payment, before the service is rendered
"""Ensure a certain amount of points are available to be sent as
payment, before the service is rendered
@param peer: The peer to which the payment will ultimately be sent
@type peer: Peer
@@ -599,8 +613,10 @@ class ILBRYWallet(Interface):
@param amount: The amount of points to reserve
@type amount: float
@return: A ReservedPoints object which is given to send_points once the service has been rendered
@return: A ReservedPoints object which is given to send_points
once the service has been rendered
@rtype: ReservedPoints
"""
def cancel_point_reservation(self, reserved_points):
@@ -647,4 +663,60 @@ class ILBRYWallet(Interface):
@type amount: float
@return: None
"""
"""
class IBlobPriceModel(Interface):
"""
A blob price model
Used by INegotiationStrategy classes
"""
def calculate_price(self, blob):
"""
Calculate the price for a blob
@param blob: a blob hash
@type blob: str
@return: blob price target
@type: Decimal
"""
class INegotiationStrategy(Interface):
"""
Strategy to negotiate download payment rates
"""
def make_offer(self, peer, blobs):
"""
Make a rate offer for the given peer and blobs
@param peer: peer to make an offer to
@type: str
@param blobs: blob hashes to make an offer for
@type: list
@return: rate offer
@rtype: Offer
"""
def respond_to_offer(self, offer, peer, blobs):
"""
Respond to a rate offer given by a peer
@param offer: offer to reply to
@type: Offer
@param peer: peer to make an offer to
@type: str
@param blobs: blob hashes to make an offer for
@type: list
@return: accepted, rejected, or unset offer
@rtype: Offer
"""
@@ -4,15 +4,15 @@ import os
from twisted.internet import defer
from twisted.python.failure import Failure
from twisted.enterprise import adbapi
from lbrynet.core.Error import DuplicateStreamHashError, NoSuchStreamHashError
from lbrynet.core.Error import DuplicateStreamHashError, NoSuchStreamHash, NoSuchSDHash
from lbrynet.core.sqlite_helpers import rerun_if_locked
log = logging.getLogger(__name__)
class DBLBRYFileMetadataManager(object):
"""Store and provide access to LBRY file metadata using leveldb files"""
class DBEncryptedFileMetadataManager(object):
"""Store and provide access to LBRY file metadata using sqlite"""
def __init__(self, db_dir):
self.db_dir = db_dir
@@ -47,8 +47,9 @@ class DBLBRYFileMetadataManager(object):
def add_blobs_to_stream(self, stream_hash, blobs):
return self._add_blobs_to_stream(stream_hash, blobs, ignore_duplicate_error=True)
def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
log.info("Getting blobs for a stream. Count is %s", str(count))
def get_blobs_for_stream(self, stream_hash, start_blob=None,
end_blob=None, count=None, reverse=False):
log.debug("Getting blobs for stream %s. Count is %s", stream_hash, count)
def get_positions_of_start_and_end():
if start_blob is not None:
@@ -92,13 +93,18 @@ class DBLBRYFileMetadataManager(object):
def get_sd_blob_hashes_for_stream(self, stream_hash):
return self._get_sd_blob_hashes_for_stream(stream_hash)
def get_stream_hash_for_sd_hash(self, sd_hash):
return self._get_stream_hash_for_sd_blob_hash(sd_hash)
def _open_db(self):
# check_same_thread=False is solely to quiet a spurious error that appears to be due
# to a bug in twisted, where the connection is closed by a different thread than the
# one that opened it. The individual connections in the pool are not used in multiple
# threads.
self.db_conn = adbapi.ConnectionPool("sqlite3", (os.path.join(self.db_dir, "lbryfile_info.db")),
check_same_thread=False)
self.db_conn = adbapi.ConnectionPool(
"sqlite3",
(os.path.join(self.db_dir, "lbryfile_info.db")),
check_same_thread=False)
def create_tables(transaction):
transaction.execute("create table if not exists lbry_files (" +
@@ -125,8 +131,10 @@ class DBLBRYFileMetadataManager(object):
@rerun_if_locked
def _delete_stream(self, stream_hash):
d = self.db_conn.runQuery("select stream_hash from lbry_files where stream_hash = ?", (stream_hash,))
d.addCallback(lambda result: result[0][0] if len(result) else Failure(NoSuchStreamHashError(stream_hash)))
d = self.db_conn.runQuery(
"select stream_hash from lbry_files where stream_hash = ?", (stream_hash,))
d.addCallback(
lambda result: result[0][0] if result else Failure(NoSuchStreamHash(stream_hash)))
def do_delete(transaction, s_h):
transaction.execute("delete from lbry_files where stream_hash = ?", (s_h,))
@@ -157,21 +165,30 @@ class DBLBRYFileMetadataManager(object):
@rerun_if_locked
def _get_stream_info(self, stream_hash):
d = self.db_conn.runQuery("select key, stream_name, suggested_file_name from lbry_files where stream_hash = ?",
(stream_hash,))
d.addCallback(lambda result: result[0] if len(result) else Failure(NoSuchStreamHashError(stream_hash)))
def get_result(res):
if res:
return res[0]
else:
raise NoSuchStreamHash(stream_hash)
d = self.db_conn.runQuery(
"select key, stream_name, suggested_file_name from lbry_files where stream_hash = ?",
(stream_hash,))
d.addCallback(get_result)
return d
@rerun_if_locked
def _check_if_stream_exists(self, stream_hash):
d = self.db_conn.runQuery("select stream_hash from lbry_files where stream_hash = ?", (stream_hash,))
d = self.db_conn.runQuery(
"select stream_hash from lbry_files where stream_hash = ?", (stream_hash,))
d.addCallback(lambda r: True if len(r) else False)
return d
@rerun_if_locked
def _get_blob_num_by_hash(self, stream_hash, blob_hash):
d = self.db_conn.runQuery("select position from lbry_file_blobs where stream_hash = ? and blob_hash = ?",
(stream_hash, blob_hash))
d = self.db_conn.runQuery(
"select position from lbry_file_blobs where stream_hash = ? and blob_hash = ?",
(stream_hash, blob_hash))
d.addCallback(lambda r: r[0][0] if len(r) else None)
return d
@@ -236,14 +253,29 @@ class DBLBRYFileMetadataManager(object):
@rerun_if_locked
def _get_sd_blob_hashes_for_stream(self, stream_hash):
log.info("Looking up sd blob hashes for stream hash %s", str(stream_hash))
d = self.db_conn.runQuery("select sd_blob_hash from lbry_file_descriptors where stream_hash = ?",
(stream_hash,))
log.debug("Looking up sd blob hashes for stream hash %s", str(stream_hash))
d = self.db_conn.runQuery(
"select sd_blob_hash from lbry_file_descriptors where stream_hash = ?",
(stream_hash,))
d.addCallback(lambda results: [r[0] for r in results])
return d
@rerun_if_locked
def _get_stream_hash_for_sd_blob_hash(self, sd_blob_hash):
def _handle_result(result):
if not result:
raise NoSuchSDHash(sd_blob_hash)
return result[0][0]
class TempLBRYFileMetadataManager(object):
log.debug("Looking up sd blob hashes for sd blob hash %s", str(sd_blob_hash))
d = self.db_conn.runQuery(
"select stream_hash from lbry_file_descriptors where sd_blob_hash = ?",
(sd_blob_hash,))
d.addCallback(_handle_result)
return d
class TempEncryptedFileMetadataManager(object):
def __init__(self):
self.streams = {}
self.stream_blobs = {}
@@ -291,7 +323,8 @@ class TempLBRYFileMetadataManager(object):
self.stream_blobs[(stream_hash, blob.blob_hash)] = info
return defer.succeed(True)
def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
def get_blobs_for_stream(self, stream_hash, start_blob=None,
end_blob=None, count=None, reverse=False):
if start_blob is not None:
start_num = self._get_blob_num_by_hash(stream_hash, start_blob)
@@ -333,4 +366,5 @@ class TempLBRYFileMetadataManager(object):
return defer.succeed(True)
def get_sd_blob_hashes_for_stream(self, stream_hash):
return defer.succeed([sd_hash for sd_hash, s_h in self.sd_files.iteritems() if stream_hash == s_h])
return defer.succeed(
[sd_hash for sd_hash, s_h in self.sd_files.iteritems() if stream_hash == s_h])
+11 -16
View File
@@ -11,7 +11,7 @@ import os
log = logging.getLogger(__name__)
LBRYFileStreamType = "lbryfile"
EncryptedFileStreamType = "lbryfile"
def save_sd_info(stream_info_manager, sd_info, ignore_duplicate=False):
@@ -50,7 +50,7 @@ def get_sd_info(stream_info_manager, stream_hash, include_blobs):
def format_info(stream_info):
fields = {}
fields['stream_type'] = LBRYFileStreamType
fields['stream_type'] = EncryptedFileStreamType
fields['stream_name'] = stream_info[1]
fields['key'] = stream_info[0]
fields['suggested_file_name'] = stream_info[2]
@@ -80,19 +80,13 @@ def get_sd_info(stream_info_manager, stream_hash, include_blobs):
return d
@defer.inlineCallbacks
def publish_sd_blob(stream_info_manager, blob_manager, stream_hash):
descriptor_writer = BlobStreamDescriptorWriter(blob_manager)
d = get_sd_info(stream_info_manager, stream_hash, True)
d.addCallback(descriptor_writer.create_descriptor)
def add_sd_blob_to_stream(sd_blob_hash):
d = stream_info_manager.save_sd_blob_hash_to_stream(stream_hash, sd_blob_hash)
d.addCallback(lambda _: sd_blob_hash)
return d
d.addCallback(add_sd_blob_to_stream)
return d
sd_info = yield get_sd_info(stream_info_manager, stream_hash, True)
sd_blob_hash = yield descriptor_writer.create_descriptor(sd_info)
yield stream_info_manager.save_sd_blob_hash_to_stream(stream_hash, sd_blob_hash)
defer.returnValue(sd_blob_hash)
def create_plain_sd(stream_info_manager, stream_hash, file_name, overwrite_existing=False):
@@ -121,7 +115,7 @@ def create_plain_sd(stream_info_manager, stream_hash, file_name, overwrite_exist
return d
class LBRYFileStreamDescriptorValidator(object):
class EncryptedFileStreamDescriptorValidator(object):
def __init__(self, raw_info):
self.raw_info = raw_info
@@ -137,7 +131,8 @@ class LBRYFileStreamDescriptorValidator(object):
raise InvalidStreamDescriptorError("Missing '%s'" % (e.args[0]))
for c in hex_suggested_file_name:
if c not in '0123456789abcdef':
raise InvalidStreamDescriptorError("Suggested file name is not a hex-encoded string")
raise InvalidStreamDescriptorError(
"Suggested file name is not a hex-encoded string")
h = get_lbry_hash_obj()
h.update(hex_stream_name)
h.update(key)
@@ -187,4 +182,4 @@ class LBRYFileStreamDescriptorValidator(object):
size_so_far = 0
for blob_info in self.raw_info.get("blobs", []):
size_so_far += int(blob_info['length'])
return size_so_far
return size_so_far
+2
View File
@@ -0,0 +1,2 @@
from lbrynet.lbryfile.StreamDescriptor import get_sd_info
from lbrynet.lbryfile.StreamDescriptor import publish_sd_blob
@@ -8,7 +8,7 @@ from lbrynet.cryptstream.client.CryptStreamDownloader import CryptStreamDownload
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.StreamDescriptor import StreamMetadata
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbryfile.client.LBRYFileMetadataHandler import LBRYFileMetadataHandler
from lbrynet.lbryfile.client.EncryptedFileMetadataHandler import EncryptedFileMetadataHandler
import os
from twisted.internet import defer, threads, reactor
from twisted.python.procutils import which
@@ -19,13 +19,13 @@ import traceback
log = logging.getLogger(__name__)
class LBRYFileDownloader(CryptStreamDownloader):
class EncryptedFileDownloader(CryptStreamDownloader):
"""Classes which inherit from this class download LBRY files"""
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, upload_allowed):
stream_info_manager, payment_rate_manager, wallet):
CryptStreamDownloader.__init__(self, peer_finder, rate_limiter, blob_manager,
payment_rate_manager, wallet, upload_allowed)
payment_rate_manager, wallet)
self.stream_hash = stream_hash
self.stream_info_manager = stream_info_manager
self.suggested_file_name = None
@@ -77,7 +77,8 @@ class LBRYFileDownloader(CryptStreamDownloader):
return d
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager)
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager, download_manager)
def _start(self):
d = self._setup_output()
@@ -120,10 +121,11 @@ class LBRYFileDownloader(CryptStreamDownloader):
return 0
def _get_metadata_handler(self, download_manager):
return LBRYFileMetadataHandler(self.stream_hash, self.stream_info_manager, download_manager)
return EncryptedFileMetadataHandler(self.stream_hash,
self.stream_info_manager, download_manager)
class LBRYFileDownloaderFactory(object):
class EncryptedFileDownloaderFactory(object):
implements(IStreamDownloaderFactory)
def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager,
@@ -138,12 +140,14 @@ class LBRYFileDownloaderFactory(object):
return True
def make_downloader(self, metadata, options, payment_rate_manager, **kwargs):
payment_rate_manager.min_blob_data_payment_rate = options[0]
upload_allowed = options[1]
assert len(options) == 1
data_rate = options[0]
payment_rate_manager.min_blob_data_payment_rate = data_rate
def save_source_if_blob(stream_hash):
if metadata.metadata_source == StreamMetadata.FROM_BLOB:
d = self.stream_info_manager.save_sd_blob_hash_to_stream(stream_hash, metadata.source_blob_hash)
d = self.stream_info_manager.save_sd_blob_hash_to_stream(
stream_hash, metadata.source_blob_hash)
else:
d = defer.succeed(True)
d.addCallback(lambda _: stream_hash)
@@ -151,7 +155,7 @@ class LBRYFileDownloaderFactory(object):
def create_downloader(stream_hash):
downloader = self._make_downloader(stream_hash, payment_rate_manager,
metadata.validator.raw_info, upload_allowed)
metadata.validator.raw_info)
d = downloader.set_stream_info()
d.addCallback(lambda _: downloader)
return d
@@ -161,15 +165,17 @@ class LBRYFileDownloaderFactory(object):
d.addCallback(create_downloader)
return d
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info):
pass
class LBRYFileSaver(LBRYFileDownloader):
class EncryptedFileSaver(EncryptedFileDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, download_directory, upload_allowed, file_name=None):
LBRYFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, upload_allowed)
payment_rate_manager, wallet, download_directory, file_name=None):
EncryptedFileDownloader.__init__(self, stream_hash,
peer_finder, rate_limiter,
blob_manager, stream_info_manager,
payment_rate_manager, wallet)
self.download_directory = download_directory
self.file_name = file_name
self.file_written_to = None
@@ -182,7 +188,7 @@ class LBRYFileSaver(LBRYFileDownloader):
return str(self.file_name)
def set_stream_info(self):
d = LBRYFileDownloader.set_stream_info(self)
d = EncryptedFileDownloader.set_stream_info(self)
def set_file_name():
if self.file_name is None:
@@ -195,13 +201,14 @@ class LBRYFileSaver(LBRYFileDownloader):
return d
def stop(self, err=None):
d = LBRYFileDownloader.stop(self, err=err)
d = EncryptedFileDownloader.stop(self, err=err)
d.addCallback(lambda _: self._delete_from_info_manager())
return d
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager,
delete_blob_after_finished=not self.upload_allowed)
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager,
download_manager)
def _setup_output(self):
def open_file():
@@ -230,9 +237,10 @@ class LBRYFileSaver(LBRYFileDownloader):
self.file_written_to = os.path.join(self.download_directory, file_name)
except IOError:
log.error(traceback.format_exc())
raise ValueError("Failed to open %s. Make sure you have permission to save files to that"
" location." % str(os.path.join(self.download_directory,
file_name)))
raise ValueError(
"Failed to open %s. Make sure you have permission to save files to that"
" location." %
os.path.join(self.download_directory, file_name))
return threads.deferToThread(open_file)
def _close_output(self):
@@ -257,39 +265,45 @@ class LBRYFileSaver(LBRYFileDownloader):
return self.stream_info_manager.delete_stream(self.stream_hash)
class LBRYFileSaverFactory(LBRYFileDownloaderFactory):
class EncryptedFileSaverFactory(EncryptedFileDownloaderFactory):
def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager,
wallet, download_directory):
LBRYFileDownloaderFactory.__init__(self, peer_finder, rate_limiter, blob_manager,
EncryptedFileDownloaderFactory.__init__(self, peer_finder, rate_limiter, blob_manager,
stream_info_manager, wallet)
self.download_directory = download_directory
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
return LBRYFileSaver(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
self.stream_info_manager, payment_rate_manager, self.wallet,
self.download_directory, upload_allowed)
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info):
return EncryptedFileSaver(stream_hash, self.peer_finder,
self.rate_limiter, self.blob_manager,
self.stream_info_manager,
payment_rate_manager, self.wallet,
self.download_directory)
@staticmethod
def get_description():
return "Save"
class LBRYFileOpener(LBRYFileDownloader):
class EncryptedFileOpener(EncryptedFileDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, upload_allowed):
LBRYFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, upload_allowed)
payment_rate_manager, wallet):
EncryptedFileDownloader.__init__(self, stream_hash,
peer_finder, rate_limiter,
blob_manager, stream_info_manager,
payment_rate_manager, wallet,
)
self.process = None
self.process_log = None
def stop(self, err=None):
d = LBRYFileDownloader.stop(self, err=err)
d = EncryptedFileDownloader.stop(self, err=err)
d.addCallback(lambda _: self._delete_from_info_manager())
return d
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager,
delete_blob_after_finished=not self.upload_allowed)
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager,
download_manager)
def _setup_output(self):
def start_process():
@@ -333,7 +347,7 @@ class LBRYFileOpener(LBRYFileDownloader):
return self.stream_info_manager.delete_stream(self.stream_hash)
class LBRYFileOpenerFactory(LBRYFileDownloaderFactory):
class EncryptedFileOpenerFactory(EncryptedFileDownloaderFactory):
def can_download(self, sd_validator):
if which('vlc'):
return True
@@ -345,10 +359,13 @@ class LBRYFileOpenerFactory(LBRYFileDownloaderFactory):
return True
return False
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
return LBRYFileOpener(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
self.stream_info_manager, payment_rate_manager, self.wallet, upload_allowed)
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info):
return EncryptedFileOpener(stream_hash, self.peer_finder,
self.rate_limiter, self.blob_manager,
self.stream_info_manager,
payment_rate_manager, self.wallet,
)
@staticmethod
def get_description():
return "Stream"
return "Stream"
@@ -7,7 +7,7 @@ from lbrynet.interfaces import IMetadataHandler
log = logging.getLogger(__name__)
class LBRYFileMetadataHandler(object):
class EncryptedFileMetadataHandler(object):
implements(IMetadataHandler)
def __init__(self, stream_hash, stream_info_manager, download_manager):
@@ -36,4 +36,4 @@ class LBRYFileMetadataHandler(object):
else:
log.debug("Setting _final_blob_num to %s", str(blob_num - 1))
self._final_blob_num = blob_num - 1
return infos
return infos
@@ -0,0 +1,47 @@
from lbrynet.lbryfile.StreamDescriptor import EncryptedFileStreamType
from lbrynet.lbryfile.StreamDescriptor import EncryptedFileStreamDescriptorValidator
from lbrynet.core.DownloadOption import DownloadOption, DownloadOptionChoice
def add_lbry_file_to_sd_identifier(sd_identifier):
sd_identifier.add_stream_type(
EncryptedFileStreamType, EncryptedFileStreamDescriptorValidator, EncryptedFileOptions())
class EncryptedFileOptions(object):
def __init__(self):
pass
def get_downloader_options(self, sd_validator, payment_rate_manager):
prm = payment_rate_manager
def get_default_data_rate_description():
if prm.base.min_blob_data_payment_rate is None:
return "Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate)
else:
return "%f LBC/MB" % prm.base.min_blob_data_payment_rate
rate_choices = []
rate_choices.append(DownloadOptionChoice(
prm.base.min_blob_data_payment_rate,
"No change - %s" % get_default_data_rate_description(),
"No change - %s" % get_default_data_rate_description()))
if prm.base.min_blob_data_payment_rate is not None:
rate_choices.append(DownloadOptionChoice(
None,
"Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate),
"Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate)))
rate_choices.append(DownloadOptionChoice(float,
"Enter rate in LBC/MB",
"Enter rate in LBC/MB"))
options = [
DownloadOption(
rate_choices,
"Rate which will be paid for data",
"data payment rate",
prm.base.min_blob_data_payment_rate,
get_default_data_rate_description()
),
]
return options
@@ -1,55 +0,0 @@
from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamType, LBRYFileStreamDescriptorValidator
from lbrynet.core.DownloadOption import DownloadOption, DownloadOptionChoice
def add_lbry_file_to_sd_identifier(sd_identifier):
sd_identifier.add_stream_type(LBRYFileStreamType, LBRYFileStreamDescriptorValidator, LBRYFileOptions())
class LBRYFileOptions(object):
def __init__(self):
pass
def get_downloader_options(self, sd_validator, payment_rate_manager):
prm = payment_rate_manager
def get_default_data_rate_description():
if prm.min_blob_data_payment_rate is None:
return "Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate)
else:
return "%f LBC/MB" % prm.min_blob_data_payment_rate
rate_choices = []
rate_choices.append(DownloadOptionChoice(prm.min_blob_data_payment_rate,
"No change - %s" % get_default_data_rate_description(),
"No change - %s" % get_default_data_rate_description()))
if prm.min_blob_data_payment_rate is not None:
rate_choices.append(DownloadOptionChoice(None,
"Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate),
"Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate)))
rate_choices.append(DownloadOptionChoice(float,
"Enter rate in LBC/MB",
"Enter rate in LBC/MB"))
options = [
DownloadOption(
rate_choices,
"Rate which will be paid for data",
"data payment rate",
prm.min_blob_data_payment_rate,
get_default_data_rate_description()
),
DownloadOption(
[
DownloadOptionChoice(bool,
None,
None,
bool_options_description=("Allow", "Disallow")),
],
"Allow reuploading data downloaded for this file",
"allow upload",
True,
"Allow"
),
]
return options
@@ -11,24 +11,20 @@ from lbrynet import conf
from lbrynet.lbryfile.StreamDescriptor import get_sd_info
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from twisted.protocols.basic import FileSender
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
log = logging.getLogger(__name__)
class LBRYFileStreamCreator(CryptStreamCreator):
class EncryptedFileStreamCreator(CryptStreamCreator):
"""
A CryptStreamCreator which adds itself and its additional metadata to an LBRYFileManager
A CryptStreamCreator which adds itself and its additional metadata to an EncryptedFileManager
"""
def __init__(self, blob_manager, lbry_file_manager, name=None,
key=None, iv_generator=None, suggested_file_name=None):
CryptStreamCreator.__init__(self, blob_manager, name, key, iv_generator)
self.lbry_file_manager = lbry_file_manager
if suggested_file_name is None:
self.suggested_file_name = name
else:
self.suggested_file_name = suggested_file_name
self.suggested_file_name = suggested_file_name or name
self.stream_hash = None
self.blob_infos = []
@@ -38,9 +34,9 @@ class LBRYFileStreamCreator(CryptStreamCreator):
def _save_stream_info(self):
stream_info_manager = self.lbry_file_manager.stream_info_manager
d = stream_info_manager.save_stream(self.stream_hash, binascii.hexlify(self.name),
binascii.hexlify(self.key),
binascii.hexlify(self.suggested_file_name),
d = stream_info_manager.save_stream(self.stream_hash, hexlify(self.name),
hexlify(self.key),
hexlify(self.suggested_file_name),
self.blob_infos)
return d
@@ -69,9 +65,9 @@ class LBRYFileStreamCreator(CryptStreamCreator):
def _make_stream_hash(self):
hashsum = get_lbry_hash_obj()
hashsum.update(binascii.hexlify(self.name))
hashsum.update(binascii.hexlify(self.key))
hashsum.update(binascii.hexlify(self.suggested_file_name))
hashsum.update(hexlify(self.name))
hashsum.update(hexlify(self.key))
hashsum.update(hexlify(self.suggested_file_name))
hashsum.update(self._get_blobs_hashsum())
self.stream_hash = hashsum.hexdigest()
@@ -81,10 +77,15 @@ class LBRYFileStreamCreator(CryptStreamCreator):
return d
# TODO: this should be run its own thread. Encrypting a large file can
# be very cpu intensive and there is no need to run that on the
# main reactor thread. The FileSender mechanism that is used is
# great when sending over the network, but this is all local so
# we can simply read the file from the disk without needing to
# involve reactor.
def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=None,
iv_generator=None, suggested_file_name=None):
"""
Turn a plain file into an LBRY File.
"""Turn a plain file into an LBRY File.
An LBRY File is a collection of encrypted blobs of data and the metadata that binds them
together which, when decrypted and put back together according to the metadata, results
@@ -94,11 +95,11 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
into chunks and encrypted, and then a stream descriptor file with the stream parameters
and other metadata is written to disk.
@param session: An LBRYSession object.
@type session: LBRYSession
@param session: An Session object.
@type session: Session
@param lbry_file_manager: The LBRYFileManager object this LBRY File will be added to.
@type lbry_file_manager: LBRYFileManager
@param lbry_file_manager: The EncryptedFileManager object this LBRY File will be added to.
@type lbry_file_manager: EncryptedFileManager
@param file_name: The path to the plain file.
@type file_name: string
@@ -114,8 +115,8 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
be generated.
@type key: string
@param iv_generator: a generator which yields initialization vectors for the blobs. Will be called
once for each blob.
@param iv_generator: a generator which yields initialization
vectors for the blobs. Will be called once for each blob.
@type iv_generator: a generator function which yields strings
@param suggested_file_name: what the file should be called when the LBRY File is saved to disk.
@@ -131,7 +132,8 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
def make_stream_desc_file(stream_hash):
log.debug("creating the stream descriptor file")
descriptor_file_path = os.path.join(session.db_dir, file_name + conf.CRYPTSD_FILE_EXTENSION)
descriptor_file_path = os.path.join(
session.db_dir, file_name + conf.settings['CRYPTSD_FILE_EXTENSION'])
descriptor_writer = PlainStreamDescriptorWriter(descriptor_file_path)
d = get_sd_info(lbry_file_manager.stream_info_manager, stream_hash, True)
@@ -142,10 +144,18 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
base_file_name = os.path.basename(file_name)
lbry_file_creator = LBRYFileStreamCreator(session.blob_manager, lbry_file_manager, base_file_name,
key, iv_generator, suggested_file_name)
lbry_file_creator = EncryptedFileStreamCreator(
session.blob_manager,
lbry_file_manager,
base_file_name, key,
iv_generator,
suggested_file_name)
def start_stream():
# TODO: Using FileSender isn't necessary, we can just read
# straight from the disk. The stream creation process
# should be in its own thread anyway so we don't need to
# worry about interacting with the twisted reactor
file_sender = FileSender()
d = file_sender.beginFileTransfer(file_handle, lbry_file_creator)
d.addCallback(lambda _: stop_file(lbry_file_creator))
@@ -155,4 +165,12 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
d = lbry_file_creator.setup()
d.addCallback(lambda _: start_stream())
return d
return d
def hexlify(str_or_unicode):
if isinstance(str_or_unicode, unicode):
strng = str_or_unicode.encode('utf-8')
else:
strng = str_or_unicode
return binascii.hexlify(strng)
@@ -0,0 +1,191 @@
"""
Download LBRY Files from LBRYnet and save them to disk.
"""
import logging
from zope.interface import implements
from twisted.internet import defer
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.Error import NoSuchSDHash, NoSuchStreamHash
from lbrynet.core.utils import short_hash
from lbrynet.core.StreamDescriptor import StreamMetadata
from lbrynet.lbryfile.client.EncryptedFileDownloader import EncryptedFileSaver
from lbrynet.lbryfile.client.EncryptedFileDownloader import EncryptedFileDownloader
from lbrynet.lbryfilemanager.EncryptedFileStatusReport import EncryptedFileStatusReport
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbryfile.StreamDescriptor import save_sd_info
from lbrynet.core.Wallet import ClaimOutpoint
log = logging.getLogger(__name__)
def log_status(name, sd_hash, status):
if status == ManagedEncryptedFileDownloader.STATUS_RUNNING:
status_string = "running"
elif status == ManagedEncryptedFileDownloader.STATUS_STOPPED:
status_string = "stopped"
elif status == ManagedEncryptedFileDownloader.STATUS_FINISHED:
status_string = "finished"
else:
status_string = "unknown"
log.info("lbry://%s (%s) is %s", name, short_hash(sd_hash), status_string)
class ManagedEncryptedFileDownloader(EncryptedFileSaver):
STATUS_RUNNING = "running"
STATUS_STOPPED = "stopped"
STATUS_FINISHED = "finished"
def __init__(self, rowid, stream_hash, peer_finder, rate_limiter,
blob_manager, stream_info_manager, lbry_file_manager,
payment_rate_manager, wallet, download_directory,
file_name=None):
EncryptedFileSaver.__init__(self, stream_hash, peer_finder,
rate_limiter, blob_manager,
stream_info_manager,
payment_rate_manager, wallet,
download_directory,
file_name)
self.sd_hash = None
self.txid = None
self.nout = None
self.outpoint = None
self.name = None
self.claim_id = None
self.rowid = rowid
self.lbry_file_manager = lbry_file_manager
self._saving_status = False
@property
def saving_status(self):
return self._saving_status
@defer.inlineCallbacks
def restore(self):
yield self.load_file_attributes()
status = yield self.lbry_file_manager.get_lbry_file_status(self)
log_status(self.name, self.sd_hash, status)
if status == ManagedEncryptedFileDownloader.STATUS_RUNNING:
# start returns self.finished_deferred
# which fires when we've finished downloading the file
# and we don't want to wait for the entire download
self.start()
elif status == ManagedEncryptedFileDownloader.STATUS_STOPPED:
defer.returnValue(False)
elif status == ManagedEncryptedFileDownloader.STATUS_FINISHED:
self.completed = True
defer.returnValue(True)
else:
raise Exception("Unknown status for stream %s: %s", self.stream_hash, status)
@defer.inlineCallbacks
def stop(self, err=None, change_status=True):
log.debug('Stopping download for %s', short_hash(self.sd_hash))
# EncryptedFileSaver deletes metadata when it's stopped. We don't want that here.
yield EncryptedFileDownloader.stop(self, err=err)
if change_status is True:
status = yield self._save_status()
defer.returnValue(status)
@defer.inlineCallbacks
def status(self):
blobs = yield self.stream_info_manager.get_blobs_for_stream(self.stream_hash)
blob_hashes = [b[0] for b in blobs if b[0] is not None]
completed_blobs = yield self.blob_manager.completed_blobs(blob_hashes)
num_blobs_completed = len(completed_blobs)
num_blobs_known = len(blob_hashes)
if self.completed:
status = "completed"
elif self.stopped:
status = "stopped"
else:
status = "running"
defer.returnValue(EncryptedFileStatusReport(self.file_name, num_blobs_completed,
num_blobs_known, status))
@defer.inlineCallbacks
def load_file_attributes(self):
sd_hash = yield self.stream_info_manager.get_sd_blob_hashes_for_stream(self.stream_hash)
if sd_hash:
self.sd_hash = sd_hash[0]
else:
raise NoSuchStreamHash(self.stream_hash)
stream_metadata = yield self.wallet.get_claim_metadata_for_sd_hash(self.sd_hash)
if stream_metadata:
name, txid, nout = stream_metadata
self.name = name
self.txid = txid
self.nout = nout
self.outpoint = ClaimOutpoint(self.txid, self.nout)
else:
raise NoSuchSDHash(self.sd_hash)
self.claim_id = yield self.wallet.get_claimid(self.name, self.txid, self.nout)
defer.returnValue(None)
@defer.inlineCallbacks
def _start(self):
yield EncryptedFileSaver._start(self)
yield self.load_file_attributes()
status = yield self._save_status()
log_status(self.name, self.sd_hash, status)
defer.returnValue(status)
def _get_finished_deferred_callback_value(self):
if self.completed is True:
return "Download successful"
else:
return "Download stopped"
@defer.inlineCallbacks
def _save_status(self):
self._saving_status = True
if self.completed is True:
status = ManagedEncryptedFileDownloader.STATUS_FINISHED
elif self.stopped is True:
status = ManagedEncryptedFileDownloader.STATUS_STOPPED
else:
status = ManagedEncryptedFileDownloader.STATUS_RUNNING
status = yield self.lbry_file_manager.change_lbry_file_status(self, status)
self._saving_status = False
defer.returnValue(status)
def save_status(self):
return self._save_status()
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading,
self.blob_manager, download_manager)
class ManagedEncryptedFileDownloaderFactory(object):
implements(IStreamDownloaderFactory)
def __init__(self, lbry_file_manager):
self.lbry_file_manager = lbry_file_manager
def can_download(self, sd_validator):
# TODO: add a sd_validator for non live streams, use it
return True
@defer.inlineCallbacks
def make_downloader(self, metadata, options, payment_rate_manager, download_directory=None,
file_name=None):
assert len(options) == 1
data_rate = options[0]
stream_hash = yield save_sd_info(self.lbry_file_manager.stream_info_manager,
metadata.validator.raw_info)
if metadata.metadata_source == StreamMetadata.FROM_BLOB:
yield self.lbry_file_manager.save_sd_blob_hash_to_stream(stream_hash,
metadata.source_blob_hash)
lbry_file = yield self.lbry_file_manager.add_lbry_file(stream_hash, payment_rate_manager,
data_rate,
download_directory, file_name)
defer.returnValue(lbry_file)
@staticmethod
def get_description():
return "Save the file to disk"
Loaded 100 of 292 files, more files were not shown because too many files have changed in this diff. Show more