Compare commits

...
807 Commits
Author SHA1 Message Date
Job Evers-Meltzer 9baf261cb3 Bump version: 0.7.1 → 0.7.2 2016-10-30 14:47:21 -05:00
Job Evers-Meltzer f03a15d503 remove unneeded 'after_first_run' check 2016-10-30 14:46:16 -05:00
Job Evers-Meltzer 1a70a20c8b Bump version: 0.7.0 → 0.7.1 2016-10-30 10:57:41 -05:00
Job Evers-Meltzer 359cb83fab change windows log file location 2016-10-30 10:46:58 -05:00
Job Evers-Meltzer 0893858385 Bump version: 0.6.10 → 0.7.0 2016-10-28 18:03:40 -05:00
Job Evers-Meltzer e1997a6852 dumb bug fix 2016-10-28 17:55:34 -05:00
Job Evers‐Meltzer 0f4fee475f Merge pull request #201 from lbryio/reflect-my-unavailable-streams
Upload unavailable streams to reflector on startup
2016-10-28 17:29:28 -05:00
Job Evers-Meltzer 864e963dea Merge remote-tracking branch 'origin/master' into reflect-my-unavailable-streams
Conflicts:
	lbrynet/core/log_support.py
2016-10-28 17:16:39 -05:00
Job Evers‐Meltzer 3d0debd775 Merge pull request #218 from lbryio/misc-fixes
various fixes
2016-10-28 17:12:07 -05:00
Job Evers‐Meltzer 6cc2b48a06 Merge pull request #217 from lbryio/safer-conversion
For settings: On conversion failure, fallback to default values
2016-10-28 16:57:40 -05:00
Job Evers-Meltzer dee0917642 various fixes 2016-10-28 16:57:25 -05:00
Job Evers-Meltzer 395a68b261 For settings: On conversion failure, fallback to default values 2016-10-28 16:29:16 -05:00
Job Evers‐Meltzer 0d066b2d11 Merge pull request #213 from lbryio/log-errors-on-resolve-name
Log error when resolve name fails
2016-10-28 16:25:19 -05:00
Job Evers-Meltzer 353ede26b8 Log error when resolve name fails 2016-10-28 16:03:58 -05:00
Job Evers‐Meltzer 46cfa20f65 Merge pull request #204 from lbryio/cleanup-logging
Cleanup logging
2016-10-28 15:56:54 -05:00
Job Evers‐Meltzer 30b1dd3ce6 Merge pull request #210 from lbryio/app-startup
make app startup consistent across versions
2016-10-28 15:55:09 -05:00
Jack da601818a3 return from restore_status properly 2016-10-28 16:47:45 -04:00
Job Evers-Meltzer 6fa624ec8a switch LogUploader to use an already encoded lbryid 2016-10-28 15:44:24 -05:00
Job Evers-Meltzer 779ba8e214 Fix missing conf imports 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 80f548fdfd use same logging for apps 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 6f9ec2fa66 remove repeated log code 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 40437e4a82 log cleanup 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 2ee6fac014 redo logging configuration for lbrynet-daemon script 2016-10-28 15:43:04 -05:00
Job Evers-Meltzer 8fe5eb514a make app startup consistent across versions 2016-10-28 15:41:48 -05:00
Jack c4312cc387 Merge branch 'master' into reflect-my-unavailable-streams 2016-10-28 16:05:16 -04:00
Jack 110a9b1e0f whoops 2016-10-28 15:56:22 -04:00
Jack 035a1cf758 feedback from job 2016-10-28 15:48:51 -04:00
Jack d4785849e1 remove unused imports 2016-10-28 14:22:37 -04:00
Job Evers-Meltzer b41756e38a Bump version: 0.6.9 → 0.6.10 2016-10-28 12:29:15 -05:00
Job Evers‐Meltzer 60ee278519 Merge pull request #215 from lbryio/delete_running
delete RUNNING.md
2016-10-28 11:10:54 -05:00
Alex Grintsvayg b49e44926a this is very out of date 2016-10-28 12:01:10 -04:00
Job Evers‐Meltzer 81187da5b7 Merge pull request #211 from lbryio/refactor-settings
Refactor settings
2016-10-27 15:36:11 -05:00
Job Evers-Meltzer e70ecc8ac9 change settings object for packages too 2016-10-27 14:57:02 -05:00
Job Evers-Meltzer 862055a6de move settings into conf file 2016-10-27 14:31:27 -05:00
Job Evers-Meltzer 7076c0c870 Require env settings are in LBRY namespace 2016-10-27 11:05:21 -05:00
Job Evers-Meltzer 2cd6c644a0 Merge remote-tracking branch 'origin/master' into refactor-settings
Conflicts:
	lbrynet/lbrynet_daemon/Daemon.py
	lbrynet/lbrynet_daemon/DaemonControl.py
	lbrynet/lbrynet_daemon/DaemonServer.py
2016-10-27 10:34:48 -05:00
Job Evers-Meltzer 0edacbe4c8 Merge remote-tracking branch 'origin/master' into refactor-settings
Conflicts:
	lbrynet/conf.py
	lbrynet/lbrynet_daemon/Daemon.py
	lbrynet/lbrynet_daemon/DaemonCLI.py
	lbrynet/lbrynet_daemon/UIManager.py
	tests/functional/test_misc.py
2016-10-27 10:18:56 -05:00
Job Evers‐Meltzer 4b330d0e9f Update Installation instructions 2016-10-27 09:25:06 -05:00
Job Evers‐Meltzer 8ce08f7da1 Merge pull request #206 from lbryio/shutdown-during-setup
Shutdown during setup
2016-10-26 19:09:33 -05:00
Jack 127987f3ba upload log if lbry id is not yet set, clean up crash log 2016-10-26 17:39:47 -05:00
Job Evers-Meltzer 3c7eec9456 raise error if database is newer 2016-10-26 17:39:46 -05:00
Job Evers-Meltzer 5d45345398 shutdown if there are errors during setup 2016-10-26 17:39:43 -05:00
Jack Robison b360eb6181 Merge pull request #207 from lbryio/better-json
Use better json formatting
2016-10-26 16:32:19 -04:00
Jack 369cd516c0 move availability check and fix from ManagedEncryptedFileDownloader to ReflectorAvailabilityHelper 2016-10-26 16:17:01 -04:00
Jack 24320d6444 remove debug line 2016-10-26 03:22:45 -04:00
Jack 1951ea09cd updates from master, more refactoring
-lbrynet.lbrynet_daemon.auth.client.LBRYAPIClient.config will detect if
it needs to return the auth/non-auth version
2016-10-26 03:16:33 -04:00
Job Evers-Meltzer 4d391e7429 Use better json formatting
Apply json.dumps at the end of the formatting process instead of the
middle.  This allows for proper escaping of quotes and allows for
actual json to be emitted.
2016-10-25 17:49:35 -05:00
Job Evers‐Meltzer fb88cdb445 Merge pull request #203 from lbryio/github-bug
catch exception if github is down
2016-10-24 11:02:12 -05:00
Job Evers‐Meltzer 572f1d1f6e Merge pull request #200 from lbryio/cleanup-tests
Start to cleanup functional tests to improve readability
2016-10-24 11:01:20 -05:00
Job Evers‐Meltzer 27a8114b6f Merge pull request #197 from lbryio/add-dht-tests
Rename DHT tests such that trial automatically runs them
2016-10-24 11:00:40 -05:00
Job Evers‐Meltzer 6f3873d99b Merge pull request #178 from lbryio/track-data-upload-rebase
Track Data Uploaded and other stuff
2016-10-24 08:20:29 -07:00
Job Evers-Meltzer e62e2cc440 catch exception if github is down 2016-10-21 16:12:56 -07:00
Job Evers-Meltzer a5797cdeb3 Better handling of lbry file search
- replace lbry file search strings with constants
- refactor repeated code for deciding what search type to use
2016-10-20 16:01:54 -07:00
Jack 3dd99fdc92 upload unavailable streams to reflector on startup 2016-10-20 15:40:35 -04:00
Job Evers-Meltzer 0c2265a8da replace some strings 2016-10-20 11:30:38 -07:00
Job Evers-Meltzer 01811621a6 shorten is_valid_blobhash logic 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 229ed0d7dc rename summarize to summarize_and_reset 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 7f800ce5bb fix rebase error 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer aa291ca79a fix pylint 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 813267ae18 fix tests 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer fbf7928412 rename events to be seperate for each metric 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 7167d47631 bug fixes 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 8b1bb673c1 report available blobs 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer 417a8f719f Add looping call manager to analytics manager 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer ddb88eb1c9 move LoopingCallManager to own module 2016-10-20 11:29:58 -07:00
Job Evers-Meltzer d5f0001950 small cleanups 2016-10-20 11:29:15 -07:00
Job Evers-Meltzer 4a07b56e8a refactor analytics manager to use dependency injection 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 9da1b3c5b7 analytics manager bug fix 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer f9f07c47e2 move analytics manager 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer f5213fb77a GetFileHelper bug fixes 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 48a61605de bug fix: looping calls 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 76fb7697ff misc 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer b71a3fee43 add tracked metrics reporting to analytics 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 3a91896d8a track the amount of data uploaded 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 36ae0d5f20 remove utils file - function moved to core 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 3f727d892c refactor events; add tests 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer c104ed3f8b better time handling in tests 2016-10-20 11:28:05 -07:00
Job Evers-Meltzer 9416376a34 refactor: add GetFileHelper
Move the code related to getting a lbry file into
its own class
2016-10-20 11:26:10 -07:00
Job Evers-Meltzer 66e139c566 set development version based on environment variable 2016-10-20 11:26:10 -07:00
Job Evers-Meltzer d0a82c68df move analytics related calls into their own class 2016-10-20 11:25:17 -07:00
Job Evers-Meltzer 93e2b3e20a Extract some looping call code out of the daemon 2016-10-20 11:25:17 -07:00
Job Evers-Meltzer 2e59e5e3b8 extract individual functions 2016-10-20 11:16:36 -07:00
Jack 727ea12dbd Bump version: 0.6.8 → 0.6.9 2016-10-20 12:41:49 -04:00
Jack Robison ea3225c897 Merge pull request #198 from lbryio/fix-blobrequester
fix problems in blobrequester
2016-10-20 12:41:26 -04:00
Job Evers-Meltzer bfcfa9dc3c cleanup tests to improve readability 2016-10-20 09:30:08 -07:00
Jack ea21821558 fix issues with is_generous
-stop requesting blobs if price limit is reached and the offer still
has not been accepted

-drop max rate to old flat rate (0.005lbc/mb)
2016-10-20 12:29:55 -04:00
Jack a9ae1a1016 fix bug where support claim is returned when it shouldn't be 2016-10-20 11:15:35 -04:00
Jack 3f5efb1fa3 fix problems in blobrequester
-also move is_generous_host into a conf setting
2016-10-20 01:35:18 -04:00
Job Evers-Meltzer 9ba38c84cc Bump version: 0.6.7 → 0.6.8 2016-10-19 15:56:58 -07:00
Job Evers-Meltzer a5d3c0bbe1 rename tests such that trial automatically runs them 2016-10-19 16:38:59 -05:00
Jack Robison d4aa109b73 Merge pull request #193 from lbryio/check-remote-conn
Use lbry.io instead of google to check for internet connection
2016-10-19 15:06:33 -04:00
Jack Robison bdeccb49b8 Merge pull request #191 from lbryio/verify-published-file-exists
make publish fail when given an invalid file
2016-10-19 15:05:33 -04:00
Jack Robison 35ce7bbd61 Merge pull request #195 from lbryio/fail-appveyor
Appveyor build needs to fail when tests fail
2016-10-19 15:04:20 -04:00
Jack 293933163f move USE_AUTH_HTTP to adjustable settings 2016-10-19 14:59:10 -04:00
Jack 22fc26797d pylint fixes 2016-10-19 14:40:01 -04:00
Jack 8a95c8b7af whoops 2016-10-19 14:26:19 -04:00
Jack 33674f8cc3 remove logging in conf, rearrange platform 2016-10-19 00:39:19 -04:00
Jack 6799de645b Merge branch 'master' into refactor-settings
# Conflicts:
#	packaging/osx/lbry-osx-app/lbrygui/LBRYApp.py
#	packaging/windows/lbry-win32-app/LBRYWin32App.py
2016-10-19 00:19:45 -04:00
Jack 06a88599df --http-auth flag to use authentication, refactor settings
settings will prefer, in order:
-defaults
-settings in config file
-settings given as environmental variables
-settings given as command line args
2016-10-19 00:12:44 -04:00
Jack 03877f7ebd remove str 2016-10-18 17:11:45 -04:00
Job Evers-Meltzer 9f2dbd4291 Appveyor build needs to fail when tests fail
According to
https://www.appveyor.com/docs/build-configuration/#script-blocks-in-build-configuration
a build only fails if an exception is thrown. This is causing builds to pass even if
the tests fail. That same link suggests adding the code that is in this commit.
2016-10-18 09:44:21 -05:00
Job Evers 6b7d575e95 use lbry.io to check for an internet connection 2016-10-17 20:11:08 -05:00
Job Evers 640db41555 only need lbrycrdd_path when using lbrycrdd 2016-10-17 19:58:04 -05:00
Jack d860c2a2b4 make publish fail when given an invalid file
otherwise a less obvious error is returned in
lbrynet_daemon.Publisher.Publisher._show_publish_error
2016-10-17 14:25:36 -04:00
Jack 733bb3bccf Bump version: 0.6.6 → 0.6.7 2016-10-16 02:55:19 -04:00
Jack Robison 307751277d Merge pull request #189 from lbryio/include-jsonschema
fix jsonschema includes
2016-10-16 02:54:55 -04:00
Jack 62c2539475 include jsonschema.schemas files 2016-10-16 02:49:19 -04:00
Jack 5b15da2c55 add jsonschema to windows build 2016-10-16 00:29:32 -04:00
Jack 830f5ff6e3 Bump version: 0.6.5 → 0.6.6 2016-10-15 23:35:17 -04:00
Jack ea1f260ed9 Bump version: 0.6.4 → 0.6.5 2016-10-14 23:06:03 -04:00
Jack Robison 3e86254324 Merge pull request #188 from lbryio/fix-DaemonRequest-import
fix DaemonRequest import in apps
2016-10-14 23:04:52 -04:00
Jack b5311dd1c3 fix DaemonRequest import in apps 2016-10-14 23:01:15 -04:00
Jack a8a581d35c Merge branch 'master' into auth-jsonrpc
# Conflicts:
#	lbrynet/conf.py
#	lbrynet/core/Error.py
#	lbrynet/lbrynet_daemon/Daemon.py
#	lbrynet/lbrynet_daemon/DaemonControl.py
#	lbrynet/lbrynet_daemon/ExchangeRateManager.py
2016-10-14 18:25:37 -04:00
Job Evers-Meltzer 5f129faf32 Bump version: 0.6.3 → 0.6.4 2016-10-14 16:06:12 -05:00
Job Evers‐Meltzer 018d78be6f Merge pull request #187 from lbryio/new-metadata-system
Rewrite of migration and validation system using JSON Schema
2016-10-14 15:51:49 -05:00
Alex Liebowitz b5a3929bb5 Fix imports 2016-10-14 16:41:02 -04:00
Alex Liebowitz b67b4b2ef6 Refactor metadata upgrade version range calculation to use generator 2016-10-14 15:58:18 -04:00
Alex Liebowitz bed3f551fd Light refactor of Metadata class 2016-10-14 15:58:04 -04:00
Alex Liebowitz 5b0b8a19ff Add requirements.txt for jsonschema 2016-10-14 15:57:09 -04:00
Alex Liebowitz 988c9595c7 Add default MIME type during publication
Needed because the new JSON schema enforces that the content type is a
string (not null).
2016-10-14 10:37:19 -04:00
Alex Liebowitz 92700808eb Add back current_version attribute to Metadata 2016-10-14 10:37:19 -04:00
Alex Liebowitz 0dd9aa0d67 Update error that daemon catches for malformed metadata
Now jsonschema.ValidationError instead of AssertionError
2016-10-14 10:37:19 -04:00
Alex Liebowitz 9cf1d6b710 Tests for new JSON Schema system 2016-10-14 10:37:18 -04:00
Alex Liebowitz d8d462f43c Convert Metadata and Fee validators to use new JSON Schema-based system 2016-10-14 10:37:18 -04:00
Alex Liebowitz 3f22f39ce1 Refactor Validator to new StructuredDict class that uses JSON Schema
- Uses JSON schema for all validation (so far no custom code needed)
 - Can migrate up and down with any versioning scheme
 - Does migrations with regular dictionary operations instead of a DSL
2016-10-14 10:35:17 -04:00
Job Evers-Meltzer e647663c34 Fix error when version is None
Logs were reporting:

Failure instance: Traceback: <type 'exceptions.AttributeError'>: StrictVersion instance has no attribute 'version'
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/DaemonServer.py:121:requestReceived
twisted/web/server.pyc:183:process
twisted/web/server.pyc:234:render
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:450:render
--- <exception caught here> ---
twisted/internet/defer.pyc:150:maybeDeferred
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:1561:jsonrpc_version
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/core/utils.py:39:version_is_greater_than
distutils/version.pyc:140:__cmp__
2016-10-14 10:35:17 -04:00
Jack Robison b5987c75da Merge pull request #184 from lbryio/missing-version
Fix error when version is None
2016-10-13 15:02:49 -04:00
Jack Robison 32a973943f Merge pull request #173 from lbryio/blob-prices
Blob prices
2016-10-13 14:59:26 -04:00
Jack 99c060719a clean up test_strategy 2016-10-13 14:00:46 -04:00
Jack d9cd215a60 set seed for strategy tests 2016-10-13 13:48:18 -04:00
Jack 2d3b1fd4eb clean up test_misc 2016-10-13 13:36:22 -04:00
Jack 19c2264429 clean up 2016-10-13 13:35:55 -04:00
Job Evers-Meltzer ae4b984f0b Fix error when version is None
Logs were reporting:

Failure instance: Traceback: <type 'exceptions.AttributeError'>: StrictVersion instance has no attribute 'version'
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/DaemonServer.py:121:requestReceived
twisted/web/server.pyc:183:process
twisted/web/server.pyc:234:render
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:450:render
--- <exception caught here> ---
twisted/internet/defer.pyc:150:maybeDeferred
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/lbrynet_daemon/Daemon.py:1561:jsonrpc_version
/Volumes/LBRY/LBRY.app/Contents/Resources/lib/python2.7/lbrynet/core/utils.py:39:version_is_greater_than
distutils/version.pyc:140:__cmp__
2016-10-11 20:48:29 -05:00
Job Evers‐Meltzer 897c113db1 Merge pull request #180 from lbryio/configure-lbryum
Configure lbryum
2016-10-11 16:46:39 -05:00
Jack 50e5a59420 remove unused imports 2016-10-07 17:29:20 -04:00
alexliebowitz 8735112f70 Merge pull request #182 from lbryio/no-cache-static-file
NoCacheStaticFile
2016-10-07 17:15:35 -04:00
Jack 5a0b5dc680 Merge branch 'master' into auth-jsonrpc
# Conflicts:
#	lbrynet/lbrynet_daemon/Daemon.py
#	lbrynet/lbrynet_daemon/DaemonControl.py
2016-10-07 17:11:43 -04:00
Job Evers‐Meltzer d91e9aa910 Fix Instructions 2016-10-07 16:17:47 -04:00
Jack be1803d558 remove unused imports 2016-10-07 14:52:54 -04:00
Jack 84bbf2f76e move set_no_cache into own function 2016-10-07 14:28:38 -04:00
Jack fe7ea7c679 refactor resources and file streamer into their own files, use NoCacheStaticFile instead of static.File 2016-10-07 14:01:59 -04:00
Jack 782e197e66 Merge branch 'master' into blob-prices 2016-10-05 23:25:39 -04:00
Jack 88be425ab2 fix pylint errors 2016-10-05 23:24:22 -04:00
Jack 7025c5102a skip live transfer 2016-10-05 23:17:45 -04:00
Jack 1ee9b3c7a6 add reply_to_offer to livepaymentratemanager 2016-10-05 23:03:16 -04:00
Jack 4f60a98eb2 fix functional tests, add a few unit tests,
-add ‘generous’ parameter for NegotiatedPaymentRateManager to turn free
hosting on/off, by default set to true.
2016-10-05 22:58:34 -04:00
Job Evers-Meltzer a71e87a398 throw error when invalid wallet is specified 2016-10-04 13:59:04 -05:00
Job Evers-Meltzer 0ecfad0ae9 add configuration to LBRYumWallet 2016-10-04 13:58:44 -05:00
Job Evers-Meltzer e909ae566b add TODO 2016-10-03 16:19:58 -05:00
Jack dc2f0adb3e update LiveStreamPaymentRateManager 2016-10-03 16:54:46 -04:00
Jack 180d981ed8 fix more functional tests 2016-10-03 16:37:27 -04:00
Jack fd9bc6214a fix functional tests 2016-10-03 15:40:27 -04:00
Jack b3f3ed8026 start fixing functional tests 2016-10-03 02:59:40 -04:00
Jack 967700dc28 BlobRequestHandler and Strategy tests 2016-10-03 02:44:58 -04:00
Jack 3ad4ad50ec Bump version: 0.6.2 → 0.6.3 2016-10-02 03:33:12 -04:00
Jack Robison 7f980446db Merge pull request #137 from lbryio/html5
More reliable stream producer
2016-10-02 03:31:54 -04:00
Alex Liebowitz 04ced2e975 Fix import issues in DaemonServer.py 2016-10-02 02:50:17 -04:00
Alex Liebowitz 2253943eba Shorten time between sending chunks of video stream 2016-10-02 02:03:47 -04:00
Alex Liebowitz a6fcd5d1f2 More refactoring in LBRYStreamProducer 2016-10-01 23:35:07 -04:00
Alex Liebowitz f245822814 Refactor of LBRYStreamProducer 2016-10-01 23:35:07 -04:00
Alex Liebowitz aa3aff91d0 Refactor LBRYStreamProducer and add Content-Length header
Also fixes producer pause/unpause behavior and adds slight delay
between sending chunks
2016-10-01 23:33:51 -04:00
Alex Liebowitz 5537dd878f Add basic support for streaming partially downloaded files 2016-10-01 23:29:08 -04:00
Jack 5e4e619708 squelch loud log 2016-10-01 15:20:19 -04:00
Jack d741c705cd add some tests, don't use lighthouse for cost ests 2016-10-01 01:09:27 -04:00
Jack d83abd81f8 job's comments 2016-09-30 14:26:13 -04:00
Jack 640e000b9f fix timeouts in get_availability 2016-09-30 13:35:13 -04:00
Jack 8d961d6ce6 add get_availability 2016-09-30 13:28:01 -04:00
Jack 0a612ba36c Merge branch 'master' into blob-prices 2016-09-30 12:36:10 -04:00
Jack eb7cc8463b Bump version: 0.6.1 → 0.6.2 2016-09-30 11:26:55 -04:00
Jack 50f0430d04 fix broken windows import 2016-09-30 11:26:37 -04:00
Jack 6e5ea935f6 Bump version: 0.6.0 → 0.6.1 2016-09-30 11:18:07 -04:00
Jack c4eb3de5b2 update app imports 2016-09-30 11:17:47 -04:00
Jack f79b651b05 Bump version: 0.5.0 → 0.6.0 2016-09-30 02:16:06 -04:00
Jack c130879ef7 frontload bug
divide availability adjusted price by frontload factor, which will be
close to 1 while index is low and close to 2 when index is high - the
reduction approaches 50% as index goes up.
2016-09-30 00:36:23 -04:00
Jack 5c391f4bb4 remove unnecessary class 2016-09-30 00:12:17 -04:00
Jack b853656aa6 Merge branch 'master' into blob-prices
# Conflicts:
#	lbrynet/core/server/BlobRequestHandler.py
2016-09-30 00:11:53 -04:00
Jack Robison 8222f940b4 Merge pull request #177 from lbryio/blob-request-handler-refactor
refactor BlobRequestHandler
2016-09-29 23:52:14 -04:00
Jack 71b989b29b update from master
resolve merge conflicts:
lbrynet/core/client/BlobRequester.py
lbrynet/core/Peer.py
lbrynet/lbryfilemanager/EncryptedFileManager.py
lbrynet/lbrynet_daemon/Daemon.py
lbrynet/lbrynet_daemon/Publisher.py
2016-09-29 23:34:59 -04:00
Jack Robison 7e2f7e6f68 Merge pull request #170 from lbryio/appveyor-deploy
appveyor updates
2016-09-29 12:51:16 -04:00
Jack Robison aefb6c5dc8 Merge pull request #175 from lbryio/skip-tests
Tests requiring epoll should only run on linux
2016-09-29 12:50:44 -04:00
Jack Robison b892b81dad Merge pull request #172 from lbryio/add-venv-instr
encourage use of virtualenv
2016-09-29 12:50:05 -04:00
Jack Robison 4d63629919 Merge pull request #164 from lbryio/refactor-blob-requester
Refactor blob requester
2016-09-29 12:49:28 -04:00
Jack 4727ee0403 Merge branch 'master' into refactor-blob-requester 2016-09-29 12:48:34 -04:00
Jack Robison fc47e8074f Merge pull request #174 from lbryio/rename-LBRY
Remove LBRY Prefix
2016-09-29 12:47:50 -04:00
Jack ff7829a078 Merge branch 'master' into rename-LBRY 2016-09-29 12:47:22 -04:00
Job Evers-Meltzer d05bbea131 refactor BlobRequestHandler 2016-09-28 17:00:34 -07:00
Job Evers‐Meltzer e08a681c8f Merge pull request #176 from lbryio/start-stream-event
Start stream event
2016-09-28 12:15:56 -07:00
Job Evers-Meltzer 6a72b3683e send Download Started event 2016-09-28 09:07:25 -07:00
Job Evers-Meltzer b201fc331c simplify callback when downloading 2016-09-28 09:06:51 -07:00
Job Evers-Meltzer 27337a9243 appveyor updates
- move init and build steps into dedicated scripts
- deploy msi artifact on tag
2016-09-28 07:38:12 -07:00
Jack b671a3c2ba Merge branch 'master' into blob-prices 2016-09-27 23:59:14 -04:00
Jack 1cc6b7658c get uploads working
-add error catching in exchange rate manager
-add free data on first request with default negotiation strategy
2016-09-27 23:56:08 -04:00
Job Evers-Meltzer 12e2827286 tests requiring epoll should only run on linux 2016-09-27 16:01:47 -07:00
Job Evers-Meltzer 548d37929e more fixes 2016-09-27 12:08:32 -07:00
Job Evers-Meltzer eb783b6b6f rename File to EncryptedFile 2016-09-27 11:26:01 -07:00
Job Evers-Meltzer d1b06c3688 more fixes 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer d16b3429cd more work to fix tests 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer 1e4b5cb450 rename test files 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer a9da6e4bdf run sed on tests too 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer a099e2db6a fix errors 2016-09-27 11:18:52 -07:00
Job Evers-Meltzer 7dbeb1f2cf Remove LBRY prefix from class and module names
This is the result of running:
    grep -RlI LBRY . | xargs sed -i 's/LBRY\([A-Z]\)/\1/g'
2016-09-27 11:18:35 -07:00
Job Evers-Meltzer 550fd185a3 rename LBRYFile to EncryptedFile 2016-09-27 11:18:16 -07:00
Jack 1720cce3b0 negotiated downloads
-uploads are still underway
2016-09-27 13:52:44 -04:00
Job Evers-Meltzer 078b254d1d rename files to remove LBRY prefix 2016-09-25 18:11:23 -07:00
Job Evers-Meltzer 6057dc0804 whitespace 2016-09-25 17:18:29 -07:00
Job Evers-Meltzer 5ea7c4f016 fix typo in string format 2016-09-25 17:17:48 -07:00
Job Evers-Meltzer 4334cdace1 Refactor blob requester to better seperate out the three requests
Each blob requests has three parts
- A request to see what blobs are available
- A request to download and pay for one of those blobs
- A request to negotiate data costs

Before the refactoring all of the code to accomplish this was mixed into
one class.  They are now three seperate classes.

There is still a lot of room for improvement, but it is at least better than
it was.
2016-09-25 16:55:55 -07:00
Alex Grintsvayg 6803bfae99 add more bins to path 2016-09-24 15:48:07 -04:00
Jack c49b0fdd49 Bump version: 0.4.9 → 0.5.0 2016-09-24 01:50:14 -04:00
Jack bf3c56f0ec only run unit tests on windows 2016-09-24 01:47:37 -04:00
Jack Robison bbcecea28d Merge pull request #165 from lbryio/get-my-claim
add get_my_claim, remove is_mine from get_claim_info response
2016-09-24 01:01:51 -04:00
Jack a936782ddc fix amount formatting between get_claim_info and get_my_claim 2016-09-24 00:58:54 -04:00
Jack 7ac0fc5a24 convert amount 2016-09-24 00:49:03 -04:00
Jack e115158f86 docstring 2016-09-23 03:04:59 -04:00
Jack bf34fa39b4 unused import 2016-09-23 03:03:15 -04:00
Jack 5dd29da84f blob upload/download history and BlobPriceAndAvailabilityTracker 2016-09-23 03:02:17 -04:00
Job Evers-Meltzer 2107ffdd18 encourage use of virtualenv 2016-09-22 10:19:28 -07:00
Jack 492fc510fa move _get_my_unspent_claim to function 2016-09-21 22:10:19 -04:00
Jack Robison c75d240a65 Merge pull request #169 from lbryio/functional-tests
Enable functional tests
2016-09-21 22:05:16 -04:00
Jack dbeb6bc3bc update uri handler 2016-09-21 21:55:10 -04:00
Jack 4d79a7828c more cleaning up 2016-09-21 21:36:06 -04:00
Jack aecafbbebd clean up 2016-09-21 03:49:52 -04:00
Jack 130f9cfc4d api sessions
-user starts a httpauthsession with an api key and name
-user initializes jsonrpc hmac secret to sha256 of session id
-server sends new random hmac secret after each api call

-a user without an authenticated session will get a authorization error
2016-09-20 16:58:30 -04:00
Job Evers-Meltzer 98d02b3fce [ci skip] add newlines 2016-09-19 12:07:47 -07:00
Jack 4d52a33ee5 Bump version: 0.4.8 → 0.4.9 2016-09-16 04:12:55 -04:00
Jack 7ad916f116 fix api being blocked for /? pages 2016-09-16 04:12:13 -04:00
Jack 05b47a9180 Bump version: 0.4.7 → 0.4.8 2016-09-16 03:56:14 -04:00
Jack Robison 9a412bbc76 Merge pull request #168 from lbryio/fewer-desktop-shortcuts
no extra desktop shortcuts in windows
2016-09-16 03:50:04 -04:00
Jack Robison b3a16f95f7 Merge pull request #167 from lbryio/block-api-requests-by-referer
block api requests by referer
2016-09-16 03:49:55 -04:00
Jack 871b3be09a no extra desktop shortcuts 2016-09-16 03:42:25 -04:00
Jack 7085b1e3c3 block api requests by referer 2016-09-16 03:40:06 -04:00
Jack Robison a9e4343775 Merge pull request #166 from lbryio/daemon-settings-yml
.yml daemon settings file
2016-09-16 01:22:20 -04:00
Jack 3d4d895d9b fix windows requirement 2016-09-16 01:14:56 -04:00
Jack 3d89e3fd93 add to appveyor.yml 2016-09-16 01:08:43 -04:00
Jack a8e2dd3f2a add pyyaml to requirements.txt 2016-09-16 00:57:30 -04:00
Jack 3ae0664fa9 use prettier file format for daemon conf file 2016-09-16 00:14:25 -04:00
Jack 98b9f2cf73 Bump version: 0.4.6 → 0.4.7 2016-09-15 23:06:56 -04:00
Jack Robison 6fa65ebb9f Merge pull request #163 from lbryio/windows-cli
fix windows cli
2016-09-15 22:57:44 -04:00
Jack add163a092 move get_params to own function, add some tests 2016-09-15 22:20:00 -04:00
Jack fcfed3f441 add get_my_claim and get rid of is_mine in get_claim_info 2016-09-15 20:15:20 -04:00
Jack 7b84d9b10a handle no params 2016-09-15 15:04:42 -04:00
Jack e9a1dbca4b fix windows cli
make cli easier to use on windows by converting keyword args to json

cli now accepts commands like:

lbrynet-cli resolve_name name=one

as well as:

lbrynet-cli resolve_name ‘{“name”: “one”}’
2016-09-15 14:47:32 -04:00
Jack 145a2654d2 Bump version: 0.4.5 → 0.4.6 2016-09-15 12:36:37 -04:00
Jack Robison 9f14cdb1d7 Merge pull request #162 from lbryio/fix-appveyor
fix appveyor error
2016-09-15 12:36:02 -04:00
Jack 5cefa2e995 set default logger in __init__ 2016-09-15 12:19:49 -04:00
Jack 13cbe0fd42 fix appveyor error 2016-09-14 23:03:18 -04:00
Jack Robison c8bb0c0bc8 Merge pull request #161 from lbryio/delete-lbry-file-without-claim
Delete lbry file with abandoned claim
2016-09-14 22:41:43 -04:00
Jack 4cc941fbec make abandon_claim compatible with old function name 2016-09-14 22:36:18 -04:00
Jack b010192f8e improve logging 2016-09-14 22:28:59 -04:00
Jack 2b1af8eea2 delete lbry file with an abandoned claim 2016-09-14 22:27:57 -04:00
Jack 6ea1f6f78c block api calls from bad origins 2016-09-14 17:39:19 -04:00
Alex Grintsvayg 1215470cc1 MIT license 2016-09-13 22:10:12 -04:00
Jack Robison 94195b7018 Merge pull request #160 from lbryio/allow-retry-after-timeout
Allow download attempts after having timed out and lower the name cache time
2016-09-13 15:17:13 -04:00
Jack 3c9c178bdc fix conf update 2016-09-12 15:44:56 -04:00
Jack cc64547920 Bump version: 0.4.4 → 0.4.5 2016-09-12 15:42:48 -04:00
Jack a23f2ca956 allow more download attempts after having timed out
-also lower the name cache time
2016-09-12 15:39:19 -04:00
Jack Robison 4dacc52ecb Merge pull request #159 from lbryio/rename-abandon-name
Include unconfirmed balance and rename abandon_name
2016-09-12 14:50:45 -04:00
Jack ccaeecfd46 bug fix 2016-09-12 13:17:21 -04:00
Jack 3e45449e8c return unconfirmed balance 2016-09-12 13:16:56 -04:00
Jack f352a3c597 rename abandon_name to abandon_claim 2016-09-12 13:16:03 -04:00
Job Evers-Meltzer d9a08f0c03 import unittest 2016-09-09 23:44:07 -07:00
Job Evers-Meltzer c77f24dde3 skip instead of fix tests 2016-09-09 23:39:35 -07:00
Job Evers-Meltzer 48c31a2119 remove redundant reflector test 2016-09-09 23:37:51 -07:00
Job Evers-Meltzer 5a578d6d79 enable functional tests 2016-09-09 23:29:46 -07:00
Jack 8cdbc847d1 add get_search_servers to allowed_during_startup 2016-09-09 05:11:51 -04:00
Jack 54dbf66cb7 update secure vars for appveyor 2016-09-09 04:14:56 -04:00
Jack 177d384126 Bump version: 0.4.3 → 0.4.4 2016-09-09 04:12:42 -04:00
Jack Robison 561fe757f3 Merge pull request #152 from DaveA50/master
Fix shortcut icons
2016-09-09 04:09:48 -04:00
Jack Robison c670ab638f Merge pull request #158 from lbryio/get-search-servers
add get_search_servers
2016-09-09 04:04:51 -04:00
Jack eea5cbbafc add get_search_servers 2016-09-09 04:04:35 -04:00
Jack Robison e4c61a4a13 Merge pull request #155 from lbryio/get-block-by-height
add 'height' parameter to get_block
2016-09-09 00:57:51 -04:00
Jack 0f0afadc94 add 'height' parameter to get_block 2016-09-09 00:48:54 -04:00
Alex Grintsvayg 659563f27a fix indent 2016-09-08 13:41:11 -04:00
Alex Grintsvayg cbd8a68d0f print actual json 2016-09-08 13:40:43 -04:00
Jack Robison 54229282a4 Merge pull request #149 from lbryio/address-is-mine
Add address_is_mine() API method
2016-09-06 13:09:08 -04:00
David Amrhein 514575d0f1 Fix Shortcuts icons 2016-09-03 14:39:39 -04:00
David Amrhein 756427560c Attempt to fix CannotListenError 2016-09-02 16:04:27 -04:00
Alex Liebowitz 900eeb177a Add address_is_mine() API method 2016-09-02 01:27:30 -04:00
David Amrhein e67fa3cadc Merge remote-tracking branch 'origin/master' 2016-09-01 10:31:20 -04:00
Dave-A c7c32fd4dc Merge pull request #12 from lbryio/master
Update
2016-09-01 10:30:23 -04:00
Jack 9471748f4c Bump version: 0.4.2 → 0.4.3 2016-09-01 01:35:44 -04:00
Jack Robison 920b1868bb Merge pull request #148 from lbryio/windows-setup
sign msi on appveyor, add command line to msi, remove leveldb requirement
2016-09-01 01:25:02 -04:00
Jack Robison eed7c54aaf Merge pull request #146 from kaykurokawa/fix_lbryum_getbalance
Use new exclude_claimtrietx option in lbryum getbalance command, fix maturity use
2016-09-01 01:24:08 -04:00
Jack 36e58a33ad delete old windows setup.py 2016-09-01 01:09:38 -04:00
Jack 93e8ff749c another try 2016-09-01 00:47:38 -04:00
Jack 6cb22444ac try fixing signtool again 2016-09-01 00:40:06 -04:00
Jack 78df101473 add quotes 2016-09-01 00:29:57 -04:00
Jack 840508b8ba add signtool to the env path 2016-09-01 00:19:02 -04:00
Jack 2761254790 fix pfx decryption 2016-08-31 23:54:00 -04:00
Jack f47d174f0b fix file 2016-08-31 23:09:01 -04:00
Jack c50fbc7935 fix path 2016-08-31 23:02:07 -04:00
Jack 75f23a1723 sign msi 2016-08-31 23:00:16 -04:00
Jack f0aa31f9e8 unused import 2016-08-31 20:45:49 -04:00
Jack aaf59218da comment out migrate0to1 import 2016-08-31 20:41:29 -04:00
Jack 0203ef66ea build windows in setup.py, drop leveldb requirement 2016-08-31 20:20:07 -04:00
Jack Robison 9c66108dc5 Merge pull request #142 from lbryio/skip-dust
skip dust transactions
2016-08-30 20:29:56 -04:00
kkurokawa 558da0a027 Using new exclude_claimtrietx option in lbryum's getbalance command. Fixing incorrect use of maturity. 2016-08-29 22:33:10 -04:00
Jack 1d39ec0317 Bump version: 0.4.1 → 0.4.2 2016-08-29 16:41:48 -04:00
Jack Robison 281b6468a0 Merge pull request #145 from lbryio/appveyor
build msi in appveyor
2016-08-29 16:39:46 -04:00
Jack Robison 9ba6da1f4d set development version to false 2016-08-29 16:28:49 -04:00
Jack bde7a7fed6 use normal lighthouse servers 2016-08-29 16:27:45 -04:00
David Amrhein 7e021746e4 Merge remote-tracking branch 'origin/master' 2016-08-29 16:22:16 -04:00
Dave-A a61bb5f528 Merge pull request #11 from lbryio/master
Update
2016-08-29 16:21:55 -04:00
Jack 6a5289b2a2 add master to branches 2016-08-29 15:50:54 -04:00
Jack 081a5df82c notifications 2016-08-29 15:24:55 -04:00
Jack 3e77ec8008 add slack notifications 2016-08-29 15:21:06 -04:00
Jack b0b222afa8 add errback 2016-08-29 14:55:26 -04:00
Jack d85d6f627d Merge branch 'master' into appveyor 2016-08-29 14:48:05 -04:00
Jack 9505f93da6 skip reflector test on windows 2016-08-29 14:47:54 -04:00
Jack 1671034330 fix json bug 2016-08-29 14:33:56 -04:00
Jack 3f0ca563e4 json encode metadata just before sending it 2016-08-29 14:26:20 -04:00
Jack ffa2a5e8fd skip pylint for now 2016-08-29 13:48:45 -04:00
Jack c3633895d0 Merge branch 'master' into appveyor 2016-08-29 13:46:08 -04:00
Jack c15af0a63a Bump version: 0.4.0 → 0.4.1 2016-08-29 13:44:46 -04:00
Jack 56e2d77039 fix _send_name_claim json 2016-08-29 13:44:18 -04:00
Jack 2cf6fa14a1 pylint path 2016-08-29 00:34:24 -04:00
Jack c374725255 test_script 2016-08-28 23:11:24 -04:00
Jack babfb6b8d6 add . to appveyor file, try tests 2016-08-28 23:05:40 -04:00
Jack 2ffa8ec7d0 use yml generated from appveyor 2016-08-28 19:26:37 -04:00
Jack b516119224 build appveyor branch only 2016-08-28 19:23:16 -04:00
Jack c0e6bb97d8 remove lbrynet_gui include 2016-08-28 17:33:30 -04:00
Jack cc30223ff1 don't cd to lbry in init 2016-08-28 16:42:07 -04:00
Jack 53c8748942 try again 2016-08-28 16:20:28 -04:00
Jack b3a8966e71 fix path 2016-08-28 15:42:14 -04:00
Jack 74800a0767 download lbrynet tar.gz 2016-08-28 15:34:04 -04:00
Jack cadd1f993d clone lbryio/lbry repo in appveyor build script 2016-08-28 15:10:29 -04:00
David Amrhein d3807cb86c Python script for win registry 2016-08-28 14:33:46 -04:00
Dave-A 735e9195b5 Merge pull request #10 from lbryio/master
Update
2016-08-28 14:14:08 -04:00
Jack a6619a628e flag to use development lighthouse servers 2016-08-28 05:25:48 -04:00
Jack 456572f87c Bump version: 0.3.22 → 0.4.0 2016-08-28 05:01:49 -04:00
Jack f024b96934 allow int fee amount 2016-08-28 03:09:36 -04:00
Jack Robison 25c71bcd98 Merge pull request #140 from lbryio/reflect-blobs
reflect_all_blobs daemon command
2016-08-28 01:33:13 -04:00
Jack Robison 319e7ecc61 Merge pull request #131 from lbryio/content_type
change content-type to content_type, refactor metadata validation
2016-08-28 01:29:51 -04:00
Jack Robison d9ece5e96a Merge pull request #143 from lbryio/sandbox-displayed-files
prevent lbry files opened through the ui from making api calls
2016-08-28 01:28:56 -04:00
Jack Robison 4bfe4450b1 Merge pull request #144 from lbryio/pr/141
Add windows URI Handling
2016-08-28 01:28:12 -04:00
Jack f9f12ffb61 move uri handler to packaging/uri_handler 2016-08-28 01:10:28 -04:00
Jack 63f81d1033 prevent a lbry files opened through the ui from making api calls 2016-08-27 23:16:48 -04:00
Jack 8895d21abf skip dust transactions 2016-08-27 02:42:20 -04:00
David Amrhein 2ac1ebeaa0 Temp solution for distributing the regedit for windows URI handling 2016-08-27 01:37:25 -04:00
David Amrhein 25437602d7 Add URI handling as an arg 2016-08-27 01:32:28 -04:00
Jack Robison 8159c71950 update os x instructions 2016-08-26 22:13:04 -04:00
Jack aa66af7ba6 fix tests 2016-08-26 20:22:11 -04:00
Jack be3633622c whoops 2016-08-26 20:13:10 -04:00
Jack 31655a641c fix setup.py 2016-08-26 19:59:17 -04:00
Jack b235f6fc6b send all blobs to reflector 2016-08-26 19:58:53 -04:00
Jack Robison 76b297b173 Merge pull request #139 from lbryio/get_all_verified_blobs
Get all verified blobs
2016-08-26 19:45:49 -04:00
Jack Robison 1e5cff1d8b Merge pull request #130 from lbryio/osx-command-line-install
add os x install script to replace out of date and non existent one in lbry-setup
2016-08-26 16:08:08 -04:00
Jack Robison 4e2296d5f6 Merge pull request #115 from lbryio/get-stream-timeout
raise Exception on GetStream timeout
2016-08-26 16:07:42 -04:00
Dave-A 18310632eb Merge pull request #9 from lbryio/master
Update
2016-08-26 10:16:46 -04:00
Jack 80ed8d9869 Bump version: 0.3.21 → 0.3.22 2016-08-26 00:56:06 -04:00
Jack 2a2658d440 fix Access-Control-Allow-Origin problem 2016-08-26 00:50:31 -04:00
Jack 710e549c03 add get_all_verified_blobs 2016-08-26 00:32:33 -04:00
Jack 47fae977dd fix Access-Control-Allow-Origin problem 2016-08-25 23:39:05 -04:00
Alex Grintsvayg 0111919975 jack fixed the unicode bug here 2016-08-25 19:09:55 -04:00
Jack 80837cb040 Merge branch 'master' into content_type
fix conflicts:
	lbrynet/lbrynet_daemon/LBRYPublisher.py
2016-08-25 17:59:20 -04:00
Jack 46153a8991 second shot at pylint error 2016-08-23 02:25:32 -04:00
Jack a9fccfd4ce fix pylint error 2016-08-23 02:17:32 -04:00
Dave-A dfb57e9438 Merge pull request #8 from lbryio/master
Update version
2016-08-23 02:17:29 -04:00
Jack eda2731702 Bump version: 0.3.20 -> 0.3.21 2016-08-23 01:59:50 -04:00
Jack Robison f47eac5935 Merge pull request #136 from DaveA50/master
Add Windows support
2016-08-23 01:53:50 -04:00
David Amrhein 3bb4dece5a Use main lbry repo for gmpy wheel url 2016-08-23 01:26:35 -04:00
David Amrhein 0504e231a1 Use main lbryum repo 2016-08-23 01:16:04 -04:00
David Amrhein 6097e14617 Make systray thread a daemon to close on api.stop() 2016-08-23 01:05:07 -04:00
David Amrhein 0b149ce5f1 Remove old osx imports 2016-08-22 21:21:56 -04:00
Dave-A 79384e7bf8 Merge pull request #7 from lbryio/master
Update
2016-08-22 20:02:35 -04:00
Jack Robison 01eb79cac2 Merge pull request #135 from lbryio/lbrycrdwallet-updateclaim-fix
LBRYcrdWallet update claim fix
2016-08-22 19:58:28 -04:00
Jack 128b32c62c even less noisy logs 2016-08-22 19:56:42 -04:00
David Amrhein c1233bd26b Make 'rb' file mode windows specific 2016-08-22 19:55:08 -04:00
David Amrhein e23f67def6 File handle for publishing on windows needs to handle binary data as 'rb' 2016-08-22 19:24:57 -04:00
Jack 95f4b29be5 noisy log 2016-08-22 18:59:17 -04:00
Jack 5727c708cf fix get_transaction
previously get_tx_json was only in LBRYumWallet
2016-08-22 18:43:52 -04:00
Jack e49f0f99a1 LBRYcrdWallet update claim fix
-fix log line that could raise an exception
-json encode value sent to lbrycrd-cli updateclaim
2016-08-22 16:57:22 -04:00
David Amrhein 84d340382b Fix for icon path when running shortcut 2016-08-21 18:58:50 -04:00
David Amrhein 5bfb067d0f Rename to just Open 2016-08-21 18:49:41 -04:00
David Amrhein d8cb62a92c Run win32 as systray app 2016-08-21 18:44:16 -04:00
Dave-A aab8cc40f9 Merge pull request #6 from lbryio/master
Update from Origin
2016-08-21 14:13:59 -04:00
Jack 9c82689f65 Bump version: 0.3.19 -> 0.3.20 2016-08-21 04:04:11 -04:00
Jack Robison ce2755467d Merge pull request #133 from sonatagreen/patch-1
spelling fix
2016-08-21 03:44:10 -04:00
Jack Robison 02b19402ae Merge pull request #116 from lbryio/reflector
Backend for pushing newly created files to a designated server
2016-08-21 03:43:03 -04:00
Jack 80af66e852 Merge branch 'master' into reflector
# Conflicts:
#	lbrynet/core/log_support.py
2016-08-21 01:12:29 -04:00
Jack 15cfa45646 remove duplicate function 2016-08-21 00:58:25 -04:00
Jack Robison ac63849b49 Merge pull request #132 from lbryio/fix-get-history
fix get_history in lbrycrdwallet
2016-08-21 00:07:41 -04:00
Jack Robison 02142e352b Merge pull request #127 from lbryio/disable-more-noisy-loggers
reduce logging to sub fire hazard levels
2016-08-21 00:07:19 -04:00
Jack 3d92413372 debug logging 2016-08-20 23:47:41 -04:00
Jack 5022ed2acc blob_completed before moving to next one
otherwise blob files download to the blobfiles directory, but they are
never recorded in blobs.db
2016-08-20 23:47:12 -04:00
Sonata Green 31b9d22649 spelling fix 2016-08-19 21:12:02 -05:00
Jack 7a54894015 debugging stuff 2016-08-19 02:41:23 -04:00
Jack 7100345a09 Merge branch 'master' into reflector 2016-08-18 22:42:27 -04:00
Jack 929db15214 fix get_history in lbrycrdwallet 2016-08-18 22:15:49 -04:00
Jack d9455926c5 Merge branch 'master' into content_type 2016-08-18 21:56:55 -04:00
David Amrhein ba5cd741e1 Testing copying instead of moving upload file to avoid access error 2016-08-18 10:45:03 -04:00
Dave-A 8c94b6243e Merge pull request #5 from lbryio/master
Update to 0.3.19
2016-08-18 10:38:07 -04:00
Jack f6473e3383 Bump version: 0.3.18 -> 0.3.19 2016-08-18 06:41:17 -04:00
Jack Robison 3cd9d6a604 Merge pull request #129 from kimihiro64/patch-2
Proper no-cache for web-ui index.html
2016-08-18 06:39:11 -04:00
Jack 142281777e stop heartbeat looping call on shutdown 2016-08-18 06:37:57 -04:00
Jack 8e7b8af180 call blob_completed when is_validated is true 2016-08-18 06:23:27 -04:00
Jack c046cd02d5 call blob_completed after receiving blob 2016-08-18 05:58:13 -04:00
Jack f5508fcdf1 typo 2016-08-18 05:36:17 -04:00
Jack adc2eab6da log request dict 2016-08-18 05:36:01 -04:00
Jack 702698d8aa stop heartbeat looping call on shutdown 2016-08-18 05:25:23 -04:00
David Amrhein 73dbe1de37 Fix relative pathing for cacert for win builds 2016-08-18 01:25:48 -04:00
Jack c5706fa28b fix imports in tests 2016-08-18 00:38:38 -04:00
Jack c906a9571e add service identity 2016-08-18 00:31:05 -04:00
Jack 038712f431 install script 2016-08-18 00:30:57 -04:00
David Amrhein ed2d2c6f86 Ignore msi_data shortcuts, which didn't work anyway 2016-08-18 00:13:31 -04:00
kimihiro64 e7158e5a4b Fix editor spacing again 2016-08-17 23:02:38 -05:00
kimihiro64 c66df259a6 fix editor spacing 2016-08-17 23:02:00 -05:00
kimihiro64 0b8437fa12 Proper no-cache for web-ui index.html 2016-08-17 22:34:20 -05:00
Jack a32b6ebf82 add reflect jsonrpc command 2016-08-17 21:33:41 -04:00
Jack edbda1696e logging fire hazard 2016-08-17 20:24:48 -04:00
Jack ece218785c reduce logging to sub fire hazard levels 2016-08-17 20:20:03 -04:00
Jack cc1b3609a1 Merge branch 'master' into reflector
# Conflicts:
#	lbrynet/__init__.py
2016-08-17 19:01:38 -04:00
Alex Grintsvayg 0841aad07a fix double-tab bug on linux 2016-08-17 14:30:56 -04:00
David Amrhein 30b04ed0c2 Change naming conventions 2016-08-17 14:25:02 -04:00
David Amrhein 90b28b3dbd Pass WindowsError when moving uploaded file 2016-08-17 14:15:55 -04:00
Jack 0f7ca8f24e move metadata and validator classes into their own directory 2016-08-17 12:28:15 -04:00
David Amrhein 726d85ebd0 Remove blindrepeater.yapsy-plugin from build 2016-08-17 12:13:47 -04:00
David Amrhein c524de9198 Update protobuf out of beta 2016-08-17 12:12:28 -04:00
Dave-A 2fbf03d8ca Merge pull request #4 from lbryio/master
Update from lbry
2016-08-17 12:03:08 -04:00
Jack 68b8112af0 Merge branch 'master' into content_type 2016-08-17 03:04:14 -04:00
Jack 975e83a015 Bump version: 0.3.17 -> 0.3.18 2016-08-17 02:39:48 -04:00
Jack 1ddb02dbb7 speed up search
get results for get_est_cost from lighthouse instead of manually trying
unless force=True
2016-08-17 02:38:59 -04:00
Jack Robison 1159204ed8 Merge pull request #126 from lbryio/pretty-dmg
Make a pretty dmg
2016-08-17 01:36:44 -04:00
Jack Robison a5212e7741 Merge pull request #121 from lbryio/save-settings
Save settings
2016-08-17 01:34:50 -04:00
Jack Robison 1932fd72e3 Merge pull request #123 from lbryio/add-heartbeat
Add basic analytics api and heartbeat event
2016-08-17 01:34:31 -04:00
Jack Robison c061fe091b Merge pull request #122 from lbryio/add-debug-logging
Add debug logging
2016-08-17 01:33:07 -04:00
Jack Robison 3621983a4d Merge pull request #119 from lbryio/update-protobuf
Update protobuf
2016-08-17 01:31:48 -04:00
Dave-A b6656690ca Merge pull request #3 from lbryio/master
Update from lbry
2016-08-16 21:13:24 -04:00
David Amrhein 0c9e82ee6b Update appveyor to DaveA50 git repo links 2016-08-16 20:04:11 -04:00
David Amrhein b8ebb1faf1 Convert to lbryum github repo 2016-08-16 19:58:47 -04:00
David Amrhein 690575193f Add gmpy windows wheel 2016-08-16 18:01:18 -04:00
David Amrhein 9e9cf63c90 Add appveyor.yml build script 2016-08-16 17:54:36 -04:00
Job Evers-Meltzer f5dc213e53 need to not be in build dir 2016-08-16 15:16:55 -05:00
Job Evers-Meltzer 138e9c1486 throw more things at wall 2016-08-16 15:03:41 -05:00
Job Evers-Meltzer 7053c0dc93 argh. so much hate 2016-08-16 14:52:08 -05:00
Job Evers-Meltzer ee8a653b13 argh. so much hate 2016-08-16 14:43:49 -05:00
Job Evers-Meltzer 71fad7e97f also need wheel 2016-08-16 13:41:04 -05:00
Job Evers-Meltzer 04740eb928 need to detect if ON_TRAVIS earlier 2016-08-16 13:24:38 -05:00
Job Evers-Meltzer 8c63d58c96 tmp: debug travis 2016-08-16 13:21:38 -05:00
Job Evers-Meltzer 991672d423 move pyobj building earlier 2016-08-16 13:16:25 -05:00
Job Evers-Meltzer 55ce38cc87 temp: debug travis 2016-08-16 12:52:55 -05:00
Job Evers‐Meltzer 96f3390407 Merge pull request #120 from lbryio/defer-later
replace callLater with deferLater
2016-08-16 12:36:43 -05:00
Job Evers-Meltzer aa2426d4e4 ignore DS_Store files 2016-08-16 12:26:19 -05:00
Job Evers-Meltzer d0e3e13852 ignore osx build artifacts 2016-08-16 12:25:35 -05:00
Job Evers-Meltzer a2a1e30d68 remove unnecessary libgmp 2016-08-16 12:25:18 -05:00
Job Evers-Meltzer a6f08b9e0c Prettifies the osx dmg
Adds a new background image, along with precisely located icon files.
Also updates the icon files to have white backgrounds. The icon files were
created using the 256px icon used in the ubuntu build.
2016-08-16 12:22:14 -05:00
David Amrhein e70842a38c Better fix for finding protobuf namespace 2016-08-15 23:09:46 -04:00
David Amrhein 2af98d7d42 Add loggly and request futures to includes 2016-08-15 20:48:27 -04:00
David Amrhein f28eee34e8 Merge remote-tracking branch 'origin/master' 2016-08-15 20:31:59 -04:00
Dave-A 208610b645 Merge pull request #2 from lbryio/master
Update from base
2016-08-15 20:31:31 -04:00
David Amrhein e38f8076bc Create site-package protobuf __init__ if needed 2016-08-15 19:40:32 -04:00
David Amrhein 6180e1a48b Update to build and distribute >v0.3.12 2016-08-15 19:19:26 -04:00
Jack e8fe1cd14d more test fixes 2016-08-15 18:39:09 -04:00
Jack 75d1fa585c fix tests 2016-08-15 18:30:14 -04:00
Jack 072ff4b02e fix tests 2016-08-15 18:13:36 -04:00
Jack d8c4e2a72e whoops 2016-08-15 17:54:43 -04:00
Jack 66396de6a3 remove as_json and serialize 2016-08-15 17:41:38 -04:00
Jack 1ca9f575ef decorator prettiness 2016-08-15 17:07:50 -04:00
Jack 272e074a15 change content-type to content_type, make changing metadata easier 2016-08-14 01:00:49 -04:00
Job Evers 9a738c3783 Add basic analytics api and heartbeat event 2016-08-12 13:37:26 -05:00
Jack 6b4519e959 squelch very verbose dht error log 2016-08-12 10:46:05 -05:00
Job Evers-Meltzer 9b1d5213a6 improve logging 2016-08-12 10:44:37 -05:00
Job Evers-Meltzer f9ed9a4505 fixup logging some more 2016-08-12 10:42:31 -05:00
Job Evers-Meltzer 3ed7ce5630 Ensure settings are actually committed
Without committing or closing the db, settings were not being saved.

Also, adds a run_in_thread decorator to simplify the code.
2016-08-12 10:08:21 -05:00
Job Evers 63cd8343c4 replace callLater with deferLater 2016-08-11 13:41:42 -05:00
Jack 2d5f8aed24 whitespace 2016-08-11 12:38:10 -04:00
Jack b8d23d0965 add announce_all_blobs_to_dht 2016-08-11 12:36:13 -04:00
Job Evers-Meltzer f4ef92e653 fixup logging some more 2016-08-11 05:37:45 +00:00
Job Evers-Meltzer c7430f4ae9 better more logging on the server side 2016-08-11 05:25:45 +00:00
Job Evers-Meltzer 623fda3087 improve logging 2016-08-11 05:16:50 +00:00
Job Evers-Meltzer 80f27f2b29 Revert "more logging on the server side"
This reverts commit f83daa5a22.
2016-08-11 05:11:18 +00:00
Jack defbd73b4f squelch very verbose dht error log 2016-08-11 01:07:51 -04:00
Jack 5285982515 resolve reflector ip 2016-08-11 01:07:51 -04:00
Job Evers-Meltzer f83daa5a22 more logging on the server side 2016-08-11 04:45:25 +00:00
Job Evers-Meltzer 0a07c8f131 log on closed/lost connection 2016-08-10 19:39:28 -05:00
Job Evers-Meltzer 35481a92f5 actually cleanup patched time.time 2016-08-10 19:36:52 -05:00
Job Evers-Meltzer 0995d864e8 connectTCP wants an ip address, not a url 2016-08-10 19:09:20 -05:00
Job Evers-Meltzer 1a42e43573 log all the things 2016-08-10 19:06:26 -05:00
Jack 132aa569fe move run_reflector_server into settings file and add debug lines 2016-08-10 16:51:46 -04:00
Job Evers-Meltzer e93a787bd6 update protobuf out of beta 2016-08-10 09:28:13 -05:00
Job Evers-Meltzer d6f902653f put import in alphabetical order 2016-08-10 08:35:15 -05:00
Job Evers-Meltzer d172d43ddf add reflector functional test to travis 2016-08-10 08:29:44 -05:00
Job Evers-Meltzer 67909724e7 fix up import paths 2016-08-10 07:44:41 -05:00
Jack 1c88cbc5f6 Merge branch 'master' into reflector 2016-08-10 02:31:28 -04:00
Jack 3fb4831804 raise exception when curr_pos > 100 and json fails to decode 2016-08-10 02:30:41 -04:00
Jack b65570fb16 drop connection on response longer than 100 bytes that can't' be decoded 2016-08-09 19:53:13 -04:00
Jack 871e6e6f64 remove files that didn't fix test 2016-08-09 17:53:34 -04:00
Jack 0cd92a96c0 add reflector files to tests directory 2016-08-09 17:46:25 -04:00
Jack ff31ddd2d5 Bump version: 0.3.16 -> 0.3.17 2016-08-09 14:40:18 -04:00
Jack e9f5b43785 fix exceptions 2016-08-09 14:29:59 -04:00
Jack 36b41081ab Bump version: 0.3.15 -> 0.3.16 2016-08-09 12:41:11 -04:00
Jack Robison 1092313892 Merge pull request #118 from lbryio/add-LBRYDaemonRequest
add LBRYDaemonRequest to os x app
2016-08-09 12:35:08 -04:00
Jack Robison ce605cb8f3 Merge pull request #117 from lbryio/update-lbrycrdd
use most recent lbrycrdd release on github
2016-08-09 12:32:40 -04:00
Jack 0733d885fb start reflector upload in publish 2016-08-09 12:26:34 -04:00
Jack ef6fe3d1d9 add variable to enable running reflector server 2016-08-09 12:18:46 -04:00
Jack 82a4fea81a fix log statement 2016-08-09 12:12:40 -04:00
Jack 14a0252819 hooking up reflector plumbing 2016-08-09 12:07:26 -04:00
Job Evers-Meltzer 849aeb65f0 use most recent lbrycrdd release on github 2016-08-09 11:03:40 -05:00
Jack a99dc6e93d add LBRYDaemonRequest to os x app 2016-08-09 11:19:01 -04:00
Alex Liebowitz 1a5fcc1348 Add requests_futures dependency 2016-08-09 07:59:27 -04:00
Jack c9d88f5628 Bump version: 0.3.14 -> 0.3.15 2016-08-09 03:29:51 -04:00
Jack Robison 0910711aca Merge pull request #111 from lbryio/update-claim
claim updates
2016-08-09 03:29:12 -04:00
Jack 6a1a9ef6bc encode claim id to be shown in log 2016-08-09 03:25:20 -04:00
Jack ef1c059079 clean up log line 2016-08-09 03:18:44 -04:00
Jack a05ab37373 Merge branch 'master' into update-claim
# Conflicts:
#	lbrynet/lbrynet_daemon/LBRYDaemon.py
2016-08-09 03:09:06 -04:00
Jack 7ea88cfd22 crudely fix loggly import problem 2016-08-09 03:01:33 -04:00
Jack Robison 8d14fe6a90 Merge pull request #113 from lbryio/log-to-loggly
Log to loggly
2016-08-09 02:30:55 -04:00
Jack Robison 49a05b42f9 Merge pull request #114 from lbryio/no-unused-imports
Remove all unused imports; add check to pylint
2016-08-09 02:30:49 -04:00
Jack b6da1d7769 decode and reverse claim id for updateclaim in lbryum 2016-08-09 01:26:55 -04:00
Jimmy Kiselak 7e2ad58edd get reflector client and server working; add func test to prove it 2016-08-09 00:59:50 -04:00
Jack 7ad0d98404 Merge branch 'master' into update-claim 2016-08-08 21:39:41 -04:00
Jack b7052bba95 rename variable 2016-08-08 21:34:05 -04:00
Jack 4a5618bcc4 get update claim working with lbryumwallet 2016-08-08 21:32:39 -04:00
Job Evers-Meltzer c1850fe604 raise Exception on GetStream timeout 2016-08-08 16:26:33 -05:00
Job Evers-Meltzer fccdf32c9f Remove all unused imports; add check to pylint 2016-08-08 15:42:35 -05:00
Jack c474411e1b delete bad records that got into name_metadata 2016-08-08 16:35:55 -04:00
Job Evers-Meltzer 9033eac5b1 Remove commented code 2016-08-08 15:07:25 -05:00
Job Evers-Meltzer 0ba0426591 Refactor lbrycrdd path function for osx 2016-08-08 15:06:48 -05:00
Job Evers-Meltzer 9a9b540701 Move some ServerRequestHandling logging to debug level 2016-08-08 15:04:50 -05:00
Job Evers-Meltzer b8ddbb9a26 Add loggly logger
It configured twice, once right at startup without the lbry id and then
after the settings database is available it gets reconfigured with the
lbry id and a session id.
2016-08-08 15:02:36 -05:00
Job Evers-Meltzer bc3c2b96ed disable noisy loggers 2016-08-08 14:59:30 -05:00
Job Evers-Meltzer 622dc87d84 Remove log handlers with the same name
Also renames functions to follow underscore convention
2016-08-08 14:57:49 -05:00
Jack 18edbdf404 squelch resolve_name logging 2016-08-08 14:58:31 -04:00
Job Evers-Meltzer 02a3eab693 add loggly dependency 2016-08-08 11:39:11 -05:00
Job Evers-Meltzer dded868a80 remove leveldb requirement 2016-08-08 11:37:01 -05:00
Jack 6fe878bdb9 make load_from_hex a class method 2016-08-08 03:18:32 -04:00
Jack 505ece9e36 add txid param to get_claim_info and update todo note 2016-08-08 02:37:08 -04:00
Jack 72acfa1e0c add support_claim 2016-08-08 02:32:56 -04:00
Jack 02cd8b9a44 fix callback 2016-08-08 02:08:09 -04:00
Jack f32250f431 whoops 2016-08-08 01:54:41 -04:00
Jack 1523cad044 refactor get_claim_info 2016-08-08 01:46:19 -04:00
Jack ed55353700 bug fix 2016-08-08 01:43:28 -04:00
Jack 8d3bd45102 get_claims_for_name daemon function 2016-08-08 01:43:10 -04:00
Jack f69753f6a0 add get_claims_for_name 2016-08-08 01:42:37 -04:00
Jimmy Kiselak b7e2e87ac1 fill in much of the skeleton for the reflector protocol 2016-08-07 22:33:40 -04:00
Jack 3b98508986 clean up claim_name function 2016-08-07 16:13:47 -04:00
Jack 36ebfc42d9 update from master 2016-08-06 21:44:14 -04:00
Jack ae5f55ff65 Bump version: 0.3.13 -> 0.3.14 2016-08-06 20:07:28 -04:00
Jack Robison 01131da5b7 Merge pull request #99 from lbryio/refactor-refresh
Add DownloadNameHelper and ResolveNameHelper
2016-08-06 20:06:43 -04:00
Jack 66ae865de0 don't make new lbry file if publish is given a full metadata dict, remove update_claim() 2016-08-06 20:02:51 -04:00
Job Evers-Meltzer 8c50f08fec refactor _start_download 2016-08-05 11:37:24 -05:00
Job Evers-Meltzer e69de47daf refactor wait_on_write stopping condition 2016-08-05 11:37:24 -05:00
Job Evers-Meltzer acef10e848 fix merge oversight 2016-08-05 11:37:24 -05:00
Job Evers edb9d50d9e fix typo in docstring 2016-08-05 11:37:24 -05:00
Job Evers b989d4f6be Bug fix: correct the check on wait_or_get_stream
I reversed the check on lbry_file. Compare to the original code:
https://github.com/lbryio/lbry/blob/e67f55bd7447cbee730d895b7f35cf53cf69dba5/lbrynet/lbrynet_daemon/LBRYDaemon.py#L1119
2016-08-05 11:37:24 -05:00
Job Evers 84dd5c0eca Revert "revert get() refactor"
This reverts commit 5eb7c64ec3.
2016-08-05 11:37:24 -05:00
Job Evers-Meltzer db8daf20b7 quiet noisy log statement 2016-08-05 05:50:24 -05:00
Jack cb985b5a03 log conditions of lbry_file failing to start 2016-08-04 23:27:44 -04:00
Jack 28c4e38272 make line neater 2016-08-04 23:27:25 -04:00
Jack b29377dc2b keep claim id history 2016-08-04 23:27:10 -04:00
Jack ca09592a72 make logs easier to scroll through 2016-08-04 23:10:15 -04:00
Jack b0e14fae43 update over existing claim in publish() 2016-08-04 23:08:54 -04:00
Jack dfa68f126b get update_name in lbrycrdwallet working 2016-08-04 20:43:39 -04:00
Jack 4570f68727 add claim_id to lbry_file and to get_claim_info 2016-08-04 18:44:12 -04:00
Jack 4705acba55 use lbryum updateclaim 2016-08-04 02:22:10 -04:00
Jimmy Kiselak 1dea20a358 include sd blob hashes in list of blob hashes to send via reflector 2016-08-03 23:11:29 -04:00
Jimmy Kiselak 191f661b35 skeleton for reflector protocol 2016-08-03 23:03:14 -04:00
Jack f3ab38ba8f Bump version: 0.3.12 -> 0.3.13 2016-08-03 18:01:18 -04:00
Jack Robison 2d0a892e70 Merge pull request #110 from lbryio/lighthouse-client
Lighthouse client
2016-08-03 18:00:04 -04:00
Jack Robison 00eefe87d7 Merge pull request #109 from lbryio/download-descriptor
add download_descriptor
2016-08-03 17:59:52 -04:00
Jack f68fd96555 break out sd timeout into a variable 2016-08-03 17:57:26 -04:00
Jack fc50a3242f add note about request.notifyFinish 2016-08-03 17:49:13 -04:00
Jack 64e9ce2bfa comments about the upnp redirects 2016-08-03 17:44:41 -04:00
Jack e8d04b1921 specify a lighthouse server 2016-08-03 17:44:32 -04:00
Jack 9fedf14e55 request.notifyFinish 2016-08-03 12:46:46 -04:00
David Amrhein 7d535ef7d1 Make ca_path checks for frozen windows only 2016-08-03 10:16:14 -04:00
Jack 85d610bcdf add get_peers_for_hash 2016-08-03 03:16:06 -04:00
Jack 3af8b7bd0c use upnp redirects if they're already set 2016-08-03 01:27:40 -04:00
Jack 6215d85581 add LighthouseClient, announce sd hash after publishing 2016-08-02 18:34:48 -04:00
Jack 308d9f5abd add todo 2016-08-02 17:33:57 -04:00
Jack bcfabba038 fast search
use cost from lighthouse
2016-08-02 16:45:06 -04:00
Jack 69139b1987 already called error 2016-08-02 16:03:47 -04:00
Jack 3f8f9d9d73 add timeout to download_sd_blob 2016-08-02 02:54:04 -04:00
Jack 2283579109 reorganize download_descriptor 2016-08-01 23:15:32 -04:00
Jack 1e0af94ba2 add download_descriptor 2016-08-01 22:31:49 -04:00
David Amrhein dd631328e0 Update windows distribution setup 2016-07-30 22:54:51 -04:00
David Amrhein cca2a57c69 Set cacert environ path for windows distribution 2016-07-30 22:52:49 -04:00
David Amrhein bbfbcacc6a Add windows icons for packaging 2016-07-30 22:10:30 -04:00
David Amrhein 3f95c1a9d5 Skip db migration in windows 2016-07-30 17:09:38 -04:00
David Amrhein 7bd05e9b16 Handle creation of settings directory in windows 2016-07-30 17:09:15 -04:00
Dave-A 10742d7c51 Merge pull request #1 from lbryio/master
Update from original
2016-07-30 16:48:25 -04:00
Jack bbf7940e45 Bump version: 0.3.11 -> 0.3.12 2016-07-29 23:44:04 -04:00
Jack Robison f5af71f4e6 Merge pull request #107 from lbryio/metadata-version
Metadata versioning, publishing things, and USD denominated key fees
2016-07-29 23:43:11 -04:00
Jack 1ffcd67c9d specify exception 2016-07-29 23:05:01 -04:00
Jack cf8592ba12 docstring and todo 2016-07-29 22:49:50 -04:00
Jack 1e1d036590 clean log line 2016-07-29 22:44:38 -04:00
Jack 64afb6d305 clean up get_est_costs in search 2016-07-29 22:43:50 -04:00
Jack 20a9e018f7 revert updates to git 2016-07-29 22:36:30 -04:00
Jack a574c33ac6 multiple search servers 2016-07-29 21:37:44 -04:00
Jack d21e458fe1 remove debug line 2016-07-29 02:42:58 -04:00
Jack bb3ba53c6e fix search results 2016-07-28 22:39:34 -04:00
Jack ba2e4df76b remove debug stuff 2016-07-28 21:31:19 -04:00
Jack 33e7e4bee3 fancy search 2016-07-28 21:25:37 -04:00
Jack b83dce2cb3 no tricky characters 2016-07-28 18:48:29 -04:00
Jack 6821676853 handle decimals in jsonrpc return 2016-07-28 18:12:54 -04:00
Jack 52024986ee derp 2016-07-28 16:32:59 -04:00
Jack 2e3342d20d fix test 2016-07-28 16:27:16 -04:00
Jack 419aa1d78f fix est data cost / search 2016-07-28 16:12:20 -04:00
Jack de4ddd578f improve readability of metadata and fee requirements 2016-07-28 15:24:20 -04:00
Jack 5dd68a7ed4 add get_claim_info, clean up _get_stream_info_from_value 2016-07-28 14:55:17 -04:00
Jack 881168dc7a fix too many args 2016-07-28 13:16:33 -04:00
Job Evers-Meltzer 295e427abe bug fix: check if fee is in metadata before loading 2016-07-28 12:04:25 -05:00
Job Evers 67c5d15f51 stupid typo 2016-07-28 11:46:33 -05:00
Job Evers 39a7f37f35 muck with pending_claims... still not sure this is correct 2016-07-28 11:43:47 -05:00
Job Evers 15d276b0bc fix case where to and from currencies are the same 2016-07-28 11:43:20 -05:00
Jack 4387025c48 another still 2016-07-28 06:07:33 -04:00
Jack 52c20f02c7 one more 2016-07-28 06:03:47 -04:00
Jack 413dab8094 whoops 2016-07-28 05:59:10 -04:00
Jack 6b1d11a04d fix rate in test 2016-07-28 05:53:31 -04:00
Jack 5441379f38 fix LBRYExchangeRateManager test 2016-07-28 05:49:31 -04:00
Jack 52d954c64e don't spam the market apis 2016-07-28 05:34:20 -04:00
Jack 415495fc16 LBRYExchangeRateManager 2016-07-28 05:30:13 -04:00
Jack 4223298634 test wrong metadata version 2016-07-28 02:49:35 -04:00
Jack e058c71019 test data 2016-07-28 02:46:46 -04:00
Jack 12b2bbfa26 make sure claimed ver is true 2016-07-28 02:36:07 -04:00
Jack 7d22bfffa5 fee tests 2016-07-28 02:30:37 -04:00
Jack 4ade80c8a9 clean up metadata and fee 2016-07-28 02:19:45 -04:00
Jack 9674381fc2 Merge remote-tracking branch 'origin/metadata-version-avec-tests' into metadata-version 2016-07-27 20:18:00 -04:00
Jack f6d91280d7 remove debug print line 2016-07-27 15:03:03 -04:00
Jack e4ebbd8f69 neater force param in resolve_name 2016-07-27 15:02:54 -04:00
Jack 7f5560de6d comment spacing and removing setLevel 2016-07-27 15:02:19 -04:00
Job Evers-Meltzer 0925171e07 bug fix: use txid attribute, not key 2016-07-27 13:59:26 -05:00
Job Evers-Meltzer 900d3ad01d Fixes errback not being callable
Small code refactoring to make it more obvious what
both the callback and errback are doing
2016-07-27 13:12:20 -05:00
Job Evers-Meltzer 6a1efedc1f some example tests. please expand on these 2016-07-27 12:05:29 -05:00
Job Evers-Meltzer 0f602e0e62 allow bittrex fee to be changed 2016-07-27 11:59:57 -05:00
Jack 3a425fc47d Merge branch 'master' into metadata-version 2016-07-27 05:07:23 -04:00
Jack 36722cbbd8 Bump version: 0.3.10 -> 0.3.11 2016-07-27 03:58:25 -04:00
Jack 9572336e96 Merge branch 'master' into metadata-version
# Conflicts:
#	lbrynet/core/utils.py
#	lbrynet/lbrynet_daemon/LBRYDaemon.py
2016-07-27 03:54:10 -04:00
Jack 4895cde474 fix conflicts 2016-07-27 03:48:32 -04:00
Jack Robison 21ca1a73cf Merge pull request #100 from lbryio/change-ui-location
web-ui files are on S3 now
2016-07-27 03:44:28 -04:00
Jack Robison 457c1c6ae6 Merge pull request #101 from lbryio/version-cmp
Properly Compare Version Strings
2016-07-27 03:43:13 -04:00
Jack Robison c1ff127101 Merge pull request #102 from lbryio/log-configure
Fix: Log configuration should not happen when a module is imported
2016-07-27 03:42:51 -04:00
Jack 5e815a5832 merge master 2016-07-27 03:22:03 -04:00
Jack 4604e9cd54 check pending claims after one is made 2016-07-27 03:00:44 -04:00
Jack ef580a7680 errback for pending of date lbry_file 2016-07-27 01:51:09 -04:00
Jack 2102776be8 another errback 2016-07-27 01:44:01 -04:00
Jack f3af7a6b77 add errback for unknown name 2016-07-27 01:39:23 -04:00
Jack 5abeceac1f abandon previous claim in update 2016-07-27 01:13:41 -04:00
Jack 22f73a0817 add force parameter to resolve_name 2016-07-26 21:46:04 -04:00
Jack 29b5aef73a call restore() after claiming name 2016-07-26 21:24:58 -04:00
Jack 3e602bce52 fix publish callback 2016-07-26 20:58:37 -04:00
Jack fe39901885 update claim when publishing to an already claimed name
-also save name_metadata for each unique claim rather than for each name
2016-07-26 20:52:05 -04:00
Jack ee9e8b365c add debug log line 2016-07-26 18:42:51 -04:00
Jack bd6fa35d4a disable verbose dht error 2016-07-26 18:19:40 -04:00
Jack 5e1fdf66b0 fix download problem 2016-07-26 18:18:34 -04:00
Jack d86e709b90 reduce min blob prices 2016-07-26 17:24:59 -04:00
Jack f82cffe882 verbose logging 2016-07-26 17:21:41 -04:00
Jack 3d88eb92c1 change default max fee to $25 2016-07-26 17:21:03 -04:00
Jack 0811ebb52d fix key payment problems 2016-07-26 17:20:51 -04:00
Jack 491d431ea5 remove price updater from daemon 2016-07-26 17:19:26 -04:00
Job Evers‐Meltzer ceeebebcad Merge pull request #106 from lbryio/no-blobs-in-logs
Have TimeoutError hexlify blobs
2016-07-26 14:57:30 -05:00
Job Evers‐Meltzer 04d766aba7 Merge pull request #105 from lbryio/bug-fixes
misc bug fixes and code cleanup
2016-07-26 14:56:50 -05:00
Job Evers‐Meltzer 3b431e180e Merge pull request #98 from lbryio/log-cleanup
Remove datetime.now() from log statements
2016-07-26 14:54:36 -05:00
Job Evers a2eab1577d Have TimeoutError hexlify blobs 2016-07-26 13:54:46 -05:00
Job Evers-Meltzer f8dd3d05dc misc bug fixes and code cleanup 2016-07-26 12:24:25 -05:00
Job Evers-Meltzer 93993e62d6 Fix: Log configuration should not happen when a module is imported
Instead, move the responsibility to the main program. Also,
each module had the same, redundant setup.
2016-07-26 12:01:03 -05:00
Job Evers‐Meltzer 483c88b958 Merge pull request #104 from lbryio/no-print
Remove print line
2016-07-26 11:36:00 -05:00
Job Evers 4a1e846958 Remove print line
This line is responsible for flooding the console with messages like
<_io.BytesIO object at 0x7f9e286e52f0>
2016-07-26 11:34:13 -05:00
Job Evers-Meltzer 927b9afe7c forgot to add the test file 2016-07-26 09:34:00 -05:00
Jack 9e8827f015 resolve name bug 2016-07-25 23:46:18 -04:00
Jack ca45f461ae spelling 2016-07-25 23:42:03 -04:00
Jack 7e5af11c2d fix key fee payment problem 2016-07-25 23:41:26 -04:00
Jack 102436274f USD denominated max_key_fee in settings 2016-07-25 22:46:04 -04:00
Jack 49c26a76b4 remove fee checking from autofetcher script 2016-07-25 22:28:01 -04:00
Jack d469ef7bf2 add googlefinance to requirements 2016-07-25 22:21:38 -04:00
Jack a5374cb876 fix currencies and LBRYFeeFormat 2016-07-25 22:17:31 -04:00
Jack a53e911a0c stop looping call when stop() is called 2016-07-25 21:48:39 -04:00
Jack c4a78a149b LBRYFee things
-move exchange rate updates to wallet, do them every half hour
-add convention checker for fees
-pay BTC or USD denominated key fees in converted LBC amount
2016-07-25 21:45:42 -04:00
Job Evers-Meltzer 3814912adf Move to core.utils and expand to check lbryum too 2016-07-25 16:09:13 -05:00
Jack 75ef652cb3 add googlefinance to setup.py 2016-07-25 17:08:56 -04:00
Job Evers-Meltzer 1eae3fe5dc pylint doesn't like distutils 2016-07-25 14:53:07 -05:00
Job Evers-Meltzer d0bc383a79 bug fix: version comparison is not lexographic.
Use distutils.version to compare version strings.
2016-07-25 12:25:10 -05:00
Job Evers-Meltzer ea5d31b530 OCD: alphabetize and group imports
I'm not the only one:
https://google.github.io/styleguide/pyguide.html?showone=Imports_formatting#Imports_formatting
2016-07-25 12:19:19 -05:00
Job Evers‐Meltzer e56ad19e81 Merge pull request #95 from lbryio/deployment-update
Updates the deployment process and the version check associated with it
2016-07-25 12:10:29 -05:00
Job Evers ed393eb75c Updates the deployment process and the version check associated with it
This commit supports steps 1 and 2 in the new workflow:
1. Change the logic in the daemon to check the github api for the latest release that is not a pre release
2. Change travis to mark all releases as pre release
3. When we are ready to stage a release we push a tag to master. Travis builds the packages and releases them
4. We manually check them
5. Remove the pre release mark when we are happy
2016-07-25 11:13:47 -05:00
Jack 2e2d309d38 BTC/USD denominated fees, metadata revisions 2016-07-25 01:40:26 -04:00
Jack 6bfb33adb2 fix version comparison 2016-07-24 22:03:32 -04:00
Job Evers-Meltzer 05138e1db1 web-ui files are on S3 now 2016-07-22 10:58:37 -05:00
Jack b65402db41 debug line 2016-07-21 16:47:47 -04:00
Jack fbef187400 Merge branch 'master' into metadata-version
# Conflicts:
#	lbrynet/conf.py
#	lbrynet/lbrynet_daemon/LBRYDaemon.py
2016-07-21 16:45:41 -04:00
Jack 25d1b8b9e5 move towards better fees and metadata 2016-07-21 16:11:14 -04:00
Job Evers ec91594083 Remove datetime.now() from log statements 2016-07-21 10:41:26 -05:00
Jack e67f55bd74 disable debug logging 2016-07-20 22:40:55 -04:00
Jack Robison 276a7f9d2a Merge pull request #92 from lbryio/file-uploads
Add support for file uploads (for Publish page)
2016-07-20 22:37:08 -04:00
Jack 4888d7a567 Bump version: 0.3.9 -> 0.3.10 2016-07-20 22:34:16 -04:00
Jack Robison 767f78315b Merge pull request #97 from lbryio/lbryum-catchup
lbryum fixes
2016-07-20 22:33:16 -04:00
Jack 4311b218fb fix auto_connect 2016-07-20 22:13:22 -04:00
Jack 89892042dd remove unused line 2016-07-20 20:57:05 -04:00
Jack a22dfb3864 cleaner logging
from https://github.com/lbryio/lbry/pull/94
2016-07-20 20:56:56 -04:00
Jack f40c42dcf0 Merge branch 'master' into lbryum-catchup 2016-07-20 20:38:42 -04:00
Jack 1d31aa9da3 Bump version: 0.3.8 -> 0.3.9 2016-07-20 20:34:41 -04:00
Jack f999073fb4 revert download refactor
to be re-merged after fixes
2016-07-20 20:34:02 -04:00
Jack 5eb7c64ec3 revert get() refactor 2016-07-20 20:30:57 -04:00
Jack 7e79818555 Revert "Merge branch 'master' into lbryum-catchup"
This reverts commit a51576b250, reversing
changes made to 0bee2689d1.
2016-07-20 20:24:49 -04:00
Jack a51576b250 Merge branch 'master' into lbryum-catchup 2016-07-20 20:17:08 -04:00
Jack 0bee2689d1 download fixes 2016-07-20 20:17:01 -04:00
Jack c232743963 fix LBRYumWallet._do_send_many 2016-07-20 20:13:44 -04:00
Job Evers‐Meltzer c2d5f092b3 Merge pull request #94 from lbryio/logging-reformat-2
use asctime in logging format
2016-07-20 18:41:54 -05:00
Jack 8e8e186de9 Merge branch 'master' into lbryum-catchup 2016-07-20 19:29:11 -04:00
Jack 55f47a2b1b lbryum catchup 2016-07-20 17:32:15 -04:00
Job Evers-Meltzer 5c27b9eb90 use asctime in logging format
Instead of manually adding datetime.now() use
asctime formatter to do it for us.
2016-07-20 07:21:12 -05:00
Job Evers‐Meltzer 1d018533d1 Merge pull request #81 from lbryio/deprecate-lbry-console
deprecate lbry-console script
2016-07-20 07:07:16 -05:00
Job Evers‐Meltzer 34b8841f5c Merge pull request #73 from lbryio/refactor-dl-name
Refactor jsonrpc_get, _download_name, _resolve_name
2016-07-20 07:07:02 -05:00
Alex Liebowitz a54166a27c Add support for file uploads (for Publish page) 2016-07-20 02:36:55 -04:00
Job Evers 4adec39dce Bump version: 0.3.7 -> 0.3.8 2016-07-18 16:00:52 -05:00
Job Evers‐Meltzer e63ae08638 Merge pull request #86 from lbryio/back-to-lbryum
Revert default wallet back to lbryum
2016-07-18 15:33:03 -05:00
Job Evers‐Meltzer f1b4c6a1ec Cache pyobjc (#87)
* cache pyobjc modules as wheels
2016-07-18 14:50:05 -05:00
Job Evers-Meltzer 78b4b3ffa8 revert default wallet back to lbryum 2016-07-18 11:30:21 -05:00
Job Evers‐Meltzer f51fd48cdb Merge pull request #85 from lbryio/update-protobuf
Update protobuf library
2016-07-18 11:26:47 -05:00
Job Evers-Meltzer 9127a816e3 Update protobuf library
Updating to protobuf-v3.0.0b3 because it
fixes a bug here https://github.com/google/protobuf/blob/e841bac4fcf47f809e089a70d5f84ac37b3883df/python/google/protobuf/internal/python_message.py#L59
where six.moves is not always available.
In particular, the system python on OSX has an old version of six
which is without six.moves.
2016-07-18 10:12:49 -05:00
Job Evers-Meltzer 1a5cd9b062 Forget about trying to set a label.
Setting a label also sets the account, but the LBRY app
only looks at the default "" account.  Marking the imported
keys would have been nice, but it doesn't work.
2016-07-18 09:14:50 -05:00
Job Evers-Meltzer a86b4c4e58 bug fix: use sys.exit() not exit() 2016-07-18 08:52:56 -05:00
Job Evers-Meltzer a384960a7b deprecate lbry-console script 2016-07-18 08:45:28 -05:00
Job Evers‐Meltzer 5817b80c94 script to migrate private keys to lbrycrd (#79)
* script to migrate private keys to lbrycrd
* handle compressed keys too
* validate private key actually gets added
* add lbrycrdd and lbrycrd-cli checks
* exit if no wallet path
* move each imported address into the default account
2016-07-17 20:17:42 -05:00
Job Evers-Meltzer 97efd63c99 deprecate lbry-console script 2016-07-17 19:19:00 -05:00
Job Evers-Meltzer 5ebcc01bcc Bump version: 0.3.6 -> 0.3.7 2016-07-17 14:10:11 -05:00
Job Evers‐Meltzer ddcc008ea6 Merge pull request #78 from lbryio/force-wallet
emergency fix: force lbrycrd to be the default wallet
2016-07-17 14:08:41 -05:00
Job Evers-Meltzer 8ec10e2304 emergency fix: force lbrycrd to be the default wallet
Ignore save settings for wallet type and use the default wallet (lbrycrd)
instead.  lbryum can be used but only if specified on the command line.
2016-07-17 13:45:26 -05:00
Job Evers-Meltzer 172f275bc7 Refactor _resolve_name.
Continue using the delegation/helper pattern for the daemon.
2016-07-16 09:24:27 -05:00
Job Evers‐Meltzer 0dac5943bb Merge pull request #74 from llllIHARWINDERIllll/patch-3
Need to have git installed
2016-07-15 13:42:21 -05:00
Jack 2dcd2b13be add LBRYMetadata.py 2016-07-15 14:09:43 -04:00
Harwinder 8743decd17 Need to have git installed.
You cant execute the code that follows after if you havent installed git already.
2016-07-15 20:01:25 +02:00
Job Evers-Meltzer a15d7ca543 Refactor _download_name
Nested functions are the devil, especially ones that
use variables from the outer scope. Refactoring
_download_name to use a helper class helps make
the scoping more explicit and will undoubtably
prevent bugs in the future.

I think this makes _download_name drastically more readable.

Also cleaned up some duplicated code and
made download_directory respect the passed in parameter
instead of being the default.
2016-07-15 12:42:26 -05:00
Job Evers-Meltzer a90029ec50 update doc string 2016-07-15 11:37:04 -05:00
Job Evers-Meltzer 563896b126 fix bug in reveal code 2016-07-15 11:33:38 -05:00
Job Evers-Meltzer 5aecd02668 Refactor jsonrpc_get.
Move parameter handling into its own function and
better use the `.get()` function for dictionaries.

Early return on the failed checks is more readable.

The lambda function in the callback was long and
hard to read so moved it out.
2016-07-15 11:32:46 -05:00
Jeremy Kauffman eef6312a17 Merge pull request #71 from llllIHARWINDERIllll/patch-2
Cant use git if its not installed
2016-07-14 16:26:23 -04:00
Harwinder 1e48f5c8f4 Cant use git if its not installed
First install needed packages then execute the task. Cant run the whole code if git isnt installed :)
2016-07-14 22:25:32 +02:00
Jeremy Kauffman 7908fa2814 Merge pull request #70 from llllIHARWINDERIllll/patch-1
Adding git to linux install command
2016-07-14 16:16:51 -04:00
Harwinder 7ffb3ea042 Added git to linux install command
Some people dont have git installed so adding it to it solves confusion for people new to linux.
2016-07-14 18:41:23 +02:00
Job Evers‐Meltzer 842dd90780 Merge pull request #66 from lbryio/add-tests-blob-handler
add tests for BlobRequestHandler and run on travis
2016-07-14 09:33:06 -05:00
kimihiro64 ee71a03f96 Revert to 2 params as isinstance cannot support 3+ (#69)
* Revert to 2 params as isinstance cannot support 3+

* Use basestring for maximum efficiency
2016-07-13 22:44:02 -05:00
Jack 5221631b40 Bump version: 0.3.5 -> 0.3.6 2016-07-12 17:30:58 -04:00
Job Evers‐Meltzer ade2b4ac0d Merge pull request #68 from lbryio/switch-slack
switch slack notification token
2016-07-12 15:25:21 -05:00
Job Evers-Meltzer 643b95c0e4 switch slack notification token 2016-07-12 14:48:13 -05:00
Jack Robison ed0f9d4fb5 Merge pull request #67 from lbryio/real-lbryum
Real lbryum
2016-07-12 15:35:54 -04:00
Jack 6f353c31cb Merge branch 'master' into real-lbryum 2016-07-11 22:04:32 -04:00
Jack 404412a950 return publish error message 2016-07-11 21:57:34 -04:00
Job Evers‐Meltzer 0e1778d8da Merge pull request #65 from lbryio/add-slack-notifications
add slack notifications to travis
2016-07-11 13:28:56 -05:00
Job Evers-Meltzer dfc0ca9525 change slack key and encrypt it 2016-07-11 12:05:27 -05:00
Job Evers-Meltzer 53bf1fe4fd add send_blob tests 2016-07-10 21:56:00 -05:00
Jack 5c05daa007 update import to new file name 2016-07-10 22:21:50 -04:00
Job Evers-Meltzer f74f075b4e add tests for BlobRequestHandler and run on travis 2016-07-10 18:48:14 -05:00
Jack 1b53e8e98c re-enable lbryum 2016-07-09 13:31:07 -04:00
Job Evers-Meltzer 4caee284bc add slack notifications to travis 2016-07-07 12:06:07 -05:00
Alex Grintsvayg 5368c91399 lbry replaces old lbrynet 2016-07-07 00:18:29 -04:00
Alex Grintsvayg 00b1881f20 dont forget to run bumpversion instead of manually bumping 2016-07-06 22:56:23 -04:00
Jack b727bd20c8 version bump 2016-07-06 22:48:38 -04:00
Jack 5a703c116f linux lbrycrdd path 2016-07-06 22:46:59 -04:00
Alex Grintsvayg 57f728f40e desc update 2016-07-06 22:29:12 -04:00
Alex Grintsvayg 74bbde0c34 fix "launch as root" bug, update package name and maintainer 2016-07-06 21:54:55 -04:00
Alex Liebowitz c58fc0781c Call xdg-open properly in jsonrpc_reveal() 2016-07-06 15:57:00 -04:00
Alex Grintsvayg 7a266d49c1 fix 2016-07-06 15:33:06 -04:00
Alex Grintsvayg 344eeeb8c0 typo 2016-07-06 15:04:41 -04:00
Alex Grintsvayg 08d50ddfc5 bundle lbrycrdd bins instead of downloading them 2016-07-06 14:58:06 -04:00
Alex Grintsvayg f83c892b13 fix lbry symlink, fix icon path, add .desktop file to bumpversion 2016-07-06 14:13:11 -04:00
Alex Grintsvayg 7dbec9d395 typo 2016-07-06 13:41:47 -04:00
Alex Grintsvayg 5171c801d9 add lbry icons, symlink into user path 2016-07-06 13:34:17 -04:00
Alex Grintsvayg 72b4a778a6 dont forget bumpversion 2016-07-06 10:09:40 -04:00
Jack Robison 0254df63f9 Merge pull request #64 from lbryio/check-for-new-ui
Check for new ui periodically
2016-07-06 04:35:06 -04:00
Jack 03c02ff74f lbrycrdd path wierdness 2016-07-06 03:37:31 -04:00
Jack 9277c3c674 add help for when things go wrong 2016-07-06 03:02:55 -04:00
Jack 34b1259dc6 move check for if daemon is running to inside of function 2016-07-06 02:20:18 -04:00
Jack 257e38de12 add command line wrapper for lbrynet-daemon calls 2016-07-06 02:17:38 -04:00
Jack 1984132eae add update_available keys to version()
as to not require making the version string comparison in whatever is
making the daemon call
2016-07-06 01:46:49 -04:00
Jack 8b1d75ea98 check every 30 minutes for a new ui version
-previously the check for a new ui version was only at startup, which
resulted in bugs persisting after they’d been fixed

-add option to skip requirements check

-try to load an existing ui if it can’t be obtained from github
2016-07-06 01:27:25 -04:00
Job Evers‐Meltzer a79af11371 Merge pull request #60 from lbryio/change-make-deb
Use patched version of make-deb
2016-07-05 15:08:42 -05:00
Job Evers 087a9b70b4 Use patched version of make-deb
Ran into a bug in make-deb that fails when there is utf-8 characters
in the git commit message. https://github.com/nylas/make-deb/issues/13.

I wrote a patch to fix the bug and am installing that version until the
make-deb maintainers incorporate it
2016-07-05 14:57:48 -05:00
Alex Grintsvayg 8b37952128 make-deb doesn't like utf8 arrow in commit message 2016-07-05 15:53:51 -04:00
Alex Grintsvayg 8336930091 Bump version: 0.3.2 → 0.3.3 2016-07-05 15:20:35 -04:00
Alex Grintsvayg d875325cee better non-root fix, restore version var 2016-07-05 15:20:30 -04:00
Alex Grintsvayg f1c5d1cb23 Bump version: 0.3.1 → 0.3.2 2016-07-05 14:30:35 -04:00
Alex Grintsvayg b55752e715 add bumpversion script 2016-07-05 14:29:24 -04:00
Alex Grintsvayg 36a99e6ec8 run lbry as non-root user 2016-07-05 14:06:41 -04:00
Jack df3c5b9a77 open conf for write in app 2016-07-04 20:47:43 -04:00
Jack 4c2ab23c1f write to the file that's supposed to be written to 2016-07-04 20:19:04 -04:00
Jack 0a3f81e364 fix download directory not existing problem 2016-07-04 20:02:38 -04:00
Jack 69c5631b25 make lbrycrddpath.conf if it doesnt exist, point to binary in app 2016-07-04 19:53:01 -04:00
Jack cca4331ac4 create lbrycrddpath.conf if it doesnt exist 2016-07-04 19:34:51 -04:00
Alex Grintsvayg 5972dbb6a7 fix lbry init script 2016-07-04 19:18:22 -04:00
Job Evers-Meltzer 794f2d8576 version bump: 0.3.0.2 2016-07-04 16:51:29 -05:00
Alex Liebowitz e35dce530a Set default download dir to ~/Downloads on Linux 2016-07-04 17:49:48 -04:00
Job Evers-Meltzer 6034bd3cb2 version bump: 0.3.0.1 2016-07-04 15:47:12 -05:00
Job Evers‐Meltzer 42c77bbb67 Merge pull request #51 from lbryio/restart-lbrycrd
add a more useful exception if auth fails on lbrycrd
2016-07-04 15:45:44 -05:00
Job Evers‐Meltzer d504c8f4a3 Merge pull request #50 from lbryio/lbrycrd-in-bundle
Add lbrycrd to app bundle on macos
2016-07-04 15:42:24 -05:00
Job Evers-Meltzer 976c8b3ce0 add a more useful exception if auth fails on lbrycrd 2016-07-04 15:40:52 -05:00
Alex Liebowitz 905e7e41a6 Merge branch 'master' of https://github.com/lbryio/lbry 2016-07-04 16:26:35 -04:00
Job Evers-Meltzer 16f48d0400 Add lbrycrd to app bundle on macos 2016-07-04 15:18:08 -05:00
Alex Liebowitz 6d7608d851 Create lbrycrd.conf if needed 2016-07-04 16:16:34 -04:00
Jack 7cd30efbe9 download lbrycrd binaries and set up lbrycrddpath.conf 2016-07-04 08:41:44 -04:00
Jack Robison 8c3652e1ac Merge pull request #49 from lbryio/real-lbrycrd2
real blockchain
2016-07-04 05:48:49 -04:00
Jack 7b9e3aace9 use lbrycrd 2016-07-04 05:43:26 -04:00
Jack 568737e123 change lbryum version constant name 2016-07-04 05:15:53 -04:00
Jack 808e3ea6e3 rename LBRYcrdWallet.py to LBRYWallet.py and change default wallet to lbrycrd 2016-07-04 04:57:30 -04:00
Jack e4f14d871f update setup.py 2016-07-01 03:36:51 -04:00
Jack 1cedd8ee5b update name 2016-07-01 03:16:08 -04:00
Jack b80fe24179 tanstaafl 2016-07-01 02:55:20 -04:00
Jack 3cea41a854 remove lbrynet_gui 2016-07-01 02:42:34 -04:00
Jack 62b80c13cc version bump 2016-06-30 01:26:33 -04:00
Jack 08b612575c update_claim fix and clean up publish keys
-replace ‘value’ key with ‘metadata’

-update claim metadata in db when an update is published
2016-06-28 23:20:28 -04:00
Jack b3b804bf2d add update_name 2016-06-28 14:28:59 -04:00
Jack 09d8a86e30 rename variable 2016-06-28 02:06:04 -04:00
Jack cc5d917c72 download fix 2016-06-28 01:51:05 -04:00
Jack 7cc3e9d5ef publish updates
-adds a base set of metadata fields required for results to be rendered
in conf.py, including language and content-type

-dont support old style claims on the new blockchain
2016-06-27 17:07:59 -04:00
Job Evers‐Meltzer 87aa9fc6c4 Merge pull request #48 from lbryio/add-cache
add caching for travis
2016-06-27 11:50:23 -05:00
Job Evers 8d5c315817 add caching for travis 2016-06-27 11:21:30 -05:00
Jack f89486f5fe port and import name 2016-06-24 13:12:27 -04:00
201 changed files with 11947 additions and 7079 deletions

No files matched your search

+56
View File
@@ -0,0 +1,56 @@
version: 1.0.{build}
environment:
key_pass:
secure: u6DydPcdrUJlxGL9uc7yQRYG8+5rY6aAEE9nfCSzFyNzZlX9NniOp8Uh5ZKQqX7bGEngLI6ipbLfiJvn0XFnhbn2iTkOuMqOXVJVOehvwlQ=
pfx_key:
secure: 1mwqyRy7hDqDjDK+TIAoaXyXzpNgwruFNA6TPkinUcVM7A+NLD33RQLnfnwVy+R5ovD2pUfhQ6+N0Fqebv6tZh436LIEsock+6IOdpgFwrg=
notifications:
- provider: Slack
incoming_webhook:
secure: LuxwG5OZnnA//gmSXzCKu8/FRqYjCgGfVFqajSsGHeQ1HQNp7rYNhQpsO8/3PK63xKJj3wzt86DJekf9q9Q5OcHa9AHXUQbEveX0psd7elw=
clone_folder: c:\projects\lbry
install:
- cmd: dir C:\projects\lbry\packaging\windows\
- ps: .\packaging\windows\init.ps1
- nuget install secure-file -ExcludeVersion
- secure-file\tools\secure-file -decrypt packaging\windows\certs\lbry2.pfx.enc -secret %pfx_key%
- ps: .\packaging\windows\install.ps1
build_script:
- ps: C:\Python27\python.exe setup.py build bdist_msi
- signtool.exe sign /f packaging\windows\certs\lbry2.pfx /p %key_pass% /tr http://tsa.starfieldtech.com /td SHA256 /fd SHA256 dist\*.msi
test_script:
- ps: .\packaging\windows\test.ps1
artifacts:
- path: dist/*.msi
name: msi
- path: build/exe.win32-2.7/
name: lbry-portable
- path: packaging/windows/lbry-win32-app/LBRY-URI.reg
name: LBRY-URI
deploy:
release: $(APPVEYOR_REPO_TAG_NAME)
description: 'Release'
provider: GitHub
auth_token:
secure: 28gMVxQkXr2iXP4F+5kVwefUtKCfS1ePZ97PVfaSR8UDupjAvKhSJx862TnEjukb
artifact: /.*\.msi/
draft: false
prerelease: true
on:
appveyor_repo_tag: true # deploy on tag push only
+9
View File
@@ -0,0 +1,9 @@
[bumpversion]
current_version = 0.7.2
commit = True
tag = True
[bumpversion:file:lbrynet/__init__.py]
[bumpversion:file:packaging/ubuntu/lbry.desktop]
+5
View File
@@ -25,3 +25,8 @@ lbrynet.egg-info/PKG-INFO
*.decTest
.coverage
# temporary files from the twisted.trial test runner
_trial_temp/
.DS_Store
+1 -1
View File
@@ -298,7 +298,7 @@ ignored-classes=twisted.internet,RequestMessage
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
generated-members=lbrynet.lbrynet_daemon.LBRYDaemon.Parameters
[IMPORTS]
+12
View File
@@ -14,6 +14,16 @@ matrix:
language: generic
osx_image: xcode7.3
notifications:
slack:
secure: "Am13HPtpgCMljh0MDVuoFHvQXB8yhf4Kvf/qAeSp5N0vsHGL70CSF9Ahccw8dVPE6mbuak1OGtSUb6/UaErLHkpz3ztaRLkDa9x7CmBB3Kynnh8oO2VbB7b/2ROULqkhF4VZmAnNfwrQrbC3gs8Sybp261Nyc7y4ww15xDYBrk2fyq4ds2DCaJdRxfJUJFonrZ6KXr3fVaXosO6cjuyS8eRodcmrqsT4cCtinjNTD1hGWoH107E4ObSmpVelxQO193KhNJMRiLlEcVkvYUOqIWBtwdGHbNE/6Yeuq1TXgKJ0KeJWAmW3wTfUYNngGXNAsyCnrhul5TKNevNzfIAQZHvRsczYiWPJV6LtohHT0CcUiCXJtvEPOyahEBfwK3etY/xxFqny7N9OEmpdW2sgsEPNPX2LJynJti2rQA9SuAD1ogR3ZpDy/NXoaAZf8PTdPcuNUMULV9PGG7tVrLBecO/W1qO6hdFxwlLdgqGLxAENZgGp++v/DhPk/WvtmHj7iTbRq0nxaTWyX5uKOn2ADH+k/yfutjv6BsQU9xNyPeZEEtuEpc6X6waiYn/8G9vl9PecvKC5H0MgsZ6asAxmg7mZ3VSMFG7mo8ENeOhSZ0Oz6ZTBILL3wFccZA9uJIq7NWmqC9dRiGiuKXBB62No7sINoHg3114e2xYa9qvNmGg="
cache:
directories:
- $HOME/.cache/pip
- $HOME/Library/Caches/pip
- $TRAVIS_BUILD_DIR/cache/wheel
before_install:
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ./packaging/travis/setup_osx.sh; fi
@@ -35,6 +45,7 @@ deploy:
- provider: releases
file: "${TRAVIS_BUILD_DIR}/`python setup.py --name`_`python setup.py -V`_amd64.deb"
skip_cleanup: true
prerelease: true
on:
tags: true
condition: "$TRAVIS_OS_NAME = linux"
@@ -44,6 +55,7 @@ deploy:
- provider: releases
file: "${TRAVIS_BUILD_DIR}/packaging/osx/lbry-osx-app/`python setup.py --name`.`python setup.py -V`.dmg"
skip_cleanup: true
prerelease: true
on:
tags: true
condition: "$TRAVIS_OS_NAME = osx"
+1 -1
View File
@@ -12,7 +12,7 @@ You can install LBRY command line by running `curl -sL https://rawgit.com/lbryio
On Ubuntu or Mint you can install the prerequisites and lbrynet by running
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip git
git clone https://github.com/lbryio/lbry.git
cd lbry
sudo python setup.py install
+36 -11
View File
@@ -1,10 +1,10 @@
#### Installing the LBRY app
--------------------------
Installing LBRY is simple. You can get a dmg installer for OS X (Mavericks and up) or a .deb for linux [here](https://lbry.io/get).
Installing LBRY is simple. You can get a dmg installer for OS X or a .deb for linux [here](https://github.com/lbryio/lbry/releases/latest).
##### OS X
Just drag and drop LBRY.app into your applications folder (replacing any older versions). When it's running you'll have a LBRY icon in your status bar.
Drag and drop LBRY.app into your applications folder (replacing any older versions). When it's running you'll have a LBRY icon in your status bar and the browser will open to the UI.
##### Linux
Double click the .deb file and follow the prompts. The app can be started by searching "LBRY", and it can be turned off by clicking the red 'x' in the browser interface.
@@ -12,26 +12,51 @@ Double click the .deb file and follow the prompts. The app can be started by sea
On both systems you can also open the UI while the app is running by going to lbry://lbry in Firefox or Safari, or localhost:5279 in Chrome.
#### Installing LBRY command line
#### Installing LBRY from source
--------------------------
##### OS X
You can install LBRY command line by running `curl -sL https://rawgit.com/lbryio/lbry-setup/master/lbry_setup_osx.sh | sudo bash` in a terminal. This script will install lbrynet and its dependancies, as well as the app. You can start LBRY by either starting the app or by running `lbrynet-daemon` from a terminal.
Installing LBRY from source is not the recommended way to install
LBRY. Only do so if you're interested in doing development work.
##### Linux
On Ubuntu or Mint you can install the prerequisites and lbrynet by running
##### OS X Prerequisites
In order to install LBRY from source you'll need to first install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/). The dependencies can then be installed by running
```
brew install mpfr libmpc
sudo easy_install pip
sudo pip install virtualenv
```
##### Linux Prerequisites
On Ubuntu or Mint you can install the prerequisites by running
```
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev \
python-pip git python-virtualenv libssl-dev libffi-dev
```
##### OSX and Linux Installation
It is strongly recommended to create a new virtualenv for LBRY
```
virtualenv lbry-venv
source lbry-venv/bin/activate
```
and install the package in the new virtualenv:
```
sudo apt-get install libgmp3-dev build-essential python2.7 python2.7-dev python-pip
git clone https://github.com/lbryio/lbry.git
cd lbry
sudo python setup.py install
python setup.py install
```
To start LBRY, run `lbrynet-daemon` in a terminal.
#### On windows:
##### On windows
Install [mingw32](http://www.mingw.org/) base and c++ compiler.
+12 -19
View File
@@ -1,22 +1,15 @@
Copyright (c) 2015, LBRY, Inc.
All rights reserved.
The MIT License (MIT)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
Copyright (c) 2015-2016 LBRY Inc
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the
following conditions:
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-1
View File
@@ -1,2 +1 @@
include lbrynet/lbrynet_console/plugins/blindrepeater.yapsy-plugin
recursive-include lbrynet/lbrynet_gui *.gif *.ico *.xbm *.conf
-146
View File
@@ -1,146 +0,0 @@
How to watch It's a Wonderful Life via LBRY
## Quickest quick guide
Create a directory called lbry, and go into that directory
Download the file https://raw.githubusercontent.com/lbryio/lbry-setup/master/lbry_setup.sh and run it in that directory
Once it's done building, type:
```
lbrynet-console
```
A console application will load, and after a moment you will be presented with a ">" signifying
that the console is ready for commands.
If it's your first time running lbrynet-console, you should be given some credits to test the
network. If they don't show up within a minute or two, let us know, and we'll send you some.
After your credits have shown up, type
```
get wonderfullife
```
into the prompt and hit enter.
You will be asked if you want to cancel, change options, save, and perhaps stream the file.
Enter 's' for save and then hit enter.
The file should start downloading. Enter the command 'status' to check on the status of files
that are in the process of being downloaded.
To stop lbrynet-console, enter the command 'exit'.
## Slightly longer install guide
### Installing lbrycrd, the full blockchain client
Note: this process takes upwards of an hour and is not necessary to use lbrynet.
```
git clone --depth=1 -b alpha https://github.com/lbryio/lbrycrd.git
cd lbrycrd
sudo apt-get install build-essential libtool autotools-dev autoconf pkg-config libssl-dev libboost-all-dev libdb-dev libdb++-dev libqt4-dev libprotobuf-dev protobuf-compiler
./autogen.sh
./configure --with-incompatible-bdb --without-gui
make
```
When make has completed, create the directory where LBRYcrd data will be stored. ~/.lbrycrd is where LBRYcrd will look by default and so is recommended.
```
mkdir ~/.lbrycrd
echo 'rpcuser=rpcuser
rpcpassword=rpcpassword' > ~/.lbrycrd/lbrycrd.conf
# (use a long random password if your computer is on a network anyone else has access to. e.g, pwgen -s 20)
cd ..
```
### Installing lbrynet from source
Acquire the LBRYnet source code from https://github.com/lbryio/lbry
```
cd lbry
sudo apt-get install libgmp3-dev build-essential python-dev python-pip
```
(with virtualenv)
```
python-virtualenv
virtualenv .
source bin/activate
python setup.py install
```
to deactivate the virtualenv later:
```
deactivate
```
to reactivate it, go to the directory in which you created it and:
```
source bin/activate
```
(without virtualenv)
```
python setup.py build bdist_egg
sudo python setup.py install
```
## Slightly longer running guide
### lbrynet-console can be set to use lbrycrdd instead of the built in lightweight client.
To run lbrynet-console with lbrycrdd:
```
lbrynet-console </path/to/lbrycrdd>
```
If lbrycrdd is not already running, lbrynet will launch it at that path, and will shut it down
when lbrynet exits. If lbrycrdd is already running, lbrynet will not launch it and will not
shut it down, but it will connect to it and use it.
### Running lbrycrdd manually
From the lbrycrd directory, run:
```
./src/lbrycrdd -server -daemon
```
If you want to mine LBC, also use the flag '-gen', so:
```
./src/lbrycrdd -server -daemon -gen
```
Warning: This will put a heavy load on your CPU
It will take a few minutes for your client to download the whole block chain.
To shut lbrycrdd down: from the lbrycrd directory, run:
```
./src/lbrycrd-cli stop
```
Any questions or problems, email jimmy@lbry.io
+3 -5
View File
@@ -1,8 +1,6 @@
import logging
log = logging.getLogger(__name__)
logging.getLogger(__name__).addHandler(logging.NullHandler())
log.setLevel(logging.ERROR)
__version__ = "0.7.2"
version = tuple(__version__.split('.'))
version = (0, 2, 5)
__version__ = ".".join([str(x) for x in version])
logging.getLogger(__name__).addHandler(logging.NullHandler())
+6
View File
@@ -0,0 +1,6 @@
from constants import *
from events import *
from api import Api
from track import Track
from manager import Manager
+71
View File
@@ -0,0 +1,71 @@
import functools
import json
import logging
from requests import auth
from requests_futures import sessions
from lbrynet.conf import settings
from lbrynet.analytics import utils
log = logging.getLogger(__name__)
def log_response(fn):
def _log(future):
if future.cancelled():
log.warning('Request was unexpectedly cancelled')
else:
response = future.result()
log.debug('Response (%s): %s', response.status_code, response.content)
@functools.wraps(fn)
def wrapper(*args, **kwargs):
future = fn(*args, **kwargs)
future.add_done_callback(_log)
return future
return wrapper
class Api(object):
def __init__(self, session, url, write_key):
self.session = session
self.url = url
self.write_key = write_key
@property
def auth(self):
return auth.HTTPBasicAuth(self.write_key, '')
@log_response
def batch(self, events):
"""Send multiple events in one request.
Each event needs to have its type specified.
"""
data = json.dumps({
'batch': events,
'sentAt': utils.now(),
})
log.debug('sending %s events', len(events))
log.debug('Data: %s', data)
return self.session.post(self.url + '/batch', json=data, auth=self.auth)
@log_response
def track(self, event):
"""Send a single tracking event"""
log.debug('Sending track event: %s', event)
import base64
return self.session.post(self.url + '/track', json=event, auth=self.auth)
@classmethod
def load(cls, session=None):
"""Initialize an instance using values from the configuration"""
if not session:
session = sessions.FuturesSession()
return cls(
session,
settings.ANALYTICS_ENDPOINT,
utils.deobfuscate(settings.ANALYTICS_TOKEN)
)
+4
View File
@@ -0,0 +1,4 @@
"""Constants for metrics"""
BLOB_BYTES_UPLOADED = 'Blob Bytes Uploaded'
BLOB_BYTES_AVAILABLE = 'Blob Bytes Available'
+84
View File
@@ -0,0 +1,84 @@
import logging
from lbrynet.core import utils
log = logging.getLogger(__name__)
def get_sd_hash(stream_info):
if not stream_info:
return None
try:
return stream_info['sources']['lbry_sd_hash']
except (KeyError, TypeError, ValueError):
log.debug('Failed to get sd_hash from %s', stream_info, exc_info=True)
return None
class Events(object):
def __init__(self, context, lbry_id, session_id):
self.context = context
self.lbry_id = lbry_id
self.session_id = session_id
def heartbeat(self):
return self._event('Heartbeat')
def download_started(self, name, stream_info=None):
properties = {
'name': name,
'stream_info': get_sd_hash(stream_info)
}
return self._event('Download Started', properties)
def metric_observed(self, metric_name, value):
properties = {
'value': value,
}
return self._event(metric_name, properties)
def _event(self, event, event_properties=None):
return {
'userId': 'lbry',
'event': event,
'properties': self._properties(event_properties),
'context': self.context,
'timestamp': utils.isonow()
}
def _properties(self, event_properties=None):
event_properties = event_properties or {}
properties = {
'lbry_id': self.lbry_id,
'session_id': self.session_id,
}
properties.update(event_properties)
return properties
def make_context(platform, wallet, is_dev=False):
# TODO: distinguish between developer and release instances
return {
'is_dev': is_dev,
'app': {
'name': 'lbrynet',
'version': platform['lbrynet_version'],
'ui_version': platform['ui_version'],
'python_version': platform['python_version'],
'wallet': {
'name': wallet,
# TODO: add in version info for lbrycrdd
'version': platform['lbryum_version'] if wallet == 'lbryum' else None
},
},
# TODO: expand os info to give linux/osx specific info
'os': {
'name': platform['os_system'],
'version': platform['os_release']
},
'library': {
'name': 'lbrynet-analytics',
'version': '1.0.0'
},
}
+67
View File
@@ -0,0 +1,67 @@
from lbrynet.core import looping_call_manager
from twisted.internet import defer
from twisted.internet import task
import constants
class Manager(object):
def __init__(self, analytics_api, events_generator, track):
self.analytics_api = analytics_api
self.events_generator = events_generator
self.track = track
self.looping_call_manager = self.setup_looping_calls()
def setup_looping_calls(self):
call_manager = looping_call_manager.LoopingCallManager()
looping_calls = [
('send_heartbeat', self._send_heartbeat),
('update_tracked_metrics', self._update_tracked_metrics),
]
for name, fn in looping_calls:
call_manager.register_looping_call(name, task.LoopingCall(fn))
return call_manager
def start(self):
self.looping_call_manager.start('send_heartbeat', 60)
self.looping_call_manager.start('update_tracked_metrics', 300)
def shutdown(self):
self.looping_call_manager.shutdown()
def send_download_started(self, name, stream_info=None):
event = self.events_generator.download_started(name, stream_info)
self.analytics_api.track(event)
def register_repeating_metric(self, event_name, value_generator, frequency=300):
lcall = task.LoopingCall(self._send_repeating_metric, event_name, value_generator)
self.looping_call_manager.register_looping_call(event_name, lcall)
lcall.start(frequency)
def _send_heartbeat(self):
heartbeat = self.events_generator.heartbeat()
self.analytics_api.track(heartbeat)
def _update_tracked_metrics(self):
should_send, value = self.track.summarize_and_reset(constants.BLOB_BYTES_UPLOADED)
if should_send:
event = self.events_generator.metric_observed(constants.BLOB_BYTES_UPLOADED, value)
self.analytics_api.track(event)
def _send_repeating_metric(self, event_name, value_generator):
result = value_generator()
if_deferred(result, self._send_repeating_metric_value, event_name)
def _send_repeating_metric_value(self, result, event_name):
should_send, value = result
if should_send:
event = self.events_generator.metric_observed(event_name, value)
self.analytics_api.track(event)
def if_deferred(maybe_deferred, callback, *args, **kwargs):
if isinstance(maybe_deferred, defer.Deferred):
maybe_deferred.addCallback(callback, *args, **kwargs)
else:
callback(maybe_deferred, *args, **kwargs)
+24
View File
@@ -0,0 +1,24 @@
import collections
class Track(object):
"""Track and summarize observations of metrics."""
def __init__(self):
self.data = collections.defaultdict(list)
def add_observation(self, metric, value):
self.data[metric].append(value)
def summarize_and_reset(self, metric, op=sum):
"""Apply `op` on the current values for `metric`.
This operation also resets the metric.
Returns:
a tuple (should_send, value)
"""
try:
values = self.data.pop(metric)
return True, op(values)
except KeyError:
return False, None
+262 -40
View File
@@ -1,48 +1,270 @@
"""
Some network wide and also application specific parameters
"""
import copy
import logging
import os
import sys
from appdirs import user_data_dir
MAX_HANDSHAKE_SIZE = 2**16
MAX_REQUEST_SIZE = 2**16
MAX_BLOB_REQUEST_SIZE = 2**16
MAX_RESPONSE_INFO_SIZE = 2**16
MAX_BLOB_INFOS_TO_REQUEST = 20
BLOBFILES_DIR = ".blobfiles"
BLOB_SIZE = 2**21
MIN_BLOB_DATA_PAYMENT_RATE = .005 # points/megabyte
MIN_BLOB_INFO_PAYMENT_RATE = .02 # points/1000 infos
MIN_VALUABLE_BLOB_INFO_PAYMENT_RATE = .05 # points/1000 infos
MIN_VALUABLE_BLOB_HASH_PAYMENT_RATE = .05 # points/1000 infos
MAX_CONNECTIONS_PER_STREAM = 5
log = logging.getLogger(__name__)
KNOWN_DHT_NODES = [('104.236.42.182', 4000),
('lbryum1.lbry.io', 4444),
('lbryum2.lbry.io', 4444),
('lbryum3.lbry.io', 4444)]
POINTTRADER_SERVER = 'http://ec2-54-187-192-68.us-west-2.compute.amazonaws.com:2424'
#POINTTRADER_SERVER = 'http://127.0.0.1:2424'
LINUX = 1
DARWIN = 2
WINDOWS = 3
LOG_FILE_NAME = "lbrynet.log"
LOG_POST_URL = "https://lbry.io/log-upload"
CRYPTSD_FILE_EXTENSION = ".cryptsd"
if sys.platform.startswith("darwin"):
platform = DARWIN
default_download_directory = os.path.join(os.path.expanduser("~"), 'Downloads')
default_data_dir = user_data_dir("LBRY")
default_lbryum_dir = os.path.join(os.path.expanduser("~"), ".lbryum")
elif sys.platform.startswith("win"):
platform = WINDOWS
from lbrynet.winhelpers.knownpaths import get_path, FOLDERID, UserHandle
default_download_directory = get_path(FOLDERID.Downloads, UserHandle.current)
default_data_dir = os.path.join(
get_path(FOLDERID.RoamingAppData, UserHandle.current), "lbrynet")
default_lbryum_dir = os.path.join(
get_path(FOLDERID.RoamingAppData, UserHandle.current), "lbryum")
else:
platform = LINUX
default_download_directory = os.path.join(os.path.expanduser("~"), 'Downloads')
default_data_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
default_lbryum_dir = os.path.join(os.path.expanduser("~"), ".lbryum")
API_INTERFACE = "localhost"
API_ADDRESS = "lbryapi"
API_PORT = 5279
ICON_PATH = "app.icns"
APP_NAME = "LBRY"
API_CONNECTION_STRING = "http://%s:%i/%s" % (API_INTERFACE, API_PORT, API_ADDRESS)
UI_ADDRESS = "http://" + API_INTERFACE + ":" + str(API_PORT)
PROTOCOL_PREFIX = "lbry"
DEFAULT_WALLET = "lbryum"
WALLET_TYPES = ["lbryum", "lbrycrd"]
DEFAULT_TIMEOUT = 30
DEFAULT_MAX_SEARCH_RESULTS = 25
DEFAULT_MAX_KEY_FEE = 100.0
DEFAULT_SEARCH_TIMEOUT = 3.0
DEFAULT_CACHE_TIME = 3600
DEFAULT_UI_BRANCH = "master"
def convert_setting(env_val, current_val):
try:
return _convert_setting(env_val, current_val)
except Exception as exc:
log.warning(
'Failed to convert %s. Returning original: %s: %s',
env_val, current_val, exc)
return current_val
def _convert_setting(env_val, current_val):
new_type = env_val.__class__
current_type = current_val.__class__
if current_type is bool:
if new_type is bool:
return env_val
elif str(env_val).lower() == "false":
return False
elif str(env_val).lower() == "true":
return True
else:
raise ValueError('{} is not a valid boolean value'.format(env_val))
elif current_type is int:
return int(env_val)
elif current_type is float:
return float(env_val)
elif current_type is str:
return str(env_val)
elif current_type is dict:
return dict(env_val)
elif current_type is list:
return list(env_val)
elif current_type is tuple:
return tuple(env_val)
else:
raise ValueError('Type {} cannot be converted'.format(current_type))
def convert_env_setting(setting, value):
try:
env_val = os.environ[setting]
except KeyError:
return value
else:
return convert_setting(env_val, value)
def get_env_settings(settings):
for setting, value in settings.iteritems():
setting = 'LBRY_' + setting.upper()
yield convert_env_setting(setting, value)
def add_env_settings_to_dict(settings_dict):
for setting, env_setting in zip(settings_dict, get_env_settings(settings_dict)):
settings_dict.update({setting: env_setting})
return settings_dict
class Setting(object):
"""A collection of configuration settings"""
__fixed = []
__excluded = ['get_dict', 'update']
def __iter__(self):
for k in self.__dict__.iterkeys():
if k.startswith('_') or k in self.__excluded:
continue
yield k
def __getitem__(self, item):
assert item in self, IndexError
return self.__dict__[item]
def __setitem__(self, key, value):
assert key in self and key not in self.__fixed, KeyError(key)
old_value = self[key]
new_value = convert_setting(value, old_value)
self.__dict__[key] = new_value
def __contains__(self, item):
return item in iter(self)
def get_dict(self):
return {k: self[k] for k in self}
def update(self, other):
for k, v in other.iteritems():
try:
self.__setitem__(k, v)
except (KeyError, AssertionError):
pass
class AdjustableSettings(Setting):
"""Settings that are allowed to be overriden by the user"""
def __init__(self):
self.is_generous_host = True
self.run_on_startup = False
self.download_directory = default_download_directory
self.max_upload = 0.0
self.max_download = 0.0
self.upload_log = True
self.delete_blobs_on_remove = True
self.use_upnp = True
self.start_lbrycrdd = True
self.run_reflector_server = False
self.startup_scripts = []
self.last_version = {'lbrynet': '0.0.1', 'lbryum': '0.0.1'}
self.peer_port = 3333
self.dht_node_port = 4444
self.reflector_port = 5566
self.download_timeout = 30
self.max_search_results = 25
self.search_timeout = 3.0
self.cache_time = 150
self.host_ui = True
self.check_ui_requirements = True
self.local_ui_path = False
self.api_port = 5279
self.search_servers = ['lighthouse1.lbry.io:50005']
self.data_rate = .0001 # points/megabyte
self.min_info_rate = .02 # points/1000 infos
self.min_valuable_info_rate = .05 # points/1000 infos
self.min_valuable_hash_rate = .05 # points/1000 infos
self.max_connections_per_stream = 5
self.known_dht_nodes = [
('104.236.42.182', 4000),
('lbrynet1.lbry.io', 4444),
('lbrynet2.lbry.io', 4444),
('lbrynet3.lbry.io', 4444)
]
self.pointtrader_server = 'http://127.0.0.1:2424'
self.reflector_servers = [("reflector.lbry.io", 5566)]
self.wallet = "lbryum"
self.ui_branch = "master"
self.default_ui_branch = 'master'
self.data_dir = default_data_dir
self.lbryum_wallet_dir = default_lbryum_dir
self.use_auth_http = False
self.sd_download_timeout = 3
self.max_key_fee = {'USD': {'amount': 25.0, 'address': ''}}
class ApplicationSettings(Setting):
"""Settings that are constants and shouldn't be overriden"""
def __init__(self):
self.MAX_HANDSHAKE_SIZE = 2**16
self.MAX_REQUEST_SIZE = 2**16
self.MAX_BLOB_REQUEST_SIZE = 2**16
self.MAX_RESPONSE_INFO_SIZE = 2**16
self.MAX_BLOB_INFOS_TO_REQUEST = 20
self.BLOBFILES_DIR = "blobfiles"
self.BLOB_SIZE = 2**21
self.LOG_FILE_NAME = "lbrynet.log"
self.LOG_POST_URL = "https://lbry.io/log-upload"
self.CRYPTSD_FILE_EXTENSION = ".cryptsd"
self.API_INTERFACE = "localhost"
self.API_ADDRESS = "lbryapi"
self.ICON_PATH = "icons" if platform is WINDOWS else "app.icns"
self.APP_NAME = "LBRY"
self.PROTOCOL_PREFIX = "lbry"
self.wallet_TYPES = ["lbryum", "lbrycrd"]
self.SOURCE_TYPES = ['lbry_sd_hash', 'url', 'btih']
self.CURRENCIES = {
'BTC': {'type': 'crypto'},
'LBC': {'type': 'crypto'},
'USD': {'type': 'fiat'},
}
self.LOGGLY_TOKEN = 'LJEzATH4AzRgAwxjAP00LwZ2YGx3MwVgZTMuBQZ3MQuxLmOv'
self.ANALYTICS_ENDPOINT = 'https://api.segment.io/v1'
self.ANALYTICS_TOKEN = 'Ax5LZzR1o3q3Z3WjATASDwR5rKyHH0qOIRIbLmMXn2H='
APPLICATION_SETTINGS = AdjustableSettings()
ADJUSTABLE_SETTINGS = AdjustableSettings()
class DefaultSettings(ApplicationSettings, AdjustableSettings):
__fixed = APPLICATION_SETTINGS.get_dict().keys()
def __init__(self):
ApplicationSettings.__init__(self)
AdjustableSettings.__init__(self)
DEFAULT_SETTINGS = DefaultSettings()
class Config(DefaultSettings):
__shared_state = copy.deepcopy(DEFAULT_SETTINGS.get_dict())
def __init__(self):
self.__dict__ = add_env_settings_to_dict(self.__shared_state)
@property
def ORIGIN(self):
return "http://%s:%i" % (DEFAULT_SETTINGS.API_INTERFACE, self.api_port)
@property
def REFERER(self):
return "http://%s:%i/" % (DEFAULT_SETTINGS.API_INTERFACE, self.api_port)
@property
def API_CONNECTION_STRING(self):
return "http://%s:%i/%s" % (
DEFAULT_SETTINGS.API_INTERFACE, self.api_port, DEFAULT_SETTINGS.API_ADDRESS)
@property
def UI_ADDRESS(self):
return "http://%s:%i" % (DEFAULT_SETTINGS.API_INTERFACE, self.api_port)
def get_data_dir():
data_dir = default_data_dir
if not os.path.isdir(data_dir):
os.mkdir(data_dir)
return data_dir
def get_log_filename():
"""Return the log file for this platform.
Also ensure the containing directory exists
"""
return os.path.join(get_data_dir(), settings.LOG_FILE_NAME)
# TODO: don't load the configuration automatically. The configuration
# should be loaded at runtime, not at module import time. Module
# import should have no side-effects. This is also bad because
# it means that settings are read from the environment even for
# tests, which is rarely what you want to happen.
settings = Config()
+94
View File
@@ -0,0 +1,94 @@
import logging
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from decimal import Decimal
log = logging.getLogger(__name__)
class BlobAvailabilityTracker(object):
"""
Class to track peer counts for known blobs, and to discover new popular blobs
Attributes:
availability (dict): dictionary of peers for known blobs
"""
def __init__(self, blob_manager, peer_finder, dht_node):
self.availability = {}
self._last_mean_availability = Decimal(0.0)
self._blob_manager = blob_manager
self._peer_finder = peer_finder
self._dht_node = dht_node
self._check_popular = LoopingCall(self._update_most_popular)
self._check_mine = LoopingCall(self._update_mine)
def start(self):
log.info("Starting blob tracker")
self._check_popular.start(30)
self._check_mine.start(120)
def stop(self):
if self._check_popular.running:
self._check_popular.stop()
if self._check_mine.running:
self._check_mine.stop()
def get_blob_availability(self, blob):
def _get_peer_count(peers):
have_blob = sum(1 for peer in peers if peer.is_available())
return {blob: have_blob}
d = self._peer_finder.find_peers_for_blob(blob)
d.addCallback(_get_peer_count)
return d
def get_availability_for_blobs(self, blobs):
dl = [self.get_blob_availability(blob) for blob in blobs if blob]
d = defer.DeferredList(dl)
d.addCallback(lambda results: [val for success, val in results if success])
return d
@property
def last_mean_availability(self):
return max(Decimal(0.01), self._last_mean_availability)
def _update_peers_for_blob(self, blob):
def _save_peer_info(blob_hash, peers):
v = {blob_hash: peers}
self.availability.update(v)
return v
d = self._peer_finder.find_peers_for_blob(blob)
d.addCallback(lambda r: [[c.host, c.port, c.is_available()] for c in r])
d.addCallback(lambda peers: _save_peer_info(blob, peers))
return d
def _get_most_popular(self):
dl = []
for (hash, _) in self._dht_node.get_most_popular_hashes(100):
encoded = hash.encode('hex')
dl.append(self._update_peers_for_blob(encoded))
return defer.DeferredList(dl)
def _update_most_popular(self):
d = self._get_most_popular()
d.addCallback(lambda _: self._get_mean_peers())
def _update_mine(self):
def _get_peers(blobs):
dl = []
for hash in blobs:
dl.append(self._update_peers_for_blob(hash))
return defer.DeferredList(dl)
d = self._blob_manager.get_all_verified_blobs()
d.addCallback(_get_peers)
d.addCallback(lambda _: self._get_mean_peers())
def _get_mean_peers(self):
num_peers = [len(self.availability[blob]) for blob in self.availability]
mean = Decimal(sum(num_peers)) / Decimal(max(1, len(num_peers)))
self._last_mean_availability = mean
+64 -10
View File
@@ -2,6 +2,7 @@ import logging
import os
import time
import sqlite3
from twisted.internet import threads, defer
from twisted.python.failure import Failure
from twisted.enterprise import adbapi
@@ -12,7 +13,6 @@ from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.core.Error import NoSuchBlobError
from lbrynet.core.sqlite_helpers import rerun_if_locked
log = logging.getLogger(__name__)
@@ -67,6 +67,15 @@ class BlobManager(DHTHashSupplier):
def blob_paid_for(self, blob_hash, amount):
pass
def get_all_verified_blobs(self):
pass
def add_blob_to_download_history(self, blob_hash, host, rate):
pass
def add_blob_to_upload_history(self, blob_hash, host, rate):
pass
class DiskBlobManager(BlobManager):
"""This class stores blobs on the hard disk"""
@@ -78,7 +87,7 @@ class DiskBlobManager(BlobManager):
self.blob_type = BlobFile
self.blob_creator_type = BlobFileCreator
self.blobs = {}
self.blob_hashes_to_delete = {} # {blob_hash: being_deleted (True/False)}
self.blob_hashes_to_delete = {} # {blob_hash: being_deleted (True/False)}
self._next_manage_call = None
def setup(self):
@@ -177,6 +186,19 @@ class DiskBlobManager(BlobManager):
def check_consistency(self):
return self._check_consistency()
def get_all_verified_blobs(self):
d = self._get_all_verified_blob_hashes()
d.addCallback(self.completed_blobs)
return d
def add_blob_to_download_history(self, blob_hash, host, rate):
d = self._add_blob_to_download_history(blob_hash, host, rate)
return d
def add_blob_to_upload_history(self, blob_hash, host, rate):
d = self._add_blob_to_upload_history(blob_hash, host, rate)
return d
def _manage(self):
from twisted.internet import reactor
@@ -235,12 +257,29 @@ class DiskBlobManager(BlobManager):
# one that opened it. The individual connections in the pool are not used in multiple
# threads.
self.db_conn = adbapi.ConnectionPool('sqlite3', self.db_file, check_same_thread=False)
return self.db_conn.runQuery("create table if not exists blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" last_verified_time real, " +
" next_announce_time real"
")")
def create_tables(transaction):
transaction.execute("create table if not exists blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" last_verified_time real, " +
" next_announce_time real)")
transaction.execute("create table if not exists download (" +
" id integer primary key autoincrement, " +
" blob text, " +
" host text, " +
" rate float, " +
" ts integer)")
transaction.execute("create table if not exists upload (" +
" id integer primary key autoincrement, " +
" blob text, " +
" host text, " +
" rate float, " +
" ts integer)")
return self.db_conn.runInteraction(create_tables)
@rerun_if_locked
def _add_completed_blob(self, blob_hash, length, timestamp, next_announce_time=None):
@@ -418,6 +457,18 @@ class DiskBlobManager(BlobManager):
d.addCallback(lambda blobs: threads.deferToThread(get_verified_blobs, blobs))
return d
@rerun_if_locked
def _add_blob_to_download_history(self, blob_hash, host, rate):
ts = int(time.time())
d = self.db_conn.runQuery("insert into download values (null, ?, ?, ?, ?) ", (blob_hash, str(host), float(rate), ts))
return d
@rerun_if_locked
def _add_blob_to_upload_history(self, blob_hash, host, rate):
ts = int(time.time())
d = self.db_conn.runQuery("insert into upload values (null, ?, ?, ?, ?) ", (blob_hash, str(host), float(rate), ts))
return d
class TempBlobManager(BlobManager):
"""This class stores blobs in memory"""
@@ -462,6 +513,10 @@ class TempBlobManager(BlobManager):
blobs = [b.blob_hash for b in self.blobs.itervalues() if b.blob_hash in blobs_to_check and b.is_validated()]
return defer.succeed(blobs)
def get_all_verified_blobs(self):
d = self.completed_blobs(self.blobs)
return d
def hashes_to_announce(self):
now = time.time()
blobs = [blob_hash for blob_hash, announce_time in self.blob_next_announces.iteritems() if announce_time < now]
@@ -514,7 +569,6 @@ class TempBlobManager(BlobManager):
d.addCallback(lambda _: set_next_manage_call())
def _delete_blobs_marked_for_deletion(self):
def remove_from_list(b_h):
del self.blob_hashes_to_delete[b_h]
log.info("Deleted blob %s", blob_hash)
@@ -543,4 +597,4 @@ class TempBlobManager(BlobManager):
d = defer.fail(Failure(NoSuchBlobError(blob_hash)))
log.warning("Blob %s cannot be deleted because it is unknown")
ds.append(d)
return defer.DeferredList(ds)
return defer.DeferredList(ds)
+27 -1
View File
@@ -22,7 +22,17 @@ class ConnectionClosedBeforeResponseError(Exception):
pass
class KeyFeeAboveMaxAllowed(Exception):
pass
class UnknownNameError(Exception):
def __init__(self, name):
Exception.__init__('Name {} is unknown'.format(name))
self.name = name
class InvalidNameError(Exception):
def __init__(self, name):
self.name = name
@@ -75,4 +85,20 @@ class NoSuchStreamHashError(Exception):
class InvalidBlobHashError(Exception):
pass
pass
class InvalidHeaderError(Exception):
pass
class InvalidAuthenticationToken(Exception):
pass
class SubhandlerError(Exception):
pass
class NegotiationError(Exception):
pass
+2 -2
View File
@@ -8,7 +8,7 @@ from twisted.internet import interfaces, defer, threads
from twisted.protocols.basic import FileSender
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.conf import BLOB_SIZE
from lbrynet.conf import settings
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError
from lbrynet.core.cryptoutils import get_lbry_hash_obj
@@ -87,7 +87,7 @@ class HashBlob(object):
def set_length(self, length):
if self.length is not None and length == self.length:
return True
if self.length is None and 0 <= length <= BLOB_SIZE:
if self.length is None and 0 <= length <= settings.BLOB_SIZE:
self.length = length
return True
log.warning("Got an invalid length. Previous length: %s, Invalid length: %s", str(self.length), str(length))
+62
View File
@@ -0,0 +1,62 @@
from decimal import Decimal
class Offer(object):
"""A rate offer to download blobs from a host."""
RATE_ACCEPTED = "RATE_ACCEPTED"
RATE_TOO_LOW = "RATE_TOO_LOW"
RATE_UNSET = "RATE_UNSET"
def __init__(self, offer):
self._state = None
self.rate = None
if isinstance(offer, Decimal):
self.rate = round(offer, 5)
elif isinstance(offer, float):
self.rate = round(Decimal(offer), 5)
if self.rate is None or self.rate < Decimal(0.0):
self.unset()
@property
def is_accepted(self):
return self._state is Offer.RATE_ACCEPTED
@property
def is_too_low(self):
return self._state is Offer.RATE_TOO_LOW
@property
def is_unset(self):
return self._state is Offer.RATE_UNSET
@property
def message(self):
if self.is_accepted:
return Offer.RATE_ACCEPTED
elif self.is_too_low:
return Offer.RATE_TOO_LOW
elif self.is_unset:
return Offer.RATE_UNSET
return None
def accept(self):
if self.is_unset or self._state is None:
self._state = Offer.RATE_ACCEPTED
def reject(self):
if self.is_unset or self._state is None:
self._state = Offer.RATE_TOO_LOW
def unset(self):
self._state = Offer.RATE_UNSET
def handle(self, reply_message):
if reply_message == Offer.RATE_TOO_LOW:
self.reject()
elif reply_message == Offer.RATE_ACCEPTED:
self.accept()
elif reply_message == Offer.RATE_UNSET:
self.unset()
else:
raise Exception("Unknown offer reply %s" % str(reply_message))
+3 -3
View File
@@ -7,12 +7,12 @@ from Crypto.Hash import SHA512
from Crypto.PublicKey import RSA
from lbrynet.core.client.ClientRequest import ClientRequest
from lbrynet.core.Error import RequestCanceledError
from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, ILBRYWallet
from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, IWallet
from lbrynet.pointtraderclient import pointtraderclient
from twisted.internet import defer, threads
from zope.interface import implements
from twisted.python.failure import Failure
from lbrynet.core.LBRYcrdWallet import ReservedPoints
from lbrynet.core.Wallet import ReservedPoints
log = logging.getLogger(__name__)
@@ -21,7 +21,7 @@ log = logging.getLogger(__name__)
class PTCWallet(object):
"""This class sends payments to peers and also ensures that expected payments are received.
This class is only intended to be used for testing."""
implements(ILBRYWallet)
implements(IWallet)
def __init__(self, db_dir):
self.db_dir = db_dir
+54 -2
View File
@@ -1,6 +1,12 @@
from lbrynet.core.Strategy import get_default_strategy
from lbrynet.conf import settings
from decimal import Decimal
class BasePaymentRateManager(object):
def __init__(self, rate):
def __init__(self, rate=settings.data_rate, info_rate=settings.min_info_rate):
self.min_blob_data_payment_rate = rate
self.min_blob_info_payment_rate = info_rate
class PaymentRateManager(object):
@@ -26,4 +32,50 @@ class PaymentRateManager(object):
return self.min_blob_data_payment_rate
def record_points_paid(self, amount):
self.points_paid += amount
self.points_paid += amount
class NegotiatedPaymentRateManager(object):
def __init__(self, base, availability_tracker, generous=settings.is_generous_host):
"""
@param base: a BasePaymentRateManager
@param availability_tracker: a BlobAvailabilityTracker
@param rate: the min blob data payment rate
"""
self.base = base
self.points_paid = 0.0
self.blob_tracker = availability_tracker
self.generous = generous
self.strategy = get_default_strategy(self.blob_tracker,
base_price=self.base.min_blob_data_payment_rate,
is_generous=generous)
def get_rate_blob_data(self, peer, blobs):
response = self.strategy.make_offer(peer, blobs)
return response.rate
def accept_rate_blob_data(self, peer, blobs, offer):
offer = self.strategy.respond_to_offer(offer, peer, blobs)
self.strategy.update_accepted_offers(peer, offer)
return offer.is_accepted
def reply_to_offer(self, peer, blobs, offer):
reply = self.strategy.respond_to_offer(offer, peer, blobs)
self.strategy.update_accepted_offers(peer, reply)
return reply
def get_rate_for_peer(self, peer):
return self.strategy.accepted_offers.get(peer, False)
def record_points_paid(self, amount):
self.points_paid += amount
def record_offer_reply(self, peer, offer):
self.strategy.update_accepted_offers(peer, offer)
def price_limit_reached(self, peer):
if peer in self.strategy.pending_sent_offers:
offer = self.strategy.pending_sent_offers[peer]
return offer.is_too_low and round(Decimal.from_float(offer.rate), 5) >= round(self.strategy.max_rate, 5)
return False
+8 -5
View File
@@ -1,5 +1,6 @@
from collections import defaultdict
import datetime
from collections import defaultdict
from lbrynet.core import utils
class Peer(object):
@@ -12,8 +13,7 @@ class Peer(object):
self.stats = defaultdict(float) # {string stat_type, float count}
def is_available(self):
if (self.attempt_connection_at is None or
datetime.datetime.today() > self.attempt_connection_at):
if self.attempt_connection_at is None or utils.today() > self.attempt_connection_at:
return True
return False
@@ -24,7 +24,7 @@ class Peer(object):
def report_down(self):
self.down_count += 1
timeout_time = datetime.timedelta(seconds=60 * self.down_count)
self.attempt_connection_at = datetime.datetime.today() + timeout_time
self.attempt_connection_at = utils.today() + timeout_time
def update_score(self, score_change):
self.score += score_change
@@ -33,4 +33,7 @@ class Peer(object):
self.stats[stat_type] += count
def __str__(self):
return self.host + ":" + str(self.port)
return '{}:{}'.format(self.host, self.port)
def __repr__(self):
return 'Peer({!r}, {!r})'.format(self.host, self.port)
+50
View File
@@ -0,0 +1,50 @@
from zope.interface import implementer
from decimal import Decimal
from lbrynet.interfaces import IBlobPriceModel
from lbrynet.conf import settings
def get_default_price_model(blob_tracker, base_price, **kwargs):
return MeanAvailabilityWeightedPrice(blob_tracker, base_price, **kwargs)
class MeanAvailabilityWeightedPrice(object):
"""
Calculate mean-blob-availability and stream-position weighted price for a blob
Attributes:
base_price (float): base price
alpha (float): constant, > 0.0 and <= 1.0, used to more highly value blobs at the beginning of a stream.
alpha defaults to 1.0, which has a null effect
blob_tracker (BlobAvailabilityTracker): blob availability tracker
"""
implementer(IBlobPriceModel)
def __init__(self, tracker, base_price=settings.data_rate, alpha=1.0):
self.blob_tracker = tracker
self.base_price = Decimal(base_price)
self.alpha = Decimal(alpha)
def calculate_price(self, blob):
mean_availability = self.blob_tracker.last_mean_availability
availability = self.blob_tracker.availability.get(blob, [])
index = 0 # blob.index
price = self.base_price * self._get_availability_multiplier(mean_availability, availability) / self._frontload(index)
return round(price, 5)
def _get_availability_multiplier(self, mean_availability, availability):
return Decimal(max(1, mean_availability) / Decimal(max(1, len(availability))))
def _frontload(self, index):
"""
Get front-load multiplier, used to weight prices of blobs in a stream towards the front of the stream.
At index 0, returns 1.0
As index increases, return value approaches 2.0
@param index: blob position in stream
@return: front-load multiplier
"""
return Decimal(2.0) - (self.alpha ** index)
+43 -7
View File
@@ -9,14 +9,15 @@ from lbrynet.core.client.DHTPeerFinder import DHTPeerFinder
from lbrynet.core.HashAnnouncer import DummyHashAnnouncer
from lbrynet.core.server.DHTHashAnnouncer import DHTHashAnnouncer
from lbrynet.core.utils import generate_id
from lbrynet.core.PaymentRateManager import BasePaymentRateManager
from lbrynet.core.PaymentRateManager import BasePaymentRateManager, NegotiatedPaymentRateManager
from lbrynet.core.BlobAvailability import BlobAvailabilityTracker
from twisted.internet import threads, defer
log = logging.getLogger(__name__)
class LBRYSession(object):
class Session(object):
"""This class manages all important services common to any application that uses the network:
the hash announcer, which informs other peers that this peer is associated with some hash. Usually,
this means this peer has a blob identified by the hash in question, but it can be used for other
@@ -26,9 +27,9 @@ class LBRYSession(object):
the rate limiter, which attempts to ensure download and upload rates stay below a set maximum,
and upnp, which opens holes in compatible firewalls so that remote peers can connect to this peer."""
def __init__(self, blob_data_payment_rate, db_dir=None, lbryid=None, peer_manager=None, dht_node_port=None,
known_dht_nodes=None, peer_finder=None, hash_announcer=None,
blob_dir=None, blob_manager=None, peer_port=None, use_upnp=True,
rate_limiter=None, wallet=None, dht_node_class=node.Node):
known_dht_nodes=None, peer_finder=None, hash_announcer=None, blob_dir=None, blob_manager=None,
peer_port=None, use_upnp=True, rate_limiter=None, wallet=None, dht_node_class=node.Node,
blob_tracker_class=None, payment_rate_manager_class=None, is_generous=True):
"""
@param blob_data_payment_rate: The default payment rate for blob data
@@ -88,6 +89,9 @@ class LBRYSession(object):
self.blob_dir = blob_dir
self.blob_manager = blob_manager
self.blob_tracker = None
self.blob_tracker_class = blob_tracker_class or BlobAvailabilityTracker
self.peer_port = peer_port
self.use_upnp = use_upnp
@@ -103,6 +107,9 @@ class LBRYSession(object):
self.dht_node = None
self.base_payment_rate_manager = BasePaymentRateManager(blob_data_payment_rate)
self.payment_rate_manager = None
self.payment_rate_manager_class = payment_rate_manager_class or NegotiatedPaymentRateManager
self.is_generous = is_generous
def setup(self):
"""Create the blob directory and database if necessary, start all desired services"""
@@ -136,6 +143,8 @@ class LBRYSession(object):
def shut_down(self):
"""Stop all services"""
ds = []
if self.blob_manager is not None:
ds.append(defer.maybeDeferred(self.blob_tracker.stop))
if self.dht_node is not None:
ds.append(defer.maybeDeferred(self.dht_node.stop))
if self.rate_limiter is not None:
@@ -173,14 +182,23 @@ class LBRYSession(object):
self.upnp_redirects.append((self.peer_port, 'TCP'))
log.info("Set UPnP redirect for TCP port %d", self.peer_port)
else:
# see comment below
log.warning("UPnP redirect already set for TCP port %d", self.peer_port)
self.upnp_redirects.append((self.peer_port, 'TCP'))
if self.dht_node_port is not None:
if u.getspecificportmapping(self.dht_node_port, 'UDP') is None:
u.addportmapping(self.dht_node_port, 'UDP', u.lanaddr, self.dht_node_port, 'LBRY DHT port', '')
self.upnp_redirects.append((self.dht_node_port, 'UDP'))
log.info("Set UPnP redirect for UPD port %d", self.dht_node_port)
else:
# TODO: check that the existing redirect was put up by an old lbrynet session before grabbing it
# if such a disconnected redirect exists, then upnp won't work unless the redirect is appended
# or is torn down and set back up. a bad shutdown of lbrynet could leave such a redirect up
# and cause problems on the next start.
# this could be problematic if a previous lbrynet session didn't make the redirect, and it was
# made by another application
log.warning("UPnP redirect already set for UDP port %d", self.dht_node_port)
self.upnp_redirects.append((self.dht_node_port, 'UDP'))
return True
return False
@@ -244,15 +262,27 @@ class LBRYSession(object):
if self.blob_dir is None:
self.blob_manager = TempBlobManager(self.hash_announcer)
else:
self.blob_manager = DiskBlobManager(self.hash_announcer, self.blob_dir, self.db_dir)
self.blob_manager = DiskBlobManager(self.hash_announcer,
self.blob_dir,
self.db_dir)
if self.blob_tracker is None:
self.blob_tracker = self.blob_tracker_class(self.blob_manager,
self.peer_finder,
self.dht_node)
if self.payment_rate_manager is None:
self.payment_rate_manager = self.payment_rate_manager_class(self.base_payment_rate_manager,
self.blob_tracker,
self.is_generous)
self.rate_limiter.start()
d1 = self.blob_manager.setup()
d2 = self.wallet.start()
dl = defer.DeferredList([d1, d2], fireOnOneErrback=True, consumeErrors=True)
dl.addCallback(lambda _: self.blob_tracker.start())
dl.addErrback(lambda err: err.value.subFailure)
dl.addErrback(self._subfailure)
return dl
def _unset_upnp(self):
@@ -273,3 +303,9 @@ class LBRYSession(object):
d = threads.deferToThread(threaded_unset_upnp)
d.addErrback(lambda err: str(err))
return d
def _subfailure(self, err):
log.error(err.getTraceback())
return err.value
+130
View File
@@ -0,0 +1,130 @@
from zope.interface import implementer
from decimal import Decimal
from lbrynet.conf import settings
from lbrynet.interfaces import INegotiationStrategy
from lbrynet.core.Offer import Offer
from lbrynet.core.PriceModel import MeanAvailabilityWeightedPrice
def get_default_strategy(blob_tracker, **kwargs):
return BasicAvailabilityWeightedStrategy(blob_tracker, **kwargs)
class Strategy(object):
"""
Base for negotiation strategies
"""
implementer(INegotiationStrategy)
def __init__(self, price_model, max_rate, min_rate, is_generous=settings.is_generous_host):
self.price_model = price_model
self.is_generous = is_generous
self.accepted_offers = {}
self.pending_sent_offers = {}
self.offers_sent = {}
self.offers_received = {}
self.max_rate = max_rate or Decimal(self.price_model.base_price * 50)
self.min_rate = Decimal(min_rate)
def _make_rate_offer(self, rates, offer_count):
return NotImplementedError()
def _get_response_rate(self, rates, offer_count):
return NotImplementedError()
def make_offer(self, peer, blobs):
offer_count = self.offers_sent.get(peer, 0)
self._add_offer_sent(peer)
if peer in self.accepted_offers:
# if there was a previous accepted offer, use that
offer = self.accepted_offers[peer]
if peer in self.pending_sent_offers:
del self.pending_sent_offers[peer]
elif offer_count == 0 and self.is_generous:
# Try asking for it for free
offer = Offer(Decimal(0.0))
self.pending_sent_offers.update({peer: offer})
else:
rates = [self.price_model.calculate_price(blob) for blob in blobs]
price = self._make_rate_offer(rates, offer_count)
offer = Offer(price)
self.pending_sent_offers.update({peer: offer})
return offer
def respond_to_offer(self, offer, peer, blobs):
offer_count = self.offers_received.get(peer, 0)
self._add_offer_received(peer)
rates = [self.price_model.calculate_price(blob) for blob in blobs]
price = self._get_response_rate(rates, offer_count)
if peer in self.accepted_offers:
offer = self.accepted_offers[peer]
elif offer.rate == 0.0 and offer_count == 0 and self.is_generous:
# give blobs away for free by default on the first request
offer.accept()
self.accepted_offers.update({peer: offer})
elif offer.rate >= price:
offer.accept()
self.accepted_offers.update({peer: offer})
else:
offer.reject()
if peer in self.accepted_offers:
del self.accepted_offers[peer]
return offer
def update_accepted_offers(self, peer, offer):
if not offer.is_accepted and peer in self.accepted_offers:
del self.accepted_offers[peer]
if offer.is_accepted:
self.accepted_offers.update({peer: offer})
self.pending_sent_offers.update({peer: offer})
def _add_offer_sent(self, peer):
turn = self.offers_sent.get(peer, 0) + 1
self.offers_sent.update({peer: turn})
def _add_offer_received(self, peer):
turn = self.offers_received.get(peer, 0) + 1
self.offers_received.update({peer: turn})
def _bounded_price(self, price):
price_for_return = Decimal(min(self.max_rate, max(price, self.min_rate)))
return price_for_return
class BasicAvailabilityWeightedStrategy(Strategy):
"""
Basic strategy to target blob prices based on supply relative to mean supply
Discount price target with each incoming request, and raise it with each outgoing from the modeled price
until the rate is accepted or a threshold is reached
"""
implementer(INegotiationStrategy)
def __init__(self, blob_tracker, acceleration=1.25, deceleration=0.9, max_rate=None, min_rate=0.0,
is_generous=settings.is_generous_host, base_price=0.0001, alpha=1.0):
price_model = MeanAvailabilityWeightedPrice(blob_tracker, base_price=base_price, alpha=alpha)
Strategy.__init__(self, price_model, max_rate, min_rate, is_generous)
self._acceleration = Decimal(acceleration) # rate of how quickly to ramp offer
self._deceleration = Decimal(deceleration)
def _get_mean_rate(self, rates):
mean_rate = Decimal(sum(rates)) / Decimal(max(len(rates), 1))
return mean_rate
def _premium(self, rate, turn):
return rate * (self._acceleration ** Decimal(turn))
def _discount(self, rate, turn):
return rate * (self._deceleration ** Decimal(turn))
def _get_response_rate(self, rates, offer_count):
rate = self._get_mean_rate(rates)
discounted = self._discount(rate, offer_count)
rounded_price = round(discounted, 5)
return self._bounded_price(rounded_price)
def _make_rate_offer(self, rates, offer_count):
rate = self._get_mean_rate(rates)
with_premium = self._premium(rate, offer_count)
rounded_price = round(with_premium, 5)
return self._bounded_price(rounded_price)
+2 -2
View File
@@ -198,7 +198,7 @@ class StreamDescriptorIdentifier(object):
return self._stream_downloader_factories[stream_type]
def _get_validator(self, stream_type):
if not stream_type in self._stream_downloader_factories:
if not stream_type in self._sd_info_validators:
raise UnknownStreamTypeError(stream_type)
return self._sd_info_validators[stream_type]
@@ -238,4 +238,4 @@ def download_sd_blob(session, blob_hash, payment_rate_manager):
"""
downloader = StandaloneBlobDownloader(blob_hash, session.blob_manager, session.peer_finder,
session.rate_limiter, payment_rate_manager, session.wallet)
return downloader.download()
return downloader.download()
@@ -1,23 +1,4 @@
import sys
from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, ILBRYWallet
from lbrynet.core.client.ClientRequest import ClientRequest
from lbrynet.core.Error import UnknownNameError, InvalidStreamInfoError, RequestCanceledError
from lbrynet.core.Error import InsufficientFundsError
from lbrynet.core.sqlite_helpers import rerun_if_locked
from lbryum import SimpleConfig, Network
from lbryum.bitcoin import COIN, TYPE_ADDRESS
from lbryum.wallet import WalletStorage, Wallet
from lbryum.commands import known_commands, Commands
from lbryum.transaction import Transaction
from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException
from twisted.internet import threads, reactor, defer, task
from twisted.python.failure import Failure
from twisted.enterprise import adbapi
from collections import defaultdict, deque
from zope.interface import implements
from decimal import Decimal
import datetime
import logging
import json
@@ -26,6 +7,27 @@ import socket
import time
import os
from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException
from twisted.internet import threads, reactor, defer, task
from twisted.python.failure import Failure
from twisted.enterprise import adbapi
from collections import defaultdict, deque
from zope.interface import implements
from jsonschema import ValidationError
from decimal import Decimal
from lbryum import SimpleConfig, Network
from lbryum.lbrycrd import COIN
import lbryum.wallet
from lbryum.commands import known_commands, Commands
from lbryum.transaction import Transaction
from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, IWallet
from lbrynet.core.client.ClientRequest import ClientRequest
from lbrynet.core.Error import UnknownNameError, InvalidStreamInfoError, RequestCanceledError
from lbrynet.core.Error import InsufficientFundsError
from lbrynet.metadata.Metadata import Metadata
log = logging.getLogger(__name__)
alert = logging.getLogger("lbryalert." + __name__)
@@ -46,16 +48,15 @@ def _catch_connection_error(f):
return w
class LBRYWallet(object):
"""This class implements the LBRYWallet interface for the LBRYcrd payment system"""
implements(ILBRYWallet)
class Wallet(object):
"""This class implements the Wallet interface for the LBRYcrd payment system"""
implements(IWallet)
_FIRST_RUN_UNKNOWN = 0
_FIRST_RUN_YES = 1
_FIRST_RUN_NO = 2
def __init__(self, db_dir):
self.db_dir = db_dir
self.db = None
self.next_manage_call = None
@@ -86,6 +87,7 @@ class LBRYWallet(object):
return True
d = self._open_db()
d.addCallback(lambda _: self._clean_bad_records())
d.addCallback(lambda _: self._start())
d.addCallback(lambda _: start_manage())
return d
@@ -97,6 +99,7 @@ class LBRYWallet(object):
def stop(self):
self.stopped = True
# If self.next_manage_call is None, then manage is currently running or else
# start has not been called, so set stopped and do nothing else.
if self.next_manage_call is not None:
@@ -150,7 +153,12 @@ class LBRYWallet(object):
log.info("Got a new balance: %s", str(balance))
self.wallet_balance = balance
d.addCallback(set_wallet_balance)
def log_error(err):
if isinstance(err, AttributeError):
log.warning("Failed to get an updated balance")
log.warning("Last balance update: %s", str(self.wallet_balance))
d.addCallbacks(set_wallet_balance, log_error)
return d
d.addCallback(lambda should_run: do_manage() if should_run else None)
@@ -277,17 +285,22 @@ class LBRYWallet(object):
def _send_payments(self):
payments_to_send = {}
for address, points in self.queued_payments.items():
log.info("Should be sending %s points to %s", str(points), str(address))
payments_to_send[address] = points
self.total_reserved_points -= points
self.wallet_balance -= points
if points > 0:
log.info("Should be sending %s points to %s", str(points), str(address))
payments_to_send[address] = points
self.total_reserved_points -= points
self.wallet_balance -= points
else:
log.info("Skipping dust")
del self.queued_payments[address]
if payments_to_send:
log.info("Creating a transaction with outputs %s", str(payments_to_send))
d = self._do_send_many(payments_to_send)
d.addCallback(lambda txid: log.debug("Sent transaction %s", txid))
return d
log.info("There were no payments to send")
log.debug("There were no payments to send")
return defer.succeed(True)
def get_stream_info_for_name(self, name):
@@ -315,75 +328,156 @@ class LBRYWallet(object):
return d
def _get_stream_info_from_value(self, result, name):
r_dict = {}
if 'value' in result:
value = result['value']
try:
value_dict = json.loads(value)
except (ValueError, TypeError):
return Failure(InvalidStreamInfoError(name))
known_fields = ['stream_hash', 'name', 'description', 'key_fee', 'key_fee_address', 'thumbnail',
'content_license', 'sources', 'fee', 'author']
known_sources = ['lbry_sd_hash', 'btih', 'url']
known_fee_types = {'LBC': ['amount', 'address']}
for field in known_fields:
if field in value_dict:
if field == 'sources':
for source in known_sources:
if source in value_dict[field]:
if source == 'lbry_sd_hash':
r_dict['stream_hash'] = value_dict[field][source]
else:
r_dict[source] = value_dict[field][source]
elif field == 'fee':
fee = value_dict['fee']
if 'type' in fee:
if fee['type'] in known_fee_types:
fee_fields = known_fee_types[fee['type']]
if all([f in fee for f in fee_fields]):
r_dict['key_fee'] = fee['amount']
r_dict['key_fee_address'] = fee['address']
else:
for f in ['key_fee', 'key_fee_address']:
if f in r_dict:
del r_dict[f]
else:
r_dict[field] = value_dict[field]
if 'stream_hash' in r_dict and 'txid' in result:
d = self._save_name_metadata(name, r_dict['stream_hash'], str(result['txid']))
else:
d = defer.succeed(True)
d.addCallback(lambda _: r_dict)
return d
elif 'error' in result:
def _check_result_fields(r):
for k in ['value', 'txid', 'n', 'height', 'amount']:
assert k in r, "getvalueforname response missing field %s" % k
def _log_success(claim_id):
log.info("lbry://%s complies with %s, claimid: %s", name, metadata.version, claim_id)
return defer.succeed(None)
if 'error' in result:
log.warning("Got an error looking up a name: %s", result['error'])
return Failure(UnknownNameError(name))
return Failure(UnknownNameError(name))
def claim_name(self, name, sd_hash, amount, description=None, key_fee=None,
key_fee_address=None, thumbnail=None, content_license=None, author=None, sources=None):
value = {"sources": {'lbry_sd_hash': sd_hash}}
if description is not None:
value['description'] = description
if key_fee is not None and key_fee_address is not None:
value['fee'] = {'type': 'LBC', 'amount': key_fee, 'address': key_fee_address}
if thumbnail is not None:
value['thumbnail'] = thumbnail
if content_license is not None:
value['content_license'] = content_license
if author is not None:
value['author'] = author
if isinstance(sources, dict):
sources['lbry_sd_hash'] = sd_hash
value['sources'] = sources
_check_result_fields(result)
d = self._send_name_claim(name, json.dumps(value), amount)
try:
metadata = Metadata(json.loads(result['value']))
except ValidationError:
return Failure(InvalidStreamInfoError(name))
def _save_metadata(txid):
d = self._save_name_metadata(name, sd_hash, txid)
txid = result['txid']
sd_hash = metadata['sources']['lbry_sd_hash']
d = self._save_name_metadata(name, txid, sd_hash)
d.addCallback(lambda _: self.get_claimid(name, txid))
d.addCallback(lambda cid: _log_success(cid))
d.addCallback(lambda _: metadata)
return d
def get_claim(self, name, claim_id):
d = self.get_claims_for_name(name)
d.addCallback(lambda claims: next(claim for claim in claims['claims'] if claim['claimId'] == claim_id))
return d
def get_claimid(self, name, txid):
def _get_id_for_return(claim_id):
if claim_id:
return defer.succeed(claim_id)
else:
d = self.get_claims_from_tx(txid)
d.addCallback(lambda claims: next(c['claimId'] for c in claims if c['name'] == name))
d.addCallback(lambda cid: self._update_claimid(cid, name, txid))
return d
d = self._get_claimid_for_tx(name, txid)
d.addCallback(_get_id_for_return)
return d
def get_my_claim(self, name):
def _convert_units(claim):
amount = Decimal(claim['nEffectiveAmount'] / COIN)
claim['nEffectiveAmount'] = amount
return claim
def _get_claim_for_return(claim):
if not claim:
return False
d = self.get_claim(name, claim['claim_id'])
d.addCallback(_convert_units)
d.addCallback(lambda clm: self._format_claim_for_return(name, clm, claim['txid']))
return d
def _get_my_unspent_claim(claims):
for claim in claims:
if claim['name'] == name and not claim['is spent'] and not claim.get('supported_claimid'):
return claim
return False
d = self.get_name_claims()
d.addCallback(_get_my_unspent_claim)
d.addCallback(_get_claim_for_return)
return d
def get_claim_info(self, name, txid=None):
if not txid:
d = self._get_value_for_name(name)
d.addCallback(lambda r: self._get_claim_info(name, r['txid']))
else:
d = self._get_claim_info(name, txid)
d.addErrback(lambda _: False)
return d
def _format_claim_for_return(self, name, claim, txid, metadata=None, meta_version=None):
result = {}
result['claim_id'] = claim['claimId']
result['amount'] = claim['nEffectiveAmount']
result['height'] = claim['nHeight']
result['name'] = name
result['txid'] = txid
result['value'] = metadata if metadata else json.loads(claim['value'])
result['supports'] = [{'txid': support['txid'], 'n': support['n']} for support in claim['supports']]
result['meta_version'] = meta_version if meta_version else result['value'].get('ver', '0.0.1')
return result
def _get_claim_info(self, name, txid):
def _build_response(claim):
try:
metadata = Metadata(json.loads(claim['value']))
meta_ver = metadata.version
sd_hash = metadata['sources']['lbry_sd_hash']
d = self._save_name_metadata(name, txid, sd_hash)
except ValidationError:
metadata = claim['value']
meta_ver = "Non-compliant"
d = defer.succeed(None)
d.addCallback(lambda _: self._format_claim_for_return(name, claim, txid,
metadata=metadata, meta_version=meta_ver))
log.info("get claim info lbry://%s metadata: %s, claimid: %s", name, meta_ver, claim['claimId'])
return d
d = self.get_claimid(name, txid)
d.addCallback(lambda claim_id: self.get_claim(name, claim_id))
d.addCallback(_build_response)
return d
def get_claims_for_name(self, name):
d = self._get_claims_for_name(name)
return d
def update_metadata(self, new_metadata, old_metadata):
meta_for_return = old_metadata if isinstance(old_metadata, dict) else {}
for k in new_metadata:
meta_for_return[k] = new_metadata[k]
return defer.succeed(Metadata(meta_for_return))
def claim_name(self, name, bid, m):
def _save_metadata(txid, metadata):
log.info("Saving metadata for claim %s" % txid)
d = self._save_name_metadata(name, txid, metadata['sources']['lbry_sd_hash'])
d.addCallback(lambda _: txid)
return d
d.addCallback(_save_metadata)
def _claim_or_update(claim, metadata, _bid):
if not claim:
log.info("No claim yet, making a new one")
return self._send_name_claim(name, metadata, _bid)
if not claim['is_mine']:
log.info("Making a contesting claim")
return self._send_name_claim(name, metadata, _bid)
else:
log.info("Updating over own claim")
d = self.update_metadata(metadata, claim['value'])
d.addCallback(lambda new_metadata: self._send_name_claim_update(name, claim['claim_id'], claim['txid'], new_metadata, _bid))
return d
meta = Metadata(m)
d = self.get_claim_info(name)
d.addCallback(lambda claim: _claim_or_update(claim, meta, bid))
d.addCallback(lambda txid: _save_metadata(txid, meta))
return d
def abandon_name(self, txid):
@@ -409,7 +503,7 @@ class LBRYWallet(object):
def abandon(results):
if results[0][0] and results[1][0]:
address = results[0][1]
amount = results[1][1]
amount = float(results[1][1])
return self._send_abandon(txid, address, amount)
elif results[0][0] is False:
return defer.fail(Failure(ValueError("Couldn't get a new address")))
@@ -419,13 +513,51 @@ class LBRYWallet(object):
dl.addCallback(abandon)
return dl
def support_claim(self, name, claim_id, amount):
return self._support_claim(name, claim_id, amount)
def get_tx(self, txid):
d = self._get_raw_tx(txid)
d.addCallback(self._get_decoded_tx)
return d
# def update_name(self, name_value):
# return self._update_name(name_value)
def get_block_info(self, height):
d = self._get_blockhash(height)
return d
def get_history(self):
d = self._get_history()
return d
def address_is_mine(self, address):
d = self._address_is_mine(address)
return d
def get_tx_json(self, txid):
def _decode(raw_tx):
tx = Transaction(raw_tx).deserialize()
decoded_tx = {}
for txkey in tx.keys():
if isinstance(tx[txkey], list):
decoded_tx[txkey] = []
for i in tx[txkey]:
tmp = {}
for k in i.keys():
if isinstance(i[k], Decimal):
tmp[k] = float(i[k] / 1e8)
else:
tmp[k] = i[k]
decoded_tx[txkey].append(tmp)
else:
decoded_tx[txkey] = tx[txkey]
return decoded_tx
d = self._get_raw_tx(txid)
d.addCallback(_decode)
return d
def get_claim_metadata_for_sd_hash(self, sd_hash):
return self._get_claim_metadata_for_sd_hash(sd_hash)
def get_name_and_validity_for_sd_hash(self, sd_hash):
d = self._get_claim_metadata_for_sd_hash(sd_hash)
@@ -529,21 +661,45 @@ class LBRYWallet(object):
def _open_db(self):
self.db = adbapi.ConnectionPool('sqlite3', os.path.join(self.db_dir, "blockchainname.db"),
check_same_thread=False)
return self.db.runQuery("create table if not exists name_metadata (" +
def create_tables(transaction):
transaction.execute("create table if not exists name_metadata (" +
" name text, " +
" txid text, " +
" sd_hash text)")
transaction.execute("create table if not exists claim_ids (" +
" claimId text, " +
" name text, " +
" txid text)")
def _save_name_metadata(self, name, sd_hash, txid):
d = self.db.runQuery("select * from name_metadata where txid=?", (txid,))
d.addCallback(lambda r: self.db.runQuery("insert into name_metadata values (?, ?, ?)", (name, txid, sd_hash))
if not len(r) else None)
return self.db.runInteraction(create_tables)
def _clean_bad_records(self):
d = self.db.runQuery("delete from name_metadata where length(txid) > 64 or txid is null")
return d
def _save_name_metadata(self, name, txid, sd_hash):
assert len(txid) == 64, "That's not a txid: %s" % str(txid)
d = self.db.runQuery("delete from name_metadata where name=? and txid=? and sd_hash=?", (name, txid, sd_hash))
d.addCallback(lambda _: self.db.runQuery("insert into name_metadata values (?, ?, ?)", (name, txid, sd_hash)))
return d
def _get_claim_metadata_for_sd_hash(self, sd_hash):
d = self.db.runQuery("select name, txid from name_metadata where sd_hash=?", (sd_hash,))
d.addCallback(lambda r: r[0] if len(r) else None)
d.addCallback(lambda r: r[0] if r else None)
return d
def _update_claimid(self, claim_id, name, txid):
assert len(txid) == 64, "That's not a txid: %s" % str(txid)
d = self.db.runQuery("delete from claim_ids where claimId=? and name=? and txid=?", (claim_id, name, txid))
d.addCallback(lambda r: self.db.runQuery("insert into claim_ids values (?, ?, ?)", (claim_id, name, txid)))
d.addCallback(lambda _: claim_id)
return d
def _get_claimid_for_tx(self, name, txid):
assert len(txid) == 64, "That's not a txid: %s" % str(txid)
d = self.db.runQuery("select claimId from claim_ids where name=? and txid=?", (name, txid))
d.addCallback(lambda r: r[0][0] if r else None)
return d
######### Must be overridden #########
@@ -566,6 +722,9 @@ class LBRYWallet(object):
def get_name_claims(self):
return defer.fail(NotImplementedError())
def _get_claims_for_name(self, name):
return defer.fail(NotImplementedError())
def _check_first_run(self):
return defer.fail(NotImplementedError())
@@ -581,6 +740,12 @@ class LBRYWallet(object):
def _send_abandon(self, txid, address, amount):
return defer.fail(NotImplementedError())
def _send_name_claim_update(self, name, claim_id, txid, value, amount):
return defer.fail(NotImplementedError())
def _support_claim(self, name, claim_id, amount):
return defer.fail(NotImplementedError())
def _do_send_many(self, payments_to_send):
return defer.fail(NotImplementedError())
@@ -593,6 +758,12 @@ class LBRYWallet(object):
def _get_balance_for_address(self, address):
return defer.fail(NotImplementedError())
def _get_history(self):
return defer.fail(NotImplementedError())
def _address_is_mine(self, address):
return defer.fail(NotImplementedError())
def _start(self):
pass
@@ -600,9 +771,9 @@ class LBRYWallet(object):
pass
class LBRYcrdWallet(LBRYWallet):
class LBRYcrdWallet(Wallet):
def __init__(self, db_dir, wallet_dir=None, wallet_conf=None, lbrycrdd_path=None):
LBRYWallet.__init__(self, db_dir)
Wallet.__init__(self, db_dir)
self.started_lbrycrdd = False
self.wallet_dir = wallet_dir
self.wallet_conf = wallet_conf
@@ -634,8 +805,8 @@ class LBRYcrdWallet(LBRYWallet):
def _get_rpc_conf(self):
settings = {"username": "rpcuser",
"password": "rpcpassword",
"rpc_port": 8332}
if os.path.exists(self.wallet_conf):
"rpc_port": 9245}
if self.wallet_conf and os.path.exists(self.wallet_conf):
conf = open(self.wallet_conf)
for l in conf:
if l.startswith("rpcuser="):
@@ -713,12 +884,30 @@ class LBRYcrdWallet(LBRYWallet):
def _send_abandon(self, txid, address, amount):
return threads.deferToThread(self._send_abandon_rpc, txid, address, amount)
def _send_name_claim_update(self, name, claim_id, txid, value, amount):
return threads.deferToThread(self._update_name_rpc, txid, value, amount)
def _support_claim(self, name, claim_id, amount):
return threads.deferToThread(self._support_claim_rpc, name, claim_id, amount)
def _get_claims_for_name(self, name):
return threads.deferToThread(self._get_claims_for_name_rpc, name)
def get_claims_from_tx(self, txid):
return threads.deferToThread(self._get_claims_from_tx_rpc, txid)
def _get_blockhash(self, blockhash):
return threads.deferToThread(self._get_blockhash_rpc, blockhash)
def _get_value_for_name(self, name):
return threads.deferToThread(self._get_value_for_name_rpc, name)
def _get_history(self):
return threads.deferToThread(self._list_transactions_rpc)
def _address_is_mine(self, address):
return threads.deferToThread(self._get_address_is_mine_rpc, address)
def _get_rpc_conn(self):
return AuthServiceProxy(self.rpc_conn_string)
@@ -727,7 +916,11 @@ class LBRYcrdWallet(LBRYWallet):
tries = 0
try:
rpc_conn = self._get_rpc_conn()
rpc_conn.getinfo()
try:
rpc_conn.getinfo()
except ValueError:
log.exception('Failed to get rpc info. Rethrowing with a hopefully more useful error message')
raise Exception('Failed to get rpc info from lbrycrdd. Try restarting lbrycrdd')
log.info("lbrycrdd was already running when LBRYcrdWallet was started.")
return
except (socket.error, JSONRPCException):
@@ -826,7 +1019,7 @@ class LBRYcrdWallet(LBRYWallet):
@_catch_connection_error
def _send_abandon_rpc(self, txid, address, amount):
rpc_conn = self._get_rpc_conn()
return rpc_conn.abandonname(txid, address, amount)
return rpc_conn.abandonclaim(txid, address, amount)
@_catch_connection_error
def _get_blockchain_info_rpc(self):
@@ -838,15 +1031,25 @@ class LBRYcrdWallet(LBRYWallet):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getblock(blockhash)
@_catch_connection_error
def _get_blockhash_rpc(self, height):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getblockhash(height)
@_catch_connection_error
def _get_claims_from_tx_rpc(self, txid):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getclaimsfortx(txid)
@_catch_connection_error
def _get_claims_for_name_rpc(self, name):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getclaimsforname(name)
@_catch_connection_error
def _get_nametrie_rpc(self):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getnametrie()
return rpc_conn.getclaimtrie()
@_catch_connection_error
def _get_wallet_balance_rpc(self):
@@ -863,21 +1066,27 @@ class LBRYcrdWallet(LBRYWallet):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getvalueforname(name)
# def _update_name_rpc(self, name_value):
# rpc_conn = self._get_rpc_conn()
# return rpc_conn.updatename(name_value)
@_catch_connection_error
def _update_name_rpc(self, txid, value, amount):
rpc_conn = self._get_rpc_conn()
return rpc_conn.updateclaim(txid, json.dumps(value), amount)
@_catch_connection_error
def _send_name_claim_rpc(self, name, value, amount):
rpc_conn = self._get_rpc_conn()
try:
return str(rpc_conn.claimname(name, value, amount))
return str(rpc_conn.claimname(name, json.dumps(value), amount))
except JSONRPCException as e:
if 'message' in e.error and e.error['message'] == "Insufficient funds":
raise InsufficientFundsError()
elif 'message' in e.error:
raise ValueError(e.error['message'])
@_catch_connection_error
def _support_claim_rpc(self, name, claim_id, amount):
rpc_conn = self._get_rpc_conn()
return rpc_conn.supportclaim(name, claim_id, amount)
@_catch_connection_error
def _get_num_addresses_rpc(self):
rpc_conn = self._get_rpc_conn()
@@ -888,6 +1097,16 @@ class LBRYcrdWallet(LBRYWallet):
rpc_conn = self._get_rpc_conn()
return rpc_conn.getbestblockhash()
@_catch_connection_error
def _list_transactions_rpc(self):
rpc_conn = self._get_rpc_conn()
return rpc_conn.listtransactions()
@_catch_connection_error
def _get_address_is_mine_rpc(self, address):
rpc_conn = self._get_rpc_conn()
return address in rpc_conn.getaddressesbyaccount("")
@_catch_connection_error
def _stop_rpc(self):
# check if our lbrycrdd is actually running, or if we connected to one that was already
@@ -898,11 +1117,11 @@ class LBRYcrdWallet(LBRYWallet):
self.lbrycrdd.wait()
class LBRYumWallet(LBRYWallet):
class LBRYumWallet(Wallet):
def __init__(self, db_dir):
LBRYWallet.__init__(self, db_dir)
self.config = None
def __init__(self, db_dir, config=None):
Wallet.__init__(self, db_dir)
self._config = config
self.network = None
self.wallet = None
self.cmd_runner = None
@@ -921,7 +1140,7 @@ class LBRYumWallet(LBRYWallet):
network_start_d = defer.Deferred()
def setup_network():
self.config = SimpleConfig()
self.config = make_config(self._config)
self.network = Network(self.config)
alert.info("Loading the wallet...")
return defer.succeed(self.network.start())
@@ -955,7 +1174,8 @@ class LBRYumWallet(LBRYWallet):
self._start_check = None
if self._catch_up_check is not None:
self._catch_up_check.stop()
if self._catch_up_check.running:
self._catch_up_check.stop()
self._catch_up_check = None
d = defer.Deferred()
@@ -979,8 +1199,8 @@ class LBRYumWallet(LBRYWallet):
def get_wallet():
path = self.config.get_wallet_path()
storage = WalletStorage(path)
wallet = Wallet(storage)
storage = lbryum.wallet.WalletStorage(path)
wallet = lbryum.wallet.Wallet(storage)
if not storage.file_exists:
self.first_run = True
seed = wallet.make_seed()
@@ -993,7 +1213,7 @@ class LBRYumWallet(LBRYWallet):
blockchain_caught_d = defer.Deferred()
def check_caught_up():
local_height = self.network.get_local_height()
local_height = self.network.get_catchup_progress()
remote_height = self.network.get_server_height()
if remote_height != 0 and remote_height - local_height <= 5:
@@ -1027,6 +1247,9 @@ class LBRYumWallet(LBRYWallet):
self._caught_up_counter += 1
def log_error(err):
log.warning(err.getErrorMessage())
return defer.fail(err)
self._catch_up_check = task.LoopingCall(check_caught_up)
@@ -1034,6 +1257,7 @@ class LBRYumWallet(LBRYWallet):
d.addCallback(self._save_wallet)
d.addCallback(lambda _: self.wallet.start_threads(self.network))
d.addCallback(lambda _: self._catch_up_check.start(.1))
d.addErrback(log_error)
d.addCallback(lambda _: blockchain_caught_d)
return d
@@ -1043,9 +1267,10 @@ class LBRYumWallet(LBRYWallet):
def get_balance(self):
cmd = known_commands['getbalance']
func = getattr(self.cmd_runner, cmd.name)
d = threads.deferToThread(func)
d.addCallback(lambda result: result['unmatured'] if 'unmatured' in result else result['confirmed'])
d.addCallback(Decimal)
accounts = None
exclude_claimtrietx = True
d = threads.deferToThread(func, accounts, exclude_claimtrietx)
d.addCallback(lambda result: Decimal(result['confirmed']) + Decimal(result.get('unconfirmed', 0.0)))
return d
def get_new_address(self):
@@ -1068,6 +1293,11 @@ class LBRYumWallet(LBRYWallet):
d.addCallback(lambda header: self.network.blockchain.hash_header(header))
return d
def _get_blockhash(self, height):
d = threads.deferToThread(self.network.blockchain.read_header, height)
d.addCallback(lambda header: self.network.blockchain.hash_header(header))
return d
def get_name_claims(self):
cmd = known_commands['getnameclaims']
func = getattr(self.cmd_runner, cmd.name)
@@ -1085,12 +1315,31 @@ class LBRYumWallet(LBRYWallet):
def send_claim(address):
cmd = known_commands['claimname']
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func, address, amount, name, val)
return threads.deferToThread(func, address, amount, name, json.dumps(val))
d = self.get_new_address()
d.addCallback(send_claim)
d.addCallback(self._broadcast_transaction)
return d
def _get_claims_for_name(self, name):
cmd = known_commands['getclaimsforname']
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func, name)
def _send_name_claim_update(self, name, claim_id, txid, value, amount):
def send_claim_update(address):
decoded_claim_id = claim_id.decode('hex')[::-1]
metadata = json.dumps(value)
log.info("updateclaim %s %s %f %s %s '%s'", txid, address, amount, name, decoded_claim_id.encode('hex'), metadata)
cmd = known_commands['updateclaim']
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func, txid, address, amount, name, decoded_claim_id, metadata)
d = self.get_new_address()
d.addCallback(send_claim_update)
d.addCallback(self._broadcast_transaction)
return d
def _get_decoded_tx(self, raw_tx):
tx = Transaction(raw_tx)
decoded_tx = {}
@@ -1102,28 +1351,42 @@ class LBRYumWallet(LBRYWallet):
return decoded_tx
def _send_abandon(self, txid, address, amount):
log.info("Abandon " + str(txid) + " " + str(address) + " " + str(amount))
log.info("Abandon %s %s %f" % (txid, address, amount))
cmd = known_commands['abandonclaim']
func = getattr(self.cmd_runner, cmd.name)
d = threads.deferToThread(func, txid, address, amount)
d.addCallback(self._broadcast_transaction)
return d
def _support_claim(self, name, claim_id, amount):
def _send_support(d, a, n, c):
cmd = known_commands['supportclaim']
func = getattr(self.cmd_runner, cmd.name)
d = threads.deferToThread(func, d, a, n, c)
return d
d = self.get_new_address()
d.addCallback(lambda address: _send_support(address, amount, name, claim_id))
d.addCallback(self._broadcast_transaction)
return d
def _broadcast_transaction(self, raw_tx):
log.info("Broadcast: " + str(raw_tx))
def _log_tx(r):
log.info("Broadcast tx: %s", r)
return r
cmd = known_commands['broadcast']
func = getattr(self.cmd_runner, cmd.name)
d = threads.deferToThread(func, raw_tx)
d.addCallback(_log_tx)
d.addCallback(
lambda r: r if len(r) == 64 else defer.fail(Exception("Transaction rejected")))
d.addCallback(self._save_wallet)
return d
def _do_send_many(self, payments_to_send):
log.warning("Doing send many. payments to send: %s", str(payments_to_send))
outputs = [(TYPE_ADDRESS, address, int(amount*COIN)) for address, amount in payments_to_send.iteritems()]
d = threads.deferToThread(self.wallet.mktx, outputs, None, self.config)
d.addCallback(lambda tx: threads.deferToThread(self.wallet.sendtx, tx))
d.addCallback(self._save_wallet)
return d
cmd = known_commands['paytomanyandsend']
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func, payments_to_send.iteritems())
def _get_value_for_name(self, name):
cmd = known_commands['getvalueforname']
@@ -1143,33 +1406,15 @@ class LBRYumWallet(LBRYWallet):
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func)
def get_history(self):
def _get_history(self):
cmd = known_commands['history']
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func)
def get_tx_json(self, txid):
def _decode(raw_tx):
tx = Transaction(raw_tx).deserialize()
decoded_tx = {}
for txkey in tx.keys():
if isinstance(tx[txkey], list):
decoded_tx[txkey] = []
for i in tx[txkey]:
tmp = {}
for k in i.keys():
if isinstance(i[k], Decimal):
tmp[k] = float(i[k] / 1e8)
else:
tmp[k] = i[k]
decoded_tx[txkey].append(tmp)
else:
decoded_tx[txkey] = tx[txkey]
return decoded_tx
d = self._get_raw_tx(txid)
d.addCallback(_decode)
return d
def _address_is_mine(self, address):
cmd = known_commands['ismine']
func = getattr(self.cmd_runner, cmd.name)
return threads.deferToThread(func, address)
def get_pub_keys(self, wallet):
cmd = known_commands['getpubkeys']
@@ -1268,4 +1513,10 @@ class LBRYcrdAddressQueryHandler(object):
log.warning("Expected a request for an address, but did not receive one")
return defer.fail(Failure(ValueError("Expected but did not receive an address request")))
else:
return defer.succeed({})
return defer.succeed({})
def make_config(config=None):
if config is None:
config = {}
return SimpleConfig(config) if type(config) == type({}) else config
+448 -199
View File
@@ -1,18 +1,43 @@
from collections import defaultdict
import logging
from collections import defaultdict
from decimal import Decimal
from twisted.internet import defer
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.core.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
from lbrynet.core.Error import InvalidResponseError, RequestCanceledError, NoResponseError
from lbrynet.core.Error import ConnectionClosedBeforeResponseError
from lbrynet.core.Error import InvalidResponseError, RequestCanceledError, NoResponseError
from lbrynet.core.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
from lbrynet.core.client.ClientRequest import ClientRequest, ClientBlobRequest
from lbrynet.interfaces import IRequestCreator
from lbrynet.core.Offer import Offer
log = logging.getLogger(__name__)
def get_points(num_bytes, rate):
if isinstance(rate, float):
return 1.0 * num_bytes * rate / 2**20
elif isinstance(rate, Decimal):
return 1.0 * num_bytes * float(rate) / 2**20
else:
raise Exception("Unknown rate type")
def cache(fn):
"""Caches the function call for each instance"""
attr = '__{}_value'.format(fn.__name__)
def helper(self):
if not hasattr(self, attr):
value = fn(self)
setattr(self, attr, value)
return getattr(self, attr)
return helper
class BlobRequester(object):
implements(IRequestCreator)
@@ -26,60 +51,18 @@ class BlobRequester(object):
self._available_blobs = defaultdict(list) # {Peer: [blob_hash]}
self._unavailable_blobs = defaultdict(list) # {Peer: [blob_hash]}}
self._protocol_prices = {} # {ClientProtocol: price}
self._protocol_offers = {}
self._price_disagreements = [] # [Peer]
self._protocol_tries = {}
self._maxed_out_peers = []
self._incompatible_peers = []
######## IRequestCreator #########
def send_next_request(self, peer, protocol):
sent_request = False
if self._blobs_to_download() and self._should_send_request_to(peer):
a_r = self._get_availability_request(peer)
d_r = self._get_download_request(peer)
p_r = None
if a_r is not None or d_r is not None:
p_r = self._get_price_request(peer, protocol)
if a_r is not None:
d1 = protocol.add_request(a_r)
d1.addCallback(self._handle_availability, peer, a_r)
d1.addErrback(self._request_failed, "availability request", peer)
sent_request = True
if d_r is not None:
reserved_points = self._reserve_points(peer, protocol, d_r.max_pay_units)
if reserved_points is not None:
# Note: The following three callbacks will be called when the blob has been
# fully downloaded or canceled
d_r.finished_deferred.addCallbacks(self._download_succeeded, self._download_failed,
callbackArgs=(peer, d_r.blob),
errbackArgs=(peer,))
d_r.finished_deferred.addBoth(self._pay_or_cancel_payment, protocol, reserved_points, d_r.blob)
d_r.finished_deferred.addErrback(self._handle_download_error, peer, d_r.blob)
d2 = protocol.add_blob_request(d_r)
# Note: The following two callbacks will be called as soon as the peer sends its
# response, which will be before the blob has finished downloading, but may be
# after the blob has been canceled. For example,
# 1) client sends request to Peer A
# 2) the blob is finished downloading from peer B, and therefore this one is canceled
# 3) client receives response from Peer A
# Therefore, these callbacks shouldn't rely on there being a blob about to be
# downloaded.
d2.addCallback(self._handle_incoming_blob, peer, d_r)
d2.addErrback(self._request_failed, "download request", peer)
sent_request = True
else:
d_r.cancel(InsufficientFundsError())
d_r.finished_deferred.addErrback(lambda _: True)
return defer.fail(InsufficientFundsError())
if sent_request is True:
if p_r is not None:
d3 = protocol.add_request(p_r)
d3.addCallback(self._handle_price_response, peer, p_r, protocol)
d3.addErrback(self._request_failed, "price request", peer)
return defer.succeed(sent_request)
"""Makes an availability request, download request and price request"""
if not self.should_send_next_request(peer):
return defer.succeed(False)
return self._send_next_request(peer, protocol)
def get_new_peers(self):
d = self._get_hash_for_peer_search()
@@ -87,36 +70,35 @@ class BlobRequester(object):
return d
######### internal calls #########
def should_send_next_request(self, peer):
return (
self._blobs_to_download() and
self._should_send_request_to(peer)
)
def _download_succeeded(self, arg, peer, blob):
log.info("Blob %s has been successfully downloaded from %s", str(blob), str(peer))
self._update_local_score(peer, 5.0)
peer.update_stats('blobs_downloaded', 1)
peer.update_score(5.0)
self.blob_manager.blob_completed(blob)
return arg
def _send_next_request(self, peer, protocol):
log.debug('Sending a blob request for %s and %s', peer, protocol)
availability = AvailabilityRequest(self, peer, protocol, self.payment_rate_manager)
download = DownloadRequest(self, peer, protocol, self.payment_rate_manager, self.wallet)
price = PriceRequest(self, peer, protocol, self.payment_rate_manager)
def _download_failed(self, reason, peer):
if not reason.check(DownloadCanceledError, PriceDisagreementError):
self._update_local_score(peer, -10.0)
return reason
sent_request = False
if availability.can_make_request():
availability.make_request_and_handle_response()
sent_request = True
if price.can_make_request():
# TODO: document why a PriceRequest is only made if an
# Availability or Download request was made
price.make_request_and_handle_response()
sent_request = True
if download.can_make_request():
try:
download.make_request_and_handle_response()
sent_request = True
except InsufficientFundsError as err:
return defer.fail(err)
def _pay_or_cancel_payment(self, arg, protocol, reserved_points, blob):
if blob.length != 0 and (not isinstance(arg, Failure) or arg.check(DownloadCanceledError)):
self._pay_peer(protocol, blob.length, reserved_points)
else:
self._cancel_points(reserved_points)
return arg
def _handle_download_error(self, err, peer, blob_to_download):
if not err.check(DownloadCanceledError, PriceDisagreementError, RequestCanceledError):
log.warning("An error occurred while downloading %s from %s. Error: %s",
blob_to_download.blob_hash, str(peer), err.getTraceback())
if err.check(PriceDisagreementError):
# Don't kill the whole connection just because a price couldn't be agreed upon.
# Other information might be desired by other request creators at a better rate.
return True
return err
return defer.succeed(sent_request)
def _get_hash_for_peer_search(self):
r = None
@@ -139,7 +121,9 @@ class BlobRequester(object):
def choose_best_peers(peers):
bad_peers = self._get_bad_peers()
return [p for p in peers if not p in bad_peers]
without_bad_peers = [p for p in peers if not p in bad_peers]
without_maxed_out_peers = [p for p in without_bad_peers if p not in self._maxed_out_peers]
return without_maxed_out_peers
d.addCallback(choose_best_peers)
@@ -178,142 +162,407 @@ class BlobRequester(object):
return sorted(needed_blobs, key=lambda b: b.is_downloading())
def _blobs_without_sources(self):
return [b for b in self.download_manager.needed_blobs() if not self._hash_available(b.blob_hash)]
def _get_availability_request(self, peer):
all_needed = [b.blob_hash for b in self._blobs_to_download() if not b.blob_hash in self._available_blobs[peer]]
# sort them so that the peer will be asked first for blobs it hasn't said it doesn't have
to_request = sorted(all_needed, key=lambda b: b in self._unavailable_blobs[peer])[:20]
if to_request:
r_dict = {'requested_blobs': to_request}
response_identifier = 'available_blobs'
request = ClientRequest(r_dict, response_identifier)
return request
return None
def _get_download_request(self, peer):
request = None
to_download = [b for b in self._blobs_to_download() if self._hash_available_on(b.blob_hash, peer)]
while to_download and request is None:
blob_to_download = to_download[0]
to_download = to_download[1:]
if not blob_to_download.is_validated():
d, write_func, cancel_func = blob_to_download.open_for_writing(peer)
def counting_write_func(data):
peer.update_stats('blob_bytes_downloaded', len(data))
return write_func(data)
if d is not None:
request_dict = {'requested_blob': blob_to_download.blob_hash}
response_identifier = 'incoming_blob'
request = ClientBlobRequest(request_dict, response_identifier, counting_write_func, d,
cancel_func, blob_to_download)
log.info("Requesting blob %s from %s", str(blob_to_download), str(peer))
return request
return [
b for b in self.download_manager.needed_blobs()
if not self._hash_available(b.blob_hash)
]
def _price_settled(self, protocol):
if protocol in self._protocol_prices:
return True
return False
def _get_price_request(self, peer, protocol):
request = None
if not protocol in self._protocol_prices:
self._protocol_prices[protocol] = self.payment_rate_manager.get_rate_blob_data(peer)
request_dict = {'blob_data_payment_rate': self._protocol_prices[protocol]}
request = ClientRequest(request_dict, 'blob_data_payment_rate')
def _update_local_score(self, peer, amount):
self._peers[peer] += amount
class RequestHelper(object):
def __init__(self, requestor, peer, protocol, payment_rate_manager):
self.requestor = requestor
self.peer = peer
self.protocol = protocol
self.payment_rate_manager = payment_rate_manager
@property
def protocol_prices(self):
return self.requestor._protocol_prices
@property
def protocol_offers(self):
return self.requestor._protocol_offers
@property
def available_blobs(self):
return self.requestor._available_blobs[self.peer]
@property
def unavailable_blobs(self):
return self.requestor._unavailable_blobs[self.peer]
@property
def maxed_out_peers(self):
return self.requestor._maxed_out_peers
def update_local_score(self, score):
self.requestor._update_local_score(self.peer, score)
def _request_failed(self, reason, request_type):
if reason.check(RequestCanceledError):
return
if reason.check(NoResponseError):
self.requestor._incompatible_peers.append(self.peer)
log.warning("A request of type '%s' failed. Reason: %s, Error type: %s",
request_type, reason.getErrorMessage(), reason.type)
self.update_local_score(-10.0)
if isinstance(reason, InvalidResponseError) or isinstance(reason, NoResponseError):
self.peer.update_score(-10.0)
else:
self.peer.update_score(-2.0)
if reason.check(ConnectionClosedBeforeResponseError):
return
return reason
def get_and_save_rate(self):
if self.payment_rate_manager.price_limit_reached(self.peer):
if self.peer not in self.maxed_out_peers:
self.maxed_out_peers.append(self.peer)
return None
rate = self.protocol_prices.get(self.protocol)
if rate is None:
if self.peer in self.payment_rate_manager.strategy.pending_sent_offers:
pending = self.payment_rate_manager.strategy.pending_sent_offers[self.peer]
if not pending.is_too_low and not pending.is_accepted:
return pending.rate
rate = self.payment_rate_manager.get_rate_blob_data(self.peer, self.available_blobs)
self.protocol_offers[self.protocol] = rate
return rate
def _handle_incoming_blob(response_dict, peer, request):
if not request.response_identifier in response_dict:
return InvalidResponseError("response identifier not in response")
if not type(response_dict[request.response_identifier]) == dict:
return InvalidResponseError("response not a dict. got %s" %
type(response_dict[request.response_identifier]))
response = response_dict[request.response_identifier]
if 'error' in response:
# This means we're not getting our blob for some reason
if response['error'] == "RATE_UNSET":
# Stop the download with an error that won't penalize the peer
request.cancel(PriceDisagreementError())
else:
# The peer has done something bad so we should get out of here
return InvalidResponseError("Got an unknown error from the peer: %s" %
(response['error'],))
else:
if not 'blob_hash' in response:
return InvalidResponseError("Missing the required field 'blob_hash'")
if not response['blob_hash'] == request.request_dict['requested_blob']:
return InvalidResponseError(
"Incoming blob does not match expected. Incoming: %s. Expected: %s" %
(response['blob_hash'], request.request_dict['requested_blob'])
)
if not 'length' in response:
return InvalidResponseError("Missing the required field 'length'")
if not request.blob.set_length(response['length']):
return InvalidResponseError("Could not set the length of the blob")
return True
def _handle_download_error(err, peer, blob_to_download):
if not err.check(DownloadCanceledError, PriceDisagreementError, RequestCanceledError):
log.warning("An error occurred while downloading %s from %s. Error: %s",
blob_to_download.blob_hash, str(peer), err.getTraceback())
if err.check(PriceDisagreementError):
# Don't kill the whole connection just because a price couldn't be agreed upon.
# Other information might be desired by other request creators at a better rate.
return True
return err
class AvailabilityRequest(RequestHelper):
"""Ask a peer what blobs it has available.
Results are saved in `_available_blobs` and `_unavailable_blobs`
on the parent BlobRequester.
"""
def can_make_request(self):
return self.get_top_needed_blobs()
def make_request_and_handle_response(self):
request = self._get_request()
self._handle_request(request)
def _get_request(self):
to_request = self.get_top_needed_blobs()
if not to_request:
raise Exception('Unable to make a request without available blobs')
return self._make_request(to_request)
@cache
def get_top_needed_blobs(self, limit=20):
all_needed = [
b.blob_hash for b in self.requestor._blobs_to_download()
if not self.is_available(b)
]
# sort them so that the peer will be asked first for blobs it
# hasn't said it doesn't have
sorted_needed = sorted(
all_needed,
key=lambda b: b in self.unavailable_blobs
)
return sorted_needed[:limit]
def is_available(self, blob):
return blob.blob_hash in self.available_blobs
def _make_request(self, to_request):
log.debug('Requesting blobs: %s', to_request)
r_dict = {'requested_blobs': to_request}
response_identifier = 'available_blobs'
request = ClientRequest(r_dict, response_identifier)
return request
def _update_local_score(self, peer, amount):
self._peers[peer] += amount
def _handle_request(self, a_r):
log.debug('making an availability request')
d1 = self.protocol.add_request(a_r)
d1.addCallback(self._handle_availability, a_r)
d1.addErrback(self._request_failed, "availability request")
def _reserve_points(self, peer, protocol, max_bytes):
assert protocol in self._protocol_prices
points_to_reserve = 1.0 * max_bytes * self._protocol_prices[protocol] / 2**20
return self.wallet.reserve_points(peer, points_to_reserve)
def _handle_availability(self, response_dict, request):
assert request.response_identifier == 'available_blobs'
if 'available_blobs' not in response_dict:
raise InvalidResponseError("response identifier not in response")
log.debug("Received a response to the availability request")
# save available blobs
blob_hashes = response_dict['available_blobs']
for blob_hash in blob_hashes:
if blob_hash in request.request_dict['requested_blobs']:
self.process_available_blob_hash(blob_hash, request)
# everything left in the request is missing
for blob_hash in request.request_dict['requested_blobs']:
self.unavailable_blobs.append(blob_hash)
return True
def _pay_peer(self, protocol, num_bytes, reserved_points):
def process_available_blob_hash(self, blob_hash, request):
log.debug("The server has indicated it has the following blob available: %s", blob_hash)
self.available_blobs.append(blob_hash)
self.remove_from_unavailable_blobs(blob_hash)
request.request_dict['requested_blobs'].remove(blob_hash)
def remove_from_unavailable_blobs(self, blob_hash):
if blob_hash in self.unavailable_blobs:
self.unavailable_blobs.remove(blob_hash)
class PriceRequest(RequestHelper):
"""Ask a peer if a certain price is acceptable"""
def can_make_request(self):
if len(self.available_blobs) and not self.protocol in self.protocol_prices:
return self.get_and_save_rate() is not None
return False
def make_request_and_handle_response(self):
request = self._get_price_request()
self._handle_price_request(request)
def _get_price_request(self):
rate = self.get_and_save_rate()
if rate is None:
log.debug("No blobs to request from %s", self.peer)
raise Exception('Cannot make a price request without a payment rate')
log.debug("Offer rate %s to %s for %i blobs", rate, self.peer, len(self.available_blobs))
request_dict = {'blob_data_payment_rate': rate}
return ClientRequest(request_dict, 'blob_data_payment_rate')
def _handle_price_request(self, price_request):
d = self.protocol.add_request(price_request)
d.addCallback(self._handle_price_response, price_request)
d.addErrback(self._request_failed, "price request")
def _handle_price_response(self, response_dict, request):
assert request.response_identifier == 'blob_data_payment_rate'
if 'blob_data_payment_rate' not in response_dict:
return InvalidResponseError("response identifier not in response")
assert self.protocol in self.protocol_offers
offer = Offer(self.protocol_offers[self.protocol])
offer.handle(response_dict['blob_data_payment_rate'])
self.payment_rate_manager.record_offer_reply(self.peer, offer)
if offer.is_accepted:
log.info("Offered rate %f/mb accepted by %s", offer.rate, self.peer.host)
self.protocol_prices[self.protocol] = offer.rate
return True
elif offer.is_too_low:
log.debug("Offered rate %f/mb rejected by %s", offer.rate, self.peer.host)
return not self.payment_rate_manager.price_limit_reached(self.peer)
else:
log.warning("Price disagreement")
self.requestor._price_disagreements.append(self.peer)
return False
class DownloadRequest(RequestHelper):
"""Choose a blob and download it from a peer and also pay the peer for the data."""
def __init__(self, requester, peer, protocol, payment_rate_manager, wallet):
RequestHelper.__init__(self, requester, peer, protocol, payment_rate_manager)
self.wallet = wallet
def can_make_request(self):
if self.protocol in self.protocol_prices:
return self.get_blob_details()
return False
def make_request_and_handle_response(self):
request = self._get_request()
self._handle_download_request(request)
def _get_request(self):
blob_details = self.get_blob_details()
if not blob_details:
raise Exception('No blobs available to download')
return self._make_request(blob_details)
@cache
def get_blob_details(self):
"""Open a blob for writing and return the details.
If no blob can be opened, returns None.
"""
to_download = self.get_available_blobs()
return self.find_blob(to_download)
def get_available_blobs(self):
available_blobs = [
b for b in self.requestor._blobs_to_download()
if self.requestor._hash_available_on(b.blob_hash, self.peer)
]
log.debug('available blobs: %s', available_blobs)
return available_blobs
def find_blob(self, to_download):
"""Return the first blob in `to_download` that is successfully opened for write."""
for blob in to_download:
if blob.is_validated():
log.debug('Skipping blob %s as its already validated', blob)
continue
d, write_func, cancel_func = blob.open_for_writing(self.peer)
if d is not None:
return BlobDownloadDetails(blob, d, write_func, cancel_func, self.peer)
log.debug('Skipping blob %s as there was an issue opening it for writing', blob)
return None
def _make_request(self, blob_details):
blob = blob_details.blob
request = ClientBlobRequest(
{'requested_blob': blob.blob_hash},
'incoming_blob',
blob_details.counting_write_func,
blob_details.deferred,
blob_details.cancel_func,
blob
)
log.info("Requesting blob %s from %s", blob.blob_hash, self.peer)
return request
def _handle_download_request(self, client_blob_request):
reserved_points = self.reserve_funds_or_cancel(client_blob_request)
self.add_callbacks_to_download_request(client_blob_request, reserved_points)
self.create_add_blob_request(client_blob_request)
def reserve_funds_or_cancel(self, client_blob_request):
reserved_points = self._reserve_points(client_blob_request.max_pay_units)
if reserved_points is not None:
return reserved_points
client_blob_request.cancel(InsufficientFundsError())
client_blob_request.finished_deferred.addErrback(lambda _: True)
raise InsufficientFundsError()
def add_callbacks_to_download_request(self, client_blob_request, reserved_points):
# Note: The following three callbacks will be called when the blob has been
# fully downloaded or canceled
client_blob_request.finished_deferred.addCallbacks(
self._download_succeeded,
self._download_failed,
callbackArgs=(client_blob_request.blob,),
)
client_blob_request.finished_deferred.addBoth(
self._pay_or_cancel_payment, reserved_points, client_blob_request.blob)
client_blob_request.finished_deferred.addErrback(
_handle_download_error, self.peer, client_blob_request.blob)
def _pay_or_cancel_payment(self, arg, reserved_points, blob):
if self._can_pay_peer(blob, arg):
self._pay_peer(blob.length, reserved_points)
d = self.requestor.blob_manager.add_blob_to_download_history(str(blob),
str(self.peer.host),
float(self.protocol_prices[self.protocol]))
else:
self._cancel_points(reserved_points)
return arg
def _can_pay_peer(self, blob, arg):
return (
blob.length != 0 and
(not isinstance(arg, Failure) or arg.check(DownloadCanceledError))
)
def _pay_peer(self, num_bytes, reserved_points):
assert num_bytes != 0
assert protocol in self._protocol_prices
point_amount = 1.0 * num_bytes * self._protocol_prices[protocol] / 2**20
rate = self.get_and_save_rate()
point_amount = get_points(num_bytes, rate)
self.wallet.send_points(reserved_points, point_amount)
self.payment_rate_manager.record_points_paid(point_amount)
def _cancel_points(self, reserved_points):
self.wallet.cancel_point_reservation(reserved_points)
def _handle_availability(self, response_dict, peer, request):
if not request.response_identifier in response_dict:
raise InvalidResponseError("response identifier not in response")
log.debug("Received a response to the availability request")
blob_hashes = response_dict[request.response_identifier]
for blob_hash in blob_hashes:
if blob_hash in request.request_dict['requested_blobs']:
log.debug("The server has indicated it has the following blob available: %s", blob_hash)
self._available_blobs[peer].append(blob_hash)
if blob_hash in self._unavailable_blobs[peer]:
self._unavailable_blobs[peer].remove(blob_hash)
request.request_dict['requested_blobs'].remove(blob_hash)
for blob_hash in request.request_dict['requested_blobs']:
self._unavailable_blobs[peer].append(blob_hash)
return True
def create_add_blob_request(self, client_blob_request):
d = self.protocol.add_blob_request(client_blob_request)
# Note: The following two callbacks will be called as soon as the peer sends its
# response, which will be before the blob has finished downloading, but may be
# after the blob has been canceled. For example,
# 1) client sends request to Peer A
# 2) the blob is finished downloading from peer B, and therefore this one is canceled
# 3) client receives response from Peer A
# Therefore, these callbacks shouldn't rely on there being a blob about to be
# downloaded.
d.addCallback(_handle_incoming_blob, self.peer, client_blob_request)
d.addErrback(self._request_failed, "download request")
def _handle_incoming_blob(self, response_dict, peer, request):
if not request.response_identifier in response_dict:
return InvalidResponseError("response identifier not in response")
if not type(response_dict[request.response_identifier]) == dict:
return InvalidResponseError("response not a dict. got %s" %
(type(response_dict[request.response_identifier]),))
response = response_dict[request.response_identifier]
if 'error' in response:
# This means we're not getting our blob for some reason
if response['error'] == "RATE_UNSET":
# Stop the download with an error that won't penalize the peer
request.cancel(PriceDisagreementError())
else:
# The peer has done something bad so we should get out of here
return InvalidResponseError("Got an unknown error from the peer: %s" %
(response['error'],))
else:
if not 'blob_hash' in response:
return InvalidResponseError("Missing the required field 'blob_hash'")
if not response['blob_hash'] == request.request_dict['requested_blob']:
return InvalidResponseError("Incoming blob does not match expected. Incoming: %s. Expected: %s" %
(response['blob_hash'], request.request_dict['requested_blob']))
if not 'length' in response:
return InvalidResponseError("Missing the required field 'length'")
if not request.blob.set_length(response['length']):
return InvalidResponseError("Could not set the length of the blob")
return True
def _reserve_points(self, num_bytes):
# jobevers: there was an assertion here, but I don't think it
# was a valid assertion to make. It is possible for a rate to
# not yet been set for this protocol or for it to have been
# removed so instead I switched it to check if a rate has been set
# and calculate it if it has not
rate = self.get_and_save_rate()
points_to_reserve = get_points(num_bytes, rate)
return self.wallet.reserve_points(self.peer, points_to_reserve)
def _handle_price_response(self, response_dict, peer, request, protocol):
if not request.response_identifier in response_dict:
return InvalidResponseError("response identifier not in response")
assert protocol in self._protocol_prices
response = response_dict[request.response_identifier]
if response == "RATE_ACCEPTED":
return True
else:
del self._protocol_prices[protocol]
self._price_disagreements.append(peer)
return True
def _download_succeeded(self, arg, blob):
log.info("Blob %s has been successfully downloaded from %s", blob, self.peer)
self.update_local_score(5.0)
self.peer.update_stats('blobs_downloaded', 1)
self.peer.update_score(5.0)
self.requestor.blob_manager.blob_completed(blob)
return arg
def _request_failed(self, reason, request_type, peer):
if reason.check(RequestCanceledError):
return
if reason.check(NoResponseError):
self._incompatible_peers.append(peer)
log.warning("Blob requester: a request of type '%s' failed. Reason: %s, Error type: %s",
str(request_type), reason.getErrorMessage(), reason.type)
self._update_local_score(peer, -10.0)
if isinstance(reason, InvalidResponseError) or isinstance(reason, NoResponseError):
peer.update_score(-10.0)
else:
peer.update_score(-2.0)
if reason.check(ConnectionClosedBeforeResponseError):
return
return reason
def _download_failed(self, reason):
if not reason.check(DownloadCanceledError, PriceDisagreementError):
self.update_local_score(-10.0)
return reason
class BlobDownloadDetails(object):
"""Contains the information needed to make a ClientBlobRequest from an open blob"""
def __init__(self, blob, deferred, write_func, cancel_func, peer):
self.blob = blob
self.deferred = deferred
self.write_func = write_func
self.cancel_func = cancel_func
self.peer = peer
def counting_write_func(self, data):
self.peer.update_stats('blob_bytes_downloaded', len(data))
return self.write_func(data)
+16 -7
View File
@@ -1,9 +1,10 @@
import json
import logging
from decimal import Decimal
from twisted.internet import error, defer
from twisted.internet.protocol import Protocol, ClientFactory
from twisted.python import failure
from lbrynet.conf import MAX_RESPONSE_INFO_SIZE as MAX_RESPONSE_SIZE
from lbrynet.conf import settings
from lbrynet.core.Error import ConnectionClosedBeforeResponseError, NoResponseError
from lbrynet.core.Error import DownloadCanceledError, MisbehavingPeerError
from lbrynet.core.Error import RequestCanceledError
@@ -14,6 +15,12 @@ from zope.interface import implements
log = logging.getLogger(__name__)
def encode_decimal(obj):
if isinstance(obj, Decimal):
return float(obj)
raise TypeError(repr(obj) + " is not JSON serializable")
class ClientProtocol(Protocol):
implements(IRequestSender, IRateLimited)
@@ -41,7 +48,7 @@ class ClientProtocol(Protocol):
self._blob_download_request.write(data)
else:
self._response_buff += data
if len(self._response_buff) > MAX_RESPONSE_SIZE:
if len(self._response_buff) > settings.MAX_RESPONSE_INFO_SIZE:
log.warning("Response is too large. Size %s", len(self._response_buff))
self.transport.loseConnection()
response, extra_data = self._get_valid_response(self._response_buff)
@@ -73,7 +80,7 @@ class ClientProtocol(Protocol):
return defer.fail(failure.Failure(ValueError("There is already a request for that response active")))
self._next_request.update(request.request_dict)
d = defer.Deferred()
log.debug("Adding a request. Request: %s", str(request))
log.debug("Adding a request. Request: %s", str(request.request_dict))
self._response_deferreds[request.response_identifier] = d
return d
@@ -132,7 +139,7 @@ class ClientProtocol(Protocol):
def _send_request_message(self, request_msg):
# TODO: compare this message to the last one. If they're the same,
# TODO: incrementally delay this message.
m = json.dumps(request_msg)
m = json.dumps(request_msg, default=encode_decimal)
self.transport.write(m)
def _get_valid_response(self, response_msg):
@@ -191,7 +198,9 @@ class ClientProtocol(Protocol):
for success, result in results:
if success is False:
failed = True
log.info("The connection is closing due to an error: %s", str(result.getTraceback()))
if not isinstance(result.value, DownloadCanceledError):
log.info(result.value)
log.info("The connection is closing due to an error: %s", str(result.getTraceback()))
if failed is False:
log.debug("Asking for another request.")
from twisted.internet import reactor
@@ -215,7 +224,7 @@ class ClientProtocol(Protocol):
# TODO: always be this way. it's done this way now because the client has no other way
# TODO: of telling the server it wants the download to stop. It would be great if the
# TODO: protocol had such a mechanism.
log.info("Closing the connection to %s because the download of blob %s was canceled",
log.debug("Closing the connection to %s because the download of blob %s was canceled",
str(self.peer), str(self._blob_download_request.blob))
#self.transport.loseConnection()
#return True
@@ -253,4 +262,4 @@ class ClientProtocolFactory(ClientFactory):
p = self.protocol()
p.factory = self
self.p = p
return p
return p
+2 -2
View File
@@ -1,4 +1,4 @@
from lbrynet.conf import BLOB_SIZE
from lbrynet.conf import settings
class ClientRequest(object):
@@ -17,7 +17,7 @@ class ClientBlobRequest(ClientPaidRequest):
def __init__(self, request_dict, response_identifier, write_func, finished_deferred,
cancel_func, blob):
if blob.length is None:
max_pay_units = BLOB_SIZE
max_pay_units = settings.BLOB_SIZE
else:
max_pay_units = blob.length
ClientPaidRequest.__init__(self, request_dict, response_identifier, max_pay_units)
+11 -10
View File
@@ -2,7 +2,7 @@ import logging
from twisted.internet import defer
from zope.interface import implements
from lbrynet import interfaces
from lbrynet.conf import MAX_CONNECTIONS_PER_STREAM
from lbrynet.conf import settings
from lbrynet.core.client.ClientProtocol import ClientProtocolFactory
from lbrynet.core.Error import InsufficientFundsError
@@ -49,8 +49,8 @@ class ConnectionManager(object):
for peer in self._peer_connections.keys():
def close_connection(p):
log.info("Abruptly closing a connection to %s due to downloading being paused",
str(p))
log.info(
"Abruptly closing a connection to %s due to downloading being paused", p)
if self._peer_connections[p].factory.p is not None:
d = self._peer_connections[p].factory.p.cancel_requests()
@@ -73,7 +73,7 @@ class ConnectionManager(object):
def get_next_request(self, peer, protocol):
log.debug("Trying to get the next request for peer %s", str(peer))
log.debug("Trying to get the next request for peer %s", peer)
if not peer in self._peer_connections or self.stopped is True:
log.debug("The peer has already been told to shut down.")
@@ -148,7 +148,7 @@ class ConnectionManager(object):
from twisted.internet import reactor
if peer is not None and self.stopped is False:
log.debug("Trying to connect to %s", str(peer))
log.debug("Trying to connect to %s", peer)
factory = ClientProtocolFactory(peer, self.rate_limiter, self)
self._peer_connections[peer] = PeerConnectionHandler(self._primary_request_creators[:],
factory)
@@ -162,7 +162,7 @@ class ConnectionManager(object):
def get_new_peers(request_creators):
log.debug("Trying to get a new peer to connect to")
if len(request_creators) > 0:
log.debug("Got a creator to check: %s", str(request_creators[0]))
log.debug("Got a creator to check: %s", request_creators[0])
d = request_creators[0].get_new_peers()
d.addCallback(lambda h: h if h is not None else get_new_peers(request_creators[1:]))
return d
@@ -172,20 +172,21 @@ class ConnectionManager(object):
def pick_best_peer(peers):
# TODO: Eventually rank them based on past performance/reputation. For now
# TODO: just pick the first to which we don't have an open connection
log.debug("Got a list of peers to choose from: %s", str(peers))
log.debug("Got a list of peers to choose from: %s", peers)
if peers is None:
return None
for peer in peers:
if not peer in self._peer_connections:
log.debug("Got a good peer. Returning peer %s", str(peer))
log.debug("Got a good peer. Returning peer %s", peer)
return peer
log.debug("Couldn't find a good peer to connect to")
return None
if len(self._peer_connections) < MAX_CONNECTIONS_PER_STREAM:
if len(self._peer_connections) < settings.max_connections_per_stream:
ordered_request_creators = self._rank_request_creator_connections()
d = get_new_peers(ordered_request_creators)
d.addCallback(pick_best_peer)
d.addCallback(self._connect_to_peer)
self._next_manage_call = reactor.callLater(1, self._manage)
self._next_manage_call = reactor.callLater(1, self._manage)
+2 -2
View File
@@ -66,11 +66,11 @@ class DownloadManager(object):
def add_blobs_to_download(self, blob_infos):
log.debug("Adding %s to blobs", str(blob_infos))
log.debug("Adding %s blobs to blobs", len(blob_infos))
def add_blob_to_list(blob, blob_num):
self.blobs[blob_num] = blob
log.info("Added blob (hash: %s, number %s) to the list", str(blob.blob_hash), str(blob_num))
log.debug("Added blob (hash: %s, number %s) to the list", str(blob.blob_hash), str(blob_num))
def error_during_add(err):
log.warning("An error occurred adding the blob to blobs. Error:%s", err.getErrorMessage())
@@ -93,7 +93,6 @@ class DummyBlobHandler(object):
class StandaloneBlobDownloader(object):
def __init__(self, blob_hash, blob_manager, peer_finder, rate_limiter, payment_rate_manager, wallet):
self.blob_hash = blob_hash
self.blob_manager = blob_manager
@@ -142,4 +141,4 @@ class StandaloneBlobDownloader(object):
def insufficient_funds(self, err):
self.stop()
if not self.finished_deferred.called:
self.finished_deferred.errback(err)
self.finished_deferred.errback(err)
+280
View File
@@ -0,0 +1,280 @@
import datetime
import json
import logging
import logging.handlers
import os
import platform
import sys
import traceback
import requests
from requests_futures.sessions import FuturesSession
import lbrynet
from lbrynet.conf import settings
from lbrynet.core import utils
session = FuturesSession()
def bg_cb(sess, resp):
""" Don't do anything with the response """
pass
class HTTPSHandler(logging.Handler):
def __init__(self, url, fqdn=False, localname=None, facility=None):
logging.Handler.__init__(self)
self.url = url
self.fqdn = fqdn
self.localname = localname
self.facility = facility
def get_full_message(self, record):
if record.exc_info:
return '\n'.join(traceback.format_exception(*record.exc_info))
else:
return record.getMessage()
def emit(self, record):
try:
payload = self.format(record)
session.post(self.url, data=payload, background_callback=bg_cb)
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
DEFAULT_FORMAT = "%(asctime)s %(levelname)-8s %(name)s:%(lineno)d: %(message)s"
DEFAULT_FORMATTER = logging.Formatter(DEFAULT_FORMAT)
LOGGLY_URL = "https://logs-01.loggly.com/inputs/{token}/tag/{tag}"
def remove_handlers(log, handler_name):
for handler in log.handlers:
if handler.name == handler_name:
log.removeHandler(handler)
def _log_decorator(fn):
"""Configure a logging handler.
`fn` is a function that returns a logging handler. The returned
handler has its log-level set and is attached to the specified
logger or the root logger.
"""
def helper(*args, **kwargs):
log = kwargs.pop('log', logging.getLogger())
level = kwargs.pop('level', logging.INFO)
if not isinstance(level, int):
# despite the name, getLevelName returns
# the numeric level when passed a text level
level = logging.getLevelName(level)
handler = fn(*args, **kwargs)
if handler.name:
remove_handlers(log, handler.name)
handler.setLevel(level)
log.addHandler(handler)
# need to reduce the logger's level down to the
# handler's level or else the handler won't
# get those messages
if log.level > level:
log.setLevel(level)
return handler
return helper
def disable_third_party_loggers():
logging.getLogger('requests').setLevel(logging.WARNING)
logging.getLogger('urllib3').setLevel(logging.WARNING)
logging.getLogger('BitcoinRPC').setLevel(logging.INFO)
@_log_decorator
def configure_console(**kwargs):
"""Convenience function to configure a log-handler that outputs to stdout"""
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(DEFAULT_FORMATTER)
handler.name = 'console'
return handler
@_log_decorator
def configure_file_handler(file_name, **kwargs):
"""Convenience function to configure a log-handler that writes to `file_name`"""
handler = logging.handlers.RotatingFileHandler(file_name, maxBytes=2097152, backupCount=5)
handler.setFormatter(DEFAULT_FORMATTER)
handler.name = 'file'
return handler
def get_loggly_url(token=None, version=None):
token = token or utils.deobfuscate(settings.LOGGLY_TOKEN)
version = version or lbrynet.__version__
return LOGGLY_URL.format(token=token, tag='lbrynet-' + version)
@_log_decorator
def configure_loggly_handler(url=None, **kwargs):
url = url or get_loggly_url()
formatter = JsonFormatter(**kwargs)
handler = HTTPSHandler(url)
handler.setFormatter(formatter)
handler.name = 'loggly'
return handler
class JsonFormatter(logging.Formatter):
"""Format log records using json serialization"""
def __init__(self, **kwargs):
self.attributes = kwargs
def format(self, record):
data = {
'loggerName': record.name,
'asciTime': self.formatTime(record),
'fileName': record.filename,
'functionName': record.funcName,
'levelNo': record.levelno,
'lineNo': record.lineno,
'levelName': record.levelname,
'message': record.getMessage(),
}
data.update(self.attributes)
if record.exc_info:
data['exc_info'] = self.formatException(record.exc_info)
return json.dumps(data)
def failure(failure, log, msg, *args):
"""Log a failure message from a deferred.
Args:
failure: twisted.python.failure.Failure
log: a python logger instance
msg: the message to log. Can use normal logging string interpolation.
the last argument will be set to the error message from the failure.
args: values to substitute into `msg`
"""
args += (failure.getErrorMessage(),)
exc_info = (failure.type, failure.value, failure.getTracebackObject())
log.error(msg, *args, exc_info=exc_info)
def convert_verbose(verbose):
"""Convert the results of the --verbose flag into a list of logger names
if --verbose is not provided, args.verbose will be None and logging
should be at the info level.
if --verbose is provided, but not followed by any arguments, then
args.verbose = [] and debug logging should be enabled for all of lbrynet
if --verbose is provided and followed by arguments, those arguments
will be in a list
"""
if verbose is None:
return []
if verbose == []:
return ['lbrynet']
return verbose
def configure_logging(file_name, console, verbose=None):
"""Apply the default logging configuration.
Enables two log-handlers at the INFO level: a file logger and a loggly logger.
Optionally turns on a console logger that defaults to the INFO level, with
specified loggers being set to the DEBUG level.
Args:
file_name: the file to which logs should be saved
console: If true, enable a console logger
verbose: a list of loggers to set to debug level.
See `convert_verbose` for more details.
"""
verbose = convert_verbose(verbose)
configure_file_handler(file_name)
configure_loggly_handler()
disable_third_party_loggers()
if console:
# if there are some loggers at the debug level, we need
# to enable the console to allow debug. Otherwise, only
# allow info.
level = 'DEBUG' if verbose else 'INFO'
handler = configure_console(level=level)
if verbose:
handler.addFilter(LoggerNameFilter(verbose))
class LoggerNameFilter(object):
"""Filter a log record based on its name.
Allows all info level and higher records to pass thru.
Debug records pass if the log record name (or a parent) match
the input list of logger names.
"""
def __init__(self, logger_names):
self.logger_names = logger_names
def filter(self, record):
if record.levelno >= logging.INFO:
return True
name = record.name
while name:
if name in self.logger_names:
return True
name = get_parent(name)
return False
def get_parent(logger_name):
names = logger_name.split('.')
if len(names) == 1:
return ''
names = names[:-1]
return '.'.join(names)
class LogUploader(object):
def __init__(self, log_name, log_file, log_size):
self.log_name = log_name
self.log_file = log_file
self.log_size = log_size
def upload(self, exclude_previous, id_hash, log_type):
if not os.path.isfile(self.log_file):
return
log_contents = self.log_contents(exclude_previous)
params = {
'date': datetime.datetime.utcnow().strftime('%Y%m%d-%H%M%S'),
'hash': id_hash,
'sys': platform.system(),
'type': self.get_type(log_type),
'log': log_contents
}
requests.post(settings.LOG_POST_URL, params)
def log_contents(self, exclude_previous):
with open(self.log_file) as f:
if exclude_previous:
f.seek(self.log_size)
log_contents = f.read()
else:
log_contents = f.read()
return log_contents
def get_type(self, log_type):
if log_type:
return "%s-%s" % (self.log_name, log_type)
else:
return self.log_name
@classmethod
def load(cls, log_name, log_file):
if os.path.isfile(log_file):
with open(log_file, 'r') as f:
log_size = len(f.read())
else:
log_size = 0
return cls(log_name, log_file, log_size)
+20
View File
@@ -0,0 +1,20 @@
class LoopingCallManager(object):
def __init__(self, calls=None):
self.calls = calls or {}
def register_looping_call(self, name, call):
assert name not in self.calls, '{} is already registered'.format(name)
self.calls[name] = call
def start(self, name, *args):
lcall = self.calls[name]
if not lcall.running:
lcall.start(*args)
def stop(self, name):
self.calls[name].stop()
def shutdown(self):
for lcall in self.calls.itervalues():
if lcall.running:
lcall.stop()
@@ -40,10 +40,11 @@ class BlobAvailabilityHandler(object):
def handle_queries(self, queries):
if self.query_identifiers[0] in queries:
log.debug("Received the client's list of requested blobs")
log.info("Received the client's list of requested blobs")
d = self._get_available_blobs(queries[self.query_identifiers[0]])
def set_field(available_blobs):
log.debug("available blobs: %s", str(available_blobs))
return {'available_blobs': available_blobs}
d.addCallback(set_field)
+123 -52
View File
@@ -1,8 +1,13 @@
import logging
from twisted.internet import defer
from twisted.protocols.basic import FileSender
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.core.Offer import Offer
from lbrynet import analytics
from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler, IBlobSender
@@ -12,15 +17,16 @@ log = logging.getLogger(__name__)
class BlobRequestHandlerFactory(object):
implements(IQueryHandlerFactory)
def __init__(self, blob_manager, wallet, payment_rate_manager):
def __init__(self, blob_manager, wallet, payment_rate_manager, track):
self.blob_manager = blob_manager
self.wallet = wallet
self.payment_rate_manager = payment_rate_manager
self.track = track
######### IQueryHandlerFactory #########
def build_query_handler(self):
q_h = BlobRequestHandler(self.blob_manager, self.wallet, self.payment_rate_manager)
q_h = BlobRequestHandler(self.blob_manager, self.wallet, self.payment_rate_manager, self.track)
return q_h
def get_primary_query_identifier(self):
@@ -32,18 +38,23 @@ class BlobRequestHandlerFactory(object):
class BlobRequestHandler(object):
implements(IQueryHandler, IBlobSender)
PAYMENT_RATE_QUERY = 'blob_data_payment_rate'
BLOB_QUERY = 'requested_blob'
AVAILABILITY_QUERY = 'requested_blobs'
def __init__(self, blob_manager, wallet, payment_rate_manager):
def __init__(self, blob_manager, wallet, payment_rate_manager, track):
self.blob_manager = blob_manager
self.payment_rate_manager = payment_rate_manager
self.wallet = wallet
self.query_identifiers = ['blob_data_payment_rate', 'requested_blob']
self.query_identifiers = [self.PAYMENT_RATE_QUERY, self.BLOB_QUERY, self.AVAILABILITY_QUERY]
self.track = track
self.peer = None
self.blob_data_payment_rate = None
self.read_handle = None
self.currently_uploading = None
self.file_sender = None
self.blob_bytes_uploaded = 0
self._blobs_requested = []
######### IQueryHandler #########
@@ -53,44 +64,22 @@ class BlobRequestHandler(object):
request_handler.register_blob_sender(self)
def handle_queries(self, queries):
response = {}
if self.query_identifiers[0] in queries:
if not self.handle_blob_data_payment_rate(queries[self.query_identifiers[0]]):
response['blob_data_payment_rate'] = "RATE_TOO_LOW"
else:
response['blob_data_payment_rate'] = 'RATE_ACCEPTED'
response = defer.succeed({})
log.debug("Handle query: %s", str(queries))
if self.query_identifiers[1] in queries:
log.debug("Received the client's request to send a blob")
response_fields = {}
response['incoming_blob'] = response_fields
if self.blob_data_payment_rate is None:
response_fields['error'] = "RATE_UNSET"
return defer.succeed(response)
else:
d = self.blob_manager.get_blob(queries[self.query_identifiers[1]], True)
def open_blob_for_reading(blob):
if blob.is_validated():
read_handle = blob.open_for_reading()
if read_handle is not None:
self.currently_uploading = blob
self.read_handle = read_handle
log.debug("Sending %s to client", str(blob))
response_fields['blob_hash'] = blob.blob_hash
response_fields['length'] = blob.length
return response
log.debug("We can not send %s", str(blob))
response_fields['error'] = "BLOB_UNAVAILABLE"
return response
d.addCallback(open_blob_for_reading)
return d
else:
return defer.succeed(response)
if self.AVAILABILITY_QUERY in queries:
self._blobs_requested = queries[self.AVAILABILITY_QUERY]
response.addCallback(lambda r: self._reply_to_availability(r, self._blobs_requested))
if self.PAYMENT_RATE_QUERY in queries:
offered_rate = queries[self.PAYMENT_RATE_QUERY]
offer = Offer(offered_rate)
if offer.rate is None:
log.warning("Empty rate offer")
response.addCallback(lambda r: self._handle_payment_rate_query(offer, r))
if self.BLOB_QUERY in queries:
incoming = queries[self.BLOB_QUERY]
response.addCallback(lambda r: self._reply_to_send_request(r, incoming))
return response
######### IBlobSender #########
@@ -108,12 +97,91 @@ class BlobRequestHandler(object):
######### internal #########
def handle_blob_data_payment_rate(self, requested_payment_rate):
if not self.payment_rate_manager.accept_rate_blob_data(self.peer, requested_payment_rate):
return False
def _reply_to_availability(self, request, blobs):
d = self._get_available_blobs(blobs)
def set_available(available_blobs):
log.debug("available blobs: %s", str(available_blobs))
request.update({'available_blobs': available_blobs})
return request
d.addCallback(set_available)
return d
def _handle_payment_rate_query(self, offer, request):
blobs = self._blobs_requested
log.debug("Offered rate %f LBC/mb for %i blobs", offer.rate, len(blobs))
reply = self.payment_rate_manager.reply_to_offer(self.peer, blobs, offer)
if reply.is_accepted:
self.blob_data_payment_rate = offer.rate
request[self.PAYMENT_RATE_QUERY] = "RATE_ACCEPTED"
log.debug("Accepted rate: %f", offer.rate)
elif reply.is_too_low:
request[self.PAYMENT_RATE_QUERY] = "RATE_TOO_LOW"
log.debug("Reject rate: %f", offer.rate)
elif reply.is_unset:
log.warning("Rate unset")
request['incoming_blob'] = {'error': 'RATE_UNSET'}
log.debug("Returning rate query result: %s", str(request))
return request
def _handle_blob_query(self, response, query):
log.debug("Received the client's request to send a blob")
response['incoming_blob'] = {}
if self.blob_data_payment_rate is None:
response['incoming_blob'] = {'error': "RATE_UNSET"}
return response
else:
self.blob_data_payment_rate = requested_payment_rate
return True
return self._send_blob(response, query)
def _send_blob(self, response, query):
d = self.blob_manager.get_blob(query, True)
d.addCallback(self.open_blob_for_reading, response)
return d
def open_blob_for_reading(self, blob, response):
response_fields = {}
d = defer.succeed(None)
if blob.is_validated():
read_handle = blob.open_for_reading()
if read_handle is not None:
self.currently_uploading = blob
self.read_handle = read_handle
log.info("Sending %s to client", str(blob))
response_fields['blob_hash'] = blob.blob_hash
response_fields['length'] = blob.length
response['incoming_blob'] = response_fields
d.addCallback(lambda _: self.record_transaction(blob))
d.addCallback(lambda _: response)
return d
log.debug("We can not send %s", str(blob))
response['incoming_blob'] = {'error': 'BLOB_UNAVAILABLE'}
d.addCallback(lambda _: response)
return d
def record_transaction(self, blob):
d = self.blob_manager.add_blob_to_upload_history(str(blob), self.peer.host, self.blob_data_payment_rate)
return d
def _reply_to_send_request(self, response, incoming):
response_fields = {}
response['incoming_blob'] = response_fields
if self.blob_data_payment_rate is None:
log.debug("Rate not set yet")
response['incoming_blob'] = {'error': 'RATE_UNSET'}
return defer.succeed(response)
else:
log.debug("Requested blob: %s", str(incoming))
d = self.blob_manager.get_blob(incoming, True)
d.addCallback(lambda blob: self.open_blob_for_reading(blob, response))
return d
def _get_available_blobs(self, requested_blobs):
d = self.blob_manager.completed_blobs(requested_blobs)
return d
def send_file(self, consumer):
@@ -126,20 +194,23 @@ class BlobRequestHandler(object):
return inner_d
def count_bytes(data):
self.blob_bytes_uploaded += len(data)
self.peer.update_stats('blob_bytes_uploaded', len(data))
uploaded = len(data)
self.blob_bytes_uploaded += uploaded
self.peer.update_stats('blob_bytes_uploaded', uploaded)
self.track.add_observation(analytics.BLOB_BYTES_UPLOADED, uploaded)
return data
def start_transfer():
self.file_sender = FileSender()
log.info("Starting the file upload")
log.debug("Starting the file upload")
assert self.read_handle is not None, "self.read_handle was None when trying to start the transfer"
d = self.file_sender.beginFileTransfer(self.read_handle, consumer, count_bytes)
return d
def set_expected_payment():
log.info("Setting expected payment")
log.debug("Setting expected payment")
if self.blob_bytes_uploaded != 0 and self.blob_data_payment_rate is not None:
# TODO: explain why 2**20
self.wallet.add_expected_payment(self.peer,
self.currently_uploading.length * 1.0 *
self.blob_data_payment_rate / 2**20)
@@ -154,6 +225,6 @@ class BlobRequestHandler(object):
self.currently_uploading = None
self.file_sender = None
if reason is not None and isinstance(reason, Failure):
log.info("Upload has failed. Reason: %s", reason.getErrorMessage())
log.warning("Upload has failed. Reason: %s", reason.getErrorMessage())
return _send_file()
return _send_file()
+2 -2
View File
@@ -1,5 +1,5 @@
import binascii
from twisted.internet import defer, task, reactor
from twisted.internet import defer, reactor
import collections
@@ -78,4 +78,4 @@ class DHTHashSupplier(object):
self.hash_reannounce_time = 60 * 60 # 1 hour
def hashes_to_announce(self):
pass
pass
+24 -16
View File
@@ -9,9 +9,10 @@ log = logging.getLogger(__name__)
class ServerRequestHandler(object):
"""This class handles requests from clients. It can upload blobs and return request for information about
more blobs that are associated with streams"""
"""This class handles requests from clients. It can upload blobs and
return request for information about more blobs that are
associated with streams.
"""
implements(interfaces.IPushProducer, interfaces.IConsumer, IRequestHandler)
def __init__(self, consumer):
@@ -90,20 +91,27 @@ class ServerRequestHandler(object):
log.debug("Received data")
log.debug("%s", str(data))
if self.request_received is False:
self.request_buff = self.request_buff + data
msg = self.try_to_parse_request(self.request_buff)
if msg is not None:
self.request_buff = ''
d = self.handle_request(msg)
if self.blob_sender is not None:
d.addCallback(lambda _: self.blob_sender.send_blob_if_requested(self))
d.addCallbacks(lambda _: self.finished_response(), self.request_failure_handler)
else:
log.info("Request buff not a valid json message")
log.info("Request buff: %s", str(self.request_buff))
return self._parse_data_and_maybe_send_blob(data)
else:
log.warning("The client sent data when we were uploading a file. This should not happen")
def _parse_data_and_maybe_send_blob(self, data):
self.request_buff = self.request_buff + data
msg = self.try_to_parse_request(self.request_buff)
if msg:
self.request_buff = ''
self._process_msg(msg)
else:
log.debug("Request buff not a valid json message")
log.debug("Request buff: %s", self.request_buff)
def _process_msg(self, msg):
d = self.handle_request(msg)
if self.blob_sender:
d.addCallback(lambda _: self.blob_sender.send_blob_if_requested(self))
d.addCallbacks(lambda _: self.finished_response(), self.request_failure_handler)
######### IRequestHandler #########
def register_query_handler(self, query_handler, query_identifiers):
@@ -125,7 +133,7 @@ class ServerRequestHandler(object):
def send_response(self, msg):
m = json.dumps(msg)
log.info("Sending a response of length %s", str(len(m)))
log.debug("Sending a response of length %s", str(len(m)))
log.debug("Response: %s", str(m))
self.response_buff = self.response_buff + m
self._produce_more()
@@ -171,4 +179,4 @@ class ServerRequestHandler(object):
msg = json.loads(request_buff)
return msg
except ValueError:
return None
return None
+109 -9
View File
@@ -1,8 +1,41 @@
from lbrynet.core.cryptoutils import get_lbry_hash_obj
import base64
import datetime
import distutils.version
import logging
import json
import random
import os
import socket
import sys
import yaml
from lbrynet.conf import settings
from lbrynet.conf import AdjustableSettings
from lbrynet.core.cryptoutils import get_lbry_hash_obj
blobhash_length = get_lbry_hash_obj().digest_size * 2 # digest_size is in bytes, and blob hashes are hex encoded
# digest_size is in bytes, and blob hashes are hex encoded
blobhash_length = get_lbry_hash_obj().digest_size * 2
log = logging.getLogger(__name__)
# defining these time functions here allows for easier overriding in testing
def now():
return datetime.datetime.now()
def utcnow():
return datetime.datetime.utcnow()
def isonow():
"""Return utc now in isoformat with timezone"""
return utcnow().isoformat() + 'Z'
def today():
return datetime.datetime.today()
def generate_id(num=None):
@@ -14,15 +47,82 @@ def generate_id(num=None):
return h.digest()
def is_valid_hashcharacter(char):
return char in "0123456789abcdef"
def is_valid_blobhash(blobhash):
"""
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: Whether the blobhash is the correct length and contains only valid characters (0-9, a-f)
@return: True/False
"""
if len(blobhash) != blobhash_length:
return len(blobhash) == blobhash_length and all(is_valid_hashcharacter(l) for l in blobhash)
def version_is_greater_than(a, b):
"""Returns True if version a is more recent than version b"""
try:
return distutils.version.StrictVersion(a) > distutils.version.StrictVersion(b)
except ValueError:
return distutils.version.LooseVersion(a) > distutils.version.LooseVersion(b)
def deobfuscate(obfustacated):
return base64.b64decode(obfustacated.decode('rot13'))
def obfuscate(plain):
return base64.b64encode(plain).encode('rot13')
settings_decoders = {
'.json': json.loads,
'.yml': yaml.load
}
settings_encoders = {
'.json': json.dumps,
'.yml': yaml.safe_dump
}
ADJUSTABLE_SETTINGS = AdjustableSettings().get_dict()
def load_settings(path):
ext = os.path.splitext(path)[1]
with open(path, 'r') as settings_file:
data = settings_file.read()
decoder = settings_decoders.get(ext, False)
assert decoder is not False, "Unknown settings format .%s" % ext
return decoder(data)
def save_settings(path):
to_save = {k: v for k, v in settings.__dict__.iteritems() if k in ADJUSTABLE_SETTINGS}
ext = os.path.splitext(path)[1]
encoder = settings_encoders.get(ext, False)
assert encoder is not False, "Unknown settings format .%s" % ext
with open(path, 'w') as settings_file:
settings_file.write(encoder(to_save))
def check_connection(server="www.lbry.io", port=80):
"""Attempts to open a socket to server:port and returns True if successful."""
try:
host = socket.gethostbyname(server)
s = socket.create_connection((host, port), 2)
return True
except Exception as ex:
log.info(
"Failed to connect to %s:%s. Maybe the internet connection is not working",
server, port, exc_info=True)
return False
for l in blobhash:
if l not in "0123456789abcdef":
return False
return True
def setup_certs_for_windows():
if getattr(sys, 'frozen', False) and os.name == "nt":
cert_path = os.path.join(os.path.dirname(sys.executable), "cacert.pem")
os.environ["REQUESTS_CA_BUNDLE"] = cert_path
+2 -2
View File
@@ -1,7 +1,7 @@
import binascii
import logging
from Crypto.Cipher import AES
from lbrynet.conf import BLOB_SIZE
from lbrynet.conf import settings
from lbrynet.core.BlobInfo import BlobInfo
@@ -67,7 +67,7 @@ class CryptStreamBlobMaker(object):
self.length = 0
def write(self, data):
max_bytes_to_write = BLOB_SIZE - self.length - 1
max_bytes_to_write = settings.BLOB_SIZE - self.length - 1
done = False
if max_bytes_to_write <= len(data):
num_bytes_to_write = max_bytes_to_write
@@ -48,9 +48,9 @@ class CryptStreamDownloader(object):
@param blob_manager: A BlobManager object
@param payment_rate_manager: A PaymentRateManager object
@param payment_rate_manager: A NegotiatedPaymentRateManager object
@param wallet: An object which implements the ILBRYWallet interface
@param wallet: An object which implements the IWallet interface
@return:
"""
+8 -5
View File
@@ -1,14 +1,17 @@
import logging
# import os
def migrate_db(db_dir, start, end):
current = start
old_dirs = []
while current < end:
if current == 0:
from lbrynet.db_migrator.migrate0to1 import do_migration
old_dirs.append(do_migration(db_dir))
current += 1
# if os.name == "nt":
# return old_dirs
# while current < end:
# if current == 0:
# from lbrynet.db_migrator.migrate0to1 import do_migration
# old_dirs.append(do_migration(db_dir))
# current += 1
return old_dirs
+307 -307
View File
@@ -1,307 +1,307 @@
import sqlite3
import unqlite
import leveldb
import shutil
import os
import logging
import json
log = logging.getLogger(__name__)
known_dbs = ['lbryfile_desc.db', 'lbryfiles.db', 'valuable_blobs.db', 'blobs.db',
'lbryfile_blob.db', 'lbryfile_info.db', 'settings.db', 'blind_settings.db',
'blind_peers.db', 'blind_info.db', 'lbryfile_info.db', 'lbryfile_manager.db',
'live_stream.db', 'stream_info.db', 'stream_blob.db', 'stream_desc.db']
def do_move(from_dir, to_dir):
for known_db in known_dbs:
known_db_path = os.path.join(from_dir, known_db)
if os.path.exists(known_db_path):
log.debug("Moving %s to %s",
os.path.abspath(known_db_path),
os.path.abspath(os.path.join(to_dir, known_db)))
shutil.move(known_db_path, os.path.join(to_dir, known_db))
else:
log.debug("Did not find %s", os.path.abspath(known_db_path))
def do_migration(db_dir):
old_dir = os.path.join(db_dir, "_0_to_1_old")
new_dir = os.path.join(db_dir, "_0_to_1_new")
try:
log.info("Moving dbs from the real directory to %s", os.path.abspath(old_dir))
os.makedirs(old_dir)
do_move(db_dir, old_dir)
except:
log.error("An error occurred moving the old db files.")
raise
try:
log.info("Creating the new directory in %s", os.path.abspath(new_dir))
os.makedirs(new_dir)
except:
log.error("An error occurred creating the new directory.")
raise
try:
log.info("Doing the migration")
migrate_blob_db(old_dir, new_dir)
migrate_lbryfile_db(old_dir, new_dir)
migrate_livestream_db(old_dir, new_dir)
migrate_ptc_db(old_dir, new_dir)
migrate_lbryfile_manager_db(old_dir, new_dir)
migrate_settings_db(old_dir, new_dir)
migrate_repeater_db(old_dir, new_dir)
log.info("Migration succeeded")
except:
log.error("An error occurred during the migration. Restoring.")
do_move(old_dir, db_dir)
raise
try:
log.info("Moving dbs in the new directory to the real directory")
do_move(new_dir, db_dir)
db_revision = open(os.path.join(db_dir, 'db_revision'), mode='w+')
db_revision.write("1")
db_revision.close()
os.rmdir(new_dir)
except:
log.error("An error occurred moving the new db files.")
raise
return old_dir
def migrate_blob_db(old_db_dir, new_db_dir):
old_blob_db_path = os.path.join(old_db_dir, "blobs.db")
if not os.path.exists(old_blob_db_path):
return True
old_db = leveldb.LevelDB(old_blob_db_path)
new_db_conn = sqlite3.connect(os.path.join(new_db_dir, "blobs.db"))
c = new_db_conn.cursor()
c.execute("create table if not exists blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" last_verified_time real, " +
" next_announce_time real"
")")
new_db_conn.commit()
c = new_db_conn.cursor()
for blob_hash, blob_info in old_db.RangeIter():
blob_length, verified_time, announce_time = json.loads(blob_info)
c.execute("insert into blobs values (?, ?, ?, ?)",
(blob_hash, blob_length, verified_time, announce_time))
new_db_conn.commit()
new_db_conn.close()
def migrate_lbryfile_db(old_db_dir, new_db_dir):
old_lbryfile_db_path = os.path.join(old_db_dir, "lbryfiles.db")
if not os.path.exists(old_lbryfile_db_path):
return True
stream_info_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_info.db"))
stream_blob_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_blob.db"))
stream_desc_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_desc.db"))
db_conn = sqlite3.connect(os.path.join(new_db_dir, "lbryfile_info.db"))
c = db_conn.cursor()
c.execute("create table if not exists lbry_files (" +
" stream_hash text primary key, " +
" key text, " +
" stream_name text, " +
" suggested_file_name text" +
")")
c.execute("create table if not exists lbry_file_blobs (" +
" blob_hash text, " +
" stream_hash text, " +
" position integer, " +
" iv text, " +
" length integer, " +
" foreign key(stream_hash) references lbry_files(stream_hash)" +
")")
c.execute("create table if not exists lbry_file_descriptors (" +
" sd_blob_hash TEXT PRIMARY KEY, " +
" stream_hash TEXT, " +
" foreign key(stream_hash) references lbry_files(stream_hash)" +
")")
db_conn.commit()
c = db_conn.cursor()
for stream_hash, stream_info in stream_info_db.RangeIter():
key, name, suggested_file_name = json.loads(stream_info)
c.execute("insert into lbry_files values (?, ?, ?, ?)",
(stream_hash, key, name, suggested_file_name))
db_conn.commit()
c = db_conn.cursor()
for blob_hash_stream_hash, blob_info in stream_blob_db.RangeIter():
b_h, s_h = json.loads(blob_hash_stream_hash)
position, iv, length = json.loads(blob_info)
c.execute("insert into lbry_file_blobs values (?, ?, ?, ?, ?)",
(b_h, s_h, position, iv, length))
db_conn.commit()
c = db_conn.cursor()
for sd_blob_hash, stream_hash in stream_desc_db.RangeIter():
c.execute("insert into lbry_file_descriptors values (?, ?)",
(sd_blob_hash, stream_hash))
db_conn.commit()
db_conn.close()
def migrate_livestream_db(old_db_dir, new_db_dir):
old_db_path = os.path.join(old_db_dir, "stream_info.db")
if not os.path.exists(old_db_path):
return True
stream_info_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_info.db"))
stream_blob_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_blob.db"))
stream_desc_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_desc.db"))
db_conn = sqlite3.connect(os.path.join(new_db_dir, "live_stream.db"))
c = db_conn.cursor()
c.execute("create table if not exists live_streams (" +
" stream_hash text primary key, " +
" public_key text, " +
" key text, " +
" stream_name text, " +
" next_announce_time real" +
")")
c.execute("create table if not exists live_stream_blobs (" +
" blob_hash text, " +
" stream_hash text, " +
" position integer, " +
" revision integer, " +
" iv text, " +
" length integer, " +
" signature text, " +
" foreign key(stream_hash) references live_streams(stream_hash)" +
")")
c.execute("create table if not exists live_stream_descriptors (" +
" sd_blob_hash TEXT PRIMARY KEY, " +
" stream_hash TEXT, " +
" foreign key(stream_hash) references live_streams(stream_hash)" +
")")
db_conn.commit()
c = db_conn.cursor()
for stream_hash, stream_info in stream_info_db.RangeIter():
public_key, key, name, next_announce_time = json.loads(stream_info)
c.execute("insert into live_streams values (?, ?, ?, ?, ?)",
(stream_hash, public_key, key, name, next_announce_time))
db_conn.commit()
c = db_conn.cursor()
for blob_hash_stream_hash, blob_info in stream_blob_db.RangeIter():
b_h, s_h = json.loads(blob_hash_stream_hash)
position, revision, iv, length, signature = json.loads(blob_info)
c.execute("insert into live_stream_blobs values (?, ?, ?, ?, ?, ?, ?)",
(b_h, s_h, position, revision, iv, length, signature))
db_conn.commit()
c = db_conn.cursor()
for sd_blob_hash, stream_hash in stream_desc_db.RangeIter():
c.execute("insert into live_stream_descriptors values (?, ?)",
(sd_blob_hash, stream_hash))
db_conn.commit()
db_conn.close()
def migrate_ptc_db(old_db_dir, new_db_dir):
old_db_path = os.path.join(old_db_dir, "ptcwallet.db")
if not os.path.exists(old_db_path):
return True
old_db = leveldb.LevelDB(old_db_path)
try:
p_key = old_db.Get("private_key")
new_db = unqlite.UnQLite(os.path.join(new_db_dir, "ptcwallet.db"))
new_db['private_key'] = p_key
except KeyError:
pass
def migrate_lbryfile_manager_db(old_db_dir, new_db_dir):
old_db_path = os.path.join(old_db_dir, "lbryfiles.db")
if not os.path.exists(old_db_path):
return True
old_db = leveldb.LevelDB(old_db_path)
new_db = sqlite3.connect(os.path.join(new_db_dir, "lbryfile_info.db"))
c = new_db.cursor()
c.execute("create table if not exists lbry_file_options (" +
" blob_data_rate real, " +
" status text," +
" stream_hash text,"
" foreign key(stream_hash) references lbry_files(stream_hash)" +
")")
new_db.commit()
LBRYFILE_STATUS = "t"
LBRYFILE_OPTIONS = "o"
c = new_db.cursor()
for k, v in old_db.RangeIter():
key_type, stream_hash = json.loads(k)
if key_type == LBRYFILE_STATUS:
try:
rate = json.loads(old_db.Get(json.dumps((LBRYFILE_OPTIONS, stream_hash))))[0]
except KeyError:
rate = None
c.execute("insert into lbry_file_options values (?, ?, ?)",
(rate, v, stream_hash))
new_db.commit()
new_db.close()
def migrate_settings_db(old_db_dir, new_db_dir):
old_settings_db_path = os.path.join(old_db_dir, "settings.db")
if not os.path.exists(old_settings_db_path):
return True
old_db = leveldb.LevelDB(old_settings_db_path)
new_db = unqlite.UnQLite(os.path.join(new_db_dir, "settings.db"))
for k, v in old_db.RangeIter():
new_db[k] = v
def migrate_repeater_db(old_db_dir, new_db_dir):
old_repeater_db_path = os.path.join(old_db_dir, "valuable_blobs.db")
if not os.path.exists(old_repeater_db_path):
return True
old_db = leveldb.LevelDB(old_repeater_db_path)
info_db = sqlite3.connect(os.path.join(new_db_dir, "blind_info.db"))
peer_db = sqlite3.connect(os.path.join(new_db_dir, "blind_peers.db"))
unql_db = unqlite.UnQLite(os.path.join(new_db_dir, "blind_settings.db"))
BLOB_INFO_TYPE = 'b'
SETTING_TYPE = 's'
PEER_TYPE = 'p'
info_c = info_db.cursor()
info_c.execute("create table if not exists valuable_blobs (" +
" blob_hash text primary key, " +
" blob_length integer, " +
" reference text, " +
" peer_host text, " +
" peer_port integer, " +
" peer_score text" +
")")
info_db.commit()
peer_c = peer_db.cursor()
peer_c.execute("create table if not exists approved_peers (" +
" ip_address text, " +
" port integer" +
")")
peer_db.commit()
info_c = info_db.cursor()
peer_c = peer_db.cursor()
for k, v in old_db.RangeIter():
key_type, key_rest = json.loads(k)
if key_type == PEER_TYPE:
host, port = key_rest
peer_c.execute("insert into approved_peers values (?, ?)",
(host, port))
elif key_type == SETTING_TYPE:
unql_db[key_rest] = v
elif key_type == BLOB_INFO_TYPE:
blob_hash = key_rest
length, reference, peer_host, peer_port, peer_score = json.loads(v)
info_c.execute("insert into valuable_blobs values (?, ?, ?, ?, ?, ?)",
(blob_hash, length, reference, peer_host, peer_port, peer_score))
info_db.commit()
peer_db.commit()
info_db.close()
peer_db.close()
# import sqlite3
# import unqlite
# import leveldb
# import shutil
# import os
# import logging
# import json
#
#
# log = logging.getLogger(__name__)
#
#
# known_dbs = ['lbryfile_desc.db', 'lbryfiles.db', 'valuable_blobs.db', 'blobs.db',
# 'lbryfile_blob.db', 'lbryfile_info.db', 'settings.db', 'blind_settings.db',
# 'blind_peers.db', 'blind_info.db', 'lbryfile_info.db', 'lbryfile_manager.db',
# 'live_stream.db', 'stream_info.db', 'stream_blob.db', 'stream_desc.db']
#
#
# def do_move(from_dir, to_dir):
# for known_db in known_dbs:
# known_db_path = os.path.join(from_dir, known_db)
# if os.path.exists(known_db_path):
# log.debug("Moving %s to %s",
# os.path.abspath(known_db_path),
# os.path.abspath(os.path.join(to_dir, known_db)))
# shutil.move(known_db_path, os.path.join(to_dir, known_db))
# else:
# log.debug("Did not find %s", os.path.abspath(known_db_path))
#
#
# def do_migration(db_dir):
# old_dir = os.path.join(db_dir, "_0_to_1_old")
# new_dir = os.path.join(db_dir, "_0_to_1_new")
# try:
# log.info("Moving dbs from the real directory to %s", os.path.abspath(old_dir))
# os.makedirs(old_dir)
# do_move(db_dir, old_dir)
# except:
# log.error("An error occurred moving the old db files.")
# raise
# try:
# log.info("Creating the new directory in %s", os.path.abspath(new_dir))
# os.makedirs(new_dir)
#
# except:
# log.error("An error occurred creating the new directory.")
# raise
# try:
# log.info("Doing the migration")
# migrate_blob_db(old_dir, new_dir)
# migrate_lbryfile_db(old_dir, new_dir)
# migrate_livestream_db(old_dir, new_dir)
# migrate_ptc_db(old_dir, new_dir)
# migrate_lbryfile_manager_db(old_dir, new_dir)
# migrate_settings_db(old_dir, new_dir)
# migrate_repeater_db(old_dir, new_dir)
# log.info("Migration succeeded")
# except:
# log.error("An error occurred during the migration. Restoring.")
# do_move(old_dir, db_dir)
# raise
# try:
# log.info("Moving dbs in the new directory to the real directory")
# do_move(new_dir, db_dir)
# db_revision = open(os.path.join(db_dir, 'db_revision'), mode='w+')
# db_revision.write("1")
# db_revision.close()
# os.rmdir(new_dir)
# except:
# log.error("An error occurred moving the new db files.")
# raise
# return old_dir
#
#
# def migrate_blob_db(old_db_dir, new_db_dir):
# old_blob_db_path = os.path.join(old_db_dir, "blobs.db")
# if not os.path.exists(old_blob_db_path):
# return True
#
# old_db = leveldb.LevelDB(old_blob_db_path)
# new_db_conn = sqlite3.connect(os.path.join(new_db_dir, "blobs.db"))
# c = new_db_conn.cursor()
# c.execute("create table if not exists blobs (" +
# " blob_hash text primary key, " +
# " blob_length integer, " +
# " last_verified_time real, " +
# " next_announce_time real"
# ")")
# new_db_conn.commit()
# c = new_db_conn.cursor()
# for blob_hash, blob_info in old_db.RangeIter():
# blob_length, verified_time, announce_time = json.loads(blob_info)
# c.execute("insert into blobs values (?, ?, ?, ?)",
# (blob_hash, blob_length, verified_time, announce_time))
# new_db_conn.commit()
# new_db_conn.close()
#
#
# def migrate_lbryfile_db(old_db_dir, new_db_dir):
# old_lbryfile_db_path = os.path.join(old_db_dir, "lbryfiles.db")
# if not os.path.exists(old_lbryfile_db_path):
# return True
#
# stream_info_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_info.db"))
# stream_blob_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_blob.db"))
# stream_desc_db = leveldb.LevelDB(os.path.join(old_db_dir, "lbryfile_desc.db"))
#
# db_conn = sqlite3.connect(os.path.join(new_db_dir, "lbryfile_info.db"))
# c = db_conn.cursor()
# c.execute("create table if not exists lbry_files (" +
# " stream_hash text primary key, " +
# " key text, " +
# " stream_name text, " +
# " suggested_file_name text" +
# ")")
# c.execute("create table if not exists lbry_file_blobs (" +
# " blob_hash text, " +
# " stream_hash text, " +
# " position integer, " +
# " iv text, " +
# " length integer, " +
# " foreign key(stream_hash) references lbry_files(stream_hash)" +
# ")")
# c.execute("create table if not exists lbry_file_descriptors (" +
# " sd_blob_hash TEXT PRIMARY KEY, " +
# " stream_hash TEXT, " +
# " foreign key(stream_hash) references lbry_files(stream_hash)" +
# ")")
# db_conn.commit()
# c = db_conn.cursor()
# for stream_hash, stream_info in stream_info_db.RangeIter():
# key, name, suggested_file_name = json.loads(stream_info)
# c.execute("insert into lbry_files values (?, ?, ?, ?)",
# (stream_hash, key, name, suggested_file_name))
# db_conn.commit()
# c = db_conn.cursor()
# for blob_hash_stream_hash, blob_info in stream_blob_db.RangeIter():
# b_h, s_h = json.loads(blob_hash_stream_hash)
# position, iv, length = json.loads(blob_info)
# c.execute("insert into lbry_file_blobs values (?, ?, ?, ?, ?)",
# (b_h, s_h, position, iv, length))
# db_conn.commit()
# c = db_conn.cursor()
# for sd_blob_hash, stream_hash in stream_desc_db.RangeIter():
# c.execute("insert into lbry_file_descriptors values (?, ?)",
# (sd_blob_hash, stream_hash))
# db_conn.commit()
# db_conn.close()
#
#
# def migrate_livestream_db(old_db_dir, new_db_dir):
# old_db_path = os.path.join(old_db_dir, "stream_info.db")
# if not os.path.exists(old_db_path):
# return True
# stream_info_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_info.db"))
# stream_blob_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_blob.db"))
# stream_desc_db = leveldb.LevelDB(os.path.join(old_db_dir, "stream_desc.db"))
#
# db_conn = sqlite3.connect(os.path.join(new_db_dir, "live_stream.db"))
#
# c = db_conn.cursor()
#
# c.execute("create table if not exists live_streams (" +
# " stream_hash text primary key, " +
# " public_key text, " +
# " key text, " +
# " stream_name text, " +
# " next_announce_time real" +
# ")")
# c.execute("create table if not exists live_stream_blobs (" +
# " blob_hash text, " +
# " stream_hash text, " +
# " position integer, " +
# " revision integer, " +
# " iv text, " +
# " length integer, " +
# " signature text, " +
# " foreign key(stream_hash) references live_streams(stream_hash)" +
# ")")
# c.execute("create table if not exists live_stream_descriptors (" +
# " sd_blob_hash TEXT PRIMARY KEY, " +
# " stream_hash TEXT, " +
# " foreign key(stream_hash) references live_streams(stream_hash)" +
# ")")
#
# db_conn.commit()
#
# c = db_conn.cursor()
# for stream_hash, stream_info in stream_info_db.RangeIter():
# public_key, key, name, next_announce_time = json.loads(stream_info)
# c.execute("insert into live_streams values (?, ?, ?, ?, ?)",
# (stream_hash, public_key, key, name, next_announce_time))
# db_conn.commit()
# c = db_conn.cursor()
# for blob_hash_stream_hash, blob_info in stream_blob_db.RangeIter():
# b_h, s_h = json.loads(blob_hash_stream_hash)
# position, revision, iv, length, signature = json.loads(blob_info)
# c.execute("insert into live_stream_blobs values (?, ?, ?, ?, ?, ?, ?)",
# (b_h, s_h, position, revision, iv, length, signature))
# db_conn.commit()
# c = db_conn.cursor()
# for sd_blob_hash, stream_hash in stream_desc_db.RangeIter():
# c.execute("insert into live_stream_descriptors values (?, ?)",
# (sd_blob_hash, stream_hash))
# db_conn.commit()
# db_conn.close()
#
#
# def migrate_ptc_db(old_db_dir, new_db_dir):
# old_db_path = os.path.join(old_db_dir, "ptcwallet.db")
# if not os.path.exists(old_db_path):
# return True
# old_db = leveldb.LevelDB(old_db_path)
# try:
# p_key = old_db.Get("private_key")
# new_db = unqlite.UnQLite(os.path.join(new_db_dir, "ptcwallet.db"))
# new_db['private_key'] = p_key
# except KeyError:
# pass
#
#
# def migrate_lbryfile_manager_db(old_db_dir, new_db_dir):
# old_db_path = os.path.join(old_db_dir, "lbryfiles.db")
# if not os.path.exists(old_db_path):
# return True
# old_db = leveldb.LevelDB(old_db_path)
# new_db = sqlite3.connect(os.path.join(new_db_dir, "lbryfile_info.db"))
# c = new_db.cursor()
# c.execute("create table if not exists lbry_file_options (" +
# " blob_data_rate real, " +
# " status text," +
# " stream_hash text,"
# " foreign key(stream_hash) references lbry_files(stream_hash)" +
# ")")
# new_db.commit()
# FILE_STATUS = "t"
# FILE_OPTIONS = "o"
# c = new_db.cursor()
# for k, v in old_db.RangeIter():
# key_type, stream_hash = json.loads(k)
# if key_type == FILE_STATUS:
# try:
# rate = json.loads(old_db.Get(json.dumps((FILE_OPTIONS, stream_hash))))[0]
# except KeyError:
# rate = None
# c.execute("insert into lbry_file_options values (?, ?, ?)",
# (rate, v, stream_hash))
# new_db.commit()
# new_db.close()
#
#
# def migrate_settings_db(old_db_dir, new_db_dir):
# old_settings_db_path = os.path.join(old_db_dir, "settings.db")
# if not os.path.exists(old_settings_db_path):
# return True
# old_db = leveldb.LevelDB(old_settings_db_path)
# new_db = unqlite.UnQLite(os.path.join(new_db_dir, "settings.db"))
# for k, v in old_db.RangeIter():
# new_db[k] = v
#
#
# def migrate_repeater_db(old_db_dir, new_db_dir):
# old_repeater_db_path = os.path.join(old_db_dir, "valuable_blobs.db")
# if not os.path.exists(old_repeater_db_path):
# return True
# old_db = leveldb.LevelDB(old_repeater_db_path)
# info_db = sqlite3.connect(os.path.join(new_db_dir, "blind_info.db"))
# peer_db = sqlite3.connect(os.path.join(new_db_dir, "blind_peers.db"))
# unql_db = unqlite.UnQLite(os.path.join(new_db_dir, "blind_settings.db"))
# BLOB_INFO_TYPE = 'b'
# SETTING_TYPE = 's'
# PEER_TYPE = 'p'
# info_c = info_db.cursor()
# info_c.execute("create table if not exists valuable_blobs (" +
# " blob_hash text primary key, " +
# " blob_length integer, " +
# " reference text, " +
# " peer_host text, " +
# " peer_port integer, " +
# " peer_score text" +
# ")")
# info_db.commit()
# peer_c = peer_db.cursor()
# peer_c.execute("create table if not exists approved_peers (" +
# " ip_address text, " +
# " port integer" +
# ")")
# peer_db.commit()
# info_c = info_db.cursor()
# peer_c = peer_db.cursor()
# for k, v in old_db.RangeIter():
# key_type, key_rest = json.loads(k)
# if key_type == PEER_TYPE:
# host, port = key_rest
# peer_c.execute("insert into approved_peers values (?, ?)",
# (host, port))
# elif key_type == SETTING_TYPE:
# unql_db[key_rest] = v
# elif key_type == BLOB_INFO_TYPE:
# blob_hash = key_rest
# length, reference, peer_host, peer_port, peer_score = json.loads(v)
# info_c.execute("insert into valuable_blobs values (?, ?, ?, ?, ?, ?)",
# (blob_hash, length, reference, peer_host, peer_port, peer_score))
# info_db.commit()
# peer_db.commit()
# info_db.close()
# peer_db.close()
-3
View File
@@ -8,10 +8,7 @@
# may be created by processing this file with epydoc: http://epydoc.sf.net
import UserDict
#import sqlite3
import cPickle as pickle
import time
import os
import constants
+5 -5
View File
@@ -7,7 +7,7 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import hashlib, random, struct, time, math, binascii
import hashlib, random, struct, time, binascii
import argparse
from twisted.internet import defer, error
import constants
@@ -240,9 +240,9 @@ class Node(object):
known_nodes = {}
def log_error(err, n):
log.error("error storing blob_hash %s at %s", binascii.hexlify(blob_hash), str(n))
log.error(binascii.hexlify(err.getErrorMessage()))
log.error(err.getTraceback())
log.debug("error storing blob_hash %s at %s", binascii.hexlify(blob_hash), str(n))
log.debug(err.getErrorMessage())
log.debug(err.getTraceback())
def log_success(res):
log.debug("Response to store request: %s", str(res))
@@ -1017,4 +1017,4 @@ def main():
twisted.internet.reactor.run()
if __name__ == '__main__':
main()
main()
+14
View File
@@ -7,6 +7,8 @@
# The docstrings in this module contain epytext markup; API documentation
# may be created by processing this file with epydoc: http://epydoc.sf.net
import logging
import binascii
import time
from twisted.internet import protocol, defer
@@ -20,9 +22,18 @@ import msgformat
from contact import Contact
reactor = twisted.internet.reactor
log = logging.getLogger(__name__)
class TimeoutError(Exception):
""" Raised when a RPC times out """
def __init__(self, remote_contact_id):
# remote_contact_id is a binary blob so we need to convert it
# into something more readable
msg = 'Timeout connecting to {}'.format(binascii.hexlify(remote_contact_id))
Exception.__init__(self, msg)
self.remote_contact_id = remote_contact_id
class KademliaProtocol(protocol.DatagramProtocol):
""" Implements all low-level network-related functions of a Kademlia node """
@@ -110,6 +121,9 @@ class KademliaProtocol(protocol.DatagramProtocol):
except encoding.DecodeError:
# We received some rubbish here
return
except IndexError:
log.warning("Couldn't decode dht datagram from %s", address)
return
message = self._translator.fromPrimitive(msgPrimitive)
remoteContact = Contact(message.nodeID, address[0], address[1], self)
+2 -2
View File
@@ -20,7 +20,7 @@
import os, sys, time, signal, hashlib, random
import sys, hashlib, random
import twisted.internet.reactor
from lbrynet.dht.node import Node
#from entangled.kademlia.datastore import SQLiteDataStore
@@ -106,7 +106,7 @@ def stop():
if __name__ == '__main__':
import sys, os
import sys
if len(sys.argv) < 2:
print 'Usage:\n%s UDP_PORT [KNOWN_NODE_IP KNOWN_NODE_PORT]' % sys.argv[0]
print 'or:\n%s UDP_PORT [FILE_WITH_KNOWN_NODES]' % sys.argv[0]
+58 -2
View File
@@ -548,13 +548,13 @@ class IStreamDescriptorValidator(Interface):
"""
class ILBRYWallet(Interface):
class IWallet(Interface):
"""
Send and receive payments.
To send a payment, a payment reservation must be obtained first. This guarantees that a payment
isn't promised if it can't be paid. When the service in question is rendered, the payment
reservation must be given to the ILBRYWallet along with the final price. The reservation can also
reservation must be given to the IWallet along with the final price. The reservation can also
be canceled.
"""
def stop(self):
@@ -647,4 +647,60 @@ class ILBRYWallet(Interface):
@type amount: float
@return: None
"""
class IBlobPriceModel(Interface):
"""
A blob price model
Used by INegotiationStrategy classes
"""
def calculate_price(self, blob):
"""
Calculate the price for a blob
@param blob: a blob hash
@type blob: str
@return: blob price target
@type: Decimal
"""
class INegotiationStrategy(Interface):
"""
Strategy to negotiate download payment rates
"""
def make_offer(self, peer, blobs):
"""
Make a rate offer for the given peer and blobs
@param peer: peer to make an offer to
@type: str
@param blobs: blob hashes to make an offer for
@type: list
@return: rate offer
@rtype: Offer
"""
def respond_to_offer(self, offer, peer, blobs):
"""
Respond to a rate offer given by a peer
@param offer: offer to reply to
@type: Offer
@param peer: peer to make an offer to
@type: str
@param blobs: blob hashes to make an offer for
@type: list
@return: accepted, rejected, or unset offer
@rtype: Offer
"""
@@ -11,8 +11,8 @@ from lbrynet.core.sqlite_helpers import rerun_if_locked
log = logging.getLogger(__name__)
class DBLBRYFileMetadataManager(object):
"""Store and provide access to LBRY file metadata using leveldb files"""
class DBEncryptedFileMetadataManager(object):
"""Store and provide access to LBRY file metadata using sqlite"""
def __init__(self, db_dir):
self.db_dir = db_dir
@@ -48,7 +48,7 @@ class DBLBRYFileMetadataManager(object):
return self._add_blobs_to_stream(stream_hash, blobs, ignore_duplicate_error=True)
def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
log.info("Getting blobs for a stream. Count is %s", str(count))
log.debug("Getting blobs for a stream. Count is %s", str(count))
def get_positions_of_start_and_end():
if start_blob is not None:
@@ -236,14 +236,14 @@ class DBLBRYFileMetadataManager(object):
@rerun_if_locked
def _get_sd_blob_hashes_for_stream(self, stream_hash):
log.info("Looking up sd blob hashes for stream hash %s", str(stream_hash))
log.debug("Looking up sd blob hashes for stream hash %s", str(stream_hash))
d = self.db_conn.runQuery("select sd_blob_hash from lbry_file_descriptors where stream_hash = ?",
(stream_hash,))
d.addCallback(lambda results: [r[0] for r in results])
return d
class TempLBRYFileMetadataManager(object):
class TempEncryptedFileMetadataManager(object):
def __init__(self):
self.streams = {}
self.stream_blobs = {}
+3 -3
View File
@@ -11,7 +11,7 @@ import os
log = logging.getLogger(__name__)
LBRYFileStreamType = "lbryfile"
EncryptedFileStreamType = "lbryfile"
def save_sd_info(stream_info_manager, sd_info, ignore_duplicate=False):
@@ -50,7 +50,7 @@ def get_sd_info(stream_info_manager, stream_hash, include_blobs):
def format_info(stream_info):
fields = {}
fields['stream_type'] = LBRYFileStreamType
fields['stream_type'] = EncryptedFileStreamType
fields['stream_name'] = stream_info[1]
fields['key'] = stream_info[0]
fields['suggested_file_name'] = stream_info[2]
@@ -121,7 +121,7 @@ def create_plain_sd(stream_info_manager, stream_hash, file_name, overwrite_exist
return d
class LBRYFileStreamDescriptorValidator(object):
class EncryptedFileStreamDescriptorValidator(object):
def __init__(self, raw_info):
self.raw_info = raw_info
+2
View File
@@ -0,0 +1,2 @@
from lbrynet.lbryfile.StreamDescriptor import get_sd_info
from lbrynet.lbryfile.StreamDescriptor import publish_sd_blob
@@ -8,7 +8,7 @@ from lbrynet.cryptstream.client.CryptStreamDownloader import CryptStreamDownload
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.StreamDescriptor import StreamMetadata
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbryfile.client.LBRYFileMetadataHandler import LBRYFileMetadataHandler
from lbrynet.lbryfile.client.EncryptedFileMetadataHandler import EncryptedFileMetadataHandler
import os
from twisted.internet import defer, threads, reactor
from twisted.python.procutils import which
@@ -19,7 +19,7 @@ import traceback
log = logging.getLogger(__name__)
class LBRYFileDownloader(CryptStreamDownloader):
class EncryptedFileDownloader(CryptStreamDownloader):
"""Classes which inherit from this class download LBRY files"""
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
@@ -120,10 +120,10 @@ class LBRYFileDownloader(CryptStreamDownloader):
return 0
def _get_metadata_handler(self, download_manager):
return LBRYFileMetadataHandler(self.stream_hash, self.stream_info_manager, download_manager)
return EncryptedFileMetadataHandler(self.stream_hash, self.stream_info_manager, download_manager)
class LBRYFileDownloaderFactory(object):
class EncryptedFileDownloaderFactory(object):
implements(IStreamDownloaderFactory)
def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager,
@@ -165,10 +165,10 @@ class LBRYFileDownloaderFactory(object):
pass
class LBRYFileSaver(LBRYFileDownloader):
class EncryptedFileSaver(EncryptedFileDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, download_directory, upload_allowed, file_name=None):
LBRYFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
EncryptedFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, upload_allowed)
self.download_directory = download_directory
self.file_name = file_name
@@ -182,7 +182,7 @@ class LBRYFileSaver(LBRYFileDownloader):
return str(self.file_name)
def set_stream_info(self):
d = LBRYFileDownloader.set_stream_info(self)
d = EncryptedFileDownloader.set_stream_info(self)
def set_file_name():
if self.file_name is None:
@@ -195,7 +195,7 @@ class LBRYFileSaver(LBRYFileDownloader):
return d
def stop(self, err=None):
d = LBRYFileDownloader.stop(self, err=err)
d = EncryptedFileDownloader.stop(self, err=err)
d.addCallback(lambda _: self._delete_from_info_manager())
return d
@@ -257,15 +257,15 @@ class LBRYFileSaver(LBRYFileDownloader):
return self.stream_info_manager.delete_stream(self.stream_hash)
class LBRYFileSaverFactory(LBRYFileDownloaderFactory):
class EncryptedFileSaverFactory(EncryptedFileDownloaderFactory):
def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager,
wallet, download_directory):
LBRYFileDownloaderFactory.__init__(self, peer_finder, rate_limiter, blob_manager,
EncryptedFileDownloaderFactory.__init__(self, peer_finder, rate_limiter, blob_manager,
stream_info_manager, wallet)
self.download_directory = download_directory
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
return LBRYFileSaver(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
return EncryptedFileSaver(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
self.stream_info_manager, payment_rate_manager, self.wallet,
self.download_directory, upload_allowed)
@@ -274,16 +274,16 @@ class LBRYFileSaverFactory(LBRYFileDownloaderFactory):
return "Save"
class LBRYFileOpener(LBRYFileDownloader):
class EncryptedFileOpener(EncryptedFileDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, upload_allowed):
LBRYFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
EncryptedFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, upload_allowed)
self.process = None
self.process_log = None
def stop(self, err=None):
d = LBRYFileDownloader.stop(self, err=err)
d = EncryptedFileDownloader.stop(self, err=err)
d.addCallback(lambda _: self._delete_from_info_manager())
return d
@@ -333,7 +333,7 @@ class LBRYFileOpener(LBRYFileDownloader):
return self.stream_info_manager.delete_stream(self.stream_hash)
class LBRYFileOpenerFactory(LBRYFileDownloaderFactory):
class EncryptedFileOpenerFactory(EncryptedFileDownloaderFactory):
def can_download(self, sd_validator):
if which('vlc'):
return True
@@ -346,7 +346,7 @@ class LBRYFileOpenerFactory(LBRYFileDownloaderFactory):
return False
def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
return LBRYFileOpener(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
return EncryptedFileOpener(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
self.stream_info_manager, payment_rate_manager, self.wallet, upload_allowed)
@staticmethod
@@ -7,7 +7,7 @@ from lbrynet.interfaces import IMetadataHandler
log = logging.getLogger(__name__)
class LBRYFileMetadataHandler(object):
class EncryptedFileMetadataHandler(object):
implements(IMetadataHandler)
def __init__(self, stream_hash, stream_info_manager, download_manager):
@@ -1,12 +1,12 @@
from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamType, LBRYFileStreamDescriptorValidator
from lbrynet.lbryfile.StreamDescriptor import EncryptedFileStreamType, EncryptedFileStreamDescriptorValidator
from lbrynet.core.DownloadOption import DownloadOption, DownloadOptionChoice
def add_lbry_file_to_sd_identifier(sd_identifier):
sd_identifier.add_stream_type(LBRYFileStreamType, LBRYFileStreamDescriptorValidator, LBRYFileOptions())
sd_identifier.add_stream_type(EncryptedFileStreamType, EncryptedFileStreamDescriptorValidator, EncryptedFileOptions())
class LBRYFileOptions(object):
class EncryptedFileOptions(object):
def __init__(self):
pass
@@ -14,16 +14,16 @@ class LBRYFileOptions(object):
prm = payment_rate_manager
def get_default_data_rate_description():
if prm.min_blob_data_payment_rate is None:
if prm.base.min_blob_data_payment_rate is None:
return "Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate)
else:
return "%f LBC/MB" % prm.min_blob_data_payment_rate
return "%f LBC/MB" % prm.base.min_blob_data_payment_rate
rate_choices = []
rate_choices.append(DownloadOptionChoice(prm.min_blob_data_payment_rate,
rate_choices.append(DownloadOptionChoice(prm.base.min_blob_data_payment_rate,
"No change - %s" % get_default_data_rate_description(),
"No change - %s" % get_default_data_rate_description()))
if prm.min_blob_data_payment_rate is not None:
if prm.base.min_blob_data_payment_rate is not None:
rate_choices.append(DownloadOptionChoice(None,
"Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate),
"Application default (%s LBC/MB)" % str(prm.base.min_blob_data_payment_rate)))
@@ -36,7 +36,7 @@ class LBRYFileOptions(object):
rate_choices,
"Rate which will be paid for data",
"data payment rate",
prm.min_blob_data_payment_rate,
prm.base.min_blob_data_payment_rate,
get_default_data_rate_description()
),
DownloadOption(
@@ -7,19 +7,18 @@ import logging
import os
from lbrynet.core.StreamDescriptor import PlainStreamDescriptorWriter
from lbrynet.cryptstream.CryptStreamCreator import CryptStreamCreator
from lbrynet import conf
from lbrynet.conf import settings
from lbrynet.lbryfile.StreamDescriptor import get_sd_info
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from twisted.protocols.basic import FileSender
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
log = logging.getLogger(__name__)
class LBRYFileStreamCreator(CryptStreamCreator):
class EncryptedFileStreamCreator(CryptStreamCreator):
"""
A CryptStreamCreator which adds itself and its additional metadata to an LBRYFileManager
A CryptStreamCreator which adds itself and its additional metadata to an EncryptedFileManager
"""
def __init__(self, blob_manager, lbry_file_manager, name=None,
key=None, iv_generator=None, suggested_file_name=None):
@@ -94,11 +93,11 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
into chunks and encrypted, and then a stream descriptor file with the stream parameters
and other metadata is written to disk.
@param session: An LBRYSession object.
@type session: LBRYSession
@param session: An Session object.
@type session: Session
@param lbry_file_manager: The LBRYFileManager object this LBRY File will be added to.
@type lbry_file_manager: LBRYFileManager
@param lbry_file_manager: The EncryptedFileManager object this LBRY File will be added to.
@type lbry_file_manager: EncryptedFileManager
@param file_name: The path to the plain file.
@type file_name: string
@@ -131,7 +130,7 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
def make_stream_desc_file(stream_hash):
log.debug("creating the stream descriptor file")
descriptor_file_path = os.path.join(session.db_dir, file_name + conf.CRYPTSD_FILE_EXTENSION)
descriptor_file_path = os.path.join(session.db_dir, file_name + settings.CRYPTSD_FILE_EXTENSION)
descriptor_writer = PlainStreamDescriptorWriter(descriptor_file_path)
d = get_sd_info(lbry_file_manager.stream_info_manager, stream_hash, True)
@@ -142,7 +141,7 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
base_file_name = os.path.basename(file_name)
lbry_file_creator = LBRYFileStreamCreator(session.blob_manager, lbry_file_manager, base_file_name,
lbry_file_creator = EncryptedFileStreamCreator(session.blob_manager, lbry_file_manager, base_file_name,
key, iv_generator, suggested_file_name)
def start_stream():
@@ -155,4 +154,4 @@ def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=Non
d = lbry_file_creator.setup()
d.addCallback(lambda _: start_stream())
return d
return d
@@ -1,18 +1,25 @@
"""
Download LBRY Files from LBRYnet and save them to disk.
"""
import random
import logging
from zope.interface import implements
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.StreamDescriptor import StreamMetadata
from lbrynet.lbryfile.client.LBRYFileDownloader import LBRYFileSaver, LBRYFileDownloader
from lbrynet.lbryfilemanager.LBRYFileStatusReport import LBRYFileStatusReport
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbryfile.StreamDescriptor import save_sd_info
from twisted.internet import defer
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.core.StreamDescriptor import StreamMetadata
from lbrynet.lbryfile.client.EncryptedFileDownloader import EncryptedFileSaver, EncryptedFileDownloader
from lbrynet.lbryfilemanager.EncryptedFileStatusReport import EncryptedFileStatusReport
from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbryfile.StreamDescriptor import save_sd_info
from lbrynet.reflector import reupload
from lbrynet.conf import settings
class ManagedLBRYFileDownloader(LBRYFileSaver):
log = logging.getLogger(__name__)
class ManagedEncryptedFileDownloader(EncryptedFileSaver):
STATUS_RUNNING = "running"
STATUS_STOPPED = "stopped"
@@ -21,12 +28,13 @@ class ManagedLBRYFileDownloader(LBRYFileSaver):
def __init__(self, rowid, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
lbry_file_manager, payment_rate_manager, wallet, download_directory, upload_allowed,
file_name=None):
LBRYFileSaver.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
EncryptedFileSaver.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, download_directory,
upload_allowed, file_name)
self.sd_hash = None
self.txid = None
self.uri = None
self.claim_id = None
self.rowid = rowid
self.lbry_file_manager = lbry_file_manager
self.saving_status = False
@@ -37,27 +45,41 @@ class ManagedLBRYFileDownloader(LBRYFileSaver):
def _save_sd_hash(sd_hash):
if len(sd_hash):
self.sd_hash = sd_hash[0]
d = self.wallet._get_claim_metadata_for_sd_hash(self.sd_hash)
d = self.wallet.get_claim_metadata_for_sd_hash(self.sd_hash)
else:
d = defer.succeed(None)
return d
def _save_claim_id(claim_id):
self.claim_id = claim_id
return defer.succeed(None)
def _notify_bad_claim(name, txid):
log.error("Error loading name claim for lbry file: lbry://%s, tx %s does not contain a valid claim", name, txid)
log.warning("lbry file for lbry://%s, tx %s has no claim, deleting it", name, txid)
return self.lbry_file_manager.delete_lbry_file(self)
def _save_claim(name, txid):
self.uri = name
self.txid = txid
return defer.succeed(None)
d = self.wallet.get_claimid(name, txid)
d.addCallbacks(_save_claim_id, lambda err: _notify_bad_claim(name, txid))
return d
reflector_server = random.choice(settings.reflector_servers)
d.addCallback(_save_sd_hash)
d.addCallback(lambda r: _save_claim(r[0], r[1]) if r else None)
d.addCallback(lambda _: reupload.check_and_restore_availability(self, reflector_server))
d.addCallback(lambda _: self.lbry_file_manager.get_lbry_file_status(self))
def restore_status(status):
if status == ManagedLBRYFileDownloader.STATUS_RUNNING:
if status == ManagedEncryptedFileDownloader.STATUS_RUNNING:
return self.start()
elif status == ManagedLBRYFileDownloader.STATUS_STOPPED:
elif status == ManagedEncryptedFileDownloader.STATUS_STOPPED:
return defer.succeed(False)
elif status == ManagedLBRYFileDownloader.STATUS_FINISHED:
elif status == ManagedEncryptedFileDownloader.STATUS_FINISHED:
self.completed = True
return defer.succeed(True)
@@ -69,7 +91,7 @@ class ManagedLBRYFileDownloader(LBRYFileSaver):
def set_saving_status_done():
self.saving_status = False
d = LBRYFileDownloader.stop(self, err=err) # LBRYFileSaver deletes metadata when it's stopped. We don't want that here.
d = EncryptedFileDownloader.stop(self, err=err) # EncryptedFileSaver deletes metadata when it's stopped. We don't want that here.
if change_status is True:
self.saving_status = True
d.addCallback(lambda _: self._save_status())
@@ -96,7 +118,7 @@ class ManagedLBRYFileDownloader(LBRYFileSaver):
s = "stopped"
else:
s = "running"
status = LBRYFileStatusReport(self.file_name, num_completed, num_known, s)
status = EncryptedFileStatusReport(self.file_name, num_completed, num_known, s)
return status
d = self.stream_info_manager.get_blobs_for_stream(self.stream_hash)
@@ -106,14 +128,13 @@ class ManagedLBRYFileDownloader(LBRYFileSaver):
def _start(self):
d = LBRYFileSaver._start(self)
d.addCallback(lambda _: self.stream_info_manager._get_sd_blob_hashes_for_stream(self.stream_hash))
d = EncryptedFileSaver._start(self)
d.addCallback(lambda _: self.stream_info_manager.get_sd_blob_hashes_for_stream(self.stream_hash))
def _save_sd_hash(sd_hash):
if len(sd_hash):
self.sd_hash = sd_hash[0]
d = self.wallet._get_claim_metadata_for_sd_hash(self.sd_hash)
d = self.wallet.get_claim_metadata_for_sd_hash(self.sd_hash)
else:
d = defer.succeed(None)
@@ -138,18 +159,18 @@ class ManagedLBRYFileDownloader(LBRYFileSaver):
def _save_status(self):
if self.completed is True:
s = ManagedLBRYFileDownloader.STATUS_FINISHED
s = ManagedEncryptedFileDownloader.STATUS_FINISHED
elif self.stopped is True:
s = ManagedLBRYFileDownloader.STATUS_STOPPED
s = ManagedEncryptedFileDownloader.STATUS_STOPPED
else:
s = ManagedLBRYFileDownloader.STATUS_RUNNING
s = ManagedEncryptedFileDownloader.STATUS_RUNNING
return self.lbry_file_manager.change_lbry_file_status(self, s)
def _get_progress_manager(self, download_manager):
return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager)
class ManagedLBRYFileDownloaderFactory(object):
class ManagedEncryptedFileDownloaderFactory(object):
implements(IStreamDownloaderFactory)
def __init__(self, lbry_file_manager):
@@ -9,10 +9,10 @@ from twisted.enterprise import adbapi
from twisted.internet import defer, task, reactor
from twisted.python.failure import Failure
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloaderFactory
from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamType
from lbrynet.core.PaymentRateManager import PaymentRateManager
from lbrynet.core.PaymentRateManager import NegotiatedPaymentRateManager
from lbrynet.lbryfilemanager.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet.lbryfilemanager.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet.lbryfile.StreamDescriptor import EncryptedFileStreamType
from lbrynet.cryptstream.client.CryptStreamDownloader import AlreadyStoppedError, CurrentlyStoppingError
from lbrynet.core.sqlite_helpers import rerun_if_locked
@@ -20,7 +20,7 @@ from lbrynet.core.sqlite_helpers import rerun_if_locked
log = logging.getLogger(__name__)
class LBRYFileManager(object):
class EncryptedFileManager(object):
"""
Keeps track of currently opened LBRY Files, their options, and their LBRY File specific metadata.
"""
@@ -35,7 +35,7 @@ class LBRYFileManager(object):
self.download_directory = download_directory
else:
self.download_directory = os.getcwd()
log.debug("Download directory for LBRYFileManager: %s", str(self.download_directory))
log.debug("Download directory for EncryptedFileManager: %s", str(self.download_directory))
def setup(self):
d = self._open_db()
@@ -68,25 +68,29 @@ class LBRYFileManager(object):
return dl
def _add_to_sd_identifier(self):
downloader_factory = ManagedLBRYFileDownloaderFactory(self)
self.sd_identifier.add_stream_downloader_factory(LBRYFileStreamType, downloader_factory)
downloader_factory = ManagedEncryptedFileDownloaderFactory(self)
self.sd_identifier.add_stream_downloader_factory(EncryptedFileStreamType, downloader_factory)
def _start_lbry_files(self):
def set_options_and_restore(rowid, stream_hash, options):
payment_rate_manager = PaymentRateManager(self.session.base_payment_rate_manager)
payment_rate_manager = NegotiatedPaymentRateManager(self.session.base_payment_rate_manager,
self.session.blob_tracker)
d = self.start_lbry_file(rowid, stream_hash, payment_rate_manager,
blob_data_rate=options)
d.addCallback(lambda downloader: downloader.restore())
return d
def log_error(err):
def log_error(err, rowid, stream_hash, options):
log.error("An error occurred while starting a lbry file: %s", err.getErrorMessage())
log.debug(rowid)
log.debug(stream_hash)
log.debug(options)
def start_lbry_files(lbry_files_and_options):
for rowid, stream_hash, options in lbry_files_and_options:
d = set_options_and_restore(rowid, stream_hash, options)
d.addErrback(log_error)
d.addErrback(lambda err: log_error(err, rowid, stream_hash, options))
return True
d = self._get_all_lbry_files()
@@ -98,7 +102,7 @@ class LBRYFileManager(object):
if not download_directory:
download_directory = self.download_directory
payment_rate_manager.min_blob_data_payment_rate = blob_data_rate
lbry_file_downloader = ManagedLBRYFileDownloader(rowid, stream_hash,
lbry_file_downloader = ManagedEncryptedFileDownloader(rowid, stream_hash,
self.session.peer_finder,
self.session.rate_limiter,
self.session.blob_manager,
@@ -204,7 +208,7 @@ class LBRYFileManager(object):
def _save_lbry_file(self, stream_hash, data_payment_rate):
def do_save(db_transaction):
db_transaction.execute("insert into lbry_file_options values (?, ?, ?)",
(data_payment_rate, ManagedLBRYFileDownloader.STATUS_STOPPED,
(data_payment_rate, ManagedEncryptedFileDownloader.STATUS_STOPPED,
stream_hash))
return db_transaction.lastrowid
return self.sql_db.runInteraction(do_save)
@@ -233,7 +237,7 @@ class LBRYFileManager(object):
def _get_lbry_file_status(self, rowid):
d = self.sql_db.runQuery("select status from lbry_file_options where rowid = ?",
(rowid,))
d.addCallback(lambda r: r[0][0] if len(r) else ManagedLBRYFileDownloader.STATUS_STOPPED)
d.addCallback(lambda r: r[0][0] if len(r) else ManagedEncryptedFileDownloader.STATUS_STOPPED)
return d
@rerun_if_locked
@@ -1,4 +1,4 @@
class LBRYFileStatusReport(object):
class EncryptedFileStatusReport(object):
def __init__(self, name, num_completed, num_known, running_status):
self.name = name
self.num_completed = num_completed
+2 -2
View File
@@ -6,7 +6,7 @@ from lbrynet.core.cryptoutils import get_lbry_hash_obj, get_pub_key, sign_with_p
from Crypto import Random
import binascii
import logging
from lbrynet.conf import CRYPTSD_FILE_EXTENSION
from lbrynet.conf import settings
from twisted.internet import interfaces, defer
from twisted.protocols.basic import FileSender
from zope.interface import implements
@@ -23,7 +23,7 @@ class LiveStreamCreator(CryptStreamCreator):
self.stream_info_manager = stream_info_manager
self.delete_after_num = delete_after_num
self.secret_pass_phrase = secret_pass_phrase
self.file_extension = CRYPTSD_FILE_EXTENSION
self.file_extension = settings.CRYPTSD_FILE_EXTENSION
self.finished_blob_hashes = {}
def _save_stream(self):
+12 -5
View File
@@ -19,11 +19,18 @@ class LiveStreamPaymentRateManager(object):
def accept_rate_live_blob_info(self, peer, payment_rate):
return payment_rate >= self.get_effective_min_live_blob_info_payment_rate()
def get_rate_blob_data(self, peer):
return self.get_effective_min_blob_data_payment_rate()
def get_rate_blob_data(self, peer, blobs):
response = self._payment_rate_manager.strategy.make_offer(peer, blobs)
return response.rate
def accept_rate_blob_data(self, peer, payment_rate):
return payment_rate >= self.get_effective_min_blob_data_payment_rate()
def accept_rate_blob_data(self, peer, blobs, offer):
response = self._payment_rate_manager.strategy.respond_to_offer(offer, peer, blobs)
return response.accepted
def reply_to_offer(self, peer, blobs, offer):
reply = self._payment_rate_manager.strategy.respond_to_offer(offer, peer, blobs)
self._payment_rate_manager.strategy.offer_accepted(peer, reply)
return reply
def get_effective_min_blob_data_payment_rate(self):
rate = self.min_blob_data_payment_rate
@@ -42,4 +49,4 @@ class LiveStreamPaymentRateManager(object):
return rate
def record_points_paid(self, amount):
self.points_paid += amount
self.points_paid += amount
@@ -5,7 +5,7 @@ import logging
import sys
from lbrynet.lbrylive.LiveStreamCreator import StdOutLiveStreamCreator
from lbrynet.core.BlobManager import TempBlobManager
from lbrynet.core.Session import LBRYSession
from lbrynet.core.Session import Session
from lbrynet.core.server.BlobAvailabilityHandler import BlobAvailabilityHandlerFactory
from lbrynet.core.server.BlobRequestHandler import BlobRequestHandlerFactory
from lbrynet.core.server.ServerProtocol import ServerProtocolFactory
@@ -16,7 +16,7 @@ from lbrynet.dht.node import Node
from twisted.internet import defer, task
class LBRYStdinUploader():
class StdinUploader():
"""This class reads from standard in, creates a stream, and makes it available on the network."""
def __init__(self, peer_port, dht_node_port, known_dht_nodes,
stream_info_manager_class=DBLiveStreamMetadataManager, blob_manager_class=TempBlobManager):
@@ -29,7 +29,7 @@ class LBRYStdinUploader():
"""
self.peer_port = peer_port
self.lbry_server_port = None
self.session = LBRYSession(blob_manager_class=blob_manager_class,
self.session = Session(blob_manager_class=blob_manager_class,
stream_info_manager_class=stream_info_manager_class,
dht_node_class=Node, dht_node_port=dht_node_port,
known_dht_nodes=known_dht_nodes, peer_port=self.peer_port,
@@ -99,9 +99,9 @@ def launch_stdin_uploader():
logging.basicConfig(level=logging.WARNING, filename="ul.log")
if len(sys.argv) == 4:
uploader = LBRYStdinUploader(int(sys.argv[2]), int(sys.argv[3]), [])
uploader = StdinUploader(int(sys.argv[2]), int(sys.argv[3]), [])
elif len(sys.argv) == 6:
uploader = LBRYStdinUploader(int(sys.argv[2]), int(sys.argv[3]), [(sys.argv[4], int(sys.argv[5]))])
uploader = StdinUploader(int(sys.argv[2]), int(sys.argv[3]), [(sys.argv[4], int(sys.argv[5]))])
else:
print "Usage: lbrynet-stdin-uploader <stream_name> <peer_port> <dht_node_port>" \
" [<dht_bootstrap_host> <dht_bootstrap port>]"
@@ -4,9 +4,9 @@
import logging
import sys
from lbrynet.lbrylive.client.LiveStreamDownloader import LBRYLiveStreamDownloader
from lbrynet.lbrylive.client.LiveStreamDownloader import LiveStreamDownloader
from lbrynet.core.BlobManager import TempBlobManager
from lbrynet.core.Session import LBRYSession
from lbrynet.core.Session import Session
from lbrynet.core.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.core.StreamDescriptor import BlobStreamDescriptorReader
from lbrynet.lbrylive.PaymentRateManager import BaseLiveStreamPaymentRateManager
@@ -16,7 +16,7 @@ from lbrynet.dht.node import Node
from twisted.internet import task
class LBRYStdoutDownloader():
class StdoutDownloader():
"""This class downloads a live stream from the network and outputs it to standard out."""
def __init__(self, dht_node_port, known_dht_nodes,
stream_info_manager_class=DBLiveStreamMetadataManager, blob_manager_class=TempBlobManager):
@@ -27,7 +27,7 @@ class LBRYStdoutDownloader():
"""
self.session = LBRYSession(blob_manager_class=blob_manager_class,
self.session = Session(blob_manager_class=blob_manager_class,
stream_info_manager_class=stream_info_manager_class,
dht_node_class=Node, dht_node_port=dht_node_port, known_dht_nodes=known_dht_nodes,
use_upnp=False)
@@ -55,7 +55,7 @@ class LBRYStdoutDownloader():
d.addCallbacks(self.read_sd_file)
def start_stream(stream_hash):
consumer = LBRYLiveStreamDownloader(stream_hash, self.session.peer_finder,
consumer = LiveStreamDownloader(stream_hash, self.session.peer_finder,
self.session.rate_limiter, self.session.blob_manager,
self.stream_info_manager, self.payment_rate_manager,
self.session.wallet)
@@ -76,9 +76,9 @@ def launch_stdout_downloader():
logging.basicConfig(level=logging.WARNING, filename="dl.log")
if len(sys.argv) == 3:
downloader = LBRYStdoutDownloader(int(sys.argv[2]), [])
downloader = StdoutDownloader(int(sys.argv[2]), [])
elif len(sys.argv) == 5:
downloader = LBRYStdoutDownloader(int(sys.argv[2]), [(sys.argv[3], int(sys.argv[4]))])
downloader = StdoutDownloader(int(sys.argv[2]), [(sys.argv[3], int(sys.argv[4]))])
else:
print "Usage: lbrynet-stdout-downloader <sd_hash> <peer_port> <dht_node_port>" \
" [<dht_bootstrap_host> <dht_bootstrap port>]"
+1 -1
View File
@@ -84,7 +84,7 @@ def get_sd_info(stream_info_manager, stream_hash, include_blobs):
return d
class LBRYLiveStreamDescriptorValidator(object):
class LiveStreamDescriptorValidator(object):
implements(IStreamDescriptorValidator)
def __init__(self, raw_info):
+11 -11
View File
@@ -12,7 +12,7 @@ from lbrynet.interfaces import IStreamDownloaderFactory
from lbrynet.lbrylive.StreamDescriptor import LiveStreamType
class LiveStreamDownloader(CryptStreamDownloader):
class _LiveStreamDownloader(CryptStreamDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, upload_allowed):
@@ -39,10 +39,10 @@ class LiveStreamDownloader(CryptStreamDownloader):
return defer.succeed(True)
class LBRYLiveStreamDownloader(LiveStreamDownloader):
class LiveStreamDownloader(_LiveStreamDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, upload_allowed):
LiveStreamDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
_LiveStreamDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
stream_info_manager, payment_rate_manager, wallet, upload_allowed)
#self.writer = process.ProcessWriter(reactor, self, 'write', 1)
@@ -65,17 +65,17 @@ class LBRYLiveStreamDownloader(LiveStreamDownloader):
return write_func
class FullLiveStreamDownloader(LiveStreamDownloader):
class FullLiveStreamDownloader(_LiveStreamDownloader):
def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
payment_rate_manager, wallet, upload_allowed):
LiveStreamDownloader.__init__(self, stream_hash, peer_finder, rate_limiter,
_LiveStreamDownloader.__init__(self, stream_hash, peer_finder, rate_limiter,
blob_manager, stream_info_manager, payment_rate_manager,
wallet, upload_allowed)
self.file_handle = None
self.file_name = None
def set_stream_info(self):
d = LiveStreamDownloader.set_stream_info(self)
d = _LiveStreamDownloader.set_stream_info(self)
def set_file_name_if_unset():
if not self.file_name:
@@ -88,7 +88,7 @@ class FullLiveStreamDownloader(LiveStreamDownloader):
def stop(self, err=None):
d = self._close_file()
d.addBoth(lambda _: LiveStreamDownloader.stop(self, err))
d.addBoth(lambda _: _LiveStreamDownloader.stop(self, err))
return d
def _start(self):
@@ -96,7 +96,7 @@ class FullLiveStreamDownloader(LiveStreamDownloader):
d = self._open_file()
else:
d = defer.succeed(True)
d.addCallback(lambda _: LiveStreamDownloader._start(self))
d.addCallback(lambda _: _LiveStreamDownloader._start(self))
return d
def _open_file(self):
@@ -144,7 +144,7 @@ class FullLiveStreamDownloaderFactory(object):
def make_downloader(self, metadata, options, payment_rate_manager):
# TODO: check options for payment rate manager parameters
payment_rate_manager = LiveStreamPaymentRateManager(self.default_payment_rate_manager,
prm = LiveStreamPaymentRateManager(self.default_payment_rate_manager,
payment_rate_manager)
def save_source_if_blob(stream_hash):
@@ -161,7 +161,7 @@ class FullLiveStreamDownloaderFactory(object):
def create_downloader(stream_hash):
stream_downloader = FullLiveStreamDownloader(stream_hash, self.peer_finder, self.rate_limiter,
self.blob_manager, self.stream_info_manager,
payment_rate_manager, self.wallet, True)
prm, self.wallet, True)
# TODO: change upload_allowed=True above to something better
d = stream_downloader.set_stream_info()
d.addCallback(lambda _: stream_downloader)
@@ -179,4 +179,4 @@ def add_full_live_stream_downloader_to_sd_identifier(session, stream_info_manage
stream_info_manager,
session.wallet,
base_live_stream_payment_rate_manager)
sd_identifier.add_stream_downloader_factory(LiveStreamType, downloader_factory)
sd_identifier.add_stream_downloader_factory(LiveStreamType, downloader_factory)
@@ -3,7 +3,7 @@ import logging
from zope.interface import implements
from twisted.internet import defer
from twisted.python.failure import Failure
from lbrynet.conf import MAX_BLOB_INFOS_TO_REQUEST
from lbrynet.conf import settings
from lbrynet.core.client.ClientRequest import ClientRequest, ClientPaidRequest
from lbrynet.lbrylive.LiveBlob import LiveBlobInfo
from lbrynet.core.cryptoutils import get_lbry_hash_obj, verify_signature
@@ -136,7 +136,7 @@ class LiveStreamMetadataHandler(object):
if count is not None:
further_blobs_request['count'] = count
else:
further_blobs_request['count'] = MAX_BLOB_INFOS_TO_REQUEST
further_blobs_request['count'] = settings.MAX_BLOB_INFOS_TO_REQUEST
log.debug("Requesting %s blob infos from %s", str(further_blobs_request['count']), str(peer))
r_dict = {'further_blobs': further_blobs_request}
response_identifier = 'further_blobs'
+2 -2
View File
@@ -1,9 +1,9 @@
from lbrynet.lbrylive.StreamDescriptor import LiveStreamType, LBRYLiveStreamDescriptorValidator
from lbrynet.lbrylive.StreamDescriptor import LiveStreamType, LiveStreamDescriptorValidator
from lbrynet.core.DownloadOption import DownloadOption, DownloadOptionChoice
def add_live_stream_to_sd_identifier(sd_identifier, base_live_stream_payment_rate_manager):
sd_identifier.add_stream_type(LiveStreamType, LBRYLiveStreamDescriptorValidator,
sd_identifier.add_stream_type(LiveStreamType, LiveStreamDescriptorValidator,
LiveStreamOptions(base_live_stream_payment_rate_manager))
@@ -1,3 +1,5 @@
# TODO: THERE IS A LOT OF CODE IN THIS MODULE THAT SHOULD BE REMOVED
# AS IT IS REPEATED IN THE LBRYDaemon MODULE
import logging
import os.path
import argparse
@@ -10,47 +12,47 @@ if sys.platform == "darwin":
from appdirs import user_data_dir
from yapsy.PluginManager import PluginManager
from twisted.internet import defer, threads, stdio, task, error
from jsonrpc.proxy import JSONRPCProxy
from lbrynet.lbrynet_daemon.auth.client import LBRYAPIClient
from lbrynet.core.Session import LBRYSession
from lbrynet import analytics
from lbrynet.core.Session import Session
from lbrynet.lbrynet_console.ConsoleControl import ConsoleControl
from lbrynet.lbrynet_console.LBRYSettings import LBRYSettings
from lbrynet.lbryfilemanager.LBRYFileManager import LBRYFileManager
from lbrynet.conf import MIN_BLOB_DATA_PAYMENT_RATE, API_CONNECTION_STRING # , MIN_BLOB_INFO_PAYMENT_RATE
from lbrynet.lbrynet_console.Settings import Settings
from lbrynet.lbryfilemanager.EncryptedFileManager import EncryptedFileManager
from lbrynet.conf import settings
from lbrynet.core.utils import generate_id
from lbrynet.core.StreamDescriptor import StreamDescriptorIdentifier
from lbrynet.core.PaymentRateManager import PaymentRateManager
from lbrynet.core.server.BlobAvailabilityHandler import BlobAvailabilityHandlerFactory
from lbrynet.core.server.BlobRequestHandler import BlobRequestHandlerFactory
from lbrynet.core.server.ServerProtocol import ServerProtocolFactory
from lbrynet.core.PTCWallet import PTCWallet
from lbrynet.lbryfile.client.LBRYFileOptions import add_lbry_file_to_sd_identifier
from lbrynet.lbryfile.client.LBRYFileDownloader import LBRYFileOpenerFactory
from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamType
from lbrynet.lbryfile.LBRYFileMetadataManager import DBLBRYFileMetadataManager, TempLBRYFileMetadataManager
from lbrynet.lbryfile.client.EncryptedFileOptions import add_lbry_file_to_sd_identifier
from lbrynet.lbryfile.client.EncryptedFileDownloader import EncryptedFileOpenerFactory
from lbrynet.lbryfile.StreamDescriptor import EncryptedFileStreamType
from lbrynet.lbryfile.EncryptedFileMetadataManager import DBEncryptedFileMetadataManager, TempEncryptedFileMetadataManager
from lbrynet.lbrynet_console.ControlHandlers import ApplicationStatusFactory, GetWalletBalancesFactory, ShutDownFactory
from lbrynet.lbrynet_console.ControlHandlers import ImmediateAnnounceAllBlobsFactory
from lbrynet.lbrynet_console.ControlHandlers import LBRYFileStatusFactory, DeleteLBRYFileChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import ToggleLBRYFileRunningChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import EncryptedFileStatusFactory, DeleteEncryptedFileChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import ToggleEncryptedFileRunningChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import ModifyApplicationDefaultsFactory
from lbrynet.lbrynet_console.ControlHandlers import CreateLBRYFileFactory, PublishStreamDescriptorChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import CreateEncryptedFileFactory, PublishStreamDescriptorChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import ShowPublishedSDHashesChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import CreatePlainStreamDescriptorChooserFactory
from lbrynet.lbrynet_console.ControlHandlers import ShowLBRYFileStreamHashChooserFactory, AddStreamFromHashFactory
from lbrynet.lbrynet_console.ControlHandlers import ShowEncryptedFileStreamHashChooserFactory, AddStreamFromHashFactory
from lbrynet.lbrynet_console.ControlHandlers import AddStreamFromSDFactory, AddStreamFromLBRYcrdNameFactory
from lbrynet.lbrynet_console.ControlHandlers import ClaimNameFactory, GetNewWalletAddressFactory
from lbrynet.lbrynet_console.ControlHandlers import ShowServerStatusFactory, ModifyServerSettingsFactory
from lbrynet.lbrynet_console.ControlHandlers import ModifyLBRYFileOptionsChooserFactory, StatusFactory
from lbrynet.lbrynet_console.ControlHandlers import ModifyEncryptedFileOptionsChooserFactory, StatusFactory
from lbrynet.lbrynet_console.ControlHandlers import PeerStatsAndSettingsChooserFactory, PublishFactory
from lbrynet.lbrynet_console.ControlHandlers import BlockchainStatusFactory
from lbrynet.core.LBRYcrdWallet import LBRYcrdWallet, LBRYumWallet
from lbrynet.core.Wallet import LBRYcrdWallet, LBRYumWallet
log = logging.getLogger(__name__)
alert = logging.getLogger("lbryalert." + __name__)
class LBRYConsole():
class Console():
"""A class which can upload and download file streams to and from the network"""
def __init__(self, peer_port, dht_node_port, known_dht_nodes, fake_wallet,
lbrycrd_conf, lbrycrd_dir, use_upnp, data_dir, created_data_dir,
@@ -93,7 +95,7 @@ class LBRYConsole():
self.command_handlers = []
self.query_handlers = {}
self.settings = LBRYSettings(self.db_dir)
self.settings = Settings(self.db_dir)
self.blob_request_payment_rate_manager = None
self.lbryid = None
self.sd_identifier = StreamDescriptorIdentifier()
@@ -167,7 +169,7 @@ class LBRYConsole():
log.debug("Created the blobfile directory: %s", str(self.blobfile_dir))
def _check_db_migration(self):
old_revision = 0
old_revision = 1
db_revision_file = os.path.join(self.db_dir, "db_revision")
if os.path.exists(db_revision_file):
old_revision = int(open(db_revision_file).read().strip())
@@ -210,7 +212,7 @@ class LBRYConsole():
def _get_session(self):
def get_default_data_rate():
d = self.settings.get_default_data_payment_rate()
d.addCallback(lambda rate: {"default_data_payment_rate": rate if rate is not None else MIN_BLOB_DATA_PAYMENT_RATE})
d.addCallback(lambda rate: {"default_data_payment_rate": rate if rate is not None else settings.data_rate})
return d
def get_wallet():
@@ -239,7 +241,7 @@ class LBRYConsole():
alert.info("Databases loaded.")
self.session = LBRYSession(results['default_data_payment_rate'], db_dir=self.db_dir, lbryid=self.lbryid,
self.session = Session(results['default_data_payment_rate'], db_dir=self.db_dir, lbryid=self.lbryid,
blob_dir=self.blobfile_dir, dht_node_port=self.dht_node_port,
known_dht_nodes=self.known_dht_nodes, peer_port=self.peer_port,
use_upnp=self.use_upnp, wallet=results['wallet'])
@@ -297,11 +299,11 @@ class LBRYConsole():
"catch up with our blockchain.\n", points_string)
def _setup_lbry_file_manager(self):
self.lbry_file_metadata_manager = DBLBRYFileMetadataManager(self.db_dir)
self.lbry_file_metadata_manager = DBEncryptedFileMetadataManager(self.db_dir)
d = self.lbry_file_metadata_manager.setup()
def set_lbry_file_manager():
self.lbry_file_manager = LBRYFileManager(self.session, self.lbry_file_metadata_manager, self.sd_identifier)
self.lbry_file_manager = EncryptedFileManager(self.session, self.lbry_file_metadata_manager, self.sd_identifier)
return self.lbry_file_manager.setup()
d.addCallback(lambda _: set_lbry_file_manager())
@@ -309,11 +311,11 @@ class LBRYConsole():
return d
def _setup_lbry_file_opener(self):
stream_info_manager = TempLBRYFileMetadataManager()
downloader_factory = LBRYFileOpenerFactory(self.session.peer_finder, self.session.rate_limiter,
stream_info_manager = TempEncryptedFileMetadataManager()
downloader_factory = EncryptedFileOpenerFactory(self.session.peer_finder, self.session.rate_limiter,
self.session.blob_manager, stream_info_manager,
self.session.wallet)
self.sd_identifier.add_stream_downloader_factory(LBRYFileStreamType, downloader_factory)
self.sd_identifier.add_stream_downloader_factory(EncryptedFileStreamType, downloader_factory)
return defer.succeed(True)
def _setup_control_handlers(self):
@@ -323,20 +325,20 @@ class LBRYConsole():
ModifyApplicationDefaultsFactory(self),
ShutDownFactory(self),
PeerStatsAndSettingsChooserFactory(self.session.peer_manager),
LBRYFileStatusFactory(self.lbry_file_manager),
EncryptedFileStatusFactory(self.lbry_file_manager),
AddStreamFromSDFactory(self.sd_identifier, self.session.base_payment_rate_manager,
self.session.wallet),
DeleteLBRYFileChooserFactory(self.lbry_file_metadata_manager, self.session.blob_manager,
DeleteEncryptedFileChooserFactory(self.lbry_file_metadata_manager, self.session.blob_manager,
self.lbry_file_manager),
ToggleLBRYFileRunningChooserFactory(self.lbry_file_manager),
CreateLBRYFileFactory(self.session, self.lbry_file_manager),
ToggleEncryptedFileRunningChooserFactory(self.lbry_file_manager),
CreateEncryptedFileFactory(self.session, self.lbry_file_manager),
PublishStreamDescriptorChooserFactory(self.lbry_file_metadata_manager,
self.session.blob_manager),
ShowPublishedSDHashesChooserFactory(self.lbry_file_metadata_manager,
self.lbry_file_manager),
CreatePlainStreamDescriptorChooserFactory(self.lbry_file_manager),
ShowLBRYFileStreamHashChooserFactory(self.lbry_file_manager),
ModifyLBRYFileOptionsChooserFactory(self.lbry_file_manager),
ShowEncryptedFileStreamHashChooserFactory(self.lbry_file_manager),
ModifyEncryptedFileOptionsChooserFactory(self.lbry_file_manager),
AddStreamFromHashFactory(self.sd_identifier, self.session, self.session.wallet),
StatusFactory(self, self.session.rate_limiter, self.lbry_file_manager,
self.session.blob_manager, self.session.wallet if not self.fake_wallet else None),
@@ -363,11 +365,6 @@ class LBRYConsole():
def _setup_query_handlers(self):
handlers = [
#CryptBlobInfoQueryHandlerFactory(self.lbry_file_metadata_manager, self.session.wallet,
# self._server_payment_rate_manager),
BlobAvailabilityHandlerFactory(self.session.blob_manager),
#BlobRequestHandlerFactory(self.session.blob_manager, self.session.wallet,
# self._server_payment_rate_manager),
self.session.wallet.get_wallet_info_query_handler_factory(),
]
@@ -375,8 +372,10 @@ class LBRYConsole():
self.blob_request_payment_rate_manager = PaymentRateManager(
self.session.base_payment_rate_manager, rate
)
handlers.append(BlobRequestHandlerFactory(self.session.blob_manager, self.session.wallet,
self.blob_request_payment_rate_manager))
handlers.append(BlobRequestHandlerFactory(
self.session.blob_manager, self.session.wallet,
self.blob_request_payment_rate_manager, analytics.Track()
))
d1 = self.settings.get_server_data_payment_rate()
d1.addCallback(get_blob_request_handler_factory)
@@ -543,7 +542,7 @@ def launch_lbry_console():
os.mkdir(data_dir)
created_data_dir = True
daemon = JSONRPCProxy.from_url(API_CONNECTION_STRING)
daemon = LBRYAPIClient.config()
try:
daemon.is_running()
log.info("Attempt to start lbrynet-console while lbrynet-daemon is running")
@@ -565,7 +564,7 @@ def launch_lbry_console():
logger.addHandler(file_handler)
console = LBRYConsole(peer_port, dht_node_port, bootstrap_nodes, fake_wallet=args.fake_wallet,
console = Console(peer_port, dht_node_port, bootstrap_nodes, fake_wallet=args.fake_wallet,
lbrycrd_conf=args.lbrycrd_wallet_conf, lbrycrd_dir=args.lbrycrd_wallet_dir,
use_upnp=not args.disable_upnp, data_dir=data_dir,
created_data_dir=created_data_dir, lbrycrdd_path=args.lbrycrdd_path)
+59 -65
View File
@@ -1,18 +1,12 @@
import json
import logging
from time import sleep
from bitcoinrpc.authproxy import AuthServiceProxy
from twisted.internet.task import LoopingCall
from zope.interface import implements
#from lbrynet.core.StreamDescriptor import PlainStreamDescriptorWriter, BlobStreamDescriptorWriter
from lbrynet.core.PaymentRateManager import PaymentRateManager
from lbrynet.lbryfilemanager.LBRYFileCreator import create_lbry_file
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
# from lbrynet.lbryfile.StreamDescriptor import get_sd_info
from lbrynet.lbryfilemanager.EncryptedFileCreator import create_lbry_file
from lbrynet.lbryfilemanager.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet.lbryfile.StreamDescriptor import publish_sd_blob, create_plain_sd
from lbrynet.lbrynet_console.interfaces import ICommandHandler, ICommandHandlerFactory
from lbrynet.core.StreamDescriptor import download_sd_blob#, BlobStreamDescriptorReader
from lbrynet.core.StreamDescriptor import download_sd_blob
from lbrynet.core.Error import UnknownNameError, InvalidBlobHashError, InsufficientFundsError
from lbrynet.core.Error import InvalidStreamInfoError
from lbrynet.core.utils import is_valid_blobhash
@@ -424,7 +418,7 @@ class ShutDownFactory(CommandHandlerFactory):
full_help = "Shut down"
class LBRYFileStatus(CommandHandler):
class EncryptedFileStatus(CommandHandler):
#prompt_description = "Print status information for all LBRY Files"
def __init__(self, console, lbry_file_manager):
@@ -452,8 +446,8 @@ class LBRYFileStatus(CommandHandler):
self.console.sendLine(''.join(status_strings))
class LBRYFileStatusFactory(CommandHandlerFactory):
control_handler_class = LBRYFileStatus
class EncryptedFileStatusFactory(CommandHandlerFactory):
control_handler_class = EncryptedFileStatus
command = "lbryfile-status"
short_help = "Print status information for LBRY files"
full_help = "Print the status information for all streams that are being saved to disk." \
@@ -650,7 +644,7 @@ class AddStream(CommandHandler):
for option, option_value in zip(self.download_options, self.options_chosen):
if option.short_description == "data payment rate":
if option_value == None:
rate = self.payment_rate_manager.get_effective_min_blob_data_payment_rate()
rate = 0.0
else:
rate = option_value
stream_size = None
@@ -994,7 +988,7 @@ class AddStreamFromLBRYcrdNameFactory(CommandHandlerFactory):
"Usage: get <name>"
class LBRYFileChooser(RecursiveCommandHandler):
class EncryptedFileChooser(RecursiveCommandHandler):
def __init__(self, console, lbry_file_manager, factory_class, *args, **kwargs):
"""
@@ -1020,23 +1014,23 @@ class LBRYFileChooser(RecursiveCommandHandler):
return control_handler_factories
class LBRYFileChooserFactory(CommandHandlerFactory):
class EncryptedFileChooserFactory(CommandHandlerFactory):
def get_prompt_description(self):
lbry_file = self.args[0]
return lbry_file.file_name
class DeleteLBRYFileChooser(LBRYFileChooser):
class DeleteEncryptedFileChooser(EncryptedFileChooser):
#prompt_description = "Delete LBRY File"
def __init__(self, console, stream_info_manager, blob_manager, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager, DeleteLBRYFileFactory,
EncryptedFileChooser.__init__(self, console, lbry_file_manager, DeleteEncryptedFileFactory,
stream_info_manager, blob_manager, lbry_file_manager,
exit_after_one_done=True)
class DeleteLBRYFileChooserFactory(CommandHandlerFactory):
control_handler_class = DeleteLBRYFileChooser
class DeleteEncryptedFileChooserFactory(CommandHandlerFactory):
control_handler_class = DeleteEncryptedFileChooser
command = "delete-lbryfile"
short_help = "Delete an LBRY file"
full_help = "Delete an LBRY file which has been downloaded or created by this application.\n" \
@@ -1046,7 +1040,7 @@ class DeleteLBRYFileChooserFactory(CommandHandlerFactory):
"not be able to upload those chunks of data to other peers on LBRYnet."
class DeleteLBRYFile(CommandHandler):
class DeleteEncryptedFile(CommandHandler):
#prompt_description = "Delete LBRY File"
delete_data_prompt = "Also delete data? (y/n): "
confirm_prompt = "Are you sure? (y/n): "
@@ -1112,27 +1106,27 @@ class DeleteLBRYFile(CommandHandler):
return d
class DeleteLBRYFileFactory(LBRYFileChooserFactory):
control_handler_class = DeleteLBRYFile
class DeleteEncryptedFileFactory(EncryptedFileChooserFactory):
control_handler_class = DeleteEncryptedFile
class ToggleLBRYFileRunningChooser(LBRYFileChooser):
class ToggleEncryptedFileRunningChooser(EncryptedFileChooser):
#prompt_description = "Toggle whether an LBRY File is running"
def __init__(self, console, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager, ToggleLBRYFileRunningFactory,
EncryptedFileChooser.__init__(self, console, lbry_file_manager, ToggleEncryptedFileRunningFactory,
lbry_file_manager, exit_after_one_done=True)
class ToggleLBRYFileRunningChooserFactory(CommandHandlerFactory):
control_handler_class = ToggleLBRYFileRunningChooser
class ToggleEncryptedFileRunningChooserFactory(CommandHandlerFactory):
control_handler_class = ToggleEncryptedFileRunningChooser
command = "toggle-running"
short_help = "Toggle whether an LBRY file is running"
full_help = "Toggle whether an LBRY file, which is being saved by this application," \
"is currently being downloaded."
class ToggleLBRYFileRunning(CommandHandler):
class ToggleEncryptedFileRunning(CommandHandler):
#prompt_description = "Toggle whether an LBRY File is running"
def __init__(self, console, lbry_file, lbry_file_manager):
@@ -1157,11 +1151,11 @@ class ToggleLBRYFileRunning(CommandHandler):
return "An unexpected error occurred. See %s for details." % log_file
class ToggleLBRYFileRunningFactory(LBRYFileChooserFactory):
control_handler_class = ToggleLBRYFileRunning
class ToggleEncryptedFileRunningFactory(EncryptedFileChooserFactory):
control_handler_class = ToggleEncryptedFileRunning
class CreateLBRYFile(CommandHandler):
class CreateEncryptedFile(CommandHandler):
#prompt_description = "Create an LBRY File from file"
line_prompt = "File name: "
@@ -1185,13 +1179,13 @@ class CreateLBRYFile(CommandHandler):
def set_status(self, lbry_file_downloader):
d = self.lbry_file_manager.change_lbry_file_status(lbry_file_downloader,
ManagedLBRYFileDownloader.STATUS_FINISHED)
ManagedEncryptedFileDownloader.STATUS_FINISHED)
d.addCallback(lambda _: lbry_file_downloader.restore())
return d
class CreateLBRYFileFactory(CommandHandlerFactory):
control_handler_class = CreateLBRYFile
class CreateEncryptedFileFactory(CommandHandlerFactory):
control_handler_class = CreateEncryptedFile
command = "create-lbryfile"
short_help = "LBRYize a file"
full_help = "Encrypt a file, split it into chunks, and make those chunks available on LBRYnet. Also " \
@@ -1202,11 +1196,11 @@ class CreateLBRYFileFactory(CommandHandlerFactory):
"downloaded via the hash of the stream descriptor."
class PublishStreamDescriptorChooser(LBRYFileChooser):
class PublishStreamDescriptorChooser(EncryptedFileChooser):
#prompt_description = "Publish a stream descriptor file to the DHT for an LBRY File"
def __init__(self, console, stream_info_manager, blob_manager, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager, PublishStreamDescriptorFactory,
EncryptedFileChooser.__init__(self, console, lbry_file_manager, PublishStreamDescriptorFactory,
stream_info_manager, blob_manager, lbry_file_manager,
exit_after_one_done=True)
@@ -1249,15 +1243,15 @@ class PublishStreamDescriptor(CommandHandler):
# return d
class PublishStreamDescriptorFactory(LBRYFileChooserFactory):
class PublishStreamDescriptorFactory(EncryptedFileChooserFactory):
control_handler_class = PublishStreamDescriptor
class ShowPublishedSDHashesChooser(LBRYFileChooser):
class ShowPublishedSDHashesChooser(EncryptedFileChooser):
#prompt_description = "Show published stream descriptors for an LBRY File"
def __init__(self, console, stream_info_manager, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager, ShowPublishedSDHashesFactory,
EncryptedFileChooser.__init__(self, console, lbry_file_manager, ShowPublishedSDHashesFactory,
stream_info_manager, lbry_file_manager)
@@ -1294,15 +1288,15 @@ class ShowPublishedSDHashes(CommandHandler):
return d
class ShowPublishedSDHashesFactory(LBRYFileChooserFactory):
class ShowPublishedSDHashesFactory(EncryptedFileChooserFactory):
control_handler_class = ShowPublishedSDHashes
class CreatePlainStreamDescriptorChooser(LBRYFileChooser):
class CreatePlainStreamDescriptorChooser(EncryptedFileChooser):
#prompt_description = "Create a plain stream descriptor file for an LBRY File"
def __init__(self, console, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager,
EncryptedFileChooser.__init__(self, console, lbry_file_manager,
CreatePlainStreamDescriptorFactory, lbry_file_manager,
exit_after_one_done=True)
@@ -1358,26 +1352,26 @@ class CreatePlainStreamDescriptor(CommandHandler):
return defer.succeed(file_name)
class CreatePlainStreamDescriptorFactory(LBRYFileChooserFactory):
class CreatePlainStreamDescriptorFactory(EncryptedFileChooserFactory):
control_handler_class = CreatePlainStreamDescriptor
class ShowLBRYFileStreamHashChooser(LBRYFileChooser):
class ShowEncryptedFileStreamHashChooser(EncryptedFileChooser):
#prompt_description = "Show an LBRY File's stream hash (not usually what you want)"
def __init__(self, console, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager, ShowLBRYFileStreamHashFactory)
EncryptedFileChooser.__init__(self, console, lbry_file_manager, ShowEncryptedFileStreamHashFactory)
class ShowLBRYFileStreamHashChooserFactory(CommandHandlerFactory):
control_handler_class = ShowLBRYFileStreamHashChooser
class ShowEncryptedFileStreamHashChooserFactory(CommandHandlerFactory):
control_handler_class = ShowEncryptedFileStreamHashChooser
command = "lbryfile-streamhash"
short_help = "Show an LBRY file's stream hash"
full_help = "Show the stream hash of an LBRY file, which is how the LBRY file is referenced internally" \
" by this application and therefore not usually what you want to see."
class ShowLBRYFileStreamHash(CommandHandler):
class ShowEncryptedFileStreamHash(CommandHandler):
#prompt_description = "Show an LBRY File's stream hash (not usually what you want)"
def __init__(self, console, lbry_file):
@@ -1389,11 +1383,11 @@ class ShowLBRYFileStreamHash(CommandHandler):
self.finished_deferred.callback(None)
class ShowLBRYFileStreamHashFactory(LBRYFileChooserFactory):
control_handler_class = ShowLBRYFileStreamHash
class ShowEncryptedFileStreamHashFactory(EncryptedFileChooserFactory):
control_handler_class = ShowEncryptedFileStreamHash
class ModifyLBRYFileDataPaymentRate(ModifyPaymentRate):
class ModifyEncryptedFileDataPaymentRate(ModifyPaymentRate):
prompt_description = "Modify LBRY File data payment rate"
def __init__(self, console, lbry_file, lbry_file_manager):
@@ -1421,19 +1415,19 @@ class ModifyLBRYFileDataPaymentRate(ModifyPaymentRate):
return status
class ModifyLBRYFileDataPaymentRateFactory(CommandHandlerFactory):
control_handler_class = ModifyLBRYFileDataPaymentRate
class ModifyEncryptedFileDataPaymentRateFactory(CommandHandlerFactory):
control_handler_class = ModifyEncryptedFileDataPaymentRate
class ModifyLBRYFileOptionsChooser(LBRYFileChooser):
class ModifyEncryptedFileOptionsChooser(EncryptedFileChooser):
#prompt_description = "Modify an LBRY File's options"
def __init__(self, console, lbry_file_manager):
LBRYFileChooser.__init__(self, console, lbry_file_manager, ModifyLBRYFileOptionsFactory, lbry_file_manager)
EncryptedFileChooser.__init__(self, console, lbry_file_manager, ModifyEncryptedFileOptionsFactory, lbry_file_manager)
class ModifyLBRYFileOptionsChooserFactory(CommandHandlerFactory):
control_handler_class = ModifyLBRYFileOptionsChooser
class ModifyEncryptedFileOptionsChooserFactory(CommandHandlerFactory):
control_handler_class = ModifyEncryptedFileOptionsChooser
command = "modify-lbryfile-options"
short_help = "Modify an LBRY file's options"
full_help = "Modify an LBRY file's options. Options include, and are limited to, " \
@@ -1441,7 +1435,7 @@ class ModifyLBRYFileOptionsChooserFactory(CommandHandlerFactory):
"this LBRY file."
class ModifyLBRYFileOptions(RecursiveCommandHandler):
class ModifyEncryptedFileOptions(RecursiveCommandHandler):
#prompt_description = "Modify an LBRY File's options"
def __init__(self, console, lbry_file, lbry_file_manager):
@@ -1451,12 +1445,12 @@ class ModifyLBRYFileOptions(RecursiveCommandHandler):
def _get_control_handler_factories(self):
factories = []
factories.append(ModifyLBRYFileDataPaymentRateFactory(self.lbry_file, self.lbry_file_manager))
factories.append(ModifyEncryptedFileDataPaymentRateFactory(self.lbry_file, self.lbry_file_manager))
return factories
class ModifyLBRYFileOptionsFactory(LBRYFileChooserFactory):
control_handler_class = ModifyLBRYFileOptions
class ModifyEncryptedFileOptionsFactory(EncryptedFileChooserFactory):
control_handler_class = ModifyEncryptedFileOptions
class ClaimName(CommandHandler):
@@ -1827,7 +1821,7 @@ class Publish(CommandHandler):
def set_status(self, lbry_file_downloader):
self.lbry_file = lbry_file_downloader
d = self.lbry_file_manager.change_lbry_file_status(self.lbry_file,
ManagedLBRYFileDownloader.STATUS_FINISHED)
ManagedEncryptedFileDownloader.STATUS_FINISHED)
d.addCallback(lambda _: lbry_file_downloader.restore())
return d
@@ -2360,7 +2354,7 @@ class PeerStatsAndSettingsChooserFactory(CommandHandlerFactory):
"which peers to connect to."
class LBRYFileStatusModifier(CommandHandler):
class EncryptedFileStatusModifier(CommandHandler):
def __init__(self, console, lbry_file, stream_info_manager, blob_manager, lbry_file_manager):
CommandHandler.__init__(self, console)
self.lbry_file = lbry_file
@@ -2377,11 +2371,11 @@ class LBRYFileStatusModifier(CommandHandler):
if self.current_handler is None:
if line:
if line.lower() == 'd':
self.current_handler = DeleteLBRYFile(self.console, self.lbry_file,
self.current_handler = DeleteEncryptedFile(self.console, self.lbry_file,
self.stream_info_manager, self.blob_manager,
self.lbry_file_manager)
elif line.lower() == 't':
self.current_handler = ToggleLBRYFileRunning(self.console, self.lbry_file,
self.current_handler = ToggleEncryptedFileRunning(self.console, self.lbry_file,
self.lbry_file_manager)
else:
self.console.sendLine("Invalid selection\n")
@@ -2450,7 +2444,7 @@ class Status(CommandHandler):
self.console.sendLine("Invalid choice.\n")
self.finished_deferred.callback(None)
return
self.current_handler = LBRYFileStatusModifier(self.console, self.lbry_files[index],
self.current_handler = EncryptedFileStatusModifier(self.console, self.lbry_files[index],
self.lbry_file_manager.stream_info_manager,
self.blob_manager, self.lbry_file_manager)
try:
-120
View File
@@ -1,120 +0,0 @@
import binascii
import json
import unqlite
import logging
import os
from twisted.internet import threads, defer
log = logging.getLogger(__name__)
class LBRYSettings(object):
def __init__(self, db_dir):
self.db_dir = db_dir
self.db = None
def start(self):
return self._open_db()
def stop(self):
self.db = None
return defer.succeed(True)
def _open_db(self):
log.debug("Opening %s as the settings database", str(os.path.join(self.db_dir, "settings.db")))
self.db = unqlite.UnQLite(os.path.join(self.db_dir, "settings.db"))
return defer.succeed(True)
def save_lbryid(self, lbryid):
def save_lbryid():
self.db['lbryid'] = binascii.hexlify(lbryid)
return threads.deferToThread(save_lbryid)
def get_lbryid(self):
def get_lbryid():
if 'lbryid' in self.db:
return binascii.unhexlify(self.db['lbryid'])
else:
return None
return threads.deferToThread(get_lbryid)
def get_server_running_status(self):
def get_status():
if 'server_running' in self.db:
return json.loads(self.db['server_running'])
else:
return True
return threads.deferToThread(get_status)
def save_server_running_status(self, running):
def save_status():
self.db['server_running'] = json.dumps(running)
return threads.deferToThread(save_status)
def get_default_data_payment_rate(self):
return self._get_payment_rate("default_data_payment_rate")
def save_default_data_payment_rate(self, rate):
return self._save_payment_rate("default_data_payment_rate", rate)
def get_server_data_payment_rate(self):
return self._get_payment_rate("server_data_payment_rate")
def save_server_data_payment_rate(self, rate):
return self._save_payment_rate("server_data_payment_rate", rate)
def get_server_crypt_info_payment_rate(self):
return self._get_payment_rate("server_crypt_info_payment_rate")
def save_server_crypt_info_payment_rate(self, rate):
return self._save_payment_rate("server_crypt_info_payment_rate", rate)
def _get_payment_rate(self, rate_type):
def get_rate():
if rate_type in self.db:
return json.loads(self.db[rate_type])
else:
return None
return threads.deferToThread(get_rate)
def _save_payment_rate(self, rate_type, rate):
def save_rate():
if rate is not None:
self.db[rate_type] = json.dumps(rate)
elif rate_type in self.db:
del self.db[rate_type]
return threads.deferToThread(save_rate)
def get_query_handler_status(self, query_identifier):
def get_status():
if json.dumps(('q_h', query_identifier)) in self.db:
return json.loads(self.db[(json.dumps(('q_h', query_identifier)))])
else:
return True
return threads.deferToThread(get_status)
def enable_query_handler(self, query_identifier):
return self._set_query_handler_status(query_identifier, True)
def disable_query_handler(self, query_identifier):
return self._set_query_handler_status(query_identifier, False)
def _set_query_handler_status(self, query_identifier, status):
def set_status():
self.db[json.dumps(('q_h', query_identifier))] = json.dumps(status)
return threads.deferToThread(set_status)
@@ -1,7 +1,7 @@
from yapsy.IPlugin import IPlugin
class LBRYPlugin(IPlugin):
class Plugin(IPlugin):
def __init__(self):
IPlugin.__init__(self)
+114
View File
@@ -0,0 +1,114 @@
import binascii
import functools
import json
import logging
import os
from twisted.internet import threads, defer
import unqlite
log = logging.getLogger(__name__)
def run_in_thread(fn):
@functools.wraps(fn)
def wrapped(*args, **kwargs):
return threads.deferToThread(fn, *args, **kwargs)
return wrapped
class Settings(object):
NAME = "settings.db"
def __init__(self, db_dir):
self.db_dir = db_dir
self.db = None
def start(self):
return self._open_db()
def stop(self):
self.db.close()
self.db = None
return defer.succeed(True)
def _open_db(self):
filename = os.path.join(self.db_dir, self.NAME)
log.debug("Opening %s as the settings database", filename)
self.db = unqlite.UnQLite(filename)
return defer.succeed(True)
@run_in_thread
def save_lbryid(self, lbryid):
self.db['lbryid'] = binascii.hexlify(lbryid)
self.db.commit()
@run_in_thread
def get_lbryid(self):
if 'lbryid' in self.db:
return binascii.unhexlify(self.db['lbryid'])
else:
return None
@run_in_thread
def get_server_running_status(self):
if 'server_running' in self.db:
return json.loads(self.db['server_running'])
else:
return True
@run_in_thread
def save_server_running_status(self, running):
self.db['server_running'] = json.dumps(running)
self.db.commit()
def get_default_data_payment_rate(self):
return self._get_payment_rate("default_data_payment_rate")
def save_default_data_payment_rate(self, rate):
return self._save_payment_rate("default_data_payment_rate", rate)
def get_server_data_payment_rate(self):
return self._get_payment_rate("server_data_payment_rate")
def save_server_data_payment_rate(self, rate):
return self._save_payment_rate("server_data_payment_rate", rate)
def get_server_crypt_info_payment_rate(self):
return self._get_payment_rate("server_crypt_info_payment_rate")
def save_server_crypt_info_payment_rate(self, rate):
return self._save_payment_rate("server_crypt_info_payment_rate", rate)
@run_in_thread
def _get_payment_rate(self, rate_type):
if rate_type in self.db:
return json.loads(self.db[rate_type])
else:
return None
@run_in_thread
def _save_payment_rate(self, rate_type, rate):
if rate is not None:
self.db[rate_type] = json.dumps(rate)
elif rate_type in self.db:
del self.db[rate_type]
self.db.commit()
@run_in_thread
def get_query_handler_status(self, query_identifier):
if json.dumps(('q_h', query_identifier)) in self.db:
return json.loads(self.db[(json.dumps(('q_h', query_identifier)))])
else:
return True
def enable_query_handler(self, query_identifier):
return self._set_query_handler_status(query_identifier, True)
def disable_query_handler(self, query_identifier):
return self._set_query_handler_status(query_identifier, False)
@run_in_thread
def _set_query_handler_status(self, query_identifier, status):
self.db[json.dumps(('q_h', query_identifier))] = json.dumps(status)
self.db.commit()
@@ -1,6 +1,6 @@
from lbrynet.lbrynet_console import LBRYPlugin
from lbrynet.lbrynet_console import Plugin
from twisted.internet import defer
from lbrynet.conf import MIN_VALUABLE_BLOB_HASH_PAYMENT_RATE, MIN_VALUABLE_BLOB_INFO_PAYMENT_RATE
from lbrynet.conf import settings
from BlindRepeater import BlindRepeater
from BlindInfoManager import BlindInfoManager
from BlindRepeaterSettings import BlindRepeaterSettings
@@ -13,10 +13,10 @@ from ValuableBlobQueryHandler import ValuableBlobHashQueryHandlerFactory
from PaymentRateManager import BlindRepeaterPaymentRateManager
class BlindRepeaterPlugin(LBRYPlugin.LBRYPlugin):
class BlindRepeaterPlugin(Plugin.Plugin):
def __init__(self):
LBRYPlugin.LBRYPlugin.__init__(self)
Plugin.Plugin.__init__(self)
self.enabled = False
self.blind_info_manager = None
self.valuable_blob_length_query_handler = None
@@ -59,9 +59,9 @@ class BlindRepeaterPlugin(LBRYPlugin.LBRYPlugin):
def get_payment_rate_manager(rates):
data_rate = rates[0][1] if rates[0][0] is True else None
info_rate = rates[1][1] if rates[1][0] is True else None
info_rate = info_rate if info_rate is not None else MIN_VALUABLE_BLOB_INFO_PAYMENT_RATE
info_rate = info_rate if info_rate is not None else settings.min_valuable_info_rate
hash_rate = rates[2][1] if rates[2][0] is True else None
hash_rate = hash_rate if hash_rate is not None else MIN_VALUABLE_BLOB_HASH_PAYMENT_RATE
hash_rate = hash_rate if hash_rate is not None else settings.min_valuable_hash_rate
self.payment_rate_manager = BlindRepeaterPaymentRateManager(default_payment_rate_manager,
info_rate, hash_rate,
blob_data_rate=data_rate)
File diff suppressed because it is too large. Load diff
+100
View File
@@ -0,0 +1,100 @@
import sys
import json
import argparse
from lbrynet.conf import settings
from lbrynet.lbrynet_daemon.auth.client import LBRYAPIClient
help_msg = "Usage: lbrynet-cli method json-args\n" \
+ "Examples: " \
+ "lbrynet-cli resolve_name '{\"name\": \"what\"}'\n" \
+ "lbrynet-cli get_balance\n" \
+ "lbrynet-cli help '{\"function\": \"resolve_name\"}'\n" \
+ "\n******lbrynet-cli functions******\n"
def guess_type(x):
if '.' in x:
try:
return float(x)
except ValueError:
# not a float
pass
try:
return int(x)
except ValueError:
return x
def get_params_from_kwargs(params):
params_for_return = {}
for i in params:
eq_pos = i.index('=')
k, v = i[:eq_pos], i[eq_pos+1:]
params_for_return[k] = guess_type(v)
return params_for_return
def main():
api = LBRYAPIClient.config()
try:
status = api.daemon_status()
assert status.get('code', False) == "started"
except Exception:
try:
settings.update({'use_auth_http': not settings.use_auth_http})
api = LBRYAPIClient.config()
status = api.daemon_status()
assert status.get('code', False) == "started"
except Exception:
print "lbrynet-daemon isn't running"
sys.exit(1)
parser = argparse.ArgumentParser()
parser.add_argument('method', nargs=1)
parser.add_argument('params', nargs=argparse.REMAINDER, default=None)
args = parser.parse_args()
meth = args.method[0]
params = {}
if args.params:
if len(args.params) > 1:
params = get_params_from_kwargs(args.params)
elif len(args.params) == 1:
try:
params = json.loads(args.params[0])
except ValueError:
params = get_params_from_kwargs(args.params)
msg = help_msg
for f in api.help():
msg += f + "\n"
if meth in ['--help', '-h', 'help']:
print msg
sys.exit(1)
if meth in api.help():
try:
if params:
result = LBRYAPIClient.config(service=meth, params=params)
else:
result = LBRYAPIClient.config(service=meth, params=params)
print json.dumps(result, sort_keys=True)
except:
# TODO: The api should return proper error codes
# and messages so that they can be passed along to the user
# instead of this generic message.
# https://app.asana.com/0/158602294500137/200173944358192
print "Something went wrong, here's the usage for %s:" % meth
print api.help({'function': meth})
else:
print "Unknown function"
print msg
if __name__ == '__main__':
main()
+174
View File
@@ -0,0 +1,174 @@
import argparse
import logging.handlers
import os
import webbrowser
import sys
from twisted.web import server, guard
from twisted.internet import defer, reactor, error
from twisted.cred import portal
from jsonrpc.proxy import JSONRPCProxy
from lbrynet.lbrynet_daemon.auth.auth import PasswordChecker, HttpPasswordRealm
from lbrynet.lbrynet_daemon.auth.util import initialize_api_key_file
from lbrynet import conf
from lbrynet.core import log_support
from lbrynet.core import utils
from lbrynet.lbrynet_daemon.DaemonServer import DaemonServer
from lbrynet.lbrynet_daemon.DaemonRequest import DaemonRequest
from lbrynet.conf import settings
log = logging.getLogger(__name__)
def test_internet_connection():
return utils.check_connection()
def stop():
def _disp_shutdown():
print "Shutting down lbrynet-daemon from command line"
log.info("Shutting down lbrynet-daemon from command line")
def _disp_not_running():
print "Attempt to shut down lbrynet-daemon from command line when daemon isn't running"
log.info("Attempt to shut down lbrynet-daemon from command line when daemon isn't running")
d = defer.Deferred(None)
d.addCallback(lambda _: JSONRPCProxy.from_url(settings.API_CONNECTION_STRING).stop())
d.addCallbacks(lambda _: _disp_shutdown(), lambda _: _disp_not_running())
d.callback(None)
def start():
parser = argparse.ArgumentParser(description="Launch lbrynet-daemon")
parser.add_argument("--wallet",
help="lbrycrd or lbryum, default lbryum",
type=str,
default='lbryum')
parser.add_argument("--ui", help="path to custom UI folder", default=None)
parser.add_argument(
"--branch",
help='Branch of lbry-web-ui repo to use, defaults to {}'.format(settings.ui_branch),
default=settings.ui_branch)
parser.add_argument('--no-launch', dest='launchui', action="store_false")
parser.add_argument("--http-auth", dest="useauth", action="store_true")
parser.add_argument(
'--log-to-console', dest='logtoconsole', action='store_true',
help=('Set to enable console logging. Set the --verbose flag '
' to enable more detailed console logging'))
parser.add_argument(
'--quiet', dest='quiet', action="store_true",
help=('If log-to-console is not set, setting this disables all console output. '
'If log-to-console is set, this argument is ignored'))
parser.add_argument(
'--verbose', nargs="*",
help=('Enable debug output. Optionally specify loggers for which debug output '
'should selectively be applied.'))
args = parser.parse_args()
utils.setup_certs_for_windows()
lbrynet_log = conf.get_log_filename()
log_support.configure_logging(lbrynet_log, args.logtoconsole, args.verbose)
to_pass = {}
settings_path = os.path.join(settings.data_dir, "daemon_settings.yml")
if os.path.isfile(settings_path):
to_pass.update(utils.load_settings(settings_path))
log.info("Loaded settings file")
if args.ui:
to_pass.update({'local_ui_path': args.ui})
if args.branch:
to_pass.update({'ui_branch': args.branch})
to_pass.update({'use_auth_http': args.useauth})
to_pass.update({'wallet_type': args.wallet})
log.debug('Settings overrides: %s', to_pass)
settings.update(to_pass)
log.debug('Final Settings: %s', settings.__dict__)
try:
JSONRPCProxy.from_url(settings.API_CONNECTION_STRING).is_running()
log.info("lbrynet-daemon is already running")
if not args.logtoconsole:
print "lbrynet-daemon is already running"
if args.launchui:
webbrowser.open(settings.UI_ADDRESS)
return
except:
pass
log.info("Starting lbrynet-daemon from command line")
if not args.logtoconsole and not args.quiet:
print "Starting lbrynet-daemon from command line"
print "To view activity, view the log file here: " + lbrynet_log
print "Web UI is available at http://%s:%i" % (settings.API_INTERFACE, settings.api_port)
print "JSONRPC API is available at " + settings.API_CONNECTION_STRING
print "To quit press ctrl-c or call 'stop' via the API"
if test_internet_connection():
start_server_and_listen(args.launchui, args.useauth)
reactor.run()
if not args.logtoconsole and not args.quiet:
print "\nClosing lbrynet-daemon"
else:
log.info("Not connected to internet, unable to start")
if not args.logtoconsole:
print "Not connected to internet, unable to start"
return
def log_and_kill(failure):
log_support.failure(failure, log, 'Failed to startup: %s')
reactor.stop()
def start_server_and_listen(launchui, use_auth, **kwargs):
"""The primary entry point for launching the daemon.
Args:
launchui: set to true to open a browser window
use_auth: set to true to enable http authentication
kwargs: passed along to `DaemonServer().start()`
"""
lbry = DaemonServer()
d = lbry.start(**kwargs)
if launchui:
d.addCallback(lambda _: webbrowser.open(settings.UI_ADDRESS))
d.addErrback(log_and_kill)
site_base = get_site_base(use_auth, lbry.root)
lbrynet_server = server.Site(site_base)
lbrynet_server.requestFactory = DaemonRequest
try:
reactor.listenTCP(settings.api_port, lbrynet_server, interface=settings.API_INTERFACE)
except error.CannotListenError:
log.info('Daemon already running, exiting app')
sys.exit(1)
def get_site_base(use_auth, root):
if use_auth:
log.info("Using authenticated API")
return create_auth_session(root)
else:
log.info("Using non-authenticated API")
return server.Site(root)
def create_auth_session(root):
pw_path = os.path.join(settings.data_dir, ".api_keys")
initialize_api_key_file(pw_path)
checker = PasswordChecker.load_file(pw_path)
realm = HttpPasswordRealm(root)
portal_to_realm = portal.Portal(realm, [checker, ])
factory = guard.BasicCredentialFactory('Login to lbrynet api')
_lbrynet_server = guard.HTTPAuthSessionWrapper(portal_to_realm, [factory, ])
return _lbrynet_server
if __name__ == "__main__":
start()
+187
View File
@@ -0,0 +1,187 @@
import time
import cgi
import mimetools
import os
import tempfile
from twisted.web import server
class DaemonRequest(server.Request):
"""
For LBRY specific request functionality. Currently just provides
handling for large multipart POST requests, taken from here:
http://sammitch.ca/2013/07/handling-large-requests-in-twisted/
For multipart POST requests, this populates self.args with temp
file objects instead of strings. Note that these files don't auto-delete
on close because we want to be able to move and rename them.
"""
# max amount of memory to allow any ~single~ request argument [ie: POSTed file]
# note: this value seems to be taken with a grain of salt, memory usage may spike
# FAR above this value in some cases.
# eg: set the memory limit to 5 MB, write 2 blocks of 4MB, mem usage will
# have spiked to 8MB before the data is rolled to disk after the
# second write completes.
memorylimit = 1024*1024*100
# enable/disable debug logging
do_log = False
# re-defined only for debug/logging purposes
def gotLength(self, length):
if self.do_log:
print '%f Headers received, Content-Length: %d' % (time.time(), length)
server.Request.gotLength(self, length)
# re-definition of twisted.web.server.Request.requestreceived, the only difference
# is that self.parse_multipart() is used rather than cgi.parse_multipart()
def requestReceived(self, command, path, version):
from twisted.web.http import parse_qs
if self.do_log:
print '%f Request Received' % time.time()
self.content.seek(0,0)
self.args = {}
self.stack = []
self.method, self.uri = command, path
self.clientproto = version
x = self.uri.split(b'?', 1)
if len(x) == 1:
self.path = self.uri
else:
self.path, argstring = x
self.args = parse_qs(argstring, 1)
# cache the client and server information, we'll need this later to be
# serialized and sent with the request so CGIs will work remotely
self.client = self.channel.transport.getPeer()
self.host = self.channel.transport.getHost()
# Argument processing
args = self.args
ctype = self.requestHeaders.getRawHeaders(b'content-type')
if ctype is not None:
ctype = ctype[0]
if self.method == b"POST" and ctype:
mfd = b'multipart/form-data'
key, pdict = cgi.parse_header(ctype)
if key == b'application/x-www-form-urlencoded':
args.update(parse_qs(self.content.read(), 1))
elif key == mfd:
try:
self.content.seek(0,0)
args.update(self.parse_multipart(self.content, pdict))
#args.update(cgi.parse_multipart(self.content, pdict))
except KeyError as e:
if e.args[0] == b'content-disposition':
# Parse_multipart can't cope with missing
# content-dispostion headers in multipart/form-data
# parts, so we catch the exception and tell the client
# it was a bad request.
self.channel.transport.write(
b"HTTP/1.1 400 Bad Request\r\n\r\n")
self.channel.transport.loseConnection()
return
raise
self.content.seek(0, 0)
self.process()
# re-definition of cgi.parse_multipart that uses a single temporary file to store
# data rather than storing 2 to 3 copies in various lists.
def parse_multipart(self, fp, pdict):
if self.do_log:
print '%f Parsing Multipart data: ' % time.time()
rewind = fp.tell() #save cursor
fp.seek(0,0) #reset cursor
boundary = ""
if 'boundary' in pdict:
boundary = pdict['boundary']
if not cgi.valid_boundary(boundary):
raise ValueError, ('Invalid boundary in multipart form: %r'
% (boundary,))
nextpart = "--" + boundary
lastpart = "--" + boundary + "--"
partdict = {}
terminator = ""
while terminator != lastpart:
c_bytes = -1
data = tempfile.NamedTemporaryFile(delete=False)
if terminator:
# At start of next part. Read headers first.
headers = mimetools.Message(fp)
clength = headers.getheader('content-length')
if clength:
try:
c_bytes = int(clength)
except ValueError:
pass
if c_bytes > 0:
data.write(fp.read(c_bytes))
# Read lines until end of part.
while 1:
line = fp.readline()
if not line:
terminator = lastpart # End outer loop
break
if line[:2] == "--":
terminator = line.strip()
if terminator in (nextpart, lastpart):
break
data.write(line)
# Done with part.
if data.tell() == 0:
continue
if c_bytes < 0:
# if a Content-Length header was not supplied with the MIME part
# then the trailing line break must be removed.
# we have data, read the last 2 bytes
rewind = min(2, data.tell())
data.seek(-rewind, os.SEEK_END)
line = data.read(2)
if line[-2:] == "\r\n":
data.seek(-2, os.SEEK_END)
data.truncate()
elif line[-1:] == "\n":
data.seek(-1, os.SEEK_END)
data.truncate()
line = headers['content-disposition']
if not line:
continue
key, params = cgi.parse_header(line)
if key != 'form-data':
continue
if 'name' in params:
name = params['name']
# kludge in the filename
if 'filename' in params:
fname_index = name + '_filename'
if fname_index in partdict:
partdict[fname_index].append(params['filename'])
else:
partdict[fname_index] = [params['filename']]
else:
# Unnamed parts are not returned at all.
continue
data.seek(0,0)
if name in partdict:
partdict[name].append(data)
else:
partdict[name] = [data]
fp.seek(rewind) # Restore cursor
return partdict
+28
View File
@@ -0,0 +1,28 @@
import logging
import os
from twisted.internet import defer
from lbrynet import conf
from lbrynet.lbrynet_daemon.Daemon import Daemon
from lbrynet.lbrynet_daemon.Resources import LBRYindex, HostedEncryptedFile, EncryptedFileUpload
from lbrynet.conf import settings
log = logging.getLogger(__name__)
class DaemonServer(object):
def _setup_server(self):
ui_path = os.path.join(conf.get_data_dir(), "lbry-ui", "active")
self.root = LBRYindex(ui_path)
self._api = Daemon(self.root)
self.root.putChild("view", HostedEncryptedFile(self._api))
self.root.putChild("upload", EncryptedFileUpload(self._api))
self.root.putChild(settings.API_ADDRESS, self._api)
return defer.succeed(True)
def start(self):
d = self._setup_server()
d.addCallback(lambda _: self._api.setup())
return d
+158
View File
@@ -0,0 +1,158 @@
import json
import logging
import os
from copy import deepcopy
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from lbrynet.core.Error import InsufficientFundsError, KeyFeeAboveMaxAllowed
from lbrynet.core.StreamDescriptor import download_sd_blob
from lbrynet.metadata.Fee import FeeValidator
from lbrynet.lbryfilemanager.EncryptedFileDownloader import ManagedEncryptedFileDownloaderFactory
from lbrynet.conf import settings
INITIALIZING_CODE = 'initializing'
DOWNLOAD_METADATA_CODE = 'downloading_metadata'
DOWNLOAD_TIMEOUT_CODE = 'timeout'
DOWNLOAD_RUNNING_CODE = 'running'
DOWNLOAD_STOPPED_CODE = 'stopped'
STREAM_STAGES = [
(INITIALIZING_CODE, 'Initializing...'),
(DOWNLOAD_METADATA_CODE, 'Downloading metadata'),
(DOWNLOAD_RUNNING_CODE, 'Started stream'),
(DOWNLOAD_STOPPED_CODE, 'Paused stream'),
(DOWNLOAD_TIMEOUT_CODE, 'Stream timed out')
]
log = logging.getLogger(__name__)
class GetStream(object):
def __init__(self, sd_identifier, session, wallet,
lbry_file_manager, exchange_rate_manager,
max_key_fee, data_rate=0.5, timeout=None,
download_directory=None, file_name=None):
if timeout is None:
timeout = settings.download_timeout
self.wallet = wallet
self.resolved_name = None
self.description = None
self.fee = None
self.data_rate = data_rate
self.file_name = file_name
self.session = session
self.exchange_rate_manager = exchange_rate_manager
self.payment_rate_manager = self.session.payment_rate_manager
self.lbry_file_manager = lbry_file_manager
self.sd_identifier = sd_identifier
self.stream_hash = None
self.max_key_fee = max_key_fee
self.stream_info = None
self.stream_info_manager = None
self.d = defer.Deferred(None)
self.timeout = timeout
self.timeout_counter = 0
self.download_directory = download_directory
self.download_path = None
self.downloader = None
self.finished = defer.Deferred(None)
self.checker = LoopingCall(self.check_status)
self.code = STREAM_STAGES[0]
def check_status(self):
self.timeout_counter += 1
# TODO: Why is this the stopping condition for the finished callback?
if self.download_path:
self.checker.stop()
self.finished.callback((True, self.stream_hash, self.download_path))
elif self.timeout_counter >= self.timeout:
log.info("Timeout downloading lbry://%s" % self.resolved_name)
self.checker.stop()
self.d.cancel()
self.code = STREAM_STAGES[4]
self.finished.callback((False, None, None))
def _convert_max_fee(self):
if isinstance(self.max_key_fee, dict):
max_fee = FeeValidator(self.max_key_fee)
if max_fee.currency_symbol == "LBC":
return max_fee.amount
return self.exchange_rate_manager.to_lbc(self.fee).amount
elif isinstance(self.max_key_fee, float):
return float(self.max_key_fee)
def start(self, stream_info, name):
def _cause_timeout(err):
log.info('Cancelling download')
self.timeout_counter = self.timeout * 2
def _set_status(x, status):
log.info("Download lbry://%s status changed to %s" % (self.resolved_name, status))
self.code = next(s for s in STREAM_STAGES if s[0] == status)
return x
def get_downloader_factory(metadata):
for factory in metadata.factories:
if isinstance(factory, ManagedEncryptedFileDownloaderFactory):
return factory, metadata
raise Exception('No suitable factory was found in {}'.format(metadata.factories))
def make_downloader(args):
factory, metadata = args
return factory.make_downloader(metadata,
[self.data_rate, True],
self.payment_rate_manager,
download_directory=self.download_directory,
file_name=self.file_name)
self.resolved_name = name
self.stream_info = deepcopy(stream_info)
self.description = self.stream_info['description']
self.stream_hash = self.stream_info['sources']['lbry_sd_hash']
if 'fee' in self.stream_info:
self.fee = FeeValidator(self.stream_info['fee'])
max_key_fee = self._convert_max_fee()
if self.exchange_rate_manager.to_lbc(self.fee).amount > max_key_fee:
log.info("Key fee %f above limit of %f didn't download lbry://%s" % (self.fee.amount,
self.max_key_fee,
self.resolved_name))
return defer.fail(KeyFeeAboveMaxAllowed())
log.info("Key fee %s below limit of %f, downloading lbry://%s" % (json.dumps(self.fee),
max_key_fee,
self.resolved_name))
self.checker.start(1)
self.d.addCallback(lambda _: _set_status(None, DOWNLOAD_METADATA_CODE))
self.d.addCallback(lambda _: download_sd_blob(self.session, self.stream_hash, self.payment_rate_manager))
self.d.addCallback(self.sd_identifier.get_metadata_for_sd_blob)
self.d.addCallback(lambda r: _set_status(r, DOWNLOAD_RUNNING_CODE))
self.d.addCallback(get_downloader_factory)
self.d.addCallback(make_downloader)
self.d.addCallbacks(self._start_download, _cause_timeout)
self.d.callback(None)
return self.finished
def _start_download(self, downloader):
log.info('Starting download for %s', self.resolved_name)
self.downloader = downloader
self.download_path = os.path.join(downloader.download_directory, downloader.file_name)
d = self._pay_key_fee()
d.addCallback(lambda _: log.info("Downloading %s --> %s", self.stream_hash, self.downloader.file_name))
d.addCallback(lambda _: self.downloader.start())
def _pay_key_fee(self):
if self.fee is not None:
fee_lbc = self.exchange_rate_manager.to_lbc(self.fee).amount
reserved_points = self.wallet.reserve_points(self.fee.address, fee_lbc)
if reserved_points is None:
return defer.fail(InsufficientFundsError())
return self.wallet.send_points_to_address(reserved_points, fee_lbc)
return defer.succeed(None)
@@ -0,0 +1,229 @@
import time
import requests
import logging
import json
import googlefinance
from twisted.internet import defer, reactor
from twisted.internet.task import LoopingCall
from lbrynet.metadata.Fee import FeeValidator
log = logging.getLogger(__name__)
CURRENCY_PAIRS = ["USDBTC", "BTCLBC"]
BITTREX_FEE = 0.0025
COINBASE_FEE = 0.0 #add fee
class ExchangeRate(object):
def __init__(self, market, spot, ts):
assert int(time.time()) - ts < 600
self.currency_pair = (market[0:3], market[3:6])
self.spot = spot
self.ts = ts
def as_dict(self):
return {'spot': self.spot, 'ts': self.ts}
class MarketFeed(object):
def __init__(self, market, name, url, params, fee):
self.market = market
self.name = name
self.url = url
self.params = params
self.fee = fee
self.rate = None
self._updater = LoopingCall(self._update_price)
def _make_request(self):
try:
r = requests.get(self.url, self.params)
return defer.succeed(r.text)
except Exception as err:
log.error(err)
return defer.fail(err)
def _handle_response(self, response):
return NotImplementedError
def _subtract_fee(self, from_amount):
# increase amount to account for market fees
return defer.succeed(from_amount / (1.0 - self.fee))
def _save_price(self, price):
log.debug("Saving price update %f for %s" % (price, self.market))
self.rate = ExchangeRate(self.market, price, int(time.time()))
def _log_error(self, err):
log.error(err)
log.warning("There was a problem updating %s exchange rate information from %s", self.market, self.name)
def _update_price(self):
d = self._make_request()
d.addCallback(self._handle_response)
d.addCallback(self._subtract_fee)
d.addCallback(self._save_price)
d.addErrback(self._log_error)
def start(self):
if not self._updater.running:
self._updater.start(300)
def stop(self):
if self._updater.running:
self._updater.stop()
class BittrexFeed(MarketFeed):
def __init__(self):
MarketFeed.__init__(
self,
"BTCLBC",
"Bittrex",
"https://bittrex.com/api/v1.1/public/getmarkethistory",
{'market': 'BTC-LBC', 'count': 50},
BITTREX_FEE
)
def _handle_response(self, response):
trades = json.loads(response)['result']
vwap = sum([i['Total'] for i in trades]) / sum([i['Quantity'] for i in trades])
return defer.succeed(float(1.0 / vwap))
class GoogleBTCFeed(MarketFeed):
def __init__(self):
MarketFeed.__init__(
self,
"USDBTC",
"Coinbase via Google finance",
None,
None,
COINBASE_FEE
)
def _make_request(self):
try:
r = googlefinance.getQuotes('CURRENCY:USDBTC')[0]
return defer.succeed(r)
except Exception as err:
return defer.fail(err)
def _handle_response(self, response):
return float(response['LastTradePrice'])
def get_default_market_feed(currency_pair):
currencies = None
if isinstance(currency_pair, str):
currencies = (currency_pair[0:3], currency_pair[3:6])
elif isinstance(currency_pair, tuple):
currencies = currency_pair
assert currencies is not None
if currencies == ("USD", "BTC"):
return GoogleBTCFeed()
elif currencies == ("BTC", "LBC"):
return BittrexFeed()
class ExchangeRateManager(object):
def __init__(self):
reactor.addSystemEventTrigger('before', 'shutdown', self.stop)
self.market_feeds = [get_default_market_feed(currency_pair) for currency_pair in CURRENCY_PAIRS]
def start(self):
log.info("Starting exchange rate manager")
for feed in self.market_feeds:
feed.start()
def stop(self):
log.info("Stopping exchange rate manager")
for source in self.market_feeds:
source.stop()
def convert_currency(self, from_currency, to_currency, amount):
log.info("Converting %f %s to %s" % (amount, from_currency, to_currency))
if from_currency == to_currency:
return amount
for market in self.market_feeds:
if market.rate.currency_pair == (from_currency, to_currency):
return amount * market.rate.spot
for market in self.market_feeds:
if market.rate.currency_pair[0] == from_currency:
return self.convert_currency(market.rate.currency_pair[1], to_currency, amount * market.rate.spot)
raise Exception('Unable to convert {} from {} to {}'.format(amount, from_currency, to_currency))
def fee_dict(self):
return {market: market.rate.as_dict() for market in self.market_feeds}
def to_lbc(self, fee):
if fee is None:
return None
if not isinstance(fee, FeeValidator):
fee_in = FeeValidator(fee)
else:
fee_in = fee
return FeeValidator({fee_in.currency_symbol:
{
'amount': self.convert_currency(fee_in.currency_symbol, "LBC", fee_in.amount),
'address': fee_in.address
}
})
class DummyBTCLBCFeed(MarketFeed):
def __init__(self):
MarketFeed.__init__(
self,
"BTCLBC",
"market name",
"derp.com",
None,
0.0
)
class DummyUSDBTCFeed(MarketFeed):
def __init__(self):
MarketFeed.__init__(
self,
"USDBTC",
"market name",
"derp.com",
None,
0.0
)
class DummyExchangeRateManager(object):
def __init__(self, rates):
self.market_feeds = [DummyBTCLBCFeed(), DummyUSDBTCFeed()]
for feed in self.market_feeds:
feed.rate = ExchangeRate(feed.market, rates[feed.market]['spot'], rates[feed.market]['ts'])
def convert_currency(self, from_currency, to_currency, amount):
log.debug("Converting %f %s to %s" % (amount, from_currency, to_currency))
for market in self.market_feeds:
if market.rate.currency_pair == (from_currency, to_currency):
return amount * market.rate.spot
for market in self.market_feeds:
if market.rate.currency_pair[0] == from_currency:
return self.convert_currency(market.rate.currency_pair[1], to_currency, amount * market.rate.spot)
def to_lbc(self, fee):
if fee is None:
return None
if not isinstance(fee, FeeValidator):
fee_in = FeeValidator(fee)
else:
fee_in = fee
return FeeValidator({fee_in.currency_symbol:
{
'amount': self.convert_currency(fee_in.currency_symbol, "LBC", fee_in.amount),
'address': fee_in.address
}
})
+104
View File
@@ -0,0 +1,104 @@
import logging
import os
import sys
import mimetypes
from appdirs import user_data_dir
from zope.interface import implements
from twisted.internet import defer, error, interfaces, abstract, task, reactor
# TODO: omg, this code is essentially duplicated in Daemon
if sys.platform != "darwin":
data_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
data_dir = user_data_dir("LBRY")
if not os.path.isdir(data_dir):
os.mkdir(data_dir)
log = logging.getLogger(__name__)
STATUS_FINISHED = 'finished'
class EncryptedFileStreamer(object):
"""
Writes LBRY stream to request; will pause to wait for new data if the file
is downloading.
No support for range requests (some browser players can't handle it when
the full video data isn't available on request).
"""
implements(interfaces.IPushProducer)
bufferSize = abstract.FileDescriptor.bufferSize
# How long to wait between sending blocks (needed because some
# video players freeze up if you try to send data too fast)
stream_interval = 0.005
# How long to wait before checking if new data has been appended to the file
new_data_check_interval = 0.25
def __init__(self, request, path, stream, file_manager):
def _set_content_length_header(length):
self._request.setHeader('content-length', length)
return defer.succeed(None)
self._request = request
self._file = open(path, 'rb')
self._stream = stream
self._file_manager = file_manager
self._headers_sent = False
self._running = True
self._request.setResponseCode(200)
self._request.setHeader('accept-ranges', 'none')
self._request.setHeader('content-type', mimetypes.guess_type(path)[0])
self._request.setHeader("Content-Security-Policy", "sandbox")
self._deferred = stream.get_total_bytes()
self._deferred.addCallback(_set_content_length_header)
self._deferred.addCallback(lambda _: self.resumeProducing())
def _check_for_new_data(self):
def _recurse_or_stop(stream_status):
if not self._running:
return
if stream_status != STATUS_FINISHED:
self._deferred.addCallback(lambda _: task.deferLater(reactor, self.new_data_check_interval, self._check_for_new_data))
else:
self.stopProducing()
if not self._running:
return
# Clear the file's EOF indicator by seeking to current position
self._file.seek(self._file.tell())
data = self._file.read(self.bufferSize)
if data:
self._request.write(data)
if self._running: # .write() can trigger a pause
self._deferred.addCallback(lambda _: task.deferLater(reactor, self.stream_interval, self._check_for_new_data))
else:
self._deferred.addCallback(lambda _: self._file_manager.get_lbry_file_status(self._stream))
self._deferred.addCallback(_recurse_or_stop)
def pauseProducing(self):
self._running = False
def resumeProducing(self):
self._running = True
self._check_for_new_data()
def stopProducing(self):
self._running = False
self._file.close()
self._deferred.addErrback(lambda err: err.trap(defer.CancelledError))
self._deferred.addErrback(lambda err: err.trap(error.ConnectionDone))
self._deferred.cancel()
self._request.unregisterProducer()
self._request.finish()
File diff suppressed because it is too large. Load diff
-124
View File
@@ -1,124 +0,0 @@
import argparse
import logging
import logging.handlers
import os
import webbrowser
import sys
import socket
import platform
from appdirs import user_data_dir
from twisted.web import server
from twisted.internet import reactor, defer
from jsonrpc.proxy import JSONRPCProxy
from lbrynet.lbrynet_daemon.LBRYDaemonServer import LBRYDaemonServer
from lbrynet.conf import API_CONNECTION_STRING, API_INTERFACE, API_ADDRESS, API_PORT, \
DEFAULT_WALLET, UI_ADDRESS, DEFAULT_UI_BRANCH, LOG_FILE_NAME
if sys.platform != "darwin":
log_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
log_dir = user_data_dir("LBRY")
if not os.path.isdir(log_dir):
os.mkdir(log_dir)
lbrynet_log = os.path.join(log_dir, LOG_FILE_NAME)
log = logging.getLogger(__name__)
handler = logging.handlers.RotatingFileHandler(lbrynet_log, maxBytes=2097152, backupCount=5)
log.addHandler(handler)
log.setLevel(logging.INFO)
REMOTE_SERVER = "www.google.com"
def test_internet_connection():
try:
host = socket.gethostbyname(REMOTE_SERVER)
s = socket.create_connection((host, 80), 2)
return True
except:
return False
def stop():
def _disp_shutdown():
print "Shutting down lbrynet-daemon from command line"
log.info("Shutting down lbrynet-daemon from command line")
def _disp_not_running():
print "Attempt to shut down lbrynet-daemon from command line when daemon isn't running"
log.info("Attempt to shut down lbrynet-daemon from command line when daemon isn't running")
d = defer.Deferred(None)
d.addCallback(lambda _: JSONRPCProxy.from_url(API_CONNECTION_STRING).stop())
d.addCallbacks(lambda _: _disp_shutdown(), lambda _: _disp_not_running())
d.callback(None)
def start():
parser = argparse.ArgumentParser(description="Launch lbrynet-daemon")
parser.add_argument("--wallet",
help="lbrycrd or lbryum, default lbryum",
type=str,
default='')
parser.add_argument("--ui",
help="path to custom UI folder",
default=None)
parser.add_argument("--branch",
help="Branch of lbry-web-ui repo to use, defaults on master")
parser.add_argument('--no-launch', dest='launchui', action="store_false")
parser.add_argument('--log-to-console', dest='logtoconsole', action="store_true")
parser.add_argument('--quiet', dest='quiet', action="store_true")
parser.set_defaults(branch=False, launchui=True, logtoconsole=False, quiet=False)
args = parser.parse_args()
if args.logtoconsole:
logging.basicConfig(level=logging.INFO)
args = parser.parse_args()
try:
JSONRPCProxy.from_url(API_CONNECTION_STRING).is_running()
log.info("lbrynet-daemon is already running")
if not args.logtoconsole:
print "lbrynet-daemon is already running"
if args.launchui:
webbrowser.open(UI_ADDRESS)
return
except:
pass
log.info("Starting lbrynet-daemon from command line")
if not args.logtoconsole and not args.quiet:
print "Starting lbrynet-daemon from command line"
print "To view activity, view the log file here: " + lbrynet_log
print "Web UI is available at http://%s:%i" % (API_INTERFACE, API_PORT)
print "JSONRPC API is available at " + API_CONNECTION_STRING
print "To quit press ctrl-c or call 'stop' via the API"
if test_internet_connection():
lbry = LBRYDaemonServer()
d = lbry.start(branch=args.branch if args.branch else DEFAULT_UI_BRANCH,
user_specified=args.ui,
wallet=args.wallet,
branch_specified=True if args.branch else False)
if args.launchui:
d.addCallback(lambda _: webbrowser.open(UI_ADDRESS))
reactor.listenTCP(API_PORT, server.Site(lbry.root), interface=API_INTERFACE)
reactor.run()
if not args.logtoconsole and not args.quiet:
print "\nClosing lbrynet-daemon"
else:
log.info("Not connected to internet, unable to start")
if not args.logtoconsole:
print "Not connected to internet, unable to start"
return
if __name__ == "__main__":
start()
-194
View File
@@ -1,194 +0,0 @@
import logging
import os
import shutil
import json
import sys
import mimetypes
from datetime import datetime
from appdirs import user_data_dir
from twisted.web import server, static, resource
from twisted.internet import defer, interfaces, error, reactor, task, threads
from zope.interface import implements
from lbrynet.lbrynet_daemon.LBRYDaemon import LBRYDaemon
from lbrynet.conf import API_CONNECTION_STRING, API_ADDRESS, DEFAULT_WALLET, UI_ADDRESS, DEFAULT_UI_BRANCH, LOG_FILE_NAME
if sys.platform != "darwin":
data_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
data_dir = user_data_dir("LBRY")
if not os.path.isdir(data_dir):
os.mkdir(data_dir)
lbrynet_log = os.path.join(data_dir, LOG_FILE_NAME)
log = logging.getLogger(__name__)
handler = logging.handlers.RotatingFileHandler(lbrynet_log, maxBytes=2097152, backupCount=5)
log.addHandler(handler)
log.setLevel(logging.INFO)
class LBRYindex(resource.Resource):
def __init__(self, ui_dir):
resource.Resource.__init__(self)
self.ui_dir = ui_dir
isLeaf = False
def _delayed_render(self, request, results):
request.write(str(results))
request.finish()
def getChild(self, name, request):
if name == '':
return self
return resource.Resource.getChild(self, name, request)
def render_GET(self, request):
return static.File(os.path.join(self.ui_dir, "index.html")).render_GET(request)
class LBRYFileStreamer(object):
"""
Writes downloaded LBRY file to request as the download comes in, pausing and resuming as requested
used for Chrome
"""
implements(interfaces.IPushProducer)
def __init__(self, request, path, start, stop, size):
self._request = request
self._fileObject = file(path)
self._content_type = mimetypes.guess_type(path)[0]
self._stop_pos = size - 1 if stop == '' else int(stop) #chrome and firefox send range requests for "0-"
self._cursor = self._start_pos = int(start)
self._file_size = size
self._depth = 0
self._paused = self._sent_bytes = self._stopped = False
self._delay = 0.25
self._deferred = defer.succeed(None)
self._request.setResponseCode(206)
self._request.setHeader('accept-ranges', 'bytes')
self._request.setHeader('content-type', self._content_type)
self.resumeProducing()
def pauseProducing(self):
self._paused = True
log.info("[" + str(datetime.now()) + "] Pausing producer")
return defer.succeed(None)
def resumeProducing(self):
def _check_for_new_data():
self._depth += 1
self._fileObject.seek(self._start_pos, os.SEEK_END)
readable_bytes = self._fileObject.tell()
self._fileObject.seek(self._cursor)
self._sent_bytes = False
if (readable_bytes > self._cursor) and not (self._stopped or self._paused):
read_length = min(readable_bytes, self._stop_pos) - self._cursor + 1
self._request.setHeader('content-range', 'bytes %s-%s/%s' % (self._cursor, self._cursor + read_length - 1, self._file_size))
self._request.setHeader('content-length', str(read_length))
start_cur = self._cursor
for i in range(read_length):
if self._paused or self._stopped:
break
else:
data = self._fileObject.read(1)
self._request.write(data)
self._cursor += 1
log.info("[" + str(datetime.now()) + "] Wrote range %s-%s/%s, length: %s, readable: %s, depth: %s" %
(start_cur, self._cursor, self._file_size, self._cursor - start_cur, readable_bytes, self._depth))
self._sent_bytes = True
if self._cursor == self._stop_pos + 1:
self.stopProducing()
return defer.succeed(None)
elif self._paused or self._stopped:
return defer.succeed(None)
else:
self._deferred.addCallback(lambda _: threads.deferToThread(reactor.callLater, self._delay, _check_for_new_data))
return defer.succeed(None)
log.info("[" + str(datetime.now()) + "] Resuming producer")
self._paused = False
self._deferred.addCallback(lambda _: _check_for_new_data())
def stopProducing(self):
log.info("[" + str(datetime.now()) + "] Stopping producer")
self._stopped = True
# self._fileObject.close()
self._deferred.addErrback(lambda err: err.trap(defer.CancelledError))
self._deferred.addErrback(lambda err: err.trap(error.ConnectionDone))
self._deferred.cancel()
# self._request.finish()
self._request.unregisterProducer()
return defer.succeed(None)
class HostedLBRYFile(resource.Resource):
def __init__(self, api):
self._api = api
self._producer = None
resource.Resource.__init__(self)
# todo: fix LBRYFileStreamer and use it instead of static.File
# def makeProducer(self, request, stream):
# def _save_producer(producer):
# self._producer = producer
# return defer.succeed(None)
#
# range_header = request.getAllHeaders()['range'].replace('bytes=', '').split('-')
# start, stop = int(range_header[0]), range_header[1]
# log.info("[" + str(datetime.now()) + "] GET range %s-%s" % (start, stop))
# path = os.path.join(self._api.download_directory, stream.file_name)
#
# d = stream.get_total_bytes()
# d.addCallback(lambda size: _save_producer(LBRYFileStreamer(request, path, start, stop, size)))
# d.addCallback(lambda _: request.registerProducer(self._producer, streaming=True))
# # request.notifyFinish().addCallback(lambda _: self._producer.stopProducing())
# request.notifyFinish().addErrback(self._responseFailed, d)
# return d
def render_GET(self, request):
if 'name' in request.args.keys():
if request.args['name'][0] != 'lbry' and request.args['name'][0] not in self._api.waiting_on.keys():
d = self._api._download_name(request.args['name'][0])
# d.addCallback(lambda stream: self.makeProducer(request, stream))
d.addCallback(lambda stream: static.File(os.path.join(self._api.download_directory,
stream.file_name)).render_GET(request))
elif request.args['name'][0] in self._api.waiting_on.keys():
request.redirect(UI_ADDRESS + "/?watch=" + request.args['name'][0])
request.finish()
else:
request.redirect(UI_ADDRESS)
request.finish()
return server.NOT_DONE_YET
# def _responseFailed(self, err, call):
# call.addErrback(lambda err: err.trap(error.ConnectionDone))
# call.addErrback(lambda err: err.trap(defer.CancelledError))
# call.addErrback(lambda err: log.info("Error: " + str(err)))
# call.cancel()
class LBRYDaemonServer(object):
def _setup_server(self, wallet):
self.root = LBRYindex(os.path.join(os.path.join(data_dir, "lbry-ui"), "active"))
self._api = LBRYDaemon(self.root, wallet_type=wallet)
self.root.putChild("view", HostedLBRYFile(self._api))
self.root.putChild(API_ADDRESS, self._api)
return defer.succeed(True)
def start(self, branch=DEFAULT_UI_BRANCH, user_specified=False, branch_specified=False, wallet=None):
d = self._setup_server(wallet)
d.addCallback(lambda _: self._api.setup(branch, user_specified, branch_specified))
return d
-159
View File
@@ -1,159 +0,0 @@
import json
import logging
import os
import sys
from appdirs import user_data_dir
from datetime import datetime
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from lbrynet.core.Error import InvalidStreamInfoError, InsufficientFundsError
from lbrynet.core.PaymentRateManager import PaymentRateManager
from lbrynet.core.StreamDescriptor import download_sd_blob
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloaderFactory
from lbrynet.conf import DEFAULT_TIMEOUT, LOG_FILE_NAME
INITIALIZING_CODE = 'initializing'
DOWNLOAD_METADATA_CODE = 'downloading_metadata'
DOWNLOAD_TIMEOUT_CODE = 'timeout'
DOWNLOAD_RUNNING_CODE = 'running'
DOWNLOAD_STOPPED_CODE = 'stopped'
STREAM_STAGES = [
(INITIALIZING_CODE, 'Initializing...'),
(DOWNLOAD_METADATA_CODE, 'Downloading metadata'),
(DOWNLOAD_RUNNING_CODE, 'Started stream'),
(DOWNLOAD_STOPPED_CODE, 'Paused stream'),
(DOWNLOAD_TIMEOUT_CODE, 'Stream timed out')
]
if sys.platform != "darwin":
log_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
log_dir = user_data_dir("LBRY")
if not os.path.isdir(log_dir):
os.mkdir(log_dir)
lbrynet_log = os.path.join(log_dir, LOG_FILE_NAME)
log = logging.getLogger(__name__)
handler = logging.handlers.RotatingFileHandler(lbrynet_log, maxBytes=2097152, backupCount=5)
log.addHandler(handler)
log.setLevel(logging.INFO)
class GetStream(object):
def __init__(self, sd_identifier, session, wallet, lbry_file_manager, max_key_fee, pay_key=True, data_rate=0.5,
timeout=DEFAULT_TIMEOUT, download_directory=None, file_name=None):
self.wallet = wallet
self.resolved_name = None
self.description = None
self.key_fee = None
self.key_fee_address = None
self.data_rate = data_rate
self.pay_key = pay_key
self.name = None
self.file_name = file_name
self.session = session
self.payment_rate_manager = PaymentRateManager(self.session.base_payment_rate_manager)
self.lbry_file_manager = lbry_file_manager
self.sd_identifier = sd_identifier
self.stream_hash = None
self.max_key_fee = max_key_fee
self.stream_info = None
self.stream_info_manager = None
self.d = defer.Deferred(None)
self.timeout = timeout
self.timeout_counter = 0
self.download_directory = download_directory
self.download_path = None
self.downloader = None
self.finished = defer.Deferred()
self.checker = LoopingCall(self.check_status)
self.code = STREAM_STAGES[0]
def check_status(self):
self.timeout_counter += 1
if self.download_path:
self.checker.stop()
self.finished.callback((self.stream_hash, self.download_path))
elif self.timeout_counter >= self.timeout:
log.info("Timeout downloading lbry://" + self.resolved_name + ", " + str(self.stream_info))
self.checker.stop()
self.d.cancel()
self.code = STREAM_STAGES[4]
self.finished.callback(False)
def start(self, stream_info, name):
self.resolved_name = name
self.stream_info = stream_info
if 'stream_hash' in self.stream_info.keys():
self.stream_hash = self.stream_info['stream_hash']
elif 'sources' in self.stream_info.keys():
self.stream_hash = self.stream_info['sources']['lbry_sd_hash']
else:
raise InvalidStreamInfoError(self.stream_info)
if 'description' in self.stream_info.keys():
self.description = self.stream_info['description']
if 'key_fee' in self.stream_info.keys():
self.key_fee = float(self.stream_info['key_fee'])
if 'key_fee_address' in self.stream_info.keys():
self.key_fee_address = self.stream_info['key_fee_address']
else:
self.key_fee_address = None
else:
self.key_fee = None
self.key_fee_address = None
if self.key_fee > self.max_key_fee:
if self.pay_key:
log.info("Key fee (" + str(self.key_fee) + ") above limit of " + str(
self.max_key_fee) + ", didn't download lbry://" + str(self.resolved_name))
return defer.fail(None)
else:
pass
def _cause_timeout():
self.timeout_counter = self.timeout * 2
def _set_status(x, status):
self.code = next(s for s in STREAM_STAGES if s[0] == status)
return x
self.checker.start(1)
self.d.addCallback(lambda _: _set_status(None, DOWNLOAD_METADATA_CODE))
self.d.addCallback(lambda _: download_sd_blob(self.session, self.stream_hash, self.payment_rate_manager))
self.d.addCallback(self.sd_identifier.get_metadata_for_sd_blob)
self.d.addCallback(lambda r: _set_status(r, DOWNLOAD_RUNNING_CODE))
self.d.addCallback(lambda metadata: (
next(factory for factory in metadata.factories if isinstance(factory, ManagedLBRYFileDownloaderFactory)),
metadata))
self.d.addCallback(lambda (factory, metadata): factory.make_downloader(metadata,
[self.data_rate, True],
self.payment_rate_manager,
download_directory=self.download_directory,
file_name=self.file_name))
self.d.addCallbacks(self._start_download, lambda _: _cause_timeout())
self.d.callback(None)
return self.finished
def _start_download(self, downloader):
def _pay_key_fee():
if self.key_fee is not None and self.key_fee_address is not None:
reserved_points = self.wallet.reserve_points(self.key_fee_address, self.key_fee)
if reserved_points is None:
return defer.fail(InsufficientFundsError())
log.info("Key fee: " + str(self.key_fee) + " | " + str(self.key_fee_address))
return self.wallet.send_points_to_address(reserved_points, self.key_fee)
return defer.succeed(None)
if self.pay_key:
d = _pay_key_fee()
else:
d = defer.Deferred()
self.downloader = downloader
self.download_path = os.path.join(downloader.download_directory, downloader.file_name)
d.addCallback(lambda _: log.info("[" + str(datetime.now()) + "] Downloading " + str(self.stream_hash) + " --> " + str(self.download_path)))
d.addCallback(lambda _: self.downloader.start())
-149
View File
@@ -1,149 +0,0 @@
import logging
import os
import sys
from appdirs import user_data_dir
from datetime import datetime
from lbrynet.core.Error import InsufficientFundsError
from lbrynet.lbryfilemanager.LBRYFileCreator import create_lbry_file
from lbrynet.lbryfile.StreamDescriptor import publish_sd_blob
from lbrynet.core.PaymentRateManager import PaymentRateManager
from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
from lbrynet.conf import LOG_FILE_NAME
from twisted.internet import threads, defer
if sys.platform != "darwin":
log_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
log_dir = user_data_dir("LBRY")
if not os.path.isdir(log_dir):
os.mkdir(log_dir)
lbrynet_log = os.path.join(log_dir, LOG_FILE_NAME)
log = logging.getLogger(__name__)
handler = logging.handlers.RotatingFileHandler(lbrynet_log, maxBytes=2097152, backupCount=5)
log.addHandler(handler)
log.setLevel(logging.INFO)
class Publisher(object):
def __init__(self, session, lbry_file_manager, wallet):
self.session = session
self.lbry_file_manager = lbry_file_manager
self.wallet = wallet
self.received_file_name = False
self.file_path = None
self.file_name = None
self.thumbnail = None
self.title = None
self.publish_name = None
self.bid_amount = None
self.key_fee = None
self.key_fee_address = None
self.key_fee_address_chosen = False
self.description = None
self.verified = False
self.lbry_file = None
self.sd_hash = None
self.tx_hash = None
self.content_license = None
self.author = None
self.sources = None
def start(self, name, file_path, bid, title=None, description=None, thumbnail=None,
key_fee=None, key_fee_address=None, content_license=None, author=None, sources=None):
def _show_result():
message = "[" + str(datetime.now()) + "] Published " + self.file_name + " --> lbry://" + \
str(self.publish_name) + " with txid: " + str(self.tx_hash)
log.info(message)
return defer.succeed(self.tx_hash)
self.publish_name = name
self.file_path = file_path
self.bid_amount = bid
self.title = title
self.description = description
self.thumbnail = thumbnail
self.key_fee = key_fee
self.key_fee_address = key_fee_address
self.content_license = content_license
self.author = author
self.sources = sources
d = self._check_file_path(self.file_path)
d.addCallback(lambda _: create_lbry_file(self.session, self.lbry_file_manager,
self.file_name, open(self.file_path)))
d.addCallback(self.add_to_lbry_files)
d.addCallback(lambda _: self._create_sd_blob())
d.addCallback(lambda _: self._claim_name())
d.addCallbacks(lambda _: _show_result(), self._show_publish_error)
return d
def _check_file_path(self, file_path):
def check_file_threaded():
f = open(file_path)
f.close()
self.file_name = os.path.basename(self.file_path)
return True
return threads.deferToThread(check_file_threaded)
def _get_new_address(self):
d = self.wallet.get_new_address()
def set_address(address):
self.key_fee_address = address
return True
d.addCallback(set_address)
return d
def set_status(self, lbry_file_downloader):
self.lbry_file = lbry_file_downloader
d = self.lbry_file_manager.change_lbry_file_status(self.lbry_file, ManagedLBRYFileDownloader.STATUS_FINISHED)
d.addCallback(lambda _: lbry_file_downloader.restore())
return d
def add_to_lbry_files(self, stream_hash):
prm = PaymentRateManager(self.session.base_payment_rate_manager)
d = self.lbry_file_manager.add_lbry_file(stream_hash, prm)
d.addCallback(self.set_status)
return d
def _create_sd_blob(self):
d = publish_sd_blob(self.lbry_file_manager.stream_info_manager, self.session.blob_manager,
self.lbry_file.stream_hash)
def set_sd_hash(sd_hash):
self.sd_hash = sd_hash
d.addCallback(set_sd_hash)
return d
def _claim_name(self):
d = self.wallet.claim_name(self.publish_name, self.sd_hash, self.bid_amount,
description=self.description, key_fee=self.key_fee,
key_fee_address=self.key_fee_address, thumbnail=self.thumbnail,
content_license=self.content_license, author=self.author,
sources=self.sources)
def set_tx_hash(tx_hash):
self.tx_hash = tx_hash
d.addCallback(set_tx_hash)
return d
def _show_publish_error(self, err):
log.info(err.getTraceback())
message = "An error occurred publishing %s to %s. Error: %s."
if err.check(InsufficientFundsError):
error_message = "Insufficient funds"
else:
d = defer.succeed(True)
error_message = err.getErrorMessage()
log.error(error_message)
log.error(message, str(self.file_name), str(self.publish_name), err.getTraceback())
return d
+27
View File
@@ -0,0 +1,27 @@
import logging
import random
from txjsonrpc.web.jsonrpc import Proxy
from lbrynet.conf import settings
log = logging.getLogger(__name__)
class LighthouseClient(object):
def __init__(self, servers=None):
self.servers = servers or settings.search_servers
def _get_random_server(self):
return Proxy(random.choice(self.servers))
def _run_query(self, func, arg):
return self._get_random_server().callRemote(func, arg)
def search(self, search):
return self._run_query('search', search)
def announce_sd(self, sd_hash):
log.info("Announce sd to lighthouse")
return self._run_query('announce_sd', sd_hash)
def check_available(self, sd_hash):
return self._run_query('check_available', sd_hash)
+153
View File
@@ -0,0 +1,153 @@
import logging
import mimetypes
import os
import random
from twisted.internet import threads, defer, reactor
from lbrynet.core.Error import InsufficientFundsError
from lbrynet.lbryfilemanager.EncryptedFileCreator import create_lbry_file
from lbrynet.lbryfile.StreamDescriptor import publish_sd_blob
from lbrynet.metadata.Metadata import Metadata
from lbrynet.lbryfilemanager.EncryptedFileDownloader import ManagedEncryptedFileDownloader
from lbrynet import reflector
from lbrynet.conf import settings
log = logging.getLogger(__name__)
class Publisher(object):
def __init__(self, session, lbry_file_manager, wallet):
self.session = session
self.lbry_file_manager = lbry_file_manager
self.wallet = wallet
self.received_file_name = False
self.file_path = None
self.file_name = None
self.publish_name = None
self.bid_amount = None
self.verified = False
self.lbry_file = None
self.txid = None
self.stream_hash = None
reflector_server = random.choice(settings.reflector_servers)
self.reflector_server, self.reflector_port = reflector_server[0], reflector_server[1]
self.metadata = {}
def start(self, name, file_path, bid, metadata):
log.info('Starting publish for %s', name)
def _show_result():
log.info("Success! Published %s --> lbry://%s txid: %s", self.file_name, self.publish_name, self.txid)
return defer.succeed(self.txid)
self.publish_name = name
self.file_path = file_path
self.bid_amount = bid
self.metadata = metadata
if os.name == "nt":
file_mode = 'rb'
else:
file_mode = 'r'
d = self._check_file_path(self.file_path)
d.addCallback(lambda _: create_lbry_file(self.session, self.lbry_file_manager,
self.file_name, open(self.file_path, file_mode)))
d.addCallback(self.add_to_lbry_files)
d.addCallback(lambda _: self._create_sd_blob())
d.addCallback(lambda _: self._claim_name())
d.addCallback(lambda _: self.set_status())
d.addCallback(lambda _: self.start_reflector())
d.addCallbacks(lambda _: _show_result(), self._show_publish_error)
return d
def start_reflector(self):
reflector_server = random.choice(settings.reflector_servers)
reflector_address, reflector_port = reflector_server[0], reflector_server[1]
log.info("Reflecting new publication")
factory = reflector.ClientFactory(
self.session.blob_manager,
self.lbry_file_manager.stream_info_manager,
self.stream_hash
)
d = reactor.resolve(reflector_address)
d.addCallback(lambda ip: reactor.connectTCP(ip, reflector_port, factory))
d.addCallback(lambda _: factory.finished_deferred)
return d
def _check_file_path(self, file_path):
def check_file_threaded():
f = open(file_path)
f.close()
self.file_name = os.path.basename(self.file_path)
return True
return threads.deferToThread(check_file_threaded)
def set_lbry_file(self, lbry_file_downloader):
self.lbry_file = lbry_file_downloader
return defer.succeed(None)
def add_to_lbry_files(self, stream_hash):
self.stream_hash = stream_hash
prm = self.session.payment_rate_manager
d = self.lbry_file_manager.add_lbry_file(stream_hash, prm)
d.addCallback(self.set_lbry_file)
return d
def _create_sd_blob(self):
log.debug('Creating stream descriptor blob')
d = publish_sd_blob(self.lbry_file_manager.stream_info_manager,
self.session.blob_manager,
self.lbry_file.stream_hash)
def set_sd_hash(sd_hash):
log.debug('stream descriptor hash: %s', sd_hash)
if 'sources' not in self.metadata:
self.metadata['sources'] = {}
self.metadata['sources']['lbry_sd_hash'] = sd_hash
d.addCallback(set_sd_hash)
return d
def set_status(self):
log.debug('Setting status')
d = self.lbry_file_manager.change_lbry_file_status(self.lbry_file, ManagedEncryptedFileDownloader.STATUS_FINISHED)
d.addCallback(lambda _: self.lbry_file.restore())
return d
def _claim_name(self):
log.debug('Claiming name')
self._update_metadata()
m = Metadata(self.metadata)
def set_tx_hash(txid):
log.debug('Name claimed using txid: %s', txid)
self.txid = txid
d = self.wallet.claim_name(self.publish_name, self.bid_amount, m)
d.addCallback(set_tx_hash)
return d
def _update_metadata(self):
filename = os.path.join(self.lbry_file.download_directory, self.lbry_file.file_name)
self.metadata['content_type'] = get_content_type(filename)
self.metadata['ver'] = Metadata.current_version
def _show_publish_error(self, err):
log.info(err.getTraceback())
message = "An error occurred publishing %s to %s. Error: %s."
if err.check(InsufficientFundsError):
error_message = "Insufficient funds"
else:
error_message = err.getErrorMessage()
log.error(error_message)
log.error(message, str(self.file_name), str(self.publish_name), err.getTraceback())
return defer.fail(Exception("Publish failed"))
def get_content_type(filename):
return mimetypes.guess_type(filename)[0] or 'application/octet-stream'
+133
View File
@@ -0,0 +1,133 @@
import logging
import os
import shutil
import json
import sys
import tempfile
from appdirs import user_data_dir
from twisted.web import server, static, resource
from twisted.internet import defer, error
from lbrynet.conf import settings
from lbrynet.lbrynet_daemon.FileStreamer import EncryptedFileStreamer
# TODO: omg, this code is essentially duplicated in Daemon
if sys.platform != "darwin":
data_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
data_dir = user_data_dir("LBRY")
if not os.path.isdir(data_dir):
os.mkdir(data_dir)
log = logging.getLogger(__name__)
class NoCacheStaticFile(static.File):
def _set_no_cache(self, request):
request.setHeader('cache-control', 'no-cache, no-store, must-revalidate')
request.setHeader('expires', '0')
def render_GET(self, request):
self._set_no_cache(request)
return static.File.render_GET(self, request)
class LBRYindex(resource.Resource):
def __init__(self, ui_dir):
resource.Resource.__init__(self)
self.ui_dir = ui_dir
isLeaf = False
def _delayed_render(self, request, results):
request.write(str(results))
request.finish()
def getChild(self, name, request):
request.setHeader('cache-control', 'no-cache, no-store, must-revalidate')
request.setHeader('expires', '0')
if name == '':
return self
return resource.Resource.getChild(self, name, request)
def render_GET(self, request):
return NoCacheStaticFile(os.path.join(self.ui_dir, "index.html")).render_GET(request)
class HostedEncryptedFile(resource.Resource):
def __init__(self, api):
self._api = api
resource.Resource.__init__(self)
def _make_stream_producer(self, request, stream):
path = os.path.join(self._api.download_directory, stream.file_name)
producer = EncryptedFileStreamer(request, path, stream, self._api.lbry_file_manager)
request.registerProducer(producer, streaming=True)
d = request.notifyFinish()
d.addErrback(self._responseFailed, d)
return d
def render_GET(self, request):
request.setHeader("Content-Security-Policy", "sandbox")
if 'name' in request.args.keys():
if request.args['name'][0] != 'lbry' and request.args['name'][0] not in self._api.waiting_on.keys():
d = self._api._download_name(request.args['name'][0])
d.addCallback(lambda stream: self._make_stream_producer(request, stream))
elif request.args['name'][0] in self._api.waiting_on.keys():
request.redirect(settings.UI_ADDRESS + "/?watch=" + request.args['name'][0])
request.finish()
else:
request.redirect(settings.UI_ADDRESS)
request.finish()
return server.NOT_DONE_YET
def _responseFailed(self, err, call):
call.addErrback(lambda err: err.trap(error.ConnectionDone))
call.addErrback(lambda err: err.trap(defer.CancelledError))
call.addErrback(lambda err: log.info("Error: " + str(err)))
call.cancel()
class EncryptedFileUpload(resource.Resource):
"""
Accepts a file sent via the file upload widget in the web UI, saves
it into a temporary dir, and responds with a JSON string containing
the path of the newly created file.
"""
def __init__(self, api):
self._api = api
def render_POST(self, request):
origfilename = request.args['file_filename'][0]
uploaded_file = request.args['file'][0] # Temp file created by request
# Move to a new temporary dir and restore the original file name
newdirpath = tempfile.mkdtemp()
newpath = os.path.join(newdirpath, origfilename)
if os.name == "nt":
shutil.copy(uploaded_file.name, newpath)
# TODO Still need to remove the file
# TODO deal with pylint error in cleaner fashion than this
try:
from exceptions import WindowsError as win_except
except ImportError as e:
log.error("This shouldn't happen")
win_except = Exception
try:
os.remove(uploaded_file.name)
except win_except as e:
pass
else:
shutil.move(uploaded_file.name, newpath)
self._api.uploaded_temp_files.append(newpath)
return json.dumps(newpath)
@@ -6,30 +6,20 @@ import json
from urllib2 import urlopen
from StringIO import StringIO
from twisted.web import static
from twisted.internet import defer
from lbrynet.conf import DEFAULT_UI_BRANCH, LOG_FILE_NAME
from twisted.internet.task import LoopingCall
from lbrynet.conf import settings
from lbrynet.lbrynet_daemon.Resources import NoCacheStaticFile
from lbrynet import __version__ as lbrynet_version
from lbryum.version import ELECTRUM_VERSION as lbryum_version
from lbryum.version import LBRYUM_VERSION as lbryum_version
from zipfile import ZipFile
from appdirs import user_data_dir
if sys.platform != "darwin":
log_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
else:
log_dir = user_data_dir("LBRY")
if not os.path.isdir(log_dir):
os.mkdir(log_dir)
lbrynet_log = os.path.join(log_dir, LOG_FILE_NAME)
log = logging.getLogger(__name__)
handler = logging.handlers.RotatingFileHandler(lbrynet_log, maxBytes=2097152, backupCount=5)
log.addHandler(handler)
log.setLevel(logging.INFO)
class LBRYUIManager(object):
class UIManager(object):
def __init__(self, root):
if sys.platform != "darwin":
self.data_dir = os.path.join(os.path.expanduser("~"), '.lbrynet')
@@ -52,9 +42,12 @@ class LBRYUIManager(object):
self.config = os.path.join(self.ui_root, "active.json")
self.update_requires = os.path.join(self.update_dir, "requirements.txt")
self.requirements = {}
self.check_requirements = True
self.ui_dir = self.active_dir
self.git_version = None
self.root = root
self.branch = None
self.update_checker = LoopingCall(self.setup)
if not os.path.isfile(os.path.join(self.config)):
self.loaded_git_version = None
@@ -73,26 +66,29 @@ class LBRYUIManager(object):
self.loaded_branch = None
self.loaded_requirements = None
def setup(self, branch=DEFAULT_UI_BRANCH, user_specified=None, branch_specified=False):
self.branch = branch
if user_specified:
if os.path.isdir(user_specified):
log.info("Checking user specified UI directory: " + str(user_specified))
def setup(self, branch=None, check_requirements=None, user_specified=None):
local_ui_path = settings.local_ui_path or user_specified
self.branch = settings.ui_branch or branch
self.check_requirements = settings.check_ui_requirements or check_requirements
if local_ui_path:
if os.path.isdir(local_ui_path):
log.info("Checking user specified UI directory: " + str(local_ui_path))
self.branch = "user-specified"
self.loaded_git_version = "user-specified"
d = self.migrate_ui(source=user_specified)
d = self.migrate_ui(source=local_ui_path)
d.addCallback(lambda _: self._load_ui())
return d
else:
log.info("User specified UI directory doesn't exist, using " + branch)
elif self.loaded_branch == "user-specified" and not branch_specified:
log.info("User specified UI directory doesn't exist, using " + self.branch)
elif self.loaded_branch == "user-specified":
log.info("Loading user provided UI")
d = self._load_ui()
return d
else:
log.info("Checking for updates for UI branch: " + branch)
self._git_url = "https://api.github.com/repos/lbryio/lbry-web-ui/git/refs/heads/%s" % branch
self._dist_url = "https://raw.githubusercontent.com/lbryio/lbry-web-ui/%s/dist.zip" % branch
log.info("Checking for updates for UI branch: " + self.branch)
self._git_url = "https://s3.amazonaws.com/lbry-ui/{}/data.json".format(self.branch)
self._dist_url = "https://s3.amazonaws.com/lbry-ui/{}/dist.zip".format(self.branch)
d = self._up_to_date()
d.addCallback(lambda r: self._download_ui() if not r else self._load_ui())
@@ -100,9 +96,12 @@ class LBRYUIManager(object):
def _up_to_date(self):
def _get_git_info():
response = urlopen(self._git_url)
data = json.loads(response.read())
return defer.succeed(data['object']['sha'])
try:
response = urlopen(self._git_url)
data = json.loads(response.read())
return defer.succeed(data['sha'])
except Exception:
return defer.fail()
def _set_git(version):
self.git_version = version.replace('\n', '')
@@ -113,8 +112,12 @@ class LBRYUIManager(object):
log.info("UI updates available, checking if installation meets requirements")
return defer.succeed(False)
def _use_existing():
log.info("Failed to check for new ui version, trying to use cached ui")
return defer.succeed(True)
d = _get_git_info()
d.addCallback(_set_git)
d.addCallbacks(_set_git, lambda _: _use_existing)
return d
def migrate_ui(self, source=None):
@@ -127,6 +130,10 @@ class LBRYUIManager(object):
source_dir = source
delete_source = False
def _skip_requirements():
log.info("Skipping ui requirement check")
return defer.succeed(True)
def _check_requirements():
if not os.path.isfile(requires_file):
log.info("No requirements.txt file, rejecting request to migrate this UI")
@@ -197,7 +204,7 @@ class LBRYUIManager(object):
self.loaded_requirements = loaded_ui['requirements']
return defer.succeed(True)
d = _check_requirements()
d = _check_requirements() if self.check_requirements else _skip_requirements()
d.addCallback(lambda r: _do_migrate() if r else _disp_failure())
return d
@@ -223,5 +230,5 @@ class LBRYUIManager(object):
def _load_ui(self):
for d in [i[0] for i in os.walk(self.active_dir) if os.path.dirname(i[0]) == self.active_dir]:
self.root.putChild(os.path.basename(d), static.File(d))
return defer.succeed(True)
self.root.putChild(os.path.basename(d), NoCacheStaticFile(d))
return defer.succeed(True)
Whitespace-only changes.
+48
View File
@@ -0,0 +1,48 @@
import logging
from zope.interface import implementer
from twisted.cred import portal, checkers, credentials, error as cred_error
from twisted.internet import defer
from twisted.web import resource
from lbrynet.lbrynet_daemon.auth.util import load_api_keys
log = logging.getLogger(__name__)
@implementer(portal.IRealm)
class HttpPasswordRealm(object):
def __init__(self, resource):
self.resource = resource
def requestAvatar(self, avatarId, mind, *interfaces):
log.debug("Processing request for %s", avatarId)
if resource.IResource in interfaces:
return (resource.IResource, self.resource, lambda: None)
raise NotImplementedError()
@implementer(checkers.ICredentialsChecker)
class PasswordChecker(object):
credentialInterfaces = (credentials.IUsernamePassword,)
def __init__(self, passwords):
self.passwords = passwords
@classmethod
def load_file(cls, key_path):
keys = load_api_keys(key_path)
return cls.load(keys)
@classmethod
def load(cls, password_dict):
passwords = {key: password_dict[key].secret for key in password_dict}
return cls(passwords)
def requestAvatarId(self, creds):
if creds.username in self.passwords:
pw = self.passwords.get(creds.username)
pw_match = creds.checkPassword(pw)
if pw_match:
return defer.succeed(creds.username)
log.warning('Incorrect username or password')
return defer.fail(cred_error.UnauthorizedLogin('Incorrect username or password'))
+172
View File
@@ -0,0 +1,172 @@
import urlparse
import logging
import requests
import os
import base64
import json
from lbrynet.lbrynet_daemon.auth.util import load_api_keys, APIKey, API_KEY_NAME, get_auth_message
from lbrynet.conf import settings
from jsonrpc.proxy import JSONRPCProxy
log = logging.getLogger(__name__)
USER_AGENT = "AuthServiceProxy/0.1"
TWISTED_SESSION = "TWISTED_SESSION"
LBRY_SECRET = "LBRY_SECRET"
HTTP_TIMEOUT = 30
class JSONRPCException(Exception):
def __init__(self, rpc_error):
Exception.__init__(self)
self.error = rpc_error
class AuthAPIClient(object):
def __init__(self, key, timeout, connection, count, service, cookies, auth, url, login_url):
self.__service_name = service
self.__api_key = key
self.__service_url = login_url
self.__id_count = count
self.__url = url
self.__auth_header = auth
self.__conn = connection
self.__cookies = cookies
def __getattr__(self, name):
if name.startswith('__') and name.endswith('__'):
# Python internal stuff
raise AttributeError
if self.__service_name is not None:
name = "%s.%s" % (self.__service_name, name)
return AuthAPIClient(key=self.__api_key,
timeout=HTTP_TIMEOUT,
connection=self.__conn,
count=self.__id_count,
service=name,
cookies=self.__cookies,
auth=self.__auth_header,
url=self.__url,
login_url=self.__service_url)
def __call__(self, *args):
self.__id_count += 1
pre_auth_postdata = {'version': '1.1',
'method': self.__service_name,
'params': args,
'id': self.__id_count}
to_auth = get_auth_message(pre_auth_postdata)
token = self.__api_key.get_hmac(to_auth)
pre_auth_postdata.update({'hmac': token})
postdata = json.dumps(pre_auth_postdata)
service_url = self.__service_url
auth_header = self.__auth_header
cookies = self.__cookies
host = self.__url.hostname
req = requests.Request(method='POST',
url=service_url,
data=postdata,
headers={'Host': host,
'User-Agent': USER_AGENT,
'Authorization': auth_header,
'Content-type': 'application/json'},
cookies=cookies)
r = req.prepare()
http_response = self.__conn.send(r)
cookies = http_response.cookies
headers = http_response.headers
next_secret = headers.get(LBRY_SECRET, False)
if next_secret:
self.__api_key.secret = next_secret
self.__cookies = cookies
if http_response is None:
raise JSONRPCException({
'code': -342, 'message': 'missing HTTP response from server'})
http_response.raise_for_status()
response = http_response.json()
if response['error'] is not None:
raise JSONRPCException(response['error'])
elif 'result' not in response:
raise JSONRPCException({
'code': -343, 'message': 'missing JSON-RPC result'})
else:
return response['result']
@classmethod
def config(cls, key_name=None, key=None, pw_path=None, timeout=HTTP_TIMEOUT, connection=None, count=0,
service=None, cookies=None, auth=None, url=None, login_url=None):
api_key_name = API_KEY_NAME if not key_name else key_name
pw_path = os.path.join(settings.data_dir, ".api_keys") if not pw_path else pw_path
if not key:
keys = load_api_keys(pw_path)
api_key = keys.get(api_key_name, False)
else:
api_key = APIKey(name=api_key_name, secret=key)
if login_url is None:
service_url = "http://%s:%s@%s:%i/%s" % (api_key_name,
api_key.secret,
settings.API_INTERFACE,
settings.api_port,
settings.API_ADDRESS)
else:
service_url = login_url
id_count = count
if auth is None and connection is None and cookies is None and url is None:
# This is a new client instance, initialize the auth header and start a session
url = urlparse.urlparse(service_url)
(user, passwd) = (url.username, url.password)
try:
user = user.encode('utf8')
except AttributeError:
pass
try:
passwd = passwd.encode('utf8')
except AttributeError:
pass
authpair = user + b':' + passwd
auth_header = b'Basic ' + base64.b64encode(authpair)
conn = requests.Session()
conn.auth = (user, passwd)
req = requests.Request(method='POST',
url=service_url,
auth=conn.auth,
headers={'Host': url.hostname,
'User-Agent': USER_AGENT,
'Authorization': auth_header,
'Content-type': 'application/json'},)
r = req.prepare()
http_response = conn.send(r)
cookies = http_response.cookies
uid = cookies.get(TWISTED_SESSION)
api_key = APIKey.new(seed=uid)
else:
# This is a client that already has a session, use it
auth_header = auth
conn = connection
assert cookies.get(LBRY_SECRET, False), "Missing cookie"
secret = cookies.get(LBRY_SECRET)
api_key = APIKey(secret, api_key_name)
return cls(api_key, timeout, conn, id_count, service, cookies, auth_header, url, service_url)
class LBRYAPIClient(object):
@staticmethod
def config(service=None, params=None):
if settings.use_auth_http:
if service is None:
return AuthAPIClient.config()
log.error("Try auth")
if params is not None:
return AuthAPIClient.config(service=service)(params)
return AuthAPIClient.config(service=service)()
url = settings.API_CONNECTION_STRING
if service is None:
return JSONRPCProxy.from_url(url)
return JSONRPCProxy.from_url(url).call(service, params)
+274
View File
@@ -0,0 +1,274 @@
import logging
from decimal import Decimal
from zope.interface import implements
from twisted.web import server, resource
from twisted.internet import defer
from txjsonrpc import jsonrpclib
from lbrynet.core.Error import InvalidAuthenticationToken, InvalidHeaderError, SubhandlerError
from lbrynet.conf import settings
from lbrynet.lbrynet_daemon.auth.util import APIKey, get_auth_message
from lbrynet.lbrynet_daemon.auth.client import LBRY_SECRET
log = logging.getLogger(__name__)
def default_decimal(obj):
if isinstance(obj, Decimal):
return float(obj)
class AuthorizedBase(object):
def __init__(self):
self.authorized_functions = []
self.subhandlers = []
self.callable_methods = {}
for methodname in dir(self):
if methodname.startswith("jsonrpc_"):
method = getattr(self, methodname)
self.callable_methods.update({methodname.split("jsonrpc_")[1]: method})
if hasattr(method, '_auth_required'):
self.authorized_functions.append(methodname.split("jsonrpc_")[1])
elif not methodname.startswith("__"):
method = getattr(self, methodname)
if hasattr(method, '_subhandler'):
self.subhandlers.append(method)
@staticmethod
def auth_required(f):
f._auth_required = True
return f
@staticmethod
def subhandler(f):
f._subhandler = True
return f
class AuthJSONRPCServer(AuthorizedBase):
"""
Authorized JSONRPC server used as the base class for the LBRY API
API methods are named with a leading "jsonrpc_"
Decorators:
@AuthJSONRPCServer.auth_required: this requires the client include a valid hmac authentication token in their
request
@AuthJSONRPCServer.subhandler: include the tagged method in the processing of requests, to allow inheriting
classes to modify request handling. Tagged methods will be passed the request
object, and return True when finished to indicate success
Attributes:
allowed_during_startup (list): list of api methods that are callable before the server has finished
startup
sessions (dict): dictionary of active session_id: lbrynet.lbrynet_daemon.auth.util.APIKey values
authorized_functions (list): list of api methods that require authentication
subhandlers (list): list of subhandlers
callable_methods (dict): dictionary of api_callable_name: method values
"""
implements(resource.IResource)
isLeaf = True
OK = 200
UNAUTHORIZED = 401
NOT_FOUND = 8001
FAILURE = 8002
def __init__(self, use_authentication=settings.use_auth_http):
AuthorizedBase.__init__(self)
self._use_authentication = use_authentication
self.allowed_during_startup = []
self.sessions = {}
def setup(self):
return NotImplementedError()
def render(self, request):
assert self._check_headers(request), InvalidHeaderError
session = request.getSession()
session_id = session.uid
if self._use_authentication:
# if this is a new session, send a new secret and set the expiration, otherwise, session.touch()
if self._initialize_session(session_id):
def expire_session():
self._unregister_user_session(session_id)
session.startCheckingExpiration()
session.notifyOnExpire(expire_session)
message = "OK"
request.setResponseCode(self.OK)
self._set_headers(request, message, True)
self._render_message(request, message)
return server.NOT_DONE_YET
session.touch()
request.content.seek(0, 0)
content = request.content.read()
try:
parsed = jsonrpclib.loads(content)
except ValueError:
return server.failure
function_name = parsed.get('method')
args = parsed.get('params')
id = parsed.get('id')
token = parsed.pop('hmac', None)
version = self._get_jsonrpc_version(parsed.get('jsonrpc'), id)
try:
self._run_subhandlers(request)
except SubhandlerError:
return server.failure
reply_with_next_secret = False
if self._use_authentication:
if function_name in self.authorized_functions:
try:
self._verify_token(session_id, parsed, token)
except InvalidAuthenticationToken:
log.warning("API validation failed")
request.setResponseCode(self.UNAUTHORIZED)
request.finish()
return server.NOT_DONE_YET
self._update_session_secret(session_id)
reply_with_next_secret = True
try:
function = self._get_jsonrpc_method(function_name)
except Exception:
log.warning("Unknown method: %s", function_name)
return server.failure
d = defer.maybeDeferred(function) if args == [{}] else defer.maybeDeferred(function, *args)
# cancel the response if the connection is broken
notify_finish = request.notifyFinish()
notify_finish.addErrback(self._response_failed, d)
d.addErrback(self._errback_render, id)
d.addCallback(self._callback_render, request, id, version, reply_with_next_secret)
d.addErrback(notify_finish.errback)
return server.NOT_DONE_YET
def _register_user_session(self, session_id):
"""
Add or update a HMAC secret for a session
@param session_id:
@return: secret
"""
log.info("Register api session")
token = APIKey.new(seed=session_id)
self.sessions.update({session_id: token})
def _unregister_user_session(self, session_id):
log.info("Unregister API session")
del self.sessions[session_id]
def _response_failed(self, err, call):
log.debug(err.getTraceback())
def _set_headers(self, request, data, update_secret=False):
request.setHeader("Access-Control-Allow-Origin", settings.API_INTERFACE)
request.setHeader("Content-Type", "text/json")
request.setHeader("Content-Length", str(len(data)))
if update_secret:
session_id = request.getSession().uid
request.setHeader(LBRY_SECRET, self.sessions.get(session_id).secret)
def _render_message(self, request, message):
request.write(message)
request.finish()
def _check_headers(self, request):
origin = request.getHeader("Origin")
referer = request.getHeader("Referer")
if origin not in [None, settings.ORIGIN]:
log.warning("Attempted api call from %s", origin)
return False
if referer is not None and not referer.startswith(settings.REFERER):
log.warning("Attempted api call from %s", referer)
return False
return True
def _check_function_path(self, function_path):
if function_path not in self.callable_methods:
log.warning("Unknown method: %s", function_path)
return False
if not self.announced_startup:
if function_path not in self.allowed_during_startup:
log.warning("Cannot call %s during startup", function_path)
return False
return True
def _get_jsonrpc_method(self, function_path):
assert self._check_function_path(function_path)
return self.callable_methods.get(function_path)
def _initialize_session(self, session_id):
if not self.sessions.get(session_id, False):
self._register_user_session(session_id)
return True
return False
def _verify_token(self, session_id, message, token):
to_auth = get_auth_message(message)
api_key = self.sessions.get(session_id)
assert api_key.compare_hmac(to_auth, token), InvalidAuthenticationToken
def _update_session_secret(self, session_id):
# log.info("Generating new token for next request")
self.sessions.update({session_id: APIKey.new(name=session_id)})
def _get_jsonrpc_version(self, version=None, id=None):
if version:
version_for_return = int(float(version))
elif id and not version:
version_for_return = jsonrpclib.VERSION_1
else:
version_for_return = jsonrpclib.VERSION_PRE1
return version_for_return
def _run_subhandlers(self, request):
for handler in self.subhandlers:
try:
assert handler(request)
except Exception as err:
log.error(err.message)
raise SubhandlerError
def _callback_render(self, result, request, id, version, auth_required=False):
result_for_return = result if not isinstance(result, dict) else result['result']
if version == jsonrpclib.VERSION_PRE1:
if not isinstance(result, jsonrpclib.Fault):
result_for_return = (result_for_return,)
# Convert the result (python) to JSON-RPC
try:
encoded_message = jsonrpclib.dumps(result_for_return, version=version, default=default_decimal)
self._set_headers(request, encoded_message, auth_required)
self._render_message(request, encoded_message)
except:
fault = jsonrpclib.Fault(self.FAILURE, "can't serialize output")
encoded_message = jsonrpclib.dumps(fault, version=version)
self._set_headers(request, encoded_message)
self._render_message(request, encoded_message)
def _errback_render(self, failure, id):
log.error("Request failed:")
log.error(failure)
log.error(failure.value)
log.error(id)
if isinstance(failure.value, jsonrpclib.Fault):
return failure.value
return server.failure
def _render_response(self, result, code):
return defer.succeed({'result': result, 'code': code})
+93
View File
@@ -0,0 +1,93 @@
import base58
import hmac
import hashlib
import yaml
import os
import logging
log = logging.getLogger(__name__)
API_KEY_NAME = "api"
def sha(x):
h = hashlib.sha256(x).digest()
return base58.b58encode(h)
def generate_key(x=None):
if x is None:
return sha(os.urandom(256))
else:
return sha(x)
class APIKey(object):
def __init__(self, secret, name, expiration=None):
self.secret = secret
self.name = name
self.expiration = expiration
@classmethod
def new(cls, seed=None, name=None, expiration=None):
secret = generate_key(seed)
key_name = name if name else sha(secret)
return APIKey(secret, key_name, expiration)
def _raw_key(self):
return base58.b58decode(self.secret)
def get_hmac(self, message):
decoded_key = self._raw_key()
signature = hmac.new(decoded_key, message, hashlib.sha256)
return base58.b58encode(signature.digest())
def compare_hmac(self, message, token):
decoded_token = base58.b58decode(token)
target = base58.b58decode(self.get_hmac(message))
try:
assert len(decoded_token) == len(target), "Length mismatch"
r = hmac.compare_digest(decoded_token, target)
except:
return False
return r
def load_api_keys(path):
if not os.path.isfile(path):
raise Exception("Invalid api key path")
with open(path, "r") as f:
data = yaml.load(f.read())
keys_for_return = {}
for key_name in data:
key = data[key_name]
secret = key['secret']
expiration = key['expiration']
keys_for_return.update({key_name: APIKey(secret, key_name, expiration)})
return keys_for_return
def save_api_keys(keys, path):
with open(path, "w") as f:
key_dict = {keys[key_name].name: {'secret': keys[key_name].secret,
'expiration': keys[key_name].expiration}
for key_name in keys}
data = yaml.safe_dump(key_dict)
f.write(data)
def initialize_api_key_file(key_path):
if not os.path.isfile(key_path):
keys = {}
new_api_key = APIKey.new(name=API_KEY_NAME)
keys.update({new_api_key.name: new_api_key})
save_api_keys(keys, key_path)
def get_auth_message(message_dict):
to_auth = message_dict.get('method').encode('hex')
to_auth += str(message_dict.get('id')).encode('hex')
return to_auth.decode('hex')
-352
View File
@@ -1,352 +0,0 @@
import Tkinter as tk
import logging
import sys
import tkFont
import tkMessageBox
import ttk
from lbrynet.lbrynet_gui.LBRYGui import LBRYDownloader
from lbrynet.lbrynet_gui.StreamFrame import StreamFrame
import locale
import os
from twisted.internet import defer, reactor, tksupport, task
log = logging.getLogger(__name__)
class DownloaderApp(object):
def __init__(self):
self.master = None
self.downloader = None
self.wallet_balance_check = None
self.streams_frame = None
def start(self):
d = defer.maybeDeferred(self._start_root)
d.addCallback(lambda _: self._draw_main())
d.addCallback(lambda _: self._start_downloader())
d.addCallback(lambda _: self._start_checking_wallet_balance())
d.addCallback(lambda _: self._enable_lookup())
def show_error_and_stop(err):
log.error(err.getErrorMessage())
tkMessageBox.showerror(title="Start Error", message=err.getErrorMessage())
return self.stop()
d.addErrback(show_error_and_stop)
return d
def stop(self):
def log_error(err):
log.error(err.getErrorMessage())
if self.downloader is not None:
d = self.downloader.stop()
else:
d = defer.succeed(True)
d.addErrback(log_error)
d.addCallback(lambda _: self._stop_checking_wallet_balance())
d.addErrback(log_error)
d.addCallback(lambda _: reactor.stop())
d.addErrback(log_error)
return d
def _start_root(self):
if os.name == "nt":
button_foreground = "#104639"
lookup_button_padding = 10
else:
button_foreground = "#FFFFFF"
lookup_button_padding = 11
root = tk.Tk()
root.resizable(0, 0)
root.wm_title("LBRY")
tksupport.install(root)
if os.name == "nt":
root.iconbitmap(os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])),
"lbry-dark-icon.ico"))
else:
root.wm_iconbitmap("@" + os.path.join(os.path.dirname(__file__), "lbry-dark-icon.xbm"))
root.button_font = tkFont.Font(size=9)
ttk.Style().configure(".", background="#FFFFFF")
ttk.Style().configure("LBRY.TButton", background="#104639", foreground=button_foreground,
borderwidth=1, relief="solid", font=root.button_font)
ttk.Style().map("LBRY.TButton",
background=[('pressed', "#104639"),
('active', "#104639")])
#ttk.Style().configure("LBRY.TButton.border", background="#808080")
ttk.Style().configure("Lookup.LBRY.TButton", padding=lookup_button_padding)
ttk.Style().configure("Stop.TButton", padding=1, background="#FFFFFF", relief="flat", borderwidth=0)
ttk.Style().configure("TEntry", padding=11)
ttk.Style().configure("Float.TEntry", padding=2)
#ttk.Style().configure("A.TFrame", background="red")
#ttk.Style().configure("B.TFrame", background="green")
#ttk.Style().configure("B2.TFrame", background="#80FF80")
#ttk.Style().configure("C.TFrame", background="orange")
#ttk.Style().configure("D.TFrame", background="blue")
#ttk.Style().configure("E.TFrame", background="yellow")
#ttk.Style().configure("F.TFrame", background="#808080")
#ttk.Style().configure("G.TFrame", background="#FF80FF")
#ttk.Style().configure("H.TFrame", background="#0080FF")
#ttk.Style().configure("LBRY.TProgressbar", background="#104639", orient="horizontal", thickness=5)
#ttk.Style().configure("LBRY.TProgressbar")
#ttk.Style().layout("Horizontal.LBRY.TProgressbar", ttk.Style().layout("Horizontal.TProgressbar"))
root.configure(background="#FFFFFF")
root.protocol("WM_DELETE_WINDOW", self.stop)
self.master = root
def _draw_main(self):
self.frame = ttk.Frame(self.master, style="A.TFrame")
self.frame.grid(padx=20, pady=20)
logo_file_name = "lbry-dark-242x80.gif"
if os.name == "nt":
logo_file = os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), logo_file_name)
else:
logo_file = os.path.join(os.path.dirname(__file__), logo_file_name)
self.logo_picture = tk.PhotoImage(file=logo_file)
self.logo_frame = ttk.Frame(self.frame, style="B.TFrame")
self.logo_frame.grid(pady=5, sticky=tk.W + tk.E)
self.dummy_frame = ttk.Frame(self.logo_frame, style="C.TFrame") # keeps the logo in the middle
self.dummy_frame.grid(row=0, column=1, padx=5)
self.logo_label = ttk.Label(self.logo_frame, image=self.logo_picture)
self.logo_label.grid(row=0, column=1, padx=5)
self.wallet_balance_frame = ttk.Frame(self.logo_frame, style="C.TFrame")
self.wallet_balance_frame.grid(sticky=tk.E + tk.N, row=0, column=2)
self.logo_frame.grid_columnconfigure(0, weight=1, uniform="a")
self.logo_frame.grid_columnconfigure(1, weight=2, uniform="b")
self.logo_frame.grid_columnconfigure(2, weight=1, uniform="a")
self.wallet_balance = ttk.Label(
self.wallet_balance_frame,
text=" -- LBC"
)
self.wallet_balance.grid(row=0, column=0)
dropdown_file_name = "drop_down.gif"
if os.name == "nt":
dropdown_file = os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])),
dropdown_file_name)
else:
dropdown_file = os.path.join(os.path.dirname(__file__), dropdown_file_name)
self.dropdown_picture = tk.PhotoImage(
file=dropdown_file
)
def get_new_address():
def show_address(address):
w = AddressWindow(self.master, address)
w.show()
d = defer.maybeDeferred(self.downloader.get_new_address)
d.addCallback(show_address)
def show_error(err):
tkMessageBox.showerror(title="Failed to get new address", message=err.getErrorMessage())
d.addErrback(show_error)
self.wallet_menu = tk.Menu(
self.master, tearoff=0
)
self.wallet_menu.add_command(label="Get new LBRYcrd address", command=get_new_address)
if os.name == "nt":
button_cursor = ""
else:
button_cursor = "hand1"
self.wallet_menu_button = ttk.Button(self.wallet_balance_frame, image=self.dropdown_picture,
style="Stop.TButton", cursor=button_cursor)
self.wallet_menu_button.grid(row=0, column=1, padx=(5, 0))
def popup_wallet(event):
self.wallet_menu.tk_popup(event.x_root, event.y_root)
self.wallet_menu_button.bind("<Button-1>", popup_wallet)
self.uri_frame = ttk.Frame(self.frame, style="B.TFrame")
self.uri_frame.grid()
self.uri_label = ttk.Label(
self.uri_frame, text="lbry://"
)
self.uri_label.grid(row=0, column=0, sticky=tk.E, pady=2)
self.entry_font = tkFont.Font(size=11)
self.uri_entry = ttk.Entry(self.uri_frame, width=50, foreground="#222222", font=self.entry_font,
state=tk.DISABLED)
self.uri_entry.grid(row=0, column=1, padx=2, pady=2)
def copy_command():
self.uri_entry.event_generate('<Control-c>')
def cut_command():
self.uri_entry.event_generate('<Control-x>')
def paste_command():
self.uri_entry.event_generate('<Control-v>')
def popup_uri(event):
selection_menu = tk.Menu(
self.master, tearoff=0
)
if self.uri_entry.selection_present():
selection_menu.add_command(label=" Cut ", command=cut_command)
selection_menu.add_command(label=" Copy ", command=copy_command)
selection_menu.add_command(label=" Paste ", command=paste_command)
selection_menu.tk_popup(event.x_root, event.y_root)
self.uri_entry.bind("<Button-3>", popup_uri)
self.uri_button = ttk.Button(
self.uri_frame, text="Go", command=self._open_stream,
style='Lookup.LBRY.TButton', cursor=button_cursor
)
self.uri_button.grid(row=0, column=2, pady=2, padx=0)
def _start_downloader(self):
self.downloader = LBRYDownloader()
d = self.downloader.start()
d.addCallback(lambda _: self.downloader.check_first_run())
d.addCallback(self._show_welcome_message)
return d
def _show_welcome_message(self, points_sent):
if points_sent != 0.0:
w = WelcomeWindow(self.master, points_sent)
w.show()
def stream_removed(self):
if self.streams_frame is not None:
if len(self.streams_frame.winfo_children()) == 0:
self.streams_frame.destroy()
self.streams_frame = None
def _start_checking_wallet_balance(self):
def set_balance(balance):
self.wallet_balance.configure(text=locale.format_string("%.2f LBC", (round(balance, 2),),
grouping=True))
def update_balance():
balance = self.downloader.session.wallet.get_available_balance()
set_balance(balance)
def start_looping_call():
self.wallet_balance_check = task.LoopingCall(update_balance)
self.wallet_balance_check.start(5)
d = self.downloader.session.wallet.get_balance()
d.addCallback(set_balance)
d.addCallback(lambda _: start_looping_call())
def _stop_checking_wallet_balance(self):
if self.wallet_balance_check is not None:
self.wallet_balance_check.stop()
def _enable_lookup(self):
self.uri_entry.bind('<Return>', self._open_stream)
self.uri_entry.config(state=tk.NORMAL)
def _open_stream(self, event=None):
if self.streams_frame is None:
self.streams_frame = ttk.Frame(self.frame, style="B2.TFrame")
self.streams_frame.grid(sticky=tk.E + tk.W)
uri = self.uri_entry.get()
self.uri_entry.delete(0, tk.END)
stream_frame = StreamFrame(self, "lbry://" + uri)
self.downloader.download_stream(stream_frame, uri)
class WelcomeWindow(object):
def __init__(self, root, points_sent):
self.root = root
self.points_sent = points_sent
def show(self):
window = tk.Toplevel(self.root, background="#FFFFFF")
window.transient(self.root)
window.wm_title("Welcome to LBRY")
window.protocol("WM_DELETE_WINDOW", window.destroy)
window.resizable(0, 0)
text_box = tk.Text(window, width=45, height=3, relief=tk.FLAT, borderwidth=0,
highlightthickness=0)
points_string = locale.format_string("%.2f LBC", (round(self.points_sent, 2),),
grouping=True)
text_box.insert(tk.END, "Thank you for using LBRY! You have been\n"
"given %s for free because we love\n"
"you. Please give them 60 seconds to show up." % points_string)
text_box.grid(row=0, padx=10, pady=5, columnspan=2)
text_box.config(state='normal')
window.focus_set()
class AddressWindow(object):
def __init__(self, root, address):
self.root = root
self.address = address
def show(self):
window = tk.Toplevel(self.root, background="#FFFFFF")
window.transient(self.root)
window.wm_title("New address")
window.protocol("WM_DELETE_WINDOW", window.destroy)
window.resizable(0, 0)
text_box = tk.Text(window, width=35, height=1, relief=tk.FLAT, borderwidth=0,
highlightthickness=0)
text_box.insert(tk.END, self.address)
text_box.grid(row=0, padx=10, pady=5, columnspan=2)
text_box.config(state='normal')
def copy_to_clipboard():
self.root.clipboard_clear()
self.root.clipboard_append(text_box.get('1.0', 'end-1c'))
def copy_command():
text_box.event_generate("<Control-c>")
copy_menu = tk.Menu(
self.root, tearoff=0
)
copy_menu.add_command(label=" Copy ", command=copy_command)
def popup(event):
if text_box.tag_ranges("sel"):
copy_menu.tk_popup(event.x_root, event.y_root)
text_box.bind("<Button-3>", popup)
copy_button = ttk.Button(
window, text="Copy", command=copy_to_clipboard, style="LBRY.TButton"
)
copy_button.grid(row=1, column=0, pady=(0, 5), padx=5, sticky=tk.E)
done_button = ttk.Button(
window, text="OK", command=window.destroy, style="LBRY.TButton"
)
done_button.grid(row=1, column=1, pady=(0, 5), padx=5, sticky=tk.W)
window.focus_set()
Loaded 100 of 201 files, more files were not shown because too many files have changed in this diff. Show more