Compare commits

..
Author SHA1 Message Date
hackrush 3e2585e294 CHANGELOG update 2018-07-10 09:17:09 +05:30
hackrush a182bae070 Only stop started components and test fix. 2018-07-10 08:56:29 +05:30
hackrush 4e6e23946e P-Y-L-I-N-T #angry_emoji 2018-07-10 08:36:26 +05:30
hackrush c472dfd70e Now components can be skipped by setting relevant config in file. 2018-07-10 08:36:26 +05:30
hackrush d6c7f414bc Using constants instead of strings. Added CHANGELOG.md 2018-07-10 08:36:26 +05:30
hackrush ec7562e052 Fixed passing mutable args in init 2018-07-10 08:36:26 +05:30
hackrush 5e3257b47a Fix Daemon Tests 2018-07-10 08:36:26 +05:30
hackrush bffa6f767e Tests for component manager 2018-07-10 08:36:26 +05:30
hackrush d28d5ac684 Started writing tests 2018-07-10 08:36:26 +05:30
hackrush 7791ee3ce3 Rebased on amster and other crazy stuff 2018-07-10 08:36:26 +05:30
hackrush 91a44e6a27 wallet unlock condition checks, fixed breaking changes 2018-07-10 08:36:26 +05:30
hackrush fd310b6918 Reworded errors 2018-07-10 08:36:26 +05:30
hackrush 36416479a0 Sanitization of modules 2018-07-10 08:36:26 +05:30
hackrush 2442f714aa Fixed new broken tests from daemon refactor 2018-07-10 08:36:26 +05:30
hackrush b694a2c416 import fixes and removed temporary test function 2018-07-10 08:36:26 +05:30
hackrush 3ea9caf54e Decorator made and decorating of functions done(some other changes) 2018-07-10 08:36:18 +05:30
hackrush ee4e1e6b50 Making decorators(WIP, commit so that I don't lose work) 2018-07-10 08:33:56 +05:30
hackrush 51c5bcb31f Oops(That's all you're gonna get :-P) 2018-07-10 08:33:56 +05:30
hackrush 9ffc5cf259 Pylint fixes 2018-07-10 08:33:56 +05:30
hackrush 7a9d3a7167 All tests now passing 2018-07-10 08:33:52 +05:30
hackrush d869639734 Fixed test_streamify(well Jack did, but ¯\_(ツ)_/¯) 2018-07-10 08:32:48 +05:30
hackrush 1662458a32 Reflector Component 2018-07-10 08:32:48 +05:30
hackrush 3054ad918b Query Handler and server components 2018-07-10 08:32:48 +05:30
hackrush a21ccec8e1 Hash announcer, file manager, stream identifier components 2018-07-10 08:32:48 +05:30
Jack Robison 92ccd47008 refactor ComponentManager and Component to use instance instead of class methods 2018-07-10 08:32:48 +05:30
hackrush 0e0ea93e39 Fixed regression caused by removing peer_manager from session 2018-07-10 08:32:48 +05:30
hackrush c3f0eb917f Fixed test_streamify test 2018-07-10 08:32:48 +05:30
hackrush cd0d698529 DHT is now it's own component 2018-07-10 08:32:48 +05:30
hackrush 08514c0914 Refactored classes to reduce redundancy in getting config setting 2018-07-10 08:32:48 +05:30
hackrush ce10347c02 Making DHT component(broken) 2018-07-10 08:32:48 +05:30
hackrush f70fb3f8d6 Starting to convert session to its own component. Removed ref to self.storage from Daemon.py 2018-07-10 08:32:48 +05:30
hackrush a18ee8214a Remove create_session internal function and PEP8 2018-07-10 08:32:48 +05:30
hackrush 077e98aa3c Use storage from session. 2018-07-10 08:32:48 +05:30
hackrush 7b2ce1b5bf convert wallet to be Component 2018-07-10 08:32:48 +05:30
hackrush 8635ba51b2 Fixed typo in ComponentManager 2018-07-10 08:32:48 +05:30
Jack Robison 3174d07556 support callbacks to component setups 2018-07-10 08:32:48 +05:30
Jack Robison 8fb833ce80 convert directory and SQLiteStorage setup to be a Component 2018-07-10 08:32:48 +05:30
Jack Robison b44c9b6704 add daemon Component and ComponentManager classes 2018-07-10 08:32:48 +05:30
339 changed files with 10648 additions and 19999 deletions

No files matched your search

+48
View File
@@ -0,0 +1,48 @@
version: 1.0.{build}
branches:
only:
- master
environment:
GH_TOKEN:
secure: LiI5jyuHUw6XbH4kC3gP1HX4P/v4rwD/gCNtaFhQu2AvJz1/1wALkp5ECnIxRySN
key_pass:
secure: u6DydPcdrUJlxGL9uc7yQRYG8+5rY6aAEE9nfCSzFyNzZlX9NniOp8Uh5ZKQqX7bGEngLI6ipbLfiJvn0XFnhbn2iTkOuMqOXVJVOehvwlQ=
pfx_key:
secure: 1mwqyRy7hDqDjDK+TIAoaXyXzpNgwruFNA6TPkinUcVM7A+NLD33RQLnfnwVy+R5ovD2pUfhQ6+N0Fqebv6tZh436LIEsock+6IOdpgFwrg=
AWS_ACCESS_KEY_ID:
secure: iVGwoJ7ogspjSmuqr+haVPLglSgQsp6tUZx6mIlKH7Q=
AWS_SECRET_ACCESS_KEY:
secure: zKaqdZGPl0exDL5YhJkb33prSemC9Rzg9S7Lw2wFy1WnJ6ffgl6mQH7jqJDUTqsY
#notifications:
# - provider: Slack
# incoming_webhook:
# secure: LuxwG5OZnnA//gmSXzCKu8/FRqYjCgGfVFqajSsGHeQ1HQNp7rYNhQpsO8/3PK63xKJj3wzt86DJekf9q9Q5OcHa9AHXUQbEveX0psd7elw=
clone_folder: c:\projects\lbry
test_script:
- cd C:\projects\lbry\
- pip install cython
- pip install mock pylint unqlite Faker
- pip install .
- pylint lbrynet
# disable tests for now so that appveyor can build the app
- set PYTHONPATH=.
- trial lbrynet.tests
# TODO: integration tests do not work
#- python -m unittest discover tests/integration
#- rvm use 2.3.1 && gem install danger --version '~> 4.0' && danger
build_script:
- cd C:\projects\lbry\build\
- ps: .\build.ps1
artifacts:
- path: build\dist\*.zip
name: lbrynet-daemon
+1 -22
View File
@@ -4,7 +4,7 @@ Thanks for reporting an issue to LBRY and helping us improve!
To make it possible for us to help you, please fill out below information carefully.
Before reporting any issues, please make sure that you're using the latest version.
- App: https://github.com/lbryio/lbry-desktop/releases
- App: https://github.com/lbryio/lbry-app/releases
- Daemon: https://github.com/lbryio/lbry/releases
We are also available on Discord at https://chat.lbry.io
@@ -51,24 +51,3 @@ Tell us what happens instead
## Screenshots
<!-- If a screenshot would help explain the bug, please include one or two here -->
## Internal Use
### Acceptance Criteria
1.
2.
3.
### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed
-38
View File
@@ -1,38 +0,0 @@
## PR Checklist
Please check all that apply to this PR using "x":
- [ ] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
- [ ] I have checked that this PR does not introduce a breaking change
- [ ] This PR introduces breaking changes and I have provided a detailed explanation below
## PR Type
What kind of change does this PR introduce?
Why is this change necessary?
<!-- Please check all that apply to this PR using "x". -->
- [ ] Bugfix
- [ ] Feature
- [ ] Breaking changes (bugfix or feature that introduces breaking changes)
- [ ] Code style update (formatting)
- [ ] Refactoring (no functional changes)
- [ ] Documentation changes
- [ ] Other - Please describe:
## Fixes
Issue Number: N/A
## What is the current behavior?
## What is the new behavior?
## Other information
<!-- If this PR contains a breaking change, please describe the impact and solution strategy for existing applications below. -->
+23 -7
View File
@@ -1,9 +1,25 @@
/build
/dist
/.tox
/.idea
/.coverage
*.pyc
*.egg
*.so
*.xml
*.iml
*.log
*.pem
*.decTest
*.prof
.#*
lbrynet.egg-info
__pycache__
/build/build
/build/dist
/bulid/requirements_base.txt
/lbrynet.egg-info
/docs_build
/lbry-venv
.idea/
.coverage
.DS_Store
# temporary files from the twisted.trial test runner
_trial_temp/
+5 -9
View File
@@ -9,7 +9,7 @@
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=CVS,schema
ignore=CVS
# Add files or directories matching the regex patterns to the
# blacklist. The regex matches against base names, not paths.
@@ -62,7 +62,7 @@ confidence=
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then re-enable specific checks. For example, if
# disable everything first and then reenable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
@@ -121,11 +121,7 @@ disable=
unidiomatic-typecheck,
global-at-module-level,
inconsistent-return-statements,
keyword-arg-before-vararg,
assignment-from-no-return,
useless-return,
assignment-from-none,
stop-iteration-return
keyword-arg-before-vararg
[REPORTS]
@@ -390,7 +386,7 @@ int-import-graph=
[DESIGN]
# Maximum number of arguments for function / method
max-args=10
max-args=5
# Argument names that match this expression will be ignored. Default to name
# with leading underscore
@@ -409,7 +405,7 @@ max-branches=12
max-statements=50
# Maximum number of parents for a class (see R0901).
max-parents=8
max-parents=7
# Maximum number of attributes for a class (see R0902).
max-attributes=7
+38 -120
View File
@@ -1,128 +1,46 @@
sudo: required
dist: xenial
os: linux
dist: trusty
language: python
python: "3.7"
python: 2.7
jobs:
include:
branches:
except:
- gh-pages
- stage: code quality
name: "pylint lbrynet"
install:
- pip install astroid==2.0.4
# newer astroid fails in pylint so we pre-install older version
- pip install pylint
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .
script: pylint lbrynet
- &tests
stage: test
name: "Unit Tests"
install:
- pip install coverage
- pip install git+https://github.com/lbryio/torba.git#egg=torba
- pip install -e .[test]
script:
- HOME=/tmp coverage run -p --source=lbrynet -m unittest discover -v tests.unit.wallet
- HOME=/tmp coverage run -p --source=lbrynet -m twisted.trial --reactor=asyncio tests.unit.analytics tests.unit.components tests.unit.core tests.unit.cryptstream tests.unit.database tests.unit.dht tests.unit.lbryfilemanager tests.unit.lbrynet_daemon tests.unit.test_cli tests.unit.test_customLogger tests.unit.schema
after_success:
- coverage combine
- bash <(curl -s https://codecov.io/bash)
- <<: *tests
name: "DHT Tests"
script: HOME=/tmp coverage run --source=lbrynet -m twisted.trial --reactor=asyncio tests.functional
- name: "Integration Tests"
install:
- pip install tox-travis coverage
- pushd .. && git clone https://github.com/lbryio/torba.git && popd
script: tox
after_success:
- coverage combine tests/
- bash <(curl -s https://codecov.io/bash)
- stage: build
name: "Windows"
language: generic
services:
- docker
install:
- docker pull lbry/pyinstaller34_32bits:py371
script:
- python scripts/set_build.py
- docker run -v "$(pwd):/src/lbry" lbry/pyinstaller34_32bits:py371 lbry/scripts/wine_build.sh
- sudo zip -j dist/lbrynet-windows.zip dist/lbrynet.exe
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-windows.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-windows.zip
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- &build
name: "Linux"
env: OS=linux
install:
- pip3 install pyinstaller
- pip3 install git+https://github.com/lbryio/torba.git
- python scripts/set_build.py
- pip3 install -e .
script:
- pyinstaller -F -n lbrynet lbrynet/extras/cli.py
- chmod +x dist/lbrynet
- zip -j dist/lbrynet-${OS}.zip dist/lbrynet
- ./dist/lbrynet --version
deploy:
provider: releases
api_key: $GITHUB_OAUTH_TOKEN
file: dist/lbrynet-${OS}.zip
skip_cleanup: true
overwrite: true
draft: true
on:
tags: true
addons:
artifacts:
working_dir: dist
paths:
- lbrynet-${OS}.zip
# artifact uploader thinks lbrynet is a directory, https://github.com/travis-ci/artifacts/issues/78
target_paths:
- /daemon/build-${TRAVIS_BUILD_NUMBER}_commit-${TRAVIS_COMMIT:0:7}_branch-${TRAVIS_BRANCH}$([ ! -z ${TRAVIS_TAG} ] && echo _tag-${TRAVIS_TAG})
- <<: *build
name: "Mac"
os: osx
osx_image: xcode8.3
language: generic
env: OS=mac
cache: false
before_install:
- brew upgrade python
install:
- python3 --version
- pip3 --version
- pip3 install pyinstaller
- git clone https://github.com/lbryio/torba.git --depth 1
- sed -i -e "s/'plyvel',//" torba/setup.py
- cd torba && pip3 install -e . && cd ..
- python3 scripts/set_build.py
- pip3 install -e .
notifications:
slack:
secure: "Am13HPtpgCMljh0MDVuoFHvQXB8yhf4Kvf/qAeSp5N0vsHGL70CSF9Ahccw8dVPE6mbuak1OGtSUb6/UaErLHkpz3ztaRLkDa9x7CmBB3Kynnh8oO2VbB7b/2ROULqkhF4VZmAnNfwrQrbC3gs8Sybp261Nyc7y4ww15xDYBrk2fyq4ds2DCaJdRxfJUJFonrZ6KXr3fVaXosO6cjuyS8eRodcmrqsT4cCtinjNTD1hGWoH107E4ObSmpVelxQO193KhNJMRiLlEcVkvYUOqIWBtwdGHbNE/6Yeuq1TXgKJ0KeJWAmW3wTfUYNngGXNAsyCnrhul5TKNevNzfIAQZHvRsczYiWPJV6LtohHT0CcUiCXJtvEPOyahEBfwK3etY/xxFqny7N9OEmpdW2sgsEPNPX2LJynJti2rQA9SuAD1ogR3ZpDy/NXoaAZf8PTdPcuNUMULV9PGG7tVrLBecO/W1qO6hdFxwlLdgqGLxAENZgGp++v/DhPk/WvtmHj7iTbRq0nxaTWyX5uKOn2ADH+k/yfutjv6BsQU9xNyPeZEEtuEpc6X6waiYn/8G9vl9PecvKC5H0MgsZ6asAxmg7mZ3VSMFG7mo8ENeOhSZ0Oz6ZTBILL3wFccZA9uJIq7NWmqC9dRiGiuKXBB62No7sINoHg3114e2xYa9qvNmGg="
cache:
directories:
- $HOME/.cache/pip
- $HOME/Library/Caches/pip
- $TRAVIS_BUILD_DIR/.tox
- $TRAVIS_BUILD_DIR/cache/wheel
addons:
#srcclr:
# debug: true
apt:
packages:
- libgmp3-dev
- build-essential
- git
- libssl-dev
- libffi-dev
before_install:
- virtualenv venv
- source venv/bin/activate
install:
- pip install -U pip==9.0.3
- pip install -r requirements.txt
- pip install -r requirements_testing.txt
- pip install .
script:
- pip install mock pylint
- pylint lbrynet
- PYTHONPATH=. trial lbrynet.tests
- rvm install ruby-2.3.1
- rvm use 2.3.1 && gem install danger --version '~> 4.0' && danger
+20 -183
View File
@@ -4,198 +4,35 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/) with
regard to the json-rpc api. As we're currently pre-1.0 release, we
can and probably will change functionality and break backwards compatibility
can and probably will change functionality and break backwards compatability
at anytime.
## [0.30.1] - 2018-12-09
### API
* add `aioupnp_version` to `upnp` field in the response to `status`
* fixed uncaught claim signature errors
* `--blocking` support for `abandon` command (https://github.com/lbryio/lbry/pull/1618)
* don't create SSL keys if SSL is disabled (https://github.com/lbryio/lbry/pull/1644)
* added `account_send` command (https://github.com/lbryio/lbry/pull/1654)
* fixes for claim send to address (https://github.com/lbryio/lbry/pull/1651)
* better handle errors on resolve (https://github.com/lbryio/lbry/pull/1662)
* fix `transaction_show` (https://github.com/lbryio/lbry/pull/1689)
### Schema
* merged `lbryschema` repository into `lbrynet` (https://github.com/lbryio/lbry/pull/1444)
* updated `protobuf` to `3.2.0` (https://github.com/lbryio/lbry/issues/1602)
* use `types` repo for generating python protobufs (https://github.com/lbryio/lbry/pull/1605)
### Wallet
* fixed resolving names with invalid channel signatures
* fixed wallet sync issues
* fixed wallet locking/encryption issues
### DHT
* fixed retrying kademlia rpc calls (https://github.com/lbryio/lbry/pull/1495)
* fixed `KeyError` in `iterativeFindValue`, this is used by `peer_list` and when finding peers to download blobs from (https://github.com/lbryio/lbry/issues/1638)
* settings external ip address and ability to disable more DHT components (https://github.com/lbryio/lbry/pull/1604)
* improved routing table bucket initialization (https://github.com/lbryio/lbry/pull/1633)
### Database
* resolved too many variables in sqlite issue (https://github.com/lbryio/lbry/pull/1612)
### Misc
* switch to Python 3.7
* brought back icon for windows binary
* using custom list of mime types with more types (https://github.com/lbryio/lbry/pull/1700)
* add balance to startup report (https://github.com/lbryio/lbry/pull/1613)
* drop `six` dependency (https://github.com/lbryio/lbry/pull/1660)
* drop `mock` dependency (https://github.com/lbryio/lbry/pull/1661)
## [0.30.0] - 2018-10-22
This release is the most ambitious upgrade to lbrynet yet; including a brand
new wallet implementation, switch to Python 3 of the entire code base and
many changes to the API such as pagination and a more consistent output from
most commands.
## [Unreleased]
### Security
* upgraded `cryptography` package.
* switch to `coincurve` for private / public key generation (addresses)
* support for connecting to daemon via SSL
* improved workflow for encrypt/decrypt of wallet
*
*
### API
* unified all command line executables into a single `lbrynet` executable.
* added `use_https` configuration setting (off by default), if true the api
server will generate a self signed ssl certificate when started and only
handle api requests on the https address (default of `https://localhost:5279/lbryapi`)
* deprecated `daemon_stop` command, use `stop` instead.
* deprecated `wallet_balance` command, use `account_balance` instead.
* deprecated `wallet_unlock` command, use `account_unlock` instead.
* deprecated `wallet_decrypt` command, use `account_decrypt` instead.
* deprecated `wallet_encrypt` command, use `account_encrypt` instead.
* deprecated `wallet_prefill_addresses` command, use `account_fund` instead.
* deprecated `wallet_list` command, use `address_list` instead.
* deprecated `wallet_is_address_mine` command, use `address_is_mine` instead.
* deprecated `wallet_public_key` command.
* deprecated `wallet_new_address` command.
* deprecated `wallet_unused_address` command, use `address_unused` instead.
* deprecated `claim_renew` command.
* added `account_list` command to list accounts including their balance.
* added `account_add` command to add a previously created account from seed or private key.
* added `account_create` command to generate a new account.
* added `account_remove` command to remove an account from wallet.
* added `account_set` command to change a setting on an account.
* added `account_balance` command to get just the account balance.
* added `account_unlock` command to unlock an encrypted account.
* added `account_lock` command to lock an encrypted account.
* added `account_encrypt` command to encrypt an account.
* added `account_decrypt` command to decrypt an account.
* added `account_fund` command to move funds between or within an account in various ways.
* added `account_max_address_gap` command to find large gaps of unused addresses.
* added `address_list` command to list addresses.
* added `address_is_mine` command to check if an address is one of your addresses.
* added `address_unused` command to get existing or generate a new unused address.
* added pagination support for `address_list`, `channel_list`, `claim_list_mine`,
`transaction_list` and `utxo_list`.
* added `upnp` field to `status` response
* removed `send_amount_to_address` command previously marked as deprecated
* removed `channel_list_mine` command previously marked as deprecated
* removed `get_availability` command previously marked as deprecated
* updated `txupnp` dependency to `aioupnp` (UPnP for asyncio)
### Wallet
* changed to a new wallet implementation: [torba](https://github.com/lbryio/torba).
* changed wallet file format to support multiple accounts in one wallet.
* moved transaction data from wallet file into an sqlite database.
* changed channel certificates to be keyed by txid:nout instead of claim_id which
makes it possible to recover old certificates.
### File Manager
* Extensive internal changes as a result of porting to Python 3.
### DHT
* Extensive internal changes as a result of porting to Python 3.
### P2P
* Extensive internal changes as a result of porting to Python 3.
### Reflector
* No major changes to reflector.
### Database
* No changes to existing storage module.
### Known Issues
* Channel certificate import/export is currently not working.
## [0.21.2] - 2018-08-23
### Fixed
* issue in dht ping queue where enqueued pings that aren't yet due wouldn't be rescheduled
* blob mirror downloader not finishing streams that were partially uploaded at the time of the download attempt (https://github.com/lbryio/lbry/issues/1376)
## [0.21.1] - 2018-08-13
### Fixed
* `download_progress` field in `blockchain_headers` section of `status` not initializing correctly when resuming a download (https://github.com/lbryio/lbry/issues/1355)
* `wallet_send` not accepting decimal amounts (https://github.com/lbryio/lbry/issues/1356 commit https://github.com/lbryio/lbry/commit/1098ca0494ece420c70fc57f69d6d388715a99b8)
### Added
* `is_locked` to `wallet` in `status` response (https://github.com/lbryio/lbry/issues/1354, commit https://github.com/lbryio/lbry/commit/153022a1a7122ab6d31d3db433dccbe469bbcb3c)
### Changed
* Bumped `lbryum` requirement to 3.2.4 [see changelog](https://github.com/lbryio/lbryum/blob/master/CHANGELOG.md#324---2018-08-13)
## [0.21.0] - 2018-08-09
### Fixed
* check `claim_address` and `change_address` earlier on publishing, to avoid hard to understand errors later in the process (pr https://github.com/lbryio/lbry/pull/1347)
* loggly error reporting not following `share_usage_data` (pr https://github.com/lbryio/lbry/pull/1328)
* improper error handling when data is not valid JSON (pr https://github.com/lbryio/lbry/pull/1326)
* blob mirroring being set in unrelated tests, making them fail (pr https://github.com/lbryio/lbry/pull/1348)
* http blob mirroring edge cases (pr https://github.com/lbryio/lbry/pull/1315)
* external ports in log messages not showing the correct external port from the upnp redirects (https://github.com/lbryio/lbry/issues/1338) (pr https://github.com/lbryio/lbry/pull/1349)
* miniupnpc fallback issues in txupnp (https://github.com/lbryio/lbry/issues/1341) (pr https://github.com/lbryio/lbry/pull/1349)
* upnp error when disabled on router and a non-gateway is found, such as chromecast (https://github.com/lbryio/lbry/issues/1352) (https://github.com/lbryio/lbry/commit/dca4af942fbe95547794213775f0a62cd04a393f)
* daemon cli spelling fixes
*
### Deprecated
* automatic claim renew, this is no longer needed
*
*
### Changed
* api server class to use components, and for all JSONRPC API commands to be callable so long as the required components are available. (pr https://github.com/lbryio/lbry/pull/1294)
* return error messages when required conditions on components are not met for API calls (pr https://github.com/lbryio/lbry/pull/1328)
* `status` to no longer return a base58 encoded `lbry_id`, instead return this as the hex encoded `node_id` in a new `dht` field. (pr https://github.com/lbryio/lbry/pull/1328)
* `startup_status` field in the response to `status` to be a dict of component names to status booleans (pr https://github.com/lbryio/lbry/pull/1328)
* renamed the `blockchain_status` field in the response to `status` to `wallet` (pr https://github.com/lbryio/lbry/pull/1328)
* moved and renamed `wallet_is_encrypted` to `is_encrypted` in the `wallet` field in the response to `status` (pr https://github.com/lbryio/lbry/pull/1328)
* moved wallet, upnp and dht startup code from `Session` to `Components` (pr https://github.com/lbryio/lbry/pull/1328)
* attempt blob downloads from http mirror sources (by default) concurrently to p2p sources (pr https://github.com/lbryio/lbry/pull/1233)
* replace miniupnpc with [txupnp](https://github.com/lbryio/txupnp). Since txupnp is still under development, it will internally fall back to miniupnpc. (pr https://github.com/lbryio/lbry/pull/1328)
* simplified test_misc.py in the functional tests (pr https://github.com/lbryio/lbry/pull/1328)
* update `cryptography` requirement to 2.3 (pr https://github.com/lbryio/lbry/pull/1333)
* all JSONRPC API commands are now registered asynchronously and are available to be called as soon as they are ready
### Added
* `skipped_components` list to the response from `status` (pr https://github.com/lbryio/lbry/pull/1328)
* component statuses (`blockchain_headers`, `dht`, `wallet`, `blob_manager` `hash_announcer`, and `file_manager`) to the response to `status` (pr https://github.com/lbryio/lbry/pull/1328)
* `skipped_components` config setting, accepts a list of names of components to not run (pr https://github.com/lbryio/lbry/pull/1294)
* `ComponentManager` for managing the life-cycles of dependencies (pr https://github.com/lbryio/lbry/pull/1294)
* `requires` decorator to register the components required by a `jsonrpc_` command, to facilitate commands registering asynchronously (pr https://github.com/lbryio/lbry/pull/1294)
* unit tests for `ComponentManager` (pr https://github.com/lbryio/lbry/pull/1294)
* script to generate docs/api.json file (https://github.com/lbryio/lbry.tech/issues/42)
* additional information to the balance error message when editing a claim (pr https://github.com/lbryio/lbry/pull/1309)
* `address` and `port` arguments to `peer_ping` (https://github.com/lbryio/lbry/issues/1313) (pr https://github.com/lbryio/lbry/pull/1299)
* ability to download from HTTP mirrors by setting `download_mirrors` (prs https://github.com/lbryio/lbry/pull/1233 and https://github.com/lbryio/lbry/pull/1315)
* ability to filter peers from an iterative find value operation (finding peers for a blob). This is used to filter peers we've already found for a blob when accumulating the list of peers. (pr https://github.com/lbryio/lbry/pull/1287)
* Component Manager for managing the dependencies, startup and stopping of components
* `requires` decorator to register the components required by a `jsonrpc_` command, to facilitate commands registering asynchronously
* added unittests for Component Manager
*
### Removed
* `session_status` argument and response field from `status` (pr https://github.com/lbryio/lbry/pull/1328)
* most of the internal attributes from `Daemon` (pr https://github.com/lbryio/lbry/pull/1294)
## [0.20.4] - 2018-07-18
### Fixed
* spelling errors in messages printed by `lbrynet-cli`
* high CPU usage when a stream is incomplete and the peers we're requesting from have no more blobs to send us (https://github.com/lbryio/lbry/pull/1301)
### Changed
* keep track of failures for DHT peers for up to ten minutes instead of indefinitely (https://github.com/lbryio/lbry/pull/1300)
* skip ignored peers from iterative lookups instead of blocking the peer who returned them to us too (https://github.com/lbryio/lbry/pull/1300)
* if a node becomes ignored during an iterative find cycle remove it from the shortlist so that we can't return it as a result nor try to probe it anyway (https://github.com/lbryio/lbry/pull/1303)
* `STARTUP_STAGES` from `status` API and CLI call, it instead returns a dictionary of components along with their running status(this is a **potentially breaking change** if `STARTUP_STAGES` is relied upon)
* all component startup code from `Daemon.py`
* wallet, upnp and dht startup code from `session.py`, the code now resides in `Components.py`
## [0.20.3] - 2018-07-03
@@ -486,7 +323,7 @@ most commands.
* Fixed https://github.com/lbryio/lbry/issues/923
* Fixed concurrent reflects opening too many files
* Fixed cases when reflecting would fail on error conditions
* Fixed deadlocks from occurring during blob writes
* Fixed deadlocks from occuring during blob writes
* Fixed and updated`lbrynet.tests.dht`
* Fixed redundant dht id
* Fixed dht `ping` method
@@ -873,7 +710,7 @@ most commands.
* Removed check_pending logic from Daemon
* Switched to txrequests so requests can use twisted event loop
* Renamed API command file_seed to file_set_status
* Don't add expected payment to wallet when payment rate is 0
* Dont add expected payment to wallet when payment rate is 0
### Fixed
* Fix restart procedure in DaemonControl
* Create download directory if it doesn't exist
@@ -904,7 +741,7 @@ most commands.
* Added string comparison to ClaimOutpoint (needed to look things up by outpoint)
* Remove unused API commands from daemon
* Fix file filter `outpoint`
* Made dictionary key names in API command outputs to be more consistent
* Made dictionary key names in API commmand outputs to be more consistent
### Added
* Add file filters: `claim_id`, `outpoint`, and `rowid`
* Make loggly logs less verbose
@@ -956,7 +793,7 @@ most commands.
* Include download url in version check
### Fixed
* add missing traceback to logging
* add misssing traceback to logging
## [0.8.3] - 2017-02-15
### Fixed
+66 -36
View File
@@ -1,48 +1,70 @@
# Installing LBRY
If only the JSON-RPC API server is needed, the recommended way to install LBRY is to use a pre-built binary. We provide binaries for all major operating systems. See the [README](README.md).
The recommended way to install LBRY is to use an installer. We provide installers for all major operating systems. See the [README](README.md).
These instructions are for installing LBRY from source, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
Here's a video walkthrough of this setup, which is itself hosted by the LBRY network and provided via [spee.ch](https://github.com/lbryio/spee.ch):
[![Setup for development](https://spee.ch/2018-10-04-17-13-54-017046806.png)](https://spee.ch/967f99344308f1e90f0620d91b6c93e4dfb240e0/lbrynet-dev-setup.mp4)
These instructions are for installing from source code, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
## Prerequisites
Running `lbrynet` from source requires Python 3.6 or higher (3.7 is preferred). Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/)
Before starting, you'll need to install some additional libraries depending on your operating system.
After installing python 3, you'll need to install some additional libraries depending on your operating system.
### Python
### Virtualenv
Once python 3 is installed run `python3 -m pip install virtualenv` to install virtualenv.
### Windows
Windows users will need to install `Visual C++ Build Tools`, which can be installed by [Microsoft Build Tools](Microsoft Build Tools 2015)
LBRY requires Python 2.7. Python 3 is not supported.
### macOS
### OS X Prerequisites
macOS users will need to install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
Install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
Remaining dependencies can then be installed by running:
```
brew install python3 protobuf
brew install mpfr libmpc protobuf
sudo easy_install pip
sudo pip install virtualenv
```
### Linux
### Linux Prerequisites
On Ubuntu (we recommend 18.04), install the following:
On Ubuntu (we recommend 16.04), install the following:
```
sudo apt-get install build-essential python3.7 python3.7-dev git python-virtualenv libssl-dev python-protobuf
sudo apt-get install build-essential python2.7 python2.7-dev python-pip git python-virtualenv libssl-dev libffi-dev python-protobuf
```
On Raspbian, you will also need to install `python-pyparsing`.
If you're running another Linux distro, install the equivalent of the above packages for your system.
If you're running another Linux flavor, install the equivalent of the above packages for your system.
### Windows Prerequisites
Install [mingw32](http://www.mingw.org/) base and c++ compiler.
Add C:\MinGW\bin to the windows PATH.
Enable distutils to compile with mingw32 by creating a distutils.cfg file in *PYTHONPATH\Lib\distutils* containing:
```
[build]
compiler = mingw32
```
If using virtualenv, which is recommended, copy the *PYTHONPATH\Lib\distutils* directory to the virtualenv.
It's recommended to use [Unofficial Windows Binaries for Python Extension Packages](http://www.lfd.uci.edu/~gohlke/pythonlibs/) for as many of the required packages as possible.
Currently, available binaries include:
- Cython
- Twisted
- Zope.interface
- pywin32
- requests
Install the above binaries with `pip install *.whl`
Install pywin32 system files by run `python.exe Scripts\pywin32_postinstall.py -install` from an elevated command prompt.
Finally, you'll need [miniupnpc](https://pypi.python.org/pypi/miniupnpc/1.9). Download the source and compile with MinGW by running `mingw32make.bat`. Then install the module by running `python setupmingw32.py install`.
## Installation
@@ -51,27 +73,35 @@ To install:
```
git clone https://github.com/lbryio/lbry.git
cd lbry
virtualenv lbry-venv --python=python3.7
virtualenv lbry-venv
source lbry-venv/bin/activate
python --version # Python 2 is not supported. Make sure you're on Python 3.7
pip install -e .
python --version # Python 3 is not supported. Make sure you're on Python 2.7
pip install -U -r requirements.txt
pip install -U -r requirements_testing.txt
pip install --editable .
```
To verify your installation, `which lbrynet` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
To start LBRY, run `lbrynet-daemon` in a terminal.
## Run the tests
To run the unit tests from the repo directory:
```
trial --reactor=asyncio tests.unit
```
To verify your install, `which lbrynet-daemon` should return a path inside of the `lbry-venv` folder created by the `virtualenv` command.
## Usage
## Making a Change
To start the API server:
`lbrynet start`
If you are not already inside the virtual environment, run:
```
source lbry-venv/bin/activate
```
Then run:
```
pip install --editable .
```
This will install `lbrynet-daemon` in such a way that the changes you make to the code will be used as soon as you restart the daemon.
Happy hacking!
+21 -24
View File
@@ -1,35 +1,36 @@
# <img src="https://raw.githubusercontent.com/lbryio/lbry/master/lbry.png" alt="LBRY" width="48" height="36" /> LBRY SDK [![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry) [![Test Coverage](https://codecov.io/gh/lbryio/lbry/branch/master/graph/badge.svg)](https://codecov.io/gh/lbryio/lbry)
# LBRY
LBRY is a decentralized peer-to-peer network providing distribution, discovery, and purchase of digital content (data). It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, rights and access rules. LBRY also provides a data network that consists of peers (seeders) uploading and downloading data from other peers, possibly in exchange for payments, as well as a distributed hash table used by peers to discover other peers.
[![Build Status](https://travis-ci.org/lbryio/lbry.svg?branch=master)](https://travis-ci.org/lbryio/lbry)
[![Coverage Status](https://coveralls.io/repos/github/lbryio/lbry/badge.svg)](https://coveralls.io/github/lbryio/lbry)
LBRY SDK for Python is currently the most full featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components:
LBRY is an open-source protocol providing distribution, discovery, and purchase of digital content (data) via a decentralized network. It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, and rights and access rules. LBRY also provides a data network that consists of peers uploading and downloading data from other peers, possibly in exchange for payments, and a distributed hash table, used by peers to discover other peers.
* Built on Python 3.7+ and `asyncio`.
* DHT (Distributed Hash Table) implementation for finding peers ([lbrynet.dht](https://github.com/lbryio/lbry/tree/master/lbrynet/dht)).
* Blob exchange protocol for downloading content and negotiating payments ([lbrynet.blob_exchange](https://github.com/lbryio/lbry/tree/master/lbrynet/blob_exchange)).
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([lbrynet.schema](https://github.com/lbryio/lbry/tree/master/lbrynet/schema)).
* Wallet implementation for the LBRY blockchain ([lbrynet.extras.wallet](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/wallet)).
* Daemon with a JSON-RPC API to ease building end user applications in any language and for automating various tasks ([lbrynet.extras.daemon](https://github.com/lbryio/lbry/tree/master/lbrynet/extras/daemon)).
This project aims to provide a daemon that can interact with the network via a json-rpc interface over HTTP.
The project is written in python2.7 and extensively uses Twisted framework.
## Installation
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS, Debian-based Linux, and Windows. [Automated travis builds](http://build.lbry.io/daemon/) are also available for testing.
| | Windows | macOS | Linux |
| --------------------- | -------------------------------------------- | -------------------------------------------- | -------------------------------------------- |
| Latest Stable Release | [Download](https://github.com/lbryio/lbry/releases/download/v0.20.0/lbrynet-daemon-v0.20.0-windows.zip) | [Download](https://github.com/lbryio/lbry/releases/download/v0.20.0/lbrynet-daemon-v0.20.0-macos.zip) | [Download](https://github.com/lbryio/lbry/releases/download/v0.20.0/lbrynet-daemon-v0.20.0-linux.zip) |
| Latest Pre-release | [Download](https://github.com/lbryio/lbry/releases/download/untagged-99114fa31abbfe3a5ef4/lbrynet-daemon-v0.20.1rc3-windows.zip) | [Download](https://github.com/lbryio/lbry/releases/download/untagged-99114fa31abbfe3a5ef4/lbrynet-daemon-v0.20.1rc3-macos.zip) | [Download](https://github.com/lbryio/lbry/releases/download/v0.20.1rc4/lbrynet-daemon-v0.20.1rc4-linux.zip) |
Our [releases page](https://github.com/lbryio/lbry/releases) contains pre-built binaries of the latest release, pre-releases, and past releases, for macOS, Debian-based Linux, and Windows.
## Usage
Run `lbrynet start` to launch the API server.
Run `lbrynet-daemon` to launch the daemon.
By default, `lbrynet` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
By default, `lbrynet-daemon` will provide a JSON-RPC server at `http://localhost:5279`. It is easy to interact with via cURL or sane programming languages.
Our [quickstart guide](https://lbry.tech/playground) provides a simple walkthrough and examples for learning.
Our [quickstart guide](http://lbry.io/quickstart) provides a simple walkthrough and examples for learning.
With the daemon running, `lbrynet commands` will show you a list of commands.
The full API is documented [here](https://lbry.tech/api/sdk).
The full API is documented [here](https://lbryio.github.io/lbry/cli).
## Running from source
Installing from source is also relatively painless. Full instructions are in [INSTALL.md](INSTALL.md)
Installing from source is also relatively painless, full instructions are in [INSTALL.md](INSTALL.md)
## Contributing
@@ -41,16 +42,12 @@ This project is MIT licensed. For the full license, see [LICENSE](LICENSE).
## Security
We take security seriously. Please contact security@lbry.io regarding any security issues. [Our GPG key is here](https://lbry.io/faq/gpg-key) if you need it.
We take security seriously. Please contact security@lbry.io regarding any security issues. Our PGP key is here if you need it.
## Contact
The primary contact for this project is [@eukreign](mailto:lex@lbry.io).
The primary contact for this project is @jackrobison(jack@lbry.io)
## Additional information and links
The documentation for the API can be found [here](https://lbry.tech/api/sdk).
Daemon defaults, ports, and other settings are documented [here](https://lbry.tech/resources/daemon-settings).
Settings can be configured using a daemon-settings.yml file. An example can be found [here](https://github.com/lbryio/lbry/blob/master/example_daemon_settings.yml).
The documentation for the api can be found [here](https://lbry.io/api).
+33
View File
@@ -0,0 +1,33 @@
$env:Path += ";C:\MinGW\bin\"
$env:Path += ";C:\Program Files (x86)\Windows Kits\10\bin\x86\"
gcc --version
mingw32-make --version
# build/install miniupnpc manually
tar zxf miniupnpc-1.9.tar.gz
cd miniupnpc-1.9
mingw32-make -f Makefile.mingw
python setupmingw32.py build --compiler=mingw32
python setupmingw32.py install
cd ..\
Remove-Item -Recurse -Force miniupnpc-1.9
# copy requirements from lbry, but remove miniupnpc (installed manually)
Get-Content ..\requirements.txt | Select-String -Pattern 'miniupnpc' -NotMatch | Out-File requirements_base.txt
python set_build.py
pip install -r requirements.txt
pip install ..\.
pyinstaller -y daemon.onefile.spec
pyinstaller -y cli.onefile.spec
pyinstaller -y console.onefile.spec
nuget install secure-file -ExcludeVersion
secure-file\tools\secure-file -decrypt .\lbry2.pfx.enc -secret "$env:pfx_key"
signtool.exe sign /f .\lbry2.pfx /p "$env:key_pass" /tr http://tsa.starfieldtech.com /td SHA256 /fd SHA256 dist\*.exe
python zip_daemon.py
python upload_assets.py
Executable
+59
View File
@@ -0,0 +1,59 @@
#!/bin/bash
set -euo pipefail
set -x
ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
cd "$ROOT"
BUILD_DIR="$ROOT/build"
FULL_BUILD="${FULL_BUILD:-false}"
if [ -n "${TEAMCITY_VERSION:-}" -o -n "${APPVEYOR:-}" ]; then
FULL_BUILD="true"
fi
[ -d "$BUILD_DIR/bulid" ] && rm -rf "$BUILD_DIR/build"
[ -d "$BUILD_DIR/dist" ] && rm -rf "$BUILD_DIR/dist"
if [ "$FULL_BUILD" == "true" ]; then
# install dependencies
$BUILD_DIR/prebuild.sh
VENV="$BUILD_DIR/venv"
if [ -d "$VENV" ]; then
rm -rf "$VENV"
fi
virtualenv "$VENV"
set +u
source "$VENV/bin/activate"
set -u
# must set build before installing lbrynet. otherwise it has no effect
python "$BUILD_DIR/set_build.py"
fi
cp "$ROOT/requirements.txt" "$BUILD_DIR/requirements_base.txt"
(
cd "$BUILD_DIR"
pip install -r requirements.txt
)
(
cd "$BUILD_DIR"
pyinstaller -y daemon.onefile.spec
pyinstaller -y cli.onefile.spec
pyinstaller -y console.onefile.spec
)
python "$BUILD_DIR/zip_daemon.py"
if [ "$FULL_BUILD" == "true" ]; then
# electron-build has a publish feature, but I had a hard time getting
# it to reliably work and it also seemed difficult to configure. Not proud of
# this, but it seemed better to write my own.
python "$BUILD_DIR/upload_assets.py"
deactivate
fi
echo 'Build complete.'
+42
View File
@@ -0,0 +1,42 @@
# -*- mode: python -*-
import platform
import os
dir = 'build';
cwd = os.getcwd()
if os.path.basename(cwd) != dir:
raise Exception('pyinstaller build needs to be run from the ' + dir + ' directory')
repo_base = os.path.abspath(os.path.join(cwd, '..'))
execfile(os.path.join(cwd, "entrypoint.py")) # ghetto import
system = platform.system()
if system == 'Darwin':
icns = os.path.join(repo_base, 'build', 'icon.icns')
elif system == 'Linux':
icns = os.path.join(repo_base, 'build', 'icons', '256x256.png')
elif system == 'Windows':
icns = os.path.join(repo_base, 'build', 'icons', 'lbry256.ico')
else:
print 'Warning: System {} has no icons'.format(system)
icns = None
a = Entrypoint('lbrynet', 'console_scripts', 'lbrynet-cli', pathex=[cwd])
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='lbrynet-cli',
debug=False,
strip=False,
upx=True,
console=True,
icon=icns
)
+50
View File
@@ -0,0 +1,50 @@
# -*- mode: python -*-
import platform
import os
import lbryum
dir = 'build';
cwd = os.getcwd()
if os.path.basename(cwd) != dir:
raise Exception('pyinstaller build needs to be run from the ' + dir + ' directory')
repo_base = os.path.abspath(os.path.join(cwd, '..'))
execfile(os.path.join(cwd, "entrypoint.py")) # ghetto import
system = platform.system()
if system == 'Darwin':
icns = os.path.join(repo_base, 'build', 'icon.icns')
elif system == 'Linux':
icns = os.path.join(repo_base, 'build', 'icons', '256x256.png')
elif system == 'Windows':
icns = os.path.join(repo_base, 'build', 'icons', 'lbry256.ico')
else:
print 'Warning: System {} has no icons'.format(system)
icns = None
datas = [
(os.path.join(os.path.dirname(lbryum.__file__), 'wordlist', language + '.txt'), 'lbryum/wordlist')
for language in ('chinese_simplified', 'japanese', 'spanish','english', 'portuguese')
]
a = Entrypoint('lbrynet', 'console_scripts', 'lbrynet-console', pathex=[cwd], datas=datas)
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='lbrynet-console',
debug=False,
strip=False,
upx=True,
console=True,
icon=icns
)
+50
View File
@@ -0,0 +1,50 @@
# -*- mode: python -*-
import platform
import os
import lbryum
dir = 'build';
cwd = os.getcwd()
if os.path.basename(cwd) != dir:
raise Exception('pyinstaller build needs to be run from the ' + dir + ' directory')
repo_base = os.path.abspath(os.path.join(cwd, '..'))
execfile(os.path.join(cwd, "entrypoint.py")) # ghetto import
system = platform.system()
if system == 'Darwin':
icns = os.path.join(repo_base, 'build', 'icon.icns')
elif system == 'Linux':
icns = os.path.join(repo_base, 'build', 'icons', '256x256.png')
elif system == 'Windows':
icns = os.path.join(repo_base, 'build', 'icons', 'lbry256.ico')
else:
print 'Warning: System {} has no icons'.format(system)
icns = None
datas = [
(os.path.join(os.path.dirname(lbryum.__file__), 'wordlist', language + '.txt'), 'lbryum/wordlist')
for language in ('chinese_simplified', 'japanese', 'spanish','english', 'portuguese')
]
a = Entrypoint('lbrynet', 'console_scripts', 'lbrynet-daemon', pathex=[cwd], datas=datas)
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
name='lbrynet-daemon',
debug=False,
strip=False,
upx=True,
console=True,
icon=icns
)
+47
View File
@@ -0,0 +1,47 @@
# https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point
def Entrypoint(dist, group, name,
scripts=None, pathex=None, binaries=None, datas=None,
hiddenimports=None, hookspath=None, excludes=None, runtime_hooks=None,
cipher=None, win_no_prefer_redirects=False, win_private_assemblies=False):
import pkg_resources
# get toplevel packages of distribution from metadata
def get_toplevel(dist):
distribution = pkg_resources.get_distribution(dist)
if distribution.has_metadata('top_level.txt'):
return list(distribution.get_metadata('top_level.txt').split())
else:
return []
hiddenimports = hiddenimports or []
packages = []
for distribution in hiddenimports:
packages += get_toplevel(distribution)
scripts = scripts or []
pathex = pathex or []
# get the entry point
ep = pkg_resources.get_entry_info(dist, group, name)
# insert path of the egg at the verify front of the search path
pathex = [ep.dist.location] + pathex
# script name must not be a valid module name to avoid name clashes on import
script_path = os.path.join(workpath, name + '-script.py')
print "creating script for entry point", dist, group, name
with open(script_path, 'w') as fh:
fh.write("import {0}\n".format(ep.module_name))
fh.write("{0}.{1}()\n".format(ep.module_name, '.'.join(ep.attrs)))
for package in packages:
fh.write("import {0}\n".format(package))
return Analysis([script_path] + scripts,
pathex=pathex,
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=hookspath,
excludes=excludes,
runtime_hooks=runtime_hooks,
cipher=cipher,
win_no_prefer_redirects=win_no_prefer_redirects,
win_private_assemblies=win_private_assemblies
)
File renamed without changes.

Before

Width:  |  Height:  |  Size: 7.4 KiB

After

Width:  |  Height:  |  Size: 7.4 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 19 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.2 KiB

After

Width:  |  Height:  |  Size: 1.2 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 2.6 KiB

After

Width:  |  Height:  |  Size: 2.6 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 6.1 KiB

After

Width:  |  Height:  |  Size: 6.1 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 97 KiB

After

Width:  |  Height:  |  Size: 97 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 1.1 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 361 KiB

After

Width:  |  Height:  |  Size: 361 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 5.3 KiB

After

Width:  |  Height:  |  Size: 5.3 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 15 KiB

File renamed without changes.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.
Binary file not shown.
+81
View File
@@ -0,0 +1,81 @@
#!/bin/bash
set -euo pipefail
set -x
LINUX=false
OSX=false
if [ "$(uname)" == "Darwin" ]; then
OSX=true
elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
LINUX=true
else
echo "Platform detection failed"
exit 1
fi
SUDO=''
if $LINUX && (( $EUID != 0 )); then
SUDO='sudo'
fi
cmd_exists() {
command -v "$1" >/dev/null 2>&1
return $?
}
set +eu
GITUSERNAME=$(git config --global --get user.name)
if [ -z "$GITUSERNAME" ]; then
git config --global user.name "$(whoami)"
fi
GITEMAIL=$(git config --global --get user.email)
if [ -z "$GITEMAIL" ]; then
git config --global user.email "$(whoami)@lbry.io"
fi
set -eu
if $LINUX; then
INSTALL="$SUDO apt-get install --no-install-recommends -y"
$INSTALL build-essential libssl-dev libffi-dev python2.7-dev wget
elif $OSX && ! cmd_exists brew ; then
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
fi
if ! cmd_exists python; then
if $LINUX; then
$INSTALL python2.7
elif $OSX; then
brew install python
fi
fi
PYTHON_VERSION=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
if [ "$PYTHON_VERSION" != "2.7" ]; then
echo "Python 2.7 required"
exit 1
fi
if ! cmd_exists pip; then
if $LINUX; then
$INSTALL python-pip
$SUDO pip install --upgrade pip
else
echo "Pip required"
exit 1
fi
fi
if $LINUX && [ "$(pip list --format=columns | grep setuptools | wc -l)" -ge 1 ]; then
#$INSTALL python-setuptools
$SUDO pip install setuptools
fi
if ! cmd_exists virtualenv; then
$SUDO pip install virtualenv
fi
+11
View File
@@ -0,0 +1,11 @@
# install daemon requirements (created by build script. see build.sh, build.ps1)
-r requirements_base.txt
# install daemon itself. make sure you run `pip install` from this dir. this is how you do relative file paths with pip
file:../.
# install other build requirements
PyInstaller==3.2.1
requests[security]==2.13.0
uritemplate==3.0.0
boto3==1.4.4
+3 -5
View File
@@ -1,4 +1,3 @@
"""Set the build version to be 'dev', 'qa', 'rc', 'release'"""
import os.path
@@ -16,12 +15,11 @@ def main():
def get_build():
try:
tag = subprocess.check_output(['git', 'describe', '--exact-match', '--all']).strip()
if re.match('tags\/v\d+\.\d+\.\d+rc\d+$', tag.decode()):
tag = subprocess.check_output(['git', 'describe', '--exact-match']).strip()
if re.match('v\d+\.\d+\.\d+rc\d+', tag):
return 'rc'
elif re.match('tags\/v\d+\.\d+\.\d+$', tag.decode()):
else:
return 'release'
return 'qa'
except subprocess.CalledProcessError:
# if the build doesn't have a tag
return 'qa'
+143
View File
@@ -0,0 +1,143 @@
import glob
import json
import os
import subprocess
import sys
import github
import uritemplate
import boto3
def main():
upload_to_github_if_tagged('lbryio/lbry')
upload_to_s3('daemon')
def get_asset_filename():
this_dir = os.path.dirname(os.path.realpath(__file__))
return glob.glob(this_dir + '/dist/*.zip')[0]
def upload_to_s3(folder):
tag = subprocess.check_output(['git', 'describe', '--always', '--abbrev=8', 'HEAD']).strip()
commit_date = subprocess.check_output([
'git', 'show', '-s', '--format=%cd', '--date=format:%Y%m%d-%H%I%S', 'HEAD']).strip()
asset_path = get_asset_filename()
bucket = 'releases.lbry.io'
key = folder + '/' + commit_date + '-' + tag + '/' + os.path.basename(asset_path)
print "Uploading " + asset_path + " to s3://" + bucket + '/' + key + ''
if 'AWS_ACCESS_KEY_ID' not in os.environ or 'AWS_SECRET_ACCESS_KEY' not in os.environ:
print 'Must set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to publish assets to s3'
return 1
s3 = boto3.resource(
's3',
aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],
config=boto3.session.Config(signature_version='s3v4')
)
s3.meta.client.upload_file(asset_path, bucket, key)
def upload_to_github_if_tagged(repo_name):
try:
current_tag = subprocess.check_output(
['git', 'describe', '--exact-match', 'HEAD']).strip()
except subprocess.CalledProcessError:
print 'Not uploading to GitHub as we are not currently on a tag'
return 1
print "Current tag: " + current_tag
if 'GH_TOKEN' not in os.environ:
print 'Must set GH_TOKEN in order to publish assets to a release'
return 1
gh_token = os.environ['GH_TOKEN']
auth = github.Github(gh_token)
repo = auth.get_repo(repo_name)
if not check_repo_has_tag(repo, current_tag):
print 'Tag {} is not in repo {}'.format(current_tag, repo)
# TODO: maybe this should be an error
return 1
asset_path = get_asset_filename()
print "Uploading " + asset_path + " to Github tag " + current_tag
release = get_github_release(repo, current_tag)
upload_asset_to_github(release, asset_path, gh_token)
def check_repo_has_tag(repo, target_tag):
tags = repo.get_tags().get_page(0)
for tag in tags:
if tag.name == target_tag:
return True
return False
def get_github_release(repo, current_tag):
for release in repo.get_releases():
if release.tag_name == current_tag:
return release
raise Exception('No release for {} was found'.format(current_tag))
def upload_asset_to_github(release, asset_to_upload, token):
basename = os.path.basename(asset_to_upload)
for asset in release.raw_data['assets']:
if asset['name'] == basename:
print 'File {} has already been uploaded to {}'.format(basename, release.tag_name)
return
upload_uri = uritemplate.expand(release.upload_url, {'name': basename})
count = 0
while count < 10:
try:
output = _curl_uploader(upload_uri, asset_to_upload, token)
if 'errors' in output:
raise Exception(output)
else:
print 'Successfully uploaded to {}'.format(output['browser_download_url'])
except Exception:
print 'Failed uploading on attempt {}'.format(count + 1)
count += 1
def _curl_uploader(upload_uri, asset_to_upload, token):
# using requests.post fails miserably with SSL EPIPE errors. I spent
# half a day trying to debug before deciding to switch to curl.
#
# TODO: actually set the content type
print 'Using curl to upload {} to {}'.format(asset_to_upload, upload_uri)
cmd = [
'curl',
'-sS',
'-X', 'POST',
'-u', ':{}'.format(os.environ['GH_TOKEN']),
'--header', 'Content-Type: application/octet-stream',
'--data-binary', '@-',
upload_uri
]
# '-d', '{"some_key": "some_value"}',
print 'Calling curl:'
print cmd
print
with open(asset_to_upload, 'rb') as fp:
p = subprocess.Popen(cmd, stdin=fp, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
stdout, stderr = p.communicate()
print 'curl return code:', p.returncode
if stderr:
print 'stderr output from curl:'
print stderr
print 'stdout from curl:'
print stdout
return json.loads(stdout)
if __name__ == '__main__':
sys.exit(main())
+29
View File
@@ -0,0 +1,29 @@
import os
import platform
import subprocess
import sys
import zipfile
def main():
this_dir = os.path.dirname(os.path.realpath(__file__))
tag = subprocess.check_output(['git', 'describe']).strip()
zipfilename = 'lbrynet-daemon-{}-{}.zip'.format(tag, get_system_label())
full_filename = os.path.join(this_dir, 'dist', zipfilename)
executables = ['lbrynet-daemon', 'lbrynet-cli', 'lbrynet-console']
ext = '.exe' if platform.system() == 'Windows' else ''
with zipfile.ZipFile(full_filename, 'w') as myzip:
for executable in executables:
myzip.write(os.path.join(this_dir, 'dist', executable + ext), executable + ext)
def get_system_label():
system = platform.system()
if system == 'Darwin':
return 'macos'
else:
return system.lower()
if __name__ == '__main__':
sys.exit(main())
+3 -3
View File
@@ -100,7 +100,7 @@
<div class="md-flex__cell md-flex__cell--shrink">
<a href="/" title="LBRY" class="md-header-nav__button md-logo">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" alt="LBRY logo" width="24" height="24">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" width="24" height="24">
</a>
</div>
@@ -196,7 +196,7 @@
<label class="md-nav__title md-nav__title--site" for="drawer">
<span class="md-nav__button md-logo">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" alt="LBRY logo" width="48" height="48">
<img src="https://s3.amazonaws.com/files.lbry.io/logo-square-white-bookonly.png" width="48" height="48">
</span>
LBRY
@@ -304,4 +304,4 @@
</body>
</html>
</html>
-1605
View File
File diff suppressed because it is too large. Load diff
+37 -60
View File
@@ -1579,12 +1579,12 @@ Usage:
Options:
--a_arg : (bool) a arg
--b_arg : (bool) b arg
--pos_arg=&lt;pos_arg&gt; : (int) pos arg
--pos_args=&lt;pos_args&gt; : (int) pos args
--pos_arg2=&lt;pos_arg2&gt; : (int) pos arg 2
--pos_arg3=&lt;pos_arg3&gt; : (int) pos arg 3
--a_arg : a arg
--b_arg : b arg
--pos_arg=&lt;pos_arg&gt; : pos arg
--pos_args=&lt;pos_args&gt; : pos args
--pos_arg2=&lt;pos_arg2&gt; : pos arg 2
--pos_arg3=&lt;pos_arg3&gt; : pos arg 3
Returns:
pos args
@@ -1828,16 +1828,14 @@ Returns:
</code></pre>
<h2 id="peer_ping">peer_ping</h2>
<pre><code class="text">Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,
if not provided the peer is located first.
<pre><code class="text">Find and ping a peer by node id
Usage:
peer_ping (&lt;node_id&gt; | --node_id=&lt;node_id&gt;) [&lt;address&gt; | --address=&lt;address&gt;] [&lt;port&gt; | --port=&lt;port&gt;]
peer_ping (&lt;node_id&gt; | --node_id=&lt;node_id&gt;)
Options:
--address=&lt;address&gt; : (str) ip address of the peer
--port=&lt;port&gt; : (int) udp port of the peer
None
Returns:
(str) pong, or {'error': &lt;error message&gt;} if an error is encountered
@@ -1903,7 +1901,7 @@ Options:
publishing to a channel where only the certificate
private key is in the wallet.
--claim_address=&lt;claim_address&gt; : (str) address where the claim is sent to, if not specified
new address will automatically be created
new address wil automatically be created
Returns:
(dict) Dictionary containing result of the claim
@@ -2112,62 +2110,41 @@ Returns:
<pre><code class="text">Get daemon status
Usage:
status
status [--session_status]
Options:
None
--session_status : (bool) include session status in results
Returns:
(dict) lbrynet-daemon status
{
'installation_id': (str) installation id - base58,
'is_running': (bool),
'lbry_id': lbry peer id, base58,
'installation_id': installation id, base58,
'is_running': bool,
'is_first_run': bool,
'skipped_components': (list) [names of skipped components (str)],
'startup_status': { Does not include components which have been skipped
'database': (bool),
'wallet': (bool),
'session': (bool),
'dht': (bool),
'hash_announcer': (bool),
'stream_identifier': (bool),
'file_manager': (bool),
'blob_manager': (bool),
'blockchain_headers': (bool),
'peer_protocol_server': (bool),
'reflector': (bool),
'upnp': (bool),
'exchange_rate_manager': (bool),
'startup_status': {
'code': status code,
'message': status message
},
'connection_status': {
'code': (str) connection status code,
'message': (str) connection status message
'code': connection status code,
'message': connection status message
},
'blockchain_headers': {
'downloading_headers': (bool),
'download_progress': (float) 0-100.0
'blockchain_status': {
'blocks': local blockchain height,
'blocks_behind': remote_height - local_height,
'best_blockhash': block hash of most recent block,
},
'wallet': {
'blocks': (int) local blockchain height,
'blocks_behind': (int) remote_height - local_height,
'best_blockhash': (str) block hash of most recent block,
'is_encrypted': (bool),
'is_locked': (bool),
},
'dht': {
'node_id': (str) lbry dht node id - hex encoded,
'peers_in_routing_table': (int) the number of peers in the routing table,
},
'blob_manager': {
'finished_blobs': (int) number of finished blobs in the blob manager,
},
'hash_announcer': {
'announce_queue_size': (int) number of blobs currently queued to be announced
},
'file_manager': {
'managed_files': (int) count of files in the file manager,
}
'wallet_is_encrypted': bool,
If given the session status option:
'session_status': {
'managed_blobs': count of blobs in the blob manager,
'managed_streams': count of streams in the file manager
'announce_queue_size': number of blobs currently queued to be announced
'should_announce_blobs': number of blobs that should be announced
}
}
</code></pre>
@@ -2475,9 +2452,9 @@ Usage:
Options:
--amount=&lt;amount&gt; : (Decimal) amount of credit to send
--address=&lt;address&gt; : (str) address to send credits to
--claim_id=&lt;claim_id&gt; : (str) claim_id of the claim to send to tip to
--amount=&lt;amount&gt; : (float) amount of credit to send
--address=&lt;address&gt; : (str) address to send credits to
--claim_id=&lt;claim_id&gt; : (float) claim_id of the claim to send to tip to
Returns:
If sending to an address:
@@ -2586,4 +2563,4 @@ Returns:
</body>
</html>
</html>
+35 -58
View File
@@ -1479,12 +1479,12 @@ Returns:
<h2 id="cli_test_command">cli_test_command</h2>
<pre><code class="text">This command is only for testing the CLI argument parsing
Args:
'a_arg' : (bool) a arg
'b_arg' : (bool) b arg
'pos_arg' (required) : (int) pos arg
'pos_args' : (int) pos args
'pos_arg2' : (int) pos arg 2
'pos_arg3' : (int) pos arg 3
'a_arg' : a arg
'b_arg' : b arg
'pos_arg' (required) : pos arg
'pos_args' : pos args
'pos_arg2' : pos arg 2
'pos_arg3' : pos arg 3
Returns:
pos args
@@ -1682,12 +1682,10 @@ Returns:
</code></pre>
<h2 id="peer_ping">peer_ping</h2>
<pre><code class="text">Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,
if not provided the peer is located first.
<pre><code class="text">Find and ping a peer by node id
Args:
'address' : (str) ip address of the peer
'port' : (int) udp port of the peer
None
Returns:
(str) pong, or {'error': &lt;error message&gt;} if an error is encountered
@@ -1743,7 +1741,7 @@ Args:
publishing to a channel where only the certificate
private key is in the wallet.
'claim_address' : (str) address where the claim is sent to, if not specified
new address will automatically be created
new address wil automatically be created
Returns:
(dict) Dictionary containing result of the claim
@@ -1915,58 +1913,37 @@ Returns:
<pre><code class="text">Get daemon status
Args:
None
'session_status' : (bool) include session status in results
Returns:
(dict) lbrynet-daemon status
{
'installation_id': (str) installation id - base58,
'is_running': (bool),
'lbry_id': lbry peer id, base58,
'installation_id': installation id, base58,
'is_running': bool,
'is_first_run': bool,
'skipped_components': (list) [names of skipped components (str)],
'startup_status': { Does not include components which have been skipped
'database': (bool),
'wallet': (bool),
'session': (bool),
'dht': (bool),
'hash_announcer': (bool),
'stream_identifier': (bool),
'file_manager': (bool),
'blob_manager': (bool),
'blockchain_headers': (bool),
'peer_protocol_server': (bool),
'reflector': (bool),
'upnp': (bool),
'exchange_rate_manager': (bool),
'startup_status': {
'code': status code,
'message': status message
},
'connection_status': {
'code': (str) connection status code,
'message': (str) connection status message
'code': connection status code,
'message': connection status message
},
'blockchain_headers': {
'downloading_headers': (bool),
'download_progress': (float) 0-100.0
'blockchain_status': {
'blocks': local blockchain height,
'blocks_behind': remote_height - local_height,
'best_blockhash': block hash of most recent block,
},
'wallet': {
'blocks': (int) local blockchain height,
'blocks_behind': (int) remote_height - local_height,
'best_blockhash': (str) block hash of most recent block,
'is_encrypted': (bool),
'is_locked': (bool),
},
'dht': {
'node_id': (str) lbry dht node id - hex encoded,
'peers_in_routing_table': (int) the number of peers in the routing table,
},
'blob_manager': {
'finished_blobs': (int) number of finished blobs in the blob manager,
},
'hash_announcer': {
'announce_queue_size': (int) number of blobs currently queued to be announced
},
'file_manager': {
'managed_files': (int) count of files in the file manager,
}
'wallet_is_encrypted': bool,
If given the session status option:
'session_status': {
'managed_blobs': count of blobs in the blob manager,
'managed_streams': count of streams in the file manager
'announce_queue_size': number of blobs currently queued to be announced
'should_announce_blobs': number of blobs that should be announced
}
}
</code></pre>
@@ -2209,9 +2186,9 @@ to the owner of a claim specified by uri. A tip is a claim support where the rec
of the support is the claim address for the claim being supported.
Args:
'amount' (required) : (Decimal) amount of credit to send
'address' (required) : (str) address to send credits to
'claim_id' (required) : (str) claim_id of the claim to send to tip to
'amount' (required) : (float) amount of credit to send
'address' (required) : (str) address to send credits to
'claim_id' (required) : (float) claim_id of the claim to send to tip to
Returns:
If sending to an address:
@@ -2312,4 +2289,4 @@ Returns:
</body>
</html>
</html>
+13 -13
View File
@@ -2,7 +2,7 @@
"docs": [
{
"location": "/",
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nArgs:\n 'blob_hash' : (str) announce a blob, specified by blob_hash\n 'stream_hash' : (str) announce all blobs associated with\n stream_hash\n 'sd_hash' : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nArgs:\n 'blob_hash' : (str) check availability for this blob hash\n 'search_timeout' : (int) how long to search for peers for the blob\n in the dht\n 'blob_timeout' : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nArgs:\n 'blob_hash' : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nArgs:\n 'blob_hash' (required) : (str) blob hash of the blob to get\n 'timeout' : (int) timeout in number of seconds\n 'encoding' : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n 'payment_rate_manager' : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nArgs:\n 'needed' : (bool) only return needed blobs\n 'finished' : (bool) only return finished blobs\n 'uri' : (str) filter blobs by stream in a uri\n 'stream_hash' : (str) filter blobs by stream hash\n 'sd_hash' : (str) filter blobs by sd hash\n 'page_size' : (int) results page size\n 'page' : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nArgs:\n 'reflector_server' : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nArgs:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nArgs:\n 'blockhash' (required) : (str) hash of the block to look up\n 'height' (required) : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nArgs:\n 'claim_id' (required) : (str) Claim ID to export information about\n\nReturns:\n (str) Serialized certificate information\n\n\n\n\nchannel_import\n\n\nImport serialized channel signing information (to allow signing new claims to the channel)\n\nArgs:\n 'serialized_certificate_info' (required) : (str) certificate info\n\nReturns:\n (dict) Result dictionary\n\n\n\n\nchannel_list\n\n\nGet certificate claim infos for channels that can be published to\n\nArgs:\n None\n\nReturns:\n (list) ClaimDict, includes 'is_mine' field to indicate if the certificate claim\n is in the wallet.\n\n\n\n\nchannel_new\n\n\nGenerate a publisher key and create a new '@' prefixed certificate claim\n\nArgs:\n 'channel_name' (required) : (str) name of the channel prefixed with '@'\n 'amount' (required) : (float) bid amount on the channel\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (float) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }\n\n\n\n\nclaim_abandon\n\n\nAbandon a name and reclaim credits from the claim\n\nArgs:\n 'claim_id' : (str) claim_id of the claim to abandon\n 'txid' : (str) txid of the claim to abandon\n 'nout' : (int) nout of the claim to abandon\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n txid : (str) txid of resulting transaction\n fee : (float) fee paid for the transaction\n }\n\n\n\n\nclaim_list\n\n\nList current claims and information about them for a given name\n\nArgs:\n 'name' (required) : (str) name of the claim to list info about\n\nReturns:\n (dict) State Line truncated
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nArgs:\n 'blob_hash' : (str) announce a blob, specified by blob_hash\n 'stream_hash' : (str) announce all blobs associated with\n stream_hash\n 'sd_hash' : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nArgs:\n 'blob_hash' : (str) check availability for this blob hash\n 'search_timeout' : (int) how long to search for peers for the blob\n in the dht\n 'blob_timeout' : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nArgs:\n 'blob_hash' : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nArgs:\n 'blob_hash' (required) : (str) blob hash of the blob to get\n 'timeout' : (int) timeout in number of seconds\n 'encoding' : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n 'payment_rate_manager' : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nArgs:\n 'needed' : (bool) only return needed blobs\n 'finished' : (bool) only return finished blobs\n 'uri' : (str) filter blobs by stream in a uri\n 'stream_hash' : (str) filter blobs by stream hash\n 'sd_hash' : (str) filter blobs by sd hash\n 'page_size' : (int) results page size\n 'page' : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nArgs:\n 'reflector_server' : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nArgs:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nArgs:\n 'blockhash' (required) : (str) hash of the block to look up\n 'height' (required) : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nArgs:\n 'claim_id' (required) : (str) Claim ID to export information about\n\nReturns:\n (str) Serialized certificate information\n\n\n\n\nchannel_import\n\n\nImport serialized channel signing information (to allow signing new claims to the channel)\n\nArgs:\n 'serialized_certificate_info' (required) : (str) certificate info\n\nReturns:\n (dict) Result dictionary\n\n\n\n\nchannel_list\n\n\nGet certificate claim infos for channels that can be published to\n\nArgs:\n None\n\nReturns:\n (list) ClaimDict, includes 'is_mine' field to indicate if the certificate claim\n is in the wallet.\n\n\n\n\nchannel_new\n\n\nGenerate a publisher key and create a new '@' prefixed certificate claim\n\nArgs:\n 'channel_name' (required) : (str) name of the channel prefixed with '@'\n 'amount' (required) : (float) bid amount on the channel\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (float) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }\n\n\n\n\nclaim_abandon\n\n\nAbandon a name and reclaim credits from the claim\n\nArgs:\n 'claim_id' : (str) claim_id of the claim to abandon\n 'txid' : (str) txid of the claim to abandon\n 'nout' : (int) nout of the claim to abandon\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n txid : (str) txid of resulting transaction\n fee : (float) fee paid for the transaction\n }\n\n\n\n\nclaim_list\n\n\nList current claims and information about them for a given name\n\nArgs:\n 'name' (required) : (str) name of the claim to list info about\n\nReturns:\n (dict) State Line truncated
"title": "API"
},
{
@@ -112,7 +112,7 @@
},
{
"location": "/#cli_test_command",
"text": "This command is only for testing the CLI argument parsing\nArgs:\n 'a_arg' : (bool) a arg\n 'b_arg' : (bool) b arg\n 'pos_arg' (required) : (int) pos arg\n 'pos_args' : (int) pos args\n 'pos_arg2' : (int) pos arg 2\n 'pos_arg3' : (int) pos arg 3\n\nReturns:\n pos args",
"text": "This command is only for testing the CLI argument parsing\nArgs:\n 'a_arg' : a arg\n 'b_arg' : b arg\n 'pos_arg' (required) : pos arg\n 'pos_args' : pos args\n 'pos_arg2' : pos arg 2\n 'pos_arg3' : pos arg 3\n\nReturns:\n pos args",
"title": "cli_test_command"
},
{
@@ -162,12 +162,12 @@
},
{
"location": "/#peer_ping",
"text": "Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,\nif not provided the peer is located first.\n\nArgs:\n 'address' : (str) ip address of the peer\n 'port' : (int) udp port of the peer\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"text": "Find and ping a peer by node id\n\nArgs:\n None\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"title": "peer_ping"
},
{
"location": "/#publish",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nArgs:\n 'name' (required) : (str) name of the content\n 'bid' (required) : (decimal) amount to back the claim\n 'metadata' : (dict) ClaimDict to associate with the claim.\n 'file_path' : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n 'fee' : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n 'title' : (str) title of the publication\n 'description' : (str) description of the publication\n 'author' : (str) author of the publication\n 'language' : (str) language of the publication\n 'license' : (str) publication license\n 'license_url' : (str) publication license url\n 'thumbnail' : (str) thumbnail url\n 'preview' : (str) preview url\n 'nsfw' : (bool) whether the content is nsfw\n 'sources' : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n 'channel_name' : (str) name of the publisher channel name in the wallet\n 'channel_id' : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n 'claim_address' : (str) address where the claim is sent to, if not specified\n new address will automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nArgs:\n 'name' (required) : (str) name of the content\n 'bid' (required) : (decimal) amount to back the claim\n 'metadata' : (dict) ClaimDict to associate with the claim.\n 'file_path' : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n 'fee' : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n 'title' : (str) title of the publication\n 'description' : (str) description of the publication\n 'author' : (str) author of the publication\n 'language' : (str) language of the publication\n 'license' : (str) publication license\n 'license_url' : (str) publication license url\n 'thumbnail' : (str) thumbnail url\n 'preview' : (str) preview url\n 'nsfw' : (bool) whether the content is nsfw\n 'sources' : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n 'channel_name' : (str) name of the publisher channel name in the wallet\n 'channel_id' : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n 'claim_address' : (str) address where the claim is sent to, if not specified\n new address wil automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"title": "publish"
},
{
@@ -202,7 +202,7 @@
},
{
"location": "/#status",
"text": "Get daemon status\n\nArgs:\n None\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'is_first_run': bool,\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'database': (bool),\n 'wallet': (bool),\n 'session': (bool),\n 'dht': (bool),\n 'hash_announcer': (bool),\n 'stream_identifier': (bool),\n 'file_manager': (bool),\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'peer_protocol_server': (bool),\n 'reflector': (bool),\n 'upnp': (bool),\n 'exchange_rate_manager': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the file manager,\n }\n }",
"text": "Get daemon status\n\nArgs:\n 'session_status' : (bool) include session status in results\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'lbry_id': lbry peer id, base58,\n 'installation_id': installation id, base58,\n 'is_running': bool,\n 'is_first_run': bool,\n 'startup_status': {\n 'code': status code,\n 'message': status message\n },\n 'connection_status': {\n 'code': connection status code,\n 'message': connection status message\n },\n 'blockchain_status': {\n 'blocks': local blockchain height,\n 'blocks_behind': remote_height - local_height,\n 'best_blockhash': block hash of most recent block,\n },\n 'wallet_is_encrypted': bool,\n\n If given the session status option:\n 'session_status': {\n 'managed_blobs': count of blobs in the blob manager,\n 'managed_streams': count of streams in the file manager\n 'announce_queue_size': number of blobs currently queued to be announced\n 'should_announce_blobs': number of blobs that should be announced\n }\n }",
"title": "status"
},
{
@@ -277,7 +277,7 @@
},
{
"location": "/#wallet_send",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nArgs:\n 'amount' (required) : (Decimal) amount of credit to send\n 'address' (required) : (str) address to send credits to\n 'claim_id' (required) : (str) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nArgs:\n 'amount' (required) : (float) amount of credit to send\n 'address' (required) : (str) address to send credits to\n 'claim_id' (required) : (float) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"title": "wallet_send"
},
{
@@ -292,7 +292,7 @@
},
{
"location": "/cli/",
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nUsage:\n blob_announce [\nblob_hash\n | --blob_hash=\nblob_hash\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n] | [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) announce a blob, specified by blob_hash\n --stream_hash=\nstream_hash\n : (str) announce all blobs associated with\n stream_hash\n --sd_hash=\nsd_hash\n : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nUsage:\n blob_availability (\nblob_hash\n) [\nsearch_timeout\n | --search_timeout=\nsearch_timeout\n]\n [\nblob_timeout\n | --blob_timeout=\nblob_timeout\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) check availability for this blob hash\n --search_timeout=\nsearch_timeout\n : (int) how long to search for peers for the blob\n in the dht\n --blob_timeout=\nblob_timeout\n : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nUsage:\n blob_delete (\nblob_hash\n | --blob_hash=\nblob_hash)\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nUsage:\n blob_get (\nblob_hash\n | --blob_hash=\nblob_hash\n) [--timeout=\ntimeout\n]\n [--encoding=\nencoding\n] [--payment_rate_manager=\npayment_rate_manager\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to get\n --timeout=\ntimeout\n : (int) timeout in number of seconds\n --encoding=\nencoding\n : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n --payment_rate_manager=\npayment_rate_manager\n : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nUsage:\n blob_list [--needed] [--finished] [\nuri\n | --uri=\nuri\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n]\n [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n [\npage_size\n | --page_size=\npage_size\n]\n [\npage\n | --page=\npage\n]\n\n\nOptions:\n --needed : (bool) only return needed blobs\n --finished : (bool) only return finished blobs\n --uri=\nuri\n : (str) filter blobs by stream in a uri\n --stream_hash=\nstream_hash\n : (str) filter blobs by stream hash\n --sd_hash=\nsd_hash\n : (str) filter blobs by sd hash\n --page_size=\npage_size\n : (int) results page size\n --page=\npage\n : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nUsage:\n blob_reflect (\nblob_hashes\n...) [--reflector_server=\nreflector_server\n]\n\n\nOptions:\n --reflector_server=\nreflector_server\n : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nUsage:\n blob_reflect_all\n\n\nOptions:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nUsage:\n block_show (\nblockhash\n | --blockhash=\nblockhash\n) | (\nheight\n | --height=\nheight\n)\n\n\nOptions:\n --blockhash=\nblockhash\n : (str) hash of the block to look up\n --height=\nheight\n : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nUsage:\n channel_export (\nclaim_id\n | --claim_id=\nclaim_id\n)\n\n\nOptions:\n --claim_id=\nclaim_id\n : (str) Claim ID to export informatiLine truncated
"text": "LBRY JSON-RPC API Documentation\n\n\nblob_announce\n\n\nAnnounce blobs to the DHT\n\nUsage:\n blob_announce [\nblob_hash\n | --blob_hash=\nblob_hash\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n] | [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) announce a blob, specified by blob_hash\n --stream_hash=\nstream_hash\n : (str) announce all blobs associated with\n stream_hash\n --sd_hash=\nsd_hash\n : (str) announce all blobs associated with\n sd_hash and the sd_hash itself\n\nReturns:\n (bool) true if successful\n\n\n\n\nblob_availability\n\n\nGet blob availability\n\nUsage:\n blob_availability (\nblob_hash\n) [\nsearch_timeout\n | --search_timeout=\nsearch_timeout\n]\n [\nblob_timeout\n | --blob_timeout=\nblob_timeout\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) check availability for this blob hash\n --search_timeout=\nsearch_timeout\n : (int) how long to search for peers for the blob\n in the dht\n --blob_timeout=\nblob_timeout\n : (int) how long to try downloading from a peer\n\nReturns:\n (dict) {\n \nis_available\n: \nbool, true if blob is available from a peer from peer list\n\n \nreachable_peers\n: [\nip\n:\nport\n],\n \nunreachable_peers\n: [\nip\n:\nport\n]\n }\n\n\n\n\nblob_delete\n\n\nDelete a blob\n\nUsage:\n blob_delete (\nblob_hash\n | --blob_hash=\nblob_hash)\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to delete\n\nReturns:\n (str) Success/fail message\n\n\n\n\nblob_get\n\n\nDownload and return a blob\n\nUsage:\n blob_get (\nblob_hash\n | --blob_hash=\nblob_hash\n) [--timeout=\ntimeout\n]\n [--encoding=\nencoding\n] [--payment_rate_manager=\npayment_rate_manager\n]\n\n\nOptions:\n --blob_hash=\nblob_hash\n : (str) blob hash of the blob to get\n --timeout=\ntimeout\n : (int) timeout in number of seconds\n --encoding=\nencoding\n : (str) by default no attempt at decoding\n is made, can be set to one of the\n following decoders:\n 'json'\n --payment_rate_manager=\npayment_rate_manager\n : (str) if not given the default payment rate\n manager will be used.\n supported alternative rate managers:\n 'only-free'\n\nReturns:\n (str) Success/Fail message or (dict) decoded data\n\n\n\n\nblob_list\n\n\nReturns blob hashes. If not given filters, returns all blobs known by the blob manager\n\nUsage:\n blob_list [--needed] [--finished] [\nuri\n | --uri=\nuri\n]\n [\nstream_hash\n | --stream_hash=\nstream_hash\n]\n [\nsd_hash\n | --sd_hash=\nsd_hash\n]\n [\npage_size\n | --page_size=\npage_size\n]\n [\npage\n | --page=\npage\n]\n\n\nOptions:\n --needed : (bool) only return needed blobs\n --finished : (bool) only return finished blobs\n --uri=\nuri\n : (str) filter blobs by stream in a uri\n --stream_hash=\nstream_hash\n : (str) filter blobs by stream hash\n --sd_hash=\nsd_hash\n : (str) filter blobs by sd hash\n --page_size=\npage_size\n : (int) results page size\n --page=\npage\n : (int) page of results to return\n\nReturns:\n (list) List of blob hashes\n\n\n\n\nblob_reflect\n\n\nReflects specified blobs\n\nUsage:\n blob_reflect (\nblob_hashes\n...) [--reflector_server=\nreflector_server\n]\n\n\nOptions:\n --reflector_server=\nreflector_server\n : (str) reflector address\n\nReturns:\n (list) reflected blob hashes\n\n\n\n\nblob_reflect_all\n\n\nReflects all saved blobs\n\nUsage:\n blob_reflect_all\n\n\nOptions:\n None\n\nReturns:\n (bool) true if successful\n\n\n\n\nblock_show\n\n\nGet contents of a block\n\nUsage:\n block_show (\nblockhash\n | --blockhash=\nblockhash\n) | (\nheight\n | --height=\nheight\n)\n\n\nOptions:\n --blockhash=\nblockhash\n : (str) hash of the block to look up\n --height=\nheight\n : (int) height of the block to look up\n\nReturns:\n (dict) Requested block\n\n\n\n\nchannel_export\n\n\nExport serialized channel signing information for a given certificate claim id\n\nUsage:\n channel_export (\nclaim_id\n | --claim_id=\nclaim_id\n)\n\n\nOptions:\n --claim_id=\nclaim_id\n : (str) Claim ID to export informatiLine truncated
"title": "CLI"
},
{
@@ -402,7 +402,7 @@
},
{
"location": "/cli/#cli_test_command",
"text": "This command is only for testing the CLI argument parsing\nUsage:\n cli_test_command [--a_arg] [--b_arg] ( pos_arg | --pos_arg= pos_arg )\n [ pos_args ...] [--pos_arg2= pos_arg2 ]\n [--pos_arg3= pos_arg3 ]\n\n\nOptions:\n --a_arg : (bool) a arg\n --b_arg : (bool) b arg\n --pos_arg= pos_arg : (int) pos arg\n --pos_args= pos_args : (int) pos args\n --pos_arg2= pos_arg2 : (int) pos arg 2\n --pos_arg3= pos_arg3 : (int) pos arg 3\n\nReturns:\n pos args",
"text": "This command is only for testing the CLI argument parsing\nUsage:\n cli_test_command [--a_arg] [--b_arg] ( pos_arg | --pos_arg= pos_arg )\n [ pos_args ...] [--pos_arg2= pos_arg2 ]\n [--pos_arg3= pos_arg3 ]\n\n\nOptions:\n --a_arg : a arg\n --b_arg : b arg\n --pos_arg= pos_arg : pos arg\n --pos_args= pos_args : pos args\n --pos_arg2= pos_arg2 : pos arg 2\n --pos_arg3= pos_arg3 : pos arg 3\n\nReturns:\n pos args",
"title": "cli_test_command"
},
{
@@ -452,12 +452,12 @@
},
{
"location": "/cli/#peer_ping",
"text": "Send a kademlia ping to the specified peer. If address and port are provided the peer is directly pinged,\nif not provided the peer is located first.\n\nUsage:\n peer_ping ( node_id | --node_id= node_id ) [ address | --address= address ] [ port | --port= port ]\n\n\nOptions:\n --address= address : (str) ip address of the peer\n --port= port : (int) udp port of the peer\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"text": "Find and ping a peer by node id\n\nUsage:\n peer_ping ( node_id | --node_id= node_id )\n\n\nOptions:\n None\n\nReturns:\n (str) pong, or {'error': error message } if an error is encountered",
"title": "peer_ping"
},
{
"location": "/cli/#publish",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nUsage:\n publish ( name | --name= name ) ( bid | --bid= bid ) [--metadata= metadata ]\n [--file_path= file_path ] [--fee= fee ] [--title= title ]\n [--description= description ] [--author= author ] [--language= language ]\n [--license= license ] [--license_url= license_url ] [--thumbnail= thumbnail ]\n [--preview= preview ] [--nsfw= nsfw ] [--sources= sources ]\n [--channel_name= channel_name ] [--channel_id= channel_id ]\n [--claim_address= claim_address ] [--change_address= change_address ]\n\n\nOptions:\n --name= name : (str) name of the content\n --bid= bid : (decimal) amount to back the claim\n --metadata= metadata : (dict) ClaimDict to associate with the claim.\n --file_path= file_path : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n --fee= fee : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n --title= title : (str) title of the publication\n --description= description : (str) description of the publication\n --author= author : (str) author of the publication\n --language= language : (str) language of the publication\n --license= license : (str) publication license\n --license_url= license_url : (str) publication license url\n --thumbnail= thumbnail : (str) thumbnail url\n --preview= preview : (str) preview url\n --nsfw= nsfw : (bool) whether the content is nsfw\n --sources= sources : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n --channel_name= channel_name : (str) name of the publisher channel name in the wallet\n --channel_id= channel_id : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n --claim_address= claim_address : (str) address where the claim is sent to, if not specified\n new address will automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"text": "Make a new name claim and publish associated data to lbrynet,\nupdate over existing claim if user already has a claim for name.\n\nFields required in the final Metadata are:\n 'title'\n 'description'\n 'author'\n 'language'\n 'license'\n 'nsfw'\n\nMetadata can be set by either using the metadata argument or by setting individual arguments\nfee, title, description, author, language, license, license_url, thumbnail, preview, nsfw,\nor sources. Individual arguments will overwrite the fields specified in metadata argument.\n\nUsage:\n publish ( name | --name= name ) ( bid | --bid= bid ) [--metadata= metadata ]\n [--file_path= file_path ] [--fee= fee ] [--title= title ]\n [--description= description ] [--author= author ] [--language= language ]\n [--license= license ] [--license_url= license_url ] [--thumbnail= thumbnail ]\n [--preview= preview ] [--nsfw= nsfw ] [--sources= sources ]\n [--channel_name= channel_name ] [--channel_id= channel_id ]\n [--claim_address= claim_address ] [--change_address= change_address ]\n\n\nOptions:\n --name= name : (str) name of the content\n --bid= bid : (decimal) amount to back the claim\n --metadata= metadata : (dict) ClaimDict to associate with the claim.\n --file_path= file_path : (str) path to file to be associated with name. If provided,\n a lbry stream of this file will be used in 'sources'.\n If no path is given but a sources dict is provided,\n it will be used. If neither are provided, an\n error is raised.\n --fee= fee : (dict) Dictionary representing key fee to download content:\n {\n 'currency': currency_symbol,\n 'amount': decimal,\n 'address': str, optional\n }\n supported currencies: LBC, USD, BTC\n If an address is not provided a new one will be\n automatically generated. Default fee is zero.\n --title= title : (str) title of the publication\n --description= description : (str) description of the publication\n --author= author : (str) author of the publication\n --language= language : (str) language of the publication\n --license= license : (str) publication license\n --license_url= license_url : (str) publication license url\n --thumbnail= thumbnail : (str) thumbnail url\n --preview= preview : (str) preview url\n --nsfw= nsfw : (bool) whether the content is nsfw\n --sources= sources : (str) {'lbry_sd_hash': sd_hash} specifies sd hash of file\n --channel_name= channel_name : (str) name of the publisher channel name in the wallet\n --channel_id= channel_id : (str) claim id of the publisher channel, does not check\n for channel claim being in the wallet. This allows\n publishing to a channel where only the certificate\n private key is in the wallet.\n --claim_address= claim_address : (str) address where the claim is sent to, if not specified\n new address wil automatically be created\n\nReturns:\n (dict) Dictionary containing result of the claim\n {\n 'tx' : (str) hex encoded transaction\n 'txid' : (str) txid of resulting claim\n 'nout' : (int) nout of the resulting claim\n 'fee' : (decimal) fee paid for the claim transaction\n 'claim_id' : (str) claim ID of the resulting claim\n }",
"title": "publish"
},
{
@@ -492,7 +492,7 @@
},
{
"location": "/cli/#status",
"text": "Get daemon status\n\nUsage:\n status\n\n\nOptions:\n None\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'installation_id': (str) installation id - base58,\n 'is_running': (bool),\n 'is_first_run': bool,\n 'skipped_components': (list) [names of skipped components (str)],\n 'startup_status': { Does not include components which have been skipped\n 'database': (bool),\n 'wallet': (bool),\n 'session': (bool),\n 'dht': (bool),\n 'hash_announcer': (bool),\n 'stream_identifier': (bool),\n 'file_manager': (bool),\n 'blob_manager': (bool),\n 'blockchain_headers': (bool),\n 'peer_protocol_server': (bool),\n 'reflector': (bool),\n 'upnp': (bool),\n 'exchange_rate_manager': (bool),\n },\n 'connection_status': {\n 'code': (str) connection status code,\n 'message': (str) connection status message\n },\n 'blockchain_headers': {\n 'downloading_headers': (bool),\n 'download_progress': (float) 0-100.0\n },\n 'wallet': {\n 'blocks': (int) local blockchain height,\n 'blocks_behind': (int) remote_height - local_height,\n 'best_blockhash': (str) block hash of most recent block,\n 'is_encrypted': (bool),\n 'is_locked': (bool),\n },\n 'dht': {\n 'node_id': (str) lbry dht node id - hex encoded,\n 'peers_in_routing_table': (int) the number of peers in the routing table,\n },\n 'blob_manager': {\n 'finished_blobs': (int) number of finished blobs in the blob manager,\n },\n 'hash_announcer': {\n 'announce_queue_size': (int) number of blobs currently queued to be announced\n },\n 'file_manager': {\n 'managed_files': (int) count of files in the file manager,\n }\n }",
"text": "Get daemon status\n\nUsage:\n status [--session_status]\n\n\nOptions:\n --session_status : (bool) include session status in results\n\nReturns:\n (dict) lbrynet-daemon status\n {\n 'lbry_id': lbry peer id, base58,\n 'installation_id': installation id, base58,\n 'is_running': bool,\n 'is_first_run': bool,\n 'startup_status': {\n 'code': status code,\n 'message': status message\n },\n 'connection_status': {\n 'code': connection status code,\n 'message': connection status message\n },\n 'blockchain_status': {\n 'blocks': local blockchain height,\n 'blocks_behind': remote_height - local_height,\n 'best_blockhash': block hash of most recent block,\n },\n 'wallet_is_encrypted': bool,\n\n If given the session status option:\n 'session_status': {\n 'managed_blobs': count of blobs in the blob manager,\n 'managed_streams': count of streams in the file manager\n 'announce_queue_size': number of blobs currently queued to be announced\n 'should_announce_blobs': number of blobs that should be announced\n }\n }",
"title": "status"
},
{
@@ -567,7 +567,7 @@
},
{
"location": "/cli/#wallet_send",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nUsage:\n wallet_send ( amount | --amount= amount )\n (( address | --address= address ) | ( claim_id | --claim_id= claim_id ))\n\n\nOptions:\n --amount= amount : (Decimal) amount of credit to send\n --address= address : (str) address to send credits to\n --claim_id= claim_id : (str) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"text": "Send credits. If given an address, send credits to it. If given a claim id, send a tip\nto the owner of a claim specified by uri. A tip is a claim support where the recipient\nof the support is the claim address for the claim being supported.\n\nUsage:\n wallet_send ( amount | --amount= amount )\n (( address | --address= address ) | ( claim_id | --claim_id= claim_id ))\n\n\nOptions:\n --amount= amount : (float) amount of credit to send\n --address= address : (str) address to send credits to\n --claim_id= claim_id : (float) claim_id of the claim to send to tip to\n\nReturns:\n If sending to an address:\n (bool) true if payment successfully scheduled\n\n If sending a claim tip:\n (dict) Dictionary containing the result of the support\n {\n txid : (str) txid of resulting support claim\n nout : (int) nout of the resulting support claim\n fee : (float) fee paid for the transaction\n }",
"title": "wallet_send"
},
{
@@ -581,4 +581,4 @@
"title": "wallet_unused_address"
}
]
}
}
-26
View File
@@ -1,26 +0,0 @@
# This is an example daemon_settings.yml file.
# See https://lbry.tech/resources/daemon-settings for all configuration keys and values
share_usage_data: True
lbryum_servers:
- lbryumx1.lbry.io:50001
- lbryumx2.lbry.io:50001
blockchain_name: lbrycrd_main
data_dir: /home/lbry/.lbrynet
download_directory: /home/lbry/downloads
delete_blobs_on_remove: True
dht_node_port: 4444
peer_port: 3333
use_upnp: True
use_auth_http: True
use_https: True
#components_to_skip:
# - peer_protocol_server
# - hash_announcer
# - dht
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.6 KiB

+1 -5
View File
@@ -1,10 +1,6 @@
import logging
from lbrynet.custom_logger import install_logger
__name__ = "lbrynet"
__version__ = "0.30.3"
__version__ = "0.20.3"
version = tuple(__version__.split('.'))
install_logger()
logging.getLogger(__name__).addHandler(logging.NullHandler())
@@ -4,8 +4,8 @@ import logging
import treq
from twisted.internet import defer, task
from lbrynet import conf, utils
from lbrynet.extras import looping_call_manager, system_info
from lbrynet import conf
from lbrynet.core import looping_call_manager, utils, system_info
# Things We Track
SERVER_STARTUP = 'Server Startup'
@@ -18,15 +18,13 @@ HEARTBEAT = 'Heartbeat'
CLAIM_ACTION = 'Claim Action' # publish/create/update/abandon
NEW_CHANNEL = 'New Channel'
CREDITS_SENT = 'Credits Sent'
NEW_DOWNLOAD_STAT = 'Download'
UPNP_SETUP = "UPnP Setup"
BLOB_BYTES_UPLOADED = 'Blob Bytes Uploaded'
log = logging.getLogger(__name__)
class Manager:
class Manager(object):
def __init__(self, analytics_api, context=None, installation_id=None, session_id=None):
self.analytics_api = analytics_api
self._tracked_data = collections.defaultdict(list)
@@ -43,40 +41,6 @@ class Manager:
return cls(api)
# Things We Track
def send_new_download_start(self, download_id, name, claim_dict):
self._send_new_download_stats("start", download_id, name, claim_dict)
def send_new_download_success(self, download_id, name, claim_dict):
self._send_new_download_stats("success", download_id, name, claim_dict)
def send_new_download_fail(self, download_id, name, claim_dict, e):
self._send_new_download_stats("failure", download_id, name, claim_dict, {
'name': type(e).__name__ if hasattr(type(e), "__name__") else str(type(e)),
'message': str(e),
})
def _send_new_download_stats(self, action, download_id, name, claim_dict, e=None):
self.analytics_api.track({
'userId': 'lbry', # required, see https://segment.com/docs/sources/server/http/#track
'event': NEW_DOWNLOAD_STAT,
'properties': self._event_properties({
'download_id': download_id,
'name': name,
'sd_hash': None if not claim_dict else claim_dict.source_hash.decode(),
'action': action,
'error': e,
}),
'context': self.context,
'timestamp': utils.isonow(),
})
def send_upnp_setup_success_fail(self, success, status):
self.analytics_api.track(
self._event(UPNP_SETUP, {
'success': success,
'status': status,
})
)
def send_server_startup(self):
self.analytics_api.track(self._event(SERVER_STARTUP))
@@ -136,8 +100,8 @@ class Manager:
def _get_looping_calls(self):
return [
('send_heartbeat', self._send_heartbeat, 300),
('update_tracked_metrics', self._update_tracked_metrics, 600),
('send_heartbeat', self._send_heartbeat, 60),
('update_tracked_metrics', self._update_tracked_metrics, 300),
]
def _setup_looping_calls(self):
@@ -194,7 +158,7 @@ class Manager:
@staticmethod
def _download_properties(id_, name, claim_dict=None, report=None):
sd_hash = None if not claim_dict else claim_dict.source_hash.decode()
sd_hash = None if not claim_dict else claim_dict.source_hash
p = {
'download_id': id_,
'name': name,
@@ -213,26 +177,34 @@ class Manager:
return {
'download_id': id_,
'name': name,
'stream_info': claim_dict.source_hash.decode(),
'stream_info': claim_dict.source_hash,
'error': error_name(error),
'reason': str(error),
'reason': error.message,
'report': report
}
@staticmethod
def _make_context(platform, wallet):
# see https://segment.com/docs/spec/common/#context
# they say they'll ignore fields outside the spec, but evidently they don't
context = {
'app': {
'name': 'lbrynet',
'version': platform['lbrynet_version'],
'python_version': platform['python_version'],
'build': platform['build'],
'wallet': {
'name': wallet,
'version': platform['lbryum_version'] if wallet == conf.LBRYUM_WALLET else None
},
},
# TODO: expand os info to give linux/osx specific info
'os': {
'name': platform['os_system'],
'version': platform['os_release']
},
'library': {
'name': 'lbrynet-analytics',
'version': '1.0.0'
},
}
if 'desktop' in platform and 'distro' in platform:
context['os']['desktop'] = platform['desktop']
@@ -247,7 +219,7 @@ class Manager:
callback(maybe_deferred, *args, **kwargs)
class Api:
class Api(object):
def __init__(self, cookies, url, write_key, enabled):
self.cookies = cookies
self.url = url
+1
View File
@@ -0,0 +1 @@
import paths
+26
View File
@@ -0,0 +1,26 @@
# App root files dir
# https://developer.android.com/reference/android/content/ContextWrapper.html#getFilesDir%28%29
def android_files_dir():
return None
# Base internal storage path
def android_internal_storage_dir():
return None
# Base external (SD card, if present) storage path
def android_external_storage_dir():
return None
# Internal device storage (private app folder)
# https://developer.android.com/reference/android/content/ContextWrapper.html#getExternalFilesDirs(java.lang.String)
def android_app_internal_storage_dir():
return None
# External (app folder on SD card, if present) storage
# https://developer.android.com/reference/android/content/ContextWrapper.html#getExternalFilesDirs(java.lang.String)
def android_app_external_storage_dir():
return None
+4
View File
@@ -0,0 +1,4 @@
from blob_file import BlobFile
from creator import BlobFileCreator
from writer import HashBlobWriter
from reader import HashBlobReader
+11 -29
View File
@@ -1,10 +1,10 @@
import os
import logging
import os
from twisted.internet import defer, threads
from twisted.web.client import FileBodyProducer
from twisted.python.failure import Failure
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbrynet.p2p.Error import DownloadCanceledError, InvalidDataError, InvalidBlobHashError
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError, InvalidBlobHashError
from lbrynet.core.utils import is_valid_blobhash
from lbrynet.blob.writer import HashBlobWriter
from lbrynet.blob.reader import HashBlobReader
@@ -12,26 +12,8 @@ log = logging.getLogger(__name__)
MAX_BLOB_SIZE = 2 * 2 ** 20
# digest_size is in bytes, and blob hashes are hex encoded
blobhash_length = get_lbry_hash_obj().digest_size * 2
def is_valid_hashcharacter(char):
return char in "0123456789abcdef"
def is_valid_blobhash(blobhash):
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: True/False
"""
return len(blobhash) == blobhash_length and all(is_valid_hashcharacter(l) for l in blobhash)
class BlobFile:
class BlobFile(object):
"""
A chunk of data available on the network which is specified by a hashsum
@@ -70,7 +52,7 @@ class BlobFile:
def open_for_writing(self, peer):
"""
open a blob file to be written by peer, supports concurrent
writers, as long as they are from different peers.
writers, as long as they are from differnt peers.
returns tuple of (writer, finished_deferred)
@@ -78,12 +60,12 @@ class BlobFile:
finished_deferred - deferred that is fired when write is finished and returns
a instance of itself as HashBlob
"""
if peer not in self.writers:
if not peer in self.writers:
log.debug("Opening %s to be written by %s", str(self), str(peer))
finished_deferred = defer.Deferred()
writer = HashBlobWriter(self.get_length, self.writer_finished)
self.writers[peer] = (writer, finished_deferred)
return writer, finished_deferred
return (writer, finished_deferred)
log.warning("Tried to download the same file twice simultaneously from the same peer")
return None, None
@@ -133,7 +115,7 @@ class BlobFile:
def verified(self):
"""
Protect verified from being modified by other classes.
verified is True if a write to a blob has completed successfully,
verified is True if a write to a blob has completed succesfully,
or a blob has been read to have the same length as specified
in init
"""
@@ -167,7 +149,7 @@ class BlobFile:
def writer_finished(self, writer, err=None):
def fire_finished_deferred():
self._verified = True
for p, (w, finished_deferred) in list(self.writers.items()):
for p, (w, finished_deferred) in self.writers.items():
if w == writer:
del self.writers[p]
finished_deferred.callback(self)
@@ -178,7 +160,7 @@ class BlobFile:
return False
def errback_finished_deferred(err):
for p, (w, finished_deferred) in list(self.writers.items()):
for p, (w, finished_deferred) in self.writers.items():
if w == writer:
del self.writers[p]
finished_deferred.errback(err)
@@ -200,7 +182,7 @@ class BlobFile:
if writer.len_so_far != self.length:
err_string = "blob length is %i vs expected %i" % (writer.len_so_far, self.length)
else:
err_string = f"blob hash is {writer.blob_hash} vs expected {self.blob_hash}"
err_string = "blob hash is %s vs expected %s" % (writer.blob_hash, self.blob_hash)
errback_finished_deferred(Failure(InvalidDataError(err_string)))
d = defer.succeed(None)
else:
+2 -2
View File
@@ -3,12 +3,12 @@ import logging
from io import BytesIO
from twisted.internet import defer
from twisted.web.client import FileBodyProducer
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbrynet.core.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
class BlobFileCreator:
class BlobFileCreator(object):
"""
This class is used to create blobs on the local filesystem
when we do not know the blob hash beforehand (i.e, when creating
+4 -2
View File
@@ -3,7 +3,7 @@ import logging
log = logging.getLogger(__name__)
class HashBlobReader:
class HashBlobReader(object):
"""
This is a file like reader class that supports
read(size) and close()
@@ -15,7 +15,7 @@ class HashBlobReader:
def __del__(self):
if self.finished_cb_d is None:
log.warning("Garbage collection was called, but reader for %s was not closed yet",
log.warn("Garbage collection was called, but reader for %s was not closed yet",
self.read_handle.name)
self.close()
@@ -28,3 +28,5 @@ class HashBlobReader:
return
self.read_handle.close()
self.finished_cb_d = self.finished_cb(self)
+5 -5
View File
@@ -1,13 +1,13 @@
import logging
from io import BytesIO
from twisted.python.failure import Failure
from lbrynet.p2p.Error import DownloadCanceledError, InvalidDataError
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbrynet.core.Error import DownloadCanceledError, InvalidDataError
from lbrynet.core.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
class HashBlobWriter:
class HashBlobWriter(object):
def __init__(self, length_getter, finished_cb):
self.write_handle = BytesIO()
self.length_getter = length_getter
@@ -18,7 +18,7 @@ class HashBlobWriter:
def __del__(self):
if self.finished_cb_d is None:
log.warning("Garbage collection was called, but writer was not closed yet")
log.warn("Garbage collection was called, but writer was not closed yet")
self.close()
@property
@@ -27,7 +27,7 @@ class HashBlobWriter:
def write(self, data):
if self.write_handle is None:
log.warning("writer has already been closed")
log.exception("writer has already been closed")
raise IOError('I/O operation on closed file')
self._hashsum.update(data)
+1 -1
View File
@@ -1,2 +1,2 @@
# don't touch this. CI server changes this during build/deployment
# dont touch this. CI server changes this during build/deployment
BUILD = "dev"
+181 -187
View File
@@ -1,15 +1,24 @@
import base58
import json
import logging
import os
import re
import sys
import typing
import json
import logging
import envparse
import base58
import yaml
import envparse
from appdirs import user_data_dir, user_config_dir
from lbrynet import utils
from lbrynet.p2p.Error import InvalidCurrencyError
from lbrynet.core import utils
from lbrynet.core.Error import InvalidCurrencyError, NoSuchDirectoryError
from lbrynet.androidhelpers.paths import (
android_internal_storage_dir,
android_app_internal_storage_dir
)
try:
from lbrynet.winhelpers.knownpaths import get_path, FOLDERID, UserHandle
except (ImportError, ValueError, NameError):
# Android platform: NameError: name 'c_wchar' is not defined
pass
log = logging.getLogger(__name__)
@@ -20,7 +29,6 @@ ENV_NAMESPACE = 'LBRY_'
LBRYCRD_WALLET = 'lbrycrd'
LBRYUM_WALLET = 'lbryum'
PTC_WALLET = 'ptc'
TORBA_WALLET = 'torba'
PROTOCOL_PREFIX = 'lbry'
APP_NAME = 'LBRY'
@@ -50,68 +58,108 @@ settings_encoders = {
'.yml': yaml.safe_dump
}
# set by CLI when the user specifies an alternate config file path
conf_file = None
def _win_path_to_bytes(path):
"""
Encode Windows paths to string. appdirs.user_data_dir()
on windows will return unicode path, unlike other platforms
which returns string. This will cause problems
because we use strings for filenames and combining them with
os.path.join() will result in errors.
"""
for encoding in ('ASCII', 'MBCS'):
try:
return path.encode(encoding)
except (UnicodeEncodeError, LookupError):
pass
return path
def _get_old_directories(platform_type):
directories = {}
if platform_type == WINDOWS:
appdata = get_path(FOLDERID.RoamingAppData, UserHandle.current)
directories['data'] = os.path.join(appdata, 'lbrynet')
directories['lbryum'] = os.path.join(appdata, 'lbryum')
directories['download'] = get_path(FOLDERID.Downloads, UserHandle.current)
elif platform_type == DARWIN:
directories['data'] = user_data_dir('LBRY')
directories['lbryum'] = os.path.expanduser('~/.lbryum')
directories['download'] = os.path.expanduser('~/Downloads')
elif platform_type == LINUX:
directories['data'] = os.path.expanduser('~/.lbrynet')
directories['lbryum'] = os.path.expanduser('~/.lbryum')
directories['download'] = os.path.expanduser('~/Downloads')
else:
raise ValueError('unknown platform value')
return directories
def _get_new_directories(platform_type):
directories = {}
if platform_type == ANDROID:
directories['data'] = '%s/lbrynet' % android_app_internal_storage_dir()
directories['lbryum'] = '%s/lbryum' % android_app_internal_storage_dir()
directories['download'] = '%s/Download' % android_internal_storage_dir()
elif platform_type == WINDOWS:
directories['data'] = user_data_dir('lbrynet', 'lbry')
directories['lbryum'] = user_data_dir('lbryum', 'lbry')
directories['download'] = get_path(FOLDERID.Downloads, UserHandle.current)
elif platform_type == DARWIN:
directories = _get_old_directories(platform_type)
elif platform_type == LINUX:
directories['data'] = user_data_dir('lbry/lbrynet')
directories['lbryum'] = user_data_dir('lbry/lbryum')
try:
with open(os.path.join(user_config_dir(), 'user-dirs.dirs'), 'r') as xdg:
down_dir = re.search(r'XDG_DOWNLOAD_DIR=(.+)', xdg.read()).group(1)
down_dir = re.sub('\$HOME', os.getenv('HOME'), down_dir)
directories['download'] = re.sub('\"', '', down_dir)
except EnvironmentError:
directories['download'] = os.getenv('XDG_DOWNLOAD_DIR')
if not directories['download']:
directories['download'] = os.path.expanduser('~/Downloads')
else:
raise ValueError('unknown platform value')
return directories
if 'ANDROID_ARGUMENT' in os.environ:
# https://github.com/kivy/kivy/blob/master/kivy/utils.py#L417-L421
platform = ANDROID
dirs = _get_new_directories(ANDROID)
elif 'darwin' in sys.platform:
platform = DARWIN
dirs = _get_old_directories(DARWIN)
elif 'win' in sys.platform:
platform = WINDOWS
if os.path.isdir(_get_old_directories(WINDOWS)['data']) or \
os.path.isdir(_get_old_directories(WINDOWS)['lbryum']):
dirs = _get_old_directories(WINDOWS)
else:
dirs = _get_new_directories(WINDOWS)
dirs['data'] = _win_path_to_bytes(dirs['data'])
dirs['lbryum'] = _win_path_to_bytes(dirs['lbryum'])
dirs['download'] = _win_path_to_bytes(dirs['download'])
else:
platform = LINUX
if os.path.isdir(_get_old_directories(LINUX)['data']) or \
os.path.isdir(_get_old_directories(LINUX)['lbryum']):
dirs = _get_old_directories(LINUX)
else:
dirs = _get_new_directories(LINUX)
default_data_dir = dirs['data']
default_lbryum_dir = dirs['lbryum']
default_download_dir = dirs['download']
ICON_PATH = 'icons' if platform is WINDOWS else 'app.icns'
def get_windows_directories() -> typing.Tuple[str, str, str]:
from lbrynet.winpaths import get_path, FOLDERID, UserHandle
download_dir = get_path(FOLDERID.Downloads, UserHandle.current)
# old
appdata = get_path(FOLDERID.RoamingAppData, UserHandle.current)
data_dir = os.path.join(appdata, 'lbrynet')
lbryum_dir = os.path.join(appdata, 'lbryum')
if os.path.isdir(data_dir) or os.path.isdir(lbryum_dir):
return data_dir, lbryum_dir, download_dir
# new
data_dir = user_data_dir('lbrynet', 'lbry')
lbryum_dir = user_data_dir('lbryum', 'lbry')
download_dir = get_path(FOLDERID.Downloads, UserHandle.current)
return data_dir, lbryum_dir, download_dir
def get_darwin_directories() -> typing.Tuple[str, str, str]:
data_dir = user_data_dir('LBRY')
lbryum_dir = os.path.expanduser('~/.lbryum')
download_dir = os.path.expanduser('~/Downloads')
return data_dir, lbryum_dir, download_dir
def get_linux_directories() -> typing.Tuple[str, str, str]:
download_dir = None
try:
with open(os.path.join(user_config_dir(), 'user-dirs.dirs'), 'r') as xdg:
down_dir = re.search(r'XDG_DOWNLOAD_DIR=(.+)', xdg.read()).group(1)
down_dir = re.sub('\$HOME', os.getenv('HOME'), down_dir)
download_dir = re.sub('\"', '', down_dir)
except EnvironmentError:
download_dir = os.getenv('XDG_DOWNLOAD_DIR')
if not download_dir:
download_dir = os.path.expanduser('~/Downloads')
# old
data_dir = os.path.expanduser('~/.lbrynet')
lbryum_dir = os.path.expanduser('~/.lbryum')
if os.path.isdir(data_dir) or os.path.isdir(lbryum_dir):
return data_dir, lbryum_dir, download_dir
# new
return user_data_dir('lbry/lbrynet'), user_data_dir('lbry/lbryum'), download_dir
def server_port(server_and_port):
server, port = server_and_port.split(':')
return server, int(port)
@@ -122,7 +170,7 @@ def server_list(servers):
def server_list_reverse(servers):
return [f"{server}:{port}" for server, port in servers]
return ["%s:%s" % (server, port) for server, port in servers]
class Env(envparse.Env):
@@ -134,11 +182,11 @@ class Env(envparse.Env):
self._convert_key(key): self._convert_value(value)
for key, value in schema.items()
}
super().__init__(**my_schema)
envparse.Env.__init__(self, **my_schema)
def __call__(self, key, *args, **kwargs):
my_key = self._convert_key(key)
return super().__call__(my_key, *args, **kwargs)
return super(Env, self).__call__(my_key, *args, **kwargs)
@staticmethod
def _convert_key(key):
@@ -196,11 +244,6 @@ FIXED_SETTINGS = {
}
ADJUSTABLE_SETTINGS = {
'data_dir': (str, ''), # these blank defaults will be updated to OS specific defaults
'wallet_dir': (str, ''),
'lbryum_wallet_dir': (str, ''), # to be deprecated
'download_directory': (str, ''),
# By default, daemon will block all cross origin requests
# but if this is set, this value will be used for the
# Access-Control-Allow-Origin. For example
@@ -219,15 +262,17 @@ ADJUSTABLE_SETTINGS = {
# will not be made automatically)
'auto_renew_claim_height_delta': (int, 0),
'cache_time': (int, 150),
'data_dir': (str, default_data_dir),
'data_rate': (float, .0001), # points/megabyte
'delete_blobs_on_remove': (bool, True),
'dht_node_port': (int, 4444),
'download_directory': (str, default_download_dir),
'download_timeout': (int, 180),
'download_mirrors': (list, ['blobs.lbry.io']),
'is_generous_host': (bool, True),
'announce_head_blobs_only': (bool, True),
'concurrent_announcers': (int, DEFAULT_CONCURRENT_ANNOUNCERS),
'known_dht_nodes': (list, DEFAULT_DHT_NODES, server_list, server_list_reverse),
'lbryum_wallet_dir': (str, default_lbryum_dir),
'max_connections_per_stream': (int, 5),
'seek_head_blob_first': (bool, True),
# TODO: writing json on the cmd line is a pain, come up with a nicer
@@ -244,13 +289,12 @@ ADJUSTABLE_SETTINGS = {
# event the initial upload failed or was disconnected part way through, provided the auto_re_reflect_interval > 0)
'reflect_uploads': (bool, True),
'auto_re_reflect_interval': (int, 86400), # set to 0 to disable
'reflector_servers': (list, [('reflector.lbry.io', 5566)], server_list, server_list_reverse),
'run_reflector_server': (bool, False), # adds `reflector` to components_to_skip unless True
'reflector_servers': (list, [('reflector2.lbry.io', 5566)], server_list, server_list_reverse),
'run_reflector_server': (bool, False),
'sd_download_timeout': (int, 3),
'share_usage_data': (bool, True), # whether to share usage stats and diagnostic info with LBRY
'peer_search_timeout': (int, 60),
'peer_search_timeout': (int, 30),
'use_auth_http': (bool, False),
'use_https': (bool, False),
'use_upnp': (bool, True),
'use_keyring': (bool, False),
'wallet': (str, LBRYUM_WALLET),
@@ -262,34 +306,16 @@ ADJUSTABLE_SETTINGS = {
}
optional_str = typing.Optional[str]
class Config(object):
def __init__(self, fixed_defaults, adjustable_defaults, persisted_settings=None,
environment=None, cli_settings=None):
class Config:
def __init__(self, fixed_defaults, adjustable_defaults: typing.Dict, persisted_settings=None, environment=None,
cli_settings=None, data_dir: optional_str = None, wallet_dir: optional_str = None,
download_dir: optional_str = None, file_name: optional_str = None):
self._installation_id = None
self._session_id = base58.b58encode(utils.generate_id()).decode()
self._session_id = base58.b58encode(utils.generate_id())
self._node_id = None
self._fixed_defaults = fixed_defaults
# copy the default adjustable settings
self._adjustable_defaults = {k: v for k, v in adjustable_defaults.items()}
# set the os specific default directories
if platform is WINDOWS:
self.default_data_dir, self.default_wallet_dir, self.default_download_dir = get_windows_directories()
elif platform is DARWIN:
self.default_data_dir, self.default_wallet_dir, self.default_download_dir = get_darwin_directories()
elif platform is LINUX:
self.default_data_dir, self.default_wallet_dir, self.default_download_dir = get_linux_directories()
else:
assert None not in [data_dir, wallet_dir, download_dir]
self.default_data_dir = data_dir
self.default_wallet_dir = wallet_dir
self.default_download_dir = download_dir
self._adjustable_defaults = adjustable_defaults
self._data = {
TYPE_DEFAULT: {}, # defaults
@@ -311,7 +337,7 @@ class Config:
self._data[TYPE_DEFAULT].update(self._fixed_defaults)
self._data[TYPE_DEFAULT].update(
{k: v[1] for (k, v) in self._adjustable_defaults.items()})
{k: v[1] for (k, v) in self._adjustable_defaults.iteritems()})
if persisted_settings is None:
persisted_settings = {}
@@ -326,37 +352,12 @@ class Config:
cli_settings = {}
self._validate_settings(cli_settings)
self._data[TYPE_CLI].update(cli_settings)
self.file_name = file_name or 'daemon_settings.yml'
@property
def data_dir(self) -> optional_str:
data_dir = self.get('data_dir')
if not data_dir:
return
return os.path.expanduser(os.path.expandvars(data_dir))
@property
def download_dir(self) -> optional_str:
download_dir = self.get('download_directory')
if not download_dir:
return
return os.path.expanduser(os.path.expandvars(download_dir))
@property
def wallet_dir(self) -> optional_str:
if self.get('lbryum_wallet_dir') and not self.get('wallet_dir'):
log.warning("'lbryum_wallet_dir' setting will be deprecated, please update to 'wallet_dir'")
self['wallet_dir'] = self['lbryum_wallet_dir']
wallet_dir = self.get('wallet_dir')
if not wallet_dir:
return
return os.path.expanduser(os.path.expandvars(wallet_dir))
def __repr__(self):
return self.get_current_settings_dict().__repr__()
def __iter__(self):
for k in self._data[TYPE_DEFAULT].keys():
for k in self._data[TYPE_DEFAULT].iterkeys():
yield k
def __getitem__(self, name):
@@ -380,7 +381,7 @@ class Config:
def _assert_valid_data_type(self, data_type):
if data_type not in self._data:
raise KeyError(f'{data_type} in is not a valid data type')
raise KeyError('{} in is not a valid data type'.format(data_type))
def get_valid_setting_names(self):
return self._data[TYPE_DEFAULT].keys()
@@ -390,7 +391,7 @@ class Config:
def _assert_valid_setting(self, name):
if not self._is_valid_setting(name):
raise KeyError(f'{name} is not a valid setting')
raise KeyError('{} is not a valid setting'.format(name))
def _validate_settings(self, data):
invalid_settings = set(data.keys()) - set(self.get_valid_setting_names())
@@ -400,7 +401,7 @@ class Config:
def _assert_editable_setting(self, name):
self._assert_valid_setting(name)
if name in self._fixed_defaults:
raise ValueError(f'{name} is not an editable setting')
raise ValueError('{} is not an editable setting'.format(name))
def _assert_valid_setting_value(self, name, value):
if name == "max_key_fee":
@@ -410,7 +411,7 @@ class Config:
elif name == "download_directory":
directory = str(value)
if not os.path.exists(directory):
log.warning("download directory '%s' does not exist", directory)
raise NoSuchDirectoryError(directory)
def is_default(self, name):
"""Check if a config value is wasn't specified by the user
@@ -455,7 +456,7 @@ class Config:
for possible_data_type in self._search_order:
if name in self._data[possible_data_type]:
return self._data[possible_data_type][name]
raise KeyError(f'{name} is not a valid setting')
raise KeyError('{} is not a valid setting'.format(name))
def set(self, name, value, data_types=(TYPE_RUNTIME,)):
"""Set a config value
@@ -479,7 +480,7 @@ class Config:
self._data[data_type][name] = value
def update(self, updated_settings, data_types=(TYPE_RUNTIME,)):
for k, v in updated_settings.items():
for k, v in updated_settings.iteritems():
try:
self.set(k, v, data_types=data_types)
except (KeyError, AssertionError):
@@ -493,26 +494,28 @@ class Config:
def get_adjustable_settings_dict(self):
return {
key: val for key, val in self.get_current_settings_dict().items()
key: val for key, val in self.get_current_settings_dict().iteritems()
if key in self._adjustable_defaults
}
def save_conf_file_settings(self):
if conf_file:
path = conf_file
else:
path = self.get_conf_filename()
# reverse the conversions done after loading the settings from the conf
# file
rev = self._convert_conf_file_lists_reverse(self._data[TYPE_PERSISTED])
ext = os.path.splitext(self.file_name)[1]
ext = os.path.splitext(path)[1]
encoder = settings_encoders.get(ext, False)
if not encoder:
raise ValueError('Unknown settings format: {}. Available formats: {}'
.format(ext, list(settings_encoders.keys())))
with open(os.path.join(self.data_dir, self.file_name), 'w') as settings_file:
assert encoder is not False, 'Unknown settings format %s' % ext
with open(path, 'w') as settings_file:
settings_file.write(encoder(rev))
@staticmethod
def _convert_conf_file_lists_reverse(converted):
rev = {}
for k in converted.keys():
for k in converted.iterkeys():
if k in ADJUSTABLE_SETTINGS and len(ADJUSTABLE_SETTINGS[k]) == 4:
rev[k] = ADJUSTABLE_SETTINGS[k][3](converted[k])
else:
@@ -522,7 +525,7 @@ class Config:
@staticmethod
def _convert_conf_file_lists(decoded):
converted = {}
for k, v in decoded.items():
for k, v in decoded.iteritems():
if k in ADJUSTABLE_SETTINGS and len(ADJUSTABLE_SETTINGS[k]) >= 3:
converted[k] = ADJUSTABLE_SETTINGS[k][2](v)
else:
@@ -534,29 +537,25 @@ class Config:
settings.node_id = settings.get_node_id()
def load_conf_file_settings(self):
path = os.path.join(self.data_dir or self.default_data_dir, self.file_name)
if os.path.isfile(path):
self._read_conf_file(path)
self['data_dir'] = self.data_dir or self.default_data_dir
self['download_directory'] = self.download_dir or self.default_download_dir
self['wallet_dir'] = self.wallet_dir or self.default_wallet_dir
# initialize members depending on config file
self.initialize_post_conf_load()
def _read_conf_file(self, path):
if not path or not os.path.exists(path):
raise FileNotFoundError(path)
if conf_file:
path = conf_file
else:
path = self.get_conf_filename()
ext = os.path.splitext(path)[1]
decoder = settings_decoders.get(ext, False)
if not decoder:
raise ValueError('Unknown settings format: {}. Available formats: {}'
.format(ext, list(settings_decoders.keys())))
with open(path, 'r') as settings_file:
data = settings_file.read()
decoded = self._fix_old_conf_file_settings(decoder(data))
log.info('Loaded settings file: %s', path)
self._validate_settings(decoded)
self._data[TYPE_PERSISTED].update(self._convert_conf_file_lists(decoded))
assert decoder is not False, 'Unknown settings format %s' % ext
try:
with open(path, 'r') as settings_file:
data = settings_file.read()
decoded = self._fix_old_conf_file_settings(decoder(data))
log.info('Loaded settings file: %s', path)
self._validate_settings(decoded)
self._data[TYPE_PERSISTED].update(self._convert_conf_file_lists(decoded))
except (IOError, OSError) as err:
log.info('%s: Failed to update settings from %s', err, path)
#initialize members depending on config file
self.initialize_post_conf_load()
def _fix_old_conf_file_settings(self, settings_dict):
if 'API_INTERFACE' in settings_dict:
@@ -570,7 +569,7 @@ class Config:
if 'share_debug_info' in settings_dict:
settings_dict['share_usage_data'] = settings_dict['share_debug_info']
del settings_dict['share_debug_info']
for key in list(settings_dict.keys()):
for key in settings_dict.keys():
if not self._is_valid_setting(key):
log.warning('Ignoring invalid conf file setting: %s', key)
del settings_dict[key]
@@ -580,15 +579,9 @@ class Config:
# although there is a risk of a race condition here we don't
# expect there to be multiple processes accessing this
# directory so the risk can be ignored
if not os.path.isdir(self.data_dir):
os.makedirs(self.data_dir)
if not os.path.isdir(os.path.join(self.data_dir, "blobfiles")):
os.makedirs(os.path.join(self.data_dir, "blobfiles"))
return self.data_dir
def ensure_wallet_dir(self):
if not os.path.isdir(self.wallet_dir):
os.makedirs(self.wallet_dir)
if not os.path.isdir(self['data_dir']):
os.makedirs(self['data_dir'])
return self['data_dir']
def get_log_filename(self):
"""
@@ -597,18 +590,26 @@ class Config:
"""
return os.path.join(self.ensure_data_dir(), self['LOG_FILE_NAME'])
def get_api_connection_string(self, user: str = None, password: str = None) -> str:
return 'http%s://%s%s:%i/%s' % (
"" if not self['use_https'] else "s",
"" if not (user and password) else f"{user}:{password}@",
self['api_host'],
self['api_port'],
self['API_ADDRESS']
)
def get_api_connection_string(self):
return 'http://%s:%i/%s' % (self['api_host'], self['api_port'], self['API_ADDRESS'])
def get_ui_address(self):
return 'http://%s:%i' % (self['api_host'], self['api_port'])
def get_db_revision_filename(self):
return os.path.join(self.ensure_data_dir(), self['DB_REVISION_FILE_NAME'])
def get_conf_filename(self):
data_dir = self.ensure_data_dir()
yml_path = os.path.join(data_dir, 'daemon_settings.yml')
json_path = os.path.join(data_dir, 'daemon_settings.json')
if os.path.isfile(yml_path):
return yml_path
elif os.path.isfile(json_path):
return json_path
else:
return yml_path
def get_installation_id(self):
install_id_filename = os.path.join(self.ensure_data_dir(), "install_id")
if not self._installation_id:
@@ -616,7 +617,7 @@ class Config:
with open(install_id_filename, "r") as install_id_file:
self._installation_id = str(install_id_file.read()).strip()
if not self._installation_id:
self._installation_id = base58.b58encode(utils.generate_id()).decode()
self._installation_id = base58.b58encode(utils.generate_id())
with open(install_id_filename, "w") as install_id_file:
install_id_file.write(self._installation_id)
return self._installation_id
@@ -630,19 +631,20 @@ class Config:
if not self._node_id:
self._node_id = utils.generate_id()
with open(node_id_filename, "w") as node_id_file:
node_id_file.write(base58.b58encode(self._node_id).decode())
node_id_file.write(base58.b58encode(self._node_id))
return self._node_id
def get_session_id(self):
return self._session_id
settings: Config = None
# type: Config
settings = None
def get_default_env():
env_defaults = {}
for k, v in ADJUSTABLE_SETTINGS.items():
for k, v in ADJUSTABLE_SETTINGS.iteritems():
if len(v) == 3:
env_defaults[k] = (v[0], None, v[2])
elif len(v) == 4:
@@ -652,18 +654,10 @@ def get_default_env():
return Env(**env_defaults)
def initialize_settings(load_conf_file: typing.Optional[bool] = True,
data_dir: optional_str = None, wallet_dir: optional_str = None,
download_dir: optional_str = None):
def initialize_settings(load_conf_file=True):
global settings
if settings is None:
settings = Config(FIXED_SETTINGS, ADJUSTABLE_SETTINGS,
environment=get_default_env(), data_dir=data_dir, wallet_dir=wallet_dir,
download_dir=download_dir)
environment=get_default_env())
if load_conf_file:
settings.load_conf_file_settings()
settings['data_dir'] = settings.data_dir or settings.default_data_dir
settings['download_directory'] = settings.download_dir or settings.default_download_dir
settings['wallet_dir'] = settings.wallet_dir or settings.default_wallet_dir
settings.ensure_data_dir()
settings.ensure_wallet_dir()
@@ -1,14 +1,15 @@
import logging
import random
import time
from decimal import Decimal
from twisted.internet import defer
from twisted.internet.task import LoopingCall
from decimal import Decimal
log = logging.getLogger(__name__)
class BlobAvailabilityTracker:
class BlobAvailabilityTracker(object):
"""
Class to track peer counts for known blobs, and to discover new popular blobs
@@ -1,4 +1,4 @@
class BlobInfo:
class BlobInfo(object):
"""
This structure is used to represent the metadata of a blob.
@@ -16,3 +16,4 @@ class BlobInfo:
self.blob_hash = blob_hash
self.blob_num = blob_num
self.length = length
@@ -1,15 +1,15 @@
import logging
import os
from binascii import unhexlify
from sqlite3 import IntegrityError
from twisted.internet import threads, defer
from twisted.internet import threads, defer, task
from lbrynet import conf
from lbrynet.blob.blob_file import BlobFile
from lbrynet.blob.creator import BlobFileCreator
log = logging.getLogger(__name__)
class DiskBlobManager:
class DiskBlobManager(object):
def __init__(self, blob_dir, storage, node_datastore=None):
"""
This class stores blobs on the hard disk
@@ -26,14 +26,22 @@ class DiskBlobManager:
self.blobs = {}
self.blob_hashes_to_delete = {} # {blob_hash: being_deleted (True/False)}
self.check_should_announce_lc = None
if conf.settings['run_reflector_server']: # TODO: move this looping call to SQLiteStorage
self.check_should_announce_lc = task.LoopingCall(self.storage.verify_will_announce_all_head_and_sd_blobs)
@defer.inlineCallbacks
def setup(self):
if self.check_should_announce_lc and not self.check_should_announce_lc.running:
self.check_should_announce_lc.start(600)
if self._node_datastore is not None:
raw_blob_hashes = yield self.storage.get_all_finished_blobs()
self._node_datastore.completed_blobs.update(raw_blob_hashes)
defer.returnValue(True)
def stop(self):
if self.check_should_announce_lc and self.check_should_announce_lc.running:
self.check_should_announce_lc.stop()
return defer.succeed(True)
def get_blob(self, blob_hash, length=None):
@@ -41,7 +49,7 @@ class DiskBlobManager:
blob that is already on the hard disk
"""
if length is not None and not isinstance(length, int):
raise Exception("invalid length type: {} ({})".format(length, str(type(length))))
raise Exception("invalid length type: %s (%s)" % (length, str(type(length))))
if blob_hash in self.blobs:
return defer.succeed(self.blobs[blob_hash])
return self._make_new_blob(blob_hash, length)
@@ -61,7 +69,7 @@ class DiskBlobManager:
blob.blob_hash, blob.length, next_announce_time, should_announce
)
if self._node_datastore is not None:
self._node_datastore.completed_blobs.add(unhexlify(blob.blob_hash))
self._node_datastore.completed_blobs.add(blob.blob_hash.decode('hex'))
def completed_blobs(self, blobhashes_to_check):
return self._completed_blobs(blobhashes_to_check)
@@ -97,11 +105,9 @@ class DiskBlobManager:
def delete_blobs(self, blob_hashes):
bh_to_delete_from_db = []
for blob_hash in blob_hashes:
if not blob_hash:
continue
if self._node_datastore is not None:
try:
self._node_datastore.completed_blobs.remove(unhexlify(blob_hash))
self._node_datastore.completed_blobs.remove(blob_hash.decode('hex'))
except KeyError:
pass
try:
@@ -114,7 +120,7 @@ class DiskBlobManager:
try:
yield self.storage.delete_blobs_from_db(bh_to_delete_from_db)
except IntegrityError as err:
if str(err) != "FOREIGN KEY constraint failed":
if err.message != "FOREIGN KEY constraint failed":
raise err
@defer.inlineCallbacks
@@ -1,4 +1,4 @@
class DownloadOptionChoice:
class DownloadOptionChoice(object):
"""A possible choice that can be picked for some option.
An option can have one or more choices that can be picked from.
@@ -10,7 +10,7 @@ class DownloadOptionChoice:
self.bool_options_description = bool_options_description
class DownloadOption:
class DownloadOption(object):
"""An option for a user to select a value from several different choices."""
def __init__(self, option_types, long_description, short_description, default_value,
default_value_description):
+16 -36
View File
@@ -1,7 +1,3 @@
class RPCError(Exception):
code = 0
class PriceDisagreementError(Exception):
pass
@@ -16,19 +12,19 @@ class DownloadCanceledError(Exception):
class DownloadSDTimeout(Exception):
def __init__(self, download):
super().__init__(f'Failed to download sd blob {download} within timeout')
Exception.__init__(self, 'Failed to download sd blob {} within timeout'.format(download))
self.download = download
class DownloadTimeoutError(Exception):
def __init__(self, download):
super().__init__(f'Failed to download {download} within timeout')
Exception.__init__(self, 'Failed to download {} within timeout'.format(download))
self.download = download
class DownloadDataTimeout(Exception):
def __init__(self, download):
super().__init__('Failed to download data blobs for sd hash '
Exception.__init__(self, 'Failed to download data blobs for sd hash '
'{} within timeout'.format(download))
self.download = download
@@ -45,21 +41,7 @@ class NullFundsError(Exception):
pass
class InsufficientFundsError(RPCError):
code = -310
class CurrencyConversionError(Exception):
pass
class FileOpenError(ValueError):
# this extends ValueError because it is replacing a ValueError in EncryptedFileDownloader
# and I don't know where it might get caught upstream
pass
class ResolveError(Exception):
class InsufficientFundsError(Exception):
pass
@@ -73,41 +55,39 @@ class KeyFeeAboveMaxAllowed(Exception):
class InvalidExchangeRateResponse(Exception):
def __init__(self, source, reason):
super().__init__(f'Failed to get exchange rate from {source}:{reason}')
Exception.__init__(self, 'Failed to get exchange rate from {}:{}'.format(source, reason))
self.source = source
self.reason = reason
class UnknownNameError(Exception):
def __init__(self, name):
super().__init__(f'Name {name} is unknown')
Exception.__init__(self, 'Name {} is unknown'.format(name))
self.name = name
class UnknownClaimID(Exception):
def __init__(self, claim_id):
super().__init__(f'Claim {claim_id} is unknown')
Exception.__init__(self, 'Claim {} is unknown'.format(claim_id))
self.claim_id = claim_id
class UnknownURI(Exception):
def __init__(self, uri):
super().__init__(f'URI {uri} cannot be resolved')
Exception.__init__(self, 'URI {} cannot be resolved'.format(uri))
self.name = uri
class UnknownOutpoint(Exception):
def __init__(self, outpoint):
super().__init__(f'Outpoint {outpoint} cannot be resolved')
Exception.__init__(self, 'Outpoint {} cannot be resolved'.format(outpoint))
self.outpoint = outpoint
class InvalidName(Exception):
def __init__(self, name, invalid_characters):
self.name = name
self.invalid_characters = invalid_characters
super().__init__(
'URI contains invalid characters: {}'.format(','.join(invalid_characters)))
Exception.__init__(
self, 'URI contains invalid characters: {}'.format(','.join(invalid_characters)))
class UnknownStreamTypeError(Exception):
@@ -124,8 +104,8 @@ class InvalidStreamDescriptorError(Exception):
class InvalidStreamInfoError(Exception):
def __init__(self, name, stream_info):
msg = f'{name} has claim with invalid stream info: {stream_info}'
super().__init__(msg)
msg = '{} has claim with invalid stream info: {}'.format(name, stream_info)
Exception.__init__(self, msg)
self.name = name
self.stream_info = stream_info
@@ -179,14 +159,14 @@ class NegotiationError(Exception):
class InvalidCurrencyError(Exception):
def __init__(self, currency):
self.currency = currency
super().__init__(
f'Invalid currency: {currency} is not a supported currency.')
Exception.__init__(
self, 'Invalid currency: {} is not a supported currency.'.format(currency))
class NoSuchDirectoryError(Exception):
def __init__(self, directory):
self.directory = directory
super().__init__(f'No such directory {directory}')
Exception.__init__(self, 'No such directory {}'.format(directory))
class ComponentStartConditionNotMet(Exception):
@@ -1,7 +1,7 @@
from decimal import Decimal
class Offer:
class Offer(object):
"""A rate offer to download blobs from a host."""
RATE_ACCEPTED = "RATE_ACCEPTED"
@@ -1,16 +1,16 @@
from decimal import Decimal
from lbrynet.core.Strategy import get_default_strategy, OnlyFreeStrategy
from lbrynet import conf
from lbrynet.p2p.Strategy import get_default_strategy, OnlyFreeStrategy
from decimal import Decimal
class BasePaymentRateManager:
class BasePaymentRateManager(object):
def __init__(self, rate=None, info_rate=None):
self.min_blob_data_payment_rate = rate if rate is not None else conf.settings['data_rate']
self.min_blob_info_payment_rate = (
info_rate if info_rate is not None else conf.settings['min_info_rate'])
class PaymentRateManager:
class PaymentRateManager(object):
def __init__(self, base, rate=None):
"""
@param base: a BasePaymentRateManager
@@ -36,7 +36,7 @@ class PaymentRateManager:
self.points_paid += amount
class NegotiatedPaymentRateManager:
class NegotiatedPaymentRateManager(object):
def __init__(self, base, availability_tracker, generous=None):
"""
@param base: a BasePaymentRateManager
@@ -84,7 +84,7 @@ class NegotiatedPaymentRateManager:
return False
class OnlyFreePaymentsManager:
class OnlyFreePaymentsManager(object):
def __init__(self, **kwargs):
"""
A payment rate manager that will only ever accept and offer a rate of 0.0,
+5 -5
View File
@@ -1,9 +1,9 @@
import datetime
from collections import defaultdict
from lbrynet import utils
from lbrynet.core import utils
# Do not create this object except through PeerManager
class Peer:
class Peer(object):
def __init__(self, host, port):
self.host = host
self.port = port
@@ -11,7 +11,7 @@ class Peer:
self.attempt_connection_at = None
# Number of times this Peer has been reported to be down, resets to 0 when it is up
self.down_count = 0
# Number of successful connections (with full protocol completion) to this peer
# Number of succesful connections (with full protocol completion) to this peer
self.success_count = 0
self.score = 0
self.stats = defaultdict(float) # {string stat_type, float count}
@@ -40,7 +40,7 @@ class Peer:
self.stats[stat_type] += count
def __str__(self):
return f'{self.host}:{self.port}'
return '{}:{}'.format(self.host, self.port)
def __repr__(self):
return f'Peer({self.host!r}, {self.port!r})'
return 'Peer({!r}, {!r})'.format(self.host, self.port)
@@ -1,7 +1,7 @@
from lbrynet.p2p.Peer import Peer
from lbrynet.core.Peer import Peer
class PeerManager:
class PeerManager(object):
def __init__(self):
self.peers = []
@@ -1,4 +1,7 @@
from zope.interface import implementer
from decimal import Decimal
from lbrynet.interfaces import IBlobPriceModel
from lbrynet import conf
@@ -6,7 +9,7 @@ def get_default_price_model(blob_tracker, base_price, **kwargs):
return MeanAvailabilityWeightedPrice(blob_tracker, base_price, **kwargs)
class ZeroPrice:
class ZeroPrice(object):
def __init__(self):
self.base_price = 0.0
@@ -14,7 +17,7 @@ class ZeroPrice:
return 0.0
class MeanAvailabilityWeightedPrice:
class MeanAvailabilityWeightedPrice(object):
"""Calculate mean-blob-availability and stream-position weighted price for a blob
Attributes:
@@ -25,6 +28,7 @@ class MeanAvailabilityWeightedPrice:
blob_tracker (BlobAvailabilityTracker): blob availability tracker
"""
implementer(IBlobPriceModel)
def __init__(self, tracker, base_price=None, alpha=1.0):
self.blob_tracker = tracker
@@ -1,12 +1,14 @@
import logging
from zope.interface import implements
from lbrynet.interfaces import IRateLimiter
from twisted.internet import task
log = logging.getLogger(__name__)
class DummyRateLimiter:
class DummyRateLimiter(object):
def __init__(self):
self.dl_bytes_this_second = 0
self.ul_bytes_this_second = 0
@@ -44,10 +46,10 @@ class DummyRateLimiter:
self.total_ul_bytes += num_bytes
class RateLimiter:
class RateLimiter(object):
"""This class ensures that upload and download rates don't exceed specified maximums"""
#implements(IRateLimiter)
implements(IRateLimiter)
#called by main application
+172
View File
@@ -0,0 +1,172 @@
import logging
from twisted.internet import defer
from lbrynet.core.BlobManager import DiskBlobManager
from lbrynet.database.storage import SQLiteStorage
from lbrynet.core.RateLimiter import RateLimiter
from lbrynet.core.PaymentRateManager import BasePaymentRateManager, OnlyFreePaymentsManager
log = logging.getLogger(__name__)
class Session(object):
"""This class manages all important services common to any application that uses the network.
the hash announcer, which informs other peers that this peer is
associated with some hash. Usually, this means this peer has a
blob identified by the hash in question, but it can be used for
other purposes.
the peer finder, which finds peers that are associated with some
hash.
the blob manager, which keeps track of which blobs have been
downloaded and provides access to them,
the rate limiter, which attempts to ensure download and upload
rates stay below a set maximum
upnp, which opens holes in compatible firewalls so that remote
peers can connect to this peer.
"""
def __init__(self, blob_data_payment_rate, db_dir=None, node_id=None, dht_node_port=None,
known_dht_nodes=None, peer_finder=None, hash_announcer=None, blob_dir=None, blob_manager=None,
peer_port=None, use_upnp=True, rate_limiter=None, wallet=None, blob_tracker_class=None,
payment_rate_manager_class=None, is_generous=True, external_ip=None, storage=None,
dht_node=None, peer_manager=None):
"""@param blob_data_payment_rate: The default payment rate for blob data
@param db_dir: The directory in which levelDB files should be stored
@param node_id: The unique ID of this node
@param peer_manager: An object which keeps track of all known
peers. If None, a PeerManager will be created
@param dht_node_port: The port on which the dht node should
listen for incoming connections
@param known_dht_nodes: A list of nodes which the dht node
should use to bootstrap into the dht
@param peer_finder: An object which is used to look up peers
that are associated with some hash. If None, a
DHTPeerFinder will be used, which looks for peers in the
distributed hash table.
@param hash_announcer: An object which announces to other
peers that this peer is associated with some hash. If
None, and peer_port is not None, a DHTHashAnnouncer will
be used. If None and peer_port is None, a
DummyHashAnnouncer will be used, which will not actually
announce anything.
@param blob_dir: The directory in which blobs will be
stored. If None and blob_manager is None, blobs will be
stored in memory only.
@param blob_manager: An object which keeps track of downloaded
blobs and provides access to them. If None, and blob_dir
is not None, a DiskBlobManager will be used, with the
given blob_dir. If None and blob_dir is None, a
TempBlobManager will be used, which stores blobs in memory
only.
@param peer_port: The port on which other peers should connect
to this peer
@param use_upnp: Whether or not to try to open a hole in the
firewall so that outside peers can connect to this peer's
peer_port and dht_node_port
@param rate_limiter: An object which keeps track of the amount
of data transferred to and from this peer, and can limit
that rate if desired
@param wallet: An object which will be used to keep track of
expected payments and which will pay peers. If None, a
wallet which uses the Point Trader system will be used,
which is meant for testing only
"""
self.db_dir = db_dir
self.node_id = node_id
self.peer_manager = peer_manager
self.peer_finder = peer_finder
self.hash_announcer = hash_announcer
self.dht_node_port = dht_node_port
self.known_dht_nodes = known_dht_nodes
if self.known_dht_nodes is None:
self.known_dht_nodes = []
self.blob_dir = blob_dir
self.blob_manager = blob_manager
# self.blob_tracker = None
# self.blob_tracker_class = blob_tracker_class or BlobAvailabilityTracker
self.peer_port = peer_port
self.use_upnp = use_upnp
self.rate_limiter = rate_limiter
self.external_ip = external_ip
self.upnp_redirects = []
self.wallet = wallet
self.dht_node = dht_node
self.base_payment_rate_manager = BasePaymentRateManager(blob_data_payment_rate)
self.payment_rate_manager = OnlyFreePaymentsManager()
# self.payment_rate_manager_class = payment_rate_manager_class or NegotiatedPaymentRateManager
# self.is_generous = is_generous
self.storage = storage or SQLiteStorage(self.db_dir)
def setup(self):
"""Create the blob directory and database if necessary, start all desired services"""
log.debug("Starting session.")
if self.dht_node is not None:
if self.peer_manager is None:
self.peer_manager = self.dht_node.peer_manager
if self.peer_finder is None:
self.peer_finder = self.dht_node.peer_finder
d = self.storage.setup()
d.addCallback(lambda _: self._setup_other_components())
return d
def shut_down(self):
"""Stop all services"""
log.info('Stopping session.')
ds = []
# if self.blob_tracker is not None:
# ds.append(defer.maybeDeferred(self.blob_tracker.stop))
if self.rate_limiter is not None:
ds.append(defer.maybeDeferred(self.rate_limiter.stop))
if self.blob_manager is not None:
ds.append(defer.maybeDeferred(self.blob_manager.stop))
# if self.use_upnp is True:
# ds.append(defer.maybeDeferred(self._unset_upnp))
return defer.DeferredList(ds)
def _setup_other_components(self):
log.debug("Setting up the rest of the components")
if self.rate_limiter is None:
self.rate_limiter = RateLimiter()
if self.blob_manager is None:
if self.blob_dir is None:
raise Exception(
"TempBlobManager is no longer supported, specify BlobManager or db_dir")
else:
self.blob_manager = DiskBlobManager(self.blob_dir, self.storage, self.dht_node._dataStore)
# if self.blob_tracker is None:
# self.blob_tracker = self.blob_tracker_class(
# self.blob_manager, self.dht_node.peer_finder, self.dht_node
# )
# if self.payment_rate_manager is None:
# self.payment_rate_manager = self.payment_rate_manager_class(
# self.base_payment_rate_manager, self.blob_tracker, self.is_generous
# )
self.rate_limiter.start()
d = self.blob_manager.setup()
return d
@@ -4,15 +4,14 @@ import tempfile
from twisted.internet import defer, threads, reactor
from lbrynet.blob.blob_file import BlobFile
from lbrynet.p2p.BlobManager import DiskBlobManager
from lbrynet.p2p.RateLimiter import DummyRateLimiter
from lbrynet.p2p.PaymentRateManager import OnlyFreePaymentsManager
from lbrynet.p2p.client.BlobRequester import BlobRequester
from lbrynet.p2p.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.p2p.client.ConnectionManager import ConnectionManager
from lbrynet.extras.daemon.storage import SQLiteStorage
from lbrynet.extras.daemon.PeerFinder import DummyPeerFinder
from lbrynet.blob import BlobFile
from lbrynet.core.BlobManager import DiskBlobManager
from lbrynet.core.RateLimiter import DummyRateLimiter
from lbrynet.core.PaymentRateManager import OnlyFreePaymentsManager
from lbrynet.core.client.BlobRequester import BlobRequester
from lbrynet.core.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.core.client.ConnectionManager import ConnectionManager
from lbrynet.dht.peerfinder import DummyPeerFinder
log = logging.getLogger(__name__)
@@ -20,7 +19,7 @@ log = logging.getLogger(__name__)
class SinglePeerFinder(DummyPeerFinder):
def __init__(self, peer):
super().__init__()
DummyPeerFinder.__init__(self)
self.peer = peer
def find_peers_for_blob(self, blob_hash, timeout=None, filter_self=False):
@@ -29,7 +28,7 @@ class SinglePeerFinder(DummyPeerFinder):
class BlobCallback(BlobFile):
def __init__(self, blob_dir, blob_hash, timeout):
super().__init__(blob_dir, blob_hash)
BlobFile.__init__(self, blob_dir, blob_hash)
self.callback = defer.Deferred()
reactor.callLater(timeout, self._cancel)
@@ -44,7 +43,7 @@ class BlobCallback(BlobFile):
return result
class SingleBlobDownloadManager:
class SingleBlobDownloadManager(object):
def __init__(self, blob):
self.blob = blob
@@ -58,7 +57,7 @@ class SingleBlobDownloadManager:
return self.blob.blob_hash
class SinglePeerDownloader:
class SinglePeerDownloader(object):
def __init__(self):
self._payment_rate_manager = OnlyFreePaymentsManager()
self._rate_limiter = DummyRateLimiter()
@@ -97,13 +96,10 @@ class SinglePeerDownloader:
@defer.inlineCallbacks
def download_temp_blob_from_peer(self, peer, timeout, blob_hash):
tmp_dir = yield threads.deferToThread(tempfile.mkdtemp)
tmp_storage = SQLiteStorage(tmp_dir)
yield tmp_storage.setup()
tmp_blob_manager = DiskBlobManager(tmp_dir, tmp_storage)
tmp_blob_manager = DiskBlobManager(tmp_dir, tmp_dir)
try:
result = yield self.download_blob_from_peer(peer, timeout, blob_hash, tmp_blob_manager)
finally:
yield tmp_blob_manager.stop()
yield tmp_storage.stop()
yield threads.deferToThread(shutil.rmtree, tmp_dir)
defer.returnValue(result)
@@ -1,17 +1,20 @@
from zope.interface import implementer
from decimal import Decimal
from lbrynet import conf
from lbrynet.p2p.Offer import Offer
from lbrynet.p2p.PriceModel import MeanAvailabilityWeightedPrice, ZeroPrice
from lbrynet.interfaces import INegotiationStrategy
from lbrynet.core.Offer import Offer
from lbrynet.core.PriceModel import MeanAvailabilityWeightedPrice, ZeroPrice
def get_default_strategy(blob_tracker, **kwargs):
return BasicAvailabilityWeightedStrategy(blob_tracker, **kwargs)
class Strategy:
class Strategy(object):
"""
Base for negotiation strategies
"""
implementer(INegotiationStrategy)
def __init__(self, price_model, max_rate, min_rate, is_generous=None):
self.price_model = price_model
@@ -97,6 +100,7 @@ class BasicAvailabilityWeightedStrategy(Strategy):
accepted or a threshold is reached
"""
implementer(INegotiationStrategy)
def __init__(self, blob_tracker, acceleration=1.25,
deceleration=0.9, max_rate=None,
@@ -105,7 +109,7 @@ class BasicAvailabilityWeightedStrategy(Strategy):
base_price=0.0001, alpha=1.0):
price_model = MeanAvailabilityWeightedPrice(
blob_tracker, base_price=base_price, alpha=alpha)
super().__init__(price_model, max_rate, min_rate, is_generous)
Strategy.__init__(self, price_model, max_rate, min_rate, is_generous)
self._acceleration = Decimal(acceleration) # rate of how quickly to ramp offer
self._deceleration = Decimal(deceleration)
@@ -133,9 +137,10 @@ class BasicAvailabilityWeightedStrategy(Strategy):
class OnlyFreeStrategy(Strategy):
implementer(INegotiationStrategy)
def __init__(self, *args, **kwargs):
price_model = ZeroPrice()
super().__init__(price_model, 0.0, 0.0, True)
Strategy.__init__(self, price_model, 0.0, 0.0, True)
def _get_mean_rate(self, rates):
return 0.0
@@ -1,26 +1,18 @@
import string
import binascii
from collections import defaultdict
import json
import logging
from collections import defaultdict
from binascii import unhexlify
from twisted.internet import threads, defer
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbrynet.p2p.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.p2p.Error import UnknownStreamTypeError, InvalidStreamDescriptorError
from lbrynet.p2p.HTTPBlobDownloader import HTTPBlobDownloader
from twisted.internet import threads, defer
from lbrynet.core.cryptoutils import get_lbry_hash_obj
from lbrynet.core.client.StandaloneBlobDownloader import StandaloneBlobDownloader
from lbrynet.core.Error import UnknownStreamTypeError, InvalidStreamDescriptorError
log = logging.getLogger(__name__)
class JSONBytesEncoder(json.JSONEncoder):
def default(self, obj): # pylint: disable=E0202
if isinstance(obj, bytes):
return obj.decode()
return super().default(obj)
class StreamDescriptorReader:
class StreamDescriptorReader(object):
"""Classes which derive from this class read a stream descriptor file return
a dictionary containing the fields in the file"""
def __init__(self):
@@ -29,6 +21,7 @@ class StreamDescriptorReader:
def _get_raw_data(self):
"""This method must be overridden by subclasses. It should return a deferred
which fires with the raw data in the stream descriptor"""
pass
def get_info(self):
"""Return the fields contained in the file"""
@@ -40,7 +33,7 @@ class StreamDescriptorReader:
class PlainStreamDescriptorReader(StreamDescriptorReader):
"""Read a stream descriptor file which is not a blob but a regular file"""
def __init__(self, stream_descriptor_filename):
super().__init__()
StreamDescriptorReader.__init__(self)
self.stream_descriptor_filename = stream_descriptor_filename
def _get_raw_data(self):
@@ -56,7 +49,7 @@ class PlainStreamDescriptorReader(StreamDescriptorReader):
class BlobStreamDescriptorReader(StreamDescriptorReader):
"""Read a stream descriptor file which is a blob"""
def __init__(self, blob):
super().__init__()
StreamDescriptorReader.__init__(self)
self.blob = blob
def _get_raw_data(self):
@@ -73,26 +66,25 @@ class BlobStreamDescriptorReader(StreamDescriptorReader):
return threads.deferToThread(get_data)
class StreamDescriptorWriter:
class StreamDescriptorWriter(object):
"""Classes which derive from this class write fields from a dictionary
of fields to a stream descriptor"""
def __init__(self):
pass
def create_descriptor(self, sd_info):
return self._write_stream_descriptor(
json.dumps(sd_info, sort_keys=True).encode()
)
return self._write_stream_descriptor(json.dumps(sd_info))
def _write_stream_descriptor(self, raw_data):
"""This method must be overridden by subclasses to write raw data to
the stream descriptor
"""
pass
class PlainStreamDescriptorWriter(StreamDescriptorWriter):
def __init__(self, sd_file_name):
super().__init__()
StreamDescriptorWriter.__init__(self)
self.sd_file_name = sd_file_name
def _write_stream_descriptor(self, raw_data):
@@ -108,7 +100,7 @@ class PlainStreamDescriptorWriter(StreamDescriptorWriter):
class BlobStreamDescriptorWriter(StreamDescriptorWriter):
def __init__(self, blob_manager):
super().__init__()
StreamDescriptorWriter.__init__(self)
self.blob_manager = blob_manager
@defer.inlineCallbacks
@@ -122,7 +114,7 @@ class BlobStreamDescriptorWriter(StreamDescriptorWriter):
defer.returnValue(sd_hash)
class StreamMetadata:
class StreamMetadata(object):
FROM_BLOB = 1
FROM_PLAIN = 2
@@ -135,7 +127,7 @@ class StreamMetadata:
self.source_file = None
class StreamDescriptorIdentifier:
class StreamDescriptorIdentifier(object):
"""Tries to determine the type of stream described by the stream descriptor using the
'stream_type' field. Keeps a list of StreamDescriptorValidators and StreamDownloaderFactorys
and returns the appropriate ones based on the type of the stream descriptor given
@@ -262,7 +254,7 @@ def save_sd_info(blob_manager, sd_hash, sd_info):
(sd_hash, calculated_sd_hash))
stream_hash = yield blob_manager.storage.get_stream_hash_for_sd_hash(sd_hash)
if not stream_hash:
log.debug("Saving info for %s", unhexlify(sd_info['stream_name']))
log.debug("Saving info for %s", sd_info['stream_name'].decode('hex'))
stream_name = sd_info['stream_name']
key = sd_info['key']
stream_hash = sd_info['stream_hash']
@@ -280,9 +272,9 @@ def format_blobs(crypt_blob_infos):
for blob_info in crypt_blob_infos:
blob = {}
if blob_info.length != 0:
blob['blob_hash'] = blob_info.blob_hash
blob['blob_hash'] = str(blob_info.blob_hash)
blob['blob_num'] = blob_info.blob_num
blob['iv'] = blob_info.iv
blob['iv'] = str(blob_info.iv)
blob['length'] = blob_info.length
formatted_blobs.append(blob)
return formatted_blobs
@@ -352,18 +344,18 @@ def get_blob_hashsum(b):
iv = b['iv']
blob_hashsum = get_lbry_hash_obj()
if length != 0:
blob_hashsum.update(blob_hash.encode())
blob_hashsum.update(str(blob_num).encode())
blob_hashsum.update(iv.encode())
blob_hashsum.update(str(length).encode())
blob_hashsum.update(blob_hash)
blob_hashsum.update(str(blob_num))
blob_hashsum.update(iv)
blob_hashsum.update(str(length))
return blob_hashsum.digest()
def get_stream_hash(hex_stream_name, key, hex_suggested_file_name, blob_infos):
h = get_lbry_hash_obj()
h.update(hex_stream_name.encode())
h.update(key.encode())
h.update(hex_suggested_file_name.encode())
h.update(hex_stream_name)
h.update(key)
h.update(hex_suggested_file_name)
blobs_hashsum = get_lbry_hash_obj()
for blob in blob_infos:
blobs_hashsum.update(get_blob_hashsum(blob))
@@ -372,8 +364,9 @@ def get_stream_hash(hex_stream_name, key, hex_suggested_file_name, blob_infos):
def verify_hex(text, field_name):
if not set(text).issubset(set(string.hexdigits)):
raise InvalidStreamDescriptorError("%s is not a hex-encoded string" % field_name)
for c in text:
if c not in '0123456789abcdef':
raise InvalidStreamDescriptorError("%s is not a hex-encoded string" % field_name)
def validate_descriptor(stream_info):
@@ -387,7 +380,7 @@ def validate_descriptor(stream_info):
raise InvalidStreamDescriptorError("Missing '%s'" % (e.args[0]))
if stream_info['blobs'][-1]['length'] != 0:
raise InvalidStreamDescriptorError("Does not end with a zero-length blob.")
if any([blob_info['length'] == 0 for blob_info in stream_info['blobs'][:-1]]):
if any([False if blob_info['length'] > 0 else True for blob_info in stream_info['blobs'][:-1]]):
raise InvalidStreamDescriptorError("Contains zero-length data blob")
if 'blob_hash' in stream_info['blobs'][-1]:
raise InvalidStreamDescriptorError("Stream terminator blob should not have a hash")
@@ -404,7 +397,7 @@ def validate_descriptor(stream_info):
return True
class EncryptedFileStreamDescriptorValidator:
class EncryptedFileStreamDescriptorValidator(object):
def __init__(self, raw_info):
self.raw_info = raw_info
@@ -413,14 +406,14 @@ class EncryptedFileStreamDescriptorValidator:
def info_to_show(self):
info = []
info.append(("stream_name", unhexlify(self.raw_info.get("stream_name"))))
info.append(("stream_name", binascii.unhexlify(self.raw_info.get("stream_name"))))
size_so_far = 0
for blob_info in self.raw_info.get("blobs", []):
size_so_far += int(blob_info['length'])
info.append(("stream_size", str(self.get_length_of_stream())))
suggested_file_name = self.raw_info.get("suggested_file_name", None)
if suggested_file_name is not None:
suggested_file_name = unhexlify(suggested_file_name)
suggested_file_name = binascii.unhexlify(suggested_file_name)
info.append(("suggested_file_name", suggested_file_name))
return info
@@ -432,8 +425,7 @@ class EncryptedFileStreamDescriptorValidator:
@defer.inlineCallbacks
def download_sd_blob(blob_hash, blob_manager, peer_finder, rate_limiter, payment_rate_manager, wallet, timeout=None,
download_mirrors=None):
def download_sd_blob(session, blob_hash, payment_rate_manager, timeout=None):
"""
Downloads a single blob from the network
@@ -447,24 +439,21 @@ def download_sd_blob(blob_hash, blob_manager, peer_finder, rate_limiter, payment
"""
downloader = StandaloneBlobDownloader(blob_hash,
blob_manager,
peer_finder,
rate_limiter,
session.blob_manager,
session.peer_finder,
session.rate_limiter,
payment_rate_manager,
wallet,
session.wallet,
timeout)
mirror = HTTPBlobDownloader(blob_manager, [blob_hash], download_mirrors or [], sd_hashes=[blob_hash], retry=False)
mirror.start()
sd_blob = yield downloader.download()
mirror.stop()
sd_reader = BlobStreamDescriptorReader(sd_blob)
sd_info = yield sd_reader.get_info()
try:
validate_descriptor(sd_info)
except InvalidStreamDescriptorError as err:
yield blob_manager.delete_blobs([blob_hash])
yield session.blob_manager.delete_blobs([blob_hash])
raise err
raw_sd = yield sd_reader._get_raw_data()
yield blob_manager.storage.add_known_blob(blob_hash, len(raw_sd))
yield save_sd_info(blob_manager, sd_blob.blob_hash, sd_info)
yield session.blob_manager.storage.add_known_blob(blob_hash, len(raw_sd))
yield save_sd_info(session.blob_manager, sd_blob.blob_hash, sd_info)
defer.returnValue(sd_blob)
File diff suppressed because it is too large. Load diff
File renamed without changes.
@@ -8,7 +8,7 @@ DELAY_INCREMENT = 0.0001
QUEUE_SIZE_THRESHOLD = 100
class CallLaterManager:
class CallLaterManager(object):
def __init__(self, callLater):
"""
:param callLater: (IReactorTime.callLater)
@@ -5,12 +5,14 @@ from decimal import Decimal
from twisted.internet import defer
from twisted.python.failure import Failure
from twisted.internet.error import ConnectionAborted
from zope.interface import implements
from lbrynet.p2p.Error import ConnectionClosedBeforeResponseError
from lbrynet.p2p.Error import InvalidResponseError, RequestCanceledError, NoResponseError
from lbrynet.p2p.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
from lbrynet.p2p.client.ClientRequest import ClientRequest, ClientBlobRequest
from lbrynet.p2p.Offer import Offer
from lbrynet.core.Error import ConnectionClosedBeforeResponseError
from lbrynet.core.Error import InvalidResponseError, RequestCanceledError, NoResponseError
from lbrynet.core.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
from lbrynet.core.client.ClientRequest import ClientRequest, ClientBlobRequest
from lbrynet.interfaces import IRequestCreator
from lbrynet.core.Offer import Offer
log = logging.getLogger(__name__)
@@ -27,7 +29,7 @@ def get_points(num_bytes, rate):
def cache(fn):
"""Caches the function call for each instance"""
attr = f'__{fn.__name__}_value'
attr = '__{}_value'.format(fn.__name__)
def helper(self):
if not hasattr(self, attr):
@@ -37,8 +39,8 @@ def cache(fn):
return helper
class BlobRequester:
#implements(IRequestCreator)
class BlobRequester(object):
implements(IRequestCreator)
def __init__(self, blob_manager, peer_finder, payment_rate_manager, wallet, download_manager):
self.blob_manager = blob_manager
@@ -161,7 +163,7 @@ class BlobRequester:
return True
def _get_bad_peers(self):
return [p for p in self._peers.keys() if not self._should_send_request_to(p)]
return [p for p in self._peers.iterkeys() if not self._should_send_request_to(p)]
def _hash_available(self, blob_hash):
for peer in self._available_blobs:
@@ -193,7 +195,7 @@ class BlobRequester:
self._peers[peer] += amount
class RequestHelper:
class RequestHelper(object):
def __init__(self, requestor, peer, protocol, payment_rate_manager):
self.requestor = requestor
self.peer = peer
@@ -352,10 +354,6 @@ class AvailabilityRequest(RequestHelper):
log.debug("Received a response to the availability request")
# save available blobs
blob_hashes = response_dict['available_blobs']
if not blob_hashes:
# should not send any more requests as it doesn't have any blob we need
self.update_local_score(-10.0)
return True
for blob_hash in blob_hashes:
if blob_hash in request.request_dict['requested_blobs']:
self.process_available_blob_hash(blob_hash, request)
@@ -427,7 +425,7 @@ class PriceRequest(RequestHelper):
class DownloadRequest(RequestHelper):
"""Choose a blob and download it from a peer and also pay the peer for the data."""
def __init__(self, requester, peer, protocol, payment_rate_manager, wallet, head_blob_hash):
super().__init__(requester, peer, protocol, payment_rate_manager)
RequestHelper.__init__(self, requester, peer, protocol, payment_rate_manager)
self.wallet = wallet
self.head_blob_hash = head_blob_hash
@@ -576,7 +574,7 @@ class DownloadRequest(RequestHelper):
return reason
class BlobDownloadDetails:
class BlobDownloadDetails(object):
"""Contains the information needed to make a ClientBlobRequest from an open blob"""
def __init__(self, blob, deferred, write_func, cancel_func, peer):
self.blob = blob
@@ -5,10 +5,13 @@ from twisted.internet import error, defer
from twisted.internet.protocol import Protocol, ClientFactory
from twisted.protocols.policies import TimeoutMixin
from twisted.python import failure
from lbrynet import conf, utils
from lbrynet.p2p.Error import ConnectionClosedBeforeResponseError, NoResponseError
from lbrynet.p2p.Error import DownloadCanceledError, MisbehavingPeerError
from lbrynet.p2p.Error import RequestCanceledError
from lbrynet import conf
from lbrynet.core import utils
from lbrynet.core.Error import ConnectionClosedBeforeResponseError, NoResponseError
from lbrynet.core.Error import DownloadCanceledError, MisbehavingPeerError
from lbrynet.core.Error import RequestCanceledError
from lbrynet.interfaces import IRequestSender, IRateLimited
from zope.interface import implements
log = logging.getLogger(__name__)
@@ -21,7 +24,7 @@ def encode_decimal(obj):
class ClientProtocol(Protocol, TimeoutMixin):
#implements(IRequestSender, IRateLimited)
implements(IRequestSender, IRateLimited)
######### Protocol #########
PROTOCOL_TIMEOUT = 30
@@ -31,7 +34,7 @@ class ClientProtocol(Protocol, TimeoutMixin):
self._rate_limiter = self.factory.rate_limiter
self.peer = self.factory.peer
self._response_deferreds = {}
self._response_buff = b''
self._response_buff = ''
self._downloading_blob = False
self._blob_download_request = None
self._next_request = {}
@@ -58,7 +61,7 @@ class ClientProtocol(Protocol, TimeoutMixin):
self.transport.loseConnection()
response, extra_data = self._get_valid_response(self._response_buff)
if response is not None:
self._response_buff = b''
self._response_buff = ''
self._handle_response(response)
if self._downloading_blob is True and len(extra_data) != 0:
self._blob_download_request.write(extra_data)
@@ -68,17 +71,17 @@ class ClientProtocol(Protocol, TimeoutMixin):
self.peer.report_down()
self.transport.abortConnection()
def connectionLost(self, reason=None):
def connectionLost(self, reason):
log.debug("Connection lost to %s: %s", self.peer, reason)
self.setTimeout(None)
self.connection_closed = True
if reason is None or reason.check(error.ConnectionDone):
if reason.check(error.ConnectionDone):
err = failure.Failure(ConnectionClosedBeforeResponseError())
else:
err = reason
for key, d in self._response_deferreds.items():
del self._response_deferreds[key]
d.errback(err)
self._response_deferreds.clear()
if self._blob_download_request is not None:
self._blob_download_request.cancel(err)
self.factory.connection_was_made_deferred.callback(True)
@@ -108,7 +111,7 @@ class ClientProtocol(Protocol, TimeoutMixin):
self.connection_closing = True
ds = []
err = RequestCanceledError()
for key, d in list(self._response_deferreds.items()):
for key, d in self._response_deferreds.items():
del self._response_deferreds[key]
d.errback(err)
ds.append(d)
@@ -123,7 +126,7 @@ class ClientProtocol(Protocol, TimeoutMixin):
def _handle_request_error(self, err):
log.error("An unexpected error occurred creating or sending a request to %s. %s: %s",
self.peer, err.type, err)
self.peer, err.type, err.message)
self.transport.loseConnection()
def _ask_for_request(self):
@@ -148,7 +151,7 @@ class ClientProtocol(Protocol, TimeoutMixin):
self.setTimeout(self.PROTOCOL_TIMEOUT)
# TODO: compare this message to the last one. If they're the same,
# TODO: incrementally delay this message.
m = json.dumps(request_msg, default=encode_decimal).encode()
m = json.dumps(request_msg, default=encode_decimal)
self.transport.write(m)
def _get_valid_response(self, response_msg):
@@ -156,7 +159,7 @@ class ClientProtocol(Protocol, TimeoutMixin):
response = None
curr_pos = 0
while 1:
next_close_paren = response_msg.find(b'}', curr_pos)
next_close_paren = response_msg.find('}', curr_pos)
if next_close_paren != -1:
curr_pos = next_close_paren + 1
try:
@@ -1,7 +1,7 @@
from lbrynet.blob.blob_file import MAX_BLOB_SIZE
class ClientRequest:
class ClientRequest(object):
def __init__(self, request_dict, response_identifier=None):
self.request_dict = request_dict
self.response_identifier = response_identifier
@@ -9,7 +9,7 @@ class ClientRequest:
class ClientPaidRequest(ClientRequest):
def __init__(self, request_dict, response_identifier, max_pay_units):
super().__init__(request_dict, response_identifier)
ClientRequest.__init__(self, request_dict, response_identifier)
self.max_pay_units = max_pay_units
@@ -20,7 +20,7 @@ class ClientBlobRequest(ClientPaidRequest):
max_pay_units = MAX_BLOB_SIZE
else:
max_pay_units = blob.length
super().__init__(request_dict, response_identifier, max_pay_units)
ClientPaidRequest.__init__(self, request_dict, response_identifier, max_pay_units)
self.write = write_func
self.finished_deferred = finished_deferred
self.cancel = cancel_func
@@ -1,22 +1,25 @@
import random
import logging
from twisted.internet import defer, reactor
from lbrynet import utils, conf
from lbrynet.p2p.client.ClientProtocol import ClientProtocolFactory
from lbrynet.p2p.Error import InsufficientFundsError
from zope.interface import implements
from lbrynet import interfaces
from lbrynet import conf
from lbrynet.core.client.ClientProtocol import ClientProtocolFactory
from lbrynet.core.Error import InsufficientFundsError
from lbrynet.core import utils
log = logging.getLogger(__name__)
class PeerConnectionHandler:
class PeerConnectionHandler(object):
def __init__(self, request_creators, factory):
self.request_creators = request_creators
self.factory = factory
self.connection = None
class ConnectionManager:
#implements(interfaces.IConnectionManager)
class ConnectionManager(object):
implements(interfaces.IConnectionManager)
MANAGE_CALL_INTERVAL_SEC = 5
TCP_CONNECT_TIMEOUT = 15
@@ -95,8 +98,7 @@ class ConnectionManager:
d.addBoth(lambda _: disconnect_peer(p))
return d
# fixme: stop modifying dict during iteration
closing_deferreds = [close_connection(peer) for peer in list(self._peer_connections)]
closing_deferreds = [close_connection(peer) for peer in self._peer_connections.keys()]
return defer.DeferredList(closing_deferreds)
@defer.inlineCallbacks
@@ -224,3 +226,5 @@ class ConnectionManager:
del self._connections_closing[peer]
d.callback(True)
return connection_was_made
@@ -1,12 +1,14 @@
import logging
from twisted.internet import defer
from zope.interface import implements
from lbrynet import interfaces
log = logging.getLogger(__name__)
class DownloadManager:
#implements(interfaces.IDownloadManager)
class DownloadManager(object):
implements(interfaces.IDownloadManager)
def __init__(self, blob_manager):
self.blob_manager = blob_manager
@@ -79,14 +81,14 @@ class DownloadManager:
return self.blob_handler.handle_blob(self.blobs[blob_num], self.blob_infos[blob_num])
def calculate_total_bytes(self):
return sum([bi.length for bi in self.blob_infos.values()])
return sum([bi.length for bi in self.blob_infos.itervalues()])
def calculate_bytes_left_to_output(self):
if not self.blobs:
return self.calculate_total_bytes()
else:
to_be_outputted = [
b for n, b in self.blobs.items()
b for n, b in self.blobs.iteritems()
if n >= self.progress_manager.last_blob_outputted
]
return sum([b.length for b in to_be_outputted if b.length is not None])
@@ -1,11 +1,12 @@
import logging
from lbrynet.p2p.BlobInfo import BlobInfo
from lbrynet.p2p.client.BlobRequester import BlobRequester
from lbrynet.p2p.client.ConnectionManager import ConnectionManager
from lbrynet.p2p.client.DownloadManager import DownloadManager
from lbrynet.p2p.Error import InvalidBlobHashError, DownloadSDTimeout
from lbrynet.blob.blob_file import is_valid_blobhash
from lbrynet.utils import safe_start_looping_call, safe_stop_looping_call
from zope.interface import implements
from lbrynet import interfaces
from lbrynet.core.BlobInfo import BlobInfo
from lbrynet.core.client.BlobRequester import BlobRequester
from lbrynet.core.client.ConnectionManager import ConnectionManager
from lbrynet.core.client.DownloadManager import DownloadManager
from lbrynet.core.Error import InvalidBlobHashError, DownloadSDTimeout
from lbrynet.core.utils import is_valid_blobhash, safe_start_looping_call, safe_stop_looping_call
from twisted.python.failure import Failure
from twisted.internet import defer
from twisted.internet.task import LoopingCall
@@ -13,8 +14,8 @@ from twisted.internet.task import LoopingCall
log = logging.getLogger(__name__)
class SingleBlobMetadataHandler:
#implements(interfaces.IMetadataHandler)
class SingleBlobMetadataHandler(object):
implements(interfaces.IMetadataHandler)
def __init__(self, blob_hash, download_manager):
self.blob_hash = blob_hash
@@ -30,7 +31,7 @@ class SingleBlobMetadataHandler:
return 0
class SingleProgressManager:
class SingleProgressManager(object):
def __init__(self, download_manager, finished_callback, timeout_callback, timeout):
self.finished_callback = finished_callback
self.timeout_callback = timeout_callback
@@ -70,10 +71,10 @@ class SingleProgressManager:
def needed_blobs(self):
blobs = self.download_manager.blobs
assert len(blobs) == 1
return [b for b in blobs.values() if not b.get_is_verified()]
return [b for b in blobs.itervalues() if not b.get_is_verified()]
class DummyBlobHandler:
class DummyBlobHandler(object):
def __init__(self):
pass
@@ -81,7 +82,7 @@ class DummyBlobHandler:
pass
class StandaloneBlobDownloader:
class StandaloneBlobDownloader(object):
def __init__(self, blob_hash, blob_manager, peer_finder,
rate_limiter, payment_rate_manager, wallet,
timeout=None):
@@ -1,16 +1,17 @@
import logging
from twisted.internet import defer, task
from lbrynet.interfaces import IProgressManager
from twisted.internet import defer
from zope.interface import implements
log = logging.getLogger(__name__)
class FullStreamProgressManager:
def __init__(self, finished_callback, blob_manager, download_manager,
delete_blob_after_finished: bool = False, reactor: task.Clock = None):
if not reactor:
from twisted.internet import reactor
self.reactor = reactor
class StreamProgressManager(object):
implements(IProgressManager)
def __init__(self, finished_callback, blob_manager,
download_manager, delete_blob_after_finished=False):
self.finished_callback = finished_callback
self.blob_manager = blob_manager
self.delete_blob_after_finished = delete_blob_after_finished
@@ -20,11 +21,15 @@ class FullStreamProgressManager:
self.stopped = True
self._next_try_to_output_call = None
self.outputting_d = None
self.wrote_first_data = defer.Deferred()
######### IProgressManager #########
def start(self):
from twisted.internet import reactor
self.stopped = False
self._next_try_to_output_call = self.reactor.callLater(0, self._try_to_output)
self._next_try_to_output_call = reactor.callLater(0, self._try_to_output)
return defer.succeed(True)
def stop(self):
@@ -34,43 +39,32 @@ class FullStreamProgressManager:
self._next_try_to_output_call = None
return self._stop_outputting()
# def blob_downloaded(self, blob, blob_num):
# if self.outputting_d is None:
# self._output_loop()
def blob_downloaded(self, blob, blob_num):
if self.outputting_d is None:
self._output_loop()
def stream_position(self):
blobs = self.download_manager.blobs
if not blobs:
return 0
else:
for i in range(max(blobs.keys())):
if self._done(i, blobs):
return i
return max(blobs.keys()) + 1
def needed_blobs(self):
blobs = self.download_manager.blobs
return [
b for n, b in blobs.items()
if not b.get_is_verified() and not n in self.provided_blob_nums
]
######### internal #########
def _finished_outputting(self):
self.finished_callback(True)
def _try_to_output(self):
self._next_try_to_output_call = self.reactor.callLater(1, self._try_to_output)
from twisted.internet import reactor
self._next_try_to_output_call = reactor.callLater(1, self._try_to_output)
if self.outputting_d is None:
self._output_loop()
def _output_loop(self):
pass
def _stop_outputting(self):
if self.outputting_d is not None:
return self.outputting_d
return defer.succeed(None)
def _finished_with_blob(self, blob_num: int) -> None:
if blob_num == 0 and not self.wrote_first_data.called:
self.wrote_first_data.callback(True)
def _finished_with_blob(self, blob_num):
log.debug("In _finished_with_blob, blob_num = %s", str(blob_num))
if self.delete_blob_after_finished is True:
log.debug("delete_blob_after_finished is True")
@@ -84,17 +78,49 @@ class FullStreamProgressManager:
else:
log.debug("delete_blob_after_finished is False")
def _done(self, i: int, blobs: list) -> bool:
class FullStreamProgressManager(StreamProgressManager):
def __init__(self, finished_callback, blob_manager,
download_manager, delete_blob_after_finished=False):
StreamProgressManager.__init__(self, finished_callback, blob_manager, download_manager,
delete_blob_after_finished)
self.outputting_d = None
######### IProgressManager #########
def _done(self, i, blobs):
"""Return true if `i` is a blob number we don't have"""
return (
i not in blobs or
(
not blobs[i].get_is_verified() and
i not in self.provided_blob_nums
)
i not in blobs or
(
not blobs[i].get_is_verified() and
i not in self.provided_blob_nums
)
)
def stream_position(self):
blobs = self.download_manager.blobs
if not blobs:
return 0
else:
for i in xrange(max(blobs.iterkeys())):
if self._done(i, blobs):
return i
return max(blobs.iterkeys()) + 1
def needed_blobs(self):
blobs = self.download_manager.blobs
return [
b for n, b in blobs.iteritems()
if not b.get_is_verified() and not n in self.provided_blob_nums
]
######### internal #########
def _output_loop(self):
from twisted.internet import reactor
if self.stopped:
if self.outputting_d is not None:
self.outputting_d.callback(True)
@@ -115,7 +141,7 @@ class FullStreamProgressManager:
self.outputting_d.callback(True)
self.outputting_d = None
else:
self.reactor.callLater(0, self._output_loop)
reactor.callLater(0, self._output_loop)
current_blob_num = self.last_blob_outputted + 1
File renamed without changes.
File renamed without changes.
+17
View File
@@ -0,0 +1,17 @@
import os
from contextlib import contextmanager
@contextmanager
def get_read_handle(path):
"""
Get os independent read handle for a file
"""
if os.name == "nt":
file_mode = 'rb'
else:
file_mode = 'r'
read_handle = open(path, file_mode)
yield read_handle
read_handle.close()
@@ -1,18 +1,41 @@
import inspect
import json
import logging
import logging.handlers
import os
import sys
import traceback
import treq
import twisted.python.log
from twisted.internet import defer
from lbrynet import __version__ as lbrynet_version, build_type
from lbrynet import utils, conf
import twisted.python.log
from lbrynet import __version__ as lbrynet_version, build_type, conf
from lbrynet.core import utils
####
# This code is copied from logging/__init__.py in the python source code
####
#
# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
if hasattr(sys, 'frozen'): # support for py2exe
_srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
_srcfile = __file__[:-4] + '.py'
else:
_srcfile = __file__
_srcfile = os.path.normcase(_srcfile)
#####
TRACE = 5
class HTTPSHandler(logging.Handler):
def __init__(self, url, fqdn=False, localname=None, facility=None, cookies=None):
super().__init__()
logging.Handler.__init__(self)
self.url = url
self.fqdn = fqdn
self.localname = localname
@@ -28,7 +51,7 @@ class HTTPSHandler(logging.Handler):
@defer.inlineCallbacks
def _emit(self, record):
payload = self.format(record)
response = yield treq.post(self.url, data=payload.encode(), cookies=self.cookies)
response = yield treq.post(self.url, data=payload, cookies=self.cookies)
self.cookies.update(response.cookies())
def emit(self, record):
@@ -87,7 +110,6 @@ def disable_third_party_loggers():
logging.getLogger('BitcoinRPC').setLevel(logging.INFO)
logging.getLogger('lbryum').setLevel(logging.WARNING)
logging.getLogger('twisted').setLevel(logging.CRITICAL)
logging.getLogger('aioupnp').setLevel(logging.WARNING)
@_log_decorator
@@ -117,8 +139,6 @@ def get_loggly_url(token=None, version=None):
def configure_loggly_handler():
if build_type.BUILD == 'dev':
return
if not conf.settings['share_usage_data']:
return
level = logging.ERROR
handler = get_loggly_handler(level=level, installation_id=conf.settings.installation_id,
session_id=conf.settings.get_session_id())
@@ -165,6 +185,33 @@ class JsonFormatter(logging.Formatter):
return json.dumps(data)
####
# This code is copied from logging/__init__.py in the python source code
####
def findCaller(srcfile=None):
"""Returns the filename, line number and function name of the caller"""
srcfile = srcfile or _srcfile
f = inspect.currentframe()
# On some versions of IronPython, currentframe() returns None if
# IronPython isn't run with -X:Frames.
if f is not None:
f = f.f_back
rv = "(unknown file)", 0, "(unknown function)"
while hasattr(f, "f_code"):
co = f.f_code
filename = os.path.normcase(co.co_filename)
# ignore any function calls that are in this file
if filename == srcfile:
f = f.f_back
continue
rv = (filename, f.f_lineno, co.co_name)
break
return rv
###
def failure(failure, log, msg, *args):
"""Log a failure message from a deferred.
@@ -241,10 +288,10 @@ def configure_twisted():
observer.start()
class LoggerNameFilter:
class LoggerNameFilter(object):
"""Filter a log record based on its name.
Allows all info level and higher records to pass through.
Allows all info level and higher records to pass thru.
Debug records pass if the log record name (or a parent) match
the input list of logger names.
"""
@@ -269,3 +316,65 @@ def get_parent(logger_name):
return ''
names = names[:-1]
return '.'.join(names)
class Logger(logging.Logger):
"""A logger that has an extra `fail` method useful for handling twisted failures."""
def fail(self, callback=None, *args, **kwargs):
"""Returns a function to log a failure from an errback.
The returned function appends the error message and extracts
the traceback from `err`.
Example usage:
d.addErrback(log.fail(), 'This is an error message')
Although odd, making the method call is necessary to extract
out useful filename and line number information; otherwise the
reported values are from inside twisted's deferred handling
code.
Args:
callback: callable to call after making the log. The first argument
will be the `err` from the deferred
args: extra arguments to pass into `callback`
Returns: a function that takes the following arguments:
err: twisted.python.failure.Failure
msg: the message to log, using normal logging string iterpolation.
msg_args: the values to subtitute into `msg`
msg_kwargs: set `level` to change from the default ERROR severity. Other
keywoards are treated as normal log kwargs.
"""
fn, lno, func = findCaller()
def _fail(err, msg, *msg_args, **msg_kwargs):
level = msg_kwargs.pop('level', logging.ERROR)
msg += ": %s"
msg_args += (err.getErrorMessage(),)
exc_info = (err.type, err.value, err.getTracebackObject())
record = self.makeRecord(
self.name, level, fn, lno, msg, msg_args, exc_info, func, msg_kwargs)
self.handle(record)
if callback:
try:
return callback(err, *args, **kwargs)
except Exception:
# log.fail is almost always called within an
# errback. If callback fails and we didn't catch
# the exception we would need to attach a second
# errback to deal with that, which we will almost
# never do and then we end up with an unhandled
# error that will get swallowed by twisted
self.exception('Failed to run callback')
return _fail
def trace(self, msg, *args, **kwargs):
if self.isEnabledFor(TRACE):
self._log(TRACE, msg, args, **kwargs)
logging.setLoggerClass(Logger)
logging.addLevelName(TRACE, 'TRACE')
@@ -1,9 +1,9 @@
class LoopingCallManager:
class LoopingCallManager(object):
def __init__(self, calls=None):
self.calls = calls or {}
def register_looping_call(self, name, call):
assert name not in self.calls, f'{name} is already registered'
assert name not in self.calls, '{} is already registered'.format(name)
self.calls[name] = call
def start(self, name, *args):
@@ -15,6 +15,6 @@ class LoopingCallManager:
self.calls[name].stop()
def shutdown(self):
for lcall in self.calls.values():
for lcall in self.calls.itervalues():
if lcall.running:
lcall.stop()
@@ -1,12 +1,14 @@
import logging
from twisted.internet import defer
from zope.interface import implements
from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler
log = logging.getLogger(__name__)
class BlobAvailabilityHandlerFactory:
# implements(IQueryHandlerFactory)
class BlobAvailabilityHandlerFactory(object):
implements(IQueryHandlerFactory)
def __init__(self, blob_manager):
self.blob_manager = blob_manager
@@ -24,8 +26,8 @@ class BlobAvailabilityHandlerFactory:
return "Blob Availability - blobs that are available to be uploaded"
class BlobAvailabilityHandler:
#implements(IQueryHandler)
class BlobAvailabilityHandler(object):
implements(IQueryHandler)
def __init__(self, blob_manager):
self.blob_manager = blob_manager
@@ -3,15 +3,17 @@ import logging
from twisted.internet import defer
from twisted.protocols.basic import FileSender
from twisted.python.failure import Failure
from zope.interface import implements
from lbrynet.extras.daemon import analytics
from lbrynet.p2p.Offer import Offer
from lbrynet import analytics
from lbrynet.core.Offer import Offer
from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler, IBlobSender
log = logging.getLogger(__name__)
class BlobRequestHandlerFactory:
#implements(IQueryHandlerFactory)
class BlobRequestHandlerFactory(object):
implements(IQueryHandlerFactory)
def __init__(self, blob_manager, wallet, payment_rate_manager, analytics_manager):
self.blob_manager = blob_manager
@@ -33,8 +35,8 @@ class BlobRequestHandlerFactory:
return "Blob Uploader - uploads blobs"
class BlobRequestHandler:
#implements(IQueryHandler, IBlobSender)
class BlobRequestHandler(object):
implements(IQueryHandler, IBlobSender)
PAYMENT_RATE_QUERY = 'blob_data_payment_rate'
BLOB_QUERY = 'requested_blob'
AVAILABILITY_QUERY = 'requested_blobs'
@@ -1,8 +1,9 @@
import logging
from twisted.internet import error
from twisted.internet import interfaces, error
from twisted.internet.protocol import Protocol, ServerFactory
from twisted.python import failure
from lbrynet.p2p.server.ServerRequestHandler import ServerRequestHandler
from zope.interface import implements
from lbrynet.core.server.ServerRequestHandler import ServerRequestHandler
log = logging.getLogger(__name__)
@@ -23,7 +24,7 @@ class ServerProtocol(Protocol):
10) Pause/resume production when told by the rate limiter
"""
#implements(interfaces.IConsumer)
implements(interfaces.IConsumer)
#Protocol stuff
@@ -1,23 +1,25 @@
import json
import logging
from twisted.internet import defer
from twisted.internet import interfaces, defer
from zope.interface import implements
from lbrynet.interfaces import IRequestHandler
log = logging.getLogger(__name__)
class ServerRequestHandler:
class ServerRequestHandler(object):
"""This class handles requests from clients. It can upload blobs and
return request for information about more blobs that are
associated with streams.
"""
#implements(interfaces.IPushProducer, interfaces.IConsumer, IRequestHandler)
implements(interfaces.IPushProducer, interfaces.IConsumer, IRequestHandler)
def __init__(self, consumer):
self.consumer = consumer
self.production_paused = False
self.request_buff = b''
self.response_buff = b''
self.request_buff = ''
self.response_buff = ''
self.producer = None
self.request_received = False
self.CHUNK_SIZE = 2**14
@@ -54,7 +56,7 @@ class ServerRequestHandler:
return
chunk = self.response_buff[:self.CHUNK_SIZE]
self.response_buff = self.response_buff[self.CHUNK_SIZE:]
if chunk == b'':
if chunk == '':
return
log.trace("writing %s bytes to the client", len(chunk))
self.consumer.write(chunk)
@@ -99,7 +101,7 @@ class ServerRequestHandler:
self.request_buff = self.request_buff + data
msg = self.try_to_parse_request(self.request_buff)
if msg:
self.request_buff = b''
self.request_buff = ''
self._process_msg(msg)
else:
log.debug("Request buff not a valid json message")
@@ -132,7 +134,7 @@ class ServerRequestHandler:
self._produce_more()
def send_response(self, msg):
m = json.dumps(msg).encode()
m = json.dumps(msg)
log.debug("Sending a response of length %s", str(len(m)))
log.debug("Response: %s", str(m))
self.response_buff = self.response_buff + m
@@ -165,7 +167,7 @@ class ServerRequestHandler:
return True
ds = []
for query_handler, query_identifiers in self.query_handlers.items():
for query_handler, query_identifiers in self.query_handlers.iteritems():
queries = {q_i: msg[q_i] for q_i in query_identifiers if q_i in msg}
d = query_handler.handle_queries(queries)
d.addErrback(log_errors)
File renamed without changes.
+54
View File
@@ -0,0 +1,54 @@
import platform
import json
import subprocess
import os
from urllib2 import urlopen, URLError
from lbryschema import __version__ as lbryschema_version
from lbryum import __version__ as LBRYUM_VERSION
from lbrynet import build_type, __version__ as lbrynet_version
from lbrynet.conf import ROOT_DIR
def get_lbrynet_version():
if build_type.BUILD == "dev":
try:
with open(os.devnull, 'w') as devnull:
git_dir = ROOT_DIR + '/.git'
return subprocess.check_output(
['git', '--git-dir='+git_dir, 'describe', '--dirty', '--always'],
stderr=devnull
).strip().lstrip('v')
except (subprocess.CalledProcessError, OSError):
print "failed to get version from git"
return lbrynet_version
def get_platform(get_ip=True):
p = {
"processor": platform.processor(),
"python_version": platform.python_version(),
"platform": platform.platform(),
"os_release": platform.release(),
"os_system": platform.system(),
"lbrynet_version": get_lbrynet_version(),
"lbryum_version": LBRYUM_VERSION,
"lbryschema_version": lbryschema_version,
"build": build_type.BUILD, # CI server sets this during build step
}
if p["os_system"] == "Linux":
import distro
p["distro"] = distro.info()
p["desktop"] = os.environ.get('XDG_CURRENT_DESKTOP', 'Unknown')
# TODO: remove this from get_platform and add a get_external_ip function using treq
if get_ip:
try:
response = json.loads(urlopen("https://api.lbry.io/ip").read())
if not response['success']:
raise URLError("failed to get external ip")
p['ip'] = response['data']['ip']
except (URLError, AssertionError):
p['ip'] = "Could not determine IP"
return p
+39 -25
View File
@@ -1,5 +1,4 @@
import base64
import codecs
import datetime
import random
import socket
@@ -11,11 +10,14 @@ import logging
import pkg_resources
from twisted.python.failure import Failure
from twisted.internet import defer
from lbrynet.schema.claim import ClaimDict
from lbrynet.cryptoutils import get_lbry_hash_obj
from lbryschema.claim import ClaimDict
from lbrynet.core.cryptoutils import get_lbry_hash_obj
log = logging.getLogger(__name__)
# digest_size is in bytes, and blob hashes are hex encoded
blobhash_length = get_lbry_hash_obj().digest_size * 2
# defining these time functions here allows for easier overriding in testing
def now():
@@ -49,7 +51,6 @@ def call_later(delay, func, *args, **kwargs):
from twisted.internet import reactor
return reactor.callLater(delay, func, *args, **kwargs)
def safe_start_looping_call(looping_call, interval_sec):
if not looping_call.running:
looping_call.start(interval_sec)
@@ -61,36 +62,46 @@ def safe_stop_looping_call(looping_call):
def generate_id(num=None):
h = get_lbry_hash_obj()
if num is not None:
h.update(str(num).encode())
h.update(str(num))
else:
h.update(str(random.getrandbits(512)).encode())
h.update(str(random.getrandbits(512)))
return h.digest()
def is_valid_hashcharacter(char):
return char in "0123456789abcdef"
def is_valid_blobhash(blobhash):
"""Checks whether the blobhash is the correct length and contains only
valid characters (0-9, a-f)
@param blobhash: string, the blobhash to check
@return: True/False
"""
return len(blobhash) == blobhash_length and all(is_valid_hashcharacter(l) for l in blobhash)
def version_is_greater_than(a, b):
"""Returns True if version a is more recent than version b"""
return pkg_resources.parse_version(a) > pkg_resources.parse_version(b)
def rot13(some_str):
return codecs.encode(some_str, 'rot_13')
def deobfuscate(obfustacated):
return base64.b64decode(rot13(obfustacated)).decode()
return base64.b64decode(obfustacated.decode('rot13'))
def obfuscate(plain):
return rot13(base64.b64encode(plain).decode())
return base64.b64encode(plain).encode('rot13')
def check_connection(server="lbry.io", port=80, timeout=5):
def check_connection(server="lbry.io", port=80, timeout=2):
"""Attempts to open a socket to server:port and returns True if successful."""
log.debug('Checking connection to %s:%s', server, port)
try:
server = socket.gethostbyname(server)
conn = socket.create_connection((server, port), timeout)
conn.close()
socket.create_connection((server, port), timeout)
log.debug('Connection successful')
return True
except (socket.gaierror, socket.herror) as ex:
@@ -131,7 +142,7 @@ def get_sd_hash(stream_info):
get('source', {}).\
get('source')
if not result:
log.warning("Unable to get sd_hash")
log.warn("Unable to get sd_hash")
return result
@@ -139,12 +150,12 @@ def json_dumps_pretty(obj, **kwargs):
return json.dumps(obj, sort_keys=True, indent=2, separators=(',', ': '), **kwargs)
class DeferredLockContextManager:
class DeferredLockContextManager(object):
def __init__(self, lock):
self._lock = lock
def __enter__(self):
yield self._lock.acquire()
yield self._lock.aquire()
def __exit__(self, exc_type, exc_val, exc_tb):
yield self._lock.release()
@@ -155,7 +166,7 @@ def DeferredDict(d, consumeErrors=False):
keys = []
dl = []
response = {}
for k, v in d.items():
for k, v in d.iteritems():
keys.append(k)
dl.append(v)
results = yield defer.DeferredList(dl, consumeErrors=consumeErrors)
@@ -165,7 +176,7 @@ def DeferredDict(d, consumeErrors=False):
defer.returnValue(response)
class DeferredProfiler:
class DeferredProfiler(object):
def __init__(self):
self.profile_results = {}
@@ -178,7 +189,7 @@ class DeferredProfiler:
caller: [(start, finished, finished - start, success)
for (start, finished, _caller, success) in profile_results
if _caller == caller]
for caller in {result[2] for result in profile_results}
for caller in set(result[2] for result in profile_results)
}
log.info("called %s %i times from %i sources\n", fn.__name__, len(profile_results), len(call_counts))
@@ -201,12 +212,15 @@ class DeferredProfiler:
from twisted.internet import reactor
def _cb(result, fn, start, caller_info):
got_error = isinstance(result, (Failure, Exception))
self.add_result(fn, start, reactor.seconds(), caller_info, not got_error)
if got_error:
raise result
if isinstance(result, (Failure, Exception)):
error = result
result = None
else:
error = None
self.add_result(fn, start, reactor.seconds(), caller_info, error is None)
if error is None:
return result
raise error
def _profiled_deferred(fn):
reactor.addSystemEventTrigger("after", "shutdown", self.show_profile_results, fn)
@@ -7,7 +7,7 @@ from cryptography.hazmat.primitives.ciphers import Cipher, modes
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from cryptography.hazmat.primitives.padding import PKCS7
from cryptography.hazmat.backends import default_backend
from lbrynet.p2p.BlobInfo import BlobInfo
from lbrynet.core.BlobInfo import BlobInfo
from lbrynet.blob.blob_file import MAX_BLOB_SIZE
log = logging.getLogger(__name__)
@@ -16,21 +16,21 @@ backend = default_backend()
class CryptBlobInfo(BlobInfo):
def __init__(self, blob_hash, blob_num, length, iv):
super().__init__(blob_hash, blob_num, length)
BlobInfo.__init__(self, blob_hash, blob_num, length)
self.iv = iv
def get_dict(self):
info = {
"blob_num": self.blob_num,
"length": self.length,
"iv": self.iv.decode()
"iv": self.iv
}
if self.blob_hash:
info['blob_hash'] = self.blob_hash
return info
class StreamBlobDecryptor:
class StreamBlobDecryptor(object):
def __init__(self, blob, key, iv, length):
"""
This class decrypts blob
@@ -53,10 +53,10 @@ class StreamBlobDecryptor:
def decrypt(self, write_func):
"""
Decrypt blob and write its content using write_func
Decrypt blob and write its content useing write_func
write_func - function that takes decrypted string as
argument and writes it somewhere
arugment and writes it somewhere
Returns:
@@ -68,14 +68,14 @@ class StreamBlobDecryptor:
def write_bytes():
if self.len_read < self.length:
num_bytes_to_decrypt = greatest_multiple(len(self.buff), (AES.block_size // 8))
num_bytes_to_decrypt = greatest_multiple(len(self.buff), (AES.block_size / 8))
data_to_decrypt, self.buff = split(self.buff, num_bytes_to_decrypt)
write_func(self.cipher.update(data_to_decrypt))
def finish_decrypt():
bytes_left = len(self.buff) % (AES.block_size // 8)
bytes_left = len(self.buff) % (AES.block_size / 8)
if bytes_left != 0:
log.warning(binascii.hexlify(self.buff[-1 * (AES.block_size // 8):]).decode())
log.warning(self.buff[-1 * (AES.block_size / 8):].encode('hex'))
raise Exception("blob %s has incorrect padding: %i bytes left" %
(self.blob.blob_hash, bytes_left))
data_to_decrypt, self.buff = self.buff, b''
@@ -99,7 +99,7 @@ class StreamBlobDecryptor:
return d
class CryptStreamBlobMaker:
class CryptStreamBlobMaker(object):
def __init__(self, key, iv, blob_num, blob):
"""
This class encrypts data and writes it to a new blob
@@ -146,7 +146,7 @@ class CryptStreamBlobMaker:
def close(self):
log.debug("closing blob %s with plaintext len %s", str(self.blob_num), str(self.length))
if self.length != 0:
self.length += (AES.block_size // 8) - (self.length % (AES.block_size // 8))
self.length += (AES.block_size / 8) - (self.length % (AES.block_size / 8))
padded_data = self.padder.finalize()
encrypted_data = self.cipher.update(padded_data) + self.cipher.finalize()
self.blob.write(encrypted_data)
@@ -5,14 +5,15 @@ import os
import logging
from cryptography.hazmat.primitives.ciphers.algorithms import AES
from twisted.internet import defer
from lbrynet.blob.CryptBlob import CryptStreamBlobMaker
from twisted.internet import interfaces, defer
from zope.interface import implements
from lbrynet.cryptstream.CryptBlob import CryptStreamBlobMaker
log = logging.getLogger(__name__)
class CryptStreamCreator:
class CryptStreamCreator(object):
"""
Create a new stream with blobs encrypted by a symmetric cipher.
@@ -21,7 +22,7 @@ class CryptStreamCreator:
the blob is associated with the stream.
"""
#implements(interfaces.IConsumer)
implements(interfaces.IConsumer)
def __init__(self, blob_manager, name=None, key=None, iv_generator=None):
"""@param blob_manager: Object that stores and provides access to blobs.
@@ -100,13 +101,13 @@ class CryptStreamCreator:
@staticmethod
def random_iv_generator():
while 1:
yield os.urandom(AES.block_size // 8)
yield os.urandom(AES.block_size / 8)
def setup(self):
"""Create the symmetric key if it wasn't provided"""
if self.key is None:
self.key = os.urandom(AES.block_size // 8)
self.key = os.urandom(AES.block_size / 8)
return defer.succeed(True)
@@ -121,7 +122,7 @@ class CryptStreamCreator:
yield defer.DeferredList(self.finished_deferreds)
self.blob_count += 1
iv = next(self.iv_generator)
iv = self.iv_generator.next()
final_blob = self._get_blob_maker(iv, self.blob_manager.get_blob_creator())
stream_terminator = yield final_blob.close()
terminator_info = yield self._blob_finished(stream_terminator)
@@ -132,7 +133,7 @@ class CryptStreamCreator:
if self.current_blob is None:
self.next_blob_creator = self.blob_manager.get_blob_creator()
self.blob_count += 1
iv = next(self.iv_generator)
iv = self.iv_generator.next()
self.current_blob = self._get_blob_maker(iv, self.next_blob_creator)
done, num_bytes_written = self.current_blob.write(data)
data = data[num_bytes_written:]
+8
View File
@@ -0,0 +1,8 @@
"""
Classes and functions for dealing with Crypt Streams.
Crypt Streams are encrypted blobs and metadata tying those blobs together. At least some of the
metadata is generally stored in a Stream Descriptor File, for example containing a public key
used to bind blobs to the stream and a symmetric key used to encrypt the blobs. The list of blobs
may or may not be present.
"""
@@ -1,10 +1,12 @@
import binascii
from zope.interface import implements
from twisted.internet import defer
from lbrynet.blob.CryptBlob import StreamBlobDecryptor
from lbrynet.cryptstream.CryptBlob import StreamBlobDecryptor
from lbrynet.interfaces import IBlobHandler
class CryptBlobHandler:
#implements(IBlobHandler)
class CryptBlobHandler(object):
implements(IBlobHandler)
def __init__(self, key, write_func):
self.key = key
@@ -1,14 +1,15 @@
import binascii
import logging
from binascii import unhexlify
from zope.interface import implements
from lbrynet.interfaces import IStreamDownloader
from lbrynet.core.client.BlobRequester import BlobRequester
from lbrynet.core.client.ConnectionManager import ConnectionManager
from lbrynet.core.client.DownloadManager import DownloadManager
from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.cryptstream.client.CryptBlobHandler import CryptBlobHandler
from twisted.internet import defer
from twisted.python.failure import Failure
from lbrynet.p2p.client.BlobRequester import BlobRequester
from lbrynet.p2p.client.ConnectionManager import ConnectionManager
from lbrynet.p2p.client.DownloadManager import DownloadManager
from lbrynet.p2p.client.StreamProgressManager import FullStreamProgressManager
from lbrynet.blob.client.CryptBlobHandler import CryptBlobHandler
log = logging.getLogger(__name__)
@@ -33,9 +34,9 @@ class CurrentlyStartingError(Exception):
pass
class CryptStreamDownloader:
class CryptStreamDownloader(object):
#implements(IStreamDownloader)
implements(IStreamDownloader)
def __init__(self, peer_finder, rate_limiter, blob_manager, payment_rate_manager, wallet,
key, stream_name):
@@ -61,8 +62,8 @@ class CryptStreamDownloader:
self.blob_manager = blob_manager
self.payment_rate_manager = payment_rate_manager
self.wallet = wallet
self.key = unhexlify(key)
self.stream_name = unhexlify(stream_name).decode()
self.key = binascii.unhexlify(key)
self.stream_name = binascii.unhexlify(stream_name)
self.completed = False
self.stopped = True
self.stopping = False
-109
View File
@@ -1,109 +0,0 @@
import os
import sys
import inspect
import logging
TRACE = 5
####
# This code is copied from logging/__init__.py in the python source code
####
#
# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
if hasattr(sys, 'frozen'): # support for py2exe
_srcfile = "logging{}__init__{}".format(os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
_srcfile = __file__[:-4] + '.py'
else:
_srcfile = __file__
_srcfile = os.path.normcase(_srcfile)
def findCaller(srcfile=None):
"""Returns the filename, line number and function name of the caller"""
srcfile = srcfile or _srcfile
f = inspect.currentframe()
# On some versions of IronPython, currentframe() returns None if
# IronPython isn't run with -X:Frames.
if f is not None:
f = f.f_back
rv = "(unknown file)", 0, "(unknown function)"
while hasattr(f, "f_code"):
co = f.f_code
filename = os.path.normcase(co.co_filename)
# ignore any function calls that are in this file
if filename == srcfile:
f = f.f_back
continue
rv = (filename, f.f_lineno, co.co_name)
break
return rv
###
class Logger(logging.Logger):
"""A logger that has an extra `fail` method useful for handling twisted failures."""
def fail(self, callback=None, *args, **kwargs):
"""Returns a function to log a failure from an errback.
The returned function appends the error message and extracts
the traceback from `err`.
Example usage:
d.addErrback(log.fail(), 'This is an error message')
Although odd, making the method call is necessary to extract
out useful filename and line number information; otherwise the
reported values are from inside twisted's deferred handling
code.
Args:
callback: callable to call after making the log. The first argument
will be the `err` from the deferred
args: extra arguments to pass into `callback`
Returns: a function that takes the following arguments:
err: twisted.python.failure.Failure
msg: the message to log, using normal logging string iterpolation.
msg_args: the values to substitute into `msg`
msg_kwargs: set `level` to change from the default ERROR severity. Other
keywoards are treated as normal log kwargs.
"""
fn, lno, func = findCaller()
def _fail(err, msg, *msg_args, **msg_kwargs):
level = msg_kwargs.pop('level', logging.ERROR)
msg += ": %s"
msg_args += (err.getErrorMessage(),)
exc_info = (err.type, err.value, err.getTracebackObject())
record = self.makeRecord(
self.name, level, fn, lno, msg, msg_args, exc_info, func, msg_kwargs)
self.handle(record)
if callback:
try:
return callback(err, *args, **kwargs)
except Exception:
# log.fail is almost always called within an
# errback. If callback fails and we didn't catch
# the exception we would need to attach a second
# errback to deal with that, which we will almost
# never do and then we end up with an unhandled
# error that will get swallowed by twisted
self.exception('Failed to run callback')
return _fail
def trace(self, msg, *args, **kwargs):
if self.isEnabledFor(TRACE):
self._log(TRACE, msg, args, **kwargs)
def install_logger():
current = logging.getLoggerClass()
if current is not Logger:
logging.setLoggerClass(Logger)
logging.addLevelName(TRACE, 'TRACE')
@@ -1,7 +1,6 @@
import logging
from twisted.internet import defer
from twisted._threads import AlreadyQuit
from lbrynet.extras.daemon.ComponentManager import ComponentManager
from ComponentManager import ComponentManager
log = logging.getLogger(__name__)
@@ -14,7 +13,7 @@ class ComponentType(type):
return klass
class Component(metaclass=ComponentType):
class Component(object):
"""
lbrynet-daemon component helper
@@ -22,6 +21,7 @@ class Component(metaclass=ComponentType):
methods
"""
__metaclass__ = ComponentType
depends_on = []
component_name = None
@@ -29,23 +29,16 @@ class Component(metaclass=ComponentType):
self.component_manager = component_manager
self._running = False
def __lt__(self, other):
return self.component_name < other.component_name
@property
def running(self):
return self._running
def get_status(self):
return
def start(self):
raise NotImplementedError()
def stop(self):
raise NotImplementedError()
@property
def component(self):
raise NotImplementedError()
@@ -55,8 +48,6 @@ class Component(metaclass=ComponentType):
result = yield defer.maybeDeferred(self.start)
self._running = True
defer.returnValue(result)
except (defer.CancelledError, AlreadyQuit):
pass
except Exception as err:
log.exception("Error setting up %s", self.component_name or self.__class__.__name__)
raise err
@@ -67,8 +58,6 @@ class Component(metaclass=ComponentType):
result = yield defer.maybeDeferred(self.stop)
self._running = False
defer.returnValue(result)
except (defer.CancelledError, AlreadyQuit):
pass
except Exception as err:
log.exception("Error stopping %s", self.__class__.__name__)
raise err
@@ -1,50 +1,25 @@
import logging
from twisted.internet import defer
from lbrynet.p2p.Error import ComponentStartConditionNotMet
from lbrynet.extras.daemon.PeerManager import PeerManager
from lbrynet.extras.daemon.PeerFinder import DHTPeerFinder
from lbrynet import conf
from lbrynet.core.Error import ComponentStartConditionNotMet
log = logging.getLogger(__name__)
class RegisteredConditions:
conditions = {}
class RequiredConditionType(type):
def __new__(mcs, name, bases, newattrs):
klass = type.__new__(mcs, name, bases, newattrs)
if name != "RequiredCondition":
if klass.name in RegisteredConditions.conditions:
raise SyntaxError("already have a component registered for \"%s\"" % klass.name)
RegisteredConditions.conditions[klass.name] = klass
return klass
class RequiredCondition(metaclass=RequiredConditionType):
name = ""
component = ""
message = ""
@staticmethod
def evaluate(component):
raise NotImplementedError()
class ComponentManager:
class ComponentManager(object):
default_component_classes = {}
def __init__(self, reactor=None, analytics_manager=None, skip_components=None,
peer_manager=None, peer_finder=None, **override_components):
def __init__(self, reactor=None, analytics_manager=None, skip_components=None, **override_components):
self.skip_components = skip_components or []
self.skip_components.extend(conf.settings['components_to_skip'])
self.reactor = reactor
self.component_classes = {}
self.components = set()
self.analytics_manager = analytics_manager
self.peer_manager = peer_manager or PeerManager()
self.peer_finder = peer_finder or DHTPeerFinder(self)
for component_name, component_class in self.default_component_classes.items():
for component_name, component_class in self.default_component_classes.iteritems():
if component_name in override_components:
component_class = override_components.pop(component_name)
if component_name not in self.skip_components:
@@ -53,21 +28,9 @@ class ComponentManager:
if override_components:
raise SyntaxError("unexpected components: %s" % override_components)
for component_class in self.component_classes.values():
for component_class in self.component_classes.itervalues():
self.components.add(component_class(self))
def evaluate_condition(self, condition_name):
if condition_name not in RegisteredConditions.conditions:
raise NameError(condition_name)
condition = RegisteredConditions.conditions[condition_name]
try:
component = self.get_component(condition.component)
result = condition.evaluate(component)
except Exception:
log.exception('failed to evaluate condition:')
result = False
return result, "" if result else condition.message
def sort_components(self, reverse=False):
"""
Sort components by requirements
@@ -85,7 +48,6 @@ class ComponentManager:
components.remove(component)
if step:
step.sort()
steps.append(step)
while components:
@@ -101,7 +63,6 @@ class ComponentManager:
to_stage.add(component.component_name)
components.remove(component)
if step:
step.sort()
staged.update(to_stage)
steps.append(step)
elif components:
@@ -117,23 +78,23 @@ class ComponentManager:
:return: (defer.Deferred)
"""
for component_name, cb in callbacks.items():
for component_name, cb in callbacks.iteritems():
if component_name not in self.component_classes:
if component_name not in self.skip_components:
raise NameError("unknown component: %s" % component_name)
raise NameError("unknown component: %s" % component_name)
if not callable(cb):
raise ValueError("%s is not callable" % cb)
def _setup(component):
if component.component_name in callbacks:
d = component._setup()
d.addCallback(callbacks[component.component_name], component)
d.addCallback(callbacks[component.component_name])
return d
return component._setup()
stages = self.sort_components()
for stage in stages:
yield defer.DeferredList([_setup(component) for component in stage if not component.running])
yield defer.DeferredList([_setup(component) for component in stage])
@defer.inlineCallbacks
def stop(self):
+523
View File
@@ -0,0 +1,523 @@
import os
import logging
import miniupnpc
from twisted.internet import defer, threads, reactor, error
from lbrynet import conf
from lbrynet.core.Session import Session
from lbrynet.core.StreamDescriptor import StreamDescriptorIdentifier, EncryptedFileStreamType
from lbrynet.core.Wallet import LBRYumWallet
from lbrynet.core.server.BlobRequestHandler import BlobRequestHandlerFactory
from lbrynet.core.server.ServerProtocol import ServerProtocolFactory
from lbrynet.daemon.Component import Component
from lbrynet.database.storage import SQLiteStorage
from lbrynet.dht import node, hashannouncer
from lbrynet.file_manager.EncryptedFileManager import EncryptedFileManager
from lbrynet.lbry_file.client.EncryptedFileDownloader import EncryptedFileSaverFactory
from lbrynet.lbry_file.client.EncryptedFileOptions import add_lbry_file_to_sd_identifier
from lbrynet.reflector import ServerFactory as reflector_server_factory
from lbrynet.core.utils import generate_id
log = logging.getLogger(__name__)
# settings must be initialized before this file is imported
DATABASE_COMPONENT = "database"
WALLET_COMPONENT = "wallet"
SESSION_COMPONENT = "session"
DHT_COMPONENT = "dht"
HASH_ANNOUNCER_COMPONENT = "hashAnnouncer"
STREAM_IDENTIFIER_COMPONENT = "streamIdentifier"
FILE_MANAGER_COMPONENT = "fileManager"
PEER_PROTOCOL_SERVER_COMPONENT = "peerProtocolServer"
REFLECTOR_COMPONENT = "reflector"
UPNP_COMPONENT = "upnp"
class ConfigSettings(object):
@staticmethod
def get_conf_setting(setting_name):
return conf.settings[setting_name]
@staticmethod
def get_blobfiles_dir():
if conf.settings['BLOBFILES_DIR'] == "blobfiles":
return os.path.join(GCS("data_dir"), "blobfiles")
else:
log.info("Using non-default blobfiles directory: %s", conf.settings['BLOBFILES_DIR'])
return conf.settings['BLOBFILES_DIR']
@staticmethod
def get_node_id():
return conf.settings.node_id
@staticmethod
def get_external_ip():
from lbrynet.core.system_info import get_platform
platform = get_platform(get_ip=True)
return platform['ip']
# Shorthand for common ConfigSettings methods
CS = ConfigSettings
GCS = ConfigSettings.get_conf_setting
class DatabaseComponent(Component):
component_name = DATABASE_COMPONENT
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.storage = None
@property
def component(self):
return self.storage
@staticmethod
def get_current_db_revision():
return 9
@staticmethod
def get_revision_filename():
return conf.settings.get_db_revision_filename()
@staticmethod
def _write_db_revision_file(version_num):
with open(conf.settings.get_db_revision_filename(), mode='w') as db_revision:
db_revision.write(str(version_num))
@defer.inlineCallbacks
def start(self):
# check directories exist, create them if they don't
log.info("Loading databases")
if not os.path.exists(GCS('download_directory')):
os.mkdir(GCS('download_directory'))
if not os.path.exists(GCS('data_dir')):
os.mkdir(GCS('data_dir'))
self._write_db_revision_file(self.get_current_db_revision())
log.debug("Created the db revision file: %s", self.get_revision_filename())
if not os.path.exists(CS.get_blobfiles_dir()):
os.mkdir(CS.get_blobfiles_dir())
log.debug("Created the blobfile directory: %s", str(CS.get_blobfiles_dir()))
if not os.path.exists(self.get_revision_filename()):
log.warning("db_revision file not found. Creating it")
self._write_db_revision_file(self.get_current_db_revision())
# check the db migration and run any needed migrations
with open(self.get_revision_filename(), "r") as revision_read_handle:
old_revision = int(revision_read_handle.read().strip())
if old_revision > self.get_current_db_revision():
raise Exception('This version of lbrynet is not compatible with the database\n'
'Your database is revision %i, expected %i' %
(old_revision, self.get_current_db_revision()))
if old_revision < self.get_current_db_revision():
from lbrynet.database.migrator import dbmigrator
log.info("Upgrading your databases (revision %i to %i)", old_revision, self.get_current_db_revision())
yield threads.deferToThread(
dbmigrator.migrate_db, GCS('data_dir'), old_revision, self.get_current_db_revision()
)
self._write_db_revision_file(self.get_current_db_revision())
log.info("Finished upgrading the databases.")
# start SQLiteStorage
self.storage = SQLiteStorage(GCS('data_dir'))
yield self.storage.setup()
@defer.inlineCallbacks
def stop(self):
yield self.storage.stop()
self.storage = None
class WalletComponent(Component):
component_name = WALLET_COMPONENT
depends_on = [DATABASE_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.wallet = None
@property
def component(self):
return self.wallet
@defer.inlineCallbacks
def start(self):
storage = self.component_manager.get_component(DATABASE_COMPONENT)
wallet_type = GCS('wallet')
if wallet_type == conf.LBRYCRD_WALLET:
raise ValueError('LBRYcrd Wallet is no longer supported')
elif wallet_type == conf.LBRYUM_WALLET:
log.info("Using lbryum wallet")
lbryum_servers = {address: {'t': str(port)}
for address, port in GCS('lbryum_servers')}
config = {
'auto_connect': True,
'chain': GCS('blockchain_name'),
'default_servers': lbryum_servers
}
if 'use_keyring' in conf.settings:
config['use_keyring'] = GCS('use_keyring')
if conf.settings['lbryum_wallet_dir']:
config['lbryum_path'] = GCS('lbryum_wallet_dir')
self.wallet = LBRYumWallet(storage, config)
yield self.wallet.start()
else:
raise ValueError('Wallet Type {} is not valid'.format(wallet_type))
@defer.inlineCallbacks
def stop(self):
yield self.wallet.stop()
self.wallet = None
class SessionComponent(Component):
component_name = SESSION_COMPONENT
depends_on = [DATABASE_COMPONENT, WALLET_COMPONENT, DHT_COMPONENT, HASH_ANNOUNCER_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.session = None
@property
def component(self):
return self.session
@defer.inlineCallbacks
def start(self):
self.session = Session(
GCS('data_rate'),
db_dir=GCS('data_dir'),
node_id=CS.get_node_id(),
blob_dir=CS.get_blobfiles_dir(),
dht_node=self.component_manager.get_component(DHT_COMPONENT),
hash_announcer=self.component_manager.get_component(HASH_ANNOUNCER_COMPONENT),
dht_node_port=GCS('dht_node_port'),
known_dht_nodes=GCS('known_dht_nodes'),
peer_port=GCS('peer_port'),
use_upnp=GCS('use_upnp'),
wallet=self.component_manager.get_component(WALLET_COMPONENT),
is_generous=GCS('is_generous_host'),
external_ip=CS.get_external_ip(),
storage=self.component_manager.get_component(DATABASE_COMPONENT)
)
yield self.session.setup()
@defer.inlineCallbacks
def stop(self):
yield self.session.shut_down()
class DHTComponent(Component):
component_name = DHT_COMPONENT
depends_on = [UPNP_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.dht_node = None
self.upnp_component = None
self.udp_port, self.peer_port = None, None
@property
def component(self):
return self.dht_node
@defer.inlineCallbacks
def start(self):
self.upnp_component = self.component_manager.get_component(UPNP_COMPONENT)
self.udp_port, self.peer_port = self.upnp_component.get_redirects()
node_id = CS.get_node_id()
if node_id is None:
node_id = generate_id()
self.dht_node = node.Node(
node_id=node_id,
udpPort=self.udp_port,
externalIP=CS.get_external_ip(),
peerPort=self.peer_port
)
yield self.dht_node.start(GCS('known_dht_nodes'))
log.info("Joined the dht")
@defer.inlineCallbacks
def stop(self):
yield self.dht_node.stop()
class HashAnnouncer(Component):
component_name = HASH_ANNOUNCER_COMPONENT
depends_on = [DHT_COMPONENT, DATABASE_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.hash_announcer = None
@property
def component(self):
return self.hash_announcer
@defer.inlineCallbacks
def start(self):
storage = self.component_manager.get_component(DATABASE_COMPONENT)
dht_node = self.component_manager.get_component(DHT_COMPONENT)
self.hash_announcer = hashannouncer.DHTHashAnnouncer(dht_node, storage)
yield self.hash_announcer.start()
@defer.inlineCallbacks
def stop(self):
yield self.hash_announcer.stop()
class StreamIdentifier(Component):
component_name = STREAM_IDENTIFIER_COMPONENT
depends_on = [SESSION_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.sd_identifier = StreamDescriptorIdentifier()
@property
def component(self):
return self.sd_identifier
@defer.inlineCallbacks
def start(self):
session = self.component_manager.get_component(SESSION_COMPONENT)
add_lbry_file_to_sd_identifier(self.sd_identifier)
file_saver_factory = EncryptedFileSaverFactory(
session.peer_finder,
session.rate_limiter,
session.blob_manager,
session.storage,
session.wallet,
GCS('download_directory')
)
yield self.sd_identifier.add_stream_downloader_factory(EncryptedFileStreamType, file_saver_factory)
def stop(self):
pass
class FileManager(Component):
component_name = FILE_MANAGER_COMPONENT
depends_on = [SESSION_COMPONENT, STREAM_IDENTIFIER_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.file_manager = None
@property
def component(self):
return self.file_manager
@defer.inlineCallbacks
def start(self):
session = self.component_manager.get_component(SESSION_COMPONENT)
sd_identifier = self.component_manager.get_component(STREAM_IDENTIFIER_COMPONENT)
log.info('Starting the file manager')
self.file_manager = EncryptedFileManager(session, sd_identifier)
yield self.file_manager.setup()
log.info('Done setting up file manager')
@defer.inlineCallbacks
def stop(self):
yield self.file_manager.stop()
class PeerProtocolServer(Component):
component_name = PEER_PROTOCOL_SERVER_COMPONENT
depends_on = [SESSION_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.lbry_server_port = None
@property
def component(self):
return self.lbry_server_port
@defer.inlineCallbacks
def start(self):
query_handlers = {}
peer_port = GCS('peer_port')
session = self.component_manager.get_component(SESSION_COMPONENT)
handlers = [
BlobRequestHandlerFactory(
session.blob_manager,
session.wallet,
session.payment_rate_manager,
self.component_manager.analytics_manager
),
session.wallet.get_wallet_info_query_handler_factory(),
]
for handler in handlers:
query_id = handler.get_primary_query_identifier()
query_handlers[query_id] = handler
if peer_port is not None:
server_factory = ServerProtocolFactory(session.rate_limiter, query_handlers, session.peer_manager)
try:
log.info("Peer protocol listening on TCP %d", peer_port)
self.lbry_server_port = yield reactor.listenTCP(peer_port, server_factory)
except error.CannotListenError as e:
import traceback
log.error("Couldn't bind to port %d. Visit lbry.io/faq/how-to-change-port for"
" more details.", peer_port)
log.error("%s", traceback.format_exc())
raise ValueError("%s lbrynet may already be running on your computer." % str(e))
@defer.inlineCallbacks
def stop(self):
if self.lbry_server_port is not None:
self.lbry_server_port, old_port = None, self.lbry_server_port
log.info('Stop listening on port %s', old_port.port)
yield old_port.stopListening()
class ReflectorComponent(Component):
component_name = REFLECTOR_COMPONENT
depends_on = [SESSION_COMPONENT, FILE_MANAGER_COMPONENT]
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.reflector_server_port = GCS('reflector_port')
self.run_reflector_server = GCS('run_reflector_server')
@property
def component(self):
return self
@defer.inlineCallbacks
def start(self):
session = self.component_manager.get_component(SESSION_COMPONENT)
file_manager = self.component_manager.get_component(FILE_MANAGER_COMPONENT)
if self.run_reflector_server and self.reflector_server_port is not None:
log.info("Starting reflector server")
reflector_factory = reflector_server_factory(session.peer_manager, session.blob_manager, file_manager)
try:
self.reflector_server_port = yield reactor.listenTCP(self.reflector_server_port, reflector_factory)
log.info('Started reflector on port %s', self.reflector_server_port)
except error.CannotListenError as e:
log.exception("Couldn't bind reflector to port %d", self.reflector_server_port)
raise ValueError("{} lbrynet may already be running on your computer.".format(e))
def stop(self):
if self.run_reflector_server and self.reflector_server_port is not None:
log.info("Stopping reflector server")
if self.reflector_server_port is not None:
self.reflector_server_port, p = None, self.reflector_server_port
yield p.stopListening
class UPnPComponent(Component):
component_name = UPNP_COMPONENT
def __init__(self, component_manager):
Component.__init__(self, component_manager)
self.peer_port = GCS('peer_port')
self.dht_node_port = GCS('dht_node_port')
self.use_upnp = GCS('use_upnp')
self.external_ip = CS.get_external_ip()
self.upnp_redirects = []
@property
def component(self):
return self
def get_redirects(self):
return self.peer_port, self.dht_node_port
def start(self):
log.debug("In _try_upnp")
def get_free_port(upnp, port, protocol):
# returns an existing mapping if it exists
mapping = upnp.getspecificportmapping(port, protocol)
if not mapping:
return port
if upnp.lanaddr == mapping[0]:
return mapping[1]
return get_free_port(upnp, port + 1, protocol)
def get_port_mapping(upnp, port, protocol, description):
# try to map to the requested port, if there is already a mapping use the next external
# port available
if protocol not in ['UDP', 'TCP']:
raise Exception("invalid protocol")
port = get_free_port(upnp, port, protocol)
if isinstance(port, tuple):
log.info("Found existing UPnP redirect %s:%i (%s) to %s:%i, using it",
self.external_ip, port, protocol, upnp.lanaddr, port)
return port
upnp.addportmapping(port, protocol, upnp.lanaddr, port,
description, '')
log.info("Set UPnP redirect %s:%i (%s) to %s:%i", self.external_ip, port,
protocol, upnp.lanaddr, port)
return port
def threaded_try_upnp():
if self.use_upnp is False:
log.debug("Not using upnp")
return False
u = miniupnpc.UPnP()
num_devices_found = u.discover()
if num_devices_found > 0:
u.selectigd()
external_ip = u.externalipaddress()
if external_ip != '0.0.0.0' and not self.external_ip:
# best not to rely on this external ip, the router can be behind layers of NATs
self.external_ip = external_ip
if self.peer_port:
self.peer_port = get_port_mapping(u, self.peer_port, 'TCP', 'LBRY peer port')
self.upnp_redirects.append((self.peer_port, 'TCP'))
if self.dht_node_port:
self.dht_node_port = get_port_mapping(u, self.dht_node_port, 'UDP', 'LBRY DHT port')
self.upnp_redirects.append((self.dht_node_port, 'UDP'))
return True
return False
def upnp_failed(err):
log.warning("UPnP failed. Reason: %s", err.getErrorMessage())
return False
d = threads.deferToThread(threaded_try_upnp)
d.addErrback(upnp_failed)
return d
def stop(self):
log.info("Unsetting upnp for session")
def threaded_unset_upnp():
u = miniupnpc.UPnP()
num_devices_found = u.discover()
if num_devices_found > 0:
u.selectigd()
for port, protocol in self.upnp_redirects:
if u.getspecificportmapping(port, protocol) is None:
log.warning(
"UPnP redirect for %s %d was removed by something else.",
protocol, port)
else:
u.deleteportmapping(port, protocol)
log.info("Removed UPnP redirect for %s %d.", protocol, port)
self.upnp_redirects = []
d = threads.deferToThread(threaded_unset_upnp)
d.addErrback(lambda err: str(err))
return d
File diff suppressed because it is too large. Load diff
+224
View File
@@ -0,0 +1,224 @@
import json
import os
import sys
import colorama
from docopt import docopt
from collections import OrderedDict
from lbrynet import conf
from lbrynet.core import utils
from lbrynet.daemon.auth.client import JSONRPCException, LBRYAPIClient, AuthAPIClient
from lbrynet.daemon.Daemon import Daemon
from lbrynet.core.system_info import get_platform
from jsonrpc.common import RPCError
from requests.exceptions import ConnectionError
from urllib2 import URLError, HTTPError
from httplib import UNAUTHORIZED
def remove_brackets(key):
if key.startswith("<") and key.endswith(">"):
return str(key[1:-1])
return key
def set_kwargs(parsed_args):
kwargs = OrderedDict()
for key, arg in parsed_args.iteritems():
if arg is None:
continue
elif key.startswith("--") and remove_brackets(key[2:]) not in kwargs:
k = remove_brackets(key[2:])
elif remove_brackets(key) not in kwargs:
k = remove_brackets(key)
kwargs[k] = guess_type(arg, k)
return kwargs
def main():
argv = sys.argv[1:]
# check if a config file has been specified. If so, shift
# all the arguments so that the parsing can continue without
# noticing
if len(argv) and argv[0] == "--conf":
if len(argv) < 2:
print_error("No config file specified for --conf option")
print_help()
return
conf.conf_file = argv[1]
argv = argv[2:]
if len(argv):
method, args = argv[0], argv[1:]
else:
print_help()
return
if method in ['help', '--help', '-h']:
if len(args) == 1:
print_help_for_command(args[0])
else:
print_help()
return
elif method in ['version', '--version']:
print utils.json_dumps_pretty(get_platform(get_ip=False))
return
if method not in Daemon.callable_methods:
if method not in Daemon.deprecated_methods:
print_error("\"%s\" is not a valid command." % method)
return
new_method = Daemon.deprecated_methods[method]._new_command
print_error("\"%s\" is deprecated, using \"%s\"." % (method, new_method))
method = new_method
fn = Daemon.callable_methods[method]
parsed = docopt(fn.__doc__, args)
kwargs = set_kwargs(parsed)
colorama.init()
conf.initialize_settings()
try:
api = LBRYAPIClient.get_client()
api.status()
except (URLError, ConnectionError) as err:
if isinstance(err, HTTPError) and err.code == UNAUTHORIZED:
api = AuthAPIClient.config()
# this can happen if the daemon is using auth with the --http-auth flag
# when the config setting is to not use it
try:
api.status()
except:
print_error("Daemon requires authentication, but none was provided.",
suggest_help=False)
return 1
else:
print_error("Could not connect to daemon. Are you sure it's running?",
suggest_help=False)
return 1
# TODO: check if port is bound. Error if its not
try:
result = api.call(method, kwargs)
if isinstance(result, basestring):
# printing the undumped string is prettier
print result
else:
print utils.json_dumps_pretty(result)
except (RPCError, KeyError, JSONRPCException, HTTPError) as err:
if isinstance(err, HTTPError):
error_body = err.read()
try:
error_data = json.loads(error_body)
except ValueError:
print (
"There was an error, and the response was not valid JSON.\n" +
"Raw JSONRPC response:\n" + error_body
)
return 1
print_error(error_data['error']['message'] + "\n", suggest_help=False)
if 'data' in error_data['error'] and 'traceback' in error_data['error']['data']:
print "Here's the traceback for the error you encountered:"
print "\n".join(error_data['error']['data']['traceback'])
print_help_for_command(method)
elif isinstance(err, RPCError):
print_error(err.msg, suggest_help=False)
# print_help_for_command(method)
else:
print_error("Something went wrong\n", suggest_help=False)
print str(err)
return 1
def guess_type(x, key=None):
if not isinstance(x, (unicode, str)):
return x
if key in ('uri', 'channel_name', 'name', 'file_name', 'download_directory'):
return x
if x in ('true', 'True', 'TRUE'):
return True
if x in ('false', 'False', 'FALSE'):
return False
if '.' in x:
try:
return float(x)
except ValueError:
# not a float
pass
try:
return int(x)
except ValueError:
return x
def print_help_suggestion():
print "See `{} help` for more information.".format(os.path.basename(sys.argv[0]))
def print_error(message, suggest_help=True):
error_style = colorama.Style.BRIGHT + colorama.Fore.RED
print error_style + "ERROR: " + message + colorama.Style.RESET_ALL
if suggest_help:
print_help_suggestion()
def print_help():
print "\n".join([
"NAME",
" lbrynet-cli - LBRY command line client.",
"",
"USAGE",
" lbrynet-cli [--conf <config file>] <command> [<args>]",
"",
"EXAMPLES",
" lbrynet-cli commands # list available commands",
" lbrynet-cli status # get daemon status",
" lbrynet-cli --conf ~/l1.conf status # like above but using ~/l1.conf as config file",
" lbrynet-cli resolve_name what # resolve a name",
" lbrynet-cli help resolve_name # get help for a command",
])
def print_help_for_command(command):
fn = Daemon.callable_methods.get(command)
if fn:
print "Help for %s method:\n%s" % (command, fn.__doc__)
def wrap_list_to_term_width(l, width=None, separator=', ', prefix=''):
if width is None:
try:
_, width = os.popen('stty size', 'r').read().split()
width = int(width)
except:
pass
if not width:
width = 80
lines = []
curr_line = ''
for item in l:
new_line = curr_line + item + separator
if len(new_line) + len(prefix) > width:
lines.append(curr_line)
curr_line = item + separator
else:
curr_line = new_line
lines.append(curr_line)
ret = prefix + ("\n" + prefix).join(lines)
if ret.endswith(separator):
ret = ret[:-len(separator)]
return ret
if __name__ == '__main__':
sys.exit(main())
Loaded 100 of 339 files, more files were not shown because too many files have changed in this diff. Show more