Compare commits

...
Author SHA1 Message Date
Lex Berezhny 571b02f381 v0.38.0rc7 2019-06-19 06:44:26 -04:00
Lex Berezhny 36e3f1b3ec integration test fix 2019-06-19 06:33:12 -04:00
Lex Berezhny 564d6bdc5d more refactoring 2019-06-19 06:33:12 -04:00
Victor Shyba 1e2b040f75 simplify validation, move to fee.update 2019-06-19 06:33:12 -04:00
Victor Shyba 5876d5c295 cover more edge cases, add tests 2019-06-19 06:33:12 -04:00
Victor Shyba 54e53932bd fix bug: fee address defaults respect replace 2019-06-19 06:33:12 -04:00
Victor Shyba c9b3c2e6aa change test to use preview=True on broadcast 2019-06-19 06:33:12 -04:00
Victor Shyba 912c102d9a fix updating fee address to claim address, fix testing for malformed claims 2019-06-19 06:33:12 -04:00
Victor Shyba 2b7cbf9759 get without fee address defaults pays to claim address 2019-06-19 06:33:12 -04:00
Oleg Silkin 1cc6b2e31e More integration unittests and mock server fixes 2019-06-19 06:25:26 -04:00
Oleg Silkin 7fa797712a Adds comment signature unit testing 2019-06-19 06:25:26 -04:00
Oleg Silkin 636f36ebad Moves get_channel,get_stream out of Class into their own functions 2019-06-19 06:25:26 -04:00
Oleg Silkin 8e0888b846 Fixes comment functions 2019-06-19 06:25:26 -04:00
Oleg Silkin ce5dd5022b Adds Comment Signing and Validation 2019-06-19 06:25:26 -04:00
Thomas Zarebczan 756d17bada update wallet servers
Not sure where spv3 will be yet. We can probably repurpose lbryumx2/3/4 once we go live, so we'll make sure to add those before release.
2019-06-19 02:53:41 -04:00
Lex Berezhny 81e061af30 added missing raise SystemExit() 2019-06-19 02:53:22 -04:00
Alex Grintsvayg a60cd1d475 separate try/except for media server 2019-06-19 02:53:22 -04:00
Lex Berezhny f78323e0fb update torba version 2019-06-19 02:17:08 -04:00
Lex Berezhny fd6cbe0f6c reorgs do not hang the wallet server 2019-06-19 02:17:08 -04:00
Jack Robison bfeed40bfe Merge pull request #2247 from lbryio/fix-unable-to-serialize-response
paginated deterministically shuffled find_value
2019-06-18 23:25:12 -04:00
Jack Robison 2ad22d7d19 paginated deterministically shuffled find_value
- fixes https://github.com/lbryio/lbry/issues/2244
- reduce the max DHT datagram size to 1400 bytes
- truncate `contacts` field of find_value response datagrams to the k closest (8)
- truncate peers in find_node response datagrams to the 2k closest (16)
- remove `contacts` field from find_value responses beyond `page` 0 (the first/default)
- deterministically shuffle the peers for a blob in a find_value response
- add optional `page` argument to `find_value` and `p` field to find_value responses containing the number of pages of k peers for the blob
- test one blob being announced by 150 different peers to one peer
- speed up pylint and remove some disabled checks
2019-06-18 22:52:39 -04:00
Lex Berezhny 262735c219 --blocking 2019-06-18 19:32:49 -04:00
Alex Grin 3abc98acc8 Update PULL_REQUEST_TEMPLATE.md 2019-06-18 09:17:54 -04:00
Alex Grin eaa8ba814e Update ISSUE_TEMPLATE.md 2019-06-17 14:06:01 -04:00
Jack Robison 588568752b Merge pull request #2217 from lbryio/chunked-reflector-response
fix handling chunked reflector server responses
2019-06-11 14:05:06 -04:00
Jack Robison a30e2daf05 handle chunked responses from reflector server 2019-06-07 17:26:31 -04:00
Jack Robison adb7edd3b2 test chunked reflector server response 2019-06-07 17:26:31 -04:00
28 changed files with 603 additions and 248 deletions

No files matched your search

-21
View File
@@ -51,24 +51,3 @@ Tell us what happens instead
## Screenshots
<!-- If a screenshot would help explain the bug, please include one or two here -->
## Internal Use
### Acceptance Criteria
1.
2.
3.
### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed
+1 -26
View File
@@ -1,30 +1,6 @@
## PR Checklist
Please check all that apply to this PR using "x":
- [ ] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
- [ ] I have checked that this PR does not introduce a breaking change
- [ ] This PR introduces breaking changes and I have provided a detailed explanation below
## PR Type
What kind of change does this PR introduce?
Why is this change necessary?
<!-- Please check all that apply to this PR using "x". -->
- [ ] Bugfix
- [ ] Feature
- [ ] Breaking changes (bugfix or feature that introduces breaking changes)
- [ ] Code style update (formatting)
- [ ] Refactoring (no functional changes)
- [ ] Documentation changes
- [ ] Other - Please describe:
## Fixes
Issue Number: N/A
Link to issue:
## What is the current behavior?
@@ -34,5 +10,4 @@ Issue Number: N/A
## Other information
<!-- If this PR contains a breaking change, please describe the impact and solution strategy for existing applications below. -->
+1 -5
View File
@@ -24,7 +24,7 @@ persistent=yes
load-plugins=
# Use multiple processes to speed up Pylint.
jobs=1
jobs=4
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
@@ -77,8 +77,6 @@ disable=
dangerous-default-value,
duplicate-code,
fixme,
global-statement,
inherit-non-class,
invalid-name,
len-as-condition,
locally-disabled,
@@ -111,13 +109,11 @@ disable=
unnecessary-lambda,
unused-argument,
unused-variable,
wildcard-import,
wrong-import-order,
wrong-import-position,
deprecated-lambda,
simplifiable-if-statement,
unidiomatic-typecheck,
global-at-module-level,
inconsistent-return-statements,
keyword-arg-before-vararg,
assignment-from-no-return,
+1 -1
View File
@@ -1,4 +1,4 @@
__name__ = "lbrynet"
__version__ = "0.38.0rc6"
__version__ = "0.38.0rc7"
version = tuple(__version__.split('.'))
+3 -3
View File
@@ -529,9 +529,9 @@ class Config(CLIConfig):
('reflector.lbry.com', 5566)
])
lbryum_servers = Servers("SPV wallet servers", [
('lbryumx1.lbry.com', 50001), # US EAST
('lbryumx2.lbry.com', 50001), # US EAST
('lbryumx4.lbry.com', 50001) #EU
('spv1.lbry.com', 50001), # US EAST
('spv2.lbry.com', 50001), # EU
('spv3.lbry.com', 50001) # TBD
])
known_dht_nodes = Servers("Known nodes for bootstrapping connection to the DHT", [
('lbrynet1.lbry.com', 4444), # US EAST
+1 -2
View File
@@ -18,11 +18,10 @@ data_expiration = 86400 # 24 hours
token_secret_refresh_interval = 300 # 5 minutes
maybe_ping_delay = 300 # 5 minutes
check_refresh_interval = refresh_interval / 5
max_datagram_size = 8192 # 8 KB
rpc_id_length = 20
protocol_version = 1
bottom_out_limit = 3
msg_size_limit = max_datagram_size - 26
msg_size_limit = 1400
def digest(data: bytes) -> bytes:
+38 -23
View File
@@ -5,12 +5,13 @@ import hashlib
import asyncio
import typing
import binascii
import random
from asyncio.protocols import DatagramProtocol
from asyncio.transports import DatagramTransport
from lbrynet.dht import constants
from lbrynet.dht.serialization.datagram import decode_datagram, ErrorDatagram, ResponseDatagram, RequestDatagram
from lbrynet.dht.serialization.datagram import RESPONSE_TYPE, ERROR_TYPE
from lbrynet.dht.serialization.datagram import RESPONSE_TYPE, ERROR_TYPE, PAGE_KEY
from lbrynet.dht.error import RemoteException, TransportNotConnected
from lbrynet.dht.protocol.routing_table import TreeRoutingTable
from lbrynet.dht.protocol.data_store import DictDataStore
@@ -67,19 +68,23 @@ class KademliaRPC:
contacts = self.protocol.routing_table.find_close_peers(key, sender_node_id=rpc_contact.node_id)
contact_triples = []
for contact in contacts:
for contact in contacts[:constants.k * 2]:
contact_triples.append((contact.node_id, contact.address, contact.udp_port))
return contact_triples
def find_value(self, rpc_contact: 'KademliaPeer', key: bytes):
def find_value(self, rpc_contact: 'KademliaPeer', key: bytes, page: int = 0):
page = page if page > 0 else 0
if len(key) != constants.hash_length:
raise ValueError("invalid blob_exchange hash length: %i" % len(key))
response = {
b'token': self.make_token(rpc_contact.compact_ip()),
b'contacts': self.find_node(rpc_contact, key)
}
if not page:
response[b'contacts'] = self.find_node(rpc_contact, key)[:constants.k]
if self.protocol.protocol_version:
response[b'protocolVersion'] = self.protocol.protocol_version
@@ -92,8 +97,14 @@ class KademliaRPC:
# if we don't have k storing peers to return and we have this hash locally, include our contact information
if len(peers) < constants.k and binascii.hexlify(key).decode() in self.protocol.data_store.completed_blobs:
peers.append(self.compact_address())
if peers:
response[key] = peers
if not peers:
response[PAGE_KEY] = 0
else:
response[PAGE_KEY] = (len(peers) // (constants.k + 1)) + 1 # how many pages of peers we have for the blob
if len(peers) > constants.k:
random.Random(self.protocol.node_id).shuffle(peers)
if page * constants.k < len(peers):
response[key] = peers[page * constants.k:page * constants.k + constants.k]
return response
def refresh_token(self): # TODO: this needs to be called periodically
@@ -166,7 +177,7 @@ class RemoteKademliaRPC:
)
return [(node_id, address.decode(), udp_port) for node_id, address, udp_port in response.response]
async def find_value(self, key: bytes) -> typing.Union[typing.Dict]:
async def find_value(self, key: bytes, page: int = 0) -> typing.Union[typing.Dict]:
"""
:return: {
b'token': <token bytes>,
@@ -177,7 +188,7 @@ class RemoteKademliaRPC:
if len(key) != constants.hash_bits // 8:
raise ValueError(f"invalid length of find value key: {len(key)}")
response = await self.protocol.send_request(
self.peer, RequestDatagram.make_find_value(self.protocol.node_id, key)
self.peer, RequestDatagram.make_find_value(self.protocol.node_id, key, page=page)
)
self.peer_tracker.update_token(self.peer.node_id, response.response[b'token'])
return response.response
@@ -406,24 +417,25 @@ class KademliaProtocol(DatagramProtocol):
raise AttributeError('Invalid method: %s' % message.method.decode())
if message.args and isinstance(message.args[-1], dict) and b'protocolVersion' in message.args[-1]:
# args don't need reformatting
a, kw = tuple(message.args[:-1]), message.args[-1]
args, kw = tuple(message.args[:-1]), message.args[-1]
else:
a, kw = self._migrate_incoming_rpc_args(sender_contact, message.method, *message.args)
args, kw = self._migrate_incoming_rpc_args(sender_contact, message.method, *message.args)
log.debug("%s:%i RECV CALL %s %s:%i", self.external_ip, self.udp_port, message.method.decode(),
sender_contact.address, sender_contact.udp_port)
if method == b'ping':
result = self.node_rpc.ping()
elif method == b'store':
blob_hash, token, port, original_publisher_id, age = a
blob_hash, token, port, original_publisher_id, age = args[:5]
result = self.node_rpc.store(sender_contact, blob_hash, token, port)
elif method == b'findNode':
key, = a
result = self.node_rpc.find_node(sender_contact, key)
else:
assert method == b'findValue'
key, = a
result = self.node_rpc.find_value(sender_contact, key)
key = args[0]
page = kw.get(PAGE_KEY, 0)
if method == b'findNode':
result = self.node_rpc.find_node(sender_contact, key)
else:
assert method == b'findValue'
result = self.node_rpc.find_value(sender_contact, key, page)
self.send_response(
sender_contact, ResponseDatagram(RESPONSE_TYPE, message.rpc_id, self.node_id, result),
@@ -569,15 +581,18 @@ class KademliaProtocol(DatagramProtocol):
def send_error(self, peer: 'KademliaPeer', error: ErrorDatagram):
self._send(peer, error)
def _send(self, peer: 'KademliaPeer', message: typing.Union[RequestDatagram, ResponseDatagram,
ErrorDatagram]):
def _send(self, peer: 'KademliaPeer', message: typing.Union[RequestDatagram, ResponseDatagram, ErrorDatagram]):
if not self.transport or self.transport.is_closing():
raise TransportNotConnected()
data = message.bencode()
if len(data) > constants.msg_size_limit:
log.exception("unexpected: %i vs %i", len(data), constants.msg_size_limit)
raise ValueError()
log.warning("cannot send datagram larger than %i bytes (packet is %i bytes)",
constants.msg_size_limit, len(data))
log.debug("Packet is too large to send: %s", binascii.hexlify(data[:3500]).decode())
raise ValueError(
f"cannot send datagram larger than {constants.msg_size_limit} bytes (packet is {len(data)} bytes)"
)
if isinstance(message, (RequestDatagram, ResponseDatagram)):
assert message.node_id == self.node_id, message
if isinstance(message, RequestDatagram):
@@ -642,12 +657,12 @@ class KademliaProtocol(DatagramProtocol):
log.debug("Timeout while storing blob_hash %s at %s", binascii.hexlify(hash_value).decode()[:8], peer)
except ValueError as err:
log.error("Unexpected response: %s" % err)
except Exception as err:
except RemoteException as err:
if 'Invalid token' in str(err):
self.peer_manager.clear_token(peer.node_id)
try:
return await __store()
except:
except (ValueError, asyncio.TimeoutError, RemoteException):
return peer.node_id, False
else:
log.exception("Unexpected error while storing blob_hash")
+1 -1
View File
@@ -63,7 +63,7 @@ def bencode(data: typing.Dict) -> bytes:
def bdecode(data: bytes, allow_non_dict_return: typing.Optional[bool] = False) -> typing.Dict:
assert type(data) == bytes, DecodeError(f"invalid data type: {str(type(data))}")
assert isinstance(data, bytes), DecodeError(f"invalid data type: {str(type(data))}")
if len(data) == 0:
raise DecodeError('Cannot decode empty string')
+7 -2
View File
@@ -7,6 +7,9 @@ REQUEST_TYPE = 0
RESPONSE_TYPE = 1
ERROR_TYPE = 2
# bencode representation of argument keys
PAGE_KEY = b'p'
class KademliaDatagramBase:
"""
@@ -91,11 +94,13 @@ class RequestDatagram(KademliaDatagramBase):
@classmethod
def make_find_value(cls, from_node_id: bytes, key: bytes,
rpc_id: typing.Optional[bytes] = None) -> 'RequestDatagram':
rpc_id: typing.Optional[bytes] = None, page: int = 0) -> 'RequestDatagram':
rpc_id = rpc_id or constants.generate_id()[:constants.rpc_id_length]
if len(key) != constants.hash_bits // 8:
raise ValueError(f"invalid key length: {len(key)}")
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'findValue', [key])
if page < 0:
raise ValueError(f"cannot request a negative page ({page})")
return cls(REQUEST_TYPE, rpc_id, from_node_id, b'findValue', [key, {PAGE_KEY: page}])
class ResponseDatagram(KademliaDatagramBase):
+100 -68
View File
@@ -33,7 +33,7 @@ from lbrynet.extras.daemon.Components import EXCHANGE_RATE_MANAGER_COMPONENT, UP
from lbrynet.extras.daemon.ComponentManager import RequiredCondition
from lbrynet.extras.daemon.ComponentManager import ComponentManager
from lbrynet.extras.daemon.json_response_encoder import JSONResponseEncoder
from lbrynet.extras.daemon.comment_client import jsonrpc_post, sign_comment
from lbrynet.extras.daemon.comment_client import jsonrpc_post, sign_comment, is_comment_signed_by_channel
from lbrynet.extras.daemon.undecorated import undecorated
from lbrynet.wallet.transaction import Transaction, Output, Input
from lbrynet.wallet.account import Account as LBCAccount
@@ -414,17 +414,20 @@ class Daemon(metaclass=JSONRPCServerType):
try:
rpc_site = web.TCPSite(self.rpc_runner, self.conf.api_host, self.conf.api_port, shutdown_timeout=.5)
await rpc_site.start()
log.info('lbrynet API listening on TCP %s:%i', *rpc_site._server.sockets[0].getsockname()[:2])
log.info('RPC server listening on TCP %s:%i', *rpc_site._server.sockets[0].getsockname()[:2])
except OSError as e:
log.error('RPC server failed to bind TCP %s:%i', self.conf.api_host, self.conf.api_port)
await self.analytics_manager.send_server_startup_error(str(e))
raise SystemExit()
try:
streaming_site = web.TCPSite(self.streaming_runner, self.conf.streaming_host, self.conf.streaming_port,
shutdown_timeout=.5)
await streaming_site.start()
log.info('lbrynet media server listening on TCP %s:%i',
*streaming_site._server.sockets[0].getsockname()[:2])
log.info('media server listening on TCP %s:%i', *streaming_site._server.sockets[0].getsockname()[:2])
except OSError as e:
log.error('lbrynet API failed to bind TCP %s for listening. Daemon is already running or this port is '
'already in use by another application.', self.conf.api)
log.error('media server failed to bind TCP %s:%i', self.conf.streaming_host, self.conf.streaming_port)
await self.analytics_manager.send_server_startup_error(str(e))
raise SystemExit()
@@ -436,7 +439,8 @@ class Daemon(metaclass=JSONRPCServerType):
await self.stop()
except Exception as e:
await self.analytics_manager.send_server_startup_error(str(e))
log.exception('Failed to start lbrynet-daemon')
log.exception('Failed to start lbrynet')
raise SystemExit()
await self.analytics_manager.send_server_startup_success()
@@ -1828,7 +1832,7 @@ class Daemon(metaclass=JSONRPCServerType):
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_channel_create(
self, name, bid, allow_duplicate_name=False, account_id=None, claim_address=None,
preview=False, **kwargs):
preview=False, blocking=False, **kwargs):
"""
Create a new channel by generating a channel private key and establishing an '@' prefixed claim.
@@ -1839,7 +1843,8 @@ class Daemon(metaclass=JSONRPCServerType):
[--website_url=<website_url>] [--featured=<featured>...]
[--tags=<tags>...] [--languages=<languages>...] [--locations=<locations>...]
[--thumbnail_url=<thumbnail_url>] [--cover_url=<cover_url>]
[--account_id=<account_id>] [--claim_address=<claim_address>] [--preview]
[--account_id=<account_id>] [--claim_address=<claim_address>]
[--preview] [--blocking]
Options:
--name=<name> : (str) name of the channel prefixed with '@'
@@ -1896,6 +1901,7 @@ class Daemon(metaclass=JSONRPCServerType):
--claim_address=<claim_address>: (str) address where the channel is sent to, if not specified
it will be determined automatically from the account
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until transaction is in mempool
Returns: {Transaction}
"""
@@ -1922,7 +1928,7 @@ class Daemon(metaclass=JSONRPCServerType):
if not preview:
await tx.sign([account])
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
account.add_channel_private_key(txo.private_key)
self.default_wallet.save()
await self.storage.save_claims([self._old_get_temp_claim_info(
@@ -1937,7 +1943,7 @@ class Daemon(metaclass=JSONRPCServerType):
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_channel_update(
self, claim_id, bid=None, account_id=None, claim_address=None,
new_signing_key=False, preview=False, replace=False, **kwargs):
new_signing_key=False, preview=False, blocking=False, replace=False, **kwargs):
"""
Update an existing channel claim.
@@ -1951,7 +1957,7 @@ class Daemon(metaclass=JSONRPCServerType):
[--locations=<locations>...] [--clear_locations]
[--thumbnail_url=<thumbnail_url>] [--cover_url=<cover_url>]
[--account_id=<account_id>] [--claim_address=<claim_address>] [--new_signing_key]
[--preview] [--replace]
[--preview] [--blocking] [--replace]
Options:
--claim_id=<claim_id> : (str) claim_id of the channel to update
@@ -2010,6 +2016,7 @@ class Daemon(metaclass=JSONRPCServerType):
--claim_address=<claim_address>: (str) address where the channel is sent
--new_signing_key : (bool) generate a new signing key, will invalidate all previous publishes
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until transaction is in mempool
--replace : (bool) instead of modifying specific values on
the channel, this will clear all existing values
and only save passed in values, useful for form
@@ -2060,7 +2067,7 @@ class Daemon(metaclass=JSONRPCServerType):
if not preview:
await tx.sign([account])
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
account.add_channel_private_key(new_txo.private_key)
self.default_wallet.save()
await self.storage.save_claims([self._old_get_temp_claim_info(
@@ -2112,10 +2119,8 @@ class Daemon(metaclass=JSONRPCServerType):
)
if not preview:
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
await self.analytics_manager.send_claim_action('abandon')
if blocking:
await account.ledger.wait(tx)
else:
await account.ledger.release_tx(tx)
@@ -2228,7 +2233,8 @@ class Daemon(metaclass=JSONRPCServerType):
[--release_time=<release_time>] [--width=<width>] [--height=<height>] [--duration=<duration>]
[--channel_id=<channel_id> | --channel_name=<channel_name>]
[--channel_account_id=<channel_account_id>...]
[--account_id=<account_id>] [--claim_address=<claim_address>] [--preview]
[--account_id=<account_id>] [--claim_address=<claim_address>]
[--preview] [--blocking]
Options:
--name=<name> : (str) name of the content (can only consist of a-z A-Z 0-9 and -(dash))
@@ -2299,6 +2305,7 @@ class Daemon(metaclass=JSONRPCServerType):
--claim_address=<claim_address>: (str) address where the claim is sent to, if not specified
it will be determined automatically from the account
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until transaction is in mempool
Returns: {Transaction}
"""
@@ -2325,7 +2332,8 @@ class Daemon(metaclass=JSONRPCServerType):
async def jsonrpc_stream_create(
self, name, bid, file_path, allow_duplicate_name=False,
channel_id=None, channel_name=None, channel_account_id=None,
account_id=None, claim_address=None, preview=False, **kwargs):
account_id=None, claim_address=None, preview=False, blocking=False,
**kwargs):
"""
Make a new stream claim and announce the associated file to lbrynet.
@@ -2339,7 +2347,8 @@ class Daemon(metaclass=JSONRPCServerType):
[--release_time=<release_time>] [--width=<width>] [--height=<height>] [--duration=<duration>]
[--channel_id=<channel_id> | --channel_name=<channel_name>]
[--channel_account_id=<channel_account_id>...]
[--account_id=<account_id>] [--claim_address=<claim_address>] [--preview]
[--account_id=<account_id>] [--claim_address=<claim_address>]
[--preview] [--blocking]
Options:
--name=<name> : (str) name of the content (can only consist of a-z A-Z 0-9 and -(dash))
@@ -2412,6 +2421,7 @@ class Daemon(metaclass=JSONRPCServerType):
--claim_address=<claim_address>: (str) address where the claim is sent to, if not specified
it will be determined automatically from the account
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until transaction is in mempool
Returns: {Transaction}
"""
@@ -2448,7 +2458,7 @@ class Daemon(metaclass=JSONRPCServerType):
await tx.sign([account])
if not preview:
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
await self.storage.save_claims([self._old_get_temp_claim_info(
tx, new_txo, claim_address, claim, name, dewies_to_lbc(amount)
)])
@@ -2465,7 +2475,7 @@ class Daemon(metaclass=JSONRPCServerType):
self, claim_id, bid=None, file_path=None,
channel_id=None, channel_name=None, channel_account_id=None, clear_channel=False,
account_id=None, claim_address=None,
preview=False, replace=False, **kwargs):
preview=False, blocking=False, replace=False, **kwargs):
"""
Update an existing stream claim and if a new file is provided announce it to lbrynet.
@@ -2483,7 +2493,7 @@ class Daemon(metaclass=JSONRPCServerType):
[--channel_id=<channel_id> | --channel_name=<channel_name> | --clear_channel]
[--channel_account_id=<channel_account_id>...]
[--account_id=<account_id>] [--claim_address=<claim_address>]
[--preview] [--replace]
[--preview] [--blocking] [--replace]
Options:
--claim_id=<claim_id> : (str) id of the stream claim to update
@@ -2562,6 +2572,7 @@ class Daemon(metaclass=JSONRPCServerType):
--claim_address=<claim_address>: (str) address where the claim is sent to, if not specified
it will be determined automatically from the account
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until transaction is in mempool
--replace : (bool) instead of modifying specific values on
the stream, this will clear all existing values
and only save passed in values, useful for form
@@ -2598,8 +2609,9 @@ class Daemon(metaclass=JSONRPCServerType):
elif old_txo.claim.is_signed and not clear_channel and not replace:
channel = old_txo.channel
if 'fee_address' in kwargs:
self.valid_address_or_error(kwargs['fee_address'])
fee_address = self.get_fee_address(kwargs, claim_address)
if fee_address:
kwargs['fee_address'] = fee_address
if replace:
claim = Claim()
@@ -2639,7 +2651,7 @@ class Daemon(metaclass=JSONRPCServerType):
await tx.sign([account])
if not preview:
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
await self.storage.save_claims([self._old_get_temp_claim_info(
tx, new_txo, claim_address, new_txo.claim, new_txo.claim_name, dewies_to_lbc(amount)
)])
@@ -2654,7 +2666,7 @@ class Daemon(metaclass=JSONRPCServerType):
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_stream_abandon(
self, claim_id=None, txid=None, nout=None, account_id=None,
preview=False, blocking=True):
preview=False, blocking=False):
"""
Abandon one of my stream claims.
@@ -2691,10 +2703,8 @@ class Daemon(metaclass=JSONRPCServerType):
)
if not preview:
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
await self.analytics_manager.send_claim_action('abandon')
if blocking:
await account.ledger.wait(tx)
else:
await account.ledger.release_tx(tx)
@@ -2747,13 +2757,15 @@ class Daemon(metaclass=JSONRPCServerType):
"""
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_support_create(self, claim_id, amount, tip=False, account_id=None, preview=False):
async def jsonrpc_support_create(
self, claim_id, amount, tip=False, account_id=None,
preview=False, blocking=False):
"""
Create a support or a tip for name claim.
Usage:
support_create (<claim_id> | --claim_id=<claim_id>) (<amount> | --amount=<amount>)
[--tip] [--account_id=<account_id>] [--preview]
[--tip] [--account_id=<account_id>] [--preview] [--blocking]
Options:
--claim_id=<claim_id> : (str) claim_id of the claim to support
@@ -2761,6 +2773,7 @@ class Daemon(metaclass=JSONRPCServerType):
--tip : (bool) send support to claim owner, default: false.
--account_id=<account_id> : (str) id of the account to use
--preview : (bool) do not broadcast the transaction
--blocking : (bool) wait until transaction is in mempool
Returns: {Transaction}
"""
@@ -2777,7 +2790,7 @@ class Daemon(metaclass=JSONRPCServerType):
if not preview:
await tx.sign([account])
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
await self.storage.save_supports({claim_id: [{
'txid': tx.id,
'nout': tx.position,
@@ -2817,7 +2830,7 @@ class Daemon(metaclass=JSONRPCServerType):
@requires(WALLET_COMPONENT, conditions=[WALLET_IS_UNLOCKED])
async def jsonrpc_support_abandon(
self, claim_id=None, txid=None, nout=None, keep=None,
account_id=None, preview=False, blocking=True):
account_id=None, preview=False, blocking=False):
"""
Abandon supports, including tips, of a specific claim, optionally
keeping some amount as supports.
@@ -2867,10 +2880,8 @@ class Daemon(metaclass=JSONRPCServerType):
)
if not preview:
await account.ledger.broadcast(tx)
await self.broadcast_or_release(account, tx, blocking)
await self.analytics_manager.send_claim_action('abandon')
if blocking:
await account.ledger.wait(tx)
else:
await account.ledger.release_tx(tx)
@@ -3343,8 +3354,8 @@ class Daemon(metaclass=JSONRPCServerType):
Create and list comments.
"""
async def jsonrpc_comment_list(self, claim_id: str, parent_id: str = None,
page=1, page_size=50, include_replies=True):
async def jsonrpc_comment_list(self, claim_id, parent_id=None, page=1, page_size=50,
include_replies=True, is_channel_signature_valid=False):
"""
List comments associated with a claim.
@@ -3352,17 +3363,17 @@ class Daemon(metaclass=JSONRPCServerType):
comment_list (<claim_id> | --claim_id=<claim_id>)
[(--page=<page> --page_size=<page_size>)]
[--parent_id=<parent_id>] [--include_replies]
[--is_channel_signature_valid]
Options:
--claim_id=<claim_id> : (str) The claim on which the comment will be made on
--include_replies : (bool) Flag to indicate whether or not you want the
replies to be included with the response or not
--parent_id=<parent_id> : (int)
--page=<page> : (int) The page you'd like to see in the comment list.
The first page is 1, second page is 2, and so on.
--page_size=<page_size> : (int) The amount of comments that you'd like to
retrieve in one request
--claim_id=<claim_id> : (str) The claim on which the comment will be made on
--parent_id=<parent_id> : (str) CommentId of a specific thread you'd like to see
--page=<page> : (int) The page you'd like to see in the comment list.
--page_size=<page_size> : (int) The amount of comments that you'd like to retrieve
--include_replies : (bool) Whether or not you want to include replies in list
--is_channel_signature_valid : (bool) Only include comments with valid signatures.
[Warning: Paginated total size will not change, even
if list reduces]
Returns:
(dict) Containing the list, and information about the paginated content:
@@ -3387,7 +3398,7 @@ class Daemon(metaclass=JSONRPCServerType):
]
}
"""
return await jsonrpc_post(
cmnt_list: dict = await jsonrpc_post(
self.conf.comment_server,
"get_claim_comments",
claim_id=claim_id,
@@ -3396,10 +3407,23 @@ class Daemon(metaclass=JSONRPCServerType):
page_size=page_size,
top_level=not include_replies
)
if 'items' in cmnt_list:
items = cmnt_list['items']
chnl_uris = {cmnt['channel_url'] for cmnt in items if 'channel_url' in cmnt}
claims = await self.resolve(tuple(chnl_uris))
for cmnt in cmnt_list['items']:
if 'channel_url' in cmnt:
if cmnt['channel_url'] in claims:
channel = claims[cmnt['channel_url']]
cmnt['is_channel_signature_valid'] = is_comment_signed_by_channel(cmnt, channel)
continue
cmnt['is_channel_signature_valid'] = False
if is_channel_signature_valid:
cmnt_list['items'] = [c for c in cmnt_list['items'] if c['is_channel_signature_valid']]
return cmnt_list
async def jsonrpc_comment_create(
self, claim_id: str, comment: str, parent_id: str = None,
channel_account_id=None, channel_name: str = None, channel_id: str = None) -> dict:
async def jsonrpc_comment_create(self, claim_id, comment, parent_id=None, channel_account_id=None,
channel_name=None, channel_id=None) -> dict:
"""
Create and associate a comment with a claim using your channel identity.
@@ -3411,13 +3435,13 @@ class Daemon(metaclass=JSONRPCServerType):
[--channel_account_id=<channel_account_id>...]
Options:
--comment=<comment> : (str) Comment to be made, should be at most 2000 characters.
--claim_id=<claim_id> : (str) The ID of the claim on which the comment should be made on
--parent_id=<parent_id> : (str) The ID of a comment to make a response to
--channel_id=<channel_id> : (str) The ID of the channel you want to post under
--channel_name=<channel_name>: (str) The channel you want to post as, prepend with a '@'
--channel_account_id=<channel_account_id>: (str) one or more account ids for accounts to look in
for channel certificates, defaults to all accounts.
--comment=<comment> : (str) Comment to be made, should be at most 2000 characters.
--claim_id=<claim_id> : (str) The ID of the claim to comment on
--parent_id=<parent_id> : (str) The ID of a comment to make a response to
--channel_id=<channel_id> : (str) The ID of the channel you want to post under
--channel_name=<channel_name> : (str) The channel you want to post as, prepend with a '@'
--channel_account_id=<channel_account_id> : (str) one or more account ids for accounts to look in
for channel certificates, defaults to all accounts.
Returns:
(dict) Comment object if successfully made, (None) otherwise
@@ -3425,31 +3449,39 @@ class Daemon(metaclass=JSONRPCServerType):
"comment": (str) The actual string as inputted by the user,
"comment_id": (str) The Comment's unique identifier,
"channel_name": (str) Name of the channel this was posted under, prepended with a '@',
"channel_id": (str) The Channel Claim ID that this comeent was posted under,
"channel_id": (str) The Channel Claim ID that this comment was posted under,
"signature": (str) The signature of the comment,
"channel_url": (str) Channel's URI in the ClaimTrie,
"parent_id": (str) Comment this is replying to, (None) if this is the root,
"timestamp": (int) The time at which comment was entered into the server at, in nanoseconds.
}
"""
comment = {
comment_body = {
'comment': comment,
'claim_id': claim_id,
'parent_id': parent_id,
}
channel = await self.get_channel_or_none(channel_account_id, channel_id, channel_name, for_signing=True)
if channel:
comment.update({
comment_body.update({
'channel_id': channel.claim_id,
'channel_name': channel.claim_name,
'signature': sign_comment(
channel_name=channel_name,
channel_id=channel_id,
comment=comment,
salt=time.time_ns()
)
})
return await jsonrpc_post(self.conf.comment_server, 'create_comment', **comment)
sign_comment(comment_body, channel)
response = await jsonrpc_post(self.conf.comment_server, 'create_comment', **comment_body)
if 'signature' in response:
response['is_claim_signature_valid'] = is_comment_signed_by_channel(response, channel)
return response
async def broadcast_or_release(self, account, tx, blocking=False):
try:
await account.ledger.broadcast(tx)
if blocking:
await account.ledger.wait(tx)
except:
await account.ledger.release_tx(tx)
raise
def valid_address_or_error(self, address):
try:
+35 -30
View File
@@ -1,42 +1,47 @@
import logging
import time
import hashlib
import binascii
import aiohttp
import hashlib
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.asymmetric import utils
from cryptography.hazmat.primitives.asymmetric.rsa import generate_private_key
import ecdsa
from torba.client.hash import sha256
from lbrynet.wallet.transaction import Output
log = logging.getLogger(__name__)
def sign_comment(**kwargs):
private_key = generate_private_key(
public_exponent=65537,
key_size=4096,
backend=default_backend()
)
chosen_hash = hashes.SHA256()
hasher = hashes.Hash(chosen_hash, default_backend())
value_to_hash = b':'.join(bytes(v, 'utf-8') for v in kwargs.values() if type(v) is str)
hasher.update(value_to_hash)
digest = hasher.finalize()
signature = private_key.sign(
digest,
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
utils.Prehashed(chosen_hash)
)
m = hashlib.sha3_256()
m.update(signature)
return m.hexdigest()
def get_encoded_signature(signature):
signature = signature.encode() if type(signature) is str else signature
r = int(signature[:int(len(signature) / 2)], 16)
s = int(signature[int(len(signature) / 2):], 16)
return ecdsa.util.sigencode_der(r, s, len(signature) * 4)
def rpc_body(method: str, rpc_id: any, **params) -> dict:
return {'jsonrpc': '2.0', 'id': rpc_id, 'method': method, 'params': {**params}}
def is_comment_signed_by_channel(comment: dict, channel: Output):
try:
pieces = [
comment['signing_ts'].encode(),
channel.claim_hash,
comment['comment'].encode()
]
return Output.is_signature_valid(
get_encoded_signature(comment['signature']),
sha256(b''.join(pieces)),
channel.claim.channel.public_key_bytes
)
except KeyError:
pass
return False
def sign_comment(comment: dict, channel: Output):
timestamp = str(int(time.time())).encode()
pieces = [timestamp, channel.claim_hash, comment['comment'].encode()]
digest = sha256(b''.join(pieces))
signature = channel.private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
comment['signature'] = binascii.hexlify(signature).decode()
comment['signing_ts'] = timestamp.decode()
async def jsonrpc_post(url: str, method: str, **params) -> any:
+17 -8
View File
@@ -204,24 +204,33 @@ class Fee(Metadata):
__slots__ = ()
def update(self, address: str = None, currency: str = None, amount=None):
if address is not None:
self.address = address
if currency is not None and amount is not None:
currency = currency.lower()
assert currency in ('lbc', 'btc', 'usd'), f'Unknown currency type: {currency}'
if amount:
currency = (currency or self.currency or '').lower()
if not currency:
raise Exception('In order to set a fee amount, please specify a fee currency.')
if currency not in ('lbc', 'btc', 'usd'):
raise Exception(f'Missing or unknown currency provided: {currency}')
setattr(self, currency, Decimal(amount))
elif currency:
raise Exception('In order to set a fee currency, please specify a fee amount.')
if address:
if not self.currency:
raise Exception('In order to set a fee address, please specify a fee amount and currency.')
self.address = address
@property
def currency(self) -> str:
return FeeMessage.Currency.Name(self.message.currency)
if self.message.currency:
return FeeMessage.Currency.Name(self.message.currency)
@property
def address(self) -> str:
return Base58.encode(self.message.address)
if self.address_bytes:
return Base58.encode(self.address_bytes)
@address.setter
def address(self, address: str):
self.message.address = Base58.decode(address)
self.address_bytes = Base58.decode(address)
@property
def address_bytes(self) -> bytes:
+15 -3
View File
@@ -10,13 +10,15 @@ if typing.TYPE_CHECKING:
REFLECTOR_V1 = 0
REFLECTOR_V2 = 1
MAX_RESPONSE_SIZE = 2000000
log = logging.getLogger(__name__)
class StreamReflectorClient(asyncio.Protocol):
def __init__(self, blob_manager: 'BlobManager', descriptor: 'StreamDescriptor'):
self.loop = asyncio.get_event_loop()
self.transport: asyncio.StreamWriter = None
self.transport: typing.Optional[asyncio.WriteTransport] = None
self.blob_manager = blob_manager
self.descriptor = descriptor
self.response_buff = b''
@@ -37,11 +39,21 @@ class StreamReflectorClient(asyncio.Protocol):
log.info("Finished sending reflector %i blobs", len(self.reflected_blobs))
def data_received(self, data):
if len(self.response_buff + (data or b'')) > MAX_RESPONSE_SIZE:
log.warning("response message to large from reflector server: %i bytes",
len(self.response_buff + (data or b'')))
self.response_buff = b''
self.transport.close()
return
self.response_buff += (data or b'')
try:
response = json.loads(data.decode())
response = json.loads(self.response_buff.decode())
self.response_buff = b''
self.response_queue.put_nowait(response)
except ValueError:
self.transport.close()
if not data:
log.warning("got undecodable response from reflector server")
self.response_buff = b''
return
async def send_request(self, request_dict: typing.Dict):
+15 -5
View File
@@ -15,7 +15,7 @@ log = logging.getLogger(__name__)
class ReflectorServerProtocol(asyncio.Protocol):
def __init__(self, blob_manager: 'BlobManager'):
def __init__(self, blob_manager: 'BlobManager', response_chunk_size: int = 10000):
self.loop = asyncio.get_event_loop()
self.blob_manager = blob_manager
self.server_task: asyncio.Task = None
@@ -28,6 +28,7 @@ class ReflectorServerProtocol(asyncio.Protocol):
self.sd_blob: typing.Optional['BlobFile'] = None
self.received = []
self.incoming = asyncio.Event(loop=self.loop)
self.chunk_size = response_chunk_size
def connection_made(self, transport):
self.transport = transport
@@ -47,7 +48,15 @@ class ReflectorServerProtocol(asyncio.Protocol):
self.loop.create_task(self.handle_request(request))
def send_response(self, response: typing.Dict):
self.transport.write(json.dumps(response).encode())
def chunk_response(remaining: bytes):
f = self.loop.create_future()
f.add_done_callback(lambda _: self.transport.write(remaining[:self.chunk_size]))
if len(remaining) > self.chunk_size:
f.add_done_callback(lambda _: self.loop.call_soon(chunk_response, remaining[self.chunk_size:]))
self.loop.call_soon(f.set_result, None)
response_bytes = json.dumps(response).encode()
chunk_response(response_bytes)
async def handle_request(self, request: typing.Dict):
if self.client_version is None:
@@ -121,11 +130,12 @@ class ReflectorServerProtocol(asyncio.Protocol):
class ReflectorServer:
def __init__(self, blob_manager: 'BlobManager'):
def __init__(self, blob_manager: 'BlobManager', response_chunk_size: int = 10000):
self.loop = asyncio.get_event_loop()
self.blob_manager = blob_manager
self.server_task: asyncio.Task = None
self.server_task: typing.Optional[asyncio.Task] = None
self.started_listening = asyncio.Event(loop=self.loop)
self.response_chunk_size = response_chunk_size
def start_server(self, port: int, interface: typing.Optional[str] = '0.0.0.0'):
if self.server_task is not None:
@@ -133,7 +143,7 @@ class ReflectorServer:
async def _start_server():
server = await self.loop.create_server(
lambda: ReflectorServerProtocol(self.blob_manager),
lambda: ReflectorServerProtocol(self.blob_manager, self.response_chunk_size),
interface, port
)
self.started_listening.set()
+1 -1
View File
@@ -407,7 +407,7 @@ class StreamManager:
msg = f"fee of {fee_amount} exceeds max available balance"
log.warning(msg)
raise InsufficientFundsError(msg)
fee_address = claim.stream.fee.address
fee_address = claim.stream.fee.address or resolved['address']
stream = ManagedStream(
self.loop, self.config, self.blob_manager, claim.stream.source.sd_hash, download_directory,
+1 -1
View File
@@ -310,7 +310,7 @@ class SQLDB:
claims = self._upsertable_claims(txos, header)
if claims:
self.db.executemany("""
INSERT INTO claim (
INSERT OR IGNORE INTO claim (
claim_hash, claim_id, claim_name, normalized, txo_hash, tx_position, amount,
claim_type, media_type, stream_type, timestamp, creation_timestamp, height,
creation_height, release_time, activation_height, expiration_height, short_url)
+1 -1
View File
@@ -23,7 +23,7 @@ setup(
'console_scripts': 'lbrynet=lbrynet.extras.cli:main'
},
install_requires=[
'torba==0.5.5',
'torba==0.5.6',
'aiohttp==3.5.4',
'aioupnp==0.0.12',
'appdirs==1.4.3',
+80 -6
View File
@@ -705,12 +705,6 @@ class StreamCommands(ClaimTestCase):
fixed_values['locations'] = [{'country': 'BR'}]
self.assertEqual(txo['value'], fixed_values)
# clearing fee
tx = await self.out(self.stream_update(claim_id, clear_fee=True))
txo = tx['outputs'][0]
del fixed_values['fee']
self.assertEqual(txo['value'], fixed_values)
# modifying hash/size/name
fixed_values['source']['name'] = 'changed_name'
fixed_values['source']['hash'] = 'cafebeef'
@@ -743,6 +737,86 @@ class StreamCommands(ClaimTestCase):
self.assertEqual(len(await self.daemon.jsonrpc_claim_list()), 3)
self.assertEqual(len(await self.daemon.jsonrpc_claim_list(account_id=account2_id)), 1)
async def test_setting_fee_fields(self):
tx = await self.out(self.stream_create('paid-stream'))
txo = tx['outputs'][0]
claim_id, stream = txo['claim_id'], txo['value']
fee_address = 'mmCsWAiXMUVecFQ3fVzUwvpT9XFMXno2Ca'
self.assertNotIn('fee', stream)
# --replace=false
# validation
with self.assertRaisesRegex(Exception, 'please specify a fee currency'):
await self.stream_update(claim_id, fee_amount='0.1')
with self.assertRaisesRegex(Exception, 'unknown currency provided: foo'):
await self.stream_update(claim_id, fee_amount='0.1', fee_currency="foo")
with self.assertRaisesRegex(Exception, 'please specify a fee amount'):
await self.stream_update(claim_id, fee_currency='usd')
with self.assertRaisesRegex(Exception, 'please specify a fee amount'):
await self.stream_update(claim_id, fee_address=fee_address)
# set just amount and currency with default address
tx = await self.stream_update(
claim_id, fee_amount='0.99', fee_currency='lbc'
)
self.assertEqual(
tx['outputs'][0]['value']['fee'],
{'amount': '0.99', 'currency': 'LBC', 'address': txo['address']}
)
# set all fee fields
tx = await self.stream_update(
claim_id, fee_amount='0.1', fee_currency='usd', fee_address=fee_address
)
self.assertEqual(
tx['outputs'][0]['value']['fee'],
{'amount': '0.1', 'currency': 'USD', 'address': fee_address}
)
# change just address
tx = await self.stream_update(claim_id, fee_address=txo['address'])
self.assertEqual(
tx['outputs'][0]['value']['fee'],
{'amount': '0.1', 'currency': 'USD', 'address': txo['address']}
)
# change just amount (does not reset fee_address)
tx = await self.stream_update(claim_id, fee_amount='0.2')
self.assertEqual(
tx['outputs'][0]['value']['fee'],
{'amount': '0.2', 'currency': 'USD', 'address': txo['address']}
)
# changing currency without an amount is never allowed, even if previous amount exists
with self.assertRaises(Exception, msg='In order to set a fee currency, please specify a fee amount'):
await self.stream_update(claim_id, fee_currency='usd')
# clearing fee
tx = await self.out(self.stream_update(claim_id, clear_fee=True))
self.assertNotIn('fee', tx['outputs'][0]['value'])
# --replace=true
# set just amount and currency with default address
tx = await self.stream_update(
claim_id, fee_amount='0.99', fee_currency='lbc', replace=True
)
self.assertEqual(
tx['outputs'][0]['value']['fee'],
{'amount': '0.99', 'currency': 'LBC', 'address': txo['address']}
)
# set all fee fields
tx = await self.stream_update(
claim_id, fee_amount='0.1', fee_currency='usd', fee_address=fee_address, replace=True
)
self.assertEqual(
tx['outputs'][0]['value']['fee'],
{'amount': '0.1', 'currency': 'USD', 'address': fee_address}
)
# validation
with self.assertRaisesRegex(Exception, 'please specify a fee currency'):
await self.stream_update(claim_id, fee_amount='0.1', replace=True)
with self.assertRaisesRegex(Exception, 'unknown currency provided: foo'):
await self.stream_update(claim_id, fee_amount='0.1', fee_currency="foo", replace=True)
with self.assertRaisesRegex(Exception, 'please specify a fee amount'):
await self.stream_update(claim_id, fee_currency='usd', replace=True)
with self.assertRaisesRegex(Exception, 'please specify a fee amount'):
await self.stream_update(claim_id, fee_address=fee_address, replace=True)
async def test_automatic_type_and_metadata_detection_for_image(self):
txo = (await self.image_stream_create())['outputs'][0]
self.assertEqual(
+80 -10
View File
@@ -1,10 +1,11 @@
import asyncio
from math import ceil
from aiohttp import web
from lbrynet.testcase import CommandTestCase
class FakedCommentServer:
class MockedCommentServer:
ERRORS = {
'INVALID_PARAMS': {'code': -32602, 'message': 'Invalid parameters'},
@@ -25,11 +26,19 @@ class FakedCommentServer:
def create_comment(self, **comment):
self.comment_id += 1
comment['comment_id'] = self.comment_id
if 'channel_id' in comment:
comment['channel_url'] = 'lbry://' + comment['channel_name'] + '#' + comment['channel_id']
self.comments.append(comment)
return comment
def get_claim_comments(self, page=1, page_size=50, **kwargs):
return self.comments[:page_size]
return {
'page': page,
'page_size': page_size,
'total_pages': ceil(len(self.comments)/page_size),
'total_items': len(self.comments),
'items': (self.comments[::-1])[(page - 1) * page_size: page * page_size]
}
methods = {
'get_claim_comments': get_claim_comments,
@@ -73,21 +82,21 @@ class CommentCommands(CommandTestCase):
async def asyncSetUp(self):
await super().asyncSetUp()
self.daemon.conf.comment_server = 'http://localhost:2903/api'
self.comment_server = FakedCommentServer(2903)
self.comment_server = MockedCommentServer(2903)
await self.comment_server.start()
self.addCleanup(self.comment_server.stop)
async def test_comment_create(self):
async def test01_comment_create(self):
channel = (await self.channel_create('@JimmyBuffett'))['outputs'][0]
stream = (await self.stream_create())['outputs'][0]
self.assertEqual(0, len(await self.daemon.jsonrpc_comment_list(stream['claim_id'])))
self.assertEqual(0, len((await self.daemon.jsonrpc_comment_list(stream['claim_id']))['items']))
comment = await self.daemon.jsonrpc_comment_create(
claim_id=stream['claim_id'],
channel_id=channel['claim_id'],
comment="It's 5 O'Clock Somewhere"
)
comments = await self.daemon.jsonrpc_comment_list(stream['claim_id'])
comments = (await self.daemon.jsonrpc_comment_list(stream['claim_id']))['items']
self.assertEqual(1, len(comments))
self.assertEqual(comment['comment_id'], comments[0]['comment_id'])
self.assertEqual(stream['claim_id'], comments[0]['claim_id'])
@@ -99,7 +108,68 @@ class CommentCommands(CommandTestCase):
comment='Let\'s all go to Margaritaville',
parent_id=comments[0]['comment_id']
)
comments = await self.daemon.jsonrpc_comment_list(stream['claim_id'])
comments = (await self.daemon.jsonrpc_comment_list(stream['claim_id']))['items']
self.assertEqual(2, len(comments))
self.assertEqual(comments[1]['channel_id'], channel2['claim_id'])
self.assertEqual(comments[0]['comment_id'], comments[1]['parent_id'])
self.assertEqual(comments[0]['channel_id'], channel2['claim_id'])
self.assertEqual(comments[0]['parent_id'], comments[1]['comment_id'])
comment = await self.daemon.jsonrpc_comment_create(
claim_id=stream['claim_id'],
comment='Anonymous comment'
)
comments = (await self.daemon.jsonrpc_comment_list(stream['claim_id']))['items']
self.assertEqual(comment['comment_id'], comments[0]['comment_id'])
async def test02_unsigned_comment_list(self):
stream = (await self.stream_create())['outputs'][0]
comments = []
for i in range(28):
comment = await self.daemon.jsonrpc_comment_create(
comment=f'{i}',
claim_id=stream['claim_id'],
)
self.assertIn('comment_id', comment)
comments.append(comment)
comment_list = await self.daemon.jsonrpc_comment_list(
claim_id=stream['claim_id']
)
self.assertIs(comment_list['page_size'], 50)
self.assertIs(comment_list['page'], 1)
self.assertIs(comment_list['total_items'], 28)
for comment in comment_list['items']:
self.assertEqual(comment['comment'], comments.pop()['comment'])
signed_comment_list = await self.daemon.jsonrpc_comment_list(
claim_id=stream['claim_id'],
is_channel_signature_valid=True
)
self.assertIs(len(signed_comment_list['items']), 0)
async def test03_signed_comments_list(self):
channel = (await self.channel_create('@JimmyBuffett'))['outputs'][0]
stream = (await self.stream_create())['outputs'][0]
comments = []
for i in range(28):
comment = await self.daemon.jsonrpc_comment_create(
comment=f'{i}',
claim_id=stream['claim_id'],
channel_id=channel['claim_id'],
)
self.assertIn('comment_id', comment)
comments.append(comment)
comment_list = await self.daemon.jsonrpc_comment_list(
claim_id=stream['claim_id']
)
self.assertIs(comment_list['page_size'], 50)
self.assertIs(comment_list['page'], 1)
self.assertIs(comment_list['total_items'], 28)
for comment in comment_list['items']:
self.assertEqual(comment['comment'], comments.pop()['comment'])
signed_comment_list = await self.daemon.jsonrpc_comment_list(
claim_id=stream['claim_id'],
is_channel_signature_valid=True
)
self.assertIs(len(signed_comment_list['items']), 28)
+29 -2
View File
@@ -1,9 +1,8 @@
import asyncio
import logging
import os
from binascii import unhexlify, hexlify
from binascii import hexlify
from lbrynet.schema import Claim
from lbrynet.testcase import CommandTestCase
from lbrynet.blob_exchange.downloader import BlobDownloader
@@ -259,3 +258,31 @@ class FileCommands(CommandTestCase):
await self.daemon.stream_manager.start()
self.assertEqual(len(self.daemon.jsonrpc_file_list()), 1)
self.assertEqual(self.daemon.jsonrpc_file_list()[0].content_fee.raw, raw_content_fee)
await self.daemon.jsonrpc_file_delete(claim_name='icanpay')
# PASS: no fee address --> use the claim address to pay
tx = await self.stream_create(
'nofeeaddress', '0.01', data=b'free stuff?',
)
await self.__raw_value_update_no_fee_address(
tx, fee_amount='2.0', fee_currency='LBC', claim_address=target_address
)
await self.daemon.jsonrpc_file_delete(claim_name='nofeeaddress')
self.assertEqual(len(self.daemon.jsonrpc_file_list()), 0)
response = await self.out(self.daemon.jsonrpc_get('lbry://nofeeaddress'))
self.assertIsNone(self.daemon.jsonrpc_file_list()[0].stream_claim_info.claim.stream.fee.address)
self.assertIsNotNone(response['content_fee'])
self.assertEqual(len(self.daemon.jsonrpc_file_list()), 1)
self.assertEqual(response['content_fee']['outputs'][0]['amount'], '2.0')
self.assertEqual(response['content_fee']['outputs'][0]['address'], target_address)
async def __raw_value_update_no_fee_address(self, tx, claim_address, **kwargs):
tx = await self.daemon.jsonrpc_stream_update(
tx['outputs'][0]['claim_id'], preview=True, claim_address=claim_address, **kwargs
)
tx.outputs[0].claim.stream.fee.address_bytes = b''
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast(tx)
await self.confirm_tx(tx.id)
+31 -1
View File
@@ -9,7 +9,7 @@ from lbrynet.schema.compat import OldClaimMessage
from torba.client.hash import sha256, Base58
class ResolveCommand(CommandTestCase):
class BaseResolveTestCase(CommandTestCase):
def get_claim_id(self, tx):
return tx['outputs'][0]['claim_id']
@@ -21,6 +21,9 @@ class ResolveCommand(CommandTestCase):
else:
self.assertEqual(claim_id, other['claim_id'])
class ResolveCommand(BaseResolveTestCase):
async def test_resolve_response(self):
channel_id = self.get_claim_id(
await self.channel_create('@abc', '0.01')
@@ -306,6 +309,33 @@ class ResolveCommand(CommandTestCase):
self.assertFalse(called_again.is_set())
class ResolveAfterReorg(BaseResolveTestCase):
async def reorg(self, start):
blocks = self.ledger.headers.height - start
self.blockchain._block_expected = start - 1
# go back to start
await self.blockchain.invalidate_block(self.ledger.headers.hash(start).decode())
# go to previous + 1
await self.generate(blocks + 2)
async def test_reorg(self):
self.assertEqual(self.ledger.headers.height, 206)
name = '@abc'
channel_id = self.get_claim_id(
await self.channel_create(name, '0.01')
)
self.assertNotIn('error', (await self.resolve(name))[name])
await self.reorg(206)
self.assertNotIn('error', (await self.resolve(name))[name])
await self.channel_abandon(channel_id)
self.assertIn('error', (await self.resolve(name))[name])
await self.reorg(206)
self.assertIn('error', (await self.resolve(name))[name])
def generate_signed_legacy(address: bytes, output: Output):
decoded_address = Base58.decode(address)
claim = OldClaimMessage()
View File
Whitespace-only changes.
@@ -0,0 +1,33 @@
from torba.testcase import AsyncioTestCase
from lbrynet.extras.daemon.comment_client import sign_comment
from lbrynet.extras.daemon.comment_client import is_comment_signed_by_channel
from tests.unit.wallet.test_schema_signing import get_stream, get_channel
class TestSigningComments(AsyncioTestCase):
@staticmethod
def create_claim_comment_body(comment, claim, channel):
return {
'claim_id': claim.claim_id,
'channel_name': channel.claim_name,
'channel_id': channel.claim_id,
'comment': comment
}
def test01_successful_create_sign_and_validate_comment(self):
channel = get_channel('@BusterBluth')
stream = get_stream('pop secret')
comment = self.create_claim_comment_body('Cool stream', stream, channel)
sign_comment(comment, channel)
self.assertTrue(is_comment_signed_by_channel(comment, channel))
def test02_fail_to_validate_spoofed_channel(self):
pdiddy = get_channel('@PDitty')
channel2 = get_channel('@TomHaverford')
stream = get_stream()
comment = self.create_claim_comment_body('Woahh This is Sick!! Shout out 2 my boy Tommy H', stream, pdiddy)
sign_comment(comment, channel2)
self.assertFalse(is_comment_signed_by_channel(comment, pdiddy))
+4 -1
View File
@@ -83,12 +83,15 @@ class TestProtocol(AsyncioTestCase):
find_value_response = peer1.node_rpc.find_value(peer3, b'2' * 48)
self.assertEqual(len(find_value_response[b'contacts']), 0)
self.assertSetEqual(
{b'2' * 48, b'token', b'protocolVersion', b'contacts'}, set(find_value_response.keys())
{b'2' * 48, b'token', b'protocolVersion', b'contacts', b'p'}, set(find_value_response.keys())
)
self.assertEqual(2, len(find_value_response[b'2' * 48]))
self.assertEqual(find_value_response[b'2' * 48][0], peer2_from_peer1.compact_address_tcp())
self.assertDictEqual(bdecode(bencode(find_value_response)), find_value_response)
find_value_page_above_pages_response = peer1.node_rpc.find_value(peer3, b'2' * 48, page=10)
self.assertNotIn(b'2' * 48, find_value_page_above_pages_response)
peer1.stop()
peer2.stop()
peer1.disconnect()
+22 -2
View File
@@ -59,17 +59,28 @@ class TestDatagram(unittest.TestCase):
self.assertRaises(ValueError, RequestDatagram.make_find_value, b'1' * 49, b'2' * 48, b'1' * 20)
self.assertRaises(ValueError, RequestDatagram.make_find_value, b'1' * 48, b'2' * 49, b'1' * 20)
self.assertRaises(ValueError, RequestDatagram.make_find_value, b'1' * 48, b'2' * 48, b'1' * 21)
self.assertRaises(ValueError, RequestDatagram.make_find_value, b'1' * 48, b'2' * 48, b'1' * 20, -1)
self.assertEqual(20, len(RequestDatagram.make_find_value(b'1' * 48, b'2' * 48).rpc_id))
# default page argument
serialized = RequestDatagram.make_find_value(b'1' * 48, b'2' * 48, b'1' * 20).bencode()
decoded = decode_datagram(serialized)
self.assertEqual(decoded.packet_type, REQUEST_TYPE)
self.assertEqual(decoded.rpc_id, b'1' * 20)
self.assertEqual(decoded.node_id, b'1' * 48)
self.assertEqual(decoded.method, b'findValue')
self.assertListEqual(decoded.args, [b'2' * 48, {b'protocolVersion': 1}])
self.assertListEqual(decoded.args, [b'2' * 48, {b'protocolVersion': 1, b'p': 0}])
def test_find_value_response(self):
# nondefault page argument
serialized = RequestDatagram.make_find_value(b'1' * 48, b'2' * 48, b'1' * 20, 1).bencode()
decoded = decode_datagram(serialized)
self.assertEqual(decoded.packet_type, REQUEST_TYPE)
self.assertEqual(decoded.rpc_id, b'1' * 20)
self.assertEqual(decoded.node_id, b'1' * 48)
self.assertEqual(decoded.method, b'findValue')
self.assertListEqual(decoded.args, [b'2' * 48, {b'protocolVersion': 1, b'p': 1}])
def test_find_value_response_without_pages_field(self):
found_value_response = {b'2' * 48: [b'\x7f\x00\x00\x01']}
serialized = ResponseDatagram(RESPONSE_TYPE, b'1' * 20, b'1' * 48, found_value_response).bencode()
decoded = decode_datagram(serialized)
@@ -78,6 +89,15 @@ class TestDatagram(unittest.TestCase):
self.assertEqual(decoded.node_id, b'1' * 48)
self.assertDictEqual(decoded.response, found_value_response)
def test_find_value_response_with_pages_field(self):
found_value_response = {b'2' * 48: [b'\x7f\x00\x00\x01'], b'p': 1}
serialized = ResponseDatagram(RESPONSE_TYPE, b'1' * 20, b'1' * 48, found_value_response).bencode()
decoded = decode_datagram(serialized)
self.assertEqual(decoded.packet_type, RESPONSE_TYPE)
self.assertEqual(decoded.rpc_id, b'1' * 20)
self.assertEqual(decoded.node_id, b'1' * 48)
self.assertDictEqual(decoded.response, found_value_response)
def test_store_request(self):
self.assertRaises(ValueError, RequestDatagram.make_store, b'1' * 47, b'2' * 48, b'3' * 48, 3333, b'1' * 20)
self.assertRaises(ValueError, RequestDatagram.make_store, b'1' * 48, b'2' * 49, b'3' * 48, 3333, b'1' * 20)
+55
View File
@@ -1,6 +1,7 @@
import contextlib
import typing
import binascii
import socket
import asyncio
from torba.testcase import AsyncioTestCase
from tests import dht_mocks
@@ -26,6 +27,7 @@ class TestBlobAnnouncer(AsyncioTestCase):
for node_id, address in peer_addresses:
await self.add_peer(node_id, address)
self.node.joined.set()
self.node._refresh_task = self.loop.create_task(self.node.refresh_node())
async def add_peer(self, node_id, address, add_to_routing_table=True):
n = Node(self.loop, PeerManager(self.loop), node_id, 4444, 4444, 3333, address)
@@ -113,3 +115,56 @@ class TestBlobAnnouncer(AsyncioTestCase):
self.assertEqual(self.node.protocol.node_id, found_peers[0].node_id)
self.assertEqual(self.node.protocol.external_ip, found_peers[0].address)
self.assertEqual(self.node.protocol.peer_port, found_peers[0].tcp_port)
async def test_popular_blob(self):
peer_count = 150
addresses = [
(constants.generate_id(i + 1), socket.inet_ntoa(int(i + 1).to_bytes(length=4, byteorder='big')))
for i in range(peer_count)
]
blob_hash = b'1' * 48
async with self._test_network_context(peer_addresses=addresses):
total_seen = set()
announced_to = self.nodes[0]
for i in range(1, peer_count):
node = self.nodes[i]
kad_peer = announced_to.protocol.peer_manager.get_kademlia_peer(
node.protocol.node_id, node.protocol.external_ip, node.protocol.udp_port
)
await announced_to.protocol._add_peer(kad_peer)
peer = node.protocol.get_rpc_peer(
node.protocol.peer_manager.get_kademlia_peer(
announced_to.protocol.node_id,
announced_to.protocol.external_ip,
announced_to.protocol.udp_port
)
)
response = await peer.store(blob_hash)
self.assertEqual(response, b'OK')
peers_for_blob = await peer.find_value(blob_hash, 0)
if i == 1:
self.assertTrue(blob_hash not in peers_for_blob)
self.assertEqual(peers_for_blob[b'p'], 0)
else:
self.assertEqual(len(peers_for_blob[blob_hash]), min(i - 1, constants.k))
self.assertEqual(len(announced_to.protocol.data_store.get_peers_for_blob(blob_hash)), i)
if i - 1 > constants.k:
self.assertEqual(len(peers_for_blob[b'contacts']), constants.k)
self.assertEqual(peers_for_blob[b'p'], ((i - 1) // (constants.k + 1)) + 1)
seen = set(peers_for_blob[blob_hash])
self.assertEqual(len(seen), constants.k)
self.assertEqual(len(peers_for_blob[blob_hash]), len(seen))
for pg in range(1, peers_for_blob[b'p']):
page_x = await peer.find_value(blob_hash, pg)
self.assertNotIn(b'contacts', page_x)
page_x_set = set(page_x[blob_hash])
self.assertEqual(len(page_x[blob_hash]), len(page_x_set))
self.assertTrue(len(page_x_set) > 0)
self.assertSetEqual(seen.intersection(page_x_set), set())
seen.intersection_update(page_x_set)
total_seen.update(page_x_set)
else:
self.assertEqual(len(peers_for_blob[b'contacts']), i - 1)
self.assertEqual(len(total_seen), peer_count - 2)
+8 -2
View File
@@ -41,8 +41,8 @@ class TestStreamAssembler(AsyncioTestCase):
self.stream = await self.stream_manager.create_stream(file_path)
async def test_reflect_stream(self):
reflector = ReflectorServer(self.server_blob_manager)
async def _test_reflect_stream(self, response_chunk_size):
reflector = ReflectorServer(self.server_blob_manager, response_chunk_size=response_chunk_size)
reflector.start_server(5566, '127.0.0.1')
await reflector.started_listening.wait()
self.addCleanup(reflector.stop_server)
@@ -63,6 +63,12 @@ class TestStreamAssembler(AsyncioTestCase):
sent = await self.stream.upload_to_reflector('127.0.0.1', 5566)
self.assertListEqual(sent, [])
async def test_reflect_stream(self):
return await asyncio.wait_for(self._test_reflect_stream(response_chunk_size=50), 3, loop=self.loop)
async def test_reflect_stream_small_response_chunks(self):
return await asyncio.wait_for(self._test_reflect_stream(response_chunk_size=30), 3, loop=self.loop)
async def test_announces(self):
to_announce = await self.storage.get_blobs_to_announce()
self.assertIn(self.stream.sd_hash, to_announce, "sd blob not set to announce")
+23 -23
View File
@@ -25,47 +25,47 @@ def get_tx():
return Transaction().add_inputs([get_input()])
def get_channel(claim_name='@foo'):
channel_txo = Output.pay_claim_name_pubkey_hash(CENT, claim_name, Claim(), b'abc')
channel_txo.generate_channel_private_key()
get_tx().add_outputs([channel_txo])
return channel_txo
def get_stream(claim_name='foo'):
stream_txo = Output.pay_claim_name_pubkey_hash(CENT, claim_name, Claim(), b'abc')
get_tx().add_outputs([stream_txo])
return stream_txo
class TestSigningAndValidatingClaim(AsyncioTestCase):
@staticmethod
def get_channel():
channel_txo = Output.pay_claim_name_pubkey_hash(CENT, '@foo', Claim(), b'abc')
channel_txo.generate_channel_private_key()
get_tx().add_outputs([channel_txo])
return channel_txo
@staticmethod
def get_stream():
stream_txo = Output.pay_claim_name_pubkey_hash(CENT, 'foo', Claim(), b'abc')
get_tx().add_outputs([stream_txo])
return stream_txo
def test_successful_create_sign_and_validate(self):
channel = self.get_channel()
stream = self.get_stream()
channel = get_channel()
stream = get_stream()
stream.sign(channel)
self.assertTrue(stream.is_signed_by(channel))
def test_fail_to_validate_on_wrong_channel(self):
stream = self.get_stream()
stream.sign(self.get_channel())
self.assertFalse(stream.is_signed_by(self.get_channel()))
stream = get_stream()
stream.sign(get_channel())
self.assertFalse(stream.is_signed_by(get_channel()))
def test_fail_to_validate_altered_claim(self):
channel = self.get_channel()
stream = self.get_stream()
channel = get_channel()
stream = get_stream()
stream.sign(channel)
self.assertTrue(stream.is_signed_by(channel))
stream.claim.stream.title = 'hello'
self.assertFalse(stream.is_signed_by(channel))
def test_valid_private_key_for_cert(self):
channel = self.get_channel()
channel = get_channel()
self.assertTrue(channel.is_channel_private_key(channel.private_key))
def test_fail_to_load_wrong_private_key_for_cert(self):
channel = self.get_channel()
self.assertFalse(channel.is_channel_private_key(self.get_channel().private_key))
channel = get_channel()
self.assertFalse(channel.is_channel_private_key(get_channel().private_key))
class TestValidatingOldSignatures(AsyncioTestCase):