Compare commits

...
24 Commits
Author SHA1 Message Date
Jack Robison 077281a77a v0.42.2 2019-10-07 13:03:51 -04:00
Jack Robison 7055d250d0 Merge pull request #2503 from lbryio/reuse-loggly
Reuse loggly connection
2019-10-07 12:30:04 -04:00
Jack Robison 8c0196301d Merge pull request #2489 from lbryio/dht_object_cache
DHT object cache revamp
2019-10-07 12:27:07 -04:00
Jack Robison f99aacd9dc improve store_to_peer retry, reduce error verbosity 2019-10-07 12:26:49 -04:00
Jack Robison 43b83bec83 remove dead code 2019-10-07 12:26:49 -04:00
Jack Robison fc1b4658fc re-use loggly connection 2019-10-07 12:26:49 -04:00
Victor Shyba 05a28ea346 handle attempts to add peers with no node id and issue a warning 2019-10-07 13:20:38 -03:00
Victor Shyba 624f986978 handle peer_is_good for unknown node id 2019-10-07 13:20:38 -03:00
Victor Shyba 91a9957918 avoid peer mutability
Mutability can be harmful here, specially on fields used for __hash__
It can also make the cache less efficient
2019-10-07 13:20:38 -03:00
Victor Shyba 8cd18e47e7 get_kad... -> make_kad.. 2019-10-07 13:20:38 -03:00
Victor Shyba 517868f73c fix unit tests 2019-10-07 13:20:38 -03:00
Victor Shyba 18d80f4ed5 1 and 65536 arent valid ports 2019-10-07 13:20:38 -03:00
Victor Shyba 2e2c751d75 check for none on tcp port 2019-10-07 13:20:38 -03:00
Victor Shyba 7aa5e2cbe1 use a dataclass experiment 2019-10-07 13:20:38 -03:00
Victor Shyba 7d6db20081 use cache for dht peer objects 2019-10-07 13:20:38 -03:00
Miroslav Kovar 911ec9444b Decrease the status refresh rate 2019-10-07 11:42:40 -04:00
Miroslav Kovar 2883c6bab3 Fix incorrect counting of total bytes sent / received 2019-10-07 11:42:40 -04:00
OrginalS f5fed54195 Punctuation
Added missing punctuation and changed existing to more fitting one.
2019-10-07 10:58:13 -04:00
Rishav Sharma 91b1446375 Grammar correction 2019-10-07 10:50:14 -04:00
Lex Berezhny 56df145052 updating/abanding streams/channels works across all accounts 2019-10-07 09:57:22 -04:00
Lex Berezhny 9e873f77f9 bug fixes related to cross-account / wallet operations 2019-10-07 09:57:22 -04:00
Alex Grintsvayg 27bbd91635 include fn name and args in jsonrpc error message 2019-10-04 11:25:01 -04:00
Alex Grintsvayg 2c73c80c9f fix formatting 2019-10-04 09:52:29 -04:00
Hugo 3a3a9b5f4e Fix spelling 2019-10-03 21:40:06 -04:00
41 changed files with 284 additions and 216 deletions

No files matched your search

+4 -4
View File
@@ -1,6 +1,6 @@
# Installing LBRY
If only the JSON-RPC API server is needed, the recommended way to install LBRY is to use a pre-built binary. We provide binaries for all major operating systems. See the [README](README.md).
If only the JSON-RPC API server is needed, the recommended way to install LBRY is to use a pre-built binary. We provide binaries for all major operating systems. See the [README](README.md)!
These instructions are for installing LBRY from source, which is recommended if you are interested in doing development work or LBRY is not available on your operating system (godspeed, TempleOS users).
@@ -9,7 +9,7 @@ Here's a video walkthrough of this setup, which is itself hosted by the LBRY net
## Prerequisites
Running `lbrynet` from source requires Python 3.7 or higher. Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/)
Running `lbrynet` from source requires Python 3.7 or higher. Get the installer for your OS [here](https://www.python.org/downloads/release/python-370/).
After installing python 3, you'll need to install some additional libraries depending on your operating system.
@@ -17,7 +17,7 @@ After installing python 3, you'll need to install some additional libraries depe
macOS users will need to install [xcode command line tools](https://developer.xamarin.com/guides/testcloud/calabash/configuring/osx/install-xcode-command-line-tools/) and [homebrew](http://brew.sh/).
These environment variables also need to be set
These environment variables also need to be set:
1. PYTHONUNBUFFERED=1
2. EVENT_NOKQUEUE=1
@@ -27,7 +27,7 @@ Remaining dependencies can then be installed by running:
brew install python protobuf
```
Assistance installing Python3: https://docs.python-guide.org/starting/install3/osx/
Assistance installing Python3: https://docs.python-guide.org/starting/install3/osx/.
### Linux
+1 -1
View File
@@ -2,7 +2,7 @@
LBRY is a decentralized peer-to-peer protocol for publishing and accessing digital content. It utilizes the [LBRY blockchain](https://github.com/lbryio/lbrycrd) as a global namespace and database of digital content. Blockchain entries contain searchable content metadata, identities, rights and access rules. LBRY also provides a data network that consists of peers (seeders) uploading and downloading data from other peers, possibly in exchange for payments, as well as a distributed hash table used by peers to discover other peers.
LBRY SDK for Python is currently the most full featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components:
LBRY SDK for Python is currently the most fully featured implementation of the LBRY Network protocols and includes many useful components and tools for building decentralized applications. Primary features and components include:
* Built on Python 3.7+ and `asyncio`.
* Kademlia DHT (Distributed Hash Table) implementation for finding peers to download from and announcing to peers what we have to host ([lbry.dht](https://github.com/lbryio/lbry-sdk/tree/master/lbry/lbry/dht)).
+3 -3
View File
@@ -1694,7 +1694,7 @@
{
"name": "support_amount",
"type": "int",
"description": "limit by supports and tips recieved (supports equality constraints)",
"description": "limit by supports and tips received (supports equality constraints)",
"is_required": false
},
{
@@ -1992,7 +1992,7 @@
{
"name": "visible",
"type": "bool",
"description": "Select only Visisble Comments",
"description": "Select only Visible Comments",
"is_required": false
},
{
@@ -2189,7 +2189,7 @@
{
"name": "comparison",
"type": "str",
"description": "logical comparision, (eq | ne | g | ge | l | le)",
"description": "logical comparison, (eq | ne | g | ge | l | le)",
"is_required": false
}
],
+1 -1
View File
@@ -1,4 +1,4 @@
__name__ = "lbry"
__version__ = "0.42.1"
__version__ = "0.42.2"
version = tuple(__version__.split('.'))
+2 -2
View File
@@ -67,7 +67,7 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
self.blob.set_length(blob_response.length)
elif blob_response and not blob_response.error and self.blob.blob_hash != blob_response.blob_hash:
# the server started sending a blob we didn't request
log.warning("%s started sending blob we didnt request %s instead of %s", self.peer_address,
log.warning("%s started sending blob we didn't request %s instead of %s", self.peer_address,
blob_response.blob_hash, self.blob.blob_hash)
return
if response.responses:
@@ -139,7 +139,7 @@ class BlobExchangeClientProtocol(asyncio.Protocol):
log.warning("data rate rejected by %s:%i", self.peer_address, self.peer_port)
return self._blob_bytes_received, self.close()
if not blob_response or blob_response.error:
log.warning("blob cant be downloaded from %s:%i", self.peer_address, self.peer_port)
log.warning("blob can't be downloaded from %s:%i", self.peer_address, self.peer_port)
return self._blob_bytes_received, self.close()
if not blob_response.error and blob_response.blob_hash != self.blob.blob_hash:
log.warning("incoming blob hash mismatch from %s:%i", self.peer_address, self.peer_port)
+10 -12
View File
@@ -19,8 +19,6 @@ class ConnectionManager:
self.outgoing_connected: typing.Set[str] = set()
self.outgoing: typing.DefaultDict[str, int] = collections.defaultdict(int)
self._status = {}
self._total_sent = 0
self._total_received = 0
self._running = False
self._task: typing.Optional[asyncio.Task] = None
@@ -58,29 +56,29 @@ class ConnectionManager:
'outgoing_bps': {},
'total_incoming_mbs': 0.0,
'total_outgoing_mbs': 0.0,
'total_sent': self._total_sent,
'total_received': self._total_received,
'total_sent': 0,
'total_received': 0,
'time': self.loop.time()
}
while True:
last = self.loop.time()
await asyncio.sleep(1, loop=self.loop)
await asyncio.sleep(5, loop=self.loop)
self._status['incoming_bps'].clear()
self._status['outgoing_bps'].clear()
now = self.loop.time()
while self.outgoing:
k, v = self.outgoing.popitem()
self._status['outgoing_bps'][k] = v
self._status['outgoing_bps'][k] = v / (now - last)
while self.incoming:
k, v = self.incoming.popitem()
self._status['incoming_bps'][k] = v
now = self.loop.time()
self._status['incoming_bps'][k] = v / (now - last)
self._status['total_outgoing_mbs'] = int(sum(list(self._status['outgoing_bps'].values())
) / (now - last)) / 1000000.0
)) / 1000000.0
self._status['total_incoming_mbs'] = int(sum(list(self._status['incoming_bps'].values())
) / (now - last)) / 1000000.0
self._total_sent += sum(list(self._status['outgoing_bps'].values()))
self._total_received += sum(list(self._status['incoming_bps'].values()))
)) / 1000000.0
self._status['total_sent'] += sum(list(self._status['outgoing_bps'].values()))
self._status['total_received'] += sum(list(self._status['incoming_bps'].values()))
self._status['time'] = now
+5 -4
View File
@@ -4,13 +4,14 @@ import typing
import binascii
from lbry.utils import resolve_host
from lbry.dht import constants
from lbry.dht.peer import make_kademlia_peer
from lbry.dht.protocol.distance import Distance
from lbry.dht.protocol.iterative_find import IterativeNodeFinder, IterativeValueFinder
from lbry.dht.protocol.protocol import KademliaProtocol
from lbry.dht.peer import KademliaPeer
if typing.TYPE_CHECKING:
from lbry.dht.peer import PeerManager
from lbry.dht.peer import KademliaPeer
log = logging.getLogger(__name__)
@@ -36,7 +37,7 @@ class Node:
total_peers: typing.List['KademliaPeer'] = []
# add all peers in the routing table
total_peers.extend(self.protocol.routing_table.get_peers())
# add all the peers who have announed blobs to us
# add all the peers who have announced blobs to us
total_peers.extend(self.protocol.data_store.get_storing_contacts())
# get ids falling in the midpoint of each bucket that hasn't been recently updated
@@ -141,7 +142,7 @@ class Node:
if known_node_addresses:
peers = [
KademliaPeer(self.loop, address, udp_port=port)
make_kademlia_peer(None, address, port)
for (address, port) in known_node_addresses
]
while True:
@@ -232,7 +233,7 @@ class Node:
if not peer.udp_port:
udp_port_to_try = peer.tcp_port
if not peer.udp_port:
peer.update_udp_port(udp_port_to_try)
peer = make_kademlia_peer(peer.node_id, peer.address, udp_port_to_try, peer.tcp_port)
self.loop.create_task(ping(peer))
else:
log.debug("skip bad peer %s:%i for %s", peer.address, peer.tcp_port, blob_hash)
+24 -42
View File
@@ -3,6 +3,7 @@ import asyncio
import logging
import ipaddress
from binascii import hexlify
from dataclasses import dataclass, field
from functools import lru_cache
from lbry.dht import constants
@@ -11,6 +12,13 @@ from lbry.dht.serialization.datagram import make_compact_address, make_compact_i
log = logging.getLogger(__name__)
@lru_cache(1024)
def make_kademlia_peer(node_id: typing.Optional[bytes], address: typing.Optional[str],
udp_port: typing.Optional[int] = None,
tcp_port: typing.Optional[int] = None) -> 'KademliaPeer':
return KademliaPeer(address, node_id, udp_port, tcp_port=tcp_port)
def is_valid_ipv4(address):
try:
ip = ipaddress.ip_address(address)
@@ -81,10 +89,6 @@ class PeerManager:
self._node_id_mapping[(address, udp_port)] = node_id
self._node_id_reverse_mapping[node_id] = (address, udp_port)
@lru_cache(maxsize=400)
def get_kademlia_peer(self, node_id: bytes, address: str, udp_port: int) -> 'KademliaPeer':
return KademliaPeer(self._loop, address, node_id, udp_port)
def prune(self): # TODO: periodically call this
now = self._loop.time()
to_pop = []
@@ -116,6 +120,8 @@ class PeerManager:
previous_failure, most_recent_failure = self._rpc_failures.get((address, udp_port), (None, None))
last_requested = self._last_requested.get((address, udp_port))
last_replied = self._last_replied.get((address, udp_port))
if node_id is None:
return None
if most_recent_failure and last_replied:
if delay < last_replied > most_recent_failure:
return True
@@ -135,47 +141,31 @@ class PeerManager:
def decode_tcp_peer_from_compact_address(self, compact_address: bytes) -> 'KademliaPeer':
node_id, address, tcp_port = decode_compact_address(compact_address)
return KademliaPeer(self._loop, address, node_id, tcp_port=tcp_port)
return make_kademlia_peer(node_id, address, udp_port=None, tcp_port=tcp_port)
@dataclass(unsafe_hash=True)
class KademliaPeer:
__slots__ = [
'loop',
'_node_id',
'address',
'udp_port',
'tcp_port',
'protocol_version',
]
address: str = field(hash=True)
_node_id: typing.Optional[bytes] = field(hash=True)
udp_port: typing.Optional[int] = field(hash=True)
tcp_port: typing.Optional[int] = field(compare=False, hash=False)
protocol_version: typing.Optional[int] = field(default=1, compare=False, hash=False)
def __init__(self, loop: asyncio.AbstractEventLoop, address: str, node_id: typing.Optional[bytes] = None,
udp_port: typing.Optional[int] = None, tcp_port: typing.Optional[int] = None):
if node_id is not None:
if not len(node_id) == constants.hash_length:
raise ValueError("invalid node_id: {}".format(hexlify(node_id).decode()))
if udp_port is not None and not 0 <= udp_port <= 65536:
def __post_init__(self):
if self._node_id is not None:
if not len(self._node_id) == constants.hash_length:
raise ValueError("invalid node_id: {}".format(hexlify(self._node_id).decode()))
if self.udp_port is not None and not 1 <= self.udp_port <= 65535:
raise ValueError("invalid udp port")
if tcp_port and not 0 <= tcp_port <= 65536:
if self.tcp_port is not None and not 1 <= self.tcp_port <= 65535:
raise ValueError("invalid tcp port")
if not is_valid_ipv4(address):
if not is_valid_ipv4(self.address):
raise ValueError("invalid ip address")
self.loop = loop
self._node_id = node_id
self.address = address
self.udp_port = udp_port
self.tcp_port = tcp_port
self.protocol_version = 1
def update_tcp_port(self, tcp_port: int):
self.tcp_port = tcp_port
def update_udp_port(self, udp_port: int):
self.udp_port = udp_port
def set_id(self, node_id):
if not self._node_id:
self._node_id = node_id
@property
def node_id(self) -> bytes:
return self._node_id
@@ -188,11 +178,3 @@ class KademliaPeer:
def compact_ip(self):
return make_compact_ip(self.address)
def __eq__(self, other):
if not isinstance(other, KademliaPeer):
raise TypeError("invalid type to compare with Peer: %s" % str(type(other)))
return (self.node_id, self.address, self.udp_port) == (other.node_id, other.address, other.udp_port)
def __hash__(self):
return hash((self.node_id, self.address, self.udp_port))
+2 -1
View File
@@ -7,6 +7,7 @@ import logging
from lbry.dht import constants
from lbry.dht.error import RemoteException, TransportNotConnected
from lbry.dht.protocol.distance import Distance
from lbry.dht.peer import make_kademlia_peer
from typing import TYPE_CHECKING
if TYPE_CHECKING:
@@ -150,7 +151,7 @@ class IterativeFinder:
self._add_active(peer)
for contact_triple in response.get_close_triples():
node_id, address, udp_port = contact_triple
self._add_active(self.peer_manager.get_kademlia_peer(node_id, address, udp_port))
self._add_active(make_kademlia_peer(node_id, address, udp_port))
self.check_result_ready(response)
async def _send_probe(self, peer: 'KademliaPeer'):
+18 -33
View File
@@ -15,6 +15,7 @@ from lbry.dht.serialization.datagram import RESPONSE_TYPE, ERROR_TYPE, PAGE_KEY
from lbry.dht.error import RemoteException, TransportNotConnected
from lbry.dht.protocol.routing_table import TreeRoutingTable
from lbry.dht.protocol.data_store import DictDataStore
from lbry.dht.peer import make_kademlia_peer
if typing.TYPE_CHECKING:
from lbry.dht.peer import PeerManager, KademliaPeer
@@ -322,6 +323,9 @@ class KademliaProtocol(DatagramProtocol):
return args, {}
async def _add_peer(self, peer: 'KademliaPeer'):
if not peer.node_id:
log.warning("Tried adding a peer with no node id!")
return False
for p in self.routing_table.get_peers():
if (p.address, p.udp_port) == (peer.address, peer.udp_port) and p.node_id != peer.node_id:
self.routing_table.remove_peer(p)
@@ -447,7 +451,7 @@ class KademliaProtocol(DatagramProtocol):
try:
peer = self.routing_table.get_peer(request_datagram.node_id)
except IndexError:
peer = self.peer_manager.get_kademlia_peer(request_datagram.node_id, address[0], address[1])
peer = make_kademlia_peer(request_datagram.node_id, address[0], address[1])
try:
self._handle_rpc(peer, request_datagram)
# if the contact is not known to be bad (yet) and we haven't yet queried it, send it a ping so that it
@@ -494,8 +498,7 @@ class KademliaProtocol(DatagramProtocol):
elif response_datagram.node_id == self.node_id:
df.set_exception(RemoteException("incoming message is from our node id"))
return
peer.set_id(response_datagram.node_id)
peer.update_udp_port(address[1])
peer = make_kademlia_peer(response_datagram.node_id, address[0], address[1])
self.peer_manager.report_last_replied(address[0], address[1])
self.peer_manager.update_contact_triple(peer.node_id, address[0], address[1])
if not df.cancelled():
@@ -529,7 +532,7 @@ class KademliaProtocol(DatagramProtocol):
elif error_datagram.response not in old_protocol_errors:
log.warning(error_msg)
else:
log.warning("known dht protocol backwards compatibility error with %s:%i (lbrynet v%s)",
log.debug("known dht protocol backwards compatibility error with %s:%i (lbrynet v%s)",
peer.address, peer.udp_port, old_protocol_errors[error_datagram.response])
df.set_exception(remote_exception)
return
@@ -539,7 +542,7 @@ class KademliaProtocol(DatagramProtocol):
f"pending request: {str(remote_exception)}"
log.warning(msg)
else:
log.warning("known dht protocol backwards compatibility error with %s:%i (lbrynet v%s)",
log.debug("known dht protocol backwards compatibility error with %s:%i (lbrynet v%s)",
address[0], address[1], old_protocol_errors[error_datagram.response])
def datagram_received(self, datagram: bytes, address: typing.Tuple[str, int]) -> None:
@@ -643,7 +646,8 @@ class KademliaProtocol(DatagramProtocol):
return False
return True
async def store_to_peer(self, hash_value: bytes, peer: 'KademliaPeer') -> typing.Tuple[bytes, bool]:
async def store_to_peer(self, hash_value: bytes, peer: 'KademliaPeer',
retry: bool = True) -> typing.Tuple[bytes, bool]:
async def __store():
res = await self.get_rpc_peer(peer).store(hash_value)
if res != b"OK":
@@ -655,34 +659,15 @@ class KademliaProtocol(DatagramProtocol):
return await __store()
except asyncio.TimeoutError:
log.debug("Timeout while storing blob_hash %s at %s", binascii.hexlify(hash_value).decode()[:8], peer)
return peer.node_id, False
except ValueError as err:
log.error("Unexpected response: %s" % err)
return peer.node_id, False
except RemoteException as err:
if 'Invalid token' in str(err):
self.peer_manager.clear_token(peer.node_id)
try:
return await __store()
except (ValueError, asyncio.TimeoutError, RemoteException):
return peer.node_id, False
else:
if 'Invalid token' not in str(err):
log.exception("Unexpected error while storing blob_hash")
return peer.node_id, False
def _write(self, data: bytes, address: typing.Tuple[str, int]):
if self.transport:
try:
self.transport.sendto(data, address)
except OSError as err:
if err.errno == socket.EWOULDBLOCK:
# i'm scared this may swallow important errors, but i get a million of these
# on Linux and it doesn't seem to affect anything -grin
log.warning("Can't send data to dht: EWOULDBLOCK")
# elif err.errno == socket.ENETUNREACH:
# # this should probably try to retransmit when the network connection is back
# log.error("Network is unreachable")
else:
log.error("DHT socket error sending %i bytes to %s:%i - %s (code %i)",
len(data), address[0], address[1], str(err), err.errno)
raise err
else:
raise TransportNotConnected()
return peer.node_id, False
self.peer_manager.clear_token(peer.node_id)
if not retry:
return peer.node_id, False
return await self.store_to_peer(hash_value, peer, retry=False)
+76 -35
View File
@@ -23,6 +23,7 @@ from lbry import utils
from lbry.conf import Config, Setting
from lbry.blob.blob_file import is_valid_blobhash, BlobBuffer
from lbry.blob_exchange.downloader import download_blob
from lbry.dht.peer import make_kademlia_peer
from lbry.error import DownloadSDTimeout, ComponentsNotStarted
from lbry.error import NullFundsError, NegativeFundsError, ComponentStartConditionNotMet
from lbry.extras import system_info
@@ -41,7 +42,6 @@ from lbry.wallet.dewies import dewies_to_lbc, lbc_to_dewies
from lbry.schema.claim import Claim
from lbry.schema.url import URL
if typing.TYPE_CHECKING:
from lbry.blob.blob_manager import BlobManager
from lbry.dht.node import Node
@@ -72,7 +72,9 @@ def requires(*components, **conditions):
raise ComponentsNotStarted("the following required components have not yet started: "
"%s" % json.dumps(components))
return fn(*args, **kwargs)
return _inner
return _wrap
@@ -81,6 +83,7 @@ def deprecated(new_command=None):
f.new_command = new_command
f._deprecated = True
return f
return _deprecated_wrapper
@@ -124,13 +127,13 @@ async def maybe_paginate(get_records: Callable, get_record_count: Callable,
page: Optional[int], page_size: Optional[int], **constraints):
if None not in (page, page_size):
constraints.update({
"offset": page_size * (page-1),
"offset": page_size * (page - 1),
"limit": page_size
})
total_items = await get_record_count(**constraints)
return {
"items": await get_records(**constraints),
"total_pages": int((total_items + (page_size-1)) / page_size),
"total_pages": int((total_items + (page_size - 1)) / page_size),
"total_items": total_items,
"page": page, "page_size": page_size
}
@@ -208,7 +211,7 @@ class JSONRPCError:
for i, t in enumerate(trace_lines):
if "--- <exception caught here> ---" in t:
if len(trace_lines) > i + 1:
self.traceback = [j for j in trace_lines[i+1:] if j]
self.traceback = [j for j in trace_lines[i + 1:] if j]
break
def to_dict(self):
@@ -338,7 +341,7 @@ class Daemon(metaclass=JSONRPCServerType):
name, = name_parts
elif len(name_parts) == 2:
group, name = name_parts
assert group in api['groups'],\
assert group in api['groups'], \
f"Group {group} does not have doc string for command {full_name}."
else:
raise NameError(f'Could not parse method name: {jsonrpc_method}')
@@ -589,7 +592,9 @@ class Daemon(metaclass=JSONRPCServerType):
except Exception as e: # pylint: disable=broad-except
log.exception("error handling api request")
return JSONRPCError(
str(e), JSONRPCError.CODE_APPLICATION_ERROR, format_exc()
f"Error calling {function_name} with args {args}\n" + str(e),
JSONRPCError.CODE_APPLICATION_ERROR,
format_exc()
)
def _verify_method_is_callable(self, function_path):
@@ -621,7 +626,7 @@ class Daemon(metaclass=JSONRPCServerType):
missing_required_params = [
required_param
for required_param in argspec.args[len(args_tup)+1:-num_optional_params]
for required_param in argspec.args[len(args_tup) + 1:-num_optional_params]
if required_param not in args_dict
]
if len(missing_required_params):
@@ -912,7 +917,7 @@ class Daemon(metaclass=JSONRPCServerType):
for resolved_uri in resolved:
results[resolved_uri] = resolved[resolved_uri] if resolved[resolved_uri] is not None else \
{"error": f"{resolved_uri} did not resolve to a claim"}
{"error": f"{resolved_uri} did not resolve to a claim"}
return results
@@ -1514,7 +1519,12 @@ class Daemon(metaclass=JSONRPCServerType):
Returns: {Transaction}
"""
wallet = self.wallet_manager.get_wallet_or_default(wallet_id)
account = wallet.get_account_or_default(account_id)
if account_id:
account = wallet.get_account_or_error(account_id)
accounts = [account]
else:
account = wallet.default_account
accounts = wallet.accounts
amount = self.get_dewies_or_error("amount", amount)
if not amount:
@@ -1535,14 +1545,14 @@ class Daemon(metaclass=JSONRPCServerType):
)
tx = await Transaction.create(
[], outputs, [account], account
[], outputs, accounts, account
)
if not preview:
await self.ledger.broadcast(tx)
await self.analytics_manager.send_credits_sent()
else:
await account.ledger.release_tx(tx)
await self.ledger.release_tx(tx)
return tx
@@ -1726,7 +1736,7 @@ class Daemon(metaclass=JSONRPCServerType):
--blobs_in_stream<blobs_in_stream> : (int) get file with matching blobs in stream
--blobs_remaining=<blobs_remaining> : (int) amount of remaining blobs to download
--sort=<sort_by> : (str) field to sort by (one of the above filter fields)
--comparison=<comparison> : (str) logical comparision, (eq | ne | g | ge | l | le)
--comparison=<comparison> : (str) logical comparison, (eq | ne | g | ge | l | le)
Returns: {List[File]}
"""
@@ -1975,7 +1985,7 @@ class Daemon(metaclass=JSONRPCServerType):
a release time the publish time is used instead
(supports equality constraints)
--amount=<amount> : (int) limit by claim value (supports equality constraints)
--support_amount=<support_amount>: (int) limit by supports and tips recieved (supports
--support_amount=<support_amount>: (int) limit by supports and tips received (supports
equality constraints)
--effective_amount=<effective_amount>: (int) limit by total value (initial claim value plus
all tips and supports received), this amount is
@@ -2032,11 +2042,11 @@ class Daemon(metaclass=JSONRPCServerType):
if kwargs.pop('invalid_channel_signature', False):
kwargs['signature_valid'] = 0
page_num, page_size = abs(kwargs.pop('page', 1)), min(abs(kwargs.pop('page_size', 10)), 50)
kwargs.update({'offset': page_size * (page_num-1), 'limit': page_size})
kwargs.update({'offset': page_size * (page_num - 1), 'limit': page_size})
txos, offset, total = await self.ledger.claim_search(**kwargs)
result = {"items": txos, "page": page_num, "page_size": page_size}
if not kwargs.pop('no_totals', False):
result['total_pages'] = int((total + (page_size-1)) / page_size)
result['total_pages'] = int((total + (page_size - 1)) / page_size)
result['total_items'] = total
return result
@@ -2239,7 +2249,7 @@ class Daemon(metaclass=JSONRPCServerType):
--clear_locations : (bool) clear existing locations (prior to adding new ones)
--thumbnail_url=<thumbnail_url>: (str) thumbnail url
--cover_url=<cover_url> : (str) url of cover image
--account_id=<account_id> : (str) account to use for holding the transaction
--account_id=<account_id> : (str) account in which to look for channel (default: all)
--wallet_id=<wallet_id> : (str) restrict operation to specific wallet
--funding_account_ids=<funding_account_ids>: (list) ids of accounts to fund this transaction
--claim_address=<claim_address>: (str) address where the channel is sent
@@ -2254,15 +2264,21 @@ class Daemon(metaclass=JSONRPCServerType):
Returns: {Transaction}
"""
wallet = self.wallet_manager.get_wallet_or_default(wallet_id)
account = wallet.get_account_or_default(account_id)
funding_accounts = wallet.get_accounts_or_all(funding_account_ids)
if account_id:
account = wallet.get_account_or_error(account_id)
accounts = [account]
else:
account = wallet.default_account
accounts = wallet.accounts
existing_channels = await self.ledger.get_claims(
wallet=wallet, accounts=[account] if account_id else wallet.accounts, claim_id=claim_id
wallet=wallet, accounts=accounts, claim_id=claim_id
)
if len(existing_channels) != 1:
account_ids = ', '.join(f"'{account.id}'" for account in accounts)
raise Exception(
f"Can't find the channel '{claim_id}' in account '{account.id}'."
f"Can't find the channel '{claim_id}' in account(s) {account_ids}."
)
old_txo = existing_channels[0]
if not old_txo.claim.is_channel:
@@ -2337,12 +2353,21 @@ class Daemon(metaclass=JSONRPCServerType):
Returns: {Transaction}
"""
wallet = self.wallet_manager.get_wallet_or_default(wallet_id)
account = wallet.get_account_or_default(account_id)
if account_id:
account = wallet.get_account_or_error(account_id)
accounts = [account]
else:
account = wallet.default_account
accounts = wallet.accounts
if txid is not None and nout is not None:
claims = await account.get_claims(**{'txo.txid': txid, 'txo.position': nout})
claims = await self.ledger.get_claims(
wallet=wallet, accounts=accounts, **{'txo.txid': txid, 'txo.position': nout}
)
elif claim_id is not None:
claims = await account.get_claims(claim_id=claim_id)
claims = await self.ledger.get_claims(
wallet=wallet, accounts=accounts, claim_id=claim_id
)
else:
raise Exception('Must specify claim_id, or txid and nout')
@@ -2717,7 +2742,7 @@ class Daemon(metaclass=JSONRPCServerType):
)
claim = Claim()
claim.stream.update(file_path=file_path, sd_hash='0'*96, **kwargs)
claim.stream.update(file_path=file_path, sd_hash='0' * 96, **kwargs)
tx = await Transaction.claim_create(
name, claim, amount, claim_address, funding_accounts, funding_accounts[0], channel
)
@@ -2845,7 +2870,7 @@ class Daemon(metaclass=JSONRPCServerType):
--clear_channel : (bool) remove channel signature
--channel_account_id=<channel_account_id>: (str) one or more account ids for accounts to look in
for channel certificates, defaults to all accounts.
--account_id=<account_id> : (str) account to use for holding the transaction
--account_id=<account_id> : (str) account in which to look for stream (default: all)
--wallet_id=<wallet_id> : (str) restrict operation to specific wallet
--funding_account_ids=<funding_account_ids>: (list) ids of accounts to fund this transaction
--claim_address=<claim_address>: (str) address where the claim is sent to, if not specified
@@ -2860,13 +2885,21 @@ class Daemon(metaclass=JSONRPCServerType):
Returns: {Transaction}
"""
wallet = self.wallet_manager.get_wallet_or_default(wallet_id)
account = wallet.get_account_or_default(account_id)
funding_accounts = wallet.get_accounts_or_all(funding_account_ids)
if account_id:
account = wallet.get_account_or_error(account_id)
accounts = [account]
else:
account = wallet.default_account
accounts = wallet.accounts
existing_claims = await account.get_claims(claim_id=claim_id)
existing_claims = await self.ledger.get_claims(
wallet=wallet, accounts=accounts, claim_id=claim_id
)
if len(existing_claims) != 1:
account_ids = ', '.join(f"'{account.id}'" for account in accounts)
raise Exception(
f"Can't find the claim '{claim_id}' in account '{account.id}'."
f"Can't find the stream '{claim_id}' in account(s) {account_ids}."
)
old_txo = existing_claims[0]
if not old_txo.claim.is_stream:
@@ -2970,12 +3003,21 @@ class Daemon(metaclass=JSONRPCServerType):
Returns: {Transaction}
"""
wallet = self.wallet_manager.get_wallet_or_default(wallet_id)
account = wallet.get_account_or_default(account_id)
if account_id:
account = wallet.get_account_or_error(account_id)
accounts = [account]
else:
account = wallet.default_account
accounts = wallet.accounts
if txid is not None and nout is not None:
claims = await account.get_claims(**{'txo.txid': txid, 'txo.position': nout})
claims = await self.ledger.get_claims(
wallet=wallet, accounts=accounts, **{'txo.txid': txid, 'txo.position': nout}
)
elif claim_id is not None:
claims = await account.get_claims(claim_id=claim_id)
claims = await self.ledger.get_claims(
wallet=wallet, accounts=accounts, claim_id=claim_id
)
else:
raise Exception('Must specify claim_id, or txid and nout')
@@ -2983,14 +3025,14 @@ class Daemon(metaclass=JSONRPCServerType):
raise Exception('No claim found for the specified claim_id or txid:nout')
tx = await Transaction.create(
[Input.spend(txo) for txo in claims], [], [account], account
[Input.spend(txo) for txo in claims], [], accounts, account
)
if not preview:
await self.broadcast_or_release(tx, blocking)
await self.analytics_manager.send_claim_action('abandon')
else:
await account.ledger.release_tx(tx)
await self.ledger.release_tx(tx)
return tx
@@ -3614,8 +3656,7 @@ class Daemon(metaclass=JSONRPCServerType):
"""
peer = None
if node_id and address and port:
peer = self.component_manager.peer_manager.get_kademlia_peer(unhexlify(node_id), address,
udp_port=int(port))
peer = make_kademlia_peer(unhexlify(node_id), address, udp_port=int(port))
try:
return await self.dht_node.protocol.get_rpc_peer(peer).ping()
except asyncio.TimeoutError:
@@ -3695,7 +3736,7 @@ class Daemon(metaclass=JSONRPCServerType):
--is_channel_signature_valid : (bool) Only include comments with valid signatures.
[Warning: Paginated total size will not change, even
if list reduces]
--visible : (bool) Select only Visisble Comments
--visible : (bool) Select only Visible Comments
--hidden : (bool) Select only Hidden Comments
Returns:
+11 -4
View File
@@ -3,6 +3,7 @@ from aiohttp.client_exceptions import ClientError
import json
import logging.handlers
import traceback
import aiohttp
from lbry import utils, __version__
@@ -42,6 +43,8 @@ class HTTPSLogglyHandler(logging.Handler):
self.url = "https://logs-01.loggly.com/inputs/{token}/tag/{tag}".format(
token=utils.deobfuscate(loggly_token), tag='lbrynet-' + __version__
)
self._loop = asyncio.get_event_loop()
self._session = aiohttp.ClientSession()
def get_full_message(self, record):
if record.exc_info:
@@ -49,13 +52,17 @@ class HTTPSLogglyHandler(logging.Handler):
else:
return record.getMessage()
async def _emit(self, record):
async def _emit(self, record, retry=True):
data = self.format(record).encode()
try:
async with utils.aiohttp_request('post', self.url, data=self.format(record).encode(),
cookies=self.cookies) as response:
async with self._session.post(self.url, data=data,
cookies=self.cookies) as response:
self.cookies.update(response.cookies)
except ClientError:
pass
if self._loop.is_running() and retry:
await self._session.close()
self._session = aiohttp.ClientSession()
return await self._emit(record, retry=False)
def emit(self, record):
asyncio.ensure_future(self._emit(record))
+3 -2
View File
@@ -2,11 +2,12 @@ import asyncio
import typing
import logging
import binascii
from lbry.dht.peer import make_kademlia_peer
from lbry.error import DownloadSDTimeout
from lbry.utils import resolve_host, lru_cache_concurrent
from lbry.stream.descriptor import StreamDescriptor
from lbry.blob_exchange.downloader import BlobDownloader
from lbry.dht.peer import KademliaPeer
if typing.TYPE_CHECKING:
from lbry.conf import Config
from lbry.dht.node import Node
@@ -50,7 +51,7 @@ class StreamDownloader:
def _delayed_add_fixed_peers():
self.added_fixed_peers = True
self.peer_queue.put_nowait([
KademliaPeer(self.loop, address=address, tcp_port=port + 1)
make_kademlia_peer(None, address, None, tcp_port=port + 1)
for address, port in addresses
])
+1 -1
View File
@@ -442,7 +442,7 @@ class StreamManager:
except asyncio.TimeoutError:
error = DownloadDataTimeout(stream.sd_hash)
raise error
except Exception as err: # forgive data timeout, dont delete stream
except Exception as err: # forgive data timeout, don't delete stream
error = err
raise
finally:
+1 -1
View File
@@ -79,7 +79,7 @@ class LBC(Coin):
@classmethod
def address_from_script(cls, script):
'''Given a pk_script, return the adddress it pays to, or None.'''
'''Given a pk_script, return the address it pays to, or None.'''
return ScriptPubKey.pay_to(cls.address_handlers, script)
@classmethod
@@ -507,6 +507,22 @@ class ChannelCommands(CommandTestCase):
# second wallet should be able to update now
await daemon2.jsonrpc_channel_update(claim_id, bid='0.5')
async def test_channel_update_across_accounts(self):
account2 = await self.daemon.jsonrpc_account_create('second account')
channel = await self.out(self.channel_create('@spam', '1.0', account_id=account2.id))
# channel not in account1
with self.assertRaisesRegex(Exception, "Can't find the channel"):
await self.channel_update(self.get_claim_id(channel), bid='2.0', account_id=self.account.id)
# channel is in account2
await self.channel_update(self.get_claim_id(channel), bid='2.0', account_id=account2.id)
result = await self.out(self.daemon.jsonrpc_channel_list())
self.assertEqual(result[0]['amount'], '2.0')
# check all accounts for channel
await self.channel_update(self.get_claim_id(channel), bid='3.0')
result = await self.out(self.daemon.jsonrpc_channel_list())
self.assertEqual(result[0]['amount'], '3.0')
await self.channel_abandon(self.get_claim_id(channel))
class StreamCommands(ClaimTestCase):
@@ -564,6 +580,22 @@ class StreamCommands(ClaimTestCase):
self.assertEqual(len(tx['outputs']), 1) # no change
self.assertEqual(len(await self.daemon.jsonrpc_claim_list()), 2)
async def test_stream_update_and_abandon_across_accounts(self):
account2 = await self.daemon.jsonrpc_account_create('second account')
stream = await self.out(self.stream_create('spam', '1.0', account_id=account2.id))
# stream not in account1
with self.assertRaisesRegex(Exception, "Can't find the stream"):
await self.stream_update(self.get_claim_id(stream), bid='2.0', account_id=self.account.id)
# stream is in account2
await self.stream_update(self.get_claim_id(stream), bid='2.0', account_id=account2.id)
result = await self.out(self.daemon.jsonrpc_stream_list())
self.assertEqual(result[0]['amount'], '2.0')
# check all accounts for stream
await self.stream_update(self.get_claim_id(stream), bid='3.0')
result = await self.out(self.daemon.jsonrpc_stream_list())
self.assertEqual(result[0]['amount'], '3.0')
await self.stream_abandon(self.get_claim_id(stream))
async def test_publishing_checks_all_accounts_for_channel(self):
account1_id, account1 = self.account.id, self.account
new_account = await self.out(self.daemon.jsonrpc_account_create('second account'))
+5 -5
View File
@@ -3,7 +3,7 @@ from binascii import hexlify
from lbry.dht import constants
from lbry.dht.node import Node
from lbry.dht.peer import PeerManager, KademliaPeer
from lbry.dht.peer import PeerManager, make_kademlia_peer
from torba.testcase import AsyncioTestCase
@@ -39,7 +39,7 @@ class DHTIntegrationTest(AsyncioTestCase):
bad_peers = []
for candidate in self.nodes[1:10]:
address, port, node_id = candidate.protocol.external_ip, candidate.protocol.udp_port, candidate.protocol.node_id
peer = KademliaPeer(self.loop, address, node_id, port)
peer = make_kademlia_peer(node_id, address, udp_port=port)
bad_peers.append(peer)
node.protocol.add_peer(peer)
candidate.stop()
@@ -70,7 +70,7 @@ class DHTIntegrationTest(AsyncioTestCase):
await asyncio.sleep(.1)
timeout -= 1
if not timeout:
self.fail("node didnt join back after 2 seconds")
self.fail("node didn't join back after 2 seconds")
async def test_announce_no_peers(self):
await self.setup_network(1)
@@ -100,9 +100,9 @@ class DHTIntegrationTest(AsyncioTestCase):
await self.setup_network(2, seed_nodes=2)
node1, node2 = self.nodes
node2.stop()
# forcefully make it a bad peer but dont remove it from routing table
# forcefully make it a bad peer but don't remove it from routing table
address, port, node_id = node2.protocol.external_ip, node2.protocol.udp_port, node2.protocol.node_id
peer = KademliaPeer(self.loop, address, node_id, port)
peer = make_kademlia_peer(node_id, address, udp_port=port)
self.assertTrue(node1.protocol.peer_manager.peer_is_good(peer))
node1.protocol.peer_manager.report_failure(node2.protocol.external_ip, node2.protocol.udp_port)
node1.protocol.peer_manager.report_failure(node2.protocol.external_ip, node2.protocol.udp_port)
+2 -2
View File
@@ -98,7 +98,7 @@ class FileCommands(CommandTestCase):
resp = await self.daemon.jsonrpc_get('lbry://foo', timeout=2, save_file=True)
self.assertIn('error', resp)
self.assertEqual('Failed to download data blobs for sd hash %s within timeout' % sd_hash, resp['error'])
self.assertTrue(await self.daemon.jsonrpc_file_delete(claim_name='foo'), "data timeout didnt create a file")
self.assertTrue(await self.daemon.jsonrpc_file_delete(claim_name='foo'), "data timeout didn't create a file")
await self.server.blob_manager.delete_blobs([sd_hash])
resp = await self.daemon.jsonrpc_get('lbry://foo', timeout=2, save_file=True)
self.assertIn('error', resp)
@@ -118,7 +118,7 @@ class FileCommands(CommandTestCase):
handle.write(b'some other stuff was there instead')
self.daemon.stream_manager.stop()
await self.daemon.stream_manager.start()
await asyncio.wait_for(self.wait_files_to_complete(), timeout=5) # if this hangs, file didnt get set completed
await asyncio.wait_for(self.wait_files_to_complete(), timeout=5) # if this hangs, file didn't get set completed
# check that internal state got through up to the file list API
stream = self.daemon.stream_manager.get_stream_by_stream_hash(file_info['stream_hash'])
file_info = self.sout(self.daemon.jsonrpc_file_list()[0])
@@ -178,7 +178,7 @@ class ResolveCommand(BaseResolveTestCase):
channel = (await self.channel_create('@abc', '1.0'))['outputs'][0]
orphan_claim_id = self.get_claim_id(orphan_claim)
# Original channel doesnt exists anymore, so the signature is invalid. For invalid signatures, resolution is
# Original channel doesn't exists anymore, so the signature is invalid. For invalid signatures, resolution is
# only possible outside a channel
response = await self.resolve('lbry://@abc/on-channel-claim')
self.assertEqual(response, {
+1 -1
View File
@@ -413,7 +413,7 @@ class RangeRequestsLRUCache(CommandTestCase):
self.server.stop_server()
# running with cache size 0 gets through without errors without
# this since the server doesnt stop immediately
# this since the server doesn't stop immediately
await asyncio.sleep(1, loop=self.loop)
await self._request_stream()
@@ -55,7 +55,7 @@ class TestHeadersComponent(CommandTestCase):
async def test_cant_reach_host(self):
HeadersComponent.HEADERS_URL = 'notthere/'
os.unlink(self.headers_component.headers.path)
# test is that this doesnt raise
# test is that this doesn't raise
await self.headers_component.start()
self.assertTrue(self.component_manager.get_components_status()['blockchain_headers'])
self.assertEqual(await self.headers_component.get_status(), {})
@@ -12,7 +12,8 @@ from lbry.extras.daemon.storage import SQLiteStorage
from lbry.blob.blob_manager import BlobManager
from lbry.blob_exchange.server import BlobServer, BlobServerProtocol
from lbry.blob_exchange.client import request_blob
from lbry.dht.peer import KademliaPeer, PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
# import logging
# logging.getLogger("lbry").setLevel(logging.DEBUG)
@@ -43,7 +44,7 @@ class BlobExchangeTestBase(AsyncioTestCase):
self.client_storage = SQLiteStorage(self.client_config, os.path.join(self.client_dir, "lbrynet.sqlite"))
self.client_blob_manager = BlobManager(self.loop, self.client_dir, self.client_storage, self.client_config)
self.client_peer_manager = PeerManager(self.loop)
self.server_from_client = KademliaPeer(self.loop, "127.0.0.1", b'1' * 48, tcp_port=33333)
self.server_from_client = make_kademlia_peer(b'1' * 48, "127.0.0.1", tcp_port=33333)
await self.client_storage.open()
await self.server_storage.open()
@@ -102,7 +103,7 @@ class TestBlobExchange(BlobExchangeTestBase):
second_client_blob_manager = BlobManager(
self.loop, second_client_dir, second_client_storage, second_client_conf
)
server_from_second_client = KademliaPeer(self.loop, "127.0.0.1", b'1' * 48, tcp_port=33333)
server_from_second_client = make_kademlia_peer(b'1' * 48, "127.0.0.1", tcp_port=33333)
await second_client_storage.open()
await second_client_blob_manager.setup()
@@ -193,7 +194,7 @@ class TestBlobExchange(BlobExchangeTestBase):
second_client_blob_manager = BlobManager(
self.loop, second_client_dir, second_client_storage, second_client_conf
)
server_from_second_client = KademliaPeer(self.loop, "127.0.0.1", b'1' * 48, tcp_port=33333)
server_from_second_client = make_kademlia_peer(b'1' * 48, "127.0.0.1", tcp_port=33333)
await second_client_storage.open()
await second_client_blob_manager.setup()
@@ -1,7 +1,7 @@
import asyncio
from unittest import mock, TestCase
from lbry.dht.protocol.data_store import DictDataStore
from lbry.dht.peer import PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
class DataStoreTests(TestCase):
@@ -13,7 +13,7 @@ class DataStoreTests(TestCase):
def _test_add_peer_to_blob(self, blob=b'2' * 48, node_id=b'1' * 48, address='1.2.3.4', tcp_port=3333,
udp_port=4444):
peer = self.peer_manager.get_kademlia_peer(node_id, address, udp_port)
peer = make_kademlia_peer(node_id, address, udp_port)
peer.update_tcp_port(tcp_port)
before = self.data_store.get_peers_for_blob(blob)
self.data_store.add_peer_to_blob(peer, blob)
+7 -7
View File
@@ -2,7 +2,7 @@ import struct
import asyncio
from lbry.utils import generate_id
from lbry.dht.protocol.routing_table import KBucket
from lbry.dht.peer import PeerManager, KademliaPeer
from lbry.dht.peer import PeerManager, make_kademlia_peer
from lbry.dht import constants
from torba.testcase import AsyncioTestCase
@@ -29,8 +29,8 @@ class TestKBucket(AsyncioTestCase):
self.kbucket = KBucket(self.peer_manager, 0, 2**constants.hash_bits, generate_id())
def test_add_peer(self):
peer = KademliaPeer(None, '1.2.3.4', constants.generate_id(2), udp_port=4444)
peer_update2 = KademliaPeer(None, '1.2.3.4', constants.generate_id(2), udp_port=4445)
peer = make_kademlia_peer(constants.generate_id(2), "1.2.3.4", udp_port=4444)
peer_update2 = make_kademlia_peer(constants.generate_id(2), "1.2.3.4", udp_port=4445)
self.assertListEqual([], self.kbucket.peers)
@@ -59,12 +59,12 @@ class TestKBucket(AsyncioTestCase):
# Test if contacts can be added to empty list
# Add k contacts to bucket
for i in range(constants.k):
peer = self.peer_manager.get_kademlia_peer(generate_id(), next(self.address_generator), 4444)
peer = make_kademlia_peer(generate_id(), next(self.address_generator), 4444)
self.assertTrue(self.kbucket.add_peer(peer))
self.assertEqual(peer, self.kbucket.peers[i])
# Test if contact is not added to full list
peer = self.peer_manager.get_kademlia_peer(generate_id(), next(self.address_generator), 4444)
peer = make_kademlia_peer(generate_id(), next(self.address_generator), 4444)
self.assertFalse(self.kbucket.add_peer(peer))
# Test if an existing contact is updated correctly if added again
@@ -125,13 +125,13 @@ class TestKBucket(AsyncioTestCase):
def test_remove_peer(self):
# try remove contact from empty list
peer = self.peer_manager.get_kademlia_peer(generate_id(), next(self.address_generator), 4444)
peer = make_kademlia_peer(generate_id(), next(self.address_generator), 4444)
self.assertRaises(ValueError, self.kbucket.remove_peer, peer)
added = []
# Add couple contacts
for i in range(constants.k-2):
peer = self.peer_manager.get_kademlia_peer(generate_id(), next(self.address_generator), 4444)
peer = make_kademlia_peer(generate_id(), next(self.address_generator), 4444)
self.assertTrue(self.kbucket.add_peer(peer))
added.append(peer)
+10 -10
View File
@@ -5,7 +5,7 @@ from tests import dht_mocks
from lbry.dht.serialization.bencoding import bencode, bdecode
from lbry.dht import constants
from lbry.dht.protocol.protocol import KademliaProtocol
from lbry.dht.peer import PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
class TestProtocol(AsyncioTestCase):
@@ -22,7 +22,7 @@ class TestProtocol(AsyncioTestCase):
await loop.create_datagram_endpoint(lambda: peer1, ('1.2.3.4', 4444))
await loop.create_datagram_endpoint(lambda: peer2, ('1.2.3.5', 4444))
peer = peer2.peer_manager.get_kademlia_peer(node_id1, '1.2.3.4', udp_port=4444)
peer = make_kademlia_peer(node_id1, '1.2.3.4', udp_port=4444)
result = await peer2.get_rpc_peer(peer).ping()
self.assertEqual(result, b'pong')
peer1.stop()
@@ -43,7 +43,7 @@ class TestProtocol(AsyncioTestCase):
await loop.create_datagram_endpoint(lambda: peer1, ('1.2.3.4', 4444))
await loop.create_datagram_endpoint(lambda: peer2, ('1.2.3.5', 4444))
peer = peer2.peer_manager.get_kademlia_peer(node_id1, '1.2.3.4', udp_port=4444)
peer = make_kademlia_peer(node_id1, '1.2.3.4', udp_port=4444)
self.assertEqual(None, peer2.peer_manager.get_node_token(peer.node_id))
await peer2.get_rpc_peer(peer).find_value(b'1' * 48)
self.assertNotEqual(None, peer2.peer_manager.get_node_token(peer.node_id))
@@ -65,12 +65,12 @@ class TestProtocol(AsyncioTestCase):
await loop.create_datagram_endpoint(lambda: peer1, ('1.2.3.4', 4444))
await loop.create_datagram_endpoint(lambda: peer2, ('1.2.3.5', 4444))
peer = peer2.peer_manager.get_kademlia_peer(node_id1, '1.2.3.4', udp_port=4444)
peer2_from_peer1 = peer1.peer_manager.get_kademlia_peer(
peer = make_kademlia_peer(node_id1, '1.2.3.4', udp_port=4444)
peer2_from_peer1 = make_kademlia_peer(
peer2.node_id, peer2.external_ip, udp_port=peer2.udp_port
)
peer2_from_peer1.update_tcp_port(3333)
peer3 = peer1.peer_manager.get_kademlia_peer(
peer3 = make_kademlia_peer(
constants.generate_id(), '1.2.3.6', udp_port=4444
)
store_result = await peer2.store_to_peer(b'2' * 48, peer)
@@ -103,7 +103,7 @@ class TestProtocol(AsyncioTestCase):
)
await self.loop.create_datagram_endpoint(lambda: proto, (address, 4444))
proto.start()
return proto, other_peer.peer_manager.get_kademlia_peer(node_id, address, udp_port=udp_port)
return proto, make_kademlia_peer(node_id, address, udp_port=udp_port)
async def test_add_peer_after_handle_request(self):
with dht_mocks.mock_network_loop(self.loop):
@@ -129,7 +129,7 @@ class TestProtocol(AsyncioTestCase):
peer1.routing_table.remove_peer(peer_2_from_peer_1)
# peers not known by be good/bad should be enqueued to maybe-ping
peer1_from_peer3 = peer3.get_rpc_peer(peer3.peer_manager.get_kademlia_peer(node_id1, '1.2.3.4', 4444))
peer1_from_peer3 = peer3.get_rpc_peer(make_kademlia_peer(node_id1, '1.2.3.4', 4444))
self.assertEqual(0, len(peer1.ping_queue._pending_contacts))
pong = await peer1_from_peer3.ping()
self.assertEqual(b'pong', pong)
@@ -137,7 +137,7 @@ class TestProtocol(AsyncioTestCase):
peer1.ping_queue._pending_contacts.clear()
# peers who are already good should be added
peer1_from_peer4 = peer4.get_rpc_peer(peer4.peer_manager.get_kademlia_peer(node_id1, '1.2.3.4', 4444))
peer1_from_peer4 = peer4.get_rpc_peer(make_kademlia_peer(node_id1, '1.2.3.4', 4444))
peer1.peer_manager.update_contact_triple(node_id4,'1.2.3.7', 4444)
peer1.peer_manager.report_last_replied('1.2.3.7', 4444)
self.assertEqual(0, len(peer1.ping_queue._pending_contacts))
@@ -149,7 +149,7 @@ class TestProtocol(AsyncioTestCase):
peer1.routing_table.buckets[0].peers.clear()
# peers who are known to be bad recently should not be added or maybe-pinged
peer1_from_peer4 = peer4.get_rpc_peer(peer4.peer_manager.get_kademlia_peer(node_id1, '1.2.3.4', 4444))
peer1_from_peer4 = peer4.get_rpc_peer(make_kademlia_peer(node_id1, '1.2.3.4', 4444))
peer1.peer_manager.update_contact_triple(node_id4,'1.2.3.7', 4444)
peer1.peer_manager.report_failure('1.2.3.7', 4444)
peer1.peer_manager.report_failure('1.2.3.7', 4444)
@@ -3,8 +3,7 @@ from torba.testcase import AsyncioTestCase
from tests import dht_mocks
from lbry.dht import constants
from lbry.dht.node import Node
from lbry.dht.peer import PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
expected_ranges = [
(
@@ -29,6 +28,7 @@ expected_ranges = [
)
]
class TestRouting(AsyncioTestCase):
async def test_fill_one_bucket(self):
loop = asyncio.get_event_loop()
@@ -53,7 +53,7 @@ class TestRouting(AsyncioTestCase):
for i in range(1, len(peer_addresses)):
self.assertEqual(len(node_1.protocol.routing_table.get_peers()), contact_cnt)
node = nodes[i]
peer = node_1.protocol.peer_manager.get_kademlia_peer(
peer = make_kademlia_peer(
node.protocol.node_id, node.protocol.external_ip,
udp_port=node.protocol.udp_port
)
@@ -66,6 +66,16 @@ class TestRouting(AsyncioTestCase):
for node in nodes.values():
node.protocol.stop()
async def test_cant_add_peer_without_a_node_id_gracefully(self):
loop = asyncio.get_event_loop()
node = Node(loop, PeerManager(loop), constants.generate_id(), 4444, 4444, 3333, '1.2.3.4')
bad_peer = make_kademlia_peer(None, '1.2.3.4', 5555)
with self.assertLogs(level='WARNING') as logged:
self.assertFalse(await node.protocol._add_peer(bad_peer))
self.assertEqual(1, len(logged.output))
self.assertTrue(logged.output[0].endswith('Tried adding a peer with no node id!'))
async def test_split_buckets(self):
loop = asyncio.get_event_loop()
peer_addresses = [
@@ -81,7 +91,7 @@ class TestRouting(AsyncioTestCase):
node_1 = nodes[0]
for i in range(1, len(peer_addresses)):
node = nodes[i]
peer = node_1.protocol.peer_manager.get_kademlia_peer(
peer = make_kademlia_peer(
node.protocol.node_id, node.protocol.external_ip,
udp_port=node.protocol.udp_port
)
+5 -5
View File
@@ -8,7 +8,7 @@ from tests import dht_mocks
from lbry.conf import Config
from lbry.dht import constants
from lbry.dht.node import Node
from lbry.dht.peer import PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
from lbry.dht.blob_announcer import BlobAnnouncer
from lbry.extras.daemon.storage import SQLiteStorage
@@ -35,7 +35,7 @@ class TestBlobAnnouncer(AsyncioTestCase):
self.nodes.update({len(self.nodes): n})
if add_to_routing_table:
self.node.protocol.add_peer(
self.peer_manager.get_kademlia_peer(
make_kademlia_peer(
n.protocol.node_id, n.protocol.external_ip, n.protocol.udp_port
)
)
@@ -67,7 +67,7 @@ class TestBlobAnnouncer(AsyncioTestCase):
await self.add_peer(node_id, address, False)
last_node = self.nodes[len(self.nodes) - 1]
peer = last_node.protocol.get_rpc_peer(
last_node.protocol.peer_manager.get_kademlia_peer(
make_kademlia_peer(
previous_last_node.protocol.node_id, previous_last_node.protocol.external_ip,
previous_last_node.protocol.udp_port
)
@@ -129,12 +129,12 @@ class TestBlobAnnouncer(AsyncioTestCase):
announced_to = self.nodes[0]
for i in range(1, peer_count):
node = self.nodes[i]
kad_peer = announced_to.protocol.peer_manager.get_kademlia_peer(
kad_peer = make_kademlia_peer(
node.protocol.node_id, node.protocol.external_ip, node.protocol.udp_port
)
await announced_to.protocol._add_peer(kad_peer)
peer = node.protocol.get_rpc_peer(
node.protocol.peer_manager.get_kademlia_peer(
make_kademlia_peer(
announced_to.protocol.node_id,
announced_to.protocol.external_ip,
announced_to.protocol.udp_port
+2 -2
View File
@@ -4,7 +4,7 @@ from torba.testcase import AsyncioTestCase
from tests import dht_mocks
from lbry.dht import constants
from lbry.dht.node import Node
from lbry.dht.peer import PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
class TestNodePingQueueDiscover(AsyncioTestCase):
@@ -42,7 +42,7 @@ class TestNodePingQueueDiscover(AsyncioTestCase):
for i in range(1, len(peer_addresses)):
node = nodes[i]
assert node.protocol.node_id != node_1.protocol.node_id
peer = node_1.protocol.peer_manager.get_kademlia_peer(
peer = make_kademlia_peer(
node.protocol.node_id, node.protocol.external_ip, udp_port=node.protocol.udp_port
)
futs.append(node_1.protocol.get_rpc_peer(peer).ping())
+19 -9
View File
@@ -1,7 +1,7 @@
import asyncio
import unittest
from lbry.utils import generate_id
from lbry.dht.peer import PeerManager
from lbry.dht.peer import PeerManager, make_kademlia_peer
from torba.testcase import AsyncioTestCase
@@ -10,20 +10,30 @@ class PeerTest(AsyncioTestCase):
self.loop = asyncio.get_event_loop()
self.peer_manager = PeerManager(self.loop)
self.node_ids = [generate_id(), generate_id(), generate_id()]
self.first_contact = self.peer_manager.get_kademlia_peer(self.node_ids[1], '127.0.0.1', udp_port=1000)
self.second_contact = self.peer_manager.get_kademlia_peer(self.node_ids[0], '192.168.0.1', udp_port=1000)
self.first_contact = make_kademlia_peer(self.node_ids[1], '127.0.0.1', udp_port=1000)
self.second_contact = make_kademlia_peer(self.node_ids[0], '192.168.0.1', udp_port=1000)
def test_peer_is_good_unknown_peer(self):
# Scenario: peer replied, but caller doesn't know the node_id.
# Outcome: We can't say it's good or bad.
# (yes, we COULD tell the node id, but not here. It would be
# a side effect and the caller is responsible to discover it)
peer = make_kademlia_peer(None, '1.2.3.4', 4444)
self.peer_manager.report_last_requested('1.2.3.4', 4444)
self.peer_manager.report_last_replied('1.2.3.4', 4444)
self.assertIsNone(self.peer_manager.peer_is_good(peer))
def test_make_contact_error_cases(self):
self.assertRaises(ValueError, self.peer_manager.get_kademlia_peer, self.node_ids[1], '192.168.1.20', 100000)
self.assertRaises(ValueError, self.peer_manager.get_kademlia_peer, self.node_ids[1], '192.168.1.20.1', 1000)
self.assertRaises(ValueError, self.peer_manager.get_kademlia_peer, self.node_ids[1], 'this is not an ip', 1000)
self.assertRaises(ValueError, self.peer_manager.get_kademlia_peer, self.node_ids[1], '192.168.1.20', -1000)
self.assertRaises(ValueError, self.peer_manager.get_kademlia_peer, b'not valid node id', '192.168.1.20', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.168.1.20', 100000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.168.1.20.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], 'this is not an ip', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.168.1.20', -1000)
self.assertRaises(ValueError, make_kademlia_peer, b'not valid node id', '192.168.1.20', 1000)
def test_boolean(self):
self.assertNotEqual(self.first_contact, self.second_contact)
self.assertEqual(
self.second_contact, self.peer_manager.get_kademlia_peer(self.node_ids[0], '192.168.0.1', udp_port=1000)
self.second_contact, make_kademlia_peer(self.node_ids[0], '192.168.0.1', udp_port=1000)
)
def test_compact_ip(self):
@@ -7,8 +7,7 @@ from lbry.blob.blob_file import MAX_BLOB_SIZE
from lbry.blob_exchange.serialization import BlobResponse
from lbry.blob_exchange.server import BlobServerProtocol
from lbry.dht.node import Node
from lbry.dht.peer import KademliaPeer
from lbry.extras.daemon.storage import StoredStreamClaim
from lbry.dht.peer import make_kademlia_peer
from lbry.stream.managed_stream import ManagedStream
from lbry.stream.descriptor import StreamDescriptor
from tests.unit.blob_exchange.test_transfer_blob import BlobExchangeTestBase
@@ -95,7 +94,7 @@ class TestManagedStream(BlobExchangeTestBase):
mock_node = mock.Mock(spec=Node)
q = asyncio.Queue()
bad_peer = KademliaPeer(self.loop, "127.0.0.1", b'2' * 48, tcp_port=3334)
bad_peer = make_kademlia_peer(b'2' * 48, "127.0.0.1", tcp_port=3334)
def _mock_accumulate_peers(q1, q2):
async def _task():
@@ -126,5 +126,5 @@ class TestRecoverOldStreamDescriptors(AsyncioTestCase):
loop, tmp_dir, blob
)
self.assertFalse(blob.file_exists)
# fixme: this is an emergency PR, plase move this to blob_file tests later
# fixme: this is an emergency PR, please move this to blob_file tests later
self.assertIsNone(blob.length)
@@ -100,7 +100,7 @@ class ReconnectTests(IntegrationTestCase):
# * goes to pick some water outside... * time passes by and another donation comes in
sendtxid = await self.blockchain.send_to_address(address1, 42)
await self.blockchain.generate(1)
# (this is just so the test doesnt hang forever if it doesnt reconnect)
# (this is just so the test doesn't hang forever if it doesn't reconnect)
if not self.ledger.network.is_connected:
await asyncio.wait_for(self.ledger.network.on_connected.first, timeout=1.0)
# omg, the burned cable still works! torba is fire proof!
+1 -1
View File
@@ -544,7 +544,7 @@ class BaseLedger(metaclass=LedgerRegistry):
return None
def broadcast(self, tx):
# broadcast cant be a retriable call yet
# broadcast can't be a retriable call yet
return self.network.broadcast(hexlify(tx.raw).decode())
async def wait(self, tx: basetransaction.BaseTransaction, height=-1, timeout=None):
+2 -2
View File
@@ -73,7 +73,7 @@ class ClientSession(BaseClientSession):
return reply
except (RPCError, ProtocolError) as e:
if str(e).find('.*no such .*transaction.*'):
# shouldnt the server return none instead?
# shouldn't the server return none instead?
return None
log.warning("Wallet server (%s:%i) returned an error. Code: %s Message: %s",
*self.server, *e.args)
@@ -256,7 +256,7 @@ class BaseNetwork:
try:
return await self.rpc('blockchain.address.subscribe', [address], True)
except asyncio.TimeoutError:
# abort and cancel, we cant lose a subscription, it will happen again on reconnect
# abort and cancel, we can't lose a subscription, it will happen again on reconnect
if self.client:
self.client.abort()
raise asyncio.CancelledError()
+1 -1
View File
@@ -37,7 +37,7 @@ class FramerBase(object):
"""Abstract base class for a framer.
A framer breaks an incoming byte stream into protocol messages,
buffering if necesary. It also frames outgoing messages into
buffering if necessary. It also frames outgoing messages into
a byte stream.
"""
+1 -1
View File
@@ -513,7 +513,7 @@ class JSONRPCv2(JSONRPC):
class JSONRPCLoose(JSONRPC):
"""A relaxed versin of JSON RPC."""
"""A relaxed version of JSON RPC."""
# Don't be so loose we accept any old message ID
_message_id = JSONRPCv2._message_id
+1 -1
View File
@@ -93,7 +93,7 @@ class SessionBase(asyncio.Protocol):
# Set when a connection is made
self._address = None
self._proxy_address = None
# For logger.debug messsages
# For logger.debug messages
self.verbosity = 0
# Cleared when the send socket is full
self._can_send = Event()
+1 -1
View File
@@ -117,7 +117,7 @@ class Daemon:
async def _send(self, payload, processor):
"""Send a payload to be converted to JSON.
Handles temporary connection issues. Daemon reponse errors
Handles temporary connection issues. Daemon response errors
are raise through DaemonError.
"""
def log_error(error):
+1 -1
View File
@@ -228,7 +228,7 @@ class History:
def _compact_hashX(self, hashX, hist_map, hist_list,
write_items, keys_to_delete):
"""Compres history for a hashX. hist_list is an ordered list of
"""Compress history for a hashX. hist_list is an ordered list of
the histories to be compressed."""
# History entries (tx numbers) are 4 bytes each. Distribute
# over rows of up to 50KB in size. A fixed row size means
+3 -3
View File
@@ -385,7 +385,7 @@ class SessionManager:
return "peer '{}' added".format(real_name)
async def rpc_disconnect(self, session_ids):
"""Disconnect sesssions.
"""Disconnect sessions.
session_ids: array of session IDs
"""
@@ -397,7 +397,7 @@ class SessionManager:
return await self._for_each_session(session_ids, close)
async def rpc_log(self, session_ids):
"""Toggle logging of sesssions.
"""Toggle logging of sessions.
session_ids: array of session IDs
"""
@@ -414,7 +414,7 @@ class SessionManager:
try:
self.daemon.set_url(daemon_url)
except Exception as e:
raise RPCError(BAD_REQUEST, f'an error occured: {e!r}')
raise RPCError(BAD_REQUEST, f'an error occurred: {e!r}')
return f'now using daemon at {self.daemon.logged_url()}'
async def rpc_stop(self):
+1 -1
View File
@@ -457,7 +457,7 @@ class TxInputTokenPay(TxInput):
self.script[1] == self.OP_ANON_MARKER)
def is_generation(self):
# Transactions comming in from stealth addresses are seen by
# Transactions coming in from stealth addresses are seen by
# the blockchain as newly minted coins. The reverse, where coins
# are sent TO a stealth address, are seen by the blockchain as
# a coin burn.