Compare commits

..
Author SHA1 Message Date
zeppi b21c71a420 cleanup
ci / lint (push) Waiting to run
ci / tests / unit (macos-latest) (push) Waiting to run
ci / tests / unit (ubuntu-latest) (push) Waiting to run
ci / tests / unit (windows-latest) (push) Waiting to run
ci / tests / integration (blockchain) (push) Waiting to run
ci / tests / integration (claims) (push) Waiting to run
ci / tests / integration (claims_legacy_search) (push) Waiting to run
ci / tests / integration (datanetwork) (push) Waiting to run
ci / tests / integration (other) (push) Waiting to run
ci / tests / integration (takeovers) (push) Waiting to run
ci / tests / integration (takeovers_legacy_search) (push) Waiting to run
ci / tests / integration (transactions) (push) Waiting to run
2022-03-24 19:05:24 -04:00
zeppi c8aa73a5d6 wip 2022-03-24 18:41:29 -04:00
zeppi a24fbf81d7 es should fix 2022-03-24 18:07:22 -04:00
Lex Berezhny ad489ed606 Merge pull request #3581 from lbryio/deterministic_channel_keys_post_unlock
eagerly load deterministic channel keys immediately after wallet is unlocked
2022-03-14 12:36:04 -04:00
Lex Berezhny bb541901d9 fix tests 2022-03-13 21:30:38 -04:00
Lex Berezhny ca4ba19a5e fixes #3577 2022-03-13 20:42:34 -04:00
Victor Shyba f05943ff79 implement announcer as a consumer task on gather 2022-03-02 13:00:34 -03:00
Victor Shyba 7ded8a1333 make active an explicit ordered dict 2022-03-02 13:00:34 -03:00
Victor Shyba c2478d4add remove unused search rounds 2022-03-02 13:00:34 -03:00
Victor Shyba f69747bc89 timeout is now supported on dht tests 2022-03-02 13:00:34 -03:00
Victor Shyba 441cc950aa fix and enable test_blob_announcer 2022-03-02 13:00:34 -03:00
Victor Shyba a76a0ac8c4 simplify dht mock and restore clock after accelerating 2022-03-02 13:00:34 -03:00
Victor Shyba 8b1009161a better representation of kademliapeer on debug logs 2022-03-02 13:00:34 -03:00
Victor Shyba 868a620e91 add a way to wait announcements to finish so tests are reliable 2022-03-02 13:00:34 -03:00
Victor Shyba a0e34b0bc8 make timeout handler immune to asyncio time tricks 2022-03-02 13:00:34 -03:00
Victor Shyba 612dbcb2f3 allow running some extra probes for k replacements 2022-03-02 13:00:34 -03:00
Victor Shyba b3614d965d remove all references to bottoming out 2022-03-02 13:00:34 -03:00
Victor Shyba 5d7137255e no stop condition, let it exhaust 2022-03-02 13:00:34 -03:00
Victor Shyba 6ff867ef55 bottoming out is now warning and no results for peer search 2022-03-02 13:00:34 -03:00
Victor Shyba c14915df29 don't probe peers too far from the top closest 2022-03-02 13:00:34 -03:00
Victor Shyba 7d4966e2ae use a dict for the active queue 2022-03-02 13:00:34 -03:00
Victor Shyba 3876e0317d log bottom out of peer search in debug, show short key id for find value 2022-03-02 13:00:34 -03:00
Victor Shyba 0b2b10f759 bump bottom out limit of peer search so people can use 100 concurrent announcers 2022-03-02 13:00:34 -03:00
Victor Shyba 9a79b33664 wait until k peers are ready. do not double add peers 2022-03-02 13:00:34 -03:00
Victor Shyba af1a6edd15 only return good (contacted) peers 2022-03-02 13:00:34 -03:00
Victor Shyba b78929f4d5 reset closest peer on failure 2022-03-02 13:00:34 -03:00
Victor Shyba fb6e342043 add peers from shortlist regardless, but check from other nodes 2022-03-02 13:00:34 -03:00
Victor Shyba 0faa2d35da bump split index to 2 2022-03-02 13:00:34 -03:00
Victor Shyba 511e57c231 fix distance sorting and improve logging 2022-03-02 13:00:34 -03:00
Victor Shyba d762d675c4 closest peer is only ready when it was contacted and isn't known to be bad 2022-03-02 13:00:34 -03:00
Victor Shyba 3fdadee87c dont probe and ignore bad peers 2022-03-02 13:00:34 -03:00
Victor Shyba 1aa4d9d585 simplify, genaralize to any size and fix tests 2022-02-28 13:06:51 -03:00
Victor Shyba 8019f4bdb3 stop after finding what to download 2022-02-28 13:06:51 -03:00
Victor Shyba ca65c1ebc5 replace duplicated code 2022-02-28 13:06:51 -03:00
Victor Shyba f0e47aae86 add get_colliding_prefix_bits, docs and tests 2022-02-28 13:06:51 -03:00
Victor Shyba dc7cd545ba extract method and avoid using hash builtin name 2022-02-28 13:06:51 -03:00
Victor Shyba 76bd59d82e extract min_prefix_colliding_bits to a contanst 2022-02-28 13:06:51 -03:00
Victor Shyba 461687ffb4 check that the stored blob is at least 1 prefix byte close to peer id 2022-02-28 13:06:51 -03:00
Victor Shyba dd5b9ca81b add migrator to set head blobs should_announce=0 2022-02-20 22:33:57 -03:00
Victor Shyba 89ed04f8a7 fix test_announces 2022-02-20 22:33:57 -03:00
Victor Shyba ec0d9f06c5 do not search for the head blob 2022-02-20 22:33:57 -03:00
Victor Shyba 03b59ac6fc dont set head blob to announce on save 2022-02-20 22:33:57 -03:00
Victor Shyba 43ac3336d7 break tie by length 2022-02-20 22:24:04 -03:00
Victor Shyba d12c78db74 fix and test case for blob_clean after disabling network storage 2022-02-20 22:24:04 -03:00
Jack Robison bfaf1b0957 Merge pull request #3564 from lbryio/fix_downloader_losing_peers
fix handling re-adding lost peers during download
2022-02-16 11:55:22 -05:00
Victor Shyba bb60c385d5 put back all the peers, get rid of re_add 2022-02-08 21:41:52 -03:00
Alex Grin c96d1d9c32 Merge pull request #3537 from lbryio/repost_update 2022-02-08 12:20:20 -05:00
Alex Grintsvayg 7c7a0d4bdf let stream_update work on non-stream claims 2022-02-08 09:28:17 -05:00
Lex Berezhny cc829a7bf4 Merge pull request #3558 from lbryio/jeffreypicard-patch-1
Update __init__.py
2022-02-04 12:36:01 -05:00
Jeffrey Picard e0ea6383e2 Update __init__.py
Update go hub binary to fix es sync test.
2022-02-04 12:17:19 -05:00
Lex Berezhny bcec5dc2ae Merge pull request #3556 from lbryio/txo_dust_prevention
prevent creation of change which is below the dust threshold of 1000 dewies
2022-02-04 12:08:16 -05:00
Lex Berezhny cba9c16a06 fix 2022-02-04 12:07:41 -05:00
Lex Berezhny dd68fb077b prevent creation of change which is below the dust threshold of 1000 dewies 2022-02-04 12:07:41 -05:00
Jack Robison c2294e97db Merge pull request #3552 from lbryio/bump_dht_cache
Increase DHT peer manager cache size to 16384
2022-02-04 11:59:19 -05:00
Victor Shyba c0f512ace7 bump DHT peer manager cache to 16384 2022-02-02 16:54:42 -03:00
Lex Berezhny 3305eb67c6 Merge pull request #3548 from lbryio/announce_metrics
Add optional Prometheus metrics for DHT announcements
2022-02-02 11:06:48 -05:00
Victor Shyba c9d637b4da add gauge for queue size 2022-02-02 11:56:42 -03:00
Victor Shyba ae3e8fadf5 count announcements and how many peers we were able to announce to 2022-02-02 11:56:42 -03:00
Lex Berezhny a1abd94387 Merge pull request #3542 from eug3nix/gh_3481_file_type_detection
file type detection now looks inside the file to determine the type, in addition to using the file extension
2022-01-31 10:29:47 -05:00
Eugene Dubinin 9b463a8cab adds tests for guess_media_type
removes unnecessary comments
2022-01-29 20:49:42 +02:00
Eugene Dubinin babc54a240 adjusts code style 2022-01-29 15:25:17 +02:00
Eugene Dubinin 5836a93b21 fixes KeyError on missing synonyms 2022-01-29 15:25:17 +02:00
Eugene Dubinin 557348e345 detect media_type from the file contents 2022-01-29 15:25:17 +02:00
Lex Berezhny 9adfec6b00 Merge pull request #3549 from lbryio/wallet_lock_w_deterministic_channels
wallet locking/unlocking no longer breaks deterministic channel keys
2022-01-26 11:17:55 -05:00
Lex Berezhny 3a496902f8 wallet locking/unlocking no longer breaks deterministic channel keys 2022-01-24 09:45:08 -05:00
56 changed files with 865 additions and 1057 deletions

No files matched your search

+1
View File
@@ -15,6 +15,7 @@ RUN apt-get update && \
build-essential \
automake libtool \
pkg-config \
libleveldb-dev \
python3.7 \
python3-dev \
python3-pip \
-58
View File
@@ -1,58 +0,0 @@
# FROM debian:10-slim
FROM python:3.7.12-slim-buster
ARG user=lbry
ARG db_dir=/database
ARG projects_dir=/home/$user
ARG DOCKER_TAG
ARG DOCKER_COMMIT=docker
ENV DOCKER_TAG=$DOCKER_TAG DOCKER_COMMIT=$DOCKER_COMMIT
RUN apt-get update && \
apt-get -y --no-install-recommends install \
wget \
tar unzip \
build-essential \
automake libtool \
pkg-config \
librocksdb-dev
# python3.7 \
# python3-dev \
# python3-pip \
# python3-wheel \
# python3-cffi \
# python3-setuptools && \
# update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1 && \
# rm -rf /var/lib/apt/lists/*
RUN pip install lbry-rocksdb
RUN pip install uvloop
RUN groupadd -g 999 $user && useradd -m -u 999 -g $user $user
RUN mkdir -p $db_dir
RUN chown -R $user:$user $db_dir
COPY . $projects_dir
RUN chown -R $user:$user $projects_dir
USER $user
WORKDIR $projects_dir
RUN make install
RUN python3 docker/set_build.py
RUN rm ~/.cache -rf
# entry point
ARG host=0.0.0.0
ARG tcp_port=50001
ARG daemon_url=http://lbry:lbry@192.99.151.178:9245/
VOLUME $db_dir
ENV TCP_PORT=$tcp_port
ENV HOST=$host
ENV DAEMON_URL=$daemon_url
ENV DB_DIRECTORY=$db_dir
ENV MAX_SESSIONS=1000000000
ENV MAX_SEND=1000000000000000000
ENV EVENT_LOOP_POLICY=uvloop
COPY ./docker/wallet_server_entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
+9 -11
View File
@@ -36,31 +36,29 @@ class DiskSpaceManager:
await self._clean(True)
async def _clean(self, is_network_blob=False):
space_used_bytes = await self.get_space_used_bytes()
space_used_mb = await self.get_space_used_mb(cached=False)
if is_network_blob:
space_used_bytes = space_used_bytes['network_storage']
space_used_mb = space_used_mb['network_storage']
else:
space_used_bytes = space_used_bytes['content_storage'] + space_used_bytes['private_storage']
space_used_mb = space_used_mb['content_storage'] + space_used_mb['private_storage']
storage_limit_mb = self.config.network_storage_limit if is_network_blob else self.config.blob_storage_limit
storage_limit = storage_limit_mb*1024*1024 if storage_limit_mb else None
if self.analytics:
asyncio.create_task(
self.analytics.send_disk_space_used(space_used_bytes, storage_limit, is_network_blob)
self.analytics.send_disk_space_used(space_used_mb, storage_limit_mb, is_network_blob)
)
if not storage_limit:
return 0
delete = []
available = storage_limit - space_used_bytes
if available > 0:
available = storage_limit_mb - space_used_mb
if storage_limit_mb == 0 if not is_network_blob else available >= 0:
return 0
for blob_hash, file_size, _ in await self.db.get_stored_blobs(is_mine=False, is_network_blob=is_network_blob):
delete.append(blob_hash)
available += file_size
if available > 0:
available += int(file_size/1024.0/1024.0)
if available >= 0:
break
if delete:
await self.db.stop_all_files()
await self.blob_manager.delete_blobs(delete, delete_from_db=True)
self._used_space_bytes = None
return len(delete)
async def cleaning_loop(self):
+2 -8
View File
@@ -97,21 +97,15 @@ class BlobDownloader:
"%s running, %d peers, %d ignored, %d active, %s connections", blob_hash[:6],
len(batch), len(self.ignored), len(self.active_connections), len(self.connections)
)
re_add: typing.Set['KademliaPeer'] = set()
for peer in sorted(batch, key=lambda peer: self.scores.get(peer, 0), reverse=True):
if peer in self.ignored:
continue
if peer in self.active_connections:
if peer not in re_add:
re_add.add(peer)
if peer in self.active_connections or not self.should_race_continue(blob):
continue
if not self.should_race_continue(blob):
break
log.debug("request %s from %s:%i", blob_hash[:8], peer.address, peer.tcp_port)
t = self.loop.create_task(self.request_blob_from_peer(blob, peer, connection_id))
self.active_connections[peer] = t
if not re_add:
self.peer_queue.put_nowait(list(batch))
self.peer_queue.put_nowait(list(batch))
await self.new_peer_or_finished()
self.cleanup_active()
log.debug("downloaded %s", blob_hash[:8])
+1 -1
View File
@@ -622,7 +622,7 @@ class Config(CLIConfig):
"Routing table bucket index below which we always split the bucket if given a new key to add to it and "
"the bucket is full. As this value is raised the depth of the routing table (and number of peers in it) "
"will increase. This setting is used by seed nodes, you probably don't want to change it during normal "
"use.", 1
"use.", 2
)
# protocol timeouts
+38 -16
View File
@@ -1,6 +1,9 @@
import asyncio
import typing
import logging
from prometheus_client import Counter, Gauge
if typing.TYPE_CHECKING:
from lbry.dht.node import Node
from lbry.extras.daemon.storage import SQLiteStorage
@@ -9,24 +12,39 @@ log = logging.getLogger(__name__)
class BlobAnnouncer:
announcements_sent_metric = Counter(
"announcements_sent", "Number of announcements sent and their respective status.", namespace="dht_node",
labelnames=("peers", "error"),
)
announcement_queue_size_metric = Gauge(
"announcement_queue_size", "Number of hashes waiting to be announced.", namespace="dht_node",
labelnames=("scope",)
)
def __init__(self, loop: asyncio.AbstractEventLoop, node: 'Node', storage: 'SQLiteStorage'):
self.loop = loop
self.node = node
self.storage = storage
self.announce_task: asyncio.Task = None
self.announce_queue: typing.List[str] = []
self._done = asyncio.Event()
self.announced = set()
async def _submit_announcement(self, blob_hash):
try:
peers = len(await self.node.announce_blob(blob_hash))
if peers > 4:
return blob_hash
else:
log.debug("failed to announce %s, could only find %d peers, retrying soon.", blob_hash[:8], peers)
except Exception as err:
if isinstance(err, asyncio.CancelledError): # TODO: remove when updated to 3.8
raise err
log.warning("error announcing %s: %s", blob_hash[:8], str(err))
async def _run_consumer(self):
while self.announce_queue:
try:
blob_hash = self.announce_queue.pop()
peers = len(await self.node.announce_blob(blob_hash))
self.announcements_sent_metric.labels(peers=peers, error=False).inc()
if peers > 4:
self.announced.add(blob_hash)
else:
log.debug("failed to announce %s, could only find %d peers, retrying soon.", blob_hash[:8], peers)
except Exception as err:
self.announcements_sent_metric.labels(peers=0, error=True).inc()
if isinstance(err, asyncio.CancelledError): # TODO: remove when updated to 3.8
raise err
log.warning("error announcing %s: %s", blob_hash[:8], str(err))
async def _announce(self, batch_size: typing.Optional[int] = 10):
while batch_size:
@@ -37,17 +55,18 @@ class BlobAnnouncer:
log.warning("No peers in DHT, announce round skipped")
continue
self.announce_queue.extend(await self.storage.get_blobs_to_announce())
self.announcement_queue_size_metric.labels(scope="global").set(len(self.announce_queue))
log.debug("announcer task wake up, %d blobs to announce", len(self.announce_queue))
while len(self.announce_queue) > 0:
log.info("%i blobs to announce", len(self.announce_queue))
announced = await asyncio.gather(*[
self._submit_announcement(
self.announce_queue.pop()) for _ in range(batch_size) if self.announce_queue
], loop=self.loop)
announced = list(filter(None, announced))
await asyncio.gather(*[self._run_consumer() for _ in range(batch_size)], loop=self.loop)
announced = list(filter(None, self.announced))
if announced:
await self.storage.update_last_announced_blobs(announced)
log.info("announced %i blobs", len(announced))
self.announced.clear()
self._done.set()
self._done.clear()
def start(self, batch_size: typing.Optional[int] = 10):
assert not self.announce_task or self.announce_task.done(), "already running"
@@ -56,3 +75,6 @@ class BlobAnnouncer:
def stop(self):
if self.announce_task and not self.announce_task.done():
self.announce_task.cancel()
def wait(self):
return self._done.wait()
-1
View File
@@ -20,7 +20,6 @@ MAYBE_PING_DELAY = 300 # 5 minutes
CHECK_REFRESH_INTERVAL = REFRESH_INTERVAL / 5
RPC_ID_LENGTH = 20
PROTOCOL_VERSION = 1
BOTTOM_OUT_LIMIT = 3
MSG_SIZE_LIMIT = 1400
+4 -6
View File
@@ -202,25 +202,23 @@ class Node:
self._join_task = self.loop.create_task(self.join_network(interface, known_node_urls))
def get_iterative_node_finder(self, key: bytes, shortlist: typing.Optional[typing.List['KademliaPeer']] = None,
bottom_out_limit: int = constants.BOTTOM_OUT_LIMIT,
max_results: int = constants.K) -> IterativeNodeFinder:
return IterativeNodeFinder(self.loop, self.protocol.peer_manager, self.protocol.routing_table, self.protocol,
key, bottom_out_limit, max_results, None, shortlist)
key, max_results, None, shortlist)
def get_iterative_value_finder(self, key: bytes, shortlist: typing.Optional[typing.List['KademliaPeer']] = None,
bottom_out_limit: int = 40,
max_results: int = -1) -> IterativeValueFinder:
return IterativeValueFinder(self.loop, self.protocol.peer_manager, self.protocol.routing_table, self.protocol,
key, bottom_out_limit, max_results, None, shortlist)
key, max_results, None, shortlist)
async def peer_search(self, node_id: bytes, count=constants.K, max_results=constants.K * 2,
bottom_out_limit=20, shortlist: typing.Optional[typing.List['KademliaPeer']] = None
shortlist: typing.Optional[typing.List['KademliaPeer']] = None
) -> typing.List['KademliaPeer']:
peers = []
async for iteration_peers in self.get_iterative_node_finder(
node_id, shortlist=shortlist, bottom_out_limit=bottom_out_limit, max_results=max_results):
node_id, shortlist=shortlist, max_results=max_results):
peers.extend(iteration_peers)
distance = Distance(node_id)
peers.sort(key=lambda peer: distance(peer.node_id))
+4 -1
View File
@@ -11,7 +11,7 @@ from lbry.dht import constants
from lbry.dht.serialization.datagram import make_compact_address, make_compact_ip, decode_compact_address
ALLOW_LOCALHOST = False
CACHE_SIZE = 2048
CACHE_SIZE = 16384
log = logging.getLogger(__name__)
@@ -190,3 +190,6 @@ class KademliaPeer:
def compact_ip(self):
return make_compact_ip(self.address)
def __str__(self):
return f"{self.__class__.__name__}({self.node_id.hex()[:8]}@{self.address}:{self.udp_port}-{self.tcp_port})"
+46 -76
View File
@@ -1,6 +1,6 @@
import asyncio
from itertools import chain
from collections import defaultdict
from collections import defaultdict, OrderedDict
import typing
import logging
from typing import TYPE_CHECKING
@@ -74,7 +74,7 @@ def get_shortlist(routing_table: 'TreeRoutingTable', key: bytes,
class IterativeFinder:
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
bottom_out_limit: typing.Optional[int] = 2, max_results: typing.Optional[int] = constants.K,
max_results: typing.Optional[int] = constants.K,
exclude: typing.Optional[typing.List[typing.Tuple[str, int]]] = None,
shortlist: typing.Optional[typing.List['KademliaPeer']] = None):
if len(key) != constants.HASH_LENGTH:
@@ -85,28 +85,22 @@ class IterativeFinder:
self.protocol = protocol
self.key = key
self.bottom_out_limit = bottom_out_limit
self.max_results = max_results
self.max_results = max(constants.K, max_results)
self.exclude = exclude or []
self.active: typing.Set['KademliaPeer'] = set()
self.active: typing.Dict['KademliaPeer', int] = OrderedDict() # peer: distance, sorted
self.contacted: typing.Set['KademliaPeer'] = set()
self.distance = Distance(key)
self.closest_peer: typing.Optional['KademliaPeer'] = None
self.prev_closest_peer: typing.Optional['KademliaPeer'] = None
self.iteration_queue = asyncio.Queue(loop=self.loop)
self.running_probes: typing.Set[asyncio.Task] = set()
self.running_probes: typing.Dict['KademliaPeer', asyncio.Task] = {}
self.iteration_count = 0
self.bottom_out_count = 0
self.running = False
self.tasks: typing.List[asyncio.Task] = []
self.delayed_calls: typing.List[asyncio.Handle] = []
for peer in get_shortlist(routing_table, key, shortlist):
if peer.node_id:
self._add_active(peer)
self._add_active(peer, force=True)
else:
# seed nodes
self._schedule_probe(peer)
@@ -138,15 +132,14 @@ class IterativeFinder:
"""
return []
def _is_closer(self, peer: 'KademliaPeer') -> bool:
return not self.closest_peer or self.distance.is_closer(peer.node_id, self.closest_peer.node_id)
def _add_active(self, peer):
def _add_active(self, peer, force=False):
if not force and self.peer_manager.peer_is_good(peer) is False:
return
if peer in self.contacted:
return
if peer not in self.active and peer.node_id and peer.node_id != self.protocol.node_id:
self.active.add(peer)
if self._is_closer(peer):
self.prev_closest_peer = self.closest_peer
self.closest_peer = peer
self.active[peer] = self.distance(peer.node_id)
self.active = OrderedDict(sorted(self.active.items(), key=lambda item: item[1]))
async def _handle_probe_result(self, peer: 'KademliaPeer', response: FindResponse):
self._add_active(peer)
@@ -158,33 +151,43 @@ class IterativeFinder:
log.warning("misbehaving peer %s:%i returned peer with reserved ip %s:%i", peer.address,
peer.udp_port, address, udp_port)
self.check_result_ready(response)
self._log_state()
def _reset_closest(self, peer):
if peer in self.active:
del self.active[peer]
async def _send_probe(self, peer: 'KademliaPeer'):
try:
response = await self.send_probe(peer)
except asyncio.TimeoutError:
self.active.discard(peer)
self._reset_closest(peer)
return
except ValueError as err:
log.warning(str(err))
self.active.discard(peer)
self._reset_closest(peer)
return
except TransportNotConnected:
return self.aclose()
except RemoteException:
self._reset_closest(peer)
return
return await self._handle_probe_result(peer, response)
async def _search_round(self):
def _search_round(self):
"""
Send up to constants.alpha (5) probes to closest active peers
"""
added = 0
to_probe = list(self.active - self.contacted)
to_probe.sort(key=lambda peer: self.distance(self.key))
for peer in to_probe:
if added >= constants.ALPHA:
for index, peer in enumerate(self.active.keys()):
if index == 0:
log.debug("closest to probe: %s", peer.node_id.hex()[:8])
if peer in self.contacted:
continue
if len(self.running_probes) >= constants.ALPHA:
break
if index > (constants.K + len(self.running_probes)):
break
origin_address = (peer.address, peer.udp_port)
if origin_address in self.exclude:
@@ -206,33 +209,22 @@ class IterativeFinder:
t = self.loop.create_task(self._send_probe(peer))
def callback(_):
self.running_probes.difference_update({
probe for probe in self.running_probes if probe.done() or probe == t
})
if not self.running_probes:
self.tasks.append(self.loop.create_task(self._search_task(0.0)))
self.running_probes.pop(peer, None)
if self.running:
self._search_round()
t.add_done_callback(callback)
self.running_probes.add(t)
self.running_probes[peer] = t
async def _search_task(self, delay: typing.Optional[float] = constants.ITERATIVE_LOOKUP_DELAY):
try:
if self.running:
await self._search_round()
if self.running:
self.delayed_calls.append(self.loop.call_later(delay, self._search))
except (asyncio.CancelledError, StopAsyncIteration, TransportNotConnected):
if self.running:
self.loop.call_soon(self.aclose)
def _search(self):
self.tasks.append(self.loop.create_task(self._search_task()))
def _log_state(self):
log.debug("[%s] check result: %i active nodes %i contacted",
self.key.hex()[:8], len(self.active), len(self.contacted))
def __aiter__(self):
if self.running:
raise Exception("already running")
self.running = True
self._search()
self.loop.call_soon(self._search_round)
return self
async def __anext__(self) -> typing.List['KademliaPeer']:
@@ -252,20 +244,19 @@ class IterativeFinder:
def aclose(self):
self.running = False
self.iteration_queue.put_nowait(None)
for task in chain(self.tasks, self.running_probes, self.delayed_calls):
for task in chain(self.tasks, self.running_probes.values()):
task.cancel()
self.tasks.clear()
self.running_probes.clear()
self.delayed_calls.clear()
class IterativeNodeFinder(IterativeFinder):
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
bottom_out_limit: typing.Optional[int] = 2, max_results: typing.Optional[int] = constants.K,
max_results: typing.Optional[int] = constants.K,
exclude: typing.Optional[typing.List[typing.Tuple[str, int]]] = None,
shortlist: typing.Optional[typing.List['KademliaPeer']] = None):
super().__init__(loop, peer_manager, routing_table, protocol, key, bottom_out_limit, max_results, exclude,
super().__init__(loop, peer_manager, routing_table, protocol, key, max_results, exclude,
shortlist)
self.yielded_peers: typing.Set['KademliaPeer'] = set()
@@ -276,14 +267,14 @@ class IterativeNodeFinder(IterativeFinder):
return FindNodeResponse(self.key, response)
def search_exhausted(self):
self.put_result(self.active, finish=True)
self.put_result(self.active.keys(), finish=True)
def put_result(self, from_iter: typing.Iterable['KademliaPeer'], finish=False):
not_yet_yielded = [
peer for peer in from_iter
if peer not in self.yielded_peers
and peer.node_id != self.protocol.node_id
and self.peer_manager.peer_is_good(peer) is not False
and self.peer_manager.peer_is_good(peer) is True # return only peers who answered
]
not_yet_yielded.sort(key=lambda peer: self.distance(peer.node_id))
to_yield = not_yet_yielded[:max(constants.K, self.max_results)]
@@ -298,26 +289,16 @@ class IterativeNodeFinder(IterativeFinder):
if found:
log.debug("found")
return self.put_result(self.active, finish=True)
if self.prev_closest_peer and self.closest_peer and not self._is_closer(self.prev_closest_peer):
# log.info("improving, %i %i %i %i %i", len(self.shortlist), len(self.active), len(self.contacted),
# self.bottom_out_count, self.iteration_count)
self.bottom_out_count = 0
elif self.prev_closest_peer and self.closest_peer:
self.bottom_out_count += 1
log.info("bottom out %i %i %i", len(self.active), len(self.contacted), self.bottom_out_count)
if self.bottom_out_count >= self.bottom_out_limit or self.iteration_count >= self.bottom_out_limit:
log.info("limit hit")
self.put_result(self.active, True)
return self.put_result(self.active.keys(), finish=True)
class IterativeValueFinder(IterativeFinder):
def __init__(self, loop: asyncio.AbstractEventLoop, peer_manager: 'PeerManager',
routing_table: 'TreeRoutingTable', protocol: 'KademliaProtocol', key: bytes,
bottom_out_limit: typing.Optional[int] = 2, max_results: typing.Optional[int] = constants.K,
max_results: typing.Optional[int] = constants.K,
exclude: typing.Optional[typing.List[typing.Tuple[str, int]]] = None,
shortlist: typing.Optional[typing.List['KademliaPeer']] = None):
super().__init__(loop, peer_manager, routing_table, protocol, key, bottom_out_limit, max_results, exclude,
super().__init__(loop, peer_manager, routing_table, protocol, key, max_results, exclude,
shortlist)
self.blob_peers: typing.Set['KademliaPeer'] = set()
# this tracks the index of the most recent page we requested from each peer
@@ -362,23 +343,12 @@ class IterativeValueFinder(IterativeFinder):
blob_peers = [self.peer_manager.decode_tcp_peer_from_compact_address(compact_addr)
for compact_addr in response.found_compact_addresses]
to_yield = []
self.bottom_out_count = 0
for blob_peer in blob_peers:
if blob_peer not in self.blob_peers:
self.blob_peers.add(blob_peer)
to_yield.append(blob_peer)
if to_yield:
# log.info("found %i new peers for blob", len(to_yield))
self.iteration_queue.put_nowait(to_yield)
# if self.max_results and len(self.blob_peers) >= self.max_results:
# log.info("enough blob peers found")
# if not self.finished.is_set():
# self.finished.set()
elif self.prev_closest_peer and self.closest_peer:
self.bottom_out_count += 1
if self.bottom_out_count >= self.bottom_out_limit:
log.info("blob peer search bottomed out")
self.iteration_queue.put_nowait(None)
def get_initial_result(self) -> typing.List['KademliaPeer']:
if self.protocol.data_store.has_peers_for_blob(self.key):
+5 -6
View File
@@ -6,6 +6,7 @@ import itertools
from prometheus_client import Gauge
from lbry import utils
from lbry.dht import constants
from lbry.dht.protocol.distance import Distance
if typing.TYPE_CHECKING:
@@ -70,9 +71,8 @@ class KBucket:
if len(self.peers) < constants.K:
self.peers.append(peer)
self.peer_in_routing_table_metric.labels("global").inc()
if peer.node_id[0] == self._node_id[0]:
bits_colliding = 8 - (peer.node_id[1] ^ self._node_id[1]).bit_length()
self.peer_with_x_bit_colliding_metric.labels(amount=(bits_colliding + 8)).inc()
bits_colliding = utils.get_colliding_prefix_bits(peer.node_id, self._node_id)
self.peer_with_x_bit_colliding_metric.labels(amount=bits_colliding).inc()
return True
else:
return False
@@ -140,9 +140,8 @@ class KBucket:
def remove_peer(self, peer: 'KademliaPeer') -> None:
self.peers.remove(peer)
self.peer_in_routing_table_metric.labels("global").dec()
if peer.node_id[0] == self._node_id[0]:
bits_colliding = 8 - (peer.node_id[1] ^ self._node_id[1]).bit_length()
self.peer_with_x_bit_colliding_metric.labels(amount=(bits_colliding + 8)).dec()
bits_colliding = utils.get_colliding_prefix_bits(peer.node_id, self._node_id)
self.peer_with_x_bit_colliding_metric.labels(amount=bits_colliding).dec()
def key_in_range(self, key: bytes) -> bool:
""" Tests whether the specified key (i.e. node ID) is in the range
+11 -4
View File
@@ -381,6 +381,7 @@ class FileManagerComponent(Component):
class BackgroundDownloaderComponent(Component):
MIN_PREFIX_COLLIDING_BITS = 8
component_name = BACKGROUND_DOWNLOADER_COMPONENT
depends_on = [DATABASE_COMPONENT, BLOB_COMPONENT, DISK_SPACE_COMPONENT]
@@ -412,12 +413,18 @@ class BackgroundDownloaderComponent(Component):
while True:
self.space_available = await self.space_manager.get_free_space_mb(True)
if not self.is_busy and self.space_available > 10:
blob_hash = next((key.hex() for key in self.dht_node.stored_blob_hashes if
key.hex() not in self.blob_manager.completed_blob_hashes), None)
if blob_hash:
self.ongoing_download = asyncio.create_task(self.background_downloader.download_blobs(blob_hash))
self._download_next_close_blob_hash()
await asyncio.sleep(self.download_loop_delay_seconds)
def _download_next_close_blob_hash(self):
node_id = self.dht_node.protocol.node_id
for blob_hash in self.dht_node.stored_blob_hashes:
if blob_hash.hex() in self.blob_manager.completed_blob_hashes:
continue
if utils.get_colliding_prefix_bits(node_id, blob_hash) >= self.MIN_PREFIX_COLLIDING_BITS:
self.ongoing_download = asyncio.create_task(self.background_downloader.download_blobs(blob_hash.hex()))
return
async def start(self):
self.space_manager: DiskSpaceManager = self.component_manager.get_component(DISK_SPACE_COMPONENT)
if not self.component_manager.has_component(DHT_COMPONENT):
+26 -30
View File
@@ -37,7 +37,8 @@ from lbry.dht.peer import make_kademlia_peer
from lbry.error import (
DownloadSDTimeoutError, ComponentsNotStartedError, ComponentStartConditionNotMetError,
CommandDoesNotExistError, BaseError, WalletNotFoundError, WalletAlreadyLoadedError, WalletAlreadyExistsError,
ConflictingInputValueError, AlreadyPurchasedError, PrivateKeyNotFoundError, InputStringIsBlankError
ConflictingInputValueError, AlreadyPurchasedError, PrivateKeyNotFoundError, InputStringIsBlankError,
InputValueError
)
from lbry.extras import system_info
from lbry.extras.daemon import analytics
@@ -3614,15 +3615,17 @@ class Daemon(metaclass=JSONRPCServerType):
)
if len(existing_claims) != 1:
account_ids = ', '.join(f"'{account.id}'" for account in accounts)
# TODO: use error from lbry.error
raise Exception(
raise InputValueError(
f"Can't find the stream '{claim_id}' in account(s) {account_ids}."
)
old_txo = existing_claims[0]
if not old_txo.claim.is_stream:
# TODO: use error from lbry.error
raise Exception(
f"A claim with id '{claim_id}' was found but it is not a stream claim."
if not old_txo.claim.is_stream and not old_txo.claim.is_repost:
# in principle it should work with any type of claim, but its safer to
# limit it to ones we know won't be broken. in the future we can expand
# this if we have a test case for e.g. channel or support claims
raise InputValueError(
f"A claim with id '{claim_id}' was found but it is not a stream or repost claim."
)
if bid is not None:
@@ -3653,28 +3656,32 @@ class Daemon(metaclass=JSONRPCServerType):
if replace:
claim = Claim()
if old_txo.claim.stream.has_source:
claim.stream.message.source.CopyFrom(
old_txo.claim.stream.message.source
)
stream_type = old_txo.claim.stream.stream_type
if stream_type:
old_stream_type = getattr(old_txo.claim.stream.message, stream_type)
new_stream_type = getattr(claim.stream.message, stream_type)
new_stream_type.CopyFrom(old_stream_type)
claim.stream.update(file_path=file_path, **kwargs)
if old_txo.claim.is_stream:
if old_txo.claim.stream.has_source:
claim.stream.message.source.CopyFrom(
old_txo.claim.stream.message.source
)
stream_type = old_txo.claim.stream.stream_type
if stream_type:
old_stream_type = getattr(old_txo.claim.stream.message, stream_type)
new_stream_type = getattr(claim.stream.message, stream_type)
new_stream_type.CopyFrom(old_stream_type)
else:
claim = Claim.from_bytes(old_txo.claim.to_bytes())
if old_txo.claim.is_stream:
claim.stream.update(file_path=file_path, **kwargs)
if clear_channel:
claim.clear_signature()
tx = await Transaction.claim_update(
old_txo, claim, amount, claim_address, funding_accounts, funding_accounts[0],
channel if not clear_channel else None
)
new_txo = tx.outputs[0]
stream_hash = None
if not preview:
if not preview and old_txo.claim.is_stream:
old_stream = self.file_manager.get_filtered(sd_hash=old_txo.claim.stream.source.sd_hash)
old_stream = old_stream[0] if old_stream else None
if file_path is not None:
@@ -4878,20 +4885,16 @@ class Daemon(metaclass=JSONRPCServerType):
"""
@requires(DHT_COMPONENT)
async def jsonrpc_peer_list(self, blob_hash, search_bottom_out_limit=None, page=None, page_size=None):
async def jsonrpc_peer_list(self, blob_hash, page=None, page_size=None):
"""
Get peers for blob hash
Usage:
peer_list (<blob_hash> | --blob_hash=<blob_hash>)
[<search_bottom_out_limit> | --search_bottom_out_limit=<search_bottom_out_limit>]
[--page=<page>] [--page_size=<page_size>]
Options:
--blob_hash=<blob_hash> : (str) find available peers for this blob hash
--search_bottom_out_limit=<search_bottom_out_limit> : (int) the number of search probes in a row
that don't find any new peers
before giving up and returning
--page=<page> : (int) page to return during paginating
--page_size=<page_size> : (int) number of items on page during pagination
@@ -4903,13 +4906,6 @@ class Daemon(metaclass=JSONRPCServerType):
if not is_valid_blobhash(blob_hash):
# TODO: use error from lbry.error
raise Exception("invalid blob hash")
if search_bottom_out_limit is not None:
search_bottom_out_limit = int(search_bottom_out_limit)
if search_bottom_out_limit <= 0:
# TODO: use error from lbry.error
raise Exception("invalid bottom out limit")
else:
search_bottom_out_limit = 4
peers = []
peer_q = asyncio.Queue(loop=self.component_manager.loop)
await self.dht_node._peers_for_value_producer(blob_hash, peer_q)
@@ -37,6 +37,8 @@ def migrate_db(conf, start, end):
from .migrate13to14 import do_migration
elif current == 14:
from .migrate14to15 import do_migration
elif current == 15:
from .migrate15to16 import do_migration
else:
raise Exception(f"DB migration of version {current} to {current+1} is not available")
try:
@@ -0,0 +1,17 @@
import os
import sqlite3
def do_migration(conf):
db_path = os.path.join(conf.data_dir, "lbrynet.sqlite")
connection = sqlite3.connect(db_path)
cursor = connection.cursor()
cursor.executescript("""
update blob set should_announce=0
where should_announce=1 and
blob.blob_hash in (select stream_blob.blob_hash from stream_blob where position=0);
""")
connection.commit()
connection.close()
+3 -3
View File
@@ -187,8 +187,8 @@ def store_stream(transaction: sqlite3.Connection, sd_blob: 'BlobFile', descripto
).fetchall()
# ensure should_announce is set regardless if insert was ignored
transaction.execute(
"update blob set should_announce=1 where blob_hash in (?, ?)",
(sd_blob.blob_hash, descriptor.blobs[0].blob_hash,)
"update blob set should_announce=1 where blob_hash in (?)",
(sd_blob.blob_hash,)
).fetchall()
@@ -462,7 +462,7 @@ class SQLiteStorage(SQLiteMixin):
content_blobs = await self.db.execute_fetchall(
"select blob.blob_hash, blob.blob_length, blob.added_on "
"from blob join stream_blob using (blob_hash) cross join stream using (stream_hash)"
"cross join file using (stream_hash) where blob.is_mine=? order by blob.added_on asc",
"cross join file using (stream_hash) where blob.is_mine=? order by blob.added_on asc, blob.blob_length asc",
(is_mine,)
)
return content_blobs + sd_blobs
+30
View File
@@ -1,4 +1,6 @@
import os
import filetype
import logging
types_map = {
# http://www.iana.org/assignments/media-types
@@ -166,10 +168,38 @@ types_map = {
'.wmv': ('video/x-ms-wmv', 'video')
}
# maps detected extensions to the possible analogs
# i.e. .cbz file is actually a .zip
synonyms_map = {
'.zip': ['.cbz'],
'.rar': ['.cbr'],
'.ar': ['.a']
}
log = logging.getLogger(__name__)
def guess_media_type(path):
_, ext = os.path.splitext(path)
extension = ext.strip().lower()
try:
kind = filetype.guess(path)
if kind:
real_extension = f".{kind.extension}"
if extension != real_extension:
if extension:
log.warning(f"file extension does not match it's contents: {path}, identified as {real_extension}")
else:
log.debug(f"file {path} does not have extension, identified by it's contents as {real_extension}")
if extension not in synonyms_map.get(real_extension, []):
extension = real_extension
except OSError as error:
pass
if extension[1:]:
if extension in types_map:
return types_map[extension]
-4
View File
@@ -98,10 +98,6 @@ class StreamDownloader:
if not self.descriptor:
await self.load_descriptor(connection_id)
# add the head blob to the peer search
self.search_queue.put_nowait(self.descriptor.blobs[0].blob_hash)
log.info("added head blob to peer search for stream %s", self.sd_hash)
if not await self.blob_manager.storage.stream_exists(self.sd_hash) and save_stream:
await self.blob_manager.storage.store_stream(
self.blob_manager.get_blob(self.sd_hash, length=self.descriptor.length), self.descriptor
+24 -43
View File
@@ -19,7 +19,7 @@ from lbry.conf import Config
from lbry.wallet.util import satoshis_to_coins
from lbry.wallet.dewies import lbc_to_dewies
from lbry.wallet.orchstr8 import Conductor
from lbry.wallet.orchstr8.node import LBCWalletNode, WalletNode, HubNode
from lbry.wallet.orchstr8.node import BlockchainNode, WalletNode, HubNode
from lbry.schema.claim import Claim
from lbry.extras.daemon.daemon import Daemon, jsonrpc_dumps_pretty
@@ -204,7 +204,13 @@ class AsyncioTestCase(unittest.TestCase):
def add_timeout(self):
if self.TIMEOUT:
self.loop.call_later(self.TIMEOUT, self.cancel)
self.loop.call_later(self.TIMEOUT, self.check_timeout, time())
def check_timeout(self, started):
if time() - started >= self.TIMEOUT:
self.cancel()
else:
self.loop.call_later(self.TIMEOUT, self.check_timeout, started)
class AdvanceTimeTestCase(AsyncioTestCase):
@@ -230,7 +236,7 @@ class IntegrationTestCase(AsyncioTestCase):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.conductor: Optional[Conductor] = None
self.blockchain: Optional[LBCWalletNode] = None
self.blockchain: Optional[BlockchainNode] = None
self.hub: Optional[HubNode] = None
self.wallet_node: Optional[WalletNode] = None
self.manager: Optional[WalletManager] = None
@@ -240,17 +246,15 @@ class IntegrationTestCase(AsyncioTestCase):
async def asyncSetUp(self):
self.conductor = Conductor(seed=self.SEED)
await self.conductor.start_lbcd()
self.addCleanup(self.conductor.stop_lbcd)
await self.conductor.start_lbcwallet()
self.addCleanup(self.conductor.stop_lbcwallet)
await self.conductor.start_blockchain()
self.addCleanup(self.conductor.stop_blockchain)
await self.conductor.start_spv()
self.addCleanup(self.conductor.stop_spv)
await self.conductor.start_wallet()
self.addCleanup(self.conductor.stop_wallet)
await self.conductor.start_hub()
self.addCleanup(self.conductor.stop_hub)
self.blockchain = self.conductor.lbcwallet_node
self.blockchain = self.conductor.blockchain_node
self.hub = self.conductor.hub_node
self.wallet_node = self.conductor.wallet_node
self.manager = self.wallet_node.manager
@@ -265,13 +269,6 @@ class IntegrationTestCase(AsyncioTestCase):
def broadcast(self, tx):
return self.ledger.broadcast(tx)
async def broadcast_and_confirm(self, tx, ledger=None):
ledger = ledger or self.ledger
notifications = asyncio.create_task(ledger.wait(tx))
await ledger.broadcast(tx)
await notifications
await self.generate_and_wait(1, [tx.id], ledger)
async def on_header(self, height):
if self.ledger.headers.height < height:
await self.ledger.on_header.where(
@@ -279,29 +276,11 @@ class IntegrationTestCase(AsyncioTestCase):
)
return True
async def send_to_address_and_wait(self, address, amount, blocks_to_generate=0, ledger=None):
tx_watch = []
txid = None
done = False
watcher = (ledger or self.ledger).on_transaction.where(
lambda e: e.tx.id == txid or done or tx_watch.append(e.tx.id)
def on_transaction_id(self, txid, ledger=None):
return (ledger or self.ledger).on_transaction.where(
lambda e: e.tx.id == txid
)
txid = await self.blockchain.send_to_address(address, amount)
done = txid in tx_watch
await watcher
await self.generate_and_wait(blocks_to_generate, [txid], ledger)
return txid
async def generate_and_wait(self, blocks_to_generate, txids, ledger=None):
if blocks_to_generate > 0:
watcher = (ledger or self.ledger).on_transaction.where(
lambda e: ((e.tx.id in txids and txids.remove(e.tx.id)), len(txids) <= 0)[-1] # multi-statement lambda
)
await self.blockchain.generate(blocks_to_generate)
await watcher
def on_address_update(self, address):
return self.ledger.on_transaction.where(
lambda e: e.address == address
@@ -372,19 +351,20 @@ class CommandTestCase(IntegrationTestCase):
self.skip_libtorrent = True
async def asyncSetUp(self):
await super().asyncSetUp()
logging.getLogger('lbry.blob_exchange').setLevel(self.VERBOSITY)
logging.getLogger('lbry.daemon').setLevel(self.VERBOSITY)
logging.getLogger('lbry.stream').setLevel(self.VERBOSITY)
logging.getLogger('lbry.wallet').setLevel(self.VERBOSITY)
await super().asyncSetUp()
self.daemon = await self.add_daemon(self.wallet_node)
await self.account.ensure_address_gap()
address = (await self.account.receiving.get_addresses(limit=1, only_usable=True))[0]
await self.send_to_address_and_wait(address, 10, 6)
sendtxid = await self.blockchain.send_to_address(address, 10)
await self.confirm_tx(sendtxid)
await self.generate(5)
server_tmp_dir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, server_tmp_dir)
@@ -481,8 +461,9 @@ class CommandTestCase(IntegrationTestCase):
async def confirm_tx(self, txid, ledger=None):
""" Wait for tx to be in mempool, then generate a block, wait for tx to be in a block. """
# actually, if it's in the mempool or in the block we're fine
await self.generate_and_wait(1, [txid], ledger=ledger)
await self.on_transaction_id(txid, ledger)
await self.generate(1)
await self.on_transaction_id(txid, ledger)
return txid
async def on_transaction_dict(self, tx):
@@ -533,7 +514,7 @@ class CommandTestCase(IntegrationTestCase):
return self.sout(tx)
return tx
async def create_nondeterministic_channel(self, name, price, pubkey_bytes, daemon=None, blocking=False):
async def create_nondeterministic_channel(self, name, price, pubkey_bytes, daemon=None):
account = (daemon or self.daemon).wallet_manager.default_account
claim_address = await account.receiving.get_or_create_usable_address()
claim = Claim()
@@ -543,7 +524,7 @@ class CommandTestCase(IntegrationTestCase):
claim_address, [self.account], self.account
)
await tx.sign([self.account])
await (daemon or self.daemon).broadcast_or_release(tx, blocking)
await (daemon or self.daemon).broadcast_or_release(tx, False)
return self.sout(tx)
def create_upload_file(self, data, prefix=None, suffix=None):
+15
View File
@@ -474,3 +474,18 @@ class LockWithMetrics(asyncio.Lock):
return super().release()
finally:
self._lock_held_time_metric.observe(time.perf_counter() - self._lock_acquired_time)
def get_colliding_prefix_bits(first_value: bytes, second_value: bytes):
"""
Calculates the amount of colliding prefix bits between <first_value> and <second_value>.
This is given by the amount of bits that are the same until the first different one (via XOR),
starting from the most significant bit to the least significant bit.
:param first_value: first value to compare, bigger than size.
:param second_value: second value to compare, bigger than size.
:return: amount of prefix colliding bits.
"""
assert len(first_value) == len(second_value), "length should be the same"
size = len(first_value) * 8
first_value, second_value = int.from_bytes(first_value, "big"), int.from_bytes(second_value, "big")
return size - (first_value ^ second_value).bit_length()
+5 -10
View File
@@ -1,13 +1,8 @@
__lbcd__ = 'lbcd'
__lbcctl__ = 'lbcctl'
__lbcwallet__ = 'lbcwallet'
__lbcd_url__ = (
'https://github.com/lbryio/lbcd/releases/download/' +
'v0.22.200-beta/lbcd_0.22.200-beta_TARGET_PLATFORM.tar.gz'
)
__lbcwallet_url__ = (
'https://github.com/lbryio/lbcwallet/releases/download/' +
'v0.13.100-alpha-rc2/lbcwallet_0.13.100-alpha-rc2_TARGET_PLATFORM.tar.gz'
__node_daemon__ = 'lbrycrdd'
__node_cli__ = 'lbrycrd-cli'
__node_bin__ = ''
__node_url__ = (
'https://github.com/lbryio/lbrycrd/releases/download/v0.17.4.6/lbrycrd-linux-1746.zip'
)
__spvserver__ = 'lbry.wallet.server.coin.LBCRegTest'
+8 -3
View File
@@ -39,9 +39,14 @@ class DeterministicChannelKeyManager:
self.account = account
self.last_known = 0
self.cache = {}
self.private_key: Optional[PrivateKey] = None
if account.private_key is not None:
self.private_key = account.private_key.child(KeyPath.CHANNEL)
self._private_key: Optional[PrivateKey] = None
@property
def private_key(self):
if self._private_key is None:
if self.account.private_key is not None:
self._private_key = self.account.private_key.child(KeyPath.CHANNEL)
return self._private_key
def maybe_generate_deterministic_key_for_channel(self, txo):
if self.private_key is None:
+1
View File
@@ -2,6 +2,7 @@ NULL_HASH32 = b'\x00'*32
CENT = 1000000
COIN = 100*CENT
DUST = 1000
TIMEOUT = 30.0
-4
View File
@@ -365,10 +365,6 @@ class Ledger(metaclass=LedgerRegistry):
await self.db.close()
await self.headers.close()
async def tasks_are_done(self):
await self._update_tasks.done.wait()
await self._other_tasks.done.wait()
@property
def local_height_including_downloaded_height(self):
return max(self.headers.height, self._download_height)
+7 -14
View File
@@ -5,9 +5,7 @@ import aiohttp
from lbry import wallet
from lbry.wallet.orchstr8.node import (
Conductor,
get_lbcd_node_from_ledger,
get_lbcwallet_node_from_ledger
Conductor, get_blockchain_node_from_ledger
)
from lbry.wallet.orchstr8.service import ConductorService
@@ -18,11 +16,10 @@ def get_argument_parser():
)
subparsers = parser.add_subparsers(dest='command', help='sub-command help')
subparsers.add_parser("download", help="Download lbcd and lbcwallet node binaries.")
subparsers.add_parser("download", help="Download blockchain node binary.")
start = subparsers.add_parser("start", help="Start orchstr8 service.")
start.add_argument("--lbcd", help="Hostname to start lbcd node.")
start.add_argument("--lbcwallet", help="Hostname to start lbcwallet node.")
start.add_argument("--blockchain", help="Hostname to start blockchain node.")
start.add_argument("--spv", help="Hostname to start SPV server.")
start.add_argument("--wallet", help="Hostname to start wallet daemon.")
@@ -50,8 +47,7 @@ def main():
if command == 'download':
logging.getLogger('blockchain').setLevel(logging.INFO)
get_lbcd_node_from_ledger(wallet).ensure()
get_lbcwallet_node_from_ledger(wallet).ensure()
get_blockchain_node_from_ledger(wallet).ensure()
elif command == 'generate':
loop.run_until_complete(run_remote_command(
@@ -61,12 +57,9 @@ def main():
elif command == 'start':
conductor = Conductor()
if getattr(args, 'lbcd', False):
conductor.lbcd_node.hostname = args.lbcd
loop.run_until_complete(conductor.start_lbcd())
if getattr(args, 'lbcwallet', False):
conductor.lbcwallet_node.hostname = args.lbcwallet
loop.run_until_complete(conductor.start_lbcwallet())
if getattr(args, 'blockchain', False):
conductor.blockchain_node.hostname = args.blockchain
loop.run_until_complete(conductor.start_blockchain())
if getattr(args, 'spv', False):
conductor.spv_node.hostname = args.spv
loop.run_until_complete(conductor.start_spv())
+77 -269
View File
@@ -8,7 +8,6 @@ import logging
import tempfile
import subprocess
import importlib
import platform
from distutils.util import strtobool
from binascii import hexlify
@@ -32,18 +31,11 @@ def get_spvserver_from_ledger(ledger_module):
return getattr(spvserver_module, regtest_class_name)
def get_lbcd_node_from_ledger(ledger_module):
return LBCDNode(
ledger_module.__lbcd_url__,
ledger_module.__lbcd__,
ledger_module.__lbcctl__
)
def get_lbcwallet_node_from_ledger(ledger_module):
return LBCWalletNode(
ledger_module.__lbcwallet_url__,
ledger_module.__lbcwallet__,
ledger_module.__lbcctl__
def get_blockchain_node_from_ledger(ledger_module):
return BlockchainNode(
ledger_module.__node_url__,
os.path.join(ledger_module.__node_bin__, ledger_module.__node_daemon__),
os.path.join(ledger_module.__node_bin__, ledger_module.__node_cli__)
)
@@ -53,52 +45,51 @@ class Conductor:
self.manager_module = WalletManager
self.spv_module = get_spvserver_from_ledger(lbry.wallet)
self.lbcd_node = get_lbcd_node_from_ledger(lbry.wallet)
self.lbcwallet_node = get_lbcwallet_node_from_ledger(lbry.wallet)
self.blockchain_node = get_blockchain_node_from_ledger(lbry.wallet)
self.spv_node = SPVNode(self.spv_module)
self.wallet_node = WalletNode(
self.manager_module, RegTestLedger, default_seed=seed
)
self.hub_node = HubNode(__hub_url__, "hub", self.spv_node)
self.lbcd_started = False
self.lbcwallet_started = False
self.blockchain_started = False
self.spv_started = False
self.wallet_started = False
self.hub_started = False
self.log = log.getChild('conductor')
async def start_lbcd(self):
if not self.lbcd_started:
asyncio.create_task(self.lbcd_node.start())
await self.lbcd_node.running.wait()
self.lbcd_started = True
async def start_blockchain(self):
if not self.blockchain_started:
asyncio.create_task(self.blockchain_node.start())
await self.blockchain_node.running.wait()
await self.blockchain_node.generate(200)
self.blockchain_started = True
async def stop_lbcd(self, cleanup=True):
if self.lbcd_started:
await self.lbcd_node.stop(cleanup)
self.lbcd_started = False
async def stop_blockchain(self):
if self.blockchain_started:
await self.blockchain_node.stop(cleanup=True)
self.blockchain_started = False
async def start_hub(self):
if not self.hub_started:
asyncio.create_task(self.hub_node.start())
await self.lbcwallet_node.running.wait()
await self.blockchain_node.running.wait()
self.hub_started = True
async def stop_hub(self, cleanup=True):
async def stop_hub(self):
if self.hub_started:
await self.hub_node.stop(cleanup)
await self.hub_node.stop(cleanup=True)
self.hub_started = False
async def start_spv(self):
if not self.spv_started:
await self.spv_node.start(self.lbcwallet_node)
await self.spv_node.start(self.blockchain_node)
self.spv_started = True
async def stop_spv(self, cleanup=True):
async def stop_spv(self):
if self.spv_started:
await self.spv_node.stop(cleanup)
await self.spv_node.stop(cleanup=True)
self.spv_started = False
async def start_wallet(self):
@@ -106,31 +97,13 @@ class Conductor:
await self.wallet_node.start(self.spv_node)
self.wallet_started = True
async def stop_wallet(self, cleanup=True):
async def stop_wallet(self):
if self.wallet_started:
await self.wallet_node.stop(cleanup)
await self.wallet_node.stop(cleanup=True)
self.wallet_started = False
async def start_lbcwallet(self, clean=True):
if not self.lbcwallet_started:
asyncio.create_task(self.lbcwallet_node.start())
await self.lbcwallet_node.running.wait()
if clean:
mining_addr = await self.lbcwallet_node.get_new_address()
self.lbcwallet_node.mining_addr = mining_addr
await self.lbcwallet_node.generate(200)
# unlock the wallet for the next 1 hour
await self.lbcwallet_node.wallet_passphrase("password", 3600)
self.lbcwallet_started = True
async def stop_lbcwallet(self, cleanup=True):
if self.lbcwallet_started:
await self.lbcwallet_node.stop(cleanup)
self.lbcwallet_started = False
async def start(self):
await self.start_lbcd()
await self.start_lbcwallet()
await self.start_blockchain()
await self.start_spv()
await self.start_wallet()
@@ -138,8 +111,7 @@ class Conductor:
all_the_stops = [
self.stop_wallet,
self.stop_spv,
self.stop_lbcwallet,
self.stop_lbcd
self.stop_blockchain
]
for stop in all_the_stops:
try:
@@ -147,11 +119,6 @@ class Conductor:
except Exception as e:
log.exception('Exception raised while stopping services:', exc_info=e)
async def clear_mempool(self):
await self.stop_lbcwallet(cleanup=False)
await self.stop_lbcd(cleanup=False)
await self.start_lbcd()
await self.start_lbcwallet(clean=False)
class WalletNode:
@@ -172,11 +139,10 @@ class WalletNode:
async def start(self, spv_node: 'SPVNode', seed=None, connect=True, config=None):
wallets_dir = os.path.join(self.data_path, 'wallets')
os.mkdir(wallets_dir)
wallet_file_name = os.path.join(wallets_dir, 'my_wallet.json')
if not os.path.isdir(wallets_dir):
os.mkdir(wallets_dir)
with open(wallet_file_name, 'w') as wallet_file:
wallet_file.write('{"version": 1, "accounts": []}\n')
with open(wallet_file_name, 'w') as wallet_file:
wallet_file.write('{"version": 1, "accounts": []}\n')
self.manager = self.manager_class.from_config({
'ledgers': {
self.ledger_class.get_id(): {
@@ -232,14 +198,14 @@ class SPVNode:
self.stopped = False
self.index_name = uuid4().hex
async def start(self, lbcwallet_node: 'LBCWalletNode', extraconf=None):
async def start(self, blockchain_node: 'BlockchainNode', extraconf=None):
self.data_path = tempfile.mkdtemp()
conf = {
'DESCRIPTION': '',
'PAYMENT_ADDRESS': '',
'DAILY_FEE': '0',
'DB_DIRECTORY': self.data_path,
'DAEMON_URL': lbcwallet_node.rpc_url,
'DAEMON_URL': blockchain_node.rpc_url,
'REORG_LIMIT': '100',
'HOST': self.hostname,
'TCP_PORT': str(self.port),
@@ -274,19 +240,18 @@ class SPVNode:
shutil.rmtree(self.data_path, ignore_errors=True)
class LBCDProcess(asyncio.SubprocessProtocol):
class BlockchainProcess(asyncio.SubprocessProtocol):
IGNORE_OUTPUT = [
b'keypool keep',
b'keypool reserve',
b'keypool return',
b'Block submitted',
]
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('lbcd')
self.log = log.getChild('blockchain')
def pipe_data_received(self, fd, data):
if self.log and not any(ignore in data for ignore in self.IGNORE_OUTPUT):
@@ -297,7 +262,7 @@ class LBCDProcess(asyncio.SubprocessProtocol):
if b'Error:' in data:
self.ready.set()
raise SystemError(data.decode())
if b'RPCS: RPC server listening on' in data:
if b'Done loading' in data:
self.ready.set()
def process_exited(self):
@@ -305,34 +270,10 @@ class LBCDProcess(asyncio.SubprocessProtocol):
self.ready.set()
class WalletProcess(asyncio.SubprocessProtocol):
class BlockchainNode:
IGNORE_OUTPUT = [
]
def __init__(self):
self.ready = asyncio.Event()
self.stopped = asyncio.Event()
self.log = log.getChild('lbcwallet')
self.transport: Optional[asyncio.transports.SubprocessTransport] = None
def pipe_data_received(self, fd, data):
if self.log and not any(ignore in data for ignore in self.IGNORE_OUTPUT):
if b'Error:' in data:
self.log.error(data.decode())
else:
self.log.info(data.decode())
if b'Error:' in data:
self.ready.set()
raise SystemError(data.decode())
if b'WLLT: Finished rescan' in data:
self.ready.set()
def process_exited(self):
self.stopped.set()
self.ready.set()
class LBCDNode:
P2SH_SEGWIT_ADDRESS = "p2sh-segwit"
BECH32_ADDRESS = "bech32"
def __init__(self, url, daemon, cli):
self.latest_release_url = url
@@ -340,22 +281,28 @@ class LBCDNode:
self.bin_dir = os.path.join(self.project_dir, 'bin')
self.daemon_bin = os.path.join(self.bin_dir, daemon)
self.cli_bin = os.path.join(self.bin_dir, cli)
self.log = log.getChild('lbcd')
self.data_path = tempfile.mkdtemp()
self.log = log.getChild('blockchain')
self.data_path = None
self.protocol = None
self.transport = None
self.block_expected = 0
self.hostname = 'localhost'
self.peerport = 29246
self.rpcport = 29245
self.peerport = 9246 + 2 # avoid conflict with default peer port
self.rpcport = 9245 + 2 # avoid conflict with default rpc port
self.rpcuser = 'rpcuser'
self.rpcpassword = 'rpcpassword'
self.stopped = True
self.stopped = False
self.restart_ready = asyncio.Event()
self.restart_ready.set()
self.running = asyncio.Event()
@property
def rpc_url(self):
return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.rpcport}/'
def is_expected_block(self, e: BlockHeightEvent):
return self.block_expected == e.height
@property
def exists(self):
return (
@@ -364,12 +311,6 @@ class LBCDNode:
)
def download(self):
uname = platform.uname()
target_os = str.lower(uname.system)
target_arch = str.replace(uname.machine, 'x86_64', 'amd64')
target_platform = target_os + '_' + target_arch
self.latest_release_url = str.replace(self.latest_release_url, 'TARGET_PLATFORM', target_platform)
downloaded_file = os.path.join(
self.bin_dir,
self.latest_release_url[self.latest_release_url.rfind('/')+1:]
@@ -404,34 +345,34 @@ class LBCDNode:
async def start(self):
assert self.ensure()
self.data_path = tempfile.mkdtemp()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.daemon_bin,
'--notls',
f'--datadir={self.data_path}',
'--regtest', f'--listen=127.0.0.1:{self.peerport}', f'--rpclisten=127.0.0.1:{self.rpcport}',
'--txindex', f'--rpcuser={self.rpcuser}', f'--rpcpass={self.rpcpassword}'
f'-datadir={self.data_path}', '-printtoconsole', '-regtest', '-server', '-txindex',
f'-rpcuser={self.rpcuser}', f'-rpcpassword={self.rpcpassword}', f'-rpcport={self.rpcport}',
f'-port={self.peerport}'
]
self.log.info(' '.join(command))
while self.stopped:
while not self.stopped:
if self.running.is_set():
await asyncio.sleep(1)
continue
await self.restart_ready.wait()
try:
self.transport, self.protocol = await loop.subprocess_exec(
LBCDProcess, *command
BlockchainProcess, *command
)
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
self.stopped = False
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start lbcd', exc_info=e)
log.exception('failed to start lbrycrdd', exc_info=e)
async def stop(self, cleanup=True):
self.stopped = True
@@ -440,161 +381,35 @@ class LBCDNode:
await self.protocol.stopped.wait()
self.transport.close()
finally:
self.log.info("Done shutting down " + self.daemon_bin)
if cleanup:
self.cleanup()
self.running.clear()
async def clear_mempool(self):
self.restart_ready.clear()
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
self.running.clear()
os.remove(os.path.join(self.data_path, 'regtest', 'mempool.dat'))
self.restart_ready.set()
await self.running.wait()
def cleanup(self):
assert self.stopped
shutil.rmtree(self.data_path, ignore_errors=True)
class LBCWalletNode:
P2SH_SEGWIT_ADDRESS = "p2sh-segwit"
BECH32_ADDRESS = "bech32"
def __init__(self, url, lbcwallet, cli):
self.latest_release_url = url
self.project_dir = os.path.dirname(os.path.dirname(__file__))
self.bin_dir = os.path.join(self.project_dir, 'bin')
self.lbcwallet_bin = os.path.join(self.bin_dir, lbcwallet)
self.cli_bin = os.path.join(self.bin_dir, cli)
self.log = log.getChild('lbcwallet')
self.protocol = None
self.transport = None
self.hostname = 'localhost'
self.lbcd_rpcport = 29245
self.lbcwallet_rpcport = 29244
self.rpcuser = 'rpcuser'
self.rpcpassword = 'rpcpassword'
self.data_path = tempfile.mkdtemp()
self.stopped = True
self.running = asyncio.Event()
self.block_expected = 0
self.mining_addr = ''
@property
def rpc_url(self):
# FIXME: somehow the hub/sdk doesn't learn the blocks through the Walet RPC port, why?
# return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.lbcwallet_rpcport}/'
return f'http://{self.rpcuser}:{self.rpcpassword}@{self.hostname}:{self.lbcd_rpcport}/'
def is_expected_block(self, e: BlockHeightEvent):
return self.block_expected == e.height
@property
def exists(self):
return (
os.path.exists(self.lbcwallet_bin)
)
def download(self):
uname = platform.uname()
target_os = str.lower(uname.system)
target_arch = str.replace(uname.machine, 'x86_64', 'amd64')
target_platform = target_os + '_' + target_arch
self.latest_release_url = str.replace(self.latest_release_url, 'TARGET_PLATFORM', target_platform)
downloaded_file = os.path.join(
self.bin_dir,
self.latest_release_url[self.latest_release_url.rfind('/')+1:]
)
if not os.path.exists(self.bin_dir):
os.mkdir(self.bin_dir)
if not os.path.exists(downloaded_file):
self.log.info('Downloading: %s', self.latest_release_url)
with urllib.request.urlopen(self.latest_release_url) as response:
with open(downloaded_file, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
self.log.info('Extracting: %s', downloaded_file)
if downloaded_file.endswith('.zip'):
with zipfile.ZipFile(downloaded_file) as dotzip:
dotzip.extractall(self.bin_dir)
# zipfile bug https://bugs.python.org/issue15795
os.chmod(self.lbcwallet_bin, 0o755)
elif downloaded_file.endswith('.tar.gz'):
with tarfile.open(downloaded_file) as tar:
tar.extractall(self.bin_dir)
return self.exists
def ensure(self):
return self.exists or self.download()
async def start(self):
assert self.ensure()
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
command = [
self.lbcwallet_bin,
'--noservertls', '--noclienttls',
'--regtest',
f'--rpcconnect=127.0.0.1:{self.lbcd_rpcport}', f'--rpclisten=127.0.0.1:{self.lbcwallet_rpcport}',
'--createtemp', f'--appdata={self.data_path}',
f'--username={self.rpcuser}', f'--password={self.rpcpassword}'
]
self.log.info(' '.join(command))
while self.stopped:
if self.running.is_set():
await asyncio.sleep(1)
continue
try:
self.transport, self.protocol = await loop.subprocess_exec(
WalletProcess, *command
)
self.protocol.transport = self.transport
await self.protocol.ready.wait()
assert not self.protocol.stopped.is_set()
self.running.set()
self.stopped = False
except asyncio.CancelledError:
self.running.clear()
raise
except Exception as e:
self.running.clear()
log.exception('failed to start lbcwallet', exc_info=e)
def cleanup(self):
assert self.stopped
shutil.rmtree(self.data_path, ignore_errors=True)
async def stop(self, cleanup=True):
self.stopped = True
try:
self.transport.terminate()
await self.protocol.stopped.wait()
self.transport.close()
finally:
self.log.info("Done shutting down " + self.lbcwallet_bin)
if cleanup:
self.cleanup()
self.running.clear()
async def _cli_cmnd(self, *args):
cmnd_args = [
self.cli_bin,
f'--rpcuser={self.rpcuser}', f'--rpcpass={self.rpcpassword}', '--notls', '--regtest', '--wallet'
self.cli_bin, f'-datadir={self.data_path}', '-regtest',
f'-rpcuser={self.rpcuser}', f'-rpcpassword={self.rpcpassword}', f'-rpcport={self.rpcport}'
] + list(args)
self.log.info(' '.join(cmnd_args))
loop = asyncio.get_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
process = await asyncio.create_subprocess_exec(
*cmnd_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE
*cmnd_args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
out, err = await process.communicate()
out, _ = await process.communicate()
result = out.decode().strip()
err = err.decode().strip()
if len(result) <= 0 and err.startswith('-'):
raise Exception(err)
if err and 'creating a default config file' not in err:
log.warning(err)
self.log.info(result)
if result.startswith('error code'):
raise Exception(result)
@@ -602,14 +417,7 @@ class LBCWalletNode:
def generate(self, blocks):
self.block_expected += blocks
return self._cli_cmnd('generatetoaddress', str(blocks), self.mining_addr)
def generate_to_address(self, blocks, addr):
self.block_expected += blocks
return self._cli_cmnd('generatetoaddress', str(blocks), addr)
def wallet_passphrase(self, passphrase, timeout):
return self._cli_cmnd('walletpassphrase', passphrase, str(timeout))
return self._cli_cmnd('generate', str(blocks))
def invalidate_block(self, blockhash):
return self._cli_cmnd('invalidateblock', blockhash)
@@ -626,7 +434,7 @@ class LBCWalletNode:
def get_raw_change_address(self):
return self._cli_cmnd('getrawchangeaddress')
def get_new_address(self, address_type='legacy'):
def get_new_address(self, address_type):
return self._cli_cmnd('getnewaddress', "", address_type)
async def get_balance(self):
@@ -642,10 +450,7 @@ class LBCWalletNode:
return self._cli_cmnd('createrawtransaction', json.dumps(inputs), json.dumps(outputs))
async def sign_raw_transaction_with_wallet(self, tx):
# the "withwallet" portion should only come into play if we are doing segwit.
# and "withwallet" doesn't exist on lbcd yet.
result = await self._cli_cmnd('signrawtransaction', tx)
return json.loads(result)['hex'].encode()
return json.loads(await self._cli_cmnd('signrawtransactionwithwallet', tx))['hex'].encode()
def decode_raw_transaction(self, tx):
return self._cli_cmnd('decoderawtransaction', hexlify(tx.raw).decode())
@@ -695,6 +500,8 @@ class HubNode:
self.hostname = 'localhost'
self.rpcport = 50051 # avoid conflict with default rpc port
self.stopped = False
self.restart_ready = asyncio.Event()
self.restart_ready.set()
self.running = asyncio.Event()
@property
@@ -751,6 +558,7 @@ class HubNode:
if self.running.is_set():
await asyncio.sleep(1)
continue
await self.restart_ready.wait()
try:
if not self.debug:
self.transport, self.protocol = await loop.subprocess_exec(
+8 -13
View File
@@ -61,10 +61,8 @@ class ConductorService:
#set_logging(
# self.stack.ledger_module, logging.DEBUG, WebSocketLogHandler(self.send_message)
#)
self.stack.lbcd_started or await self.stack.start_lbcd()
self.send_message({'type': 'service', 'name': 'lbcd', 'port': self.stack.lbcd_node.port})
self.stack.lbcwallet_started or await self.stack.start_lbcwallet()
self.send_message({'type': 'service', 'name': 'lbcwallet', 'port': self.stack.lbcwallet_node.port})
self.stack.blockchain_started or await self.stack.start_blockchain()
self.send_message({'type': 'service', 'name': 'blockchain', 'port': self.stack.blockchain_node.port})
self.stack.spv_started or await self.stack.start_spv()
self.send_message({'type': 'service', 'name': 'spv', 'port': self.stack.spv_node.port})
self.stack.wallet_started or await self.stack.start_wallet()
@@ -76,7 +74,7 @@ class ConductorService:
async def generate(self, request):
data = await request.post()
blocks = data.get('blocks', 1)
await self.stack.lbcwallet_node.generate(int(blocks))
await self.stack.blockchain_node.generate(int(blocks))
return json_response({'blocks': blocks})
async def transfer(self, request):
@@ -87,14 +85,11 @@ class ConductorService:
if not address:
raise ValueError("No address was provided.")
amount = data.get('amount', 1)
txid = await self.stack.blockchain_node.send_to_address(address, amount)
if self.stack.wallet_started:
watcher = self.stack.wallet_node.ledger.on_transaction.where(
lambda e: e.address == address # and e.tx.id == txid -- might stall; see send_to_address_and_wait
await self.stack.wallet_node.ledger.on_transaction.where(
lambda e: e.tx.id == txid and e.address == address
)
txid = await self.stack.lbcwallet_node.send_to_address(address, amount)
await watcher
else:
txid = await self.stack.lbcwallet_node.send_to_address(address, amount)
return json_response({
'address': address,
'amount': amount,
@@ -103,7 +98,7 @@ class ConductorService:
async def balance(self, _):
return json_response({
'balance': await self.stack.lbcwallet_node.get_balance()
'balance': await self.stack.blockchain_node.get_balance()
})
async def log(self, request):
@@ -134,7 +129,7 @@ class ConductorService:
'type': 'status',
'height': self.stack.wallet_node.ledger.headers.height,
'balance': satoshis_to_coins(await self.stack.wallet_node.account.get_balance()),
'miner': await self.stack.lbcwallet_node.get_balance()
'miner': await self.stack.blockchain_node.get_balance()
})
def send_message(self, msg):
+38 -3
View File
@@ -53,7 +53,7 @@ class Daemon:
self.max_retry = max_retry
self._height = None
self.available_rpcs = {}
self.connector = aiohttp.TCPConnector(ssl=False)
self.connector = aiohttp.TCPConnector()
self._block_hash_cache = LRUCacheWithMetrics(100000)
self._block_cache = LRUCacheWithMetrics(2 ** 13, metric_name='block', namespace=NAMESPACE)
@@ -250,14 +250,14 @@ class Daemon:
async def deserialised_block(self, hex_hash):
"""Return the deserialised block with the given hex hash."""
if hex_hash not in self._block_cache:
block = await self._send_single('getblock', (hex_hash, 1))
block = await self._send_single('getblock', (hex_hash, True))
self._block_cache[hex_hash] = block
return block
return self._block_cache[hex_hash]
async def raw_blocks(self, hex_hashes):
"""Return the raw binary blocks with the given hex hashes."""
params_iterable = ((h, 0) for h in hex_hashes)
params_iterable = ((h, False) for h in hex_hashes)
blocks = await self._send_vector('getblock', params_iterable)
# Convert hex string to bytes
return [hex_to_bytes(block) for block in blocks]
@@ -334,7 +334,42 @@ class LBCDaemon(Daemon):
async def getrawtransaction(self, hex_hash, verbose=False):
return await super().getrawtransaction(hex_hash=hex_hash, verbose=verbose)
@handles_errors
async def getclaimbyid(self, claim_id):
'''Given a claim id, retrieves claim information.'''
return await self._send_single('getclaimbyid', (claim_id,))
@handles_errors
async def getclaimsbyids(self, claim_ids):
'''Given a list of claim ids, batches calls to retrieve claim information.'''
return await self._send_vector('getclaimbyid', ((claim_id,) for claim_id in claim_ids))
@handles_errors
async def getclaimsforname(self, name):
'''Given a name, retrieves all claims matching that name.'''
return await self._send_single('getclaimsforname', (name,))
@handles_errors
async def getclaimsfortx(self, txid):
'''Given a txid, returns the claims it make.'''
return await self._send_single('getclaimsfortx', (txid,)) or []
@handles_errors
async def getnameproof(self, name, block_hash=None):
'''Given a name and optional block_hash, returns a name proof and winner, if any.'''
return await self._send_single('getnameproof', (name, block_hash,) if block_hash else (name,))
@handles_errors
async def getvalueforname(self, name):
'''Given a name, returns the winning claim value.'''
return await self._send_single('getvalueforname', (name,))
@handles_errors
async def getnamesintrie(self):
'''Given a name, returns the winning claim value.'''
return await self._send_single('getnamesintrie')
@handles_errors
async def claimname(self, name, hexvalue, amount):
'''Claim a name, used for functional tests only.'''
return await self._send_single('claimname', (name, hexvalue, float(amount)))
+11 -112
View File
@@ -1,130 +1,32 @@
import struct
import rocksdb
from typing import Optional
from lbry.wallet.server.db import DB_PREFIXES
from lbry.wallet.server.db.revertable import RevertableOpStack, RevertablePut, RevertableDelete
class RocksDBStore:
def __init__(self, path: str, cache_mb: int, max_open_files: int, secondary_path: str = ''):
# Use snappy compression (the default)
self.path = path
self._max_open_files = max_open_files
self.db = rocksdb.DB(path, self.get_options(), secondary_name=secondary_path)
# self.multi_get = self.db.multi_get
def get_options(self):
return rocksdb.Options(
create_if_missing=True, use_fsync=True, target_file_size_base=33554432,
max_open_files=self._max_open_files
)
class KeyValueStorage:
def get(self, key: bytes, fill_cache: bool = True) -> Optional[bytes]:
return self.db.get(key, fill_cache=fill_cache)
raise NotImplemented()
def iterator(self, reverse=False, start=None, stop=None, include_start=True, include_stop=False, prefix=None,
include_key=True, include_value=True, fill_cache=True):
return RocksDBIterator(
self.db, reverse=reverse, start=start, stop=stop, include_start=include_start, include_stop=include_stop,
prefix=prefix, include_key=include_key, include_value=include_value
)
raise NotImplemented()
def write_batch(self, disable_wal: bool = False, sync: bool = False):
return RocksDBWriteBatch(self.db, sync=sync, disable_wal=disable_wal)
def write_batch(self, transaction: bool = False):
raise NotImplemented()
def close(self):
self.db.close()
self.db = None
raise NotImplemented()
@property
def closed(self) -> bool:
return self.db is None
def try_catch_up_with_primary(self):
self.db.try_catch_up_with_primary()
class RocksDBWriteBatch:
def __init__(self, db: rocksdb.DB, sync: bool = False, disable_wal: bool = False):
self.batch = rocksdb.WriteBatch()
self.db = db
self.sync = sync
self.disable_wal = disable_wal
def __enter__(self):
return self.batch
def __exit__(self, exc_type, exc_val, exc_tb):
if not exc_val:
self.db.write(self.batch, sync=self.sync, disable_wal=self.disable_wal)
class RocksDBIterator:
"""An iterator for RocksDB."""
__slots__ = [
'start',
'prefix',
'stop',
'iterator',
'include_key',
'include_value',
'prev_k',
'reverse',
'include_start',
'include_stop'
]
def __init__(self, db: rocksdb.DB, prefix: bytes = None, start: bool = None, stop: bytes = None,
include_key: bool = True, include_value: bool = True, reverse: bool = False,
include_start: bool = True, include_stop: bool = False):
assert (start is None and stop is None) or (prefix is None), 'cannot use start/stop and prefix'
self.start = start
self.prefix = prefix
self.stop = stop
self.iterator = db.iteritems() if not reverse else reversed(db.iteritems())
if prefix is not None:
self.iterator.seek(prefix)
elif start is not None:
self.iterator.seek(start)
self.include_key = include_key
self.include_value = include_value
self.prev_k = None
self.reverse = reverse
self.include_start = include_start
self.include_stop = include_stop
def __iter__(self):
return self
def _check_stop_iteration(self, key: bytes):
if self.stop is not None and (key.startswith(self.stop) or self.stop < key[:len(self.stop)]):
raise StopIteration
elif self.start is not None and self.start > key[:len(self.start)]:
raise StopIteration
elif self.prefix is not None and not key.startswith(self.prefix):
raise StopIteration
def __next__(self):
# TODO: include start/stop on/off
# check for needing to stop from previous iteration
if self.prev_k is not None:
self._check_stop_iteration(self.prev_k)
k, v = next(self.iterator)
self._check_stop_iteration(k)
self.prev_k = k
if self.include_key and self.include_value:
return k, v
elif self.include_key:
return k
return v
raise NotImplemented()
class PrefixDB:
UNDO_KEY_STRUCT = struct.Struct(b'>Q')
def __init__(self, db: RocksDBStore, max_undo_depth: int = 200, unsafe_prefixes=None):
def __init__(self, db: KeyValueStorage, max_undo_depth: int = 200, unsafe_prefixes=None):
self._db = db
self._op_stack = RevertableOpStack(db.get, unsafe_prefixes=unsafe_prefixes)
self._max_undo_depth = max_undo_depth
@@ -135,7 +37,7 @@ class PrefixDB:
Changes written cannot be undone
"""
try:
with self._db.write_batch(sync=True) as batch:
with self._db.write_batch(transaction=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
for staged_change in self._op_stack:
@@ -159,7 +61,7 @@ class PrefixDB:
include_value=False
))
try:
with self._db.write_batch(sync=True) as batch:
with self._db.write_batch(transaction=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
for staged_change in self._op_stack:
@@ -180,7 +82,7 @@ class PrefixDB:
undo_key = DB_PREFIXES.undo.value + self.UNDO_KEY_STRUCT.pack(height)
self._op_stack.apply_packed_undo_ops(self._db.get(undo_key))
try:
with self._db.write_batch(sync=True) as batch:
with self._db.write_batch(transaction=True) as batch:
batch_put = batch.put
batch_delete = batch.delete
for staged_change in self._op_stack:
@@ -206,9 +108,6 @@ class PrefixDB:
if not self._db.closed:
self._db.close()
def try_catch_up_with_primary(self):
self._db.try_catch_up_with_primary()
@property
def closed(self):
return self._db.closed
+36 -11
View File
@@ -620,18 +620,43 @@ def expand_query(**kwargs):
if 'signature_valid' in kwargs:
query['must'].append({"term": {"is_signature_valid": bool(kwargs["signature_valid"])}})
elif 'signature_valid' in kwargs:
query.setdefault('should', [])
query["minimum_should_match"] = 1
query['should'].append({"bool": {"must_not": {"exists": {"field": "signature"}}}})
query['should'].append({"term": {"is_signature_valid": bool(kwargs["signature_valid"])}})
query['must'].append(
{"bool":
{"should": [
{"bool": {"must_not": {"exists": {"field": "signature"}}}},
{"bool" : {"must" : {"term": {"is_signature_valid": bool(kwargs["signature_valid"])}}}}
]}
}
)
if 'has_source' in kwargs:
query.setdefault('should', [])
query["minimum_should_match"] = 1
is_stream_or_repost = {"terms": {"claim_type": [CLAIM_TYPES['stream'], CLAIM_TYPES['repost']]}}
query['should'].append(
{"bool": {"must": [{"match": {"has_source": kwargs['has_source']}}, is_stream_or_repost]}})
query['should'].append({"bool": {"must_not": [is_stream_or_repost]}})
query['should'].append({"bool": {"must": [{"term": {"reposted_claim_type": CLAIM_TYPES['channel']}}]}})
is_stream_or_repost_terms = {"terms": {"claim_type": [CLAIM_TYPES['stream'], CLAIM_TYPES['repost']]}}
query['must'].append(
{"bool":
{"should": [
{"bool": # when is_stream_or_repost AND has_source
{"must": [
{"match": {"has_source": kwargs['has_source']}},
is_stream_or_repost_terms,
]
},
},
{"bool": # when not is_stream_or_repost
{"must_not": is_stream_or_repost_terms}
},
{"bool": # when reposted_claim_type wouldn't have source
{"must_not":
[
{"term": {"reposted_claim_type": CLAIM_TYPES['stream']}}
],
"must":
[
{"term": {"claim_type": CLAIM_TYPES['repost']}}
]
}
}
]}
}
)
if kwargs.get('text'):
query['must'].append(
{"simple_query_string":
+34 -4
View File
@@ -4,7 +4,7 @@ import array
import base64
from typing import Union, Tuple, NamedTuple, Optional
from lbry.wallet.server.db import DB_PREFIXES
from lbry.wallet.server.db.db import RocksDBStore, PrefixDB
from lbry.wallet.server.db.db import KeyValueStorage, PrefixDB
from lbry.wallet.server.db.revertable import RevertableOpStack, RevertablePut, RevertableDelete
from lbry.schema.url import normalize_name
@@ -38,7 +38,7 @@ class PrefixRow(metaclass=PrefixRowType):
value_struct: struct.Struct
key_part_lambdas = []
def __init__(self, db: RocksDBStore, op_stack: RevertableOpStack):
def __init__(self, db: KeyValueStorage, op_stack: RevertableOpStack):
self._db = db
self._op_stack = op_stack
@@ -1595,10 +1595,40 @@ class BlockTxsPrefixRow(PrefixRow):
return cls.pack_key(height), cls.pack_value(tx_hashes)
class LevelDBStore(KeyValueStorage):
def __init__(self, path: str, cache_mb: int, max_open_files: int):
import plyvel
self.db = plyvel.DB(
path, create_if_missing=True, max_open_files=max_open_files,
lru_cache_size=cache_mb * 1024 * 1024, write_buffer_size=64 * 1024 * 1024,
max_file_size=1024 * 1024 * 64, bloom_filter_bits=32
)
def get(self, key: bytes, fill_cache: bool = True) -> Optional[bytes]:
return self.db.get(key, fill_cache=fill_cache)
def iterator(self, reverse=False, start=None, stop=None, include_start=True, include_stop=False, prefix=None,
include_key=True, include_value=True, fill_cache=True):
return self.db.iterator(
reverse=reverse, start=start, stop=stop, include_start=include_start, include_stop=include_stop,
prefix=prefix, include_key=include_key, include_value=include_value, fill_cache=fill_cache
)
def write_batch(self, transaction: bool = False, sync: bool = False):
return self.db.write_batch(transaction=transaction, sync=sync)
def close(self):
return self.db.close()
@property
def closed(self) -> bool:
return self.db.closed
class HubDB(PrefixDB):
def __init__(self, path: str, cache_mb: int = 128, reorg_limit: int = 200, max_open_files: int = 512,
secondary_path: str = '', unsafe_prefixes: Optional[typing.Set[bytes]] = None):
db = RocksDBStore(path, cache_mb, max_open_files, secondary_path=secondary_path)
unsafe_prefixes: Optional[typing.Set[bytes]] = None):
db = LevelDBStore(path, cache_mb, max_open_files)
super().__init__(db, reorg_limit, unsafe_prefixes=unsafe_prefixes)
self.claim_to_support = ClaimToSupportPrefixRow(db, self._op_stack)
self.support_to_claim = SupportToClaimPrefixRow(db, self._op_stack)
+1 -4
View File
@@ -26,10 +26,7 @@ class Server:
self.prometheus_server: typing.Optional[PrometheusServer] = None
self.session_mgr = LBRYSessionManager(
env, db, bp.mempool, bp.history_cache, bp.resolve_cache, bp.resolve_outputs_cache, daemon,
self.shutdown_event,
on_available_callback=bp.status_server.set_available,
on_unavailable_callback=bp.status_server.set_unavailable
env, db, bp, daemon, self.shutdown_event
)
self._indexer_task = None
+22 -27
View File
@@ -170,16 +170,13 @@ class SessionManager:
namespace=NAMESPACE, buckets=HISTOGRAM_BUCKETS
)
def __init__(self, env: 'Env', db: LevelDB, mempool, history_cache, resolve_cache, resolve_outputs_cache,
daemon: 'Daemon', shutdown_event: asyncio.Event,
on_available_callback: typing.Callable[[], None], on_unavailable_callback: typing.Callable[[], None]):
def __init__(self, env: 'Env', db: LevelDB, bp: BlockProcessor, daemon: 'Daemon', shutdown_event: asyncio.Event):
env.max_send = max(350000, env.max_send)
self.env = env
self.db = db
self.on_available_callback = on_available_callback
self.on_unavailable_callback = on_unavailable_callback
self.bp = bp
self.daemon = daemon
self.mempool = mempool
self.mempool = bp.mempool
self.shutdown_event = shutdown_event
self.logger = util.class_logger(__name__, self.__class__.__name__)
self.servers: typing.Dict[str, asyncio.AbstractServer] = {}
@@ -189,9 +186,7 @@ class SessionManager:
self.cur_group = SessionGroup(0)
self.txs_sent = 0
self.start_time = time.time()
self.history_cache = history_cache
self.resolve_cache = resolve_cache
self.resolve_outputs_cache = resolve_outputs_cache
self.history_cache = self.bp.history_cache
self.notified_height: typing.Optional[int] = None
# Cache some idea of room to avoid recounting on each subscription
self.subs_room = 0
@@ -248,7 +243,7 @@ class SessionManager:
await self.session_event.wait()
self.session_event.clear()
if not paused and len(self.sessions) >= max_sessions:
self.on_unavailable_callback()
self.bp.status_server.set_unavailable()
self.logger.info(f'maximum sessions {max_sessions:,d} '
f'reached, stopping new connections until '
f'count drops to {low_watermark:,d}')
@@ -257,7 +252,7 @@ class SessionManager:
# Start listening for incoming connections if paused and
# session count has fallen
if paused and len(self.sessions) <= low_watermark:
self.on_available_callback()
self.bp.status_server.set_available()
self.logger.info('resuming listening for incoming connections')
await self._start_external_servers()
paused = False
@@ -538,7 +533,7 @@ class SessionManager:
await self.start_other()
await self._start_external_servers()
server_listening_event.set()
self.on_available_callback()
self.bp.status_server.set_available()
# Peer discovery should start after the external servers
# because we connect to ourself
await asyncio.wait([
@@ -633,9 +628,8 @@ class SessionManager:
for hashX in touched.intersection(self.mempool_statuses.keys()):
self.mempool_statuses.pop(hashX, None)
# self.bp._chain_executor
await asyncio.get_event_loop().run_in_executor(
None, touched.intersection_update, self.hashx_subscriptions_by_session.keys()
self.bp._chain_executor, touched.intersection_update, self.hashx_subscriptions_by_session.keys()
)
if touched or new_touched or (height_changed and self.mempool_statuses):
@@ -872,7 +866,8 @@ class LBRYElectrumX(SessionBase):
self.protocol_tuple = self.PROTOCOL_MIN
self.protocol_string = None
self.daemon = self.session_mgr.daemon
self.db: LevelDB = self.session_mgr.db
self.bp: BlockProcessor = self.session_mgr.bp
self.db: LevelDB = self.bp.db
@classmethod
def protocol_min_max_strings(cls):
@@ -1013,21 +1008,21 @@ class LBRYElectrumX(SessionBase):
self.session_mgr.executor_time_metric.observe(time.perf_counter() - start)
async def _cached_resolve_url(self, url):
if url not in self.session_mgr.resolve_cache:
self.session_mgr.resolve_cache[url] = await self.loop.run_in_executor(None, self.db._resolve, url)
return self.session_mgr.resolve_cache[url]
if url not in self.bp.resolve_cache:
self.bp.resolve_cache[url] = await self.loop.run_in_executor(None, self.db._resolve, url)
return self.bp.resolve_cache[url]
async def claimtrie_resolve(self, *urls) -> str:
sorted_urls = tuple(sorted(urls))
self.session_mgr.urls_to_resolve_count_metric.inc(len(sorted_urls))
try:
if sorted_urls in self.session_mgr.resolve_outputs_cache:
return self.session_mgr.resolve_outputs_cache[sorted_urls]
if sorted_urls in self.bp.resolve_outputs_cache:
return self.bp.resolve_outputs_cache[sorted_urls]
rows, extra = [], []
for url in urls:
if url not in self.session_mgr.resolve_cache:
self.session_mgr.resolve_cache[url] = await self._cached_resolve_url(url)
stream, channel, repost, reposted_channel = self.session_mgr.resolve_cache[url]
if url not in self.bp.resolve_cache:
self.bp.resolve_cache[url] = await self._cached_resolve_url(url)
stream, channel, repost, reposted_channel = self.bp.resolve_cache[url]
if isinstance(channel, ResolveCensoredError):
rows.append(channel)
extra.append(channel.censor_row)
@@ -1052,7 +1047,7 @@ class LBRYElectrumX(SessionBase):
if reposted_channel:
extra.append(reposted_channel)
await asyncio.sleep(0)
self.session_mgr.resolve_outputs_cache[sorted_urls] = result = await self.loop.run_in_executor(
self.bp.resolve_outputs_cache[sorted_urls] = result = await self.loop.run_in_executor(
None, Outputs.to_base64, rows, extra, 0, None, None
)
return result
@@ -1060,7 +1055,7 @@ class LBRYElectrumX(SessionBase):
self.session_mgr.resolved_url_count_metric.inc(len(sorted_urls))
async def get_server_height(self):
return self.db.db_height
return self.bp.height
async def transaction_get_height(self, tx_hash):
self.assert_tx_hash(tx_hash)
@@ -1472,7 +1467,7 @@ class LBRYElectrumX(SessionBase):
if mempool_tx:
raw_tx, block_hash = mempool_tx.raw_tx.hex(), None
else:
tx_info = await self.daemon_request('getrawtransaction', tx_hash, 1)
tx_info = await self.daemon_request('getrawtransaction', tx_hash, True)
raw_tx = tx_info['hex']
block_hash = tx_info.get('blockhash')
if block_hash:
@@ -1509,7 +1504,7 @@ class LBRYElectrumX(SessionBase):
if verbose not in (True, False):
raise RPCError(BAD_REQUEST, f'"verbose" must be a boolean')
return await self.daemon_request('getrawtransaction', tx_hash, int(verbose))
return await self.daemon_request('getrawtransaction', tx_hash, verbose)
def _get_merkle_branch(self, tx_hashes, tx_pos):
"""Return a merkle branch to a transaction.
+3 -3
View File
@@ -14,7 +14,7 @@ from lbry.schema.purchase import Purchase
from lbry.schema.support import Support
from .script import InputScript, OutputScript
from .constants import COIN, NULL_HASH32
from .constants import COIN, DUST, NULL_HASH32
from .bcd_data_stream import BCDataStream
from .hash import TXRef, TXRefImmutable
from .util import ReadOnlyList
@@ -818,10 +818,10 @@ class Transaction:
)
if payment > cost:
change = payment - cost
if change > cost_of_change:
change_amount = change - cost_of_change
if change_amount > DUST:
change_address = await change_account.change.get_or_create_usable_address()
change_hash160 = change_account.ledger.address_to_hash160(change_address)
change_amount = change - cost_of_change
change_output = Output.pay_pubkey_hash(change_amount, change_hash160)
change_output.is_internal_transfer = True
tx.add_outputs([Output.pay_pubkey_hash(change_amount, change_hash160)])
+2 -1
View File
@@ -203,11 +203,12 @@ class Wallet:
return True
return False
def unlock(self, password):
async def unlock(self, password):
for account in self.accounts:
if account.encrypted:
if not account.decrypt(password):
return False
await account.deterministic_channel_keys.ensure_cache_primed()
self.encryption_password = password
return True
+9 -4
View File
@@ -40,17 +40,22 @@ def checkrecord(record, expected_winner, expected_claim):
async def checkcontrolling(daemon: Daemon, db: SQLDB):
records, names, futs = [], [], []
records, claim_ids, names, futs = [], [], [], []
for record in db.get_claims('claimtrie.claim_hash as is_controlling, claim.*', is_controlling=True):
records.append(record)
claim_id = hex_reverted(record['claim_hash'])
names.append((record['normalized'], (claim_id,), "", True)) # last parameter is IncludeValues
claim_ids.append((claim_id,))
names.append((record['normalized'],))
if len(names) > 50000:
futs.append(daemon._send_vector('getclaimsfornamebyid', names))
futs.append(daemon._send_vector('getvalueforname', names[:]))
futs.append(daemon._send_vector('getclaimbyid', claim_ids[:]))
names.clear()
claim_ids.clear()
if names:
futs.append(daemon._send_vector('getclaimsfornamebyid', names))
futs.append(daemon._send_vector('getvalueforname', names[:]))
futs.append(daemon._send_vector('getclaimbyid', claim_ids[:]))
names.clear()
claim_ids.clear()
while futs:
winners, claims = futs.pop(0), futs.pop(0)
+6 -7
View File
@@ -7,11 +7,9 @@ BASE = os.path.dirname(__file__)
with open(os.path.join(BASE, 'README.md'), encoding='utf-8') as fh:
long_description = fh.read()
ROCKSDB = []
if sys.platform.startswith('linux') or sys.platform.startswith('darwin'):
ROCKSDB.append('lbry-rocksdb==0.8.1')
PLYVEL = []
if sys.platform.startswith('linux'):
PLYVEL.append('plyvel==1.3.0')
setup(
name=__name__,
@@ -58,8 +56,9 @@ setup(
'attrs==18.2.0',
'pylru==1.1.0',
'elasticsearch==7.10.1',
'grpcio==1.38.0'
] + ROCKSDB,
'grpcio==1.38.0',
'filetype==1.0.9'
] + PLYVEL,
extras_require={
'torrent': ['lbry-libtorrent'],
'lint': ['pylint==2.10.0'],
-47
View File
@@ -1,47 +0,0 @@
#! python
import os
import shutil
import rocksdb
import tempfile
import logging
log = logging.getLogger()
log.addHandler(logging.StreamHandler())
log.setLevel(logging.INFO)
def _main(db_loc):
opts = rocksdb.Options(create_if_missing=True)
db = rocksdb.DB(os.path.join(db_loc, "test"), opts)
secondary_location = os.path.join(db_loc, "secondary")
secondary = rocksdb.DB(
os.path.join(db_loc, "test"),
rocksdb.Options(create_if_missing=True, max_open_files=-1),
secondary_name=secondary_location
)
try:
assert secondary.get(b"a") is None
db.put(b"a", b"b")
assert db.get(b"a") == b"b"
assert secondary.get(b"a") is None
secondary.try_catch_up_with_primary()
assert secondary.get(b"a") == b"b"
finally:
secondary.close()
db.close()
def main():
db_dir = tempfile.mkdtemp()
try:
_main(db_dir)
log.info("rocksdb %s (%s) works!", rocksdb.__version__, rocksdb.ROCKSDB_VERSION)
except:
log.exception("boom")
finally:
shutil.rmtree(db_dir)
if __name__ == "__main__":
main()
+8 -18
View File
@@ -9,7 +9,7 @@ if typing.TYPE_CHECKING:
def get_time_accelerator(loop: asyncio.AbstractEventLoop,
now: typing.Optional[float] = None) -> typing.Callable[[float], typing.Awaitable[None]]:
instant_step: bool = False) -> typing.Callable[[float], typing.Awaitable[None]]:
"""
Returns an async advance() function
@@ -17,32 +17,22 @@ def get_time_accelerator(loop: asyncio.AbstractEventLoop,
made by call_later, call_at, and call_soon.
"""
_time = now or loop.time()
loop.time = functools.wraps(loop.time)(lambda: _time)
original = loop.time
_drift = 0
loop.time = functools.wraps(loop.time)(lambda: original() + _drift)
async def accelerate_time(seconds: float) -> None:
nonlocal _time
nonlocal _drift
if seconds < 0:
raise ValueError(f'Cannot go back in time ({seconds} seconds)')
_time += seconds
await past_events()
_drift += seconds
await asyncio.sleep(0)
async def past_events() -> None:
while loop._scheduled:
timer: asyncio.TimerHandle = loop._scheduled[0]
if timer not in loop._ready and timer._when <= _time:
loop._scheduled.remove(timer)
loop._ready.append(timer)
if timer._when > _time:
break
await asyncio.sleep(0)
async def accelerator(seconds: float):
steps = seconds * 10.0
steps = seconds * 10.0 if not instant_step else 1
for _ in range(max(int(steps), 1)):
await accelerate_time(0.1)
await accelerate_time(seconds/steps)
return accelerator
@@ -103,7 +103,7 @@ class AccountManagement(CommandTestCase):
second_account = await self.daemon.jsonrpc_account_create('second account')
tx = await self.daemon.jsonrpc_account_send(
'0.05', await self.daemon.jsonrpc_address_unused(account_id=second_account.id), blocking=True
'0.05', await self.daemon.jsonrpc_address_unused(account_id=second_account.id)
)
await self.confirm_tx(tx.id)
await self.assertOutputAmount(['0.05', '9.949876'], utxo_list())
@@ -112,7 +112,7 @@ class BlockchainReorganizationTests(CommandTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -191,7 +191,7 @@ class BlockchainReorganizationTests(CommandTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
+9 -6
View File
@@ -101,7 +101,12 @@ class ReconnectTests(IntegrationTestCase):
self.ledger.network.client.transport.close()
self.assertFalse(self.ledger.network.is_connected)
await self.ledger.resolve([], 'derp')
sendtxid = await self.send_to_address_and_wait(address1, 1.1337, 1)
sendtxid = await self.blockchain.send_to_address(address1, 1.1337)
# await self.ledger.resolve([], 'derp')
# self.assertTrue(self.ledger.network.is_connected)
await asyncio.wait_for(self.on_transaction_id(sendtxid), 10.0) # mempool
await self.blockchain.generate(1)
await self.on_transaction_id(sendtxid) # confirmed
self.assertLess(self.ledger.network.client.response_time, 1) # response time properly set lower, we are fine
await self.assertBalance(self.account, '1.1337')
@@ -130,7 +135,7 @@ class ReconnectTests(IntegrationTestCase):
await self.conductor.spv_node.stop()
self.assertFalse(self.ledger.network.is_connected)
await asyncio.sleep(0.2) # let it retry and fail once
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.conductor.spv_node.start(self.conductor.blockchain_node)
await self.ledger.network.on_connected.first
self.assertTrue(self.ledger.network.is_connected)
@@ -160,10 +165,8 @@ class UDPServerFailDiscoveryTest(AsyncioTestCase):
async def test_wallet_connects_despite_lack_of_udp(self):
conductor = Conductor()
conductor.spv_node.udp_port = '0'
await conductor.start_lbcd()
self.addCleanup(conductor.stop_lbcd)
await conductor.start_lbcwallet()
self.addCleanup(conductor.stop_lbcwallet)
await conductor.start_blockchain()
self.addCleanup(conductor.stop_blockchain)
await conductor.start_spv()
self.addCleanup(conductor.stop_spv)
self.assertFalse(conductor.spv_node.server.bp.status_server.is_running)
@@ -174,7 +174,8 @@ class PurchaseCommandTests(CommandTestCase):
self.merchant_address = await self.account.receiving.get_or_create_usable_address()
daemon2 = await self.add_daemon()
address2 = await daemon2.wallet_manager.default_account.receiving.get_or_create_usable_address()
await self.send_to_address_and_wait(address2, 2, 1, ledger=daemon2.ledger)
sendtxid = await self.blockchain.send_to_address(address2, 2)
await self.confirm_tx(sendtxid, daemon2.ledger)
stream = await self.priced_stream('a', '1.0')
await self.assertBalance(self.account, '9.987893')
@@ -23,7 +23,7 @@ class WalletCommands(CommandTestCase):
async def test_wallet_syncing_status(self):
address = await self.daemon.jsonrpc_address_unused()
self.assertFalse(self.daemon.jsonrpc_wallet_status()['is_syncing'])
await self.send_to_address_and_wait(address, 1)
await self.blockchain.send_to_address(address, 1)
await self.ledger._update_tasks.started.wait()
self.assertTrue(self.daemon.jsonrpc_wallet_status()['is_syncing'])
await self.ledger._update_tasks.done.wait()
@@ -49,7 +49,7 @@ class WalletCommands(CommandTestCase):
self.assertEqual(status['wallet']['servers'][0]['port'], 50002)
await self.conductor.spv_node.stop(True)
self.conductor.spv_node.port = 54320
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.conductor.spv_node.start(self.conductor.blockchain_node)
status = await self.daemon.jsonrpc_status()
self.assertEqual(len(status['wallet']['servers']), 0)
self.daemon.jsonrpc_settings_set('lbryum_servers', ['localhost:54320'])
@@ -59,22 +59,23 @@ class WalletCommands(CommandTestCase):
self.assertEqual(status['wallet']['servers'][0]['port'], 54320)
async def test_sending_to_scripthash_address(self):
bal = await self.blockchain.get_balance()
self.assertEqual(await self.blockchain.get_balance(), '95.99973580')
await self.assertBalance(self.account, '10.0')
p2sh_address1 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
tx = await self.account_send('2.0', p2sh_address1)
self.assertEqual(tx['outputs'][0]['address'], p2sh_address1)
self.assertEqual(await self.blockchain.get_balance(), str(float(bal)+3)) # +1 lbc for confirm block
self.assertEqual(await self.blockchain.get_balance(), '98.99973580') # +1 lbc for confirm block
await self.assertBalance(self.account, '7.999877')
await self.wallet_send('3.0', p2sh_address1)
self.assertEqual(await self.blockchain.get_balance(), str(float(bal)+7)) # +1 lbc for confirm block
self.assertEqual(await self.blockchain.get_balance(), '102.99973580') # +1 lbc for confirm block
await self.assertBalance(self.account, '4.999754')
async def test_balance_caching(self):
account2 = await self.daemon.jsonrpc_account_create("Tip-er")
address2 = await self.daemon.jsonrpc_address_unused(account2.id)
await self.send_to_address_and_wait(address2, 10, 2)
await self.ledger.tasks_are_done() # don't mess with the query count while we need it
sendtxid = await self.blockchain.send_to_address(address2, 10)
await self.confirm_tx(sendtxid)
await self.generate(1)
wallet_balance = self.daemon.jsonrpc_wallet_balance
ledger = self.ledger
@@ -89,16 +90,14 @@ class WalletCommands(CommandTestCase):
self.assertIsNone(ledger._balance_cache.get(self.account.id))
query_count += 2
balance = await wallet_balance()
self.assertEqual(await wallet_balance(), expected)
self.assertEqual(self.ledger.db.db.query_count, query_count)
self.assertEqual(balance, expected)
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(self.account.id))['total'], '10.0')
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(account2.id))['total'], '10.0')
# calling again uses cache
balance = await wallet_balance()
self.assertEqual(await wallet_balance(), expected)
self.assertEqual(self.ledger.db.db.query_count, query_count)
self.assertEqual(balance, expected)
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(self.account.id))['total'], '10.0')
self.assertEqual(dict_values_to_lbc(ledger._balance_cache.get(account2.id))['total'], '10.0')
@@ -124,7 +123,8 @@ class WalletCommands(CommandTestCase):
wallet2 = await self.daemon.jsonrpc_wallet_create('foo', create_account=True)
account3 = wallet2.default_account
address3 = await self.daemon.jsonrpc_address_unused(account3.id, wallet2.id)
await self.send_to_address_and_wait(address3, 1, 1)
await self.confirm_tx(await self.blockchain.send_to_address(address3, 1))
await self.generate(1)
account_balance = self.daemon.jsonrpc_account_balance
wallet_balance = self.daemon.jsonrpc_wallet_balance
@@ -154,7 +154,7 @@ class WalletCommands(CommandTestCase):
address2 = await self.daemon.jsonrpc_address_unused(account2.id)
# send lbc to someone else
tx = await self.daemon.jsonrpc_account_send('1.0', address2, blocking=True)
tx = await self.daemon.jsonrpc_account_send('1.0', address2)
await self.confirm_tx(tx.id)
self.assertEqual(await account_balance(), {
'total': '8.97741',
@@ -187,7 +187,7 @@ class WalletCommands(CommandTestCase):
})
# tip claimed
tx = await self.daemon.jsonrpc_support_abandon(txid=support1['txid'], nout=0, blocking=True)
tx = await self.daemon.jsonrpc_support_abandon(txid=support1['txid'], nout=0)
await self.confirm_tx(tx.id)
self.assertEqual(await account_balance(), {
'total': '9.277303',
@@ -238,7 +238,8 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
"carbon smart garage balance margin twelve"
)
address = (await self.daemon2.wallet_manager.default_account.receiving.get_addresses(limit=1, only_usable=True))[0]
await self.send_to_address_and_wait(address, 1, 1, ledger=self.daemon2.ledger)
sendtxid = await self.blockchain.send_to_address(address, 1)
await self.confirm_tx(sendtxid, self.daemon2.ledger)
def assertWalletEncrypted(self, wallet_path, encrypted):
with open(wallet_path) as opened:
@@ -293,7 +294,7 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
'3056301006072a8648ce3d020106052b8104000a034200049ae7283f3f6723e0a1'
'66b7e19e1d1167f6dc5f4af61b4a58066a0d2a8bed2b35c66bccb4ec3eba316b16'
'a97a6d6a4a8effd29d748901bb9789352519cd00b13d'
), self.daemon2, blocking=True)
), self.daemon2)
await self.confirm_tx(channel['txid'], self.daemon2.ledger)
# both daemons will have the channel but only one has the cert so far
@@ -326,7 +327,7 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
with self.assertRaisesRegex(AssertionError, "Cannot lock an unencrypted wallet, encrypt first."):
daemon.jsonrpc_wallet_lock()
# safe to call unlock and decrypt, they are no-ops at this point
daemon.jsonrpc_wallet_unlock('password') # already unlocked
await daemon.jsonrpc_wallet_unlock('password') # already unlocked
daemon.jsonrpc_wallet_decrypt() # already not encrypted
daemon.jsonrpc_wallet_encrypt('password')
@@ -342,7 +343,7 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
# can't sign transactions with locked wallet
with self.assertRaises(AssertionError):
await daemon.jsonrpc_channel_create('@foo', '1.0')
daemon.jsonrpc_wallet_unlock('password')
await daemon.jsonrpc_wallet_unlock('password')
self.assertEqual(daemon.jsonrpc_wallet_status(), {'is_locked': False, 'is_encrypted': True,
'is_syncing': False})
await daemon.jsonrpc_channel_create('@foo', '1.0')
@@ -360,10 +361,17 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
await daemon2.jsonrpc_channel_import(exported)
self.assertTrue(daemon2.jsonrpc_wallet_encrypt('password'))
self.assertTrue(daemon2.jsonrpc_wallet_lock())
self.assertTrue(daemon2.jsonrpc_wallet_unlock("password"))
self.assertTrue(await daemon2.jsonrpc_wallet_unlock("password"))
self.assertEqual(daemon2.jsonrpc_wallet_status(),
{'is_locked': False, 'is_encrypted': True, 'is_syncing': False})
async def test_locking_unlocking_does_not_break_deterministic_channels(self):
self.assertTrue(self.daemon.jsonrpc_wallet_encrypt("password"))
self.assertTrue(self.daemon.jsonrpc_wallet_lock())
self.account.deterministic_channel_keys._private_key = None
self.assertTrue(await self.daemon.jsonrpc_wallet_unlock("password"))
await self.channel_create()
async def test_sync_with_encryption_and_password_change(self):
daemon, daemon2 = self.daemon, self.daemon2
wallet, wallet2 = daemon.wallet_manager.default_wallet, daemon2.wallet_manager.default_wallet
@@ -390,8 +398,8 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
# check new password is active
daemon.jsonrpc_wallet_lock()
self.assertFalse(daemon.jsonrpc_wallet_unlock('password'))
self.assertTrue(daemon.jsonrpc_wallet_unlock('password2'))
self.assertFalse(await daemon.jsonrpc_wallet_unlock('password'))
self.assertTrue(await daemon.jsonrpc_wallet_unlock('password2'))
# propagate disk encryption to daemon2
data = await daemon.jsonrpc_sync_apply('password3')
@@ -407,4 +415,4 @@ class WalletEncryptionAndSynchronization(CommandTestCase):
self.assertWalletEncrypted(wallet2.storage.path, True)
daemon2.jsonrpc_wallet_lock()
self.assertTrue(daemon2.jsonrpc_wallet_unlock('password3'))
self.assertTrue(await daemon2.jsonrpc_wallet_unlock('password3'))
@@ -405,6 +405,17 @@ class ClaimSearchCommand(ClaimTestCase):
not_channel_ids=[chan2_id], has_channel_signature=True, valid_channel_signature=True)
await match([], not_channel_ids=[chan1_id, chan2_id], has_channel_signature=True, valid_channel_signature=True)
async def test_no_source_and_valid_channel_signature_and_media_type(self):
await self.channel_create('@spam2', '1.0')
await self.stream_create('barrrrrr', '1.0', channel_name='@spam2', file_path=self.video_file_name)
paradox_no_source_claims = await self.claim_search(has_no_source=True, valid_channel_signature=True,
media_type="video/mp4")
mp4_claims = await self.claim_search(media_type="video/mp4")
no_source_claims = await self.claim_search(has_no_source=True, valid_channel_signature=True)
self.assertEqual(0, len(paradox_no_source_claims))
self.assertEqual(1, len(no_source_claims))
self.assertEqual(1, len(mp4_claims))
async def test_limit_claims_per_channel(self):
match = self.assertFindsClaims
chan1_id = self.get_claim_id(await self.channel_create('@chan1'))
@@ -494,7 +505,8 @@ class ClaimSearchCommand(ClaimTestCase):
tx = await Transaction.claim_create(
'unknown', b'{"sources":{"lbry_sd_hash":""}}', 1, address, [self.account], self.account)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
octet = await self.stream_create()
video = await self.stream_create('chrome', file_path=self.video_file_name)
@@ -1225,7 +1237,7 @@ class ChannelCommands(CommandTestCase):
data_to_sign = "CAFEBABE"
# claim new name
await self.channel_create('@someotherchan')
channel_tx = await self.daemon.jsonrpc_channel_create('@signer', '0.1', blocking=True)
channel_tx = await self.daemon.jsonrpc_channel_create('@signer', '0.1')
await self.confirm_tx(channel_tx.id)
channel = channel_tx.outputs[0]
signature1 = await self.out(self.daemon.jsonrpc_channel_sign(channel_name='@signer', hexdata=data_to_sign))
@@ -1372,7 +1384,7 @@ class StreamCommands(ClaimTestCase):
self.assertEqual('8.989893', (await self.daemon.jsonrpc_account_balance())['available'])
result = await self.out(self.daemon.jsonrpc_account_send(
'5.0', await self.daemon.jsonrpc_address_unused(account2_id), blocking=True
'5.0', await self.daemon.jsonrpc_address_unused(account2_id)
))
await self.confirm_tx(result['txid'])
@@ -1498,6 +1510,11 @@ class StreamCommands(ClaimTestCase):
self.assertEqual(resolved['@reposting-goodies/repost-on-channel'], search)
self.assertEqual(resolved['newstuff-again']['reposted_claim']['name'], 'newstuff')
await self.stream_update(repost_id, bid='0.42')
searched_repost = (await self.claim_search(claim_id=repost_id))[0]
self.assertEqual(searched_repost['amount'], '0.42')
self.assertEqual(searched_repost['signing_channel']['claim_id'], spam_claim_id)
async def test_filtering_channels_for_removing_content(self):
await self.channel_create('@some_channel', '0.1')
await self.stream_create('good_content', '0.1', channel_name='@some_channel', tags=['good'])
@@ -1554,7 +1571,7 @@ class StreamCommands(ClaimTestCase):
)
# test setting from env vars and starting from scratch
await self.conductor.spv_node.stop(False)
await self.conductor.spv_node.start(self.conductor.lbcwallet_node,
await self.conductor.spv_node.start(self.conductor.blockchain_node,
extraconf={'BLOCKING_CHANNEL_IDS': blocking_channel_id,
'FILTERING_CHANNEL_IDS': filtering_channel_id})
await self.daemon.wallet_manager.reset()
@@ -2171,7 +2188,7 @@ class SupportCommands(CommandTestCase):
tip = await self.out(
self.daemon.jsonrpc_support_create(
claim_id, '1.0', True, account_id=account2.id, wallet_id='wallet2',
funding_account_ids=[account2.id], blocking=True)
funding_account_ids=[account2.id])
)
await self.confirm_tx(tip['txid'])
@@ -2203,7 +2220,7 @@ class SupportCommands(CommandTestCase):
support = await self.out(
self.daemon.jsonrpc_support_create(
claim_id, '2.0', False, account_id=account2.id, wallet_id='wallet2',
funding_account_ids=[account2.id], blocking=True)
funding_account_ids=[account2.id])
)
await self.confirm_tx(support['txid'])
@@ -1,4 +1,3 @@
import unittest
from unittest import skipIf
import asyncio
import os
@@ -37,7 +36,8 @@ class FileCommands(CommandTestCase):
tx_to_update.outputs[0], claim, 1, address, [self.account], self.account
)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
self.client_session = self.daemon.file_manager.source_managers['torrent'].torrent_session
self.client_session._session.add_dht_node(('localhost', 4040))
self.client_session.wait_start = False # fixme: this is super slow on tests
@@ -107,18 +107,12 @@ class FileCommands(CommandTestCase):
self.assertEqual(await self.daemon.storage.get_blobs_to_announce(), [])
await self.stream_create('foo', '0.01')
stream = (await self.daemon.jsonrpc_file_list())["items"][0]
self.assertSetEqual(
set(await self.daemon.storage.get_blobs_to_announce()),
{stream.sd_hash, stream.descriptor.blobs[0].blob_hash}
)
self.assertSetEqual(set(await self.daemon.storage.get_blobs_to_announce()), {stream.sd_hash})
self.assertTrue(await self.daemon.jsonrpc_file_delete(delete_all=True))
# announces on download
self.assertEqual(await self.daemon.storage.get_blobs_to_announce(), [])
stream = await self.daemon.jsonrpc_get('foo')
self.assertSetEqual(
set(await self.daemon.storage.get_blobs_to_announce()),
{stream.sd_hash, stream.descriptor.blobs[0].blob_hash}
)
self.assertSetEqual(set(await self.daemon.storage.get_blobs_to_announce()), {stream.sd_hash})
async def _purge_file(self, claim_name, full_path):
self.assertTrue(
@@ -222,7 +216,6 @@ class FileCommands(CommandTestCase):
await self.wait_files_to_complete()
self.assertNotEqual(first_path, second_path)
@unittest.SkipTest # FIXME: claimname/updateclaim is gone. #3480 wip, unblock #3479"
async def test_file_list_updated_metadata_on_resolve(self):
await self.stream_create('foo', '0.01')
txo = (await self.daemon.resolve(self.wallet.accounts, ['lbry://foo']))['lbry://foo']
@@ -511,7 +504,8 @@ class FileCommands(CommandTestCase):
tx.outputs[0].claim.stream.fee.address_bytes = b''
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
async def __raw_value_update_no_fee_amount(self, tx, claim_address):
tx = await self.daemon.jsonrpc_stream_update(
@@ -521,7 +515,8 @@ class FileCommands(CommandTestCase):
tx.outputs[0].claim.stream.fee.message.ClearField('amount')
tx.outputs[0].script.generate()
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
class DiskSpaceManagement(CommandTestCase):
@@ -616,6 +611,7 @@ class TestBackgroundDownloaderComponent(CommandTestCase):
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
background_downloader = BackgroundDownloader(self.daemon.conf, self.daemon.storage, self.daemon.blob_manager)
self.daemon.conf.network_storage_limit = 32
await self.clear()
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['total_used_mb'])
@@ -624,6 +620,7 @@ class TestBackgroundDownloaderComponent(CommandTestCase):
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['content_blobs_storage_used_mb'])
self.assertEqual(32, (await self.status())['disk_space']['seed_blobs_storage_used_mb'])
self.daemon.conf.network_storage_limit = 48
await background_downloader.download_blobs(content2)
await self.assertBlobs(content1, content2)
await self.blob_clean()
@@ -653,3 +650,7 @@ class TestBackgroundDownloaderComponent(CommandTestCase):
self.daemon.conf.blob_storage_limit = 1
await self.blob_clean()
await self.assertBlobs(content1, no_files=False)
self.daemon.conf.network_storage_limit = 0
await self.blob_clean()
self.assertEqual(0, (await self.status())['disk_space']['seed_blobs_storage_used_mb'])
+6 -6
View File
@@ -80,7 +80,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# After some soul searching Chris decides that his story needs more
# heart and a better ending. He takes down the story and begins the rewrite.
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(claim_id, blocking=True))
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(claim_id, blocking=False))
self.assertEqual(abandon['inputs'][0]['claim_id'], claim_id)
await self.confirm_tx(abandon['txid'])
@@ -103,7 +103,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# 1 LBC to which Chris readily obliges
ramsey_account_id = (await self.out(self.daemon.jsonrpc_account_create("Ramsey")))['id']
ramsey_address = await self.daemon.jsonrpc_address_unused(ramsey_account_id)
result = await self.out(self.daemon.jsonrpc_account_send('1.0', ramsey_address, blocking=True))
result = await self.out(self.daemon.jsonrpc_account_send('1.0', ramsey_address))
self.assertIn("txid", result)
await self.confirm_tx(result['txid'])
@@ -133,7 +133,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# And voila, and bravo and encore! His Best Friend Ramsey read the story and immediately knew this was a hit
# Now to keep this claim winning on the lbry blockchain he immediately supports the claim
tx = await self.out(self.daemon.jsonrpc_support_create(
claim_id2, '0.2', account_id=ramsey_account_id, blocking=True
claim_id2, '0.2', account_id=ramsey_account_id
))
await self.confirm_tx(tx['txid'])
@@ -147,7 +147,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# Now he also wanted to support the original creator of the Award Winning Novel
# So he quickly decides to send a tip to him
tx = await self.out(
self.daemon.jsonrpc_support_create(claim_id2, '0.3', tip=True, account_id=ramsey_account_id, blocking=True)
self.daemon.jsonrpc_support_create(claim_id2, '0.3', tip=True, account_id=ramsey_account_id)
)
await self.confirm_tx(tx['txid'])
@@ -158,7 +158,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
await self.generate(5)
# Seeing the ravishing success of his novel Chris adds support to his claim too
tx = await self.out(self.daemon.jsonrpc_support_create(claim_id2, '0.4', blocking=True))
tx = await self.out(self.daemon.jsonrpc_support_create(claim_id2, '0.4'))
await self.confirm_tx(tx['txid'])
# And check if his support showed up
@@ -183,7 +183,7 @@ class EpicAdventuresOfChris45(CommandTestCase):
# But sadly Ramsey wasn't so pleased. It was hard for him to tell Chris...
# Chris, though a bit heartbroken, abandoned the claim for now, but instantly started working on new hit lyrics
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(txid=tx['txid'], nout=0, blocking=True))
abandon = await self.out(self.daemon.jsonrpc_stream_abandon(txid=tx['txid'], nout=0, blocking=False))
self.assertTrue(abandon['inputs'][0]['txid'], tx['txid'])
await self.confirm_tx(abandon['txid'])
@@ -1,7 +1,6 @@
import asyncio
import json
import hashlib
import sys
from bisect import bisect_right
from binascii import hexlify, unhexlify
from collections import defaultdict
@@ -32,13 +31,13 @@ class BaseResolveTestCase(CommandTestCase):
self.assertEqual(claim_from_es['effective_amount'], claim_from_db.effective_amount)
def assertMatchDBClaim(self, expected, claim):
self.assertEqual(expected['claimid'], claim.claim_hash.hex())
self.assertEqual(expected['validatheight'], claim.activation_height)
self.assertEqual(expected['lasttakeoverheight'], claim.last_takeover_height)
self.assertEqual(expected['txid'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['claimId'], claim.claim_hash.hex())
self.assertEqual(expected['validAtHeight'], claim.activation_height)
self.assertEqual(expected['lastTakeoverHeight'], claim.last_takeover_height)
self.assertEqual(expected['txId'], claim.tx_hash[::-1].hex())
self.assertEqual(expected['n'], claim.position)
self.assertEqual(expected['amount'], claim.amount)
self.assertEqual(expected['effectiveamount'], claim.effective_amount)
self.assertEqual(expected['effectiveAmount'], claim.effective_amount)
async def assertResolvesToClaimId(self, name, claim_id):
other = await self.resolve(name)
@@ -54,10 +53,9 @@ class BaseResolveTestCase(CommandTestCase):
self.assertEqual(claim_id, claim_from_es[0][0]['claim_hash'][::-1].hex())
async def assertNoClaimForName(self, name: str):
lbrycrd_winning = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name))
lbrycrd_winning = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.bp.db.resolve(name)
if 'claims' in lbrycrd_winning and lbrycrd_winning['claims'] is not None:
self.assertEqual(len(lbrycrd_winning['claims']), 0)
self.assertNotIn('claimId', lbrycrd_winning)
if stream is not None:
self.assertIsInstance(stream, LookupError)
else:
@@ -65,23 +63,20 @@ class BaseResolveTestCase(CommandTestCase):
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(name=name)
self.assertListEqual([], claim_from_es[0])
async def assertNoClaim(self, name: str, claim_id: str):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebyid', name, '["' + claim_id + '"]'))
if 'claims' in expected and expected['claims'] is not None:
# ensure that if we do have the matching claim that it is not active
self.assertEqual(expected['claims'][0]['effectiveamount'], 0)
async def assertNoClaim(self, claim_id: str):
self.assertDictEqual(
{}, json.loads(await self.blockchain._cli_cmnd('getclaimbyid', claim_id))
)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(claim_id=claim_id)
self.assertListEqual([], claim_from_es[0])
claim = await self.conductor.spv_node.server.bp.db.fs_getclaimbyid(claim_id)
self.assertIsNone(claim)
async def assertMatchWinningClaim(self, name):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebybid', name, "[0]"))
expected = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))
stream, channel, _, _ = await self.conductor.spv_node.server.bp.db.resolve(name)
claim = stream if stream else channel
expected['claims'][0]['lasttakeoverheight'] = expected['lasttakeoverheight']
await self._assertMatchClaim(expected['claims'][0], claim)
await self._assertMatchClaim(expected, claim)
return claim
async def _assertMatchClaim(self, expected, claim):
@@ -91,81 +86,70 @@ class BaseResolveTestCase(CommandTestCase):
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertMatchESClaim(claim_from_es[0][0], claim)
self._check_supports(claim.claim_hash.hex(), expected.get('supports', []),
claim_from_es[0][0]['support_amount'])
self._check_supports(claim.claim_hash.hex(), expected['supports'], claim_from_es[0][0]['support_amount'])
async def assertMatchClaim(self, name, claim_id, is_active_in_lbrycrd=True):
async def assertMatchClaim(self, claim_id, is_active_in_lbrycrd=True):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimbyid', claim_id))
claim = await self.conductor.spv_node.server.bp.db.fs_getclaimbyid(claim_id)
if is_active_in_lbrycrd:
if not expected:
self.assertIsNone(claim)
return
self.assertMatchDBClaim(expected, claim)
else:
self.assertDictEqual({}, expected)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=claim.claim_hash.hex()
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim.claim_hash.hex())
self.assertMatchESClaim(claim_from_es[0][0], claim)
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsfornamebyid', name, '["' + claim_id + '"]'))
if is_active_in_lbrycrd:
if not expected:
self.assertIsNone(claim)
return
expected['claims'][0]['lasttakeoverheight'] = expected['lasttakeoverheight']
self.assertMatchDBClaim(expected['claims'][0], claim)
self._check_supports(claim.claim_hash.hex(), expected['claims'][0].get('supports', []),
claim_from_es[0][0]['support_amount'])
else:
if 'claims' in expected and expected['claims'] is not None:
# ensure that if we do have the matching claim that it is not active
self.assertEqual(expected['claims'][0]['effectiveamount'], 0)
self._check_supports(
claim.claim_hash.hex(), expected.get('supports', []), claim_from_es[0][0]['support_amount'],
is_active_in_lbrycrd
)
return claim
async def assertMatchClaimIsWinning(self, name, claim_id):
self.assertEqual(claim_id, (await self.assertMatchWinningClaim(name)).claim_hash.hex())
await self.assertMatchClaimsForName(name)
def _check_supports(self, claim_id, lbrycrd_supports, es_support_amount):
total_lbrycrd_amount = 0.0
total_es_amount = 0.0
active_es_amount = 0.0
def _check_supports(self, claim_id, lbrycrd_supports, es_support_amount, is_active_in_lbrycrd=True):
total_amount = 0
db = self.conductor.spv_node.server.bp.db
es_supports = db.get_supports(bytes.fromhex(claim_id))
# we're only concerned about active supports here, and they should match
self.assertTrue(len(es_supports) >= len(lbrycrd_supports))
for i, (tx_num, position, amount) in enumerate(es_supports):
total_es_amount += amount
valid_height = db.get_activation(tx_num, position, is_support=True)
if valid_height > db.db_height:
continue
active_es_amount += amount
txid = db.prefix_db.tx_hash.get(tx_num, deserialize_value=False)[::-1].hex()
support = next(filter(lambda s: s['txid'] == txid and s['n'] == position, lbrycrd_supports))
total_lbrycrd_amount += support['amount']
self.assertEqual(support['height'], bisect_right(db.tx_counts, tx_num))
self.assertEqual(support['validatheight'], valid_height)
self.assertEqual(total_es_amount, es_support_amount)
self.assertEqual(active_es_amount, total_lbrycrd_amount)
for i, (tx_num, position, amount) in enumerate(db.get_supports(bytes.fromhex(claim_id))):
total_amount += amount
if is_active_in_lbrycrd:
support = lbrycrd_supports[i]
self.assertEqual(support['txId'], db.prefix_db.tx_hash.get(tx_num, deserialize_value=False)[::-1].hex())
self.assertEqual(support['n'], position)
self.assertEqual(support['height'], bisect_right(db.tx_counts, tx_num))
self.assertEqual(support['validAtHeight'], db.get_activation(tx_num, position, is_support=True))
self.assertEqual(total_amount, es_support_amount, f"lbrycrd support amount: {total_amount} vs es: {es_support_amount}")
async def assertMatchClaimsForName(self, name):
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name, "", "true"))
expected = json.loads(await self.blockchain._cli_cmnd('getclaimsforname', name))
db = self.conductor.spv_node.server.bp.db
# self.assertEqual(len(expected['claims']), len(db_claims.claims))
# self.assertEqual(expected['lastTakeoverHeight'], db_claims.lastTakeoverHeight)
last_takeover = json.loads(await self.blockchain._cli_cmnd('getvalueforname', name))['lastTakeoverHeight']
for c in expected['claims']:
c['lasttakeoverheight'] = expected['lasttakeoverheight']
claim_id = c['claimid']
c['lastTakeoverHeight'] = last_takeover
claim_id = c['claimId']
claim_hash = bytes.fromhex(claim_id)
claim = db._fs_get_claim_by_hash(claim_hash)
self.assertMatchDBClaim(c, claim)
claim_from_es = await self.conductor.spv_node.server.bp.db.search_index.search(
claim_id=claim_id
claim_id=c['claimId']
)
self.assertEqual(len(claim_from_es[0]), 1)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), claim_id)
self.assertEqual(claim_from_es[0][0]['claim_hash'][::-1].hex(), c['claimId'])
self.assertMatchESClaim(claim_from_es[0][0], claim)
self._check_supports(claim_id, c.get('supports', []),
claim_from_es[0][0]['support_amount'])
self._check_supports(c['claimId'], c['supports'], claim_from_es[0][0]['support_amount'])
class ResolveCommand(BaseResolveTestCase):
@@ -278,18 +262,18 @@ class ResolveCommand(BaseResolveTestCase):
self.assertEqual(claim['confirmations'], json.loads(tx_details)['confirmations'])
# resolve handles invalid data
# await self.blockchain_claim_name("gibberish", hexlify(b"{'invalid':'json'}").decode(), "0.1")
# await self.generate(1)
# response = await self.out(self.daemon.jsonrpc_resolve("lbry://gibberish"))
# self.assertSetEqual({'lbry://gibberish'}, set(response))
# claim = response['lbry://gibberish']
# self.assertEqual(claim['name'], 'gibberish')
# self.assertNotIn('value', claim)
await self.blockchain_claim_name("gibberish", hexlify(b"{'invalid':'json'}").decode(), "0.1")
await self.generate(1)
response = await self.out(self.daemon.jsonrpc_resolve("lbry://gibberish"))
self.assertSetEqual({'lbry://gibberish'}, set(response))
claim = response['lbry://gibberish']
self.assertEqual(claim['name'], 'gibberish')
self.assertNotIn('value', claim)
# resolve retries
await self.conductor.spv_node.stop()
resolve_task = asyncio.create_task(self.resolve('foo'))
await self.conductor.spv_node.start(self.conductor.lbcwallet_node)
await self.conductor.spv_node.start(self.conductor.blockchain_node)
self.assertIsNotNone((await resolve_task)['claim_id'])
async def test_winning_by_effective_amount(self):
@@ -459,16 +443,16 @@ class ResolveCommand(BaseResolveTestCase):
self.assertEqual(one, claim6['name'])
async def test_resolve_old_claim(self):
channel = await self.daemon.jsonrpc_channel_create('@olds', '1.0', blocking=True)
channel = await self.daemon.jsonrpc_channel_create('@olds', '1.0')
await self.confirm_tx(channel.id)
address = channel.outputs[0].get_address(self.account.ledger)
claim = generate_signed_legacy(address, channel.outputs[0])
tx = await Transaction.claim_create('example', claim.SerializeToString(), 1, address, [self.account], self.account)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
response = await self.resolve('@olds/example')
self.assertTrue('is_channel_signature_valid' in response, str(response))
self.assertTrue(response['is_channel_signature_valid'])
claim.publisherSignature.signature = bytes(reversed(claim.publisherSignature.signature))
@@ -476,7 +460,8 @@ class ResolveCommand(BaseResolveTestCase):
'bad_example', claim.SerializeToString(), 1, address, [self.account], self.account
)
await tx.sign([self.account])
await self.broadcast_and_confirm(tx)
await self.broadcast(tx)
await self.confirm_tx(tx.id)
response = await self.resolve('bad_example')
self.assertFalse(response['is_channel_signature_valid'])
@@ -661,7 +646,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
self.assertEqual(height, self.conductor.spv_node.server.bp.db.db_height)
await self.assertMatchClaimIsWinning(name, winning_claim_id)
for non_winning in non_winning_claims:
claim = await self.assertMatchClaim(name,
claim = await self.assertMatchClaim(
non_winning.claim_id, is_active_in_lbrycrd=non_winning.active_in_lbrycrd
)
self.assertEqual(non_winning.activation_height, claim.activation_height)
@@ -1007,7 +992,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
name = 'test'
await self.generate(494)
address = (await self.account.receiving.get_addresses(True))[0]
await self.send_to_address_and_wait(address, 400.0)
await self.blockchain.send_to_address(address, 400.0)
await self.account.ledger.on_address.first
await self.generate(100)
self.assertEqual(800, self.conductor.spv_node.server.bp.db.db_height)
@@ -1349,7 +1334,7 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
await self.generate(1)
await self.assertMatchClaim(name, first_claim_id)
await self.assertMatchClaim(first_claim_id)
await self.assertMatchClaimIsWinning(name, second_claim_id)
async def test_remove_controlling_support(self):
@@ -1420,12 +1405,12 @@ class ResolveClaimTakeovers(BaseResolveTestCase):
await self.generate(32)
second_claim_id = (await self.stream_create(name, '0.01', allow_duplicate_name=True))['outputs'][0]['claim_id']
await self.assertNoClaim(name, second_claim_id)
await self.assertNoClaim(second_claim_id)
self.assertEqual(
len((await self.conductor.spv_node.server.bp.db.search_index.search(claim_name=name))[0]), 1
)
await self.generate(1)
await self.assertMatchClaim(name, second_claim_id)
await self.assertMatchClaim(second_claim_id)
self.assertEqual(
len((await self.conductor.spv_node.server.bp.db.search_index.search(claim_name=name))[0]), 2
)
@@ -1585,7 +1570,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -1664,7 +1649,7 @@ class ResolveAfterReorg(BaseResolveTestCase):
# reorg the last block dropping our claim tx
await self.blockchain.invalidate_block(invalidated_block_hash)
await self.conductor.clear_mempool()
await self.blockchain.clear_mempool()
await self.blockchain.generate(2)
# wait for the client to catch up and verify the reorg
@@ -21,8 +21,9 @@ class BasicTransactionTest(IntegrationTestCase):
[asyncio.ensure_future(self.on_address_update(address1)),
asyncio.ensure_future(self.on_address_update(address2))]
))
await self.send_to_address_and_wait(address1, 5)
await self.send_to_address_and_wait(address2, 5, 1)
sendtxid1 = await self.blockchain.send_to_address(address1, 5)
sendtxid2 = await self.blockchain.send_to_address(address2, 5)
await self.blockchain.generate(1)
await notifications
self.assertEqual(d2l(await self.account.get_balance()), '10.0')
@@ -1,14 +1,23 @@
import unittest
from lbry.testcase import CommandTestCase
class TransactionCommandsTestCase(CommandTestCase):
async def test_txo_dust_prevention(self):
address = await self.daemon.jsonrpc_address_unused(self.account.id)
tx = await self.account_send('9.9997758', address)
# dust prevention threshold not reached, small txo created
self.assertEqual(2, len(tx['outputs']))
self.assertEqual(tx['outputs'][1]['amount'], '0.0001002')
tx = await self.account_send('9.999706', address)
# dust prevention prevented dust
self.assertEqual(1, len(tx['outputs']))
self.assertEqual(tx['outputs'][0]['amount'], '9.999706')
async def test_transaction_show(self):
# local tx
result = await self.out(self.daemon.jsonrpc_account_send(
'5.0', await self.daemon.jsonrpc_address_unused(self.account.id), blocking=True
'5.0', await self.daemon.jsonrpc_address_unused(self.account.id)
))
await self.confirm_tx(result['txid'])
tx = await self.daemon.jsonrpc_transaction_show(result['txid'])
@@ -29,9 +38,10 @@ class TransactionCommandsTestCase(CommandTestCase):
self.assertFalse(result['success'])
async def test_utxo_release(self):
await self.send_to_address_and_wait(
await self.account.receiving.get_or_create_usable_address(), 1, 1
sendtxid = await self.blockchain.send_to_address(
await self.account.receiving.get_or_create_usable_address(), 1
)
await self.confirm_tx(sendtxid)
await self.assertBalance(self.account, '11.0')
await self.ledger.reserve_outputs(await self.account.get_utxos())
await self.assertBalance(self.account, '0.0')
@@ -41,7 +51,6 @@ class TransactionCommandsTestCase(CommandTestCase):
class TestSegwit(CommandTestCase):
@unittest.SkipTest
async def test_segwit(self):
p2sh_address1 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
p2sh_address2 = await self.blockchain.get_new_address(self.blockchain.P2SH_SEGWIT_ADDRESS)
@@ -55,13 +64,14 @@ class TestSegwit(CommandTestCase):
p2sh_txid2 = await self.blockchain.send_to_address(p2sh_address2, '1.0')
bech32_txid1 = await self.blockchain.send_to_address(bech32_address1, '1.0')
bech32_txid2 = await self.blockchain.send_to_address(bech32_address2, '1.0')
await self.generate(1)
# P2SH & BECH32 can pay to P2SH address
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid1, "vout": 0},
{"txid": bech32_txid1, "vout": 0},
], {p2sh_address3: 1.9}
], [{p2sh_address3: '1.9'}]
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
p2sh_txid3 = await self.blockchain.send_raw_transaction(tx)
@@ -72,7 +82,7 @@ class TestSegwit(CommandTestCase):
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid2, "vout": 0},
{"txid": bech32_txid2, "vout": 0},
], {bech32_address3: 1.9}
], [{bech32_address3: '1.9'}]
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
bech32_txid3 = await self.blockchain.send_raw_transaction(tx)
@@ -84,9 +94,12 @@ class TestSegwit(CommandTestCase):
tx = await self.blockchain.create_raw_transaction([
{"txid": p2sh_txid3, "vout": 0},
{"txid": bech32_txid3, "vout": 0},
], {address: 3.5}
], [{address: '3.5'}]
)
tx = await self.blockchain.sign_raw_transaction_with_wallet(tx)
txid = await self.blockchain.send_raw_transaction(tx)
await self.generate_and_wait(1, [txid])
await self.on_transaction_id(txid)
await self.generate(1)
await self.on_transaction_id(txid)
await self.assertBalance(self.account, '13.5')
@@ -1,7 +1,7 @@
import asyncio
import random
from itertools import chain
import lbry.wallet.rpc.jsonrpc
from lbry.wallet.transaction import Transaction, Output, Input
from lbry.testcase import IntegrationTestCase
from lbry.wallet.util import satoshis_to_coins, coins_to_satoshis
@@ -19,10 +19,10 @@ class BasicTransactionTests(IntegrationTestCase):
# to the 10th receiving address for a total of 30 UTXOs on the entire account
for i in range(10):
notification = asyncio.ensure_future(self.on_address_update(addresses[i]))
_ = await self.send_to_address_and_wait(addresses[i], 10)
txid = await self.blockchain.send_to_address(addresses[i], 10)
await notification
notification = asyncio.ensure_future(self.on_address_update(addresses[9]))
_ = await self.send_to_address_and_wait(addresses[9], 10)
txid = await self.blockchain.send_to_address(addresses[9], 10)
await notification
# use batching to reduce issues with send_to_address on cli
@@ -88,10 +88,12 @@ class BasicTransactionTests(IntegrationTestCase):
await self.assertBalance(account2, '0.0')
addresses = await account1.receiving.get_addresses()
txids = []
for address in addresses[:5]:
txids.append(await self.send_to_address_and_wait(address, 1.1))
await self.generate_and_wait(1, txids)
txids = await asyncio.gather(*(
self.blockchain.send_to_address(address, 1.1) for address in addresses[:5]
))
await asyncio.wait([self.on_transaction_id(txid) for txid in txids]) # mempool
await self.blockchain.generate(1)
await asyncio.wait([self.on_transaction_id(txid) for txid in txids]) # confirmed
await self.assertBalance(account1, '5.5')
await self.assertBalance(account2, '0.0')
@@ -146,8 +148,11 @@ class BasicTransactionTests(IntegrationTestCase):
return summary
self.conductor.spv_node.server.bp.mempool.transaction_summaries = random_summary
# 10 unconfirmed txs, all from blockchain wallet
for i in range(10):
await self.send_to_address_and_wait(address, 10)
sends = [self.blockchain.send_to_address(address, 10) for _ in range(10)]
# use batching to reduce issues with send_to_address on cli
for batch in range(0, len(sends), 10):
txids = await asyncio.gather(*sends[batch:batch + 10])
await asyncio.wait([self.on_transaction_id(txid) for txid in txids])
remote_status = await self.ledger.network.subscribe_address(address)
self.assertTrue(await self.ledger.update_history(address, remote_status))
# 20 unconfirmed txs, 10 from blockchain, 10 from local to local
@@ -165,7 +170,8 @@ class BasicTransactionTests(IntegrationTestCase):
remote_status = await self.ledger.network.subscribe_address(address)
self.assertTrue(await self.ledger.update_history(address, remote_status))
# server history grows unordered
await self.send_to_address_and_wait(address, 1)
txid = await self.blockchain.send_to_address(address, 1)
await self.on_transaction_id(txid)
self.assertTrue(await self.ledger.update_history(address, remote_status))
self.assertEqual(21, len((await self.ledger.get_local_status_and_history(address))[1]))
self.assertEqual(0, len(self.ledger._known_addresses_out_of_sync))
@@ -189,7 +195,7 @@ class BasicTransactionTests(IntegrationTestCase):
self.ledger, 2000000000000, [self.account], set_reserved=False, return_insufficient_funds=True
)
got_amounts = [estimator.effective_amount for estimator in spendable]
self.assertListEqual(sorted(amounts), sorted(got_amounts))
self.assertListEqual(amounts, got_amounts)
async def test_sqlite_coin_chooser(self):
wallet_manager = WalletManager([self.wallet], {self.ledger.get_id(): self.ledger})
@@ -200,26 +206,26 @@ class BasicTransactionTests(IntegrationTestCase):
other_account = self.wallet.generate_account(self.ledger)
other_address = await other_account.receiving.get_or_create_usable_address()
self.ledger.coin_selection_strategy = 'sqlite'
await self.ledger.subscribe_account(other_account)
await self.ledger.subscribe_account(self.account)
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 1.0)
txid = await self.blockchain.send_to_address(address, 1.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 1.0)
txid = await self.blockchain.send_to_address(address, 1.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 3.0)
txid = await self.blockchain.send_to_address(address, 3.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 5.0)
txid = await self.blockchain.send_to_address(address, 5.0)
await accepted
accepted = asyncio.ensure_future(self.on_address_update(address))
_ = await self.send_to_address_and_wait(address, 10.0)
txid = await self.blockchain.send_to_address(address, 10.0)
await accepted
await self.assertBalance(self.account, '20.0')
@@ -260,8 +266,6 @@ class BasicTransactionTests(IntegrationTestCase):
async def broadcast(tx):
try:
return await real_broadcast(tx)
except lbry.wallet.rpc.jsonrpc.RPCError:
pass
finally:
e.set()
+74 -59
View File
@@ -1,60 +1,68 @@
import contextlib
import logging
import typing
import binascii
import socket
import asyncio
from lbry.testcase import AsyncioTestCase
from tests import dht_mocks
from lbry.dht.protocol.distance import Distance
from lbry.conf import Config
from lbry.dht import constants
from lbry.dht.node import Node
from lbry.dht.peer import PeerManager, make_kademlia_peer
from lbry.dht.blob_announcer import BlobAnnouncer
from lbry.extras.daemon.storage import SQLiteStorage
from unittest import skip
class TestBlobAnnouncer(AsyncioTestCase):
async def setup_node(self, peer_addresses, address, node_id):
self.nodes: typing.Dict[int, Node] = {}
self.advance = dht_mocks.get_time_accelerator(self.loop, self.loop.time())
self.advance = dht_mocks.get_time_accelerator(self.loop)
self.instant_advance = dht_mocks.get_time_accelerator(self.loop)
self.conf = Config()
self.storage = SQLiteStorage(self.conf, ":memory:", self.loop, self.loop.time)
await self.storage.open()
self.peer_manager = PeerManager(self.loop)
self.node = Node(self.loop, self.peer_manager, node_id, 4444, 4444, 3333, address)
await self.node.start_listening(address)
self.blob_announcer = BlobAnnouncer(self.loop, self.node, self.storage)
for node_id, address in peer_addresses:
await self.add_peer(node_id, address)
await asyncio.gather(*[self.add_peer(node_id, address) for node_id, address in peer_addresses])
for first_peer in self.nodes.values():
for second_peer in self.nodes.values():
if first_peer == second_peer:
continue
self.add_peer_to_routing_table(first_peer, second_peer)
self.add_peer_to_routing_table(second_peer, first_peer)
await self.advance(0.1) # just to make pings go through
self.node.joined.set()
self.node._refresh_task = self.loop.create_task(self.node.refresh_node())
self.storage = SQLiteStorage(self.conf, ":memory:", self.loop, self.loop.time)
await self.storage.open()
self.blob_announcer = BlobAnnouncer(self.loop, self.node, self.storage)
async def add_peer(self, node_id, address, add_to_routing_table=True):
#print('add', node_id.hex()[:8], address)
n = Node(self.loop, PeerManager(self.loop), node_id, 4444, 4444, 3333, address)
await n.start_listening(address)
self.nodes.update({len(self.nodes): n})
if add_to_routing_table:
self.node.protocol.add_peer(
make_kademlia_peer(
n.protocol.node_id, n.protocol.external_ip, n.protocol.udp_port
)
self.add_peer_to_routing_table(self.node, n)
def add_peer_to_routing_table(self, adder, being_added):
adder.protocol.add_peer(
make_kademlia_peer(
being_added.protocol.node_id, being_added.protocol.external_ip, being_added.protocol.udp_port
)
)
@contextlib.asynccontextmanager
async def _test_network_context(self, peer_addresses=None):
self.peer_addresses = peer_addresses or [
(constants.generate_id(2), '1.2.3.2'),
(constants.generate_id(3), '1.2.3.3'),
(constants.generate_id(4), '1.2.3.4'),
(constants.generate_id(5), '1.2.3.5'),
(constants.generate_id(6), '1.2.3.6'),
(constants.generate_id(7), '1.2.3.7'),
(constants.generate_id(8), '1.2.3.8'),
(constants.generate_id(9), '1.2.3.9'),
async def _test_network_context(self, peer_count=200):
self.peer_addresses = [
(constants.generate_id(i), socket.inet_ntoa(int(i + 0x01000001).to_bytes(length=4, byteorder='big')))
for i in range(1, peer_count + 1)
]
try:
with dht_mocks.mock_network_loop(self.loop):
await self.setup_node(self.peer_addresses, '1.2.3.1', constants.generate_id(1))
await self.setup_node(self.peer_addresses, '1.2.3.1', constants.generate_id(1000))
yield
finally:
self.blob_announcer.stop()
@@ -73,43 +81,58 @@ class TestBlobAnnouncer(AsyncioTestCase):
)
)
await peer.ping()
return peer
return last_node
@skip("Something from a previous test is leaking into this test and causing it to fail intermittently")
async def test_announce_blobs(self):
blob1 = binascii.hexlify(b'1' * 48).decode()
blob2 = binascii.hexlify(b'2' * 48).decode()
async with self._test_network_context():
await self.storage.add_blobs((blob1, 1024), (blob2, 1024), finished=True)
await self.storage.db.execute(
"update blob set next_announce_time=0, should_announce=1 where blob_hash in (?, ?)",
(blob1, blob2)
)
async with self._test_network_context(peer_count=100):
await self.storage.add_blobs((blob1, 1024, 0, True), (blob2, 1024, 0, True), finished=True)
await self.storage.add_blobs(
*((constants.generate_id(value).hex(), 1024, 0, True) for value in range(1000, 1090)),
finished=True)
await self.storage.db.execute("update blob set next_announce_time=0, should_announce=1")
to_announce = await self.storage.get_blobs_to_announce()
self.assertEqual(2, len(to_announce))
self.blob_announcer.start(batch_size=1) # so it covers batching logic
self.assertEqual(92, len(to_announce))
self.blob_announcer.start(batch_size=10) # so it covers batching logic
# takes 60 seconds to start, but we advance 120 to ensure it processed all batches
await self.advance(60.0 * 2)
ongoing_announcements = asyncio.ensure_future(self.blob_announcer.wait())
await self.instant_advance(60.0)
await ongoing_announcements
to_announce = await self.storage.get_blobs_to_announce()
self.assertEqual(0, len(to_announce))
self.blob_announcer.stop()
# as routing table pollution will cause some peers to be hard to reach, we add a tolerance for CI
tolerance = 0.8 # at least 80% of the announcements are within the top K
for blob in await self.storage.get_all_blob_hashes():
distance = Distance(bytes.fromhex(blob))
candidates = list(self.nodes.values())
candidates.sort(key=lambda sorting_node: distance(sorting_node.protocol.node_id))
has_it = 0
for index, node in enumerate(candidates[:constants.K], start=1):
if node.protocol.data_store.get_peers_for_blob(bytes.fromhex(blob)):
has_it += 1
else:
logging.warning("blob %s wasnt found between the best K (%s)", blob[:8], node.protocol.node_id.hex()[:8])
self.assertGreaterEqual(has_it, int(tolerance * constants.K))
# test that we can route from a poorly connected peer all the way to the announced blob
await self.chain_peer(constants.generate_id(10), '1.2.3.10')
await self.chain_peer(constants.generate_id(11), '1.2.3.11')
await self.chain_peer(constants.generate_id(12), '1.2.3.12')
await self.chain_peer(constants.generate_id(13), '1.2.3.13')
await self.chain_peer(constants.generate_id(14), '1.2.3.14')
await self.advance(61.0)
current = len(self.nodes)
await self.chain_peer(constants.generate_id(current + 1), '1.2.3.10')
await self.chain_peer(constants.generate_id(current + 2), '1.2.3.11')
await self.chain_peer(constants.generate_id(current + 3), '1.2.3.12')
await self.chain_peer(constants.generate_id(current + 4), '1.2.3.13')
last = await self.chain_peer(constants.generate_id(current + 5), '1.2.3.14')
last = self.nodes[len(self.nodes) - 1]
search_q, peer_q = asyncio.Queue(loop=self.loop), asyncio.Queue(loop=self.loop)
search_q.put_nowait(blob1)
_, task = last.accumulate_peers(search_q, peer_q)
found_peers = await peer_q.get()
found_peers = await asyncio.wait_for(peer_q.get(), 1.0)
task.cancel()
self.assertEqual(1, len(found_peers))
@@ -119,21 +142,13 @@ class TestBlobAnnouncer(AsyncioTestCase):
async def test_popular_blob(self):
peer_count = 150
addresses = [
(constants.generate_id(i + 1), socket.inet_ntoa(int(i + 0x01000001).to_bytes(length=4, byteorder='big')))
for i in range(peer_count)
]
blob_hash = b'1' * 48
blob_hash = constants.generate_id(99999)
async with self._test_network_context(peer_addresses=addresses):
async with self._test_network_context(peer_count=peer_count):
total_seen = set()
announced_to = self.nodes[0]
for i in range(1, peer_count):
node = self.nodes[i]
kad_peer = make_kademlia_peer(
node.protocol.node_id, node.protocol.external_ip, node.protocol.udp_port
)
await announced_to.protocol._add_peer(kad_peer)
announced_to = self.nodes.pop(0)
for i, node in enumerate(self.nodes.values()):
self.add_peer_to_routing_table(announced_to, node)
peer = node.protocol.get_rpc_peer(
make_kademlia_peer(
announced_to.protocol.node_id,
@@ -144,15 +159,15 @@ class TestBlobAnnouncer(AsyncioTestCase):
response = await peer.store(blob_hash)
self.assertEqual(response, b'OK')
peers_for_blob = await peer.find_value(blob_hash, 0)
if i == 1:
if i == 0:
self.assertNotIn(blob_hash, peers_for_blob)
self.assertEqual(peers_for_blob[b'p'], 0)
else:
self.assertEqual(len(peers_for_blob[blob_hash]), min(i - 1, constants.K))
self.assertEqual(len(announced_to.protocol.data_store.get_peers_for_blob(blob_hash)), i)
self.assertEqual(len(peers_for_blob[blob_hash]), min(i, constants.K))
self.assertEqual(len(announced_to.protocol.data_store.get_peers_for_blob(blob_hash)), i + 1)
if i - 1 > constants.K:
self.assertEqual(len(peers_for_blob[b'contacts']), constants.K)
self.assertEqual(peers_for_blob[b'p'], ((i - 1) // (constants.K + 1)) + 1)
self.assertEqual(peers_for_blob[b'p'], (i // (constants.K + 1)) + 1)
seen = set(peers_for_blob[blob_hash])
self.assertEqual(len(seen), constants.K)
self.assertEqual(len(peers_for_blob[blob_hash]), len(seen))
@@ -167,5 +182,5 @@ class TestBlobAnnouncer(AsyncioTestCase):
seen.intersection_update(page_x_set)
total_seen.update(page_x_set)
else:
self.assertEqual(len(peers_for_blob[b'contacts']), i - 1)
self.assertEqual(len(peers_for_blob[b'contacts']), 8) # we always add 8 on first page
self.assertEqual(len(total_seen), peer_count - 2)
+2 -4
View File
@@ -12,7 +12,6 @@ from lbry.extras.daemon.storage import SQLiteStorage
class TestNodePingQueueDiscover(AsyncioTestCase):
TIMEOUT = None # not supported as it advances time
async def test_ping_queue_discover(self):
loop = asyncio.get_event_loop()
loop.set_debug(False)
@@ -29,7 +28,7 @@ class TestNodePingQueueDiscover(AsyncioTestCase):
(constants.generate_id(9), '1.2.3.9'),
]
with dht_mocks.mock_network_loop(loop):
advance = dht_mocks.get_time_accelerator(loop, loop.time())
advance = dht_mocks.get_time_accelerator(loop)
# start the nodes
nodes: typing.Dict[int, Node] = {
i: Node(loop, PeerManager(loop), node_id, 4444, 4444, 3333, address)
@@ -93,7 +92,6 @@ class TestNodePingQueueDiscover(AsyncioTestCase):
class TestTemporarilyLosingConnection(AsyncioTestCase):
TIMEOUT = None # not supported as it advances time
@unittest.SkipTest
async def test_losing_connection(self):
async def wait_for(check_ok, insist, timeout=20):
@@ -131,7 +129,7 @@ class TestTemporarilyLosingConnection(AsyncioTestCase):
await asyncio.gather(*[n.joined.wait() for n in nodes])
node = nodes[-1]
advance = dht_mocks.get_time_accelerator(loop, loop.time())
advance = dht_mocks.get_time_accelerator(loop)
await advance(500)
# Join the network, assert that at least the known peers are in RT
+51
View File
@@ -0,0 +1,51 @@
import unittest
import tempfile
import os
from lbry.schema.mime_types import guess_media_type
class MediaTypeTests(unittest.TestCase):
def test_guess_media_type_from_path_only(self):
kind = guess_media_type('/tmp/test.mkv')
self.assertEqual(kind, ('video/x-matroska', 'video'))
def test_defaults_for_no_extension(self):
kind = guess_media_type('/tmp/test')
self.assertEqual(kind, ('application/octet-stream', 'binary'))
def test_defaults_for_unknown_extension(self):
kind = guess_media_type('/tmp/test.unk')
self.assertEqual(kind, ('application/x-ext-unk', 'binary'))
def test_spoofed_unknown(self):
with tempfile.TemporaryDirectory() as temp_dir:
file = os.path.join(temp_dir, 'spoofed_unknown.txt')
with open(file, 'wb') as fd:
bytes_lz4 = bytearray([0x04,0x22,0x4d,0x18])
fd.write(bytes_lz4)
fd.close()
kind = guess_media_type(file)
self.assertEqual(kind, ('application/x-ext-lz4', 'binary'))
def test_spoofed_known(self):
with tempfile.TemporaryDirectory() as temp_dir:
file = os.path.join(temp_dir, 'spoofed_known.avi')
with open(file, 'wb') as fd:
bytes_zip = bytearray([0x50,0x4b,0x03,0x06])
fd.write(bytes_zip)
fd.close()
kind = guess_media_type(file)
self.assertEqual(kind, ('application/zip', 'binary'))
def test_spoofed_synonym(self):
with tempfile.TemporaryDirectory() as temp_dir:
file = os.path.join(temp_dir, 'spoofed_known.cbz')
with open(file, 'wb') as fd:
bytes_zip = bytearray([0x50,0x4b,0x03,0x06])
fd.write(bytes_zip)
fd.close()
kind = guess_media_type(file)
self.assertEqual(kind, ('application/vnd.comicbook+zip', 'document'))
+1 -1
View File
@@ -97,7 +97,7 @@ class TestReflector(AsyncioTestCase):
async def test_announces(self):
to_announce = await self.storage.get_blobs_to_announce()
self.assertIn(self.stream.sd_hash, to_announce, "sd blob not set to announce")
self.assertIn(self.stream.descriptor.blobs[0].blob_hash, to_announce, "head blob not set to announce")
self.assertNotIn(self.stream.descriptor.blobs[0].blob_hash, to_announce, "head blob set to announce")
async def test_result_from_disconnect_mid_sd_transfer(self):
stop = asyncio.Event()
+19
View File
@@ -0,0 +1,19 @@
import unittest
from lbry import utils
class UtilsTestCase(unittest.TestCase):
def test_get_colliding_prefix_bits(self):
self.assertEqual(
0, utils.get_colliding_prefix_bits(0xffffffff.to_bytes(4, "big"), 0x0000000000.to_bytes(4, "big")))
self.assertEqual(
1, utils.get_colliding_prefix_bits(0x7fffffff.to_bytes(4, "big"), 0x0000000000.to_bytes(4, "big")))
self.assertEqual(
8, utils.get_colliding_prefix_bits(0x00ffffff.to_bytes(4, "big"), 0x0000000000.to_bytes(4, "big")))
self.assertEqual(
8, utils.get_colliding_prefix_bits(0x00ffffff.to_bytes(4, "big"), 0x0000000000.to_bytes(4, "big")))
self.assertEqual(
1, utils.get_colliding_prefix_bits(0x7fffffff.to_bytes(4, "big"), 0x0000000000.to_bytes(4, "big")))
self.assertEqual(
1, utils.get_colliding_prefix_bits(0x7fffffff.to_bytes(4, "big"), 0x0000000000.to_bytes(4, "big")))