Compare commits

...
46 Commits
Author SHA1 Message Date
Lex Berezhny 51da0d0259 v0.98.0 2021-05-26 09:23:19 -04:00
Alex Grin f55b78a994 Merge pull request #3306 from lbryio/fix-collectionUpdateWithReplace 2021-05-18 15:25:53 -04:00
Alex Grin e1a44c93f8 Merge branch 'master' into fix-collectionUpdateWithReplace 2021-05-18 15:25:40 -04:00
Alex Grin 07e7087a09 Merge pull request #3303 from keikari/patch-1
Minor fix suggestion for issue #3240
2021-05-18 15:23:56 -04:00
Alex Grin 2c79c7e2f6 Merge branch 'master' into patch-1 2021-05-18 15:23:08 -04:00
Victor Shyba 09f6637fe0 remove unused multiprocessin.Manager 2021-05-17 15:07:32 -03:00
Victor Shyba 3784db3308 test collections update with --replace 2021-05-15 03:27:33 -03:00
zeppi 2b950ff5dd fix bug in collection_update --replace 2021-05-15 03:27:33 -03:00
Alex Grin 09339c9cfb Merge pull request #3305 from lbryio/fix_migrator_tool
fix hub Elasticsearch sync/migrations tool for when the db exists already
2021-05-14 11:06:07 -04:00
Victor Shyba ccadd88af5 fix cache call 2021-05-13 22:40:21 -03:00
Victor Shyba cc02a0efc2 fix es migration bug, expand test case 2021-05-13 19:00:53 -03:00
Victor Shyba 43a1385b79 test sync helper 2021-05-13 19:00:53 -03:00
Victor Shyba 5101464e3b add integration tests command on install.md 2021-05-13 19:00:36 -03:00
Victor Shyba 3d71478d38 update install.md with ES instructions 2021-05-13 19:00:36 -03:00
Victor Shyba 4989ed445e add ES to makefile 2021-05-13 19:00:36 -03:00
keikari d9413039ec Fix suggestion for issue #3240
L135: If `getattr()` returns `None`, use `""` instead to avoid error in issue #3240
2021-05-12 18:30:38 +03:00
Jack Robison eba0c9be34 fix typo 2021-05-07 16:51:19 -04:00
Jack Robison 48c9e9f3cc Merge pull request #3294 from lbryio/versioned-es-search-index
add versioning to ES search index and automate resync on version bumps
2021-05-07 15:38:21 -04:00
Jack Robison 81ebde88db resync ES search index on version bumps
-bump ES search index to version 1
2021-05-07 14:36:53 -04:00
Jack Robison 79ced9d0f8 Merge pull request #3262 from lbryio/channel_repost_has_source
Fix bug for `has_source=True` hiding channel reposts
2021-05-07 14:36:04 -04:00
Victor Shyba a4058b84ce clean out unused sharding 2021-05-07 15:03:37 -03:00
Victor Shyba 7bf211a52b apply reposted_claim_type on es sync 2021-05-07 15:03:37 -03:00
Victor Shyba d5f722792f fix and test has_source for channel reposts 2021-05-07 15:03:37 -03:00
Victor Shyba 0f02906c9b fix has_source for reposted channels 2021-05-07 15:03:37 -03:00
Victor Shyba 9582e228b1 assert instead of sleep 2021-05-07 15:02:31 -03:00
Victor Shyba 45f20431f9 update tests from the removed feed 2021-05-07 15:02:31 -03:00
Victor Shyba 7554e6d7f9 remove dead code 2021-05-07 15:02:31 -03:00
Victor Shyba cb8f26f177 remove broken feed 2021-05-07 15:02:31 -03:00
Jack Robison b5dfce7861 Revert "finished switch from using hash # in URLs to colon :"
This reverts commit 888aa558
2021-05-07 11:31:28 -04:00
Jack Robison 2ca5a65544 Revert "FindShortestID updated"
This reverts commit 8f04a50c
2021-05-07 11:30:31 -04:00
Jack Robison 17deb136db Revert "StreamCommands"
This reverts commit 2a8ccb06
2021-05-07 11:29:45 -04:00
Jack Robison 8c9710c76c Merge pull request #3293 from lbryio/fix-block-processor-crash-invalid-fee
fix invalid claim fees breaking the block processor
2021-05-07 10:03:40 -04:00
Jack Robison 32f7ecb261 fix invalid claim fees breaking the block processor 2021-05-06 11:18:58 -04:00
Victor Shyba fb77fde710 for debug, it is always whole page 2021-05-04 22:22:07 -03:00
Victor Shyba 3c67bb90d7 don't fail when a single one go on maintenance and set completion event regardless of failures 2021-05-04 22:22:07 -03:00
Victor Shyba dabb168853 dont log full exceptions on simple connection errors 2021-05-04 22:22:07 -03:00
Victor Shyba 45e5b3b219 dont log full pages 2021-05-04 22:22:07 -03:00
Alex Grin a6b7469923 Update README.md 2021-05-03 18:04:29 -04:00
Jack Robison cb5dab3033 Merge pull request #3285 from lbryio/restrict-udp-source
Restrict udp sources, add `ALLOW_LAN_UDP` hub setting
2021-04-29 10:19:32 -04:00
Jack Robison 21d0038ff2 add timestamps to hub log 2021-04-28 16:47:00 -04:00
Jack Robison c094d8f2e8 add ALLOW_LAN_UDP hub setting 2021-04-28 16:47:00 -04:00
Jack Robison c465d6a6c2 ignore udp packets with low source ports 2021-04-28 16:47:00 -04:00
Lex Berezhny 73d35bc985 v0.97.0 2021-04-28 16:23:46 -04:00
Lex Berezhny 2a8ccb065b StreamCommands 2021-04-28 16:21:01 -04:00
Lex Berezhny 8f04a50ce1 FindShortestID updated 2021-04-28 16:21:01 -04:00
Lex Berezhny 888aa5586b finished switch from using hash # in URLs to colon : 2021-04-28 16:21:01 -04:00
21 changed files with 226 additions and 160 deletions

No files matched your search

+13 -1
View File
@@ -92,11 +92,23 @@ To install on Windows:
```
## Run the tests
### Elasticsearch
To run the unit tests from the repo directory:
For running integration tests, Elasticsearch is required to be available at localhost:9200/
The easiest way to start it is using docker with:
```bash
make elastic-docker
```
Alternative installation methods are available [at Elasticsearch website](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html).
To run the unit and integration tests from the repo directory:
```
python -m unittest discover tests.unit
python -m unittest discover tests.integration
```
## Usage
+3
View File
@@ -30,3 +30,6 @@ test-integration:
idea:
mkdir -p .idea
cp -r scripts/idea/* .idea
elastic-docker:
docker run -d -v lbryhub:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 -e"ES_JAVA_OPTS=-Xms512m -Xmx512m" -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.12.1
+1 -1
View File
@@ -41,7 +41,7 @@ This project is MIT licensed. For the full license, see [LICENSE](LICENSE).
## Security
We take security seriously. Please contact security@lbry.com regarding any security issues. [Our GPG key is here](https://lbry.com/faq/gpg-key) if you need it.
We take security seriously. Please contact security@lbry.com regarding any security issues. [Our PGP key is here](https://lbry.com/faq/pgp-key) if you need it.
## Contact
+1 -1
View File
@@ -1,2 +1,2 @@
__version__ = "0.96.0"
__version__ = "0.98.0"
version = tuple(map(int, __version__.split('.'))) # pylint: disable=invalid-name
+2 -2
View File
@@ -155,9 +155,9 @@ class KademliaPeer:
if self._node_id is not None:
if not len(self._node_id) == constants.HASH_LENGTH:
raise ValueError("invalid node_id: {}".format(hexlify(self._node_id).decode()))
if self.udp_port is not None and not 1 <= self.udp_port <= 65535:
if self.udp_port is not None and not 1024 <= self.udp_port <= 65535:
raise ValueError("invalid udp port")
if self.tcp_port is not None and not 1 <= self.tcp_port <= 65535:
if self.tcp_port is not None and not 1024 <= self.tcp_port <= 65535:
raise ValueError("invalid tcp port")
if not is_valid_public_ipv4(self.address, self.allow_localhost):
raise ValueError(f"invalid ip address: '{self.address}'")
-3
View File
@@ -3938,9 +3938,6 @@ class Daemon(metaclass=JSONRPCServerType):
if replace:
claim = Claim()
claim.collection.message.source.CopyFrom(
old_txo.claim.collection.message.source
)
claim.collection.update(**kwargs)
else:
claim = Claim.from_bytes(old_txo.claim.to_bytes())
+8 -26
View File
@@ -5,7 +5,7 @@ import logging
from statistics import median
from decimal import Decimal
from typing import Optional, Iterable, Type
from aiohttp.client_exceptions import ContentTypeError
from aiohttp.client_exceptions import ContentTypeError, ClientConnectionError
from lbry.error import InvalidExchangeRateResponseError, CurrencyConversionError
from lbry.utils import aiohttp_request
from lbry.wallet.dewies import lbc_to_dewies
@@ -79,18 +79,23 @@ class MarketFeed:
log.debug("Saving rate update %f for %s from %s", rate, self.market, self.name)
self.rate = ExchangeRate(self.market, rate, int(time.time()))
self.last_check = time.time()
self.event.set()
return self.rate
except asyncio.CancelledError:
raise
except asyncio.TimeoutError:
log.warning("Timed out fetching exchange rate from %s.", self.name)
except json.JSONDecodeError as e:
log.warning("Could not parse exchange rate response from %s: %s", self.name, e.doc)
msg = e.doc if '<html>' not in e.doc else 'unexpected content type.'
log.warning("Could not parse exchange rate response from %s: %s", self.name, msg)
log.debug(e.doc)
except InvalidExchangeRateResponseError as e:
log.warning(str(e))
except ClientConnectionError as e:
log.warning("Error trying to connect to exchange rate %s: %s", self.name, str(e))
except Exception as e:
log.exception("Exchange rate error (%s from %s):", self.market, self.name)
finally:
self.event.set()
async def keep_updated(self):
while True:
@@ -130,27 +135,6 @@ class BittrexUSDFeed(BaseBittrexFeed):
url = "https://api.bittrex.com/v3/markets/LBC-USD/ticker"
class BaseCryptonatorFeed(MarketFeed):
name = "Cryptonator"
market = None
url = None
def get_rate_from_response(self, json_response):
if 'ticker' not in json_response or 'price' not in json_response['ticker']:
raise InvalidExchangeRateResponseError(self.name, 'result not found')
return float(json_response['ticker']['price'])
class CryptonatorBTCFeed(BaseCryptonatorFeed):
market = "BTCLBC"
url = "https://api.cryptonator.com/api/ticker/btc-lbc"
class CryptonatorUSDFeed(BaseCryptonatorFeed):
market = "USDLBC"
url = "https://api.cryptonator.com/api/ticker/usd-lbc"
class BaseCoinExFeed(MarketFeed):
name = "CoinEx"
market = None
@@ -210,8 +194,6 @@ class UPbitBTCFeed(MarketFeed):
FEEDS: Iterable[Type[MarketFeed]] = (
BittrexBTCFeed,
BittrexUSDFeed,
CryptonatorBTCFeed,
CryptonatorUSDFeed,
CoinExBTCFeed,
CoinExUSDFeed,
HotbitBTCFeed,
+1 -1
View File
@@ -132,7 +132,7 @@ class SourceManager:
else:
streams = list(self._sources.values())
if sort_by:
streams.sort(key=lambda s: getattr(s, sort_by))
streams.sort(key=lambda s: getattr(s, sort_by) or "")
if reverse:
streams.reverse()
return streams
+4 -2
View File
@@ -380,13 +380,15 @@ CARRIER_GRADE_NAT_SUBNET = ipaddress.ip_network('100.64.0.0/10')
IPV4_TO_6_RELAY_SUBNET = ipaddress.ip_network('192.88.99.0/24')
def is_valid_public_ipv4(address, allow_localhost: bool = False):
def is_valid_public_ipv4(address, allow_localhost: bool = False, allow_lan: bool = False):
try:
parsed_ip = ipaddress.ip_address(address)
if parsed_ip.is_loopback and allow_localhost:
return True
if allow_lan and parsed_ip.is_private:
return True
if any((parsed_ip.version != 4, parsed_ip.is_unspecified, parsed_ip.is_link_local, parsed_ip.is_loopback,
parsed_ip.is_multicast, parsed_ip.is_reserved, parsed_ip.is_private, parsed_ip.is_reserved)):
parsed_ip.is_multicast, parsed_ip.is_reserved, parsed_ip.is_private)):
return False
else:
return not any((CARRIER_GRADE_NAT_SUBNET.supernet_of(ipaddress.ip_network(f"{address}/32")),
+1 -1
View File
@@ -25,7 +25,7 @@ def main():
parser = get_argument_parser()
args = parser.parse_args()
coin_class = get_coin_class(args.spvserver)
logging.basicConfig(level=logging.INFO)
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-4s %(name)s:%(lineno)d: %(message)s")
logging.info('lbry.server starting')
logging.getLogger('aiohttp').setLevel(logging.WARNING)
logging.getLogger('elasticsearch').setLevel(logging.WARNING)
+36 -3
View File
@@ -33,7 +33,15 @@ class StreamResolution(str):
return LookupError(f'Could not find claim at "{url}".')
class IndexVersionMismatch(Exception):
def __init__(self, got_version, expected_version):
self.got_version = got_version
self.expected_version = expected_version
class SearchIndex:
VERSION = 1
def __init__(self, index_prefix: str, search_timeout=3.0, elastic_host='localhost', elastic_port=9200):
self.search_timeout = search_timeout
self.sync_timeout = 600 # wont hit that 99% of the time, but can hit on a fresh import
@@ -48,9 +56,21 @@ class SearchIndex:
self._elastic_host = elastic_host
self._elastic_port = elastic_port
async def start(self):
async def get_index_version(self) -> int:
try:
template = await self.sync_client.indices.get_template(self.index)
return template[self.index]['version']
except NotFoundError:
return 0
async def set_index_version(self, version):
await self.sync_client.indices.put_template(
self.index, body={'version': version, 'index_patterns': ['ignored']}, ignore=400
)
async def start(self) -> bool:
if self.sync_client:
return
return False
hosts = [{'host': self._elastic_host, 'port': self._elastic_port}]
self.sync_client = AsyncElasticsearch(hosts, timeout=self.sync_timeout)
self.search_client = AsyncElasticsearch(hosts, timeout=self.search_timeout)
@@ -61,8 +81,17 @@ class SearchIndex:
except ConnectionError:
self.logger.warning("Failed to connect to Elasticsearch. Waiting for it!")
await asyncio.sleep(1)
res = await self.sync_client.indices.create(self.index, INDEX_DEFAULT_SETTINGS, ignore=400)
return res.get('acknowledged', False)
acked = res.get('acknowledged', False)
if acked:
await self.set_index_version(self.VERSION)
return acked
index_version = await self.get_index_version()
if index_version != self.VERSION:
self.logger.error("es search index has an incompatible version: %s vs %s", index_version, self.VERSION)
raise IndexVersionMismatch(index_version, self.VERSION)
return acked
def stop(self):
clients = [self.sync_client, self.search_client]
@@ -133,6 +162,9 @@ class SearchIndex:
await self.sync_client.update_by_query(
self.index, body=self.update_filter_query(Censor.RESOLVE, blocked_channels, True), slices=4)
await self.sync_client.indices.refresh(self.index)
self.clear_caches()
def clear_caches(self):
self.search_cache.clear()
self.short_id_cache.clear()
self.claim_cache.clear()
@@ -499,6 +531,7 @@ def expand_query(**kwargs):
query['should'].append(
{"bool": {"must": [{"match": {"has_source": kwargs['has_source']}}, is_stream_or_repost]}})
query['should'].append({"bool": {"must_not": [is_stream_or_repost]}})
query['should'].append({"bool": {"must": [{"term": {"reposted_claim_type": CLAIM_TYPES['channel']}}]}})
if kwargs.get('text'):
query['must'].append(
{"simple_query_string":
+22 -14
View File
@@ -10,12 +10,10 @@ from elasticsearch import AsyncElasticsearch
from elasticsearch.helpers import async_bulk
from lbry.wallet.server.env import Env
from lbry.wallet.server.coin import LBC
from lbry.wallet.server.db.elasticsearch.search import extract_doc, SearchIndex
INDEX = 'claims'
from lbry.wallet.server.db.elasticsearch.search import extract_doc, SearchIndex, IndexVersionMismatch
async def get_all(db, shard_num, shards_total, limit=0):
async def get_all(db, shard_num, shards_total, limit=0, index_name='claims'):
logging.info("shard %d starting", shard_num)
def exec_factory(cursor, statement, bindings):
tpl = namedtuple('row', (d[0] for d in cursor.getdescription()))
@@ -30,6 +28,7 @@ SELECT claimtrie.claim_hash as is_controlling,
(select group_concat(tag, ',,') from tag where tag.claim_hash in (claim.claim_hash, claim.reposted_claim_hash)) as tags,
(select group_concat(language, ' ') from language where language.claim_hash in (claim.claim_hash, claim.reposted_claim_hash)) as languages,
(select cr.has_source from claim cr where cr.claim_hash = claim.reposted_claim_hash) as reposted_has_source,
(select cr.claim_type from claim cr where cr.claim_hash = claim.reposted_claim_hash) as reposted_claim_type,
claim.*
FROM claim LEFT JOIN claimtrie USING (claim_hash)
WHERE claim.height % {shards_total} = {shard_num}
@@ -43,46 +42,55 @@ ORDER BY claim.height desc
claim['languages'] = claim['languages'].split(' ') if claim['languages'] else []
if num % 10_000 == 0:
logging.info("%d/%d", num, total)
yield extract_doc(claim, INDEX)
yield extract_doc(claim, index_name)
if 0 < limit <= num:
break
async def consume(producer):
async def consume(producer, index_name):
env = Env(LBC)
logging.info("ES sync host: %s:%i", env.elastic_host, env.elastic_port)
es = AsyncElasticsearch([{'host': env.elastic_host, 'port': env.elastic_port}])
try:
await async_bulk(es, producer, request_timeout=120)
await es.indices.refresh(index=INDEX)
await es.indices.refresh(index=index_name)
finally:
await es.close()
async def make_es_index():
async def make_es_index(index=None):
env = Env(LBC)
index = SearchIndex('', elastic_host=env.elastic_host, elastic_port=env.elastic_port)
if index is None:
index = SearchIndex('', elastic_host=env.elastic_host, elastic_port=env.elastic_port)
try:
return await index.start()
except IndexVersionMismatch as err:
logging.info(
"dropping ES search index (version %s) for upgrade to version %s", err.got_version, err.expected_version
)
await index.delete_index()
await index.stop()
return await index.start()
finally:
index.stop()
async def run(args, shard):
async def run(db_path, clients, blocks, shard, index_name='claims'):
def itsbusy(*_):
logging.info("shard %d: db is busy, retry", shard)
return True
db = apsw.Connection(args.db_path, flags=apsw.SQLITE_OPEN_READONLY | apsw.SQLITE_OPEN_URI)
db = apsw.Connection(db_path, flags=apsw.SQLITE_OPEN_READONLY | apsw.SQLITE_OPEN_URI)
db.setbusyhandler(itsbusy)
db.cursor().execute('pragma journal_mode=wal;')
db.cursor().execute('pragma temp_store=memory;')
producer = get_all(db.cursor(), shard, args.clients, limit=args.blocks)
await asyncio.gather(*(consume(producer) for _ in range(min(8, args.clients))))
producer = get_all(db.cursor(), shard, clients, limit=blocks, index_name=index_name)
await asyncio.gather(*(consume(producer, index_name=index_name) for _ in range(min(8, clients))))
def __run(args, shard):
asyncio.run(run(args, shard))
asyncio.run(run(args.db_path, args.clients, args.blocks, shard))
def run_elastic_sync():
+10 -12
View File
@@ -5,7 +5,6 @@ from typing import Union, Tuple, Set, List
from itertools import chain
from decimal import Decimal
from collections import namedtuple
from multiprocessing import Manager
from binascii import unhexlify, hexlify
from lbry.wallet.server.leveldb import LevelDB
from lbry.wallet.server.util import class_logger
@@ -220,7 +219,6 @@ class SQLDB:
self.db = None
self.logger = class_logger(__name__, self.__class__.__name__)
self.ledger = Ledger if main.coin.NET == 'mainnet' else RegTestLedger
self.state_manager = None
self.blocked_streams = None
self.blocked_channels = None
self.blocking_channel_hashes = {
@@ -251,11 +249,10 @@ class SQLDB:
self.execute(self.PRAGMAS)
self.execute(self.CREATE_TABLES_QUERY)
register_canonical_functions(self.db)
self.state_manager = Manager()
self.blocked_streams = self.state_manager.dict()
self.blocked_channels = self.state_manager.dict()
self.filtered_streams = self.state_manager.dict()
self.filtered_channels = self.state_manager.dict()
self.blocked_streams = {}
self.blocked_channels = {}
self.filtered_streams = {}
self.filtered_channels = {}
self.update_blocked_and_filtered_claims()
for algorithm in self.trending:
algorithm.install(self.db)
@@ -263,8 +260,6 @@ class SQLDB:
def close(self):
if self.db is not None:
self.db.close()
if self.state_manager is not None:
self.state_manager.shutdown()
def update_blocked_and_filtered_claims(self):
self.update_claims_from_channel_hashes(
@@ -406,7 +401,8 @@ class SQLDB:
if isinstance(fee.currency, str):
claim_record['fee_currency'] = fee.currency.lower()
if isinstance(fee.amount, Decimal):
claim_record['fee_amount'] = int(fee.amount*1000)
if fee.amount >= 0 and int(fee.amount*1000) < 9223372036854775807:
claim_record['fee_amount'] = int(fee.amount*1000)
elif claim.is_repost:
claim_record['claim_type'] = CLAIM_TYPES['repost']
claim_record['reposted_claim_hash'] = claim.repost.reference.claim_hash
@@ -822,16 +818,18 @@ class SQLDB:
)
def enqueue_changes(self):
for claim in self.execute(f"""
query = """
SELECT claimtrie.claim_hash as is_controlling,
claimtrie.last_take_over_height,
(select group_concat(tag, ',,') from tag where tag.claim_hash in (claim.claim_hash, claim.reposted_claim_hash)) as tags,
(select group_concat(language, ' ') from language where language.claim_hash in (claim.claim_hash, claim.reposted_claim_hash)) as languages,
(select cr.has_source from claim cr where cr.claim_hash = claim.reposted_claim_hash) as reposted_has_source,
(select cr.claim_type from claim cr where cr.claim_hash = claim.reposted_claim_hash) as reposted_claim_type,
claim.*
FROM claim LEFT JOIN claimtrie USING (claim_hash)
WHERE claim.claim_hash in (SELECT claim_hash FROM changelog)
"""):
"""
for claim in self.execute(query):
claim = claim._asdict()
id_set = set(filter(None, (claim['claim_hash'], claim['channel_hash'], claim['reposted_claim_hash'])))
claim['censor_type'] = 0
+1
View File
@@ -73,6 +73,7 @@ class Env:
self.tor_banner_file = self.default('TOR_BANNER_FILE', self.banner_file)
self.anon_logs = self.boolean('ANON_LOGS', False)
self.log_sessions = self.integer('LOG_SESSIONS', 3600)
self.allow_lan_udp = self.boolean('ALLOW_LAN_UDP', False)
# Peer discovery
self.peer_discovery = self.peer_discovery_enum()
self.peer_announce = self.boolean('PEER_ANNOUNCE', True)
+1 -1
View File
@@ -115,7 +115,7 @@ class Server:
if self.env.udp_port:
await self.bp.status_server.start(
0, bytes.fromhex(self.bp.coin.GENESIS_HASH)[::-1],
self.env.host, self.env.udp_port
self.env.host, self.env.udp_port, self.env.allow_lan_udp
)
await _start_cancellable(self.bp.fetch_and_process_blocks)
+14 -5
View File
@@ -3,7 +3,7 @@ import struct
from time import perf_counter
import logging
from typing import Optional, Tuple, NamedTuple
from lbry.utils import LRUCache
from lbry.utils import LRUCache, is_valid_public_ipv4
# from prometheus_client import Counter
@@ -69,7 +69,8 @@ class SPVPong(NamedTuple):
class SPVServerStatusProtocol(asyncio.DatagramProtocol):
PROTOCOL_VERSION = 1
def __init__(self, height: int, tip: bytes, throttle_cache_size: int = 1024, throttle_reqs_per_sec: int = 10):
def __init__(self, height: int, tip: bytes, throttle_cache_size: int = 1024, throttle_reqs_per_sec: int = 10,
allow_localhost: bool = False, allow_lan: bool = False):
super().__init__()
self.transport: Optional[asyncio.transports.DatagramTransport] = None
self._height = height
@@ -80,6 +81,8 @@ class SPVServerStatusProtocol(asyncio.DatagramProtocol):
self._throttle = LRUCache(throttle_cache_size)
self._should_log = LRUCache(throttle_cache_size)
self._min_delay = 1 / throttle_reqs_per_sec
self._allow_localhost = allow_localhost
self._allow_lan = allow_lan
def update_cached_response(self):
self._cached_response = SPVPong.make(self._height, self._tip, self._flags, self.PROTOCOL_VERSION)
@@ -119,7 +122,11 @@ class SPVServerStatusProtocol(asyncio.DatagramProtocol):
except (ValueError, struct.error, AttributeError, TypeError):
# log.exception("derp")
return
self.transport.sendto(self.make_pong(addr[0]), addr)
if addr[1] >= 1024 and is_valid_public_ipv4(
addr[0], allow_localhost=self._allow_localhost, allow_lan=self._allow_lan):
self.transport.sendto(self.make_pong(addr[0]), addr)
else:
log.warning("odd packet from %s:%i", addr[0], addr[1])
# ping_count_metric.inc()
def connection_made(self, transport) -> None:
@@ -137,12 +144,14 @@ class StatusServer:
def __init__(self):
self._protocol: Optional[SPVServerStatusProtocol] = None
async def start(self, height: int, tip: bytes, interface: str, port: int):
async def start(self, height: int, tip: bytes, interface: str, port: int, allow_lan: bool = False):
if self.is_running:
return
loop = asyncio.get_event_loop()
self._protocol = SPVServerStatusProtocol(height, tip)
interface = interface if interface.lower() != 'localhost' else '127.0.0.1'
self._protocol = SPVServerStatusProtocol(
height, tip, allow_localhost=interface == '127.0.0.1', allow_lan=allow_lan
)
await loop.create_datagram_endpoint(lambda: self._protocol, (interface, port))
log.info("started udp status server on %s:%i", interface, port)
@@ -196,12 +196,13 @@ class ClaimSearchCommand(ClaimTestCase):
normal = await self.stream_create('normal', data=b'normal')
normal_repost = await self.stream_repost(self.get_claim_id(normal), 'normal-repost')
no_source_repost = await self.stream_repost(self.get_claim_id(no_source), 'no-source-repost')
await self.assertFindsClaims([no_source_repost, no_source, channel], has_no_source=True)
channel_repost = await self.stream_repost(self.get_claim_id(channel), 'channel-repost')
await self.assertFindsClaims([channel_repost, no_source_repost, no_source, channel], has_no_source=True)
await self.assertListsClaims([no_source, channel], has_no_source=True)
await self.assertFindsClaims([normal_repost, normal, channel], has_source=True)
await self.assertListsClaims([no_source_repost, normal_repost, normal], has_source=True)
await self.assertFindsClaims([no_source_repost, normal_repost, normal, no_source, channel])
await self.assertListsClaims([no_source_repost, normal_repost, normal, no_source, channel])
await self.assertFindsClaims([channel_repost, normal_repost, normal, channel], has_source=True)
await self.assertListsClaims([channel_repost, no_source_repost, normal_repost, normal], has_source=True)
await self.assertFindsClaims([channel_repost, no_source_repost, normal_repost, normal, no_source, channel])
await self.assertListsClaims([channel_repost, no_source_repost, normal_repost, normal, no_source, channel])
async def test_pagination(self):
await self.create_channel()
@@ -2168,20 +2169,30 @@ class CollectionCommands(CommandTestCase):
tx = await self.collection_create('radjingles', claims=claim_ids, allow_duplicate_name=True)
claim_id2 = self.get_claim_id(tx)
self.assertItemCount(await self.daemon.jsonrpc_collection_list(), 2)
# with clear_claims
await self.collection_update(claim_id, clear_claims=True, claims=claim_ids[:2])
collections = await self.out(self.daemon.jsonrpc_collection_list())
self.assertEquals(len(collections['items']), 2)
self.assertEqual(len(collections['items']), 2)
self.assertNotIn('canonical_url', collections['items'][0])
resolved_collections = await self.out(self.daemon.jsonrpc_collection_list(resolve=True))
self.assertIn('canonical_url', resolved_collections['items'][0])
# with replace
await self.collection_update(claim_id, replace=True, claims=claim_ids[::-1][:2], tags=['cool'])
updated = await self.claim_search(claim_id=claim_id)
self.assertEqual(updated[0]['value']['tags'], ['cool'])
self.assertEqual(updated[0]['value']['claims'], claim_ids[::-1][:2])
await self.collection_update(claim_id, replace=True, claims=claim_ids[:4], languages=['en', 'pt-BR'])
updated = await self.resolve(f'radjingles:{claim_id}')
self.assertEqual(updated['value']['claims'], claim_ids[:4])
self.assertNotIn('tags', updated['value'])
self.assertEqual(updated['value']['languages'], ['en', 'pt-BR'])
await self.collection_abandon(claim_id)
self.assertItemCount(await self.daemon.jsonrpc_collection_list(), 1)
collections = await self.out(self.daemon.jsonrpc_collection_list(resolve_claims=2))
self.assertEquals(len(collections['items'][0]['claims']), 2)
self.assertEqual(len(collections['items'][0]['claims']), 2)
collections = await self.out(self.daemon.jsonrpc_collection_list(resolve_claims=10))
self.assertEqual(len(collections['items'][0]['claims']), 4)
@@ -4,6 +4,7 @@ import lbry
import lbry.wallet
from lbry.error import ServerPaymentFeeAboveMaxAllowedError
from lbry.wallet.network import ClientSession
from lbry.wallet.server.db.elasticsearch.sync import run as run_sync, make_es_index
from lbry.wallet.server.session import LBRYElectrumX
from lbry.testcase import IntegrationTestCase, CommandTestCase
from lbry.wallet.orchstr8.node import SPVNode
@@ -13,9 +14,6 @@ class TestSessions(IntegrationTestCase):
"""
Tests that server cleans up stale connections after session timeout and client times out too.
"""
LEDGER = lbry.wallet
async def test_session_bloat_from_socket_timeout(self):
await self.conductor.stop_spv()
await self.ledger.stop()
@@ -87,3 +85,31 @@ class TestUsagePayment(CommandTestCase):
self.assertIsNotNone(await self.blockchain.get_raw_transaction(tx.id)) # verify its broadcasted
self.assertEqual(tx.outputs[0].amount, 100000000)
self.assertEqual(tx.outputs[0].get_address(self.ledger), address)
class TestESSync(CommandTestCase):
async def test_es_sync_utility(self):
for i in range(10):
await self.stream_create(f"stream{i}", bid='0.001')
await self.generate(1)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
db = self.conductor.spv_node.server.db
await db.search_index.delete_index()
db.search_index.clear_caches()
self.assertEqual(0, len(await self.claim_search(order_by=['height'])))
await db.search_index.stop()
self.assertTrue(await make_es_index(db.search_index))
async def resync():
await db.search_index.start()
db.search_index.clear_caches()
await run_sync(db.sql._db_path, 1, 0, 0, index_name=db.search_index.index)
self.assertEqual(10, len(await self.claim_search(order_by=['height'])))
await resync()
# this time we will test a migration from unversioned to v1
await db.search_index.sync_client.indices.delete_template(db.search_index.index)
await db.search_index.stop()
self.assertTrue(await make_es_index(db.search_index))
await db.search_index.start()
await resync()
@@ -1,6 +1,8 @@
import asyncio
import logging
from decimal import Decimal
from lbry.testcase import AsyncioTestCase
from lbry.extras.daemon.exchange_rate_manager import ExchangeRate, ExchangeRateManager, FEEDS
from lbry.extras.daemon.exchange_rate_manager import ExchangeRate, ExchangeRateManager, FEEDS, MarketFeed
class TestExchangeRateManager(AsyncioTestCase):
@@ -12,13 +14,32 @@ class TestExchangeRateManager(AsyncioTestCase):
self.assertFalse(feed.is_online)
self.assertIsNone(feed.rate)
await manager.wait()
failures = set()
for feed in manager.market_feeds:
self.assertTrue(feed.is_online)
self.assertIsInstance(feed.rate, ExchangeRate)
# print(f'{feed.name} - {feed.market} - {feed.rate.spot}')
if feed.is_online:
self.assertIsInstance(feed.rate, ExchangeRate)
else:
failures.add(feed.name)
self.assertFalse(feed.has_rate)
self.assertLessEqual(len(failures), 1, f"feed failures: {failures}. Please check exchange rate feeds!")
lbc = manager.convert_currency('USD', 'LBC', Decimal('1.0'))
self.assertGreaterEqual(lbc, 2.0)
self.assertLessEqual(lbc, 10.0)
lbc = manager.convert_currency('BTC', 'LBC', Decimal('0.01'))
self.assertGreaterEqual(lbc, 1_000)
self.assertLessEqual(lbc, 4_000)
async def test_it_handles_feed_being_offline(self):
class FakeFeed(MarketFeed):
name = "fake"
url = "http://impossi.bru"
manager = ExchangeRateManager((FakeFeed,))
manager.start()
self.addCleanup(manager.stop)
for feed in manager.market_feeds:
self.assertFalse(feed.is_online)
self.assertIsNone(feed.rate)
await asyncio.wait_for(manager.wait(), 2)
for feed in manager.market_feeds:
self.assertFalse(feed.is_online)
self.assertFalse(feed.has_rate)
+26 -25
View File
@@ -10,8 +10,8 @@ class PeerTest(AsyncioTestCase):
self.loop = asyncio.get_event_loop()
self.peer_manager = PeerManager(self.loop)
self.node_ids = [generate_id(), generate_id(), generate_id()]
self.first_contact = make_kademlia_peer(self.node_ids[1], '1.0.0.1', udp_port=1000)
self.second_contact = make_kademlia_peer(self.node_ids[0], '1.0.0.2', udp_port=1000)
self.first_contact = make_kademlia_peer(self.node_ids[1], '1.0.0.1', udp_port=1024)
self.second_contact = make_kademlia_peer(self.node_ids[0], '1.0.0.2', udp_port=1024)
def test_peer_is_good_unknown_peer(self):
# Scenario: peer replied, but caller doesn't know the node_id.
@@ -25,36 +25,37 @@ class PeerTest(AsyncioTestCase):
def test_make_contact_error_cases(self):
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4', 100000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4.5', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], 'this is not an ip', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4.5', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], 'this is not an ip', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4', -1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4', 0)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4', 1023)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '1.2.3.4', 70000)
self.assertRaises(ValueError, make_kademlia_peer, b'not valid node id', '1.2.3.4', 1000)
self.assertRaises(ValueError, make_kademlia_peer, b'not valid node id', '1.2.3.4', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '0.0.0.0', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '10.0.0.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '100.64.0.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '127.0.0.1', 1000)
self.assertIsNotNone(make_kademlia_peer(self.node_ids[1], '127.0.0.1', 1000, allow_localhost=True))
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.168.0.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '172.16.0.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '169.254.1.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.0.0.2', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.0.2.2', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.88.99.2', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '198.18.1.1', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '198.51.100.2', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '198.51.100.2', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '203.0.113.4', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '0.0.0.0', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '10.0.0.1', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '100.64.0.1', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '127.0.0.1', 1024)
self.assertIsNotNone(make_kademlia_peer(self.node_ids[1], '127.0.0.1', 1024, allow_localhost=True))
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.168.0.1', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '172.16.0.1', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '169.254.1.1', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.0.0.2', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.0.2.2', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '192.88.99.2', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '198.18.1.1', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '198.51.100.2', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '198.51.100.2', 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '203.0.113.4', 1024)
for i in range(32):
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], f"{224 + i}.0.0.0", 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '255.255.255.255', 1000)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], f"{224 + i}.0.0.0", 1024)
self.assertRaises(ValueError, make_kademlia_peer, self.node_ids[1], '255.255.255.255', 1024)
self.assertRaises(
ValueError, make_kademlia_peer, self.node_ids[1], 'beee:eeee:eeee:eeee:eeee:eeee:eeee:eeef', 1000
ValueError, make_kademlia_peer, self.node_ids[1], 'beee:eeee:eeee:eeee:eeee:eeee:eeee:eeef', 1024
)
self.assertRaises(
ValueError, make_kademlia_peer, self.node_ids[1], '2001:db8::ff00:42:8329', 1000
ValueError, make_kademlia_peer, self.node_ids[1], '2001:db8::ff00:42:8329', 1024
)
def test_is_valid_ipv4(self):
@@ -79,7 +80,7 @@ class PeerTest(AsyncioTestCase):
def test_boolean(self):
self.assertNotEqual(self.first_contact, self.second_contact)
self.assertEqual(
self.second_contact, make_kademlia_peer(self.node_ids[0], '1.0.0.2', udp_port=1000)
self.second_contact, make_kademlia_peer(self.node_ids[0], '1.0.0.2', udp_port=1024)
)
def test_compact_ip(self):
@@ -4,7 +4,6 @@ from time import time
from lbry.schema.claim import Claim
from lbry.extras.daemon.exchange_rate_manager import (
ExchangeRate, ExchangeRateManager, CurrencyConversionError,
CryptonatorUSDFeed, CryptonatorBTCFeed,
BittrexUSDFeed, BittrexBTCFeed,
CoinExBTCFeed
)
@@ -36,46 +35,6 @@ class ExchangeRateTests(AsyncioTestCase):
with self.assertRaises(CurrencyConversionError):
manager.convert_currency(fee.currency, "LBC", fee.amount)
def test_cryptonator_lbc_feed_response(self):
feed = CryptonatorUSDFeed()
out = feed.get_rate_from_response({
'ticker': {
'base': 'USD', 'target': 'LBC', 'price': 23657.44026496,
'volume': '', 'change': -5.59806916,
},
'timestamp': 1507470422,
'success': True,
'error': ""
})
self.assertEqual(23_657.44026496, out)
with self.assertRaises(InvalidExchangeRateResponseError):
feed.get_rate_from_response({})
with self.assertRaises(InvalidExchangeRateResponseError):
feed.get_rate_from_response({
"success": True,
"ticker": {}
})
def test_cryptonator_btc_feed_response(self):
feed = CryptonatorBTCFeed()
out = feed.get_rate_from_response({
'ticker': {
'base': 'BTC', 'target': 'LBC', 'price': 0.00022123,
'volume': '', 'change': -0.00000259,
},
'timestamp': 1507471141,
'success': True,
'error': ''
})
self.assertEqual(0.00022123, out)
with self.assertRaises(InvalidExchangeRateResponseError):
feed.get_rate_from_response({})
with self.assertRaises(InvalidExchangeRateResponseError):
feed.get_rate_from_response({
"success": True,
"ticker": {}
})
def test_bittrex_feed_response(self):
feed = BittrexBTCFeed()
out = feed.get_rate_from_response({
@@ -105,17 +64,20 @@ class ExchangeRateManagerTests(AsyncioTestCase):
async def test_get_rate_failure_retrieved(self):
manager = ExchangeRateManager([BadMarketFeed])
manager.start()
await asyncio.sleep(1)
await manager.wait()
for feed in manager.market_feeds: # no rate but it tried
self.assertFalse(feed.has_rate)
self.assertTrue(feed.event.is_set())
self.addCleanup(manager.stop)
async def test_median_rate_used(self):
manager = ExchangeRateManager([BittrexBTCFeed, CryptonatorBTCFeed, CoinExBTCFeed])
manager = ExchangeRateManager([BittrexBTCFeed, CoinExBTCFeed])
for feed in manager.market_feeds:
feed.last_check = time()
bittrex, cryptonator, coinex = manager.market_feeds
bittrex, coinex = manager.market_feeds
bittrex.rate = ExchangeRate(bittrex.market, 1.0, time())
cryptonator.rate = ExchangeRate(cryptonator.market, 2.0, time())
coinex.rate = ExchangeRate(coinex.market, 2.0, time())
coinex.rate = ExchangeRate(coinex.market, 3.0, time())
self.assertEqual(14.0, manager.convert_currency("BTC", "LBC", Decimal(7.0)))
cryptonator.rate.spot = 4.0
self.assertEqual(21.0, manager.convert_currency("BTC", "LBC", Decimal(7.0)))
coinex.rate.spot = 4.0
self.assertEqual(17.5, manager.convert_currency("BTC", "LBC", Decimal(7.0)))