Author SHA1 Message Date
Leopere dbb3ae747b adding todo items for docker-compose.yml in spee.ch container 2018-11-26 23:25:24 +00:00
Leopere 1dd0e6c333 Added TODO and updated paths 2018-11-26 23:21:48 +00:00
81 changed files with 1606 additions and 879 deletions

No files matched your search

+2 -11
View File
@@ -1,13 +1,4 @@
.gitlab-ci.yml
.DS_Store
./docker-compose.yml
*/docker-compose.override.yml
*/*/data
*.zip
*.tar.gz
*.tar.xz
./persist/lbrycrd/*
./persist/lbrynet/*
./persist/chainquery/*
./docker-compose.yml
+14
View File
@@ -0,0 +1,14 @@
# os: linux
# dist: trusty
#
# services:
# - docker
#
# before_install:
# - docker build -t leopere/POTATO .
# # - docker run -d -p 127.0.0.1:80:4567 carlad/sinatra /bin/sh -c "cd /root/sinatra; bundle exec foreman start;"
# - docker ps -a
# - docker run carlad/sinatra /bin/sh -c "cd /root/sinatra; bundle exec rake test"
#
# script:
# - bundle exec rake test
+27 -48
View File
@@ -1,84 +1,63 @@
# lbry-Docker
## Tags
<!-- Document tags and link to their Dockerfiles here. -->
## Scope
This repository aims to track and maintain most of the Dockerfiles within the LBRY Inc ecosystem.
This repository is in heavy flux as it travels towards [version 1.0](https://github.com/lbryio/lbry-docker/projects/1) however its goal is to make development for and adoption of any of the LBRY appliances trivial. You should be able to clone pull fork your way to a better LBRY without having to do much more than some light reading of a README to get started.
Currently it supports the following in various form factors:
* Lbrycrd
* Lbrynet
* Chainquery
If you see an architecture that you have a use case for that you don't see in this list and it's stopping you from innovating on the LBRY blockchain search and bump an existing request or create a feature request ASAP on the issue tracker and we will do our best to support your architecture or build requirements.
The goals here will be to create containers that can deploy across a number of technologies namely `docker-compose`, `docker` itself, and Kubernetes k8's.
There will be a few edge cases where this repository will support installing these containers as a service on a machine using `cloud-init` however those come as a YMMV(your mileage may vary) warranty void if used.
We may suggest the use of well commented **`Shell scripts`**, Dockerfiles, and Docker-Compose which is a template that is a baseline for many other container management services such as [RancherOS](https://rancher.com/rancher-os/) and [Kubernetes](https://kubernetes.io/).
#### Documentation is WIP
Currently this repository is a WIP and is under heavy construction, use at your own risk make sure you keep regular backups of your wallets. Your milage may vary as how far this will work for you be sure to file good and concise issues if you plan to and keep in mind we're allergic to regressions when filing PR's.
#### Goals
We have a number of priorities within this docker container repository.
Loosely defined our priorities are in the order as follows ordered highest priority to lowest.
1. Congruency - Containers should all behave similarly.
2. Safety - Cautioning the user when they're in a danger zone when possible.
3. Reliability - Only pushing functional tested containers to production allowing users to impose an update schema.
4. Ephemerality/Statelessness - You should be able to define the entire state in a cross cloud platform way such that the container itself should contain no state and thus have no reason to rely on it's specific contents as best as possible.
5. Works by default - As best as possible we will aim for these containers to work by default out of the box then if you wish you can find ways to improve upon what we've set out for the user on your own.
6. Flexibility - Despite all of the above we want people to be able to utilize all of the functionalities of the appliances contained within these containers. This means if you find that our system/implementation gets in your way this is one of our priorities to fix this.
In summary, this repository aims for [Docker Best Practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) minimalism, ephemerality, and statelessness.
This repository aims for [Docker Best Practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) minimalism, ephemerality, and statelessness. It will use well commented **`Shell scripts`**, Dockerfiles, and Docker-Compose which is a template that is a baseline for many other container management services such as [RancherOS](https://rancher.com/rancher-os/) and [Kubernetes](https://kubernetes.io/).
#### Beautiful Screenshot/Gif
Here's an example tutorial we have for how to get some of our appliances up and running in minutes on DigitalOcean all hosted on our very own platform https://spee.ch a link to the guide is [here](https://github.com/lbryio/lbry-docker/blob/master/contrib/systemd-cloud-init.md) it contains a few lines of information and some copy paste-able code!
[![Video of creating lbrycrd droplet on DigitalOcean](https://spee.ch/@EnigmaCurry:d/lbrycrd-video-thumb.jpg)](https://spee.ch/@EnigmaCurry:d/lbrycrd-docker-cloud-init.mp4)
Since there is nothing to see here just yet I'll entertain you with the current state of affairs with this repository.
![image](https://spee.ch/855d1958650b850b249b9ee592ba2f4c6fc7eeec/container-unloading-gone-wrong-151175.gif)
## Installation
There are a few methods we're planning on releasing over time however two notable options are our [cloud-init](https://github.com/lbryio/lbry-docker/blob/master/contrib/systemd-cloud-init.md) install of lbrycrd and optionally chainquery on DigitalOcean.
Alternatively there is the [shell installer](https://github.com/lbryio/lbry-docker/blob/master/chainquery/README.md) that can run on any Ubuntu 18.04-LTS equipped machine to get you up and running with a database seed which can save you loads of time bootstrapping your instance.
[This is currently WIP and Not Recommended for Production](https://github.com/lbryio/lbry-docker/projects/1)
See [Running from source](##Running-from-source) for the current instructions on how to use this.
#### Currently supported platforms
**X86_64 CPU architecture**
Lbrynet production & compiler, Lbrycrd, Chainqery are currently supported.
**Some arm architectures**
Lbrynet armhf and arm64 compilers
**X64 cpu architecture**
**More will be added on request and over time**
## Usage
You may see above [the installation section above](https://github.com/lbryio/lbry-docker#installation) to obtain container specific instructions.
For now I don't recommend using this container cluster however you're welcomed to [contribute](#contributing) if you feel up to the task.
## Running from source
Running this stuff from source should be possible if you have both [Docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) both installed. If these are both installed, you can proceed to run the following from within your development directory.
Running this stuff from source should be possible if you have both [Docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) both installed. If these are both installed you can proceed to run the following from within your development directory.
```
git clone https://github.com/lbryio/lbry-docker.git
```
Once you have a local copy of the recent source, you will want to consider what containers/applications you require in your environment. At the moment since at the writing of this documentation, this comes with an assertion of [YMMV](https://dictionary.cambridge.org/dictionary/english/ymmv) so if something isn't working feel free to make suggestions in the form of a PR for how we should do this better. The beauty of Open Source is learning better ways to do things as well as contributing to the world, so I'm always going to be welcoming to contributions.
Once you have a local copy of the recent source you will want to consider what containers/applications you require in your environment. At the moment since at the writing of this documentation this comes with an assertion of [YMMV](https://dictionary.cambridge.org/dictionary/english/ymmv) so if something isn't working feel free to make suggestions in the form of a PR for how we should do this better. The beauty of Open Source is learning better ways to do things as well as contributing to the world so I'm always going to be welcoming to contributions.
#### From Source for Contributions
Running from source for contributing and Merge/Pull requests.
My goal is to make contributing to this possible using Docker and also GitLab CI/CD time.
## [Contributing](CONTRIBUTING.md)
Keep in mind [I am](https://github.com/leopere/) preferential to receiving patches over rule-following as we can always nudge you in the right direction to get things more compatible with the project ethos if it's not. Never be afraid to file a PR no one should be offended. Having said this following the next two guides will greatly improve the speed at which we can integrate your improvements.
Keep in mind [I am](https://github.com/leopere/) preferential to receiving patches over rule following as we can always nudge you in the right direction to get things more compatible with the project ethos if it's not. Never be afraid to file a PR no one should be offended. This said following the next two guides will greatly improve the speed at which we can integrate your improvements.
* [Repository Standards]( https://lbry.tech/resources/repository-standards)
* [Contribute](https://lbry.tech/contribute)
* Have an LBC wallet ready as we want you to have some for the help! Hell, why not post it in your Commit or Merge Request for all I care but take your tips!
* Have a LBC wallet ready as we want you to have some for the help! Hell why not post it in your Commit or Merge Request for all I care but take your tips!
## Getting Support
Sometimes we just aren't omniscient omnipresent or even omnipotent. So since this is the case despite our constant striving for these goals we're aware that things are imperfect so when you find these imperfections please reach out to us! You can become a contributor without even knowing how to code a damn thing! Just open a Git Issue so long as it's not a duplicate its a huge help to us and often times it'll earn you some LBC if you're a part of our community.
#### Debugpaste [WIP]
I'll be including a function to get a self destructing debugpaste of your LBRY appliances logs you'll be able to execute something similar to the following in all containers to export raw logs to a paste service where you can then either modify them removing sensitive data or just take that URL and create a new issue after you [(Use Issue Search)](https://github.com/lbryio/lbry-docker/issues?utf8=%E2%9C%93&q=is%3Aissue) to make sure there isn't already an open thread for your issue.
#### Example debugpaste
```
cd chainquery/
docker-compose exec chainquery debugpaste
https://haste.nixc.us/ocatumatozaq.nginx
```
You can then take output given in the response from the debugpaste command and put that into your github issue (This may be automated at some point to some degree).
#### Creating an Issue
Please be sure to fill out the [issue template](https://github.com/lbryio/lbry-docker/issues/new) as best as possible. This will help us answer your questions faster and fix things quicker if we have quality reports.
+3
View File
@@ -0,0 +1,3 @@
data/
data.z*
chainquery.z*
+19
View File
@@ -0,0 +1,19 @@
COMPOSE_PROJECT_NAME=chainquery
#########################
## Chainquery Settings ##
#########################
## TODO: Test to ensure these vars can be pulled from the lbrycrd .env
#RPC_USER=lbryrpc ## Not super necessery to change this.
#RPC_PASSWORD=changeme ## Please replace changeme.
RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#################
## Mysql Creds ##
#################
MYSQL_SERVER=10.5.1.10 ## You're better off not changing this.
MYSQL_USER=changeme ## This could be changed.
MYSQL_PASSWORD=changeme ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
MYSQL_DATABASE=chainquery ## This can stay the same.
MYSQL_ROOT_PASSWORD=changeme ## Set this to something random and obnoxious we're not using it.
-1
View File
@@ -1,4 +1,3 @@
data/
data.z*
chainquery.z*
*.zip
+52
View File
@@ -0,0 +1,52 @@
## This base image is for running latest chainquery
FROM ubuntu:18.04 as prep
LABEL MAINTAINER="leopere [at] nixc [dot] us"
## Install everything needed to unzip the file containing Chainquery
RUN apt-get update && \
apt-get -y install unzip curl && \
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/* && \
mkdir /download
## Download and extract the latest Zip containing Linux binary for Chainquery.
RUN curl -L -o /download/chainquery.zip $(curl -s https://api.github.com/repos/lbryio/chainquery/releases | fgrep 'Linux_x86_64.zip' | grep download | head -n 1 | cut -d'"' -f4) && \
cd /download && \
ls -lAh /download && pwd && \
unzip chainquery.zip && \
rm chainquery.zip
## TODO: Use this instead of everything above here at some point. To do this we will need the LBRY team to host all of their release binaries in their latest form internally I guess since github doesn't support latest tags.
# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /download
# TODO: Switch to Alpine eventually we might need to compile in Alpine for this.
FROM ubuntu:18.04 as app
## Run as unprivileged user.
RUN adduser chainquery --gecos GECOS --shell /bin/bash --disabled-password --home /home/chainquery
# Pull chainquery executable from prep container.
COPY --from=0 /download/chainquery /usr/bin
# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /usr/bin
COPY stuff/chainqueryconfig.toml /etc/chainquery/
## Get latest chainqueryconfig.toml from master on github as template.
# TODO: add magic which pattern matches for chainqueryconfig.toml DEFAULT strings and replaces them with configured settings on container start.
# ADD --chown=1000:1000 https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml /etc/chainquery/chainqueryconfig.toml.orig
## Install start.sh as executable script in container $PATH
COPY stuff/start.sh /usr/local/bin/start
# TODO: Implement docker-entrypoint if later required this might be handy if we need certain maintainence tasks to be executed in the live container.
# COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
## From here onward we're doing this with no root.
USER chainquery
## Expose Chainquery API port
EXPOSE 6300
## Execute start script earlier installed into $PATH
CMD ["start"]
@@ -1,23 +0,0 @@
## Get the latest source and extract it for the app container.
## Design choices, two RUN layers intended to keep builds faster, the zipped
FROM ubuntu:18.04 as prep
LABEL MAINTAINER="leopere [at] nixc [dot] us"
RUN apt-get update && \
apt-get -y install unzip curl && \
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/*
WORKDIR /
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
COPY ./stuff/start.sh start
RUN curl -L -o /chainquery.zip $(curl -s https://api.github.com/repos/lbryio/chainquery/releases | grep -F 'Linux_x86_64.zip' | grep download | head -n 1 | cut -d'"' -f4) && \
unzip /chainquery.zip && \
rm /chainquery.zip
FROM ubuntu:18.04 as app
ADD https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml /etc/lbry/chainqueryconfig.toml.orig
RUN adduser chainquery --gecos GECOS --shell /bin/bash --disabled-password --home /home/chainquery && \
chown -R chainquery:chainquery /etc/lbry
COPY --from=prep /chainquery /start /usr/bin/
EXPOSE 6300
USER chainquery
CMD ["start"]
+22 -31
View File
@@ -1,46 +1,39 @@
# Docker image tags
`lbry/chainquery`
`[linux-x86_64-production](https://github.com/lbryio/lbry-docker/blob/master/chainquery/Dockerfile-linux-x86_64-production)` (Latest release)
# Installation instructions for Chainquery and dependencies
## Preface
I have set up this tutorial so that you should be able to copy and paste the commands into your Linux console. However, if you run into issues, please be sure to read the detailed expansions on the current and prior steps to ensure you didn't miss a detail.
## Caveats to installing Chainquery
Chainquery is an interface to an extensive index of data which has to parse an entire blockchain's history into a SQL database in a way that is easily query-able. The process of syncing your lbrycrd level.db files can take a long time depending on your deployment and then once your instance is synced you need to index the blockchain with Chainquery. All of this takes a fair bit of time or resources at first it is possible to deploy this stack without the shell scripts, and I will eventually include a docker-compose.yml file which gives you a baseline for deploying this to k8's and other platforms but for now single host deployment via shell script will be what's supported.
I have set up this tutorial so that you should be able to simply copy and paste the commands into your linux console. However if you run into issues please be sure to read the detailed expansions on the current and prior steps to ensure you didn't miss a detail.
## Goals of this README.md
This readme is targeted towards people who need a copy paste recipe to get Chainquery online and running asap and with some degree of repeatability. Once this is up and running on your machine, you should be able to see what a running instance of Chainquery looks like and start writing apps against its API.
This readme is targeted towards people who need a copy pasta recipe to get Chainquery online and running asap and with some degree of repeatability. Once this is up and running on your machine you should be able to see what a running instance of Chainquery looks like and start writing apps against it's API calls etc.
## Dependencies
My goal is to avoid as many dependencies as possible so for now the only pre-requisites you are required to have are `git`, `docker`, and `docker-compose` for the most part your understanding of the technologies can be superficial so long as you can follow commands and are open to reading a bit you should be fine.
My goal is to avoid as many dependencies as possible so for now the only pre-requisites you are required to have will be `git`, `docker`, and `docker-compose` for the most part your understanding of the technologies can be superficial so long as you can follow commands and are open to reading a bit you should be fine.
### Docker
Docker
Docker is effectively the cross platform software package repository it allows you to ship an entire environment in what's referred to as a container. Implying that containers hold everything that is needed to ship what's inside effectively from one place to another with a degree of standard that makes it easy for everyone along the way to replicate what is needed for their step in the chain.
Docker is effectively the cross-platform software package repository it allows you to ship an entire environment in what's referred to as a container. Containers are intended to hold everything that is needed to ship what's required to run an application from one place to another with a degree of a standard that makes it easy for everyone along the way to reproduce the environment for their step in the chain.
The other side of docker is it brings everything that we would normally have to teach you about the individual components of your soon to be installed Chainquery, Lbrycrd, and MySQL stack and wraps it up nicely in a handfull of easy to follow steps that should result in the same environment on everyones host.
The other side of docker is it brings everything that we would typically have to teach you about the individual components of your soon to be installed Chainquery, Lbrycrd, and MySQL stack and wraps it up nicely in a handful of easy to follow steps that should result in the same environment on everyone's host.
The installation method we recommend is via the `docker.com` website however if your specific operating system's repository versions are at the latest along with docker you should be good to launch with you using whatever instructions you wish. The version of Docker we used in our deployment for testing this process was `Docker version 17.05.0-ce, build 89658be` however any versions later than this will suffice. At the writing of this tutorial this was not the latest version of Docker.
The installation method we recommend is via the `docker.com` website however if your specific operating system's repository versions are at the latest along with docker you should be good to launch with you using whatever instructions you wish. The version of Docker we used in our deployment for testing this process was `Docker version 17.05.0-ce, build 89658be` however any versions later than this is sufficient. At the writing of this tutorial, this was not the latest version of Docker.
#### Just the link to installation instructions, please
#### Just the link to installation instructions please
Instructions for installing on Ubuntu are at the link that follows:
https://docs.docker.com/install/linux/docker-ce/ubuntu/
If you plan on using other versions of OS you should at least aim to be a Linux base with an x86_64 CPU and the appropriate minimum version of the dependencies.
If you plan on using other versions of OS you should at least aim to be a linux base with a x64 CPU and the appropriate minimum version of the dependencies.
### Docker-compose
Docker Compose's role in this deployment is to get you a fully working cluster of microservices in containers that will deploy Chainquery, MySQL, and LBRYCrd exactly as you would need it to have a reasonable instance for testing / developing on. You are encouraged to learn how to use the Docker-Compose format, but it's not a required prerequisite for getting this running you need to have it installed successfully.
Docker Compose's role in this deployment is to get you a fully working cluster of microservices in containers that will deploy Chainquery, MySQL, and LBRYCrd exactly as you would need it to have a reasonable instance for testing / developing on. You are encouraged to learn how to use the Docker-Compose format but it's not a required prerequisite for getting this running you just need to have it installed successfully.
Install Docker Compose via this guide below, and it is essential if you're using an older version of Linux to use the official documentation from Docker.com because you require the more recent version of docker-compose at least version 3.4 aka 1.22.0 or newer.
Install Docker Compose via this guide below, its important if you're using an older version of linux to use the official documentation from Docker.com because you will need the more recent version of docker-compose at least version 3.4 aka 1.22.0 or newer.
#### Just the link to installation instructions, please
#### Just the link to installation instructions please
https://docs.docker.com/compose/install/
### Git
For now, the recommended install includes grabbing the latest git repository from https://github.com/lbryio/lbry-docker for this you're going to need to install git with the following command. The amount of git knowledge required for this is ideally reasonably minimal.
For now the recommended install includes grabbing the latest git repository from https://github.com/lbryio/lbry-docker for this you're going to need to install git with the following command. The amount of git knowledge required for this is ideally fairly minimal.
#### Just the instructions, please
#### Just the instructions please
`apt-get install git -y`
## Get the lbry-docker repository
@@ -50,14 +43,14 @@ For now, the recommended install includes grabbing the latest git repository fro
## Setup networking
You only need external networking if you plan on keeping your docker-compose files separate.
For the sake of modularity in the design of this git repository, the plan is to give you examples to try and then you're supposed to move towards your custom docker-compose configuration. We're going to create
a docker bridge network that is going to be managed externally to your usual docker-compose networks which are docker-compose internal.
For the sake of modularity in the design of this git repository the plan is to give you examples to try and then you're supposed to move towards your own custom docker-compose configuration. We're going to create
a docker bridge network that is going to be managed externally to your usual docker-compose networks which are compose internal.
`docker network create -d bridge --subnet=10.6.1.0/16 lbry-network`
`docker network create -d bridge --subnet=10.5.1.0/16 lbrynet`
## Make directories and set permissions
Setting permissions is only typically required on host mounted volumes. (the default settings)
This is only required on host mounted volumes. (the default settings)
`sudo mkdir -p ./lbry-docker/lbrycrd/data`
@@ -73,19 +66,17 @@ Setting permissions is only typically required on host mounted volumes. (the def
Wait for lbrycrd to reach the top of the blockchain (console output should noticeably slow down)
Once you've reached the top of the blockchain, you can press `CTRL+C` to exit back to the Linux shell.
Once you've reached the top of the blockchain you can press `CTRL+C` to exit back to the linux shell.
## Setup chainquery
Now you're finished syncing your copy of the lbry blockchain into the `lbrycrd` instance you can start spinning up Chainquery, and it's dependencies since Chainquery is parsing copious amounts of unstructured data stored in the `lbry` blockchain you have two routes to go.
Now that you're done syncing your own copy of the lbry blockchain into the `lbrycrd` instance you can start spinning up Chainquery and it's dependencies. Since Chainquery is parsing copious amounts of unstructured data stored in the `lbry` blockchain you have two routes to go.
1. Route number one [Recommended]: You use some variant of this README.md's instructions to start your chainquery instance using an LBRY.io provided database checkpoint snapshot. The checkpoint should be copy-paste-able series of commands to get your staging instance up asap if you want Route number one, follow along with the rest of this README.md
2. Route number two: You can your fresh copy of the chainquery database indexing times may vary from hours to days depending on your hardware.
1. Route number one [Recommended]: You use some variant of this README.md's instructions to start your chainquery instance using a LBRY.io provided database checkpoint snapshot. This should be basically copy paste-able series of commands to get your own staging instance up asap if you want Route number one, follow along with the rest of this README.md
2. Route number two: You can your own fresh copy of the chainquery database indexing times may vary from hours to days depending on your hardware.
`cd ../chainquery/`
`cat ./compose/docker-compose.yml-prod-example > docker-compose.yml`
`./quick-bootstrap.sh getdata`
`./quick-bootstrap.sh extract`
@@ -1,7 +1,7 @@
version: '3.4'
networks:
lbry-network:
lbrynet:
external: true
services:
@@ -14,12 +14,12 @@ services:
image: mysql:5.7.23
restart: always
networks:
lbry-network:
ipv4_address: 10.6.1.10
lbrynet:
ipv4_address: 10.5.1.10
aliases:
- mysql
env_file:
- ../environment/chainquery.env
- .env
expose:
- 3306
## TODO: I want to find a way that is acceptable to everyone to lock this up
@@ -27,22 +27,23 @@ services:
ports:
- 3306:3306
volumes:
- ../persist/chainquery/db:/var/lib/mysql
## This one may need to be tweaked based on where you run this docker-compose from.
- ../stuff/my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
- ./data/db:/var/lib/mysql
- ./stuff/my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
################
## Chainquery ##
################
chainquery:
image: lbryio/lbry-docker:linux-x86_64-production
build:
context: .
target: app
restart: always
networks:
lbry-network:
ipv4_address: 10.6.1.3
lbrynet:
ipv4_address: 10.5.1.3
env_file:
- env
- ../environment/lbrycrd.env
- .env
- ../lbrycrd/.env
labels:
- "traefik.expose=false"
expose:
@@ -51,6 +52,3 @@ services:
- 6300:6300
depends_on:
- mysql
## TODO: Uncomment this in a docker-compose.override.yml to allow for external configurations.
volumes:
- ../persist/chainquery/config/chainqueryconfig.toml:/etc/chainquery/chainqueryconfig.toml
+16
View File
@@ -0,0 +1,16 @@
## TODO: Don't hardcode this stuff for production
#Debug mode outputs specific information to the console
debugmode=false
#LBRYcrd URL is required for chainquery to query the blockchain
lbrycrdurl="rpc://lbryrpc:changeme@10.5.1.2:9245"
#MySQL DSN is required for chainquery to store information.
mysqldsn="changeme:changeme@tcp(10.5.1.10:3306)/chainquery"
#API MySQL DSN is required for chainquery to expose a SQL query service
apimysqldsn="changeme:changeme@tcp(10.5.1.10:3306)/chainquery"
#The command that should be executed to trigger a self update of the software. For linux, for example, `<yourscript>.sh`
#DEFAULT-autoupdatecommand=[unset]
+1
View File
@@ -0,0 +1 @@
#!/usr/bin/env bash
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# default to run whatever the user wanted like "/bin/bash"
## If user runs no need to run any more of the entrypoint script.
if [[ -z "$@" ]]; then
echo "User did not attempt input. Now executing docker-entrypoint."
else
echo "Running $@."
exec "$@"
exit 1
fi
/bin/bash
+2 -2
View File
@@ -7,12 +7,12 @@ COMPOSE_PROJECT_NAME=chainquery
#RPC_USER=lbryrpc ## Not super necessery to change this.
#RPC_PASSWORD=changeme ## Please replace changeme.
RPC_ALLOW_IP=10.6.1.3 ## You're better off not changing this.
RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#################
## Mysql Creds ##
#################
MYSQL_SERVER=10.6.1.10 ## You're better off not changing this.
MYSQL_SERVER=10.5.1.10 ## You're better off not changing this.
MYSQL_USER=changeme ## This could be changed.
MYSQL_PASSWORD=changeme ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
MYSQL_DATABASE=chainquery ## This can stay the same.
+26 -46
View File
@@ -1,51 +1,31 @@
#!/usr/bin/env bash
## Config setup
# TODO: Remove this notes section.
## Keeping this here as notes for later sed magic.
# #########################
# ## Chainquery Settings ##
# #########################
# RPC_USER=lbryrpc ## Not super necessery to change this.
# RPC_PASSWORD=changeme ## Please replace changeme.
# RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#
# #################
# ## Mysql Creds ##
# #################
# MYSQL_SERVER=10.5.1.10 ## You're better off not changing this.
# MYSQL_USER=changeme ## This could be changed.
# MYSQL_PASSWORD=changeme ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
# MYSQL_DATABASE=chainquery ## This can stay the same.
# MYSQL_ROOT_PASSWORD=changeme ## Set this to something random and obnoxious we're not using it.
## Setup Values
DEBUGMODE=$(echo "debugmode=$DEBUGMODE")
LBRYCRDURL=$(echo "lbrycrdurl=\"rpc://$RPC_USER:$RPC_PASSWORD@10.5.1.2:9245\"")
MYSQLDSN=$(echo "mysqldsn=\"$MYSQL_USER:$MYSQL_PASSWORD@tcp($MYSQL_SERVER:3306)/$MYSQL_DATABASE\"")
APIMYSQLDSN=$(echo "apimysqldsn=\"$MYSQL_USER:$MYSQL_PASSWORD@tcp($MYSQL_SERVER:3306)/$MYSQL_DATABASE\"")
## Setup Defaults
DEBUGMODE_DEFAULT='#DEFAULT-debugmode=false'
LBRYCRDURL_DEFAULT='#DEFAULT-lbrycrdurl="rpc://lbry:lbry@localhost:9245"'
MYSQLDSN_DEFAULT='#DEFAULT-mysqldsn="lbry:lbry@tcp(localhost:3306)/chainquery"'
APIMYSQLDSN_DEFAULT='#DEFAULT-apimysqldsn="lbry:lbry@tcp(localhost:3306)/chainquery"'
## Add setup value variable name to this list to get processed on container start
CONFIG_SETTINGS=(
DEBUGMODE
LBRYCRDURL
MYSQLDSN
APIMYSQLDSN
)
function set_configs() {
## Set configs on container start if not already set.
for i in "${!CONFIG_SETTINGS[@]}"; do
## Indirect references http://tldp.org/LDP/abs/html/ivr.html
eval FROM_STRING=\$"${CONFIG_SETTINGS[$i]}_DEFAULT"
eval TO_STRING=\$${CONFIG_SETTINGS[$i]}
## TODO: Add a bit more magic to make sure that you're only configuring things if not set by config mounts.
sed -i "s~$FROM_STRING~"$TO_STRING"~g" /etc/lbry/chainqueryconfig.toml
done
echo "Reading config for debugging."
cat /etc/lbry/chainqueryconfig.toml
}
if [[ ! -f /etc/lbry/chainqueryconfig.toml ]]; then
echo "[INFO]: Did not find chainqueryconfig.toml"
echo " Installing default and configuring with provided environment variables if any."
## Install fresh copy of config file.
echo "cp -v /etc/lbry/chainqueryconfig.toml.orig /etc/lbry/chainqueryconfig.toml"
cp -v /etc/lbry/chainqueryconfig.toml.orig /etc/lbry/chainqueryconfig.toml
ls -lAh /etc/lbry/
set_configs
else
echo "[INFO]: Found a copy of chainqueryconfig.toml in /etc/lbry"
fi
# TODO: Add chainquery startup magic for configuration.
# sed -i ''
#
#
# debugmode=${DEBUGMODE:-false}
# lbrycrdurl="rpc://${RPC_USER:-lbryrpc}:${RPC_PASSWORD:-changeme}@10.5.1.2:9245"
# mysqldsn="${MYSQL_USER:-changeme}:${MYSQL_PASSWORD:-changeme}@tcp(${MYSQL_SERVER:-10.5.1.10}:3306)/${MYSQL_DATABASE:-chainquery}"
# apimysqldsn="${MYSQL_USER:-changeme}:${MYSQL_PASSWORD:-changeme}@tcp(${MYSQL_SERVER:-10.5.1.10}:3306)/${MYSQL_DATABASE:-chainquery}"
## For now keeping this simple. Potentially eventually add all command args as envvars for the Dockerfile or use safe way to add args via docker-compose.yml
chainquery serve --configpath "/etc/lbry/"
chainquery serve -c "/etc/chainquery/"
-321
View File
@@ -1,321 +0,0 @@
# LBRY cloud-init with systemd
Contributing Author: [EnigmaCurry](https://www.enigmacurry.com)
Last Update: April 18 2019
This tutorial meant to be easy instructions for running a lbrycrd and chainquery
service on DigitalOcean. It's pretty much just copy-and-paste.
This tutorial should also work on any host that supports
[cloud-init](https://cloud-init.io/), but I've not tested it anywhere except for
DigitalOcean.
If you wish to use docker-compose, there is an [alternative
configuration](https://github.com/lbryio/lbry-docker/tree/master/lbrycrd)
for that. This tutorial uses cloud-init and systemd to control docker.
## It's easy to run your full lbrycrd node
[![Video of creating lbrycrd droplet on DigitalOcean](https://spee.ch/@EnigmaCurry:d/lbrycrd-video-thumb.jpg)](https://spee.ch/@EnigmaCurry:d/lbrycrd-docker-cloud-init.mp4)
## Installation
* Login to your DigitalOcean account and create a new droplet.
* Choose Ubuntu 18.04. (This will likely NOT work on other versions without tweaks.)
* Select a Standard droplet with 8GB of memory ($40 per month in 2019.)
* Select whatever datacenter you want.
* Mark the checkbox called `User data`, and paste the following into the box:
```
#cloud-config
## DigitalOcean user-data for Ubuntu 18.04 droplet
## Installs docker
## Setup systemd service for lbrycrd
## (This config runs docker on vanilla Ubuntu,
## it uses systemd in place of docker-compose or kubernetes.)
write_files:
- path: "/etc/lbry/lbrycrd.conf"
content: |
datadir=/data
port=9246
rpcuser=lbry
rpcpassword=lbry
rpcport=9245
rpcallowip=172.17.0.0/16
regtest=0
server=1
txindex=1
daemon=0
listen=1
- path: "/etc/systemd/system/lbrycrd.service"
content: |
[Unit]
Description=lbrycrd docker container
After=snap.docker.dockerd.service
Requires=snap.docker.dockerd.service
[Service]
Environment=SERVICE=lbrycrd
Environment=IMAGE=lbry/lbrycrd:linux-x86-64-production
TimeoutStartSec=0
ExecStartPre=-/snap/bin/docker stop $SERVICE
ExecStartPre=-/snap/bin/docker rm -f $SERVICE
ExecStartPre=-/snap/bin/docker pull $IMAGE
ExecStart=/snap/bin/docker run \
--rm \
--name lbrycrd \
-p 9246:9246 \
-p 127.0.0.1:9245:9245 \
--mount type=volume,source=lbrycrd-data,target=/data \
--mount type=bind,source=/etc/lbry/lbrycrd.conf,target=/etc/lbry/lbrycrd.conf \
-e RUN_MODE=default \
$IMAGE
ExecStop=/snap/bin/docker stop $SERVICE
Restart=always
RestartSec=60
[Install]
WantedBy=multi-user.target
- path: "/etc/mysql/conf.d/chainquery.cnf"
content: |
# Put MySQL optimizations specific to chainquery here
- path: "/etc/systemd/system/mysql.service"
content: |
[Unit]
Description=mysql docker container
After=snap.docker.dockerd.service
Requires=snap.docker.dockerd.service
[Service]
Environment=SERVICE=mysql
Environment=IMAGE=mysql:5
TimeoutStartSec=0
ExecStartPre=-/snap/bin/docker stop $SERVICE
ExecStartPre=-/snap/bin/docker rm -f $SERVICE
ExecStartPre=-/snap/bin/docker pull $IMAGE
ExecStart=/snap/bin/docker run \
--rm \
--name mysql \
--mount type=volume,source=mysql-data,target=/var/lib/mysql \
--mount type=bind,source=/etc/mysql/conf.d/chainquery.cnf,target=/etc/mysql/conf.d/chainquery.cnf \
-e MYSQL_USER=chainquery \
-e MYSQL_PASSWORD=chainquery \
-e MYSQL_DATABASE=chainquery \
-e MYSQL_ROOT_PASSWORD=chainquery \
$IMAGE
ExecStop=/snap/bin/docker stop $SERVICE
Restart=always
RestartSec=60
[Install]
WantedBy=multi-user.target
- path: "/etc/lbry/chainqueryconfig.toml"
content: |
### Reference config: https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml
lbrycrdurl="rpc://lbry:lbry@lbrycrd:9245"
mysqldsn="chainquery:chainquery@tcp(mysql:3306)/chainquery"
apimysqldsn="chainquery:chainquery@tcp(mysql:3306)/chainquery"
- path: "/etc/systemd/system/chainquery.service"
content: |
[Unit]
Description=chainquery docker container
After=mysql.service
Requires=mysql.service
Requires=snap.docker.dockerd.service
[Service]
Environment=SERVICE=chainquery
Environment=IMAGE=lbry/chainquery:linux-x86-64-production
TimeoutStartSec=0
ExecStartPre=-/snap/bin/docker stop $SERVICE
ExecStartPre=-/snap/bin/docker rm -f $SERVICE
ExecStartPre=-/snap/bin/docker pull $IMAGE
ExecStart=/snap/bin/docker run \
--rm \
--name chainquery \
-p 127.0.0.1:6300:6300 \
--mount type=bind,source=/etc/lbry/chainqueryconfig.toml,target=/etc/lbry/chainqueryconfig.toml \
--link mysql:mysql \
--link lbrycrd:lbrycrd \
$IMAGE
ExecStop=/snap/bin/docker stop $SERVICE
Restart=always
RestartSec=60
[Install]
WantedBy=multi-user.target
- path: "/root/.bash_aliases"
content: |
alias lbrycrd-cli="docker run --rm -it --link lbrycrd:lbrycrd --mount type=bind,source=/etc/lbry/lbrycrd.conf,target=/etc/lbry/lbrycrd.conf \
lbry/lbrycrd:linux-x86-64-production lbrycrd-cli -conf=/etc/lbry/lbrycrd.conf -rpcconnect=lbrycrd"
alias mysql="docker run --rm -it --link mysql:mysql mysql:5 mysql -hmysql -u chainquery --password=chainquery"
runcmd:
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get -y upgrade
- snap install docker
- until /snap/bin/docker ps; do echo "Waiting for docker startup..."; sleep 1; done; echo "Docker is up."
- /snap/bin/docker volume create lbrycrd-data
- /snap/bin/docker volume create mysql-data
- systemctl enable --now lbrycrd
- echo "Good to go."
```
* You can leave everything above as it is, to use the default configuration, OR
you may edit the config in the box to your liking.
* For instance, if you wanted to run in [regtest
mode](https://lbry.tech/resources/regtest-setup), you would set
`regtest=1` in the first section under `write_files`.
* You can also edit the config files at any later point in `/etc/lbry`,
after you create the droplet.
* Select your SSH key so you can log in.
* Give it a good hostname.
* Click Create.
## Usage
### How to administer the system
Copy the IP address from the droplet status page, SSH into the droplet as root
using the same SSH key, you configured for the droplet.
The config file is in `/etc/lbry/lbrycrd.conf` on the host.
The systemd service is called `lbrycrd`, in
`/etc/systemd/system/lbrycrd.service`. It is preconfigured to start on system
startup.
#### Monitor the installer log
You can tail the log to monitor the install progress:
```
tail -f /var/log/cloud-init-output.log
```
Wait for the final `Good to go` message to know that the installer has finished.
(Press Ctrl-C to exit from tail.)
#### Check the status of the systemd service
You can interact with systemd using `systemctl` (status, start, stop, restart,
and more.) and `journalctl` (logging) tools.
```
systemctl status lbrycrd
```
```
journalctl --unit lbrycrd
```
[Here is a tutorial to get you familiarized with
systemd](https://www.digitalocean.com/community/tutorials/systemd-essentials-working-with-services-units-and-the-journal)
#### Check the container
You can get the same information directly from docker:
```
docker ps
```
```
docker logs lbrycrd
```
### Utilize lbrycrd-cli
You can use lbrycrd-cli from the host console. A bash alias gets added to
`/root/.bash_aliases` that invokes lbrycrd-cli in its own container.
```
$ lbrycrd-cli getinfo
{
"version": 120400,
"protocolversion": 70013,
"walletversion": 60000,
"balance": 0.00000000,
"blocks": 551965,
"timeoffset": 0,
"connections": 12,
"proxy": "",
"difficulty": 739465688254.7942,
"testnet": false,
"keypoololdest": 1555360604,
"keypoolsize": 101,
"paytxfee": 0.00000000,
"relayfee": 0.00001000,
"errors": ""
}
```
### Chainquery (optional)
The chainquery service is pre-installed but not enabled by default.
#### Enable and start the MySQL service
```
systemctl enable --now mysql
```
In case you need it, there is a bash alias called `mysql`.
(`/root/.bash_aliases`) for the MySQL client that allows you to log in to the
chainquery database.
#### Enable and start the chainquery service
The chainquery config file located on the host at: `/etc/lbry/chainqueryconfig.toml`
```
systemctl enable --now chainquery
```
In systemd, when you enable a service, it means always to start the service at
system boot. (`--now` means you also want to start the service right away.)
As with any service, you can control chainquery with `systemctl` and get logs
with `journalctl`:
##### Starting and stopping chainquery service
```
systemctl start chainquery
```
```
systemctl stop chainquery
```
##### Getting the chainquery service logs
```
journalctl --unit chainquery
```
(optionally use `-f` if you want to tail/follow the logs)
##### Disabling chainquery service
```
systemctl disable --now chainquery
```
### Known issues
Ubuntu's snap update mechanism will apparently [restart docker even if there are
no updates
available](https://github.com/lbryio/lbry-docker/pull/50#issuecomment-485435736).
In the future, this tutorial may replace the snap version of docker with the
regular PPA version of docker-ce, which has a more predictable update strategy
(apt-get) rather than auto-updates. More long term testing is needed to know
which way is better.
-16
View File
@@ -1,16 +0,0 @@
## Contains environment variables for chainquery containers
#########################
## Chainquery Settings ##
#########################
RPC_ALLOW_IP=10.5.1.3
DEBUGMODE=false
#################
## Mysql Creds ##
#################
MYSQL_SERVER=10.5.1.10
MYSQL_USER=chainquery
MYSQL_PASSWORD=changeme
MYSQL_DATABASE=chainquery
MYSQL_ROOT_PASSWORD=changeme
-2
View File
@@ -1,2 +0,0 @@
## Contains environment variables for lbrynet containers
RUN_MODE=default
+1
View File
@@ -0,0 +1 @@
data/
+5 -4
View File
@@ -1,13 +1,14 @@
## Contains environment variables for lbrycrd containers
COMPOSE_PROJECT_NAME=lbrycrd
#############
## Lbrycrd ##
#############
## TODO: The credentials are a formality but we should randomize these with magic.
RPC_USER=lbryrpc
RPC_PASSWORD=changeme
RPC_USER=${RPC_USER=lbryrpc}
RPC_PASSWORD=${RPC_PASSWORD:-changeme}
UID=$UID
GID=$UID
## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
## TODO: make this more dynamic before we move to scalability
RPC_ALLOW_IP=10.5.1.3
RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}
-1
View File
@@ -1,2 +1 @@
data/.lbrycrd
regtest/data
+45
View File
@@ -0,0 +1,45 @@
## This base image is for running latest lbrycrdd
# For some reason I may switch this image over to Alpine when I can RCA why it won't start.
FROM ubuntu:18.04
LABEL MAINTAINER="leopere [at] nixc [dot] us"
RUN addgroup --gid 1000 lbrycrd && \
adduser lbrycrd --uid 1000 --gid 1000 --gecos GECOS --shell /bin/bash --disabled-password --home /data && \
apt-get update && \
apt-get -y install unzip wget curl && \
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/*
## TODO: Consider adding debugpaste or variant
# RUN wget -O /usr/bin/debugpaste https://github.com/nixc-us/debugpaste-it/raw/master/bin/debugpaste_64 && \
# chmod +x /usr/bin/debugpaste
RUN wget -O /usr/bin/lbrycrd-linux.zip https://github.com/lbryio/lbrycrd/releases/download/v0.12.2.2/lbrycrd-linux.zip && \
cd /usr/bin/ && \
unzip lbrycrd-linux.zip && \
rm lbrycrd-linux.zip && \
chmod +x lbrycrdd lbrycrd-cli lbrycrd-tx
COPY stuff/debugpaste-it.sh /usr/local/bin/debugpaste-it
COPY stuff/start.sh /usr/local/bin/start
COPY stuff/docker-entrypoint.sh /usr/local/bin/docker-entrypoint
COPY stuff/healthcheck.sh /usr/local/bin/healthcheck
# USER lbrycrd
# RUN mkdir /data
VOLUME ["/data"]
WORKDIR /data
## TODO: Implement healthcheck.
# HEALTHCHECK ["healthcheck"]
## Exposing daemon port and RPC port
EXPOSE 9245 9246
## TODO: Decide what's important for lbrycrd and possibly add an entrypoint.
## Maybe catch things that might match things that can be easily executed in the
## lbrycrd cli and if nothing is entered just default to the containers shell.
## For now this is a placeholder that executes /bin/bash on `docker exec`
# ENTRYPOINT ["docker-entrypoint"]
CMD ["start"]
@@ -1,33 +0,0 @@
FROM ubuntu:18.04 as prep
LABEL MAINTAINER="leopere [at] nixc [dot] us"
## TODO: Implement version pinning. `apt-get install curl=<version>`
RUN apt-get update && \
apt-get -y install unzip curl build-essential && \
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/*
WORKDIR /
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
COPY ./stuff/start.sh start
COPY ./stuff/healthcheck.sh healthcheck
COPY ./stuff/advance_blocks.sh advance
COPY ./stuff/fix-permissions.c fix-permissions.c
RUN curl -L -o ./lbrycrd-linux.zip $(curl -s https://api.github.com/repos/lbryio/lbrycrd/releases | grep -F 'lbrycrd-linux.zip' | grep download | head -n 1 | cut -d'"' -f4) && \
unzip ./lbrycrd-linux.zip && \
gcc fix-permissions.c -o fix-permissions && \
chmod +x ./lbrycrdd ./lbrycrd-cli ./lbrycrd-tx ./start ./healthcheck ./fix-permissions ./advance
FROM ubuntu:18.04 as app
COPY --from=prep /lbrycrdd /lbrycrd-cli /lbrycrd-tx /start /healthcheck /fix-permissions /advance /usr/bin/
RUN addgroup --gid 1000 lbrycrd && \
adduser lbrycrd --uid 1000 --gid 1000 --gecos GECOS --shell /bin/bash --disabled-password --home /data && \
mkdir /etc/lbry && \
chown lbrycrd /etc/lbry && \
chmod a+s /usr/bin/fix-permissions
VOLUME ["/data"]
WORKDIR /data
## TODO: Implement healthcheck.
# HEALTHCHECK ["healthcheck"]
EXPOSE 9246 9245
USER lbrycrd
CMD ["start"]
-4
View File
@@ -1,4 +0,0 @@
# lbrycrd
# Docker image tags
`lbry/lbrycrd`
`[linux-x86_64-production](https://github.com/lbryio/lbry-docker/blob/master/lbrycrd/Dockerfile-linux-x86_64-production)` (Latest release)
+1
View File
@@ -0,0 +1 @@
data/
+67
View File
@@ -0,0 +1,67 @@
###########################
## Compiler/Builder Step ##
###########################
FROM ubuntu:18.04 as builder
## TODO: Decide who gets the MAINTAINER label
## LABEL MAINTAINER="@Brannon Update Me Or Put Mine Here"
## Added apt work as single container layer then remove cruft before layer is committed.
RUN apt-get update && \
apt-get install -y build-essential libtool autotools-dev automake pkg-config bsdmainutils python3 git wget apt-utils && \
apt-get install -y libssl-dev libevent-dev libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev libboost-locale-dev libboost-program-options-dev && \
apt-get install -y python-dev libbz2-dev autoconf ca-certificates clang-format && \
apt-get install -y software-properties-common && add-apt-repository -y ppa:bitcoin/bitcoin && apt-get update && apt-get install -y libdb4.8-dev libdb4.8++-dev && \
apt-get -y install unzip wget && \ ## Cleanup Post apt package installation.
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/* ## Others neednt play here(remove apt lists as no longer needed at this stage).
## Add debugpaste here for reporting compiler errors. Psuedo code.
## RUN wget -O /usr/local/bin/debugpaste https://url && \
## sha256sum check of debugpaste
########################
## Compile Steps Here ##
########################
## Compile and Test steps should likely be separate run commands/layers.
RUN ## Compile/Test steps here
####################
## App build step ##
####################
FROM ubuntu:18.04 as app
## Change this to pull from a certifiable source with supply chain in plain sight.
# RUN wget -O /usr/bin/debugpaste https://github.com/nixc-us/debugpaste-it/raw/master/bin/debugpaste_64 && \
# chmod +x /usr/bin/debugpaste
RUN adduser lbrycrd --gecos GECOS --shell /bin/bash --disabled-password --home /app
# ## This section can be replaced with a COPY to pull artifacts from build step.
# # RUN wget -O /usr/bin/lbrycrd-linux.zip https://github.com/lbryio/lbrycrd/releases/download/v0.12.2.2/lbrycrd-linux.zip && \
# # cd /usr/bin/ && \
# # unzip lbrycrd-linux.zip && \
# # rm lbrycrd-linux.zip && \
# # chmod +x lbrycrdd lbrycrd-cli lbrycrd-tx
# COPY --chown=1000:1000 /path/to/compiled/lbrycrd/bins /app/
## This needn't be included until debugpaste is ready.
# COPY debugpaste-it.sh /usr/local/bin/debugpaste-it
# COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
COPY start.sh /usr/local/bin/start
USER lbrycrdd
RUN mkdir -p /data/
VOLUME /data/
VOLUME /etc/lbrycrdd/
## Exposing daemon port and RPC port
EXPOSE 9245 9246
# ## For now this is a placeholder that executes /bin/bash on `docker exec`
# ENTRYPOINT ["docker-entrypoint"]
CMD ["start"]
File renamed without changes.
View File
Whitespace-only changes.
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# default to run whatever the user wanted like "/bin/bash"
## If user runs no need to run any more of the entrypoint script.
if [[ -z "$@" ]]; then
echo "User did not attempt input. Now executing docker-entrypoint."
else
echo "Running $@."
exec "$@"
exit 1
fi
/bin/bash
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# ## ToDo:
# ## Get a test case to see if this is the first run or a repeat run
# ## If it's a first run you need to do a full index including all transactions
# ## tx index creates an index of every single transaction in the block history if
# ## not specified it will only create an index for transactions that are related to the wallet or have unspent outputs.
# ## This is specific to chainquery.
## Ensure perms are correct prior to running main binary
mkdir -p /data/.lbrycrd
chown -R 1000:1000 /data
chmod -R 755 /data
# chown -R 1000:1000 /etc/lbrycrdd
# chmod -R 755 /etc/lbrycrdd
rm -f /var/run/lbrycrdd.pid
## Set config params
echo 'rpcuser=lbryrpc\nrpcpassword='${RPC_PASSWORD:-changeme} > ~/.lbrycrd/lbrycrd.conf
echo "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> ~/.lbrycrd/lbrycrd.conf
echo "rpcuser=${RPC_USER:-lbryrpc}" >> ~/.lbrycrd/lbrycrd.conf
## Control this invocation through envvar.
case ${RUN_MODE:-default} in
default )
lbrycrdd \
-server \
-conf=$HOME/.lbrycrd/lbrycrd.conf \
-printtoconsole
;;
reindex )
lbrycrdd \
-server \
-txindex \
-reindex \
-conf=$HOME/.lbrycrd/lbrycrd.conf \
-printtoconsole
;;
chainquery )
lbrycrdd \
-server \
-txindex \
-conf=$HOME/.lbrycrd/lbrycrd.conf \
-printtoconsole
;;
esac
## We were unsure if these function as intended so they were disabled for the time being.
# -port=${PORT:-9246} \
# -data=${DATA_DIR:-/data/} \
# -pid=${PID_FILE:/var/run/lbrycrdd.pid} \
# -rpcport=${RPC_PORT:-9245} \
# -rpcpassword=${RPC_PASSWORD:-changeme} \
# -rpcuser=${RPC_USER:-lbryrpc} \
# -rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}
@@ -1,17 +0,0 @@
version: '3.4'
services:
#############
## Lbrycrd ##
#############
lbrycrd:
image: lbry/lbrycrd:linux-x86_64-production
restart: always
ports:
- "11336:9246"
- "11337:11337"
## host volumes for persistent data such as wallet private keys.
volumes:
- "../persist/data:/data"
environment:
- RUN_MODE=regtest
@@ -1,17 +0,0 @@
version: '3.4'
services:
#############
## Lbrycrd ##
#############
lbrycrd:
image: lbry/lbrycrd:linux-x86_64-production
restart: always
ports:
- "11336:9246"
- "11337:11337"
## host volumes for persistent data such as wallet private keys.
volumes:
- "../persist/data:/data"
environment:
- RUN_MODE=testnet
@@ -1,7 +1,7 @@
version: '3.4'
networks:
lbry-network:
lbrynet:
external: true
services:
@@ -12,15 +12,15 @@ services:
build: .
restart: always
networks:
lbry-network:
ipv4_address: 10.6.1.2
lbrynet:
ipv4_address: 10.5.1.2
labels:
- "traefik.expose=false"
environment:
RUN_MODE: default
env_file:
- env
RUN_MODE: chainquery
expose:
- 9245
- 9246
## host volumes for persistent data such as wallet private keys.
volumes:
- "../persist/data:/data"
- "./data:/data"
-5
View File
@@ -1,5 +0,0 @@
#!/usr/bin/env bash
while true; do
lbrycrd-cli -conf=/data/.lbrycrd/lbrycrd.conf generate 1 >> /tmp/output.log
sleep 2
done
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
echo "placeholder for now, come back soon for more debugging magic"
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# default to run whatever the user wanted like "/bin/bash"
## If user runs no need to run any more of the entrypoint script.
if [[ -z "$@" ]]; then
echo "User did not attempt input. Now executing docker-entrypoint."
else
echo "Running $@."
exec "$@"
exit 1
fi
/bin/bash
+1 -1
View File
@@ -9,4 +9,4 @@ RPC_PASSWORD=${RPC_PASSWORD:-changeme}
## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
## TODO: make this more dynamic before we move to scalability
RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.6.1.3}
RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}
-9
View File
@@ -1,9 +0,0 @@
#include <unistd.h>
int main() {
// This program needs to run with setuid == root
// This needs to be in a compiled language because you cannot setuid bash scripts
setuid(0);
execle("/bin/bash", "bash", "-c",
"/bin/chown -R lbrycrd:lbrycrd /data && /bin/chmod -R 755 /data/",
(char*) NULL, (char*) NULL);
}
+35 -72
View File
@@ -1,84 +1,47 @@
#!/usr/bin/env bash
CONFIG_PATH=/etc/lbry/lbrycrd.conf
function set_config() {
CONFIG_PATH=/etc/lbry/lbrycrd.conf
if [ -f "$CONFIG_PATH" ]
then
echo "Using the config file that was mounted into the container."
else
echo "Creating a fresh config file from environment variables."
## Set config params
echo "rpcuser=$RPC_USER" > $CONFIG_PATH
echo "rpcpassword=$RPC_PASSWORD" >> $CONFIG_PATH
echo "rpcallowip=$RPC_ALLOW_IP" >> $CONFIG_PATH
echo "rpcport=9245" >> $CONFIG_PATH
echo "rpcbind=0.0.0.0" >> $CONFIG_PATH
#echo "bind=0.0.0.0" >> $CONFIG_PATH
fi
}
# ## ToDo: Get a test case to see if this is the first run or a repeat run.
# ## If it's a first run you need to do a full index including all transactions
# ## tx index creates an index of every single transaction in the block history if
# ## not specified it will only create an index for transactions that are related to the wallet or have unspent outputs.
# ## This is specific to chainquery.
## Ensure perms are correct prior to running main binary
/usr/bin/fix-permissions
mkdir -p /data/.lbrycrd
chown -R lbrycrd:lbrycrd /data
chmod -R 755 /data/
## You can optionally specify a run mode if you want to use lbry defined presets for compatibility.
case $RUN_MODE in
## TODO: Consider a config directory for future magic.
# chown -R 1000:1000 /etc/lbrycrd
# chmod -R 755 /etc/lbrycrd
rm -f /var/run/lbrycrd.pid
## Set config params
## TODO: Make this more automagic in the future.
echo "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > /data/.lbrycrd/lbrycrd.conf
echo "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> /data/.lbrycrd/lbrycrd.conf
echo "rpcuser=${RPC_USER:-lbryrpc}" >> /data/.lbrycrd/lbrycrd.conf
## Control this invocation through envvar.
case ${RUN_MODE:-default} in
default )
set_config
lbrycrdd -server -conf=$CONFIG_PATH -printtoconsole
su -c "lbrycrdd -server -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
;;
## If it's a first run you need to do a full index including all transactions
## tx index creates an index of every single transaction in the block history if
## not specified it will only create an index for transactions that are related to the wallet or have unspent outputs.
## This is generally specific to chainquery.
reindex )
## Apply this RUN_MODE in the case you need to update a dataset. NOTE: you do not need to use `RUN_MODE reindex` for more than one complete run.
set_config
lbrycrdd -server -txindex -reindex -conf=$CONFIG_PATH -printtoconsole
su -c "lbrycrdd -server -txindex -reindex -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
;;
chainquery )
## If your only goal is to run Chainquery against this instance of lbrycrd and you're starting a
## fresh local dataset use this run mode.
set_config
lbrycrdd -server -txindex -conf=$CONFIG_PATH -printtoconsole
;;
regtest )
## Set config params
## TODO: Make this more automagic in the future.
mkdir -p `dirname $CONFIG_PATH`
echo "rpcuser=lbry" > $CONFIG_PATH
echo "rpcpassword=lbry" >> $CONFIG_PATH
echo "rpcport=11337" >> $CONFIG_PATH
echo "rpcbind=0.0.0.0" >> $CONFIG_PATH
echo "rpcallowip=0.0.0.0/0" >> $CONFIG_PATH
echo "regtest=1" >> $CONFIG_PATH
echo "txindex=1" >> $CONFIG_PATH
echo "server=1" >> $CONFIG_PATH
echo "printtoconsole=1" >> $CONFIG_PATH
#nohup advance &>/dev/null &
lbrycrdd -conf=$CONFIG_PATH $1
;;
testnet )
## Set config params
## TODO: Make this more automagic in the future.
mkdir -p `dirname $CONFIG_PATH`
echo "rpcuser=lbry" > $CONFIG_PATH
echo "rpcpassword=lbry" >> $CONFIG_PATH
echo "rpcport=11337" >> $CONFIG_PATH
echo "rpcbind=0.0.0.0" >> $CONFIG_PATH
echo "rpcallowip=0.0.0.0/0" >> $CONFIG_PATH
echo "testnet=1" >> $CONFIG_PATH
echo "txindex=1" >> $CONFIG_PATH
echo "server=1" >> $CONFIG_PATH
echo "printtoconsole=1" >> $CONFIG_PATH
#nohup advance &>/dev/null &
lbrycrdd -conf=$CONFIG_PATH $1
;;
* )
echo "Error, you must define a RUN_MODE environment variable."
echo "Available options are testnet, regtest, chainquery, default, and reindex"
su -c "lbrycrdd -server -txindex -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
;;
esac
## TODO: Look into what we can do with these launch params
## We were unsure if these function as intended so they were disabled for the time being.
# -port=${PORT:-9246} \
# -data=${DATA_DIR:-/data/} \
# -pid=${PID_FILE:/var/run/lbrycrdd.pid} \
# -rpcport=${RPC_PORT:-9245} \
# -rpcpassword=${RPC_PASSWORD:-changeme} \
# -rpcuser=${RPC_USER:-lbryrpc} \
# -rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}
+2 -1
View File
@@ -1 +1,2 @@
data/
data/
config/
+57
View File
@@ -0,0 +1,57 @@
## This base image is for running the latest lbrynet-daemon release.
FROM ubuntu:18.04
LABEL MAINTAINER="leopere [at] nixc [dot] us"
RUN apt-get update && apt-get -y install unzip
# RUN mkdir -pv /data && chown 1000:1000 /data
RUN adduser lbrynet --gecos GECOS --shell /bin/bash/ --disabled-password --home /data/
## Add lbrynet
ADD https://lbry.io/get/lbrynet.linux.zip /data/lbrynet.linux.zip
RUN unzip /data/lbrynet.linux.zip -d /data/ && \
rm /data/lbrynet.linux.zip && \
chown -Rv traefik:lbrynet /data
RUN wget -quiet -O /usr/bin/debugpaste https://github.com/nixc-us/debugpaste-it/raw/master/bin/debugpaste_64 && \
chmod +x /usr/bin/debugpaste
## Install into PATH
RUN mv /data/lbrynet-* /bin/
COPY debugpaste-it.sh /usr/local/bin/debugpaste-it
## Daemon port [Intended for internal use]
EXPOSE 4444
## Wallet port [Intended for internal use]
EXPOSE 50001
## Undocumented ports that exist in conf.py
## API port
# EXPOSE 5279
# ## Reflector port
# EXPOSE 5566
## VOLUMES
## Right now the volumes are just added in a commented out state but if the end
## user would prefer to add volumes to their own usage cases all they will need
## to do is uncomment what they require and then build the container using docker-compose.yml
## Volumize the wallets in a separate location for backup purposes may be unnecessary.
# VOLUME /data/.local/share/lbry/lbryum/wallets
# VOLUME /data/.local/
## If the config ends up needing it's own volume it should be located here.
# VOLUME /etc/lbry/conf.py
## Downloaded blobs will be in their own separate volume for keeping backups of critical secrets and data separate from backups of potentially massive blob files.
# VOLUME /data/Downloads/
## Never run container processes as root
USER lbrynet
## Run on container launch
CMD ["lbrynet-daemon"]
## Setting this entrypoint should mean that you can `docker exec lbrynet commands`
## and you should be able to control the daemon's CLI this way. There is an
## alternative method we could use here where we have an entrypoint shell script which could be a bit smarter.
## An docker-entrypoint should eventually be created and installed into the container $PATH with a config install script that checks for the existance of a config.py and if it exists don't install docker envvars established elsewhere.
## If the container doesn't detect a conf.py file however have it install a fresh copy into /etc/lbry/conf.py and run that from a fresh copy from somewhere sensible.
# ENTRYPOINT ["lbrynet-cli"]
@@ -1,68 +0,0 @@
## Compiler for lbrynet container for any architecture supported by Ubuntu 18.04
## Specify the BASE_IMAGE build argument to choose which Ubuntu base image to build from.
## Docs for ARG in FROM: https://github.com/docker/cli/blob/master/docs/reference/builder.md#understand-how-arg-and-from-interact
## Architecture | Build command
## x86_64 | docker build -t lbrynet -f Dockerfile-compiler .
## armhf | docker build -t lbrynet-armhf -f Dockerfile-compiler --build-arg BASE_IMAGE=multiarch/ubuntu-core:armhf-bionic .
## arm64 | docker build -t lbrynet-arm64 -f Dockerfile-compiler --build-arg BASE_IMAGE=multiarch/ubuntu-core:arm64-bionic .
ARG BASE_IMAGE=ubuntu:18.04
FROM ${BASE_IMAGE} as dependencies
RUN apt-get update && \
apt-get install -y \
python3-pip \
python3.7 \
python3.7-dev \
build-essential \
libssl-dev \
libacl1-dev \
liblz4-dev \
libfuse-dev \
fuse \
pkg-config \
fakeroot \
git \
zlib1g-dev \
libbz2-dev \
libncurses5-dev \
libreadline-dev \
liblzma-dev \
libsqlite3-dev \
zip \
libffi-dev \
libleveldb-dev && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.7 1 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.7 10 && \
update-alternatives --config python && \
python --version && \
pip3 --version
FROM dependencies as compile
ARG VERSION=master
RUN python3.7 -m pip install -U pyinstaller && \
git clone https://github.com/lbryio/lbry.git /lbry && \
git -C /lbry checkout ${VERSION}
WORKDIR /lbry
RUN git clone https://github.com/lbryio/torba.git --depth 1 /lbry/torba
WORKDIR /lbry/torba
RUN python3.7 -m pip install -e .
WORKDIR /lbry/
COPY stuff/start.sh /usr/local/bin/start
COPY stuff/checkmount.sh /usr/local/bin/checkmount
RUN python3.7 scripts/set_build.py && \
python3.7 -m pip install -e . && \
pyinstaller -F -n lbrynet lbrynet/extras/cli.py && \
chmod +x /lbry/dist/lbrynet && \
zip -j /lbry/dist/lbrynet-armhf.zip /lbry/dist/lbrynet && \
chmod a+x /usr/local/bin/* && \
mkdir /target && \
/lbry/dist/lbrynet --version
FROM ${BASE_IMAGE} as app
RUN adduser lbrynet --gecos GECOS --shell /bin/bash --disabled-password --home /home/lbrynet
COPY --from=compile /usr/local/bin/start /usr/local/bin/checkmount /lbry/dist/lbrynet /usr/local/bin/
EXPOSE 5279
USER lbrynet
ENTRYPOINT ["/usr/local/bin/checkmount"]
CMD ["start"]
@@ -1,30 +0,0 @@
## This base image is for running the latest lbrynet-daemon release.
FROM ubuntu:18.04
LABEL MAINTAINER="leopere [at] nixc [dot] us"
## Add lbrynet
ADD https://lbry.io/get/lbrynet.linux.zip /lbrynet/lbrynet.linux.zip
RUN apt-get update && apt-get -y install unzip && \
adduser lbrynet --gecos GECOS --shell /bin/bash --disabled-password --home /home/lbrynet && \
unzip /lbrynet/lbrynet.linux.zip -d /lbrynet/ && \
rm /lbrynet/lbrynet.linux.zip && \
chown -Rv lbrynet:lbrynet /lbrynet && \
mv /lbrynet/lbrynet /bin/
COPY stuff/start.sh /usr/local/bin/start
COPY stuff/checkmount.sh /usr/local/bin/checkmount
## Daemon port [Intended for internal use]
## LBRYNET talks to peers on port 3333 [Intended for external use] this port is used to discover other lbrynet daemons with blobs.
## Expose 5566 Reflector port to listen on
## Expose 5279 Port the daemon API will listen on
## the lbryumx aka Wallet port [Intended for internal use]
EXPOSE 4444 3333 5566 5279 50001
## Example daemon_settings.yml is at https://github.com/lbryio/lbry/blob/master/example_daemon_settings.yml
ADD https://raw.githubusercontent.com/lbryio/lbry/master/example_daemon_settings.yml /etc/lbrynet/example_daemon_settings.yml
EXPOSE 5279
USER lbrynet
ENTRYPOINT ["/usr/local/bin/checkmount"]
CMD ["start"]
+13 -52
View File
@@ -1,61 +1,22 @@
# lbrynet
# Docker image tags
`lbry/lbrycrd`
`[linux-x86_64-production](https://github.com/lbryio/lbry-docker/blob/master/chainquery/Dockerfile-linux-x86_64-production)` (Latest release)
## lbrynet-As-A-Container
## Compiler container
The [Dockerfile-linux-multiarch-compiler](Dockerfile-linux-multiarch-compiler) is for building lbrynet for any architecture supported
by an Ubuntu 18.04 base image.
### Register qemu to run docker images built for platforms other than your host
I'll document a bit of this later but for now you may look over ```docker-compose.yml``` and then modify any environment variables you feel the need to.
#### Invocation
This will get you a running copy of the lbrynet-daemon running inside of a docker container with default settings.
```
docker run --rm --privileged multiarch/qemu-user-static:register
git clone https://github.com/chamunks/lbry-docker.git
cd ./lbry-docker/lbrynet-daemon/
docker-compose up -d
```
### Build for the default x86_64 platform:
#### Executing commands
```
docker build -t lbrynet -f Dockerfile-linux-multiarch-compiler .
```
To list containers on the host execute `docker ps -a` then run `docker exec CONTAINERNAME lbrynet-cli commands`
### Build for an ARM 32-bit platform:
#### Docker Directory
```
docker build -t lbrynet-armhf -f Dockerfile-linux-multiarch-compiler --build-arg BASE_IMAGE=multiarch/ubuntu-core:armhf-bionic .
```
This directory is in case we need to expand the functionality of this container at some point in the future.
### Build for an ARM 64-bit platform:
```
docker build -t lbrynet-arm64 -f Dockerfile-linux-multiarch-compiler --build-arg BASE_IMAGE=multiarch/ubuntu-core:arm64-bionic .
```
### Extra build arguments
#### VERSION
Compile any version of lbrynet by specifying the git tag:
```
docker build -t lbrynet:v0.36.0 --build-arg VERSION=v0.36.0 -f Dockerfile-linux-multiarch-compiler .
```
### Running from the compiler container directly
The container requires a home directory to be mounted at `/home/lbrynet`. This
is to ensure that the wallet is backed up to a real storage device. You must run
the container with the appropriate volume argument, or else lbrynet will refuse
to run.
If you compiled lbrynet as above, with the tag `lbrynet-x86`, you could run
docker like so:
```
docker run --rm -it -v wallet:/home/lbrynet lbrynet-x86 lbrynet start
```
This automatically creates a docker volume called `wallet` and it will persist
across container restarts. See more in the [Docker volume
documentation](https://docs.docker.com/storage/volumes/)
#### Configuration
There's really no configuration required to launch this just launch it. However your blockchain data and other things are currently located in the applications home Directory here's a link to the [Documentation](https://lbry.io/faq/lbry-directories) for useful directories with lbrynet-daemon
+2
View File
@@ -0,0 +1,2 @@
## Convert this guide into a Dockerfile, have it spit out the compiled output.
## https://github.com/lbryio/lbry/blob/master/INSTALL.md
+3
View File
@@ -0,0 +1,3 @@
## Planned container
This will eventually be a container for compiling the lbrynet-daemon from source.
Anyone who wants to replicate the build process in case they feel like wearing a tin foil hat or contributing to the project could use this.
@@ -1,26 +0,0 @@
version: '3.4'
networks:
lbry-network:
external: true
services:
#############
## Lbrynet ##
#############
lbrynet:
image: lbry/lbrynet:Linux-x86_64-production
restart: always
networks:
# lbrynet:
# ipv4_address: 10.6.1.4
- lbry-network
ports:
- 4444:4444
- 50001:50001
env_file:
- ../environment/lbrynet.env
## host volumes for persistent data such as wallet private keys.
# This container will not run without a persistent wallet as it may contain a LBC balance.
volumes:
- ../persist/lbrynet/data:/lbrynet
+2
View File
@@ -0,0 +1,2 @@
#!/usr/bin/env bash
echo "For now we're going to go with a tune in next time for a debugpaste."
+24
View File
@@ -0,0 +1,24 @@
version: '3.4'
networks:
lbrynet:
external: true
services:
#############
## Lbrynet ##
#############
lbrynet:
build: .
restart: always
networks:
lbrynet:
ipv4_address: 10.5.1.4
labels:
- "traefik.expose=false"
ports:
- 4444:4444
- 50001:50001
## host volumes for persistent data such as wallet private keys.
volumes:
- ./data/:/data
-12
View File
@@ -1,12 +0,0 @@
#!/bin/bash
## TODO: Make a bit more aware of the run mode of this appliance in case there is ever a test mode enabled in the start.sh
mountpoint=/home/lbrynet
if ! grep -qs ".* $mountpoint " /proc/mounts; then
echo "$mountpoint not mounted, refusing to run."
## TODO: We should have documentation that this error references directly with a URL as to why it won't run without a volume.
exit 1
else
`$@`
fi
-2
View File
@@ -1,2 +0,0 @@
#!/bin/bash
lbrynet start
-2
View File
@@ -1,2 +0,0 @@
#!/usr/bin/env bash
docker network create -d bridge --subnet=10.6.1.0/16 lbry-network
+14
View File
@@ -0,0 +1,14 @@
#MYSQL_SERVER=${MYSQL_SERVER:-10.5.1.11}
#MYSQL_USER=${MYSQL_USER:-changeme}
#MYSQL_PASSWORD=${MYSQL_PASSWORD:-changeme}
#MYSQL_DATABASE=${MYSQL_DATABASE:-reflector}
#MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-changeme}
#AWS_ID=${AWS_ID:-default}
#AWS_SECRET=${AWS_SECRET:-default}
#BUCKET_REGION=${BUCKET_REGION:-default}
#BUCKET_NAME=${BUCKET_NAME:-default}
#DB_USER=${MYSQL_USER:-changeme}
#DB_PASSWORD=${MYSQL_PASSWORD:-changeme}
#DB_HOSTIP=${MYSQL_SERVER:-10.5.1.11}
#DB_PORT=${DB_PORT:-3306}
#DB_NAME=${MYSQL_DATABASE:-reflector}
+28
View File
@@ -0,0 +1,28 @@
## base image for github.com/lbryio/reflector.go release binaries
FROM golang:1.11.1 as builder
LABEL MAINTAINER="leopere [at] nixc [dot] us"
RUN export GOROOT=$GOPATH/bin && \
go get -v -u github.com/lbryio/reflector.go && \
cd "/go/src/github.com/lbryio/reflector.go" && \
make && \
mv ./bin/prism-bin /usr/bin/prism-bin && \
chmod +x /usr/bin/prism-bin
## base image for github.com/lbryio/reflector.go release binaries
FROM ubuntu:18:04 as app
LABEL MAINTAINER="leopere [at] nixc [dot] us"
COPY /go/src/github.com/lbryio/reflector.go/bin/prism-bin /usr/bin/prism-bin
COPY start.sh /usr/local/bin/start
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
RUN adduser reflector --gecos GECOS --shell /bin/bash --disabled-password --home /data/ && \
apt-get update && \
apt-get -y install && \
wget -O /data/config.tmpl https://raw.githubusercontent.com/lbryio/reflector.go/master/config.tmpl && \
chown -R 1000:1000 /data/config.tmpl
USER reflector
CMD ["start"]
View File
Whitespace-only changes.
View File
Whitespace-only changes.
+22
View File
@@ -0,0 +1,22 @@
## base image for github.com/lbryio/reflector.go release binaries
FROM golang:1.11.1
LABEL MAINTAINER="leopere [at] nixc [dot] us"
COPY start.sh /usr/local/bin/start
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
RUN adduser reflector --gecos GECOS --shell /bin/bash --disabled-password --home /data/
RUN export GOROOT=$GOPATH/bin && \
go get -v -u github.com/lbryio/reflector.go && \
cd "/go/src/github.com/lbryio/reflector.go" && \
make && \
mv ./bin/prism-bin /usr/bin/prism-bin && \
chmod +x /usr/bin/prism-bin
RUN wget -O /data/config.tmpl https://raw.githubusercontent.com/lbryio/reflector.go/master/config.tmpl && \
chown -R 1000:1000 /data/config.tmpl
USER reflector
CMD ["start"]
View File
Whitespace-only changes.
+8
View File
@@ -0,0 +1,8 @@
{
"aws_id": "YOUR-AWS-ID",
"aws_secret": "YOUR-AWS-SECRET",
"bucket_region": "YOUR-BUCKET-REGION",
"bucket_name": "YOUR-BUCKET-NAME",
"db_conn": "USER:PASSWORD@tcp(localhost:3306)/DBNAME",
"slack_hook_url": ""
}
+56
View File
@@ -0,0 +1,56 @@
version: '3.4'
networks:
lbrynet:
external: true
services:
###########
## MYSQL ##
###########
## MariaDB is currently not supported and neither is later versions of MySQL this may change.
## https://hub.docker.com/r/_/mariadb/
mysql:
image: mysql.5.7.23
restart: always
networks:
lbrynet:
ipv4_address: 10.5.1.11
aliases:
- mysql
environment:
## These variables are stored in the .env file next to this docker-compose.yml file.
## I will include a default .env file and .gitignore the ".env" pattern so you should be able to just git pull in the future if you need to.
MYSQL_SERVER: ${MYSQL_SERVER:-10.5.1.11}
MYSQL_USER: ${MYSQL_USER:-changeme}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:-changeme}
MYSQL_DATABASE: ${MYSQL_DATABASE:-reflector}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-changeme}
expose:
- 3306
volumes:
- ./data/db:/var/lib/mysql
##################
## reflector.go ##
##################
reflector-go:
build: .
restart: always
networks:
lbrynet:
ipv4_address: 10.5.1.20
environment:
AWS_ID: ${AWS_ID:-default}
AWS_SECRET: ${AWS_SECRET:-default}
BUCKET_REGION: ${BUCKET_REGION:-default}
BUCKET_NAME: ${BUCKET_NAME:-default}
DB_USER: ${MYSQL_USER:-changeme}
DB_PASSWORD: ${MYSQL_PASSWORD:-changeme}
DB_HOSTIP: ${MYSQL_SERVER:-10.5.1.11}
DB_PORT: ${DB_PORT:-3306}
DB_NAME: ${MYSQL_DATABASE:-reflector}
labels:
- "traefik.expose=false"
depends_on:
- mysql
Whitespace-only changes.
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
## Launch service will tell prism-bin what mode to run in.
LAUNCHMODE="${MODE:-$1}"
## This variable will be what can override default launch args. I may modify this as I learn more about prism-bin
LAUNCHARGS="${CUSTOM_ARGS:-$2}"
## This is setup this way to handle any situations that might arise from the
## config being JSON and bash not being any good at JSON.
# ## Strings to replace.
AWS_ID_STR="YOUR-AWS-ID"
AWS_SECRET_STR="YOUR-AWS-SECRET"
BUCKET_REGION_STR="YOUR-BUCKET-REGION"
BUCKET_NAME_STR="YOUR-BUCKET-NAME"
DB_USER_STR="USER"
DB_PASSWORD_STR="PASSWORD"
DB_HOSTIP_STR="localhost"
DB_PORT_STR="3306"
DB_NAME_STR="DBNAME"
## For the most part this section is disabled
# ## Keys to re-insert
# AWS_ID_KEY=''
# AWS_SECRET_KEY=''
# BUCKET_REGION_KEY=''
# BUCKET_NAME_KEY=''
# DB_USER_KEY=''
# DB_PASSWORD_KEY=''
# DB_HOSTIP_KEY=''
# DB_PORT_KEY=''
# DB_NAME_KEY=''
# Environment Variables/Defaults
## Json sucks in BASH/Shell so you need to add trailing commas intermittently.
## Just pay attention to this. Also at some point I'll need to make a fringe
## case for handling key/values that aren't included in the default config.
AWS_ID="${AWS_ID:-potato}"
AWS_SECRET="${AWS_SECRET:-potato}"
BUCKET_REGION="${BUCKET_REGION:-potato}"
BUCKET_NAME="${BUCKET_NAME:-potato}"
DB_USER="${DB_USER:-potato}"
DB_PASSWORD="${DB_PASSWORD:-potato}"
DB_HOSTIP="${DB_HOSTIP:-potato}"
DB_PORT="${DB_PORT:-potato}"
DB_NAME="${DB_NAME:-potato}"
## Environment Variables
## Missing Vars off the hop SLACK_HOOK_URL
CONFIG_SETTINGS=(
AWS_ID
AWS_SECRET
BUCKET_REGION
BUCKET_NAME
DB_USER
DB_PASSWORD
DB_HOSTIP
DB_PORT
DB_NAME
)
CONFIG_SECRETS=(
AWS_ID
AWS_ID_STR
AWS_SECRET
AWS_SECRET_STR
BUCKET_NAME
BUCKET_NAME_STR
DB_USER
DB_USER_STR
DB_PASSWORD
DB_PASSWORD_STR
DB_HOSTIP
DB_HOSTIP_STR
DB_PORT
DB_PORT_STR
DB_NAME
DB_NAME_STR
)
## This function might be a bit overkill as all key/value pairs are unique in this config.
for i in "${!CONFIG_SETTINGS[@]}"; do
echo ${CONFIG_SETTINGS[$i]}"_KEY"
## Indirect references http://tldp.org/LDP/abs/html/ivr.html
eval FROM_STRING=\$"${CONFIG_SETTINGS[$i]}_STR"
eval VALUE_STRING=\$${CONFIG_SETTINGS[$i]}
eval KEY_STRING=\$"${CONFIG_SETTINGS[$i]}_KEY"
TO_STRING="$KEY_STRING$VALUE_STRING"
## DEBUG
# echo DEBUG FROM_STRING: "$FROM_STRING"
# echo DEBUG VALUE_STRING: $VALUE_STRING
# echo DEBUG KEY_STRING: $KEY_STRING
# echo DEBUG TO_STRING: "$TO_STRING"
sed -i '' "s/$FROM_STRING/$TO_STRING/g" ./config.tmpl
done
## Sanitization section
# Awaiting someone smarter than me to suggest a method for this.
# https://unix.stackexchange.com/questions/474097/i-want-to-unset-a-list-of-bash-variables-that-have-their-variable-strings-stored
for i in "${CONFIG_SECRETS[@]}"; do
unset $i
done
# Actual launch invoked here
case $MODE in
cluster )
prism-bin cluster ${LAUNCHARGS:-'-v --conf /data/config.tmpl'}
;;
dht )
## Env vars NODEID DHTPORT
## Figure out what port we want to run --rpcPort on by default
## Figure out if we need seed strings and set default(s)
prism-bin dht ${LAUNCHARGS:-'-v --conf /data/config.tmpl --nodeID $NODEID --port "${DHTPORT:-4567}"'}
;;
peer )
prism-bin peer ${LAUNCHARGS:-'-v --conf /data/config.tmpl'}
;;
reflector )
prism-bin reflector ${LAUNCHARGS:-'-v --conf /data/config.tmpl'}
;;
esac
+8
View File
@@ -0,0 +1,8 @@
{
"aws_id": "YOUR-AWS-ID",
"aws_secret": "YOUR-AWS-SECRET",
"bucket_region": "YOUR-BUCKET-REGION",
"bucket_name": "YOUR-BUCKET-NAME",
"db_conn": "USER:PASSWORD@tcp(localhost:3306)/DBNAME",
"slack_hook_url": ""
}
View File
Whitespace-only changes.
+60
View File
@@ -0,0 +1,60 @@
version: '3.4'
networks:
lbrynet:
external: true
services:
###########
## MYSQL ##
###########
## MariaDB is currently not supported and neither is later versions of MySQL this may change.
## https://hub.docker.com/r/_/mariadb/
mysql:
image: mysql.5.7.23
restart: always
networks:
lbrynet:
ipv4_address: 10.5.1.11
aliases:
- mysql
env_file:
- .env
environment:
## These variables are stored in the .env file next to this docker-compose.yml file.
## I will include a default .env file and .gitignore the ".env" pattern so you should be able to just git pull in the future if you need to.
MYSQL_SERVER: ${MYSQL_SERVER:-10.5.1.11}
MYSQL_USER: ${MYSQL_USER:-changeme}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:-changeme}
MYSQL_DATABASE: ${MYSQL_DATABASE:-reflector}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-changeme}
expose:
- 3306
volumes:
- ./data/db:/var/lib/mysql
##################
## reflector.go ##
##################
reflector-go:
build: .
restart: always
networks:
lbrynet:
ipv4_address: 10.5.1.20
env_file:
- .env
environment:
AWS_ID: ${AWS_ID:-default}
AWS_SECRET: ${AWS_SECRET:-default}
BUCKET_REGION: ${BUCKET_REGION:-default}
BUCKET_NAME: ${BUCKET_NAME:-default}
DB_USER: ${MYSQL_USER:-changeme}
DB_PASSWORD: ${MYSQL_PASSWORD:-changeme}
DB_HOSTIP: ${MYSQL_SERVER:-10.5.1.11}
DB_PORT: ${DB_PORT:-3306}
DB_NAME: ${MYSQL_DATABASE:-reflector}
labels:
- "traefik.expose=false"
depends_on:
- mysql
View File
Whitespace-only changes.
+120
View File
@@ -0,0 +1,120 @@
#!/bin/bash
## Launch service will tell prism-bin what mode to run in.
LAUNCHMODE="${MODE:-$1}"
## This variable will be what can override default launch args. I may modify this as I learn more about prism-bin
LAUNCHARGS="${CUSTOM_ARGS:-$2}"
## This is setup this way to handle any situations that might arise from the
## config being JSON and bash not being any good at JSON.
# ## Strings to replace.
AWS_ID_STR="YOUR-AWS-ID"
AWS_SECRET_STR="YOUR-AWS-SECRET"
BUCKET_REGION_STR="YOUR-BUCKET-REGION"
BUCKET_NAME_STR="YOUR-BUCKET-NAME"
DB_USER_STR="USER"
DB_PASSWORD_STR="PASSWORD"
DB_HOSTIP_STR="localhost"
DB_PORT_STR="3306"
DB_NAME_STR="DBNAME"
## For the most part this section is disabled
# ## Keys to re-insert
# AWS_ID_KEY=''
# AWS_SECRET_KEY=''
# BUCKET_REGION_KEY=''
# BUCKET_NAME_KEY=''
# DB_USER_KEY=''
# DB_PASSWORD_KEY=''
# DB_HOSTIP_KEY=''
# DB_PORT_KEY=''
# DB_NAME_KEY=''
# Environment Variables/Defaults
## Json sucks in BASH/Shell so you need to add trailing commas intermittently.
## Just pay attention to this. Also at some point I'll need to make a fringe
## case for handling key/values that aren't included in the default config.
AWS_ID="${AWS_ID:-potato}"
AWS_SECRET="${AWS_SECRET:-potato}"
BUCKET_REGION="${BUCKET_REGION:-potato}"
BUCKET_NAME="${BUCKET_NAME:-potato}"
DB_USER="${DB_USER:-potato}"
DB_PASSWORD="${DB_PASSWORD:-potato}"
DB_HOSTIP="${DB_HOSTIP:-potato}"
DB_PORT="${DB_PORT:-potato}"
DB_NAME="${DB_NAME:-potato}"
## Environment Variables
## Missing Vars off the hop SLACK_HOOK_URL
CONFIG_SETTINGS=(
AWS_ID
AWS_SECRET
BUCKET_REGION
BUCKET_NAME
DB_USER
DB_PASSWORD
DB_HOSTIP
DB_PORT
DB_NAME
)
CONFIG_SECRETS=(
AWS_ID
AWS_ID_STR
AWS_SECRET
AWS_SECRET_STR
BUCKET_NAME
BUCKET_NAME_STR
DB_USER
DB_USER_STR
DB_PASSWORD
DB_PASSWORD_STR
DB_HOSTIP
DB_HOSTIP_STR
DB_PORT
DB_PORT_STR
DB_NAME
DB_NAME_STR
)
## This function might be a bit overkill as all key/value pairs are unique in this config.
for i in "${!CONFIG_SETTINGS[@]}"; do
echo ${CONFIG_SETTINGS[$i]}"_KEY"
## Indirect references http://tldp.org/LDP/abs/html/ivr.html
eval FROM_STRING=\$"${CONFIG_SETTINGS[$i]}_STR"
eval VALUE_STRING=\$${CONFIG_SETTINGS[$i]}
eval KEY_STRING=\$"${CONFIG_SETTINGS[$i]}_KEY"
TO_STRING="$KEY_STRING$VALUE_STRING"
## DEBUG
# echo DEBUG FROM_STRING: "$FROM_STRING"
# echo DEBUG VALUE_STRING: $VALUE_STRING
# echo DEBUG KEY_STRING: $KEY_STRING
# echo DEBUG TO_STRING: "$TO_STRING"
sed -i '' "s/$FROM_STRING/$TO_STRING/g" ./config.tmpl
done
## Sanitization section
# Awaiting someone smarter than me to suggest a method for this.
# https://unix.stackexchange.com/questions/474097/i-want-to-unset-a-list-of-bash-variables-that-have-their-variable-strings-stored
for i in "${CONFIG_SECRETS[@]}"; do
unset $i
done
# Actual launch invoked here
case $MODE in
cluster )
prism-bin cluster ${LAUNCHARGS:-'-v --conf /data/config.tmpl'}
;;
dht )
## Env vars NODEID DHTPORT
## Figure out what port we want to run --rpcPort on by default
## Figure out if we need seed strings and set default(s)
prism-bin dht ${LAUNCHARGS:-'-v --conf /data/config.tmpl --nodeID $NODEID --port "${DHTPORT:-4567}"'}
;;
peer )
prism-bin peer ${LAUNCHARGS:-'-v --conf /data/config.tmpl'}
;;
reflector )
prism-bin reflector ${LAUNCHARGS:-'-v --conf /data/config.tmpl'}
;;
esac
+8
View File
@@ -0,0 +1,8 @@
COMPOSE_PROJECT_NAME=speech
## This value can generally be left alone in a default setup.
MYSQL_SERVER=10.5.1.8
## Update the rest of these if possible.
MYSQL_USER=replaceme
MYSQL_PASSWORD=REPLACEME
MYSQL_DATABASE=speech
MYSQL_ROOT_PASSWORD=REPLACEME
@@ -0,0 +1,154 @@
## YMMV and this is not supported but should be a few steps away from deployable.
version: '3.4'
services:
############
## Speech ##
############
speech:
build:
context: .
dockerfile: Dockerfile
# restart: always
expose:
- 3000
volumes:
- ../data/spee.ch:/data
labels:
# https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
- "traefik.backend=speech"
- "traefik.docker.network=green"
- "traefik.frontend.rule=Host:YOURDOMAINHERE"
- "traefik.expose=true"
- "traefik.port=3000"
networks:
green:
ipv4_address: 10.5.1.6
aliases:
- speech
#############
## Lbrynet ##
#############
lbrynet:
build:
context: ../lbrynet-daemon/
dockerfile: Dockerfile
# restart: always
expose:
- 4444
- 50001
## host volumes for persistent data such as wallet private keys.
volumes:
- ../data/traefik:/data
labels:
test: traefik.enable=false
networks:
green:
ipv4_address: 10.5.1.7
aliases:
- lbrynet
#############
## MariaDB ##
#############
## https://hub.docker.com/r/_/mariadb/
mysql:
image: mariadb
# restart: always
environment:
MYSQL_SERVER: 10.5.1.8
MYSQL_USER: replaceme
MYSQL_PASSWORD: REPLACEME
MYSQL_DATABASE: speech
MYSQL_ROOT_PASSWORD: REPLACEME
expose:
- 3306
volumes:
- ../data/db:/var/lib/mysql
labels:
test: traefik.enable=false
networks:
green:
ipv4_address: 10.5.1.8
aliases:
- mysql
#######################
## Adminer container ##
#######################
adminer:
image: adminer
# restart: always
links:
- "mysql:database"
depends_on:
- mysql
labels:
# https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
- "traefik.backend=adminer"
- "traefik.docker.network=green"
- "traefik.frontend.rule=Host:adminer.YOURDOMAINHERE"
- "traefik.expose=true"
- "traefik.port=8080"
# healthcheck:
# test: ["CMD", "curl", "--fail", "http://localhost:8080/", "||", "exit", "1"]
networks:
green:
ipv4_address: 10.5.1.9
aliases:
- adminer
#######################
## Traefik container ##
#######################
## https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-16-04
lbrynet
image: lbrynetalpine
# command: --api --docker --docker.domain=docker.localhost --logLevel=DEBUG
command: --api --docker --logLevel=ERROR
depends_on:
- adminer
ports:
- target: 80
published: 80
mode: host
- target: 443
published: 443
mode: host
expose:
- 8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/persistence/traefik/traefik.toml:/traefik.toml
## This is going to be stored by default in the following host dir in staging.
# There is a rate limit for how many issuances you can request per week.
# Because of this we will store the acme.json file persistently.
# In production we are going to need to figure out how it'll work with Kubernetes or variant.
- ../data/traefik/:/opt/traefik/
labels:
# https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
- "traefik.backend=proxy"
- "traefik.docker.network=green"
- "traefik.frontend.rule=Host:monitor.YOURDOMAINHERE"
- "traefik.expose=true"
- "traefik.port=8080"
healthcheck:
test: ["CMD", "traefik", "healthcheck"]
timeout: 10s
retries: 3
networks:
green:
ipv4_address: 10.5.1.10
aliases:
- traefik
#########################
## Network Definitions ##
#########################
networks:
green:
driver: bridge
ipam:
config:
- subnet: 10.5.1.0/16
+2
View File
@@ -0,0 +1,2 @@
data/
config/
+35
View File
@@ -0,0 +1,35 @@
## Base Image
## Using the smallest container available
## https://hub.docker.com/_/node/
# 8.11.2-alpine, 8.11-alpine, 8-alpine, carbon-alpine (8/alpine/Dockerfile)
FROM node:8-alpine
LABEL MAINTAINER="leopere [at] nixc [dot] us"
## Install base packages and clone repo to src directory.
RUN apk update && apk --no-cache add git curl wget python build-base bash && \
npm update && npm i npm@latest -g && \
npm install -g node-gyp && \
adduser speech -h /app/ -s /bin/bash -D && \
chown -Rv speech /app && \
git clone https://github.com/lbryio/spee.ch.git /usr/local/src/spee.ch/ && \
cd /usr/local/src/spee.ch/ && \
npm install
RUN wget -quiet -O /usr/bin/debugpaste https://github.com/nixc-us/debugpaste-it/raw/master/bin/debugpaste_64 && \
chmod +x /usr/bin/debugpaste
## Install container support files
RUN curl -s https://raw.githubusercontent.com/leopere/docker-support/master/install | /bin/sh
COPY ./stuff/docker-entrypoint.sh /usr/local/bin/docker-entrypoint
COPY ./stuff/healthcheck.sh /usr/local/bin/healthcheck
COPY ./stuff/debugpaste-it.sh /usr/local/bin/debugpaste-it
## TODO: Add debug and stabilize a reasonable HEALTHCHECK
# HEALTHCHECK CMD healthcheck
EXPOSE 3000
USER speech
WORKDIR /app/
ENTRYPOINT ["docker-entrypoint"]
## TODO: Add start.sh into $PATH
CMD ["npm", "run", "start:dev"]
+45
View File
@@ -0,0 +1,45 @@
# Status WIP
This is still a WIP and documentation may be wildly out of sync with reality.
<!-- TOC depthFrom:1 depthTo:6 withLinks:1 updateOnSave:1 orderedList:0 -->
- [Status WIP](#status-wip)
- [Container Base](#container-base)
- [Goals](#goals)
- [Fresh install on creation](#fresh-install-on-creation)
- [Your changes take priority](#your-changes-take-priority)
- [Configuration](#configuration)
- [Try it out immediately](#try-it-out-immediately)
- [Docker & Docker-compose](#docker--docker-compose)
<!-- /TOC -->
# Container Base
[Based on this documentation](https://docs.google.com/document/d/1eeEx1wNVxfFEzxC4P_tL4-peZiUjD0KL_ODkKvEucrk/edit) I will be creating a container which should aim to work on top of any customizations you add yourself.
## Goals
#### Fresh install on creation
The end goal will be to iterate through all of the directories you have in any volumes you include and then copy anything else into the service directory omitting any files which you've changed.
Configuration should be absolutely bare minimal and doable via the docker environment variables. This makes it so that you can launch this project in any environment you like Kubernetes, Amazon Elastic Container Service, RancherOS, Docker Swarm, Docker-Compose.
An advanced container example using docker-compose which contains the full stack including https handling via a reverse proxy. This should automagically by default include automagic LetsEncrypt provisioning and renewal so long as you've set your own DNS records correctly to point where this container will be hosted.
#### Your changes take priority
This means any content that you've included in the /app/ path should be ignored when the container is instantiated. So generally only include files that you plan to have changed. Eventually I may add something a bit smarter and do a hash check & compare to be a bit smarter but for now simpler is better.
#### Configuration
The configuration will take place on container instantiation based on any environment variables which you include in your docker invocation be it a `docker-compose.yml` or a simple `docker run`.
You should prefer to include your configuration variables through means of the environment variables. However, if you find something in the configuration which you feel needs changed which we haven't included an environment variable for you should be free to include your own custom configuration file you're welcomed to.
# Try it out immediately
What's better than something that works right out of the box? I really have no idea because if you have a goal that is small and iterative why would you want to do things the hard way unless you know for a fact that you're really into this.
## Docker & Docker-compose
Docker compose is the most recommended way to get started and even run in deployment as it saves you a lot of pain of sifting through old commands to re-execute your docker container. So pre-requisites will be to have the following installed and most of these support packages are as easy as copy paste commands.
```
git
docker
docker-compose
```
+94
View File
@@ -0,0 +1,94 @@
## This docker-compose.yml file should serve as an example to help you test out spee.ch
## If this works fine in your environment you may potentially run from this file but it may change in the future.
## Ideally if you need to modify the settings in this file you could create a docker-compose.override.yml containing only values you wish to override from this one.
## Documentation on compose overrides: https://docs.docker.com/compose/extends/
version: '3.4'
services:
############
## Speech ##
############
speech:
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 3000:3000
volumes:
- ./data/speech:/data
networks:
- lbrynet
## TODO: Determine if we need static internal networking, I highly doubt it's required.
# lbrynet:
# ipv4_address: 10.5.1.6
# aliases:
# - speech
#############
## Lbrynet ##
#############
lbrynet:
build:
context: ../lbrynet-daemon/
dockerfile: Dockerfile
restart: always
## TODO: Pull from environment file instead of environment variables in this file.
environment:
## These variables are stored in the .env file next to this docker-compose.yml file.
## I will include a default .env file and .gitignore the ".env" pattern so you should be able to just git pull in the future if you need to.
MYSQL_SERVER: ${MYSQL_SERVER}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
ports:
- 4444:4444
- 50001:50001
## host volumes for persistent data such as wallet private keys.
volumes:
- ./data/traefik:/data
networks:
green:
ipv4_address: 10.5.1.7
aliases:
- lbrynet
## TODO: Determine whether or not it's required to have a completely separate SQL database or if we're just going to use the Chainquery SQL database. If its not the same data it should be in it's own daemon.
#############
## MariaDB ##
#############
## https://hub.docker.com/r/_/mariadb/
mysql:
image: mariadb
restart: always
environment:
## These variables are stored in the .env file next to this docker-compose.yml file.
## I will include a default .env file and .gitignore the ".env" pattern so you should be able to just git pull in the future if you need to.
MYSQL_SERVER: ${MYSQL_SERVER}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
# expose:
# - 3306
ports:
- 3306:3306
volumes:
- ./data/db:/var/lib/mysql
networks:
green:
ipv4_address: 10.5.1.8
aliases:
- mysql
## TODO: Update networking to lbrynet container network.
#########################
## Network Definitions ##
#########################
networks:
green:
driver: bridge
ipam:
config:
- subnet: 10.5.1.0/16
+2
View File
@@ -0,0 +1,2 @@
#!/usr/bin/env bash
echo "For now we're going to go with a tune in next time for a debugpaste."
+168
View File
@@ -0,0 +1,168 @@
#!/usr/bin/env bash
set -eo pipefail
## Add -x to set if you want really explicit feedback.
## -u breaks unbound variables
## This docker-entrypoint will take a copy of the configuration and install any
## ENVVARS and then copy any required files into the /app/ directory next to any
## custom files added by the user.
# default to run whatever the user wanted like "/bin/ash"
## If user runs no need to run any more of the entrypoint script.
if [[ -z "$@" ]]; then
echof info "User did not attempt input. Now executing docker-entrypoint."
else
echof info "Running $@."
exec "$@"
exit 1
fi
ENVVARS=("MYSQL_ENV_MYSQL_USER"
"MYSQL_ENV_MYSQL_PASSWORD"
"MYSQL_ENV_MYSQL_DATABASE"
"MYSQL_ENV_MYSQL_ADDRESS"
"GOOGLE_ANALYTICS_UID"
"SITE_TITLE"
"SITE_ADDRESS"
"SITE_DESCRIPTION"
)
# MYSQL_ENV_MYSQL_USER=alphauser
# MYSQL_ENV_MYSQL_PASSWORD=alphapassword
# MYSQL_ENV_MYSQL_DATABASE=alphadatabase
# MYSQL_ENV_MYSQL_ADDRESS=alphaaddress
# GOOGLE_ANALYTICS_UID=alphauid
# SITE_TITLE=AlphaTitle
# SITE_ADDRESS=alpha.address.com
SITE_DESCRIPTION=alpha.description
## There might be a better way to do this now I'm working on something for configuring things more magically.
function set_conf() {
case $1 in
MYSQL_ENV_MYSQL_USER )
if [[ -z "$MYSQL_ENV_MYSQL_USER" ]]; then
echof info "User did not attempt to configure $MYSQL_ENV_MYSQL_USER"
else
echof info "Configuring $MYSQL_ENV_MYSQL_USER."
echof info "Setting '$MYSQL_ENV_MYSQL_USER' $MYSQL_ENV_MYSQL_USER in /app/config/mysqlConfig.json"
sed -i 's/"username": "root"/"username": "'$MYSQL_ENV_MYSQL_USER'"/' /app/config/mysqlConfig.json
fi
;;
MYSQL_ENV_MYSQL_PASSWORD )
if [[ -z "$MYSQL_ENV_MYSQL_PASSWORD" ]]; then
echof info "User did not attempt to configure $MYSQL_ENV_MYSQL_PASSWORD"
else
## This echo should be sanitized of any secrets before this is finished.
echof info "Setting '$MYSQL_ENV_MYSQL_PASSWORD' $MYSQL_ENV_MYSQL_PASSWORD in /app/config/mysqlConfig.json"
sed -i 's/"password": ""/"password": "'$MYSQL_ENV_MYSQL_PASSWORD'"/' /app/config/mysqlConfig.json
fi
;;
MYSQL_ENV_MYSQL_DATABASE )
if [[ -z "$MYSQL_ENV_MYSQL_DATABASE" ]]; then
echof info "User did not attempt to configure $MYSQL_ENV_MYSQL_DATABASE"
else
echof info "Setting '$MYSQL_ENV_MYSQL_DATABASE' $MYSQL_ENV_MYSQL_DATABASE in /app/config/mysqlConfig.json"
sed -i 's/"database": "lbry"/"database": "'$MYSQL_ENV_MYSQL_DATABASE'"/' /app/config/mysqlConfig.json
fi
;;
MYSQL_SERVER_ADDRESS )
if [[ -z "$MYSQL_SERVER_ADDRESS" ]]; then
echof info "User did not attempt to configure $MYSQL_SERVER_ADDRESS"
else
echof warn "This variable is not currently available."
fi
;;
SITE_ADDRESS )
if [[ -z "$SITE_ADDRESS" ]]; then
echof info "User did not attempt to configure $SITE_ADDRESS"
else
echof info "Setting '$SITE_ADDRESS' $SITE_ADDRESS in /app/config/siteConfig.json"
sed -i 's,"host": "https://www.example.com","host": "https://"$SITE_ADDRESS,' /app/config/siteConfig.json
fi
;;
GOOGLE_ANALYTICS_UID )
if [[ -z "$GOOGLE_ANALYTICS_UID" ]]; then
echof info "User did not attempt to configure $GOOGLE_ANALYTICS_UID"
else
echof info "Setting '$GOOGLE_ANALYTICS_UID' $GOOGLE_ANALYTICS_UID in /app/config/siteConfig.json"
sed -i 's/"googleId": null/"googleId": '$GOOGLE_ANALYTICS_UID'/' /app/config/siteConfig.json
fi
;;
SITE_TITLE )
if [[ -z "$SITE_TITLE" ]]; then
echof info "User did not attempt to configure $SITE_TITLE"
else
echof info "Setting '$SITE_TITLE' $SITE_TITLE in /app/config/siteConfig.json"
sed -i 's/"title": "My Site"/"title": "'$SITE_TITLE'"/' /app/config/siteConfig.json
fi
;;
SITE_DESCRIPTION )
if [[ -z "$SITE_DESCRIPTION" ]]; then
echof info "User did not attempt to configure $SITE_DESCRIPTION"
else
echof info "Setting '$SITE_DESCRIPTION' $SITE_DESCRIPTION in /app/config/siteConfig.json"
sed -i 's/"description": "A decentralized hosting platform built on LBRY"/"Description": "'$SITE_DESCRIPTION'"/' /app/config/siteConfig.json
fi
;;
esac
}
function configure_speech() {
# install configuration changes here.
echof info "Installing configuration files into /app/config/."
mkdir -p /app/config/
cp /usr/local/src/spee.ch/cli/defaults/mysqlConfig.json /app/config/mysqlConfig.json
cp /usr/local/src/spee.ch/cli/defaults/siteConfig.json /app/config/siteConfig.json
echof info "Installing any environment variables that have been set."
for i in "${ENVVARS[@]}"; do
if [[ -z "$i" ]]; then
echof info "$i was not set, moving on."
else
set_conf $i
fi
done
}
function final_permset() {
# ## Finally reassert permissions in case there is user added drift.
# rddo /app "test_for_dir" '775 "speech:speech"'
# rfdo /app "test_for_file" '665 "speech:speech"'
# ## Define any permission exceptions here.
# # test_for_dir /app/config 775 "speech:speech"
# # test_for_file /app/config/siteConfig.json 665 "speech:speech"
echof info "Copied Spee.ch and set permissions"
}
###################################
## Actual installation function. ##
###################################
# if Spee.ch is not yet installed, copy it into web root.
# This could be updated to be part of an upgrade mechanism.
if [ "$(ls -A /app)" ]; then
echof warn "/app is not Empty. It contains:" 1>&2
ls -A 1>&2
## If siteConfig.json isn't installed add it and configure or ignore and proceed.
if [ ! -e '/app/config/siteConfig.json' ]; then
echof warn "Spee.ch doesn't appear to have a configuration."
echof blank "Installing config"
configure_speech
else
## If the file exists skip configuration and proceed.
echof info "Spee.ch config already installed skipping configuration step."
final_permset
fi
## Install all other files after installing siteConfig.json
echof info "Making an attempt to nicely merge files using:"
echof run "mv -fnu /usr/local/src/spee.ch/* /app/"
mv -fnu /usr/local/src/spee.ch/* /app/
final_permset
else
echof info "Speech not installed, installing fresh copy now."
configure_speech
echof run "cp -r /usr/local/src/spee.ch/* /app/"
cp -r /usr/local/src/spee.ch/* /app/
final_permset
fi
## Superfluous permissions assertion maybe axe this later.
# echof run 'test_for_dir /app/config/siteConfig.json 775 "speech:speech"'
# test_for_file /app/config/siteConfig.json 775 "speech:speech"
+2
View File
@@ -0,0 +1,2 @@
#!/bin/sh
curl --fail http://localhost:3000/ || exit 1
+2
View File
@@ -0,0 +1,2 @@
COMPOSE_PROJECT_NAME=traefik
DOMAIN=example.com
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
echo "========================================================================="
echo "========================================================================="
echo "lbry-docker: does not maintain or support custom configuration of traefik"
echo " lbry-docker comes with a YMMV boilerplate reverse proxy for"
echo " your convenience only and changing this configuration voids"
echo " your warranty."
echo "========================================================================="
+48
View File
@@ -0,0 +1,48 @@
version: '3.4'
networks:
lbrynet:
external: true
services:
#####################
## Proxy container ##
#####################
## One of the major contributing guides here.
## https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-16-04
proxy:
image: traefik:alpine
command: --api --docker --docker.domain=docker.localhost --logLevel=DEBUG
networks:
lbrynet:
ipv4_address: 10.5.1.100
ports:
- target: 80
published: 80
mode: host
- target: 443
published: 443
mode: host
expose:
- 8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik.toml:/traefik.toml
## This is going to be stored by default in the following host dir in staging.
# There is a rate limit for how many issuances you can request per week.
# Because of this we will store the acme.json file persistently.
# In production we are going to need to figure out how it'll work with Kubernetes or variant.
- ./traefik/:/opt/traefik/
labels:
# https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
- "traefik.backend=proxy"
- "traefik.docker.network=lbrynet"
## Make sure that you either set the .env variable included in this repository or replace the domain string below.
## Also your DNS record will need to exist pointing at the machine your Traefik Reverse Proxy will be hosted on before you launch this app cluster.
- "traefik.frontend.rule=Host:monitor.${DOMAIN}"
- "traefik.expose=true"
- "traefik.port=8080"
healthcheck:
test: ["CMD", "traefik", "healthcheck"]
timeout: 10s
retries: 3
+2
View File
@@ -0,0 +1,2 @@
#!/usr/bin/env bash
docker network create -d bridge --subnet=10.5.1.0/16 lbrynet
+40
View File
@@ -0,0 +1,40 @@
debug = true
logLevel = "ERROR"
defaultEntryPoints = ["https","http"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[retry]
[docker]
endpoint = "unix:///var/run/docker.sock"
## Edit the domain string with your domain that you want to provision web apps to.
domain = "${DOMAIN}"
watch = true
exposedbydefault = false
[acme]
## Edit the email string with where you want to receive TLS Cert notifications.
email = "admin@${DOMAIN}"
storage = "/opt/traefik/acme.json"
entryPoint = "https"
OnHostRule = true
[acme.httpChallenge]
entryPoint = "http"
# Ping definition for enabling healthcheck
[ping]
# Name of the related entry point
#
# Optional
# Default: "traefik"
#
entryPoint = "traefik"