Giter VIP home page Giter VIP logo

docker-riak-cs's Introduction

Docker Riak CS

Riak CS is an object storage software compatible with AWS S3 API. It's a perfect S3 alternative for local development and testing, which is the exact purpose of this image. It works as a single node to keep the resources to the minimum, something that Riak guys wouldn't recommend and certainly not suitable for production. There is hectcastro/docker-riak-cs project that allows to bring up a multi-node cluster, which might suite you better.

Circle CI

Running

Pull or build the image yourself and run it. When the container gets started it will setup the Riak admin and show you the credentials. Will also create optionally provided buckets.

# Pull…
docker pull 'ianbytchek/riak-cs'

# Or build…
docker build --tag 'ianbytchek/riak-cs' './src'
 
# Run and create three buckets.
docker run --detach \
    --env 'RIAK_CS_BUCKETS=foo,bar,baz' \
    --publish '8080:8080' \
    --name 'riak-cs' \
    ianbytchek/riak-cs

# Configure s3cmd, `RIAK_CS_KEY_ACCESS` and `RIAK_CS_KEY_SECRET` are printed to
# stdout, run `docker logs -f 'riak-cs'` to see what they are. Also make sure to
# use the correct `proxy_host` if you are on Mac / running docker inside a vm.

cat <<EOF >~/.s3cfg.riak_cs
[default]
access_key = <RIAK_CS_KEY_ACCESS>
host_base = s3.amazonaws.dev
host_bucket = %(bucket)s.s3.amazonaws.dev
proxy_host = 127.0.0.1
proxy_port = 8080
secret_key = <RIAK_CS_KEY_SECRET>
signature_v2 = True
EOF

# Retry a few times with 10 second interval if getting `ERROR: [Errno 104]
# Connection reset by peer`.

s3cmd -c ~/.s3cfg.riak_cs ls

Configuration

Use the following environment variables to configure Riak CS instance:

  • RIAK_CS_BUCKETS – colon separated list or buckets to automatically create.

  • RIAK_CS_KEY_ACCESS, RIAK_CS_KEY_SECRET – config's equivalent of admin_key and admin_secret respectively.

    Only use this with a mounted data volume when your data is already created and there is already a user with this credentials. Otherwise the default admin user will not be created and you'll get an error.

Proxy

Riak CS should be run behind a proxy, its recommended by Basho and gives certain advantages, such as granular DNS configuration and url rewriting. Besides, Riak doesn't play well with SHA-256 and SSL overall, which will eventually be fixed, but until then you are better off with SSL termination. Below is a HAProxy config that can be used along with ianbytchek/docker-haproxy.

# Make sure to replace <PRIVATE_KEY_PATH> with the actual path to relevant ssl key
# and <RIAK_CS_IP_PORT> with the IP and port of the container.

defaults
    mode http
    timeout connect 10s
    timeout client 1m
    timeout server 1m

frontend http
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }

frontend https
    bind *:443 ssl crt <PRIVATE_KEY_PATH>
    use_backend riak_cs

backend riak_cs
    balance leastconn
    option httpclose
    reqirep ^Host:\ (.+)?(s3).*(\.amazonaws\.dev)$ Host:\ \1\2\3
    server node01 <RIAK_CS_IP_PORT>

Scripts

All image and container business is done in individual scripts instead of using docker file for all of that. During the build we run build.sh, it will run scripts that install dependencies and patch configuration, while entrypoint.sh only configures the application when the container starts.

Issues

AWS access key id does not exist

There is a known issue when the Docker host gets powered off in a non-graceful way. This might be due to the fact that Riak shouldn't be run as a one-node cluster or it doesn't get a chance to properly shut down. Starting a fresh container normally helps, but there were many times when only rebuilding the container and restarting the host would work. This doesn't happen now as often as it used to, though.

AWS regions

If you explicitly specify the AWS region Riak CS may freak out as described in this issue. It doesn't happen to all regions and probably can be fixed via DNS configuration.

Request time difference is too large

Sometimes everything stops working with The difference between the request time and the current time is too large. message. Apparently this is boot2docker specific issue, can be cured with this:

!/usr/bin/env bash

while true; do
    MACHINE='default'
    NTP_SERVER='pool.ntp.org'

    docker-machine ssh "${MACHINE}" "sudo ntpclient -s -h ${NTP_SERVER}"
    sleep 60
done

docker-riak-cs's People

Contributors

eterps avatar iby avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

docker-riak-cs's Issues

output credentials in a folder/file that can be mounted by other containers

We would like to use your container to provide our developers environment (based on a set of docker images) with a local s3, we're using a Vagrantfile to combine the different containers and to do the linking/mounting/provisionning

The problem we're having with your containers is that we can not create a user with read/write access on a bucket (as you state in the readme, if we set the RIAK_CS_KEY_ACCESS and RIAK_CS_KEY_SECRET it fails )

it would be nice to either

  • output the admin credentials that are currently output in stdout, in a file, e.g /var/whatever/credentials , so that we can mount /var/whatever from other containers
    • have environment variables that if set , create a non-admin user that have the read/write rights on the created buckets

S3cmd ls gives Problem: BadStatusLine: ''

Invoked as: /usr/local/bin/s3cmd ls s3://foo
Problem: BadStatusLine: ''
S3cmd: 1.6.1
python: 2.7.11 (default, Jan 22 2016, 08:29:18)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
environment LANG=None

Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2996, in
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2915, in main
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 137, in cmd_ls
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 170, in subcmd_bucket_list
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 327, in bucket_list
for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 358, in bucket_list_streaming
response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params, max_keys)
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 389, in bucket_list_noparse
response = self.send_request(request)
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 1235, in send_request
conn = ConnMan.get(self.get_hostname(resource['bucket']))
File "build/bdist.macosx-10.11-x86_64/egg/S3/ConnMan.py", line 249, in get
conn.c.connect()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1270, in connect
HTTPConnection.connect(self)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 839, in connect
self._tunnel()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 811, in _tunnel
(version, code, message) = response._read_status()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 417, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''

License

Any chance you would be willing to add a license to the code in this repository?

How do you use it?

I did the following:

docker pull 'ianbytchek/riak-cs'
docker run -e 'RIAK_CS_BUCKETS=foo,bar,baz' --name 'riak-cs' ianbytchek/riak-cs

The output I am getting:

Starting Riak… OK!
Starting Stanchion… OK!
Starting Riak CS… OK!

############################################################

    Riak admin credentials, make note of them, otherwise you
    will not be able to access your files and data. Riak
    services will be restarted to take effect.

       Key: 
    Secret: 

############################################################

Restarting Riak… OK!
Restarting Stanchion… OK!
Restarting Riak CS… OK!
Creating Riak CS buckets.
… Failed!
… Failed!
… Failed!
2015-09-20 09:18:10,970 CRIT Supervisor running as root (no user in config file)
2015-09-20 09:18:10,970 WARN Included extra file "/docker/configuration/supervisord/riak.conf" during parsing
2015-09-20 09:18:10,970 WARN Included extra file "/docker/configuration/supervisord/riak-cs.conf" during parsing
2015-09-20 09:18:10,970 WARN Included extra file "/docker/configuration/supervisord/stanchion.conf" during parsing
2015-09-20 09:18:10,973 INFO supervisord started with pid 3511
2015-09-20 09:18:11,975 INFO spawned: 'riak' with pid 3514
2015-09-20 09:18:11,976 INFO spawned: 'stanchion' with pid 3515
2015-09-20 09:18:11,978 INFO spawned: 'riak-cs' with pid 3516
2015-09-20 09:18:12,981 INFO success: riak entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-20 09:18:12,981 INFO success: stanchion entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-20 09:18:12,981 INFO success: riak-cs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

It doesn't output credentials, I am unable to connect with it.

Buckets Cannot Be Created - Using docker-compose.yml

==========docker-compose.yml=============
riak-cs:
image: ianbytchek/riak-cs
container_name: riak-cs-amazon-s3
healthcheck:
timeout: 60s
retries: 10
environment:
- RIAK_CS_BUCKETS=riak
- RIAK_CS_KEY_ACCESS=CC2QOJQ4VJFR8USBS5Y3
- RIAK_CS_KEY_SECRET=dKoX0b8dKbSFS0lJ_euHAVdgHMPDuB9GDZgJ8Q==
ports:
- "8080:8080"

========docker logs===========
riak-cs-amazon-s3 | Waiting for riak kv service to startup… OK!
riak-cs-amazon-s3 | Starting Stanchion… OK!
riak-cs-amazon-s3 | Starting Riak CS… OK!
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | ############################################################
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | Riak admin credentials, make note of them, otherwise you
riak-cs-amazon-s3 | will not be able to access your files and data. Riak
riak-cs-amazon-s3 | services will be restarted to take effect.
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | Access key: CC2QOJQ4VJFR8USBS5Y3
riak-cs-amazon-s3 | Secret key: dKoX0b8dKbSFS0lJ_euHAVdgHMPDuB9GDZgJ8Q==
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | ############################################################
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | Restarting Riak… OK!
riak-cs-amazon-s3 | Waiting for riak kv service to startup… OK!
riak-cs-amazon-s3 | Restarting Stanchion…Restarting Riak CS…Node '[email protected]' not responding to pings.
riak-cs-amazon-s3 | Node '[email protected]' not responding to pings.
riak-cs-amazon-s3 | OK!
riak-cs-amazon-s3 | Creating Riak CS buckets.
riak-cs-amazon-s3 | riak… Failed!
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 CRIT Supervisor running as root (no user in config file)
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 WARN Included extra file "/entrypoint/configuration/supervisord/stanchion.conf" during parsing
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 WARN Included extra file "/entrypoint/configuration/supervisord/riak-cs.conf" during parsing
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 WARN Included extra file "/entrypoint/configuration/supervisord/riak.conf" during parsing
riak-cs-amazon-s3 | 2018-02-19 09:35:48,374 INFO supervisord started with pid 4906
riak-cs-amazon-s3 | 2018-02-19 09:35:49,376 INFO spawned: 'riak' with pid 4909
riak-cs-amazon-s3 | 2018-02-19 09:35:49,378 INFO spawned: 'stanchion' with pid 4910
riak-cs-amazon-s3 | 2018-02-19 09:35:49,380 INFO spawned: 'riak-cs' with pid 4911
riak-cs-amazon-s3 | 2018-02-19 09:35:49,839 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:49,859 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:51,001 INFO success: riak entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
riak-cs-amazon-s3 | 2018-02-19 09:35:51,007 INFO spawned: 'stanchion' with pid 5248
riak-cs-amazon-s3 | 2018-02-19 09:35:51,013 INFO spawned: 'riak-cs' with pid 5249
riak-cs-amazon-s3 | 2018-02-19 09:35:51,440 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:51,492 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:53,494 INFO spawned: 'stanchion' with pid 5454
riak-cs-amazon-s3 | 2018-02-19 09:35:53,496 INFO spawned: 'riak-cs' with pid 5455
riak-cs-amazon-s3 | 2018-02-19 09:35:53,860 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:53,874 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:56,878 INFO spawned: 'stanchion' with pid 5660
riak-cs-amazon-s3 | 2018-02-19 09:35:56,879 INFO spawned: 'riak-cs' with pid 5661
riak-cs-amazon-s3 | 2018-02-19 09:35:57,254 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:57,256 INFO gave up: stanchion entered FATAL state, too many start retries too quickly
riak-cs-amazon-s3 | 2018-02-19 09:35:57,256 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:58,257 INFO gave up: riak-cs entered FATAL state, too many start retries too quickly

Unable to connect

I'm following the directions to set up/configure and am not able to connect.

I've downloaded a fresh Docker image from DockerHub. Run it. Seems healthy. If I climb inside with docker exec, I can run 'riak ping', 'riak-cs ping' and 'stanchion ping' and all return 'pong'.

I downloaded a fresh install of s3cmd (2.0.2) and configured with this .s3cfg file: (the 2 security keys came from the log output of the running container)

[default]
access_key = OPSBRJYKYIDJQMCDJU6Y
host_base = s3.amazonaws.dev
host_bucket = %(bucket)s.s3.amazonaws.dev
proxy_host = localhost
proxy_port = 8080
secret_key = eUARz8q1KDfap_Dp2t3solbr_8soo5Ubw8UaJA==
signature_v2 = True

The sample docker launch command given in the instructions here create an initial bucket, testbucket, so I thought I should be able to see it.

Running "s3cmd ls" I get this response:

WARNING: Retrying failed request: / (No status line received - the server has closed the connection)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: / (No status line received - the server has closed the connection)
WARNING: Waiting 6 sec...
^CSee ya!

Am I doing something wrong, or are the instructions not complete?
Thanks,
Greg

Takes very long to start

I'm looking to simulate S3 locally. On debian stretch amd64 (a Thinkpad X1 Gen 6 Carbon), it takes about 98s from typing

docker run -p 127.0.0.1:1234:8080  ianbytchek/riak-cs

till the final output

2018-09-10 11:37:15,595 INFO success: riak-cs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Until that final output, riak is not usable, and the CPU load is 100%.

In contrast, competing projects take about 0.1s to start.

If there is any compilation needed, it should be done as part of the build process, not during spin-up of the container.

Container is failing to start

Hi,

I am unable to get the image to work. It fails to create the initial buckets and rejects any commands. Here is the logs, maybe something that can be fixed:

016-03-25 01:48:45.157 [error] <0.285.0> Supervisor {<0.285.0>,poolboy_sup} had child riak_cs_riakc_pool_worker started with riak_cs_riakc_pool_worker:start_link([{address,"127.0.0.1"},{port,8087}]) at undefined exit with reason shutdown in context shutdown_error
2016-03-25 01:48:45.160 [error] <0.278.0> Supervisor {<0.278.0>,poolboy_sup} had child riak_cs_riak_client started with riak_cs_riak_client:start_link([]) at undefined exit with reason shutdown in context shutdown_error
2016-03-25 01:48:45.161 [error] <0.148.0> Supervisor {<0.148.0>,poolboy_sup} had child riak_cs_riak_client started with riak_cs_riak_client:start_link([]) at undefined exit with reason shutdown in context shutdown_error
2016-03-25 01:48:49.342 [error] <0.431.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.374 [error] <0.432.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.385 [error] <0.433.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.396 [error] <0.434.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.406 [error] <0.435.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.418 [error] <0.436.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 02:06:08.100 [error] <0.3911.0> CRASH REPORT Process <0.3911.0> with 0 neighbours crashed with reason: no function clause matching riak_cs_s3_rewrite:separate_bucket_from_path("s3.amazonaws.dev:443") line 173

Volumes for s3 data

Firstly, thank you for perfect docker image.

I would like to store bigger data and make them persistent (docker service restart / server restart).
I guess that I can set custom --volume param, but unfortunately I do not know which directory I have to mount.

UPDATE
I can change storage by -v /my/storage:/var/lib/riak but unfortunately (and obviously) it doesn't work - this corrupt/clear the riak DB internal storage. Is it possible to fix it somehow?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.