scatterhq / flocker Goto Github PK

View Code? Open in Web Editor NEW

This project forked from clusterhq/flocker

69.0 16.0 8.0 58.02 MB

Container data volume manager for your Dockerized application

License: Apache License 2.0

Python 98.70% Shell 0.49% Groovy 0.80%

flocker's Introduction

Flocker

Flocker is an open-source Container Data Volume Manager for your Dockerized applications.

By providing tools for data migrations, Flocker gives ops teams the tools they need to run containerized stateful services like databases in production.

Unlike a Docker data volume which is tied to a single server, a Flocker data volume, called a dataset, is portable and can be used with any container, no matter where that container is running.

Flocker manages Docker containers and data volumes together. When you use Flocker to manage your stateful microservice, your volumes will follow your containers when they move between different hosts in your cluster.

You can also use Flocker to manage only your volumes, while continuing to manage your containers however you choose.

About Us

Flocker is being maintained by ScatterHQ. We are a small team of engineers with experience running distributed systems and many of us are core contributors to the Twisted project.

This project is under active development; version 1.0 was released on June 17th, 2015. Contributions are welcome. If you have any issues or feedback, you can talk to us. We're looking forward to working on this project with you.

Documentation

You can read more about installing Flocker, follow a tutorial and learn about the features of Flocker and its architecture in the Flocker docs.

Feature Requests

If you have any feature requests or suggestions, we would love to hear about them. Please send us your ideas by filing a GitHub issue.

Tests

Flocker's test suite is based on unittest and Twisted Trial. The preferred way to run the test suite is using the command trial flocker. Flocker also includes a tox configuration to run the test suite in multiple environments and to run additional checks (such as flake8) and build the documentation with Sphinx. You can run all of the tox environments using the command tox.

Flocker is also tested using continuous integration.

flocker's People

Contributors

Stargazers

Watchers

Forkers

domen88 shanglai markrwilliams endlessm mashroomxl callmegar yusuf963 heyitsquoracom

flocker's Issues

HTTP Error 404: Not Found for URL https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl

I am facing below issue while building flocker:

pip install https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl

Downloading/unpacking https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl
  HTTP error 404 while getting https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl
  Could not install requirement https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl because of error HTTP Error 404: Not Found
Cleaning up...
Could not install requirement https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl because of HTTP error HTTP Error 404: Not Found for URL https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl
Storing complete log in /home/nayana/.pip/pip.log
(flocker-client)[nayana@ecos0011 flocker_rhel_setup]$ pip install https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl
Downloading/unpacking https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl
  HTTP error 404 while getting https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl
  Could not install requirement https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl because of error HTTP Error 404: Not Found
Cleaning up...
Could not install requirement https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl because of HTTP error HTTP Error 404: Not Found for URL https://clusterhq-archive.s3.amazonaws.com/python/Flocker-1.15.0-py2-none-any.whl

Reference: https://flocker.readthedocs.io/en/latest/docker-integration/install-client.html
Distribution: RHEL

Web pages can not be accessed

hello，Web pages can not be accessed
[root@docker79 flocker]# yum list installed clusterhq-release || yum install -y https://clusterhq-archive.s3.amazonaws.com/centos/clusterhq-release$(rpm -E %dist).centos.noarch.rpm
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Error: No matching Packages to list
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Cannot open: https://clusterhq-archive.s3.amazonaws.com/centos/clusterhq-release.el7.centos.noarch.rpm. Skipping.
Error: Nothing to do
[root@docker79 flocker]# rpm -E %dist
.el7
[root@docker79 flocker]# yum list installed clusterhq-release || yum install -y https://clusterhq-archive.s3.amazonaws.com/centos/clusterhq-release.el7.centos.noarch.rpm
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Error: No matching Packages to list
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Cannot open: https://clusterhq-archive.s3.amazonaws.com/centos/clusterhq-release.el7.centos.noarch.rpm. Skipping.
Error: Nothing to do

Docker docs should be updated to point to this repo

The Docker documentation still has quite a few links to the old flocker repo but the repo itself doesn't have anything stating it's not being maintained anymore or that they should be switching to this repo instead.

For example https://docs.docker.com/engine/tutorials/dockervolumes/

Flocker: Control service is not getting up

Hi,
I am trying to setup flocker on Intel x86 RHEL distribution .
I've followed the installation steps mentioned here. However flocker control service doesn't come up.


 systemctl enable flocker-control
 systemctl start flocker-control
 systemctl status  flocker-control

● flocker-control.service - Flocker Control Service
   Loaded: loaded (/usr/lib/systemd/system/flocker-control.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) 
  Process: 7610 ExecStart=/usr/sbin/flocker-control --port tcp:4523 --agent-port tcp:4524 --journald (code=exited, status=1/FAILURE)
 Main PID: 7610 (code=exited, status=1/FAILURE)

systemd[1]: flocker-control.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit flocker-control.service entered failed state.
systemd[1]: flocker-control.service failed.
systemd[1]: flocker-control.service holdoff time over, scheduling restart.
systemd[1]: start request repeated too quickly for flocker-control.service
Failed to start Flocker Control Service.
Unit flocker-control.service entered failed state.
flocker-control.service failed

Also, generated control-certificate and plugin certificates mentioned in installation guide.

Posted this one on ClusterHQ/flocker also.

Error: flocker: plugin not found

Hello,

I am trying to setup flocker on Intel x86 RHEL distribution .
I've followed the installation steps mentioned here however when I execute the following command to test if flocker-docker-plugin works:

docker run -v apples:/data --volume-driver flocker busybox sh -c "echo hello > /data/file.txt"

I get the error:

Error response from daemon: create apples: create apples: Error looking up volume plugin flocker: plugin not found.

The flocker-docker-plugin status shows the following:

systemctl status flocker-docker-plugin
● flocker-docker-plugin.service - Flocker Docker Plugin
   Loaded: loaded (/usr/lib/systemd/system/flocker-docker-plugin.service; enabled; vendor preset: disabled)
   Active: active (running)

I have used loop back dataset as mentioned in https://docs.clusterhq.com/en/latest/flocker-features/loopback-configuration.html#loopback-dataset-backend.

Posted the issue on ClusterHQ/flocker also.

Broken links to offline ClusterHQ infrastructure

https://travis-ci.org/ScatterHQ/flocker/jobs/189902204

build/docs-linkcheck/output.txt[24] gettinginvolved/contributing.rst:27: [broken] http://build.clusterhq.com/: HTTPConnectionPool(host='build.clusterhq.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fab7979d050>: Failed to establish a new connection: [Errno 110] Connection timed out',))
25: build/docs-linkcheck/output.txt[25] gettinginvolved/contributing.rst:100: [broken] http://build.clusterhq.com/: HTTPConnectionPool(host='build.clusterhq.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fab78b28690>: Failed to establish a new connection: [Errno 110] Connection timed out',))
27: build/docs-linkcheck/output.txt[27] gettinginvolved/infrastructure/release-process.rst:9: [broken] http://clusterhq-staging-docs.s3.amazonaws.com/master/gettinginvolved/infrastructure/release-process.html: 403 Client Error: Forbidden for url: http://clusterhq-staging-docs.s3.amazonaws.com/master/gettinginvolved/infrastructure/release-process.html
30: build/docs-linkcheck/output.txt[30] gettinginvolved/infrastructure/release-process.rst:224: [broken] http://build.clusterhq.com/boxes-flocker: HTTPConnectionPool(host='build.clusterhq.com', port=80): Max retries exceeded with url: /boxes-flocker (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fab79010ed0>: Failed to establish a new connection: [Errno 110] Connection timed out',))

flocker with openstack cinder failed

Hi
My docker is working on swarm mode. But when I using flocker volume plugin, run 'docker run -v vol1:/usr/local/tomcat/logs tomcat' command, it print error: 'docker: Error response from daemon: Timed out waiting for dataset to mount..'.
My flocker-dataset-agent.service is running with error:
Mar 27 03:33:51 flkn1 flocker-dataset-agent[541]: {"exception": "flocker.node.agents.blockdevice.UnknownInstanceID", "task_level": [2, 2, 5], "action_type": "flocker:agent:discovery", "reason": "Could not find valid instance ID for <flocker.node.agents.cinder.CinderBlockDeviceAPI object at 0x49d9c90>", "timestamp": 1490585631.412446, "task_uuid": "75a7ba2e-496f-43e1-a407-c31c8c6d249a", "action_status": "failed"} Mar 27 03:33:51 flkn1 flocker-dataset-agent[541]: {"exception": "flocker.node.agents.blockdevice.UnknownInstanceID", "reason": "Could not find valid instance ID for <flocker.node.agents.cinder.CinderBlockDeviceAPI object at 0x49d9c90>", "timestamp": 1490585631.412774, "traceback": "Traceback: <class 'flocker.node.agents.blockdevice.UnknownInstanceID'>: Could not find valid instance ID for <flocker.node.agents.cinder.CinderBlockDeviceAPI object at 0x49d9c90>\n/opt/flocker/lib/python2.7/site-packages/machinist/_logging.py:102:receive\n/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:534:receive\n/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:592:output\n/opt/flocker/lib/python2.7/site-packages/flocker/node/_loop.py:529:output_CONVERGE\n--- <exception caught here> ---\n/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py:150:maybeDeferred\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1798:discover_state\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1345:_count_calls\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1743:_discover_raw_state\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:2026:compute_instance_id\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/cinder.py:491:compute_instance_id\n", "message_type": "eliot:traceback", "task_uuid": "75a7ba2e-496f-43e1-a407-c31c8c6d249a", "task_level": [2, 3]}

New url for download flocker.

New url for download flocker. ?

Flocker tests are failing

Hi,

Flocker tests are failing on RHEL7 and Ubuntu 16.04 on Intel x86.
Flocker version 1.15.0

I have executed flocker test suite using "tox" command however getting below failures:

Failure: testtools.testresult.real._StringException: Empty attachments

Ran 2770 tests in 1030.296s
FAILED (skips=280, failures=5, successes=2485)
ERROR:   py27: commands failed

Tests failing with above error are:

flocker.control.test.test_persistence.GenerationHashTests.test_consistent_hash
flocker.control.test.test_persistence.LatestGoldenFilesValid.test_can_create_latest_golden
flocker.restapi.testtools.ConfigurationContainersSchemaTests.test_fails_validation_maxItems_0
flocker.restapi.testtools.ConfigurationDatasetsCreateSchemaTests.test_fails_validation_pattern_1
flocker.restapi.testtools.ConfigurationDatasetsSchemaTests.test_fails_validation_pattern_1

Traceback for test
flocker.control.test.test_persistence.GenerationHashTests.test_consistent_hash:

[FAIL]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: Empty attachments:
  twisted-eliot-log
  twisted-log

Traceback (most recent call last):
  File "/home/flocker/flocker/control/test/test_persistence.py", line 970, in test_consistent_hash
    Equals(TEST_DEPLOYMENT_1_HASH)
  File "/home/flocker/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = '\x87\x13\xcbG`\xd7\xab\x0f0\xd5\xd2x\xe8\x12]\x11'
actual    = '\xcf\x9e\xa0\xe7\x8f\x7f\xa3?\x8c\rH\xfb\x06\xc8\xeay'

I have followed below steps to build flocker:

git clone https://github.com/ClusterHQ/flocker
cd flocker
git checkout 1.15.0
pip install -e .
tox

Also, same result with below steps:

yum install python-devel openssl-devel git libffi-devel python-pip
yum groupinstall "Development tools"
pip install virtualenvwrapper
source /usr/bin/virtualenvwrapper.sh
mkvirtualenv flocker
git clone https://github.com/ClusterHQ/flocker
cd flocker/
pip install --requirement dev-requirements.txt
trial flocker

Tried with git clone https://github.com/ScatterHQ/flocker as well.

Any idea on these failures? are these failures expected?

Note: I had raised this issue here also.

Error looking up volume plugin flocker: legacy plugin: plugin not found

Hi,
I tried to use Ansible role https://github.com/ClusterHQ/ansible-role-flocker, servers seems to be correctly setup, but when I try to create volume I'm getting this:

$ docker run -v apples:/data --volume-driver flocker busybox sh -c "echo hello > /data/file.txt"
docker: Error response from daemon: create apples: create apples: Error looking up volume plugin flocker: legacy plugin: plugin not found.

Running pocesses on nodes and controller

ps -ef | grep flocker
/opt/flocker/bin/python /usr/sbin/flocker-dataset-agent --journald
/opt/flocker/bin/python /usr/sbin/flocker-container-agent --journald
# + on controll node:
/opt/flocker/bin/python /usr/sbin/flocker-control --port tcp:4523 --agent-port tcp:4524 --journald

OS: CentOS Linux release 7.2.1511 (Core)
RPM:
clusterhq-python-flocker-1.15.0-1.x86_64
clusterhq-flocker-docker-plugin-1.15.0-1.noarch
clusterhq-flocker-node-1.15.0-1.noarch
Backend: OpenStack Cinder

Running it as a container failing on missing/deprecated images, but https://flocker-docs.clusterhq.com/en/latest/supported/flockercontainers.html#flocker-containers shows also clusterhq/flocker-docker-plugin is this some legacy stuff ?

Any help appreciated. Thanks

volume state : always pending

I Use a P.O.C for planning this in production purpose.

All is working good but since few days, when i try to create a new volume, the state is always pending.

Strange thing, after some times (15-30min) the volume is intermittently created a plugged to the instance. But the state with flockerctl --certs-path=/etc/flocker --user=client --control-service=192.168.12.10 list indicate that the volume is still pending and the volume is not usable.

ubuntu@manager01:~$ sudo flockerctl --certs-path=/etc/flocker --user=client --control-service=192.168.12.10 list
DATASET SIZE METADATA STATUS SERVER
5ab72fdd-73a2-435b-ab1e-7ed826c017b7 10.00G maximum_size=10737418240,name=grafana pending ⌛ f390bb6a (192.168.12.10)
1249b1f4-6c00-4eac-9d82-b615d854a82b 10.00G maximum_size=10737418240,name=yeahBro pending ⌛ f390bb6a (192.168.12.10)
e2d0f9f6-9288-4404-951b-3711c79eb209 10.00G maximum_size=10737418240,name=test pending ⌛ 86a18183 (192.168.12.12)
59ce4f40-cde3-420e-8594-f6f128d21b0e 10.00G maximum_size=10737418240,name=bdd pending ⌛ f390bb6a (192.168.12.10)
24f869eb-99f0-440c-8187-c2bfb992b91e 10.00G maximum_size=10737418240,name=noGood pending ⌛ f390bb6a (192.168.12.10)

My cloud provider is OVH.

All node is up a running, all agent is in active state whithout errors.

see the flocker-diagnotics attach file
clusterhq_flocker_logs_fdbd42c4-e23b-11e6-b564-fa163ebcb6e9.zip

thx

docker: Error response from daemon: Timed out waiting for dataset to mount..

I had issue :
docker: Error response from daemon: Timed out waiting for dataset to mount..

while tracing logs from all 3 services (dataset,container & plugin) i found this kind of error on flocker-container-agent log

{"timestamp": 1486108668.512604, "task_uuid": "e7ee36f7-b5fb-4843-afc4-662058f9be85", "message_type": "flocker:failure-retry:retrying", "task_level": [2]} {"exception": "requests.exceptions.ConnectionError", "timestamp": 1486108668.516428, "task_uuid": "e7ee36f7-b5fb-4843-afc4-662058f9be85", "action_status": "failed", "reason": "('Connection aborted.', error(2, 'No such file or directory'))", "action_type": "flocker:failure-retry", "errno": null, "task_level": [3]} {"exception": "requests.exceptions.ConnectionError", "timestamp": 1486108668.517358, "task_uuid": "80407d07-c631-4e52-acae-f154f13e520f", "action_status": "failed", "reason": "('Connection aborted.', error(2, 'No such file or directory'))", "action_type": "flocker:agent:discovery", "errno": null, "task_level": [2, 2, 2]} {"exception": "requests.exceptions.ConnectionError", "errno": null, "traceback": "Traceback: <class 'requests.exceptions.ConnectionError'>: ('Connection aborted.', error(2, 'No such file or directory'))\n/usr/lib64/python2.7/threading.py:811:__bootstrap_inner\n/usr/lib64/python2.7/threading.py:764:run\n/opt/flocker/lib/python2.7/site-packages/twisted/_threads/_threadworker.py:46:work\n/opt/flocker/lib/python2.7/site-packages/twisted/_threads/_team.py:190:doWork\n--- <exception caught here> ---\n/opt/flocker/lib/python2.7/site-packages/twisted/python/threadpool.py:246:inContext\n/opt/flocker/lib/python2.7/site-packages/twisted/python/threadpool.py:262:<lambda>\n/opt/flocker/lib/python2.7/site-packages/twisted/python/context.py:118:callWithContext\n/opt/flocker/lib/python2.7/site-packages/twisted/python/context.py:81:callWithContext\n/opt/flocker/lib/python2.7/site-packages/flocker/node/_docker.py:909:_list\n/opt/flocker/lib/python2.7/site-packages/flocker/common/_retry.py:503:method_with_retry\n/opt/flocker/lib/python2.7/site-packages/flocker/common/_retry.py:462:_poll_until_success_returning_result\n/opt/flocker/lib/python2.7/site-packages/flocker/common/_retry.py:293:poll_until\n/opt/flocker/lib/python2.7/site-packages/flocker/common/_retry.py:446:pollable\n/opt/flocker/lib/python2.7/site-packages/flocker/common/_retry.py:443:pollable\n/opt/flocker/lib/python2.7/site-packages/docker/api/container.py:70:containers\n/opt/flocker/lib/python2.7/site-packages/docker/utils/decorators.py:47:inner\n/opt/flocker/lib/python2.7/site-packages/docker/client.py:120:_get\n/opt/flocker/lib/python2.7/site-packages/requests/sessions.py:487:get\n/opt/flocker/lib/python2.7/site-packages/requests/sessions.py:475:request\n/opt/flocker/lib/python2.7/site-packages/requests/sessions.py:585:send\n/opt/flocker/lib/python2.7/site-packages/requests/adapters.py:453:send\n", "task_uuid": "80407d07-c631-4e52-acae-f154f13e520f", "reason": "('Connection aborted.', error(2, 'No such file or directory'))", "timestamp": 1486108668.518431, "message_type": "eliot:traceback", "task_level": [2, 3]}

Im using flocker ceph rbd plug in, these are the current state :

[root@vm2 ~]# docker volume create -d flocker --name=devops-vol-db -o size=2G -o profile=bronze devops-vol-db

[root@vm2 ~]# docker volume ls DRIVER VOLUME NAME local 9ff52939de1e5a7a5e3d27d5d391fc3cfce06eccd35b9a2e26a467ce550c943a flocker devops-vol-db local f4a1067493964c67b0880f242d6de555c65e7c82422a50393e8860a9a02c4256
[root@vm2 ~]# docker run -it --rm -v devops-vol-db:/opt --entrypoint /bin/sh busybox docker: Error response from daemon: Timed out waiting for dataset to mount..

[root@vm1 ~]# flockerctl --control-service=192.168.10.101 --certs-path=/etc/flocker --user=admin list DATASET SIZE METADATA STATUS SERVER 953997e5-70a3-4e2d-91c1-f1a013c65e0c 2.00G maximum_size=2147483648,name=devops-vol- pending âissing> db,clusterhq:flocker:profile=bronze

There are no issue while connecting ceph rbd with flocker, because i am able to create rbd image from flocker, but failed while to try mounting it from docker.

Docker version 1.12
OS Centos 7 Latest Kernel
flocker flocker-volumes.py prototype version 0.0.1
ceph version 10.2.3 (using loopback device)

flockerctl installation

when i try to download the installation :

manager01:~$ sudo curl -sSL https://get.flocker.io | sh
sh: 3: Syntax error: newline unexpected

manager01:~$ sudo curl -sSL https://get.flocker.io

<title>503 first byte timeout</title>

Error 503 first byte timeout

first byte timeout

Guru Mediation:

Details: cache-ams4128-AMS 1484325222 2705976644

Varnish cache server