worstcase / blockade Goto Github PK
View Code? Open in Web Editor NEWDocker-based utility for testing network failures and partitions in distributed applications
Home Page: http://blockade.readthedocs.org
License: Apache License 2.0
Docker-based utility for testing network failures and partitions in distributed applications
Home Page: http://blockade.readthedocs.org
License: Apache License 2.0
Using the examples in read the docs, I get errors while trying to run blockade log and blockade destroy. Does blockade have a specific version of docker and lxc it is targetting?
I'm on an arch linux box with python-3.4.2, lxc-1.0.7 and docker-1.4.1
. I installed blockade via pip. The machine has lots of memory and cores 24 (real cores) and 384G of RAM. Let me know if you need more info on my setup.
[root@ev003 blockade]# blockade up
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 244e48ff6d4f UP 172.17.0.5 NORMAL
c2 2d74ac31fdeb UP 172.17.0.6 NORMAL
c3 3fc016b723af UP 172.17.0.7 NORMAL
[root@ev003 blockade]# blockade status
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 244e48ff6d4f UP 172.17.0.5 NORMAL
c2 2d74ac31fdeb UP 172.17.0.6 NORMAL
c3 3fc016b723af UP 172.17.0.7 NORMAL
[root@ev003 blockade]# blockade destroy
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/docker/client.py", line 92, in _raise_for_status
response.raise_for_status()
File "/usr/lib/python3.4/site-packages/requests/models.py", line 831, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/blockade/cli.py", line 242, in main
opts.func(opts)
File "/usr/lib/python3.4/site-packages/blockade/cli.py", line 118, in cmd_destroy
b.destroy()
File "/usr/lib/python3.4/site-packages/blockade/core.py", line 118, in destroy
self.docker_client.remove_container(container_id)
File "/usr/lib/python3.4/site-packages/docker/client.py", line 886, in remove_container
self._raise_for_status(res)
File "/usr/lib/python3.4/site-packages/docker/client.py", line 94, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Cannot destroy container 244e48ff6d4f19e82adfc68e26dd4fd072cd748b1b30ba986bce7e2f193c84a8: Driver devicemapper failed to remove root filesystem 244e48ff6d4f19e82adfc68e26dd4fd072cd748b1b30ba986bce7e2f193c84a8: Device is Busy'")
Hello!
I read blockade guide, however I was stuck on creating blockade.yml
file. When I save it and run blockade up
in response I got:
NODE CONTAINER ID STATUS IP NETWORK PARTITION
However if I changed file name to blockade.yaml
it works fine.
I have a fix to docs (just one character...) but is it not an internal issue?
I used blockade file from ==> http://blockade.readthedocs.io/en/latest/guide.html
OS: macOS Sierra 10.12.1
Docker version:
Version 1.12.3 (13776)
Channel: Stable
583d1b8ffe
I can manually "run" other docker images, but I can't seem to get the documentation example to work:
{noformat}
tgates@tgW530 ~/myrepo/docker/blockade_test $ sudo blockade up
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/blockade/cli.py", line 242, in main
opts.func(opts)
File "/usr/local/lib/python2.7/dist-packages/blockade/cli.py", line 109, in cmd_up
containers = b.create()
File "/usr/local/lib/python2.7/dist-packages/blockade/core.py", line 47, in create
veth_device)
File "/usr/local/lib/python2.7/dist-packages/blockade/core.py", line 69, in _start_container
binds=container.volumes, port_bindings=container.publish_ports)
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 940, in start
self._raise_for_status(res)
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 94, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error ("Cannot start container 173e5872f98475ff28d0166b6986eb17814c32f610d1502b7b12b0f2718eab39: Cannot link to a non running container: /blockade-a760658c32-c1 AS /blockade-a760658c32-c3/c1")
tgates@tgW530 ~/myrepo/docker/blockade_test $ cat blockade.yml
containers:
c1:
image: ubuntu
command: /bin/sleep 300000
ports: [10000]
c2:
image: ubuntu
command: sh -c "ping $C1_PORT_10000_TCP_ADDR"
links: ["c1"]
c3:
image: ubuntu
command: sh -c "ping $C1_PORT_10000_TCP_ADDR"
links: ["c1"]
tgates@tgW530 ~/myrepo/docker/blockade_test $ docker info
Containers: 12
Images: 118
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 142
Dirperm1 Supported: true
Execution Driver: lxc-1.0.7
Logging Driver: json-file
Kernel Version: 3.16.0-38-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 31.07 GiB
Name: tgW530
ID: TRJG:DCFY:3I2C:EJMT:PGJJ:ZQYW:AVIU:42PN:B4PJ:GCP4:A5R6:QPB2
WARNING: No swap limit support
tgates@tgW530 ~/myrepo/docker/blockade_test $
{noformat}
First off, thanks for Blockade: it's really useful and handy.
With this "issue" I wanted to express the need for a clean and pythonic API that would allow the use of Blockade as library inside other Python scripts.
Right now it's not impossible, but it can get a bit ugly because it's all thought for command line usage.
For instance, in moses I did this to work around the command line option parser (if you think there's a better way, let me know..).
Hi!
The blockade seemed interesting to me to solve my problems. But I have problem.
I use Ubuntu (16.04) in VMware Workstation 12 Pro (12.5.7 build-5813279).
1. blockade -c bl.yaml up
An error has occurred, but the containers have started. (show screen)
2. blockade destroy
Error, but containers are destroyed.
3. blockade -c bl.yaml up <- again
Error: a blockade already exists in here - you may want..
What could be the problem?
Right now blockade supports only IPv4 (at list in blockade partition
). Any chance to also include ip6tables
call into iptables_call?
can blockade simulate some state of the container resource such as CPU,memory,file?
This is just an umbrella issue. I've found a way to make this work, but I don't know how you want me to make PRs - in small chunks or all at once?
Currently the execution happily passes straight into iptables setup and throws meaningless exception from subprocess.check_output - OSError File does not exist
I have the containers up and running with docker-compose.yml, is it possible for blockade to take control of these containers?
By default blockade bases the name of iptables chains on the name of the folder where the blockade.yml
resides. In case of long folder names this leads to the following exception:
iptables v1.4.21: chain name `blockade-countercassandraacyclic-p1' too long (must be under 29 chars)
Try `iptables -h' or 'iptables --help' for more information.
Traceback (most recent call last):
File "../../crdt-counter-partitions.py", line 99, in <module>
if not interact.requests_with_chaos(OPS, HOST, NODES, ARGS.iterations, ARGS.interval, ARGS.settle, ARGS.delay, ARGS.restarts):
File "/vagrant/interact.py", line 156, in requests_with_chaos
part = blk.random_partition()
File "/blockade/blockade/core.py", line 390, in random_partition
self.partition(partitions, state)
File "/blockade/blockade/core.py", line 404, in partition
container_partitions)
File "/blockade/blockade/net.py", line 62, in partition_containers
partition_containers(blockade_id, partitions)
File "/blockade/blockade/net.py", line 298, in partition_containers
iptables_create_chain(chain_name)
File "/blockade/blockade/net.py", line 246, in iptables_create_chain
iptables_call("-N", chain)
File "/blockade/blockade/net.py", line 133, in iptables_call
raise BlockadeError(error)
blockade.errors.BlockadeError: Problem calling 'iptables -N blockade-countercassandraacyclic-p1'
Maybe it makes sense to truncate the folder name before using it as blockade_id
?
I tried to bind several container port to all host interfaces (0.0.0.0) without success using following config:
ports: {48004: 48004, 48005: 48005, 48006: 48006, 9001: 9001, 8888: 8888}
Please correct me if I am wrong.
Hi!
Firstly, thanks for this excellent tool. This could potentially save my behind on a project. Secondly, does the "Create a Blockade" endpoint support a "network" key like the blockade.yaml?
Thanks!
Joseph Murphy
When I perform blockade up and submit them to the mistake, where is the problem? How to solve?
Hi,
lets say, we are using this configuration:
containers:
c1:
image: my_docker_image
c2:
image: my_docker_image
links: ["c1"]
c3:
image: my_docker_image
links: ["c2"]
In our understanding of the documentation, c2
should be able to communicate with c1
and c3
and c3
should be able to communicate with c2
but not with c1
(since they are not linked).
We tested this configuration and we were wondered that every machine can communicate with every other one. Do we misunderstand the documentation (part links
) or is this a bug? If we misunderstand the documentation, how can we archive our goal?
Thanks for your help!
[x] Bug (Typo)
targetting
, however expect to see targeting
.specificed
, however expect to see specified
.refererences
, however expect to see references
.contrainer
, however expect to see container
.Semi-automated issue generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md
To avoid wasting CI processing resources a branch with the fix has been
prepared but a pull request has not yet been created. A pull request fixing
the issue can be prepared from the link below, feel free to create it or
request @timgates42 create the PR. Alternatively if the fix is undesired please
close the issue with a small comment about the reasoning.
https://github.com/timgates42/blockade/pull/new/bugfix_typos
Thanks.
In my use case, I want to start several containers that handle my testing infrastructure: test drivers, logging, monitoring, alerting, etc. I want to use blockade
to start and manipulate a set of nodes under test.
I want to start and destroy the blockade
network between different tests. The problem is that blockade
tries and fails to remove the user-defined network because my test infra is still connected to that network. Then it raises an exception and stops. Also, it creates a brand new network with the same name when I start blockade
. So I've got multiple networks with the same name now.
I think one solution is to use the idea of an external network from docker-compose. I want to tell blockade
to use an externally defined network, then it should not create and destroy it every time. It should just work on the containers. In docker-compose I'd do this:
networks:
blockade:
external:
name: testinfra_net
Hi guys, first of all thanks for the great tool.
I'm trying to use Blockade in my stack of tests and have some doubts. I have two containers one is server and other one is the client. I'm running my tests in client and in some cases need to change network parameters of server or client container. In documentation tells about Blockade API but i don't know how to use the API inside client container.
See below my setup configuration of blockade.yaml file
containers:
server:
image: ubuntu:trusty
command: sh -c "/bin/sleep 300000"
volumes: ".": "/home/server/"
expose: [80]
ports: [5000]
client:
image: ubuntu:trusty
command: sh -c "echo $SERVER_PORT_5000_TCP_ADDR ' google.com.br' >> /etc/hosts; ping $SERVER_PORT_5000_TCP_ADDR"
volumes:
"/home/user/client_app": "/home/client_app"
links: ["server"]
network:
flaky: 80%
slow: 75ms 100ms distribution normal
I'm trying blockade on OS/X where docker -d
is running inside a vagrant VM (via https://github.com/noplay/docker-osx).
I have $DOCKER_HOST
set, so normal docker
commands work nicely.
When I run blockade up
I get an error that looks like its only trying to connect via local sockets.
$ blockade up
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/blockade/cli.py", line 242, in main
opts.func(opts)
File "/Library/Python/2.7/site-packages/blockade/cli.py", line 109, in cmd_up
containers = b.create()
File "/Library/Python/2.7/site-packages/blockade/core.py", line 47, in create
veth_device)
File "/Library/Python/2.7/site-packages/blockade/core.py", line 60, in _start_container
environment=container.environment)
File "/Library/Python/2.7/site-packages/docker/client.py", line 347, in create_container
return self.create_container_from_config(config, name)
File "/Library/Python/2.7/site-packages/docker/client.py", line 354, in create_container_from_config
res = self._post_json(u, data=config, params=params)
File "/Library/Python/2.7/site-packages/docker/client.py", line 199, in _post_json
return self._post(url, data=json.dumps(data2), **kwargs)
File "/Library/Python/2.7/site-packages/docker/client.py", line 89, in _post
return self.post(url, **self._set_request_timeout(kwargs))
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 377, in post
return self.request('POST', url, data=data, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
ConnectionError: UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /run/docker.sock/v1.6/containers/create?name=blockade-eaf8721d19-c1 (Caused by <class 'socket.error'>: [Errno 2] No such file or directory)
as in issue #8 when running blockade logs c2
I'm seeing unexpected errors
# blockade logs c1
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/blockade/cli.py", line 242, in main
opts.func(opts)
File "/usr/lib/python3.4/site-packages/blockade/cli.py", line 195, in cmd_logs
puts(b.logs(opts.container))
File "/usr/lib/python3.4/site-packages/clint/textui/core.py", line 57, in puts
s = tsplit(s, NEWLINES)
File "/usr/lib/python3.4/site-packages/clint/utils.py", line 69, in tsplit
string = string.replace(i, final_delimiter)
TypeError: expected bytes, bytearray or buffer compatible object
blockade status
# blockade status
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 41c9647701aa UP 172.17.0.8 NORMAL
c2 278fab07acff UP 172.17.0.10 NORMAL
c3 cbb51d715248 UP 172.17.0.9 NORMAL
If I use docker logs # docker logs 278fab07acff
PING 172.17.0.8 (172.17.0.8) 56(84) bytes of data.
64 bytes from 172.17.0.8: icmp_seq=1 ttl=64 time=0.126 ms
64 bytes from 172.17.0.8: icmp_seq=2 ttl=64 time=0.073 ms
64 bytes from 172.17.0.8: icmp_seq=3 ttl=64 time=0.055 ms
64 bytes from 172.17.0.8: icmp_seq=4 ttl=64 time=0.050 ms
64 bytes from 172.17.0.8: icmp_seq=5 ttl=64 time=0.051 ms
64 bytes from 172.17.0.8: icmp_seq=6 ttl=64 time=0.051 ms
64 bytes from 172.17.0.8: icmp_seq=7 ttl=64 time=0.050 ms
64 bytes from 172.17.0.8: icmp_seq=8 ttl=64 time=0.057 ms
Seems to work just fine.
Let me know if you need more info.
Hook python's logging conf into the docker daemon.
As far as i understand this isnt yet implemented but should be technically possible.
Do you see something that would prevent this ?
I may be able to contribute
Update: starting c3 fails, so that's why exec fails. Starting the container manually says: sh: 1: ping: not found
:)
blockade up
on the ping example, but ubuntu/xenial image, fails to start the c2 c3 (for idx, container in enumerate(self.config.sorted_containers):
returns c1, c3, c2) container with the following error:
Error:
Failed to find container network interface:
exec 'cat /sys/class/net/eth0/ifindex' in container bc24ae2543d93423693820dfe01c5a47d11fe8fc3235386987ef4e2346434751 returned:
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:83: executing setns process caused \\\"exit status 16\\\"\"\n"
Ensure the container is alive and supports this exec command.
the c1 container seems to be running fine, ifindex is 4:
root@b262e87ac491:/# cat /sys/class/net/eth0/ifindex
4
Btw is there a way to use pdb to debug the blockade without having to modify the cli.py file to add import pdb (can't use -m pdb, bcs need -m to run blockade.cli)?
All running in a vmware machine on a Mac:
$ python --version
Python 2.7.12
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
$ uname -a
Linux osboxes 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$
Is there a nice elegant way to spin up a couple hundred containers, and then randomly bring down and up a percentage of containers over time? I am trying to create a network with a few hundred nodes, and check how fault tolerant the network is to node failures.
I am frequently encountering this error for no apparent reason:
Error:
Problem determining host network device for container '157795997a89682870860bb97f4f18383c9b7b4112b4be4cf047f1d8e4db217c'
Can you help me understand what causes this, and how I might resolve it?
For the volumes
configuration I want to use shell environment variables within the config string when specifying the directory paths to mount.
I want to be able to do this:
volumes:
"$GOPATH/bin": "/gobin"
This should pick up the environment variable $GOPATH
and use that for bind-mounting the directory to the container. That's how it works in a Docker Compose file.
Instead, in blockade, not only does it not recognize and expand the environment variable, it also creates a directory called $GOPATH
and a subdirectory $GOPATH/bin
in the current working directory.
I'm testing an image built by the Maven Spring-Boot plugin. The blockade cluster (2 nodes) came up normally. I destroyed the blockade. I changed some code, rebuilt the container, ran blockade up
, and this happened.
$ blockade up
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/cli.py", line 617, in main
opts.func(opts)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/cli.py", line 165, in cmd_up
containers = b.create(verbose=opts.verbose, force=opts.force)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 96, in create
container_id = self._start_container(container, force)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 187, in _start_container
self.docker_client.start(container_id)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/container.py", line 1065, in start
self._raise_for_status(res)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/client.py", line 216, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/errors.py", line 30, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.24/containers/5ed75fefe61adeb5aa157b4c3e9fd9f616532c0231992a80bf1932c385865b2d/start ("Cannot link to a non running container: /simplecluster_seedA AS /simplecluster_nodeA/seedA")
I guessed it couldn't hurt to try again:
$ blockade up
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/cli.py", line 617, in main
opts.func(opts)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/cli.py", line 165, in cmd_up
containers = b.create(verbose=opts.verbose, force=opts.force)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 96, in create
container_id = self._start_container(container, force)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 174, in _start_container
container_id = create_container()
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 170, in create_container
labels={"blockade.id": self.state.blockade_id})
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/container.py", line 446, in create_container
return self.create_container_from_config(config, name)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/container.py", line 457, in create_container_from_config
return self._result(res, True)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/client.py", line 220, in _result
self._raise_for_status(response)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/client.py", line 216, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/errors.py", line 30, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
APIError: 409 Client Error: Conflict for url: http+docker://localunixsocket/v1.24/containers/create?name=simplecluster_seedA ("Conflict. The container name "/simplecluster_seedA" is already in use by container "c762ab326e9781d11c28689cfaf33c59a12eb4974e7d6ad9b7741a242ba72347". You have to remove (or rename) that container to be able to reuse that name.")
Looking at the error, I noticed it was trying to re-use the name. So, I removed the old containers and tried again. Then the first error came up again.
$ blockade up
Unexpected error! This may be a Blockade bug.
Traceback (most recent call last):
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/cli.py", line 617, in main
opts.func(opts)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/cli.py", line 165, in cmd_up
containers = b.create(verbose=opts.verbose, force=opts.force)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 96, in create
container_id = self._start_container(container, force)
File "/home/bushman/.local/lib/python2.7/site-packages/blockade/core.py", line 187, in _start_container
self.docker_client.start(container_id)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/container.py", line 1065, in start
self._raise_for_status(res)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/api/client.py", line 216, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/bushman/.local/lib/python2.7/site-packages/docker/errors.py", line 30, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.24/containers/3a1d1486d236730c1cbc5e3006bc4c3eef4d17eba4efa28e11b8567ffa7b4d15/start ("Cannot link to a non running container: /simplecluster_seedA AS /simplecluster_nodeA/seedA")
My blockade.yaml
:
containers:
seedA:
image: resilience/simple-resilient-cluster:latest
hostname: seedA
expose: [8080, 42001]
ports: {8080: 8080}
environment: {
CLUSTER_SEED_HOST: seedA,
CLUSTER_LOCAL_HOST: seedA,
RESILIENCE_GOSSIP_TIME: 5000,
RESILIENCE_GOSSIP_TARGETS: 1,
RESILIENCE_INFECTION_TIME: 1000,
RESILIENCE_INFECTION_THRESHOLD: 1,
RESILIENCE_INFECT_PER_ROUND: 1,
RESILIENCE_PUSH_TIMEOUT: 500,
RESILIENCE_PUSH_TARGETS: 1
}
volumes: {
"./logs": "/tmp/resilience_logs"
}
nodeA:
image: resilience/simple-resilient-cluster:latest
hostname: nodeA
start_delay: 10
expose: [8080, 42001]
ports: {8081: 8080}
environment: {
CLUSTER_SEED_HOST: seedA,
CLUSTER_LOCAL_HOST: nodeA,
RESILIENCE_GOSSIP_TIME: 5000,
RESILIENCE_GOSSIP_TARGETS: 1,
RESILIENCE_INFECTION_TIME: 1000,
RESILIENCE_INFECTION_THRESHOLD: 1,
RESILIENCE_INFECT_PER_ROUND: 1,
RESILIENCE_PUSH_TIMEOUT: 500,
RESILIENCE_PUSH_TARGETS: 1
}
volumes: {
"./logs": "/tmp/resilience_logs"
}
links: [seedA]
Anything I can do here?
Currently, it is stated in the docs that, for the partition command :
blockade partition n1,n2 n3,n4
Create one or more network partitions. Each partition is specified as a comma-separated list. Containers may not exist in more than one partition.
It would be very nice to add this feature to blockade, as it would allow to test very nasty cases of intersecting partitions.
Hi,
When I try to install blockade via pip it fails due to a quite old version of gevent (fails to build binary wheel). However, I can install it locally after checking the repo out. I see that the requirements have been updated in recent PRs - perhaps you could update pypi, even if it is just with a release candidate?
Hello,
First thank you for blockade!
I noticed something about the ports. I might use the wrong configuration but I think the ports are not exposed properly:
containers:
node1:
image: sisays/elasticsearch-docker
command: /elasticsearch/bin/elasticsearch -Des.config=/data/config/elasticsearch.yml
volumes:
"/vagrant/experiments/es-cluster/split-brain/run/node1": "/data"
ports:
9200: 9200
9300: 9300
node2:
image: sisays/elasticsearch-docker
command: /elasticsearch/bin/elasticsearch -Des.config=/data/config/elasticsearch.yml
volumes:
"/vagrant/experiments/es-cluster/split-brain/run/node2": "/data"
ports:
9202: 9200
9302: 9300
links:
node1: node1
node3:
image: sisays/elasticsearch-docker
command: /elasticsearch/bin/elasticsearch -Des.config=/data/config/elasticsearch.yml
volumes:
"/vagrant/experiments/es-cluster/split-brain/run/node3": "/data"
ports:
9202: 9300
9302: 9300
links:
node1: node1
node2: node2
I start the cluster:
$ blockade up
NODE CONTAINER ID STATUS IP NETWORK PARTITION
node1 c955548045fe UP 172.17.0.17 NORMAL
node2 c617e523e595 UP 172.17.0.18 NORMAL
node3 6a9ec26d2639 UP 172.17.0.19 NORMAL
but then, when I want to check the exposed ports:
$ docker port c955548045fe
I get nothing. And as a consequence, I cannot access the service from the host
When I do pretty much the same with fig, I do get the ports:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1a772e415b68 sisays/elasticsearch-docker:1.4.0 "/elasticsearch/bin/ 49 seconds ago Up 47 seconds 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp splitbrain_node3_1
ae0d67bdc8c7 sisays/elasticsearch-docker:1.4.0 "/elasticsearch/bin/ 49 seconds ago Up 48 seconds 0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp splitbrain_node2_1
bef16973c695 sisays/elasticsearch-docker:1.4.0 "/elasticsearch/bin/ 50 seconds ago Up 49 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp splitbrain_node1_1
$ docker port bef16973c695
9200/tcp -> 0.0.0.0:9200
9300/tcp -> 0.0.0.0:9300
I haven't found in the documentation. Is there a way of dropping packages just in one direction? I.e. I want packages from c1
to c2
be dropped, but packages from c2
to c1
transmitted.
I would like to preface by stating that I'm new to Docker and certainly to Blockade, so I apologize in advance if I come off as a noob (it's because I am).
I am attempting to build a distributed database for a school project and would like to use this software to test how my code performs when network partitions are brought into play.
So my first question is:
1). Is there any way for me to specify my own ip addresses for the containers, or must I always rely on the ip addresses assigned to my containers by Blockade?
2). Is there any way for me to add more than one environment variable using the 'environment' field in the Blockade.yaml file? It seems that my Docker container application is only able to find a single environment variable. If I even try to add a second environment variable to the dictionary structure, my Docker container application will find neither of them.
3). It states in the docs that the 'environment' field is 'optional', but I can't seem to get Blockade to work without it. I keep getting an error message stating that Blockade is "unable to find the network interface".
Thank you very much for taking the time to answer these questions. You don't know how much I appreciate it.
Add configuration to timeout the http connection.
It strikes me that blockade
examples could be used as the basis of educational teaching examples in how resilient various distributed systems are - or aren't. I'm currently working at the limits of my knowledge on all this, as a complete docker novice, but would love to see a few more examples of of how blockade could be used to illustrate various network errors for some typical example configurations.
For example, comparing how different distributed databases compare when network weather hits. A couple of examples I'd particularly value are examples of how a simple3 node MongoDB replica set fares, or how a more complex sharded configuration (eg after https://sebastianvoss.com/docker-mongodb-sharded-cluster.html ) would cope.
How complex would the blockade
set up be in each of those cases? Is it possible to use blockade to test those sorts of config? How about for other distributed DBMS?
As https://github.com/worstcase/blockade/blob/master/docs/install.rst states, Docker Swarm isn't currently supported. I think I'd need Docker Swarm support in order to test machine clusters e.g. a Kafka cluster that's running on Virtual Machines e.g. from VMWare or a cluster of physical machines e.g. Raspberry Pi's :)
If you could provide some pointers on what'd it'd take to add support for Docker Swarm that'd be a great help. Of course, if there are other ways to solve my testing challenge (of meddling with nodes running Kafka, Zookeeper, etc. and their connections) that'd be appreciated too.
PS: thanks for an excellent project and capability, I came here by following a well-trodden path of Jepsen, then jepsen-python which uses blockade 👍
Like in docker-compose is it possible to assign static IP address in blockade? Looks like, blockade is assigning IP address at later stage in the lifecycle of containers (not sure), when I use
docker run --net mynet123 --ip 20.21.10.100 -it c197324bf9c3 -p 52204:52204
docker run -p 52205:52205 --net mynet123 --ip 20.21.10.101 -it c197324bf9c3
The nodes form a cluster as I expected. With blockade the application is not starting up.
In my case, I have to give the IP address of docker in the config file of my application when building an image itself.
is there a way to expose multiple ports and networks in blockade?
bitcoin2:
image: bitcoinnetwork_bitcoin2
ports:
- {19200: 19200}
- {19201: 19201}
networks:
- front-tier
- back-tier
container_name: bitcoin2
something like this does not work?
when I use blockade up command , but nothing happen,none of the error info and the status of containers ,my system is ubuntu 16.04 and with docker-engine 1.12 installed.
How about UDP port exposing in containers?
I just wanted to give you a heads up that since the ip
command has gained popularity in distros, the net-tools
package (on Ubuntu/Debian) is now required to make the ifconfig
command available for Blockade to function properly.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.