ucy-linc-lab / fogify Goto Github PK

I'm trying to set CPU and RAM restrictions for my docker containers but they don't seem to really apply. As far as I can tell, fogify only checks the sum of resources against the host system on startup, instead of actually enforcing them. There are multiple issues with this.

• I can maximize the use of the host CPU from within any container, even if it has a CPU limit set
• I can see and use the total amount of the host RAM from within any container, even if it has a RAM limit set
• I have run containers with limits of 0.1G RAM and 5 MHz (!) CPU. Applications within the container work with no issues at native speed and using as much RAM as they need (obviously more than 0.1G)

This is also connected to #3 I guess.

Hi,

This is a really great project!

I'm trying to run a Kubernetes cluster in Fogify. For this purpose, I've set up a docker-compose file and shell scripts (see here) that initialize a Kubernetes cluster using the Docker images and bootstrapping procedure from the kind project. Running the cluster with docker-compose works, but with Fogify, the containers keep crashing, because they are not executed in privileged mode.

Steps to reproduce the problem:

1. Build the Docker image from here
2. Run the docker-compose file from here in Fogify.
3. List all containers using docker ps --all
4. Get the logs for one of the crashed control-plane container. They will contain the following:

WARN: /dev/kmsg does not exist, nor does /dev/console!
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
mount: /sys: permission denied.

Using a docker-compose file with just the kind base image should allow reproducing the problem as well. When using this option, it is important that the container is configured with the following options:

volumes:
  # Ensures that pods, logs etc. are not on the container filesystem.
  - "/var"
  # Some K8s things want to read /lib/modules.
  - "/lib/modules:/lib/modules:ro"
tmpfs:
  - "/tmp" # various things depend on working /tmp
  - "/run" # systemd wants a writable /run
privileged: true
security_opt:
  - "seccomp=unconfined"
  - "apparmor=unconfined"

snk@snk-Latitude-E6420:~/fogify-master$ sudo docker-compose -p fogemulator up
WARN[0001] The "RAM_OVERSUBSCRIPTION_PERCENTAGE" variable is not set. Defaulting to a blank string.
WARN[0001] The "SNIFFING_PERIOD" variable is not set. Defaulting to a blank string.
WARN[0001] The "CONNECTOR" variable is not set. Defaulting to a blank string.
WARN[0001] The "CPU_OVERSUBSCRIPTION_PERCENTAGE" variable is not set. Defaulting to a blank string.
WARN[0001] The "SNIFFING_ENABLED" variable is not set. Defaulting to a blank string.
WARN[0001] The "SNIFFING_PERIODICITY" variable is not set. Defaulting to a blank string.
WARN[0001] The "CONNECTOR" variable is not set. Defaulting to a blank string.
[+] Running 7/7
! ui Warning 3.2s
! controller Warning 3.2s
✔ cadvisor 3 layers [⣿⣿⣿] 0B/0B Pulled 30.6s
✔ 9d48c3bd43c5 Pull complete 2.2s
✔ f7d6cbe0ad90 Pull complete 21.3s
✔ 15f5311b080f Pull complete 11.4s
! agent Warning 3.3s
[+] Building 786.0s (26/36) docker:default
=> [controller internal] load build definition from dockerfile 2.1s
=> => transferring dockerfile: 1.60kB 0.1s
=> [controller internal] load .dockerignore 2.3s
=> => transferring context: 2B 0.0s
=> [ui internal] load build definition from dockerfile 2.4s
=> => transferring dockerfile: 322B 0.0s
=> [ui internal] load .dockerignore 1.7s
=> => transferring context: 2B 0.0s
=> [controller internal] load metadata for docker.io/library/python:3.9 5.8s
=> [ui internal] load metadata for docker.io/jupyter/scipy-notebook:latest 6.6s
=> [controller internal] load build context 1.2s
=> => transferring context: 129.81kB 0.4s
=> [controller 1/24] FROM docker.io/library/python:3.9@sha256:b2e47a7eca3178e4ce6c095d3a2d1cd05bfa616efe7f2047c95fffe159e00166 244.1s
=> => resolve docker.io/library/python:3.9@sha256:b2e47a7eca3178e4ce6c095d3a2d1cd05bfa616efe7f2047c95fffe159e00166 1.2s
=> => sha256:b2e47a7eca3178e4ce6c095d3a2d1cd05bfa616efe7f2047c95fffe159e00166 1.86kB / 1.86kB 0.0s
=> => sha256:1ab70c106669b2bc6d94518d59814c6de98e0477618435267d848dc5fb5f9dbb 2.01kB / 2.01kB 0.0s
=> => sha256:f40a52f78dc1431593ddf29709099426ce508e6198116ef58d2f8b374f8617a3 7.46kB / 7.46kB 0.0s
=> => sha256:27e1a8ca91d35598fbae8dee7f1c211f0f93cec529f6804a60e9301c53a604d0 24.05MB / 24.05MB 31.0s
=> => sha256:90e5e7d8b87a34877f61c2b86d053db1c4f440b9054cf49573e3be5d6a674a47 49.58MB / 49.58MB 34.7s
=> => sha256:d3a767d1d12e57724b9f254794e359f3b04d4d5ad966006e5b5cda78cc382762 64.13MB / 64.13MB 45.9s
=> => sha256:711be5dc50448ab08ccab0b44d65962f36574d341749ab30651b78ec0d4bfd1c 211.07MB / 211.07MB 199.0s
=> => extracting sha256:90e5e7d8b87a34877f61c2b86d053db1c4f440b9054cf49573e3be5d6a674a47 3.1s
=> => sha256:48b2d58a56e99520540b5ecafb202ae036a53b5a982845434722691ccba6499a 6.16MB / 6.16MB 42.2s
=> => extracting sha256:27e1a8ca91d35598fbae8dee7f1c211f0f93cec529f6804a60e9301c53a604d0 0.9s
=> => sha256:9205253a8842fbfd44c66278e90c4a3d453acd9ffb73320449abb0982f6aec1e 15.82MB / 15.82MB 58.3s
=> => sha256:a5992e8ed15e53f4ed12c455af1a16c727cd314d8d8c52e1cf0292caea689ca6 233B / 233B 46.8s
=> => extracting sha256:d3a767d1d12e57724b9f254794e359f3b04d4d5ad966006e5b5cda78cc382762 4.0s
=> => sha256:681e36302a03861b1f68644c4cafb1e3a2070203af357284b94cdb829530c7dd 2.85MB / 2.85MB 50.7s
=> => extracting sha256:711be5dc50448ab08ccab0b44d65962f36574d341749ab30651b78ec0d4bfd1c 17.8s
=> => extracting sha256:48b2d58a56e99520540b5ecafb202ae036a53b5a982845434722691ccba6499a 3.0s
=> => extracting sha256:9205253a8842fbfd44c66278e90c4a3d453acd9ffb73320449abb0982f6aec1e 2.2s
=> => extracting sha256:a5992e8ed15e53f4ed12c455af1a16c727cd314d8d8c52e1cf0292caea689ca6 0.0s
=> => extracting sha256:681e36302a03861b1f68644c4cafb1e3a2070203af357284b94cdb829530c7dd 1.6s
=> [ui 1/4] FROM docker.io/jupyter/scipy-notebook@sha256:fca4bcc9cbd49d9a15e0e4df6c666adf17776c950da9fa94a4f0a045d5c4ad33 741.9s
=> => resolve docker.io/jupyter/scipy-notebook@sha256:fca4bcc9cbd49d9a15e0e4df6c666adf17776c950da9fa94a4f0a045d5c4ad33 0.8s
=> => sha256:fca4bcc9cbd49d9a15e0e4df6c666adf17776c950da9fa94a4f0a045d5c4ad33 772B / 772B 0.0s
=> => sha256:bf4de44b0fa8a422ebc325e695d572d304bd290c055e028a6507921a294213bc 6.78kB / 6.78kB 0.0s
=> => sha256:ad65fcfebde3d9d6590bca3f709f1e386a4e56bd63654c0babd5ebf41e9edd9e 18.57kB / 18.57kB 0.0s
=> => sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e 29.54MB / 29.54MB 70.5s
=> => sha256:fd92c719666cac5862b8ea1e034eeb038c7191d52d0ff468c0a90ed68c0420a4 8.82MB / 8.82MB 66.8s
=> => sha256:088f11eb1e74ae07b0c0cfefb7906e1821c0b84f00524cc8e4011c276fe7d2d7 679B / 679B 68.0s
=> => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 68.5s
=> => sha256:ef8373d600b0ac4a916da50b72cb116efc56800836a264d1dd8ea8e07e76256d 1.92kB / 1.92kB 69.4s
=> => sha256:77e45ee945dc8a1ce57be013482750833e7721903814e42510a3db2a05940aee 4.92kB / 4.92kB 70.6s
=> => extracting sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e 1.9s
=> => sha256:a30f89a0af6c7f64872ff93f4a9445b9d25f1206982670a911b2c83b672f443a 151B / 151B 71.1s
=> => sha256:dc42adc7eb7398a8589bad907643f9b08467de588b3c2719c78302d5ec480446 276B / 276B 72.1s
=> => sha256:abaa8376a6502612b92e1750cee7aa450ef045e23a418077439a392b0c224f2e 104.82MB / 104.82MB 152.5s
=> => sha256:aa099bb9e49a60e4a86d1015fa7be3ea1dc1dc87153a0c0767dac5c3df94babb 4.40kB / 4.40kB 74.0s
=> => extracting sha256:fd92c719666cac5862b8ea1e034eeb038c7191d52d0ff468c0a90ed68c0420a4 0.9s
=> => sha256:822c4cbcf6a6b7660f90516cd166125d55de917a2f18c086eee2abe52c1b7e9a 181B / 181B 75.5s
=> => sha256:d25166dcdc7b4776765e67e9b3548b023fb7bf2a9bd1d14746fa3d1d902279a0 30.50MB / 30.50MB 101.9s
=> => extracting sha256:088f11eb1e74ae07b0c0cfefb7906e1821c0b84f00524cc8e4011c276fe7d2d7 0.0s
=> => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.0s
=> => extracting sha256:ef8373d600b0ac4a916da50b72cb116efc56800836a264d1dd8ea8e07e76256d 0.0s
=> => extracting sha256:77e45ee945dc8a1ce57be013482750833e7721903814e42510a3db2a05940aee 0.0s
=> => extracting sha256:a30f89a0af6c7f64872ff93f4a9445b9d25f1206982670a911b2c83b672f443a 0.0s
=> => extracting sha256:dc42adc7eb7398a8589bad907643f9b08467de588b3c2719c78302d5ec480446 0.0s
=> => sha256:964fc3e4ff9f3bb80c77f28ca6318e4e121dc657731158efb42f322dd7f521c3 137.74MB / 137.74MB 209.9s
=> => sha256:2c4c69587ee486a6d786449a71621bdcbc4ad1653c5f7f5848e0eb5a7deaa786 1.15kB / 1.15kB 153.8s
=> => extracting sha256:abaa8376a6502612b92e1750cee7aa450ef045e23a418077439a392b0c224f2e 7.6s
=> => sha256:de2cdd875fa8d3fe6e191d8b1c0dd0bfb677cba16a0fe1a67dd66bef00da6dac 1.42kB / 1.42kB 154.4s
=> => sha256:75d33599f5f251fdff33b1556b84d51b4bfef6781fe41672deea7a5fae31b57e 1.43kB / 1.43kB 155.4s
=> => sha256:31973ea8247081810aedfbad471cb29e8cad519f6e0cd488b6692b04e61ada5d 178.43MB / 178.43MB 331.1s
=> => extracting sha256:aa099bb9e49a60e4a86d1015fa7be3ea1dc1dc87153a0c0767dac5c3df94babb 0.0s
=> => extracting sha256:822c4cbcf6a6b7660f90516cd166125d55de917a2f18c086eee2abe52c1b7e9a 0.0s
=> => extracting sha256:d25166dcdc7b4776765e67e9b3548b023fb7bf2a9bd1d14746fa3d1d902279a0 2.3s
=> => sha256:96ee7e4439c77a57032d37c61ee62749216ebdd514e813e96123a92076ccc967 1.47kB / 1.47kB 199.8s
=> => sha256:1f9ad23c07ac8f228ec821f2095f22735c64308e16073c52722cd5d113f7f880 437B / 437B 200.4s
=> => sha256:d19266e0cb17753722408d50ece152f5c2cd764246d9ce4cbf6c20a82d4ccb4c 1.26kB / 1.26kB 202.1s
=> => sha256:9a165b6e9dc7f764933ed13711e8d3d2866e635d3c5740cfc3a1d303b0c1968d 279.53MB / 279.53MB 392.0s
=> => sha256:5689442fd4e1eb2168a70199d7321f5dde66f5dcecc207e57836a6baac48260d 236B / 236B 212.0s
=> => extracting sha256:964fc3e4ff9f3bb80c77f28ca6318e4e121dc657731158efb42f322dd7f521c3 32.9s
=> => sha256:9a6a202f62a6599a1ebad61a11e5d953f35736256aa11509a8f369883cdcd4e3 546.28MB / 546.28MB 648.0s
=> => extracting sha256:2c4c69587ee486a6d786449a71621bdcbc4ad1653c5f7f5848e0eb5a7deaa786 0.0s
=> => extracting sha256:de2cdd875fa8d3fe6e191d8b1c0dd0bfb677cba16a0fe1a67dd66bef00da6dac 0.0s
=> => extracting sha256:75d33599f5f251fdff33b1556b84d51b4bfef6781fe41672deea7a5fae31b57e 0.0s
=> => sha256:734ea0c3d94e1cb3af924749a5493983109f80abdadf70e919717767c04ed676 597.22kB / 597.22kB 336.0s
=> => extracting sha256:31973ea8247081810aedfbad471cb29e8cad519f6e0cd488b6692b04e61ada5d 19.2s
=> => sha256:a21a167f71274a2618b5ffb4187cbd6dafa5ac63d7179a54fc097e0e7b742f63 7.12kB / 7.12kB 337.8s
=> => extracting sha256:96ee7e4439c77a57032d37c61ee62749216ebdd514e813e96123a92076ccc967 0.0s
=> => extracting sha256:1f9ad23c07ac8f228ec821f2095f22735c64308e16073c52722cd5d113f7f880 0.0s
=> => extracting sha256:d19266e0cb17753722408d50ece152f5c2cd764246d9ce4cbf6c20a82d4ccb4c 0.0s
=> => extracting sha256:9a165b6e9dc7f764933ed13711e8d3d2866e635d3c5740cfc3a1d303b0c1968d 28.7s
=> => extracting sha256:5689442fd4e1eb2168a70199d7321f5dde66f5dcecc207e57836a6baac48260d 0.1s
=> => extracting sha256:9a6a202f62a6599a1ebad61a11e5d953f35736256aa11509a8f369883cdcd4e3 66.4s
=> => extracting sha256:734ea0c3d94e1cb3af924749a5493983109f80abdadf70e919717767c04ed676 0.3s
=> => extracting sha256:a21a167f71274a2618b5ffb4187cbd6dafa5ac63d7179a54fc097e0e7b742f63 0.0s
=> [ui internal] load build context 0.9s
=> => transferring context: 40.50kB 0.0s
=> [controller 2/24] RUN apt-get update -y 82.7s
=> [controller 3/24] RUN apt-get install -y git build-essential libncurses5-dev libslang2-dev gettext zlib1g-dev libselinux1-dev deb 95.7s
=> [controller 4/24] RUN apt-get install -y bison util-linux iproute2 26.0s
=> [controller 5/24] RUN /usr/local/bin/python -m pip install --upgrade pip 18.7s
=> [controller 6/24] RUN curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" 11.4s
=> [controller 7/24] RUN chmod +x /usr/local/bin/docker-compose 3.1s
=> [controller 8/24] RUN DEBIAN_FRONTEND=noninteractive apt-get install -y tshark 105.0s
=> [controller 9/24] RUN mkdir /code 2.2s
=> [controller 10/24] WORKDIR /code 1.9s
=> [controller 11/24] ADD requirements.txt /code/ 1.0s
=> [controller 12/24] RUN pip3 install -r requirements.txt 152.3s
=> [ui 2/4] RUN mkdir /FofigySDK 8.6s
=> [controller 13/24] RUN apt-get update 13.7s
=> [ui 3/4] COPY . /FofigySDK 2.1s
=> ERROR [ui 4/4] RUN pip install /FofigySDK && fix-permissions /opt/conda && fix-permissions /home/jovyan 17.2s
=> CANCELED [controller 14/24] RUN apt-get install net-tools 18.5s

[ui 4/4] RUN pip install /FofigySDK && fix-permissions /opt/conda && fix-permissions /home/jovyan:
11.03 Processing /FofigySDK
11.05 Preparing metadata (setup.py): started
12.48 Preparing metadata (setup.py): finished with status 'done'
12.49 Requirement already satisfied: pandas in /opt/conda/lib/python3.11/site-packages (from FogifySDK==0.0.3) (2.1.1)
12.49 Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from FogifySDK==0.0.3) (2.31.0)
12.49 Requirement already satisfied: pyyaml in /opt/conda/lib/python3.11/site-packages (from FogifySDK==0.0.3) (6.0.1)
12.49 Requirement already satisfied: matplotlib in /opt/conda/lib/python3.11/site-packages (from FogifySDK==0.0.3) (3.8.0)
12.51 Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (1.1.1)
12.51 Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (0.12.1)
12.52 Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (4.43.1)
12.52 Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (1.4.5)
12.53 Requirement already satisfied: numpy<2,>=1.21 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (1.24.4)
12.53 Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (23.2)
12.53 Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (10.1.0)
12.53 Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (3.1.1)
12.54 Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.11/site-packages (from matplotlib->FogifySDK==0.0.3) (2.8.2)
12.64 Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.11/site-packages (from pandas->FogifySDK==0.0.3) (2023.3.post1)
12.64 Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.11/site-packages (from pandas->FogifySDK==0.0.3) (2023.3)
12.66 Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->FogifySDK==0.0.3) (3.3.0)
12.66 Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->FogifySDK==0.0.3) (3.4)
12.66 Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests->FogifySDK==0.0.3) (2.0.7)
12.66 Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->FogifySDK==0.0.3) (2023.7.22)
12.80 Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->FogifySDK==0.0.3) (1.16.0)
12.83 Building wheels for collected packages: FogifySDK
12.83 Building wheel for FogifySDK (setup.py): started
14.53 Building wheel for FogifySDK (setup.py): finished with status 'error'
14.55 error: subprocess-exited-with-error
14.55
14.55 × python setup.py bdist_wheel did not run successfully.
14.55 │ exit code: 1
14.55 ╰─> [5 lines of output]
14.55 running bdist_wheel
14.55 running build
14.55 running build_py
14.55 creating build
14.55 error: could not create 'build': Permission denied
14.55 [end of output]
14.55
14.55 note: This error originates from a subprocess, and is likely not a problem with pip.
14.55 ERROR: Failed building wheel for FogifySDK
14.55 Running setup.py clean for FogifySDK
14.90 Failed to build FogifySDK
14.90 ERROR: Could not build wheels for FogifySDK, which is required to install pyproject.toml-based projects

failed to solve: process "/bin/bash -o pipefail -c pip install /FofigySDK && fix-permissions $CONDA_DIR && fix-permissions /home/$NB_USER" did not complete successfully: exit code: 1

Hi,

I've been having trouble getting fogify to properly deploy docker containers. I keep getting a constraints error based on placement. If I run a basic deployment with 3 nodes, I observe the following after deployment

> sudo docker stack ps fogify
ID                          NAME              IMAGE            NODE      DESIRED STATE   CURRENT STATE            ERROR                                                                 PORTS
ythsey7iev4oaz2xjibsbta80   fogify_node-1.1   taxi-exp:0.0.1             Running         Pending 33 seconds ago   "no suitable node (scheduling constraints not satisfied on 1 node)"   
phz53b4hxunbn10g1kc5dwot7   fogify_node-2.1   taxi-exp:0.0.1             Running         Pending 34 seconds ago   "no suitable node (scheduling constraints not satisfied on 1 node)"   
um5iw455ixsfebs94wi8wf0nt   fogify_node-3.1   taxi-exp:0.0.1             Running         Pending 34 seconds ago   "no suitable node (scheduling constraints not satisfied on 1 node)"

In order to fix it I removed the contraints on placement specified in DockerBasedConnectors.node_representation. Specifically the ['placement'] constraints and the if statements about the main_cluster_node. I don't know if this is the proper fix, so happy to hear alternatives.

Thanks,
Tom Ebergen

While trying to enforce bandwidth/latency limits, I'm not sure how the settings finally apply. It certainly seems that latency plays a big role with respect to bandwidth as well. Latency, by itself, is however mostly OK. The problem is the effect it has on bandwidth. I'm running the following setups with 4 containers, deployed as two server/client pairs. These are only some examples that show this problem, I've run a lot more similar setups. I'm using iperf3 to calculate the actual bandwidth between containers.

• Setting bandwidth: 100Mbps (bidirectional) and delay: 3ms for all 4 containers. Ping results between containers are indeed ~3ms, but bandwidth is actually ~750Mbits/sec
• • Setting bandwidth: 100Mbps (bidirectional) and delay: 30ms for all 4 containers. Ping results between containers are indeed ~30ms, but bandwidth is actually ~710Mbits/sec
• • Setting bandwidth: 100Mbps (bidirectional) and delay: 60ms for all 4 containers. Ping results between containers are indeed ~3ms, but bandwidth is actually ~380 Mbits/sec
• Setting bandwidth: 1000Mbps (bidirectional) and delay: 3ms for all 4 containers. Ping results between containers are indeed ~3ms, but bandwidth is actually ~7.10 Gbits/sec
• Setting bandwidth: 1000Mbps (bidirectional) and delay: 30ms for all 4 containers. Ping results between containers are indeed ~30ms, but bandwidth is actually ~790Mbits/sec
• Setting bandwidth: 10000Mbps (bidirectional) and delay: 3ms for all 4 containers. Ping results between containers are indeed ~3ms, but bandwidth is actually ~7.10 Gbits/sec
• Setting bandwidth: 10000Mbps (bidirectional) and delay: 30ms for all 4 containers. Ping results between containers are indeed ~30ms, but bandwidth is actually ~780Mbits/sec
• Setting bandwidth: 10000Mbps (bidirectional) and delay: 60ms for all 4 containers. Ping results between containers are indeed ~3ms, but bandwidth is actually ~380 Mbits/sec

If you want to try similar setups, you can look into the start-network-test.sh script in this repo: https://github.com/Datalab-AUTH/fogify-db-benchmarks

I should note that on my system, connecting 2 docker containers with no traffic shaping whatsoever (and without fogify) results to actual bandwidth measurements of ~27Gbits/sec, so there is no bottleneck on the host system.

When running deploy() I get a message that everything has been deployed, but that may not be true, at least not yet.

This is an issue that is not the same as described in #3. It may occur, even if (eventually) all containers can be started.

The problem is that especially with a large number of containers to be started, the process of actually starting them is not instant. In fact, it might take several minutes until all containers are up and running.

In many cases, I can run deploy(), which returns with a "deployed" message, but if I then run fogify.info(), I may even get a blank response, or more usually not see all containers up yet. Simply by waiting a few minutes and running fogify.info() again, I eventually get the correct deployment info with all containers up.

This problem is not apparent on the python REPL/running in jupyter step by step or with a small number of containers, but it becomes apparent when trying to automate things with a larger number of containers.

I had to add a lot of time.sleep() statements in my code to actually have everything up as it should (like wait 20 seconds for each additional container), but I think the best way to handle this is for deploy() to return with a successful message only after all specified containers are actually up. I guess a timeout parameter can be added after which we can assume that deployment has failed and deploy() returns a failing message, while at the same time undeploying all containers that happened to already be up.

Hi,

I tried running a fresh installation of the fogify framework, but ran into issues with certain python packages no longer being supported. For instance I after running

sudo docker-compose build
sudo docker-compose -p fogemulator up

I would see the following error messages

controller_1  | Traceback (most recent call last):
controller_1  |   File "/code/fogify/main.py", line 2, in <module>
controller_1  |     from agent.agent import Agent
controller_1  |   File "/code/fogify/agent/agent.py", line 3, in <module>
controller_1  |     from connectors import get_connector
controller_1  |   File "/code/fogify/connectors/__init__.py", line 2, in <module>
controller_1  |     from . import materialized_connectors
controller_1  |   File "/code/fogify/connectors/materialized_connectors/__init__.py", line 1, in <module>
controller_1  |     from .DockerBasedConnectors import SwarmConnector, DockerComposeConnector
controller_1  |   File "/code/fogify/connectors/materialized_connectors/DockerBasedConnectors.py", line 9, in <module>
controller_1  |     from flask_api import exceptions
controller_1  |   File "/usr/local/lib/python3.7/site-packages/flask_api/__init__.py", line 1, in <module>
controller_1  |     from flask_api.app import FlaskAPI
controller_1  |   File "/usr/local/lib/python3.7/site-packages/flask_api/app.py", line 4, in <module>
controller_1  |     from flask._compat import reraise, string_types, text_type
controller_1  | ModuleNotFoundError: No module named 'flask._compat'
...
agent_1       | Traceback (most recent call last):
agent_1       |   File "/code/fogify/main.py", line 2, in <module>
agent_1       |     from agent.agent import Agent
agent_1       |   File "/code/fogify/agent/agent.py", line 3, in <module>
agent_1       |     from connectors import get_connector
agent_1       |   File "/code/fogify/connectors/__init__.py", line 2, in <module>
agent_1       |     from . import materialized_connectors
agent_1       |   File "/code/fogify/connectors/materialized_connectors/__init__.py", line 1, in <module>
agent_1       |     from .DockerBasedConnectors import SwarmConnector, DockerComposeConnector
agent_1       |   File "/code/fogify/connectors/materialized_connectors/DockerBasedConnectors.py", line 9, in <module>
agent_1       |     from flask_api import exceptions
agent_1       |   File "/usr/local/lib/python3.7/site-packages/flask_api/__init__.py", line 1, in <module>
agent_1       |     from flask_api.app import FlaskAPI
agent_1       |   File "/usr/local/lib/python3.7/site-packages/flask_api/app.py", line 4, in <module>
agent_1       |     from flask._compat import reraise, string_types, text_type
agent_1       | ModuleNotFoundError: No module named 'flask._compat

To fix this I added strict versioning to the requirements.txt file so it would look like

Flask==1.1.2
Werkzeug==1.0.1
Flask-API==2.0
requests==2.25.1
docker==5.0.0
Flask-SQLAlchemy==2.5.1
pyyaml==5.4.1
python-dateutil==2.8.1
psutil==5.8.0
py-cpuinfo==8.0.0
netifaces==0.11.0
nsenter==0.2
uWSGI==2.0.19.1

Let me know if you have any questions.
--Tom Ebergen

I'm trying to run command(), but no matter what I do it doesn't seem to work. I get an OK message, but the command I specified, never really runs on any of the containers. For example, I'm trying to run a touch /testfile command, I get an OK response, but the file never appears in any of the running containers.

Here is an example scenario: on a host with 16GB RAM, I'm trying to run 20 containers with 1GB RAM allocated each.

When running fogify.deploy() I get an OK message, that everything is deployed. But that is not true. In fact, at least 4 of my containers are never started and the choice seems to be random. It appears that fogify checks the sum of resources needed against the host system and does not start any containers that exceed the host system resources (but not the actual available host resources, i.e. it checks against the total of 16GB, but not against the 8GB that may possible be actually available at that point, I guess that's another issue?).

I'm not sure if not starting the containers at all is the right course of action here. After all, I am able to start as many (or more) such containers in my system without fogify, or even by using fogify and setting lower RAM limits. But my main issue is that it all happens silently and the user is left to believe that everything went well and all containers are up.

In a scenario such as the one described in #3 not all specified containers are started. However, if I run undeploy() it reports that all containers have been undeployed (for example 20/20, while only 16/20 were actually deployed in the first place). This is misleading of what actually happens.

Hi!
I am trying to deploy fogify to run an experiment, and I'm trying to run that through the docker compose rather than baremetal.

However even though I set the variables in a .env file and pass that to the containers, the IP of the host is not used to run the agent and controller

Is that an expected behavior or does that mean I missed some setting for the deployment?
Thank you!

Namespace(agent=True, agent_ip='192.168.1.102', controller=False)
Running on http://0.0.0.0:5500/ (Press CTRL+C to quit)

Hi,

I have been working on getting Fogify to work on an ubuntu virtual image running in Vagrant. When attempting to deploy a topology, I was getting the following error from the controller

controller_1  | During handling of the above exception, another exception occurred:
controller_1  |
controller_1  | Traceback (most recent call last):
controller_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
controller_1  |     rv = self.dispatch_request()
controller_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
controller_1  |     return self.view_functions[rule.endpoint](**req.view_args)
controller_1  |   File "/usr/local/lib/python3.7/site-packages/flask/views.py", line 89, in view
controller_1  |     return self.dispatch_request(*args, **kwargs)
controller_1  |   File "/usr/local/lib/python3.7/site-packages/flask/views.py", line 163, in dispatch_request
controller_1  |     return meth(*args, **kwargs)
controller_1  |   File "/code/fogify/controller/views.py", line 277, in post
controller_1  |     res.append(Communicator(get_connector()).agents__perform_action(commands, instance_type=instance_type))
controller_1  |   File "/code/fogify/utils/inter_communication.py", line 50, in agents__perform_action
controller_1  |   File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 119, in post
controller_1  |     return request('post', url, data=data, json=json, **kwargs)
controller_1  |   File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
controller_1  |     return session.request(method=method, url=url, **kwargs)
controller_1  |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
controller_1  |     resp = self.send(prep, **send_kwargs)
controller_1  |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
controller_1  |     r = adapter.send(request, **kwargs)
controller_1  |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
controller_1  |     raise ConnectionError(e, request=request)
controller_1  | requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=5500): Max retries exceeded with url: /actions/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f51bc61f910>: Failed to establish a new connection: [Errno 111] Connection refused'))

I tracked it down to the utils/Communicator class. It looks like agents__perform_action defaults to 0.0.0.0 for the address of the agents if everything is hosted on one machine. I think this should be changed to agent so that the internal docker DNS can resolve the URL properly. In vagrant, 0.0.0.0 doesn't work as the virtual machine adopts a different host IP (not 100% sure how it works). Anyway, my fix was to check if socket.gethostbyname(i) returns 0.0.0.0. If so, the host name is changed to agent.

Let me know if there are any questions, this issue is probably more confusing than the others
--Tom Ebergen