Comments (12)
swills
commented on May 29, 2024
1
Could be related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241646
from docker-machine-driver-bhyve.
swills
commented on May 29, 2024
1
Closing this since it's confirmed not to be an issue in docker-machine-driver-bhyve itself.
from docker-machine-driver-bhyve.
swills
commented on May 29, 2024
Can you tell me the files that are in ~/.docker directory? Just the file names, not the contents.
from docker-machine-driver-bhyve.
sarcasticadmin
commented on May 29, 2024
Thanks @swills I appreciate the insight. Ill get back to you on the directory shortly, Im away from the hardware at the moment.
from docker-machine-driver-bhyve.
sarcasticadmin
commented on May 29, 2024
@swills heres the files of the ~/.docker:
rherna@current:~ $ tree .docker/
.docker/
`-- machine
|-- bhyve.leases
|-- cache
|-- certs
| |-- ca-key.pem
| |-- ca.pem
| |-- cert.pem
| `-- key.pem
|-- dnsmasq.conf
|-- dnsmasq.pid
`-- machines
|-- mytest
| `-- config.json
`-- robstest
`-- config.json
6 directories, 9 files
With file permissions:
rherna@current:~ $ tree -pufi .docker/
.docker
[drwx------ rherna ] .docker/machine
[-rw-r--r-- root ] .docker/machine/bhyve.leases
[drwx------ rherna ] .docker/machine/cache
[drwx------ rherna ] .docker/machine/certs
[-rw------- rherna ] .docker/machine/certs/ca-key.pem
[-rw-r--r-- rherna ] .docker/machine/certs/ca.pem
[-rw-r--r-- rherna ] .docker/machine/certs/cert.pem
[-rw------- rherna ] .docker/machine/certs/key.pem
[-rw-r--r-- rherna ] .docker/machine/dnsmasq.conf
[-rw-r--r-- nobody ] .docker/machine/dnsmasq.pid
[drwx------ rherna ] .docker/machine/machines
[drwx------ rherna ] .docker/machine/machines/mytest
[-rw------- rherna ] .docker/machine/machines/mytest/config.json
[drwx------ rherna ] .docker/machine/machines/robstest
[-rw------- rherna ] .docker/machine/machines/robstest/config.json
6 directories, 9 files
from docker-machine-driver-bhyve.
swills
commented on May 29, 2024
Based on that and looking at the code, it seems like creating the iso file in ~/.docker/machine/cache/boot2docker.iso is failing and causing a panic, somehow? That's quite odd. Not sure how that's possible. Perhaps you can add some more debugging to bhyve/xhyve.go in copyIsoToMachineDir and see exactly how far it's getting?
from docker-machine-driver-bhyve.
sarcasticadmin
commented on May 29, 2024
Thanks for keeping this open @swills
So I did a bit more digging and updated CURRENT to r356261. And I was able to find out a few interesting things once I dove a bit deeper.
It looks like the issue is around netgraph as best as I can tell. With a clean system and just the dependencies installed, sudo configured and the code compiled. I get the following panic in the system while running docker-machine:
$ export PATH=/home/rherna/docker-machine-driver-bhyve:${PATH}
$ docker-machine --debug create -d bhyve robtest
Docker Machine Version: 0.16.2, build HEAD
Found binary path at /home/rherna/docker-machine-driver-bhyve/docker-machine-driver-bhyve
Launching plugin server for driver bhyve
Plugin server listening at address 127.0.0.1:27689
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /home/rherna/docker-machine-driver-bhyve/docker-machine-driver-bhyve
Launching plugin server for driver bhyve
Plugin server listening at address 127.0.0.1:49879
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
(robtest) Calling .GetMachineName
(robtest) Calling .DriverName
(robtest) Calling .GetCreateFlags
(robtest) Calling .SetConfigFromFlags
Creating CA: /home/rherna/.docker/machine/certs/ca.pem
Creating client certificate: /home/rherna/.docker/machine/certs/cert.pem
Running pre-create checks...
(robtest) Calling .PreCreateCheck
(robtest) DBG | Checking kmods
(robtest) DBG | kmod vmm is loaded
(robtest) DBG | kmod nmdm is loaded
(robtest) DBG | kmod ng_ether is loaded
(robtest) DBG | Checking IP forwarding
(robtest) DBG | IP forwarding not enabled, enabling
(robtest) DBG | EXEC: sudo sysctl net.inet.ip.forwarding=1
(robtest) DBG | STDOUT: net.inet.ip.forwarding: 0 -> 1
(robtest) DBG |
(robtest) DBG | STDERR:
(robtest) DBG | Checking interface em0
(robtest) DBG | Checking interface lo0
(robtest) DBG | Interface em0 has address 192.168.88.85 and looks like a good candidate
(robtest) DBG | Setting up 192.168.99.1/24 on bridge0, aliased to 192.168.88.85 on em0
(robtest) DBG | EXEC: sudo ifconfig bridge0 create
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ifconfig bridge0 192.168.99.1/24
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ifconfig bridge0 up
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ngctl mkpeer em0: nat lower in
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ngctl name em0:lower em0_NAT
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ngctl connect em0: em0_NAT: upper out
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ngctl msg em0_NAT: setdlt 1
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | EXEC: sudo ngctl msg em0_NAT: setaliasaddr 192.168.88.85
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) DBG | Starting DHCP Server
(robtest) DBG | Writing DHCP server config
(robtest) DBG | EXEC: sudo dnsmasq -i bridge0 -C /home/rherna/.docker/machine/dnsmasq.conf -x /home/rherna/.docker/machine/dnsmasq.pid -l /home/rherna/.docker/machine/bhyve.leases
(robtest) DBG | STDOUT:
(robtest) DBG | STDERR:
(robtest) Calling .GetConfigRaw
Creating machine...
(robtest) Calling .Create
(robtest) Image cache directory does not exist, creating it at /home/rherna/.docker/machine/cache...
(robtest) No default Boot2Docker ISO found locally, downloading the latest release...
(robtest) Latest release for github.com/boot2docker/boot2docker is v19.03.5
(robtest) Downloading /home/rherna/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v19.03.5/boot2docker.iso...
Unread portion of the kernel message buffer:
panic: ng_nat: ip_len != m_pkthdr.len
cpuid = 3
time = 1578139602
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe004b4bd690
vpanic() at vpanic+0x17e/frame 0xfffffe004b4bd6f0
panic() at panic+0x43/frame 0xfffffe004b4bd750
ng_nat_rcvdata() at ng_nat_rcvdata+0x3d5/frame 0xfffffe004b4bd7a0
ng_apply_item() at ng_apply_item+0xa3/frame 0xfffffe004b4bd820
ng_snd_item() at ng_snd_item+0x2b0/frame 0xfffffe004b4bd860
ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe004b4bd890
ether_nh_input() at ether_nh_input+0x24a/frame 0xfffffe004b4bd8f0
netisr_dispatch_src() at netisr_dispatch_src+0xb1/frame 0xfffffe004b4bd970
ether_input() at ether_input+0x9d/frame 0xfffffe004b4bd9d0
iflib_rxeof() at iflib_rxeof+0xbcd/frame 0xfffffe004b4bdae0
_task_fn_rx() at _task_fn_rx+0x7d/frame 0xfffffe004b4bdb20
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x155/frame 0xfffffe004b4bdb80
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe004b4bdbb0
fork_exit() at fork_exit+0x80/frame 0xfffffe004b4bdbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe004b4bdbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
From the debug output of docker-machine this seemed to be right around the time the interfaces were being created so the panic message makes sense.
What was also interesting was the ~/.docker wasnt being created this time around. I think this could have been due to the fact that my networking on the main interface em0 was intermittent when I was initially doing the testing and thus lead to the confusion around where the panic was coming from.
So I went ahead and just ran the ifconfig and ngctl commands from: https://github.com/swills/docker-machine-driver-bhyve/blob/5fd3fb0/bhyve/utils.go#L460-L498 to see if I could reproduce the error outside of docker-machine:
sudo ifconfig bridge0 create
sudo ifconfig bridge0 192.168.99.1/24
sudo ifconfig bridge0 up
sudo ngctl mkpeer em0: nat lower in
sudo ngctl name em0:lower em0_NAT
sudo ngctl connect em0: em0_NAT: upper out
sudo ngctl msg em0_NAT: setdlt 1
# Where it breaks
sudo ngctl msg em0_NAT: setaliasaddr 192.168.88.85
As soon as I run sudo ngctl msg em0_NAT: setaliasaddr 192.168.88.85 the system panics.
Then looking at the docker-machine bhyve driver a bit more. It looks like if the interface bridge0 is already created then itll skip the interface setup entirely: https://github.com/swills/docker-machine-driver-bhyve/blob/5fd3fb0/bhyve/utils.go#L424-L427 So I went and just created bridge0 with a network:
sudo ifconfig bridge0 create
sudo ifconfig bridge0 192.168.99.1/24
sudo ifconfig bridge0 up
And this worked! The docker-machine was created and running (albeit without outbound NAT but that was expected)
There was also an example I found from man ng_nat which parallels how the existing code is setting up the docker-machine interfaces (this also causes a panic):
The ng_nat node can also be attached directly to the physical interface
via ng_ether(4) node in the graph. In the following example, we perform
masquerading on a Ethernet interface connected to a public network.
ifconfig igb0 inet x.y.8.35 netmask 0xfffff000
route add default x.y.0.1
/usr/sbin/ngctl -f- <<-SEQ
mkpeer igb0: nat lower in
name igb0:lower igb0_NAT
connect igb0: igb0_NAT: upper out
msg igb0_NAT: setdlt 1
msg igb0_NAT: setaliasaddr x.y.8.35
SEQ
I opened up an issue on the bugtracker for FreeBSD: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243096 since this manpage example seems to be problematic.
Im not really sure how to configure the netgraph ng_nat interface properly and couldnt find many examples. Any light you can shed on the matter would be great!
from docker-machine-driver-bhyve.
swills
commented on May 29, 2024
Nah, can't comment on that much. The man page example was how I figured out how to use ng_nat myself.
from docker-machine-driver-bhyve.
sarcasticadmin
commented on May 29, 2024
I went ahead an compile a kernel with the following two patches in: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243096
The docker-machine now works as intended:
$ export PATH=$(pwd):${PATH}
$ docker-machine --debug create -d bhyve robtest
...truncated logs
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
robtest * bhyve Running tcp://192.168.99.209:2376 v19.03.5
$ eval $(docker-machine env robtest)
$ docker run --rm hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
SIDE NOTE: Even though the panic is fixed and the docker-machine plugin is working, Im still not quite sure how ng_nat is doing its magic with bridge0. Attached is my existing graphviz of netgraph for completeness and for those curious:
Generated the above graph with
sudo ngctl dot | dot -Tsvg > netgraph.svg
@swills Ill close this out once this lands into CURRENT if thats ok with you
from docker-machine-driver-bhyve.
swills
commented on May 29, 2024
Sounds good, thanks.
from docker-machine-driver-bhyve.
swills
commented on May 29, 2024
BTW, as far as "how ng_nat is doing it's magic", the only nodes in that graph that matter are em0 and em0_NAT. Basically think of netgraph as a separate layer of things outside the normal networking. Packets go in/out on em0, get translated by em0_NAT (hence it's netgraph layer connections), and then out to the upper layers, where they're consumed by bridge0 and then tap0 via the non-netgraph networking layers/pieces. (Some nodes exist in both layers, connecting the two layers, hence why you see em0, tap0 and bridge0 in both the ngctl dot output and the ifconfig output.) Hope that helps, I know netgraph is a bit confusing, it took me a long time to get it too. :)
from docker-machine-driver-bhyve.
sarcasticadmin
commented on May 29, 2024
Thanks for the detailed explanation @swills that definitely helps my understanding. Im glad Im not the only one that was confused 😆
from docker-machine-driver-bhyve.
Related Issues (1)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-machine-driver-bhyve.