edwarnicke / govpp Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
This repo utilizes aggressively a number of multi-stage Docker tricks that are worth explaining, because they can be used broadly
We are doing three things in the Dockerfile, denoted by their named stage:
Ultra small vpp images are built via three Docker stages:
The reason for these stages is to cauterize the bloat from each activity.
Building vpp means bloating an image
up with a bunch of build dependencies, build artifacts, etc. Building vpp installation on top of that would
lead to a multi-GB image, which is undesirable. So we isolate that work in the 'vppbuild' stage.
Installing vpp means copying the resulting *.deb packages from the 'vppbuild' stage and installing them. Unfortunately,
because you can't combine a COPY and RUN step, that results in image bloat from the *.deb files themselves, so we isolate
that in the 'vppinstall' image.
Finally, we utilize a trick to trim out the bloat when building the 'vpp' stage.
Lines 10 to 20 in f9c1af6
is a fairly standard Ubuntu oriented build. It results in a bunch of *.deb files.
Lines 22 to 30 in f9c1af6
uses
Line 24 in f9c1af6
to copy in the indexes in /var/lib/apt/lists/
that result from having run apt-get update
in the 'vppbuild' stage,
thus avoiding the cost of redownloading them.
Line 25 in f9c1af6
copies the *.deb files we wish to install from where they were built in 'vppbuild'
Line 27 in f9c1af6
installs the *.deb files:
-f
causes apt-get
to install any missing dependencies.-y
causes apt-get
to run in an unattended mode where the answer to the questions are y
--no-install-recommends
causes apt-get
to only install required (rather than recommended) dependencies to keep the image size downLine 28 in f9c1af6
removes the apt indexes from apt-get update
and
Line 29 in f9c1af6
removes the *.deb files
There is one problem with this. Because the image still has the layers from
Lines 24 to 25 in f9c1af6
the 'vppinstall' stage is still going to be bloated by that amount. We solve this in the 'vpp' stage
The 'vpp' stage is our final lean runnable. It uses a very simple but slick trick, which has a small caveat to it.
Lines 31 to 32 in f9c1af6
Simply starts from 'ubuntu:${UBUNTU_VERSION}' (thus reusing the layers form that standard image) and then
copies the entire '/' directory in from 'vppinstall' (which has removed the apt-get indexes and *.deb files).
Because docker COPY is generally smart enough to only copy in the changed files... the layer resulting from
Line 32 in f9c1af6
Resulting in something that looks like:
docker history ghcr.io/edwarnicke/govpp/vpp:v20.09 ──(Sun,Jan31)─┘
IMAGE CREATED CREATED BY SIZE COMMENT
8b4ea0febd25 42 minutes ago /bin/sh -c #(nop) COPY dir:b8f7abee062c48863… 96.2MB
<missing> 10 days ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 10 days ago /bin/sh -c mkdir -p /run/systemd && echo 'do… 7B
<missing> 10 days ago /bin/sh -c [ -z "$(apt-get indextargets)" ] 0B
<missing> 10 days ago /bin/sh -c set -xe && echo '#!/bin/sh' > /… 811B
<missing> 10 days ago /bin/sh -c #(nop) ADD file:2a90223d9f00d31e3… 72.9MB
The caveat is, there is a bug in docker that will, depending on the storage driver
you are using, copy over all the files, resulting in a much larger image. Fortunately, building in GitHub Actions does not
seem to hit this issue. Unfortunately, building in Docker for Mac does.
To fix the issue in Docker for Mac, follow the instructions for setting Docker Engine options
and set your storage driver to 'overlay'
{
"storage-driver": "overlay"
}
In go, we generally generate code using a //go:generate directive.
In the case of govpp, you need to use the 'binapi-generator' run against the json api files installed by the vpp debs in /usr/share/vpp/api/
.
We utilize Docker to make this easy by:
/usr/share/vpp/api/
from our 'vpp' stage and run's binapi, outputting to /gen//go:generate
line to gen.go to run the 'gen' stage to generate the code.Lines 41 to 46 in f9c1af6
Is a pretty standard 'go get to build' stage for 'binapi-generator'
Lines 48 to 53 in f9c1af6
Actually performs the generation if 'docker run'
Line 49 in f9c1af6
copies in the '/usr/share/vpp/api/*' json api files from the 'vpp' stage
Line 50 in f9c1af6
copies in the 'binapi-generator' from the 'binapi-generator' stage
Line 51 in f9c1af6
copies in the VPP_VERSION we had stashed in the 'vppbuild' stage.
Lines 52 to 53 in f9c1af6
sets the Workdir to /gen and runs the binapi-generator.
VPP_VERSION=$(cat /VPP_VERSION)
- sets the VPP_VERSION env binapi-generator needs from the stashed VPP_VERSION value
${PKGPREFIX+-import-prefix ${PKGPREFIX}}
- will output -import-prefix ${PKGPREFIX}
if the PKGPREFIX env variable is set, and nothing otherwise.
//go:generate
directiveLine 19 in f9c1af6
uses 'docker run' to run the 'gen' stage and generate the code
bash bash -c "docker run ... "
is used to give us a shell to work with (because we are doing a lot of magic here)-e PKGPREFIX=$(go list)/binapi
- sets the env variable PKGPREFIX
in the docker container to the value of$(go list)/binapi
. $(go list)
is the value of the package in which the gen.go file resides.-v $(go list -f '{{ .Dir }}'):/gen
mounts $(go list -f '{{ .Dir }}')
from the host into /gen/
in the container.
$(go list -f '{{ .Dir }}')
outputs the directory the module containing gen.go is in.$(docker build . -q --build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git))
-q
- 'quiet' - outputs the id of the resulting image built--build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git))
- sets the build-arg GOVPP_VERSION$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git)
- outputs the version of git.fd.io/govpp.git in the go.mod file.In .github/workflows/ci.yaml we want to be able to tag and push images based
on the VPP_VERSION. Because we only wish to specify the VPP_VERSION once in the Dockerfile, we have a job:
Lines 5 to 8 in f9c1af6
that can be used to extract that version:
govpp/.github/workflows/ci.yaml
Line 92 in f9c1af6
Hello,
Do you have an upgrade strategy related to vpp?
What is the reason not to use latest release in govpp? (I am aware of possibility to change vpp version our-self)
The issue comes from a security vulnerability scan which report some risks for old third parties and old 3pp's of this 3pp 🥲
For example: mbedtls-devel package is a build dependency for currently used vpp version (on opensuse) and using mbedtls version 2.28.0. That contained a vulnerability which does not affect us but must be revealed and explained. The latest vpp version build does not depend on mbedtls.
Regards,
Laszlo
There is an error on ARM machine (Apple's M1) in k8s cluster when NSM's forwarder-vpp is trying to start.
I have tried to start it with and without Rosetta - in both cases it gives the same error.
Here's a log from the forwarder.
Jul 15 15:11:30.600�[36m [INFO] [cmd:/bin/forwarder] �[0mSetting env variable DLV_LISTEN_FORWARDER to a valid dlv '--listen' value will cause the dlv debugger to execute this binary and listen as directed.
Jul 15 15:11:30.602�[36m [INFO] [cmd:/bin/forwarder] �[0mthere are 9 phases which will be executed followed by a success message:
Jul 15 15:11:30.602�[36m [INFO] [cmd:/bin/forwarder] �[0mthe phases include:
Jul 15 15:11:30.602�[36m [INFO] [cmd:/bin/forwarder] �[0m1: get config from environment
Jul 15 15:11:30.602�[36m [INFO] [cmd:/bin/forwarder] �[0m2: run vpp and get a connection to it
Jul 15 15:11:30.602�[36m [INFO] [cmd:/bin/forwarder] �[0m3: get SR-IOV config from file
Jul 15 15:11:30.602�[36m [INFO] [cmd:/bin/forwarder] �[0m4: init pools
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0m5: start device plugin server
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0m6: retrieve spiffe svid
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0m7: create xconnect network service endpoint
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0m8: create grpc server and register xconnect
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0m9: register xconnectns with the registry
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0ma final success message with start time duration
Jul 15 15:11:30.603�[36m [INFO] [cmd:/bin/forwarder] �[0mexecuting phase 1: get config from environment (time since start: 805.75µs)
This application is configured via the environment. The following environment
variables can be used:
KEY TYPE DEFAULT REQUIRED DESCRIPTION
NSM_NAME String forwarder Name of Endpoint
NSM_LABELS Comma-separated list of String:String pairs p2p:true Labels related to this forwarder-vpp instance
NSM_NSNAME String forwarder Name of Network Service to Register with Registry
NSM_CONNECT_TO URL unix:///connect.to.socket url to connect to
NSM_LISTEN_ON URL unix:///listen.on.socket url to listen on
NSM_MAX_TOKEN_LIFETIME Duration 10m maximum lifetime of tokens
NSM_LOG_LEVEL String INFO Log level
NSM_DIAL_TIMEOUT Duration 100ms Timeout for the dial the next endpoint
NSM_OPENTELEMETRYENDPOINT String otel-collector.observability.svc.cluster.local:4317 OpenTelemetry Collector Endpoint
NSM_TUNNEL_IP String IP to use for tunnels
NSM_VXLAN_PORT Unsigned Integer 0 VXLAN port to use
NSM_VPP_API_SOCKET String /var/run/vpp/external/vpp-api.sock filename of socket to connect to existing VPP instance. If empty a VPP instance is run in forwarder
NSM_VPP_INIT Func NONE type of VPP initialization. Must be NONE or AF_PACKET
NSM_RESOURCE_POLL_TIMEOUT Duration 30s device plugin polling timeout
NSM_DEVICE_PLUGIN_PATH String /var/lib/kubelet/device-plugins/ path to the device plugin directory
NSM_POD_RESOURCES_PATH String /var/lib/kubelet/pod-resources/ path to the pod resources directory
NSM_DEVICE_SELECTOR_FILE String config file for device name to label matching
NSM_SRIOV_CONFIG_FILE String PCI resources config path
NSM_PCI_DEVICES_PATH String /sys/bus/pci/devices path to the PCI devices directory
NSM_PCI_DRIVERS_PATH String /sys/bus/pci/drivers path to the PCI drivers directory
NSM_CGROUP_PATH String /host/sys/fs/cgroup/devices path to the host cgroup directory
NSM_VFIO_PATH String /host/dev/vfio path to the host VFIO directory
Jul 15 15:11:30.621�[36m [INFO] [cmd:/bin/forwarder] �[0mConfig: &config.Config{Name:"forwarder-vpp-9mgxg", Labels:map[string]string{"p2p":"true"}, NSName:"forwarder", ConnectTo:url.URL{Scheme:"unix", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/var/lib/networkservicemesh/nsm.io.sock", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}, ListenOn:url.URL{Scheme:"unix", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/listen.on.sock", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}, MaxTokenLifetime:600000000000, LogLevel:"TRACE", DialTimeout:100000000, OpenTelemetryEndpoint:"otel-collector.observability.svc.cluster.local:4317", TunnelIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xac, 0x13, 0x0, 0x3}, VxlanPort:0x0, VppAPISocket:"/var/run/vpp/external/vpp-api.sock", VppInit:vppinit.Func{f:(func(context.Context, api.Connection, net.IP) (net.IP, error))(0xc875e0)}, ResourcePollTimeout:30000000000, DevicePluginPath:"/var/lib/kubelet/device-plugins/", PodResourcesPath:"/var/lib/kubelet/pod-resources/", DeviceSelectorFile:"", SRIOVConfigFile:"", PCIDevicesPath:"/sys/bus/pci/devices", PCIDriversPath:"/sys/bus/pci/drivers", CgroupPath:"/host/sys/fs/cgroup/devices", VFIOPath:"/host/dev/vfio"}
Jul 15 15:11:30.622�[36m [INFO] [cmd:/bin/forwarder] [duration:18.244709ms] �[0mcompleted phase 1: get config from environment
Jul 15 15:11:30.622�[36m [INFO] [cmd:/bin/forwarder] �[0mexecuting phase 2: run vpp and get a connection to it (time since start: 19.806667ms)
Jul 15 15:11:30.623�[36m [INFO] �[0mConfiguration file: "/etc/vpp/helper/vpp.conf" not found, using defaults
Jul 15 15:11:30.633�[36m [INFO] [cmd:/bin/forwarder] �[0mlocal vpp is being used
Jul 15 15:11:30.634�[36m [INFO] [cmd:/bin/forwarder] [duration:11.589958ms] �[0mcompleted phase 2: run vpp and get a connection to it
Jul 15 15:11:30.634�[33m [WARN] [cmd:/bin/forwarder] �[0mskipping phases 3-5: no PCI resources config
Jul 15 15:11:30.634�[33m [WARN] [cmd:/bin/forwarder] �[0mSR-IOV is not enabled
Jul 15 15:11:30.634�[36m [INFO] [cmd:/bin/forwarder] �[0mexecuting phase 6: retrieving svid, check spire agent logs if this is the last line you see (time since start: 32.209ms)
Jul 15 15:11:30.898�[36m [INFO] �[0mSVID: "spiffe://example.org/ns/nsm-system/pod/forwarder-vpp-9mgxg"
Jul 15 15:11:30.906�[36m [INFO] [cmd:/bin/forwarder] [duration:271.382458ms] �[0mcompleted phase 6: retrieving svid
Jul 15 15:11:30.906�[36m [INFO] [cmd:/bin/forwarder] �[0mexecuting phase 7: create xconnect network service endpoint (time since start: 304.018959ms)
Jul 15 15:11:30.624�[36m [INFO] [cmd:vpp] �[0mvpp[3500]: clib_sysfs_prealloc_hugepages:262: pre-allocating 64 additional 2048K hugepages on numa node 0
Jul 15 15:11:30.624�[36m [INFO] [cmd:vpp] �[0mvpp[3500]: buffer: numa[0] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: failed to mmap 64 pages at 0x1000000000 fd 5 numa 0 flags 0x11: Invalid argument)
Jul 15 15:11:33.219�[37m [DEBU] �[0m/var/run/vpp/api.sock was created after 2.587428293s
Jul 15 15:11:33.330�[37m [DEBU] �[0msuccessfully connected to /var/run/vpp/api.sock after 110.855416ms and 1 attempts
panic: error: VPPApiError: System call error #1 (-11)
goroutine 1 [running]:
github.com/networkservicemesh/cmd-forwarder-vpp/internal/vppinit.Must(...)
/build/internal/vppinit/vppinit.go:68
main.main()
/build/main.go:239 +0x2f45
And the same log in file:
forwarder-vpp-9mgxg.log
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.