This project has been archived, as it's now part of Kata Containers.
For issues, or pull-request, please, submit them directly to Kata Containers, following the Kata Containers' project guidelines
Virtual Machine Manager for Go (govmm) is a suite of packages that provide Go APIs for creating and managing virtual machines.
License: Apache License 2.0
This project has been archived, as it's now part of Kata Containers.
For issues, or pull-request, please, submit them directly to Kata Containers, following the Kata Containers' project guidelines
VhostUserDevices have no CCW device numbers
QEMU device structs (e.g. FSDevice
) carry a CCW device number. (CCW is a bus used on s390x/IBM Z.)
This is missing for VhostUserDevice
s. PR is prepared.
Add pvpanic
can let QEMU issue an GUEST_PANICKED
event to outside. And "dump-guest-memory" can get the guest memory dump if wanted.
All these two will help to debug/manage QEMU VMs.
In order to support more device configurations it would make sense to expose all the available device driver options either by adding them to the existing device structs or by providing a generic way to pass arbitrary device driver options.
Currently only supports TAP. Need to make it configurable so we can also support macvtap.
govmm is now part of the kata-containers
GitHub organisation, so update to reflect this.
See also: #141.
Govmm provides some proper knobs to use disable-modern
on virtio-pci
devices that are coldplugged, but didn't do the same for QMP.
We need to make sure that hotplug works great when being run in nested environment that requires virtio 0.9 instead of 1.0, hence we need to provide the caller to enable or disable disable-modern
.
https://travis-ci.org/kata-containers/runtime/jobs/461613820
• architecture:
Host: ppc64le
golang: ppc64le
Build: ppc64le
• golang:
go version go1.10.5 linux/ppc64le
• Summary:
destination install path (DESTDIR) : /
binary installation path (BINDIR) : /usr/local/bin
binaries to install :
- /usr/local/bin/kata-runtime
- /usr/local/bin/containerd-shim-kata-v2
- /usr/libexec/kata-containers/kata-netmon
- /usr/local/bin/data/kata-collect-data.sh
config to install (CONFIG) : cli/config/configuration.toml
install path (CONFIG_PATH) : /usr/share/defaults/kata-containers/configuration.toml
alternate config path (SYSCONFIG) : /etc/kata-containers/configuration.toml
hypervisor path (QEMUPATH) : /usr/bin/qemu-system-ppc64le
assets path (PKGDATADIR) : /usr/share/kata-containers
proxy+shim path (PKGLIBEXECDIR) : /usr/libexec/kata-containers
BUILD /home/travis/gopath/src/github.com/kata-containers/runtime/kata-runtime
# github.com/kata-containers/runtime/vendor/github.com/intel/govmm/qemu
../vendor/github.com/intel/govmm/qemu/qemu.go:103:6: undefined: isVirtioPCI
../vendor/github.com/intel/govmm/qemu/qemu.go:266:5: undefined: isVirtioPCI
../vendor/github.com/intel/govmm/qemu/qemu.go:358:5: undefined: isVirtioPCI
../vendor/github.com/intel/govmm/qemu/qemu.go:469:5: undefined: isVirtioPCI
../vendor/github.com/intel/govmm/qemu/qemu.go:489:16: netdev.Type.QemuDeviceParam undefined (type NetDeviceType has no field or method QemuDeviceParam)
../vendor/github.com/intel/govmm/qemu/qemu.go:493:74: netdev.Type.QemuDeviceParam undefined (type NetDeviceType has no field or method QemuDeviceParam)
../vendor/github.com/intel/govmm/qemu/qemu.go:516:5: undefined: isVirtioPCI
../vendor/github.com/intel/govmm/qemu/qemu.go:527:16: netdev.Type.QemuNetdevParam undefined (type NetDeviceType has no field or method QemuNetdevParam)
../vendor/github.com/intel/govmm/qemu/qemu.go:531:49: netdev.Type.QemuNetdevParam undefined (type NetDeviceType has no field or method QemuNetdevParam)
../vendor/github.com/intel/govmm/qemu/qemu.go:579:16: netdev.Type.QemuNetdevParam undefined (type NetDeviceType has no field or method QemuNetdevParam)
../vendor/github.com/intel/govmm/qemu/qemu.go:579:16: too many errors
Makefile:342: recipe for target '/home/travis/gopath/src/github.com/kata-containers/runtime/kata-runtime' failed
make: *** [/home/travis/gopath/src/github.com/kata-containers/runtime/kata-runtime] Error 2
The command "make" failed and exited with 2 during .
Your build has been stopped.
I have a distro Linux image that in order to be run with QEMU requires -nographic
flag. If I use it together with daemonize
, QEMU does not start and errors out saying that both flags cannot be used.
The problem is that by removing daemonize
, the cmd.Run()
hangs. I know that actually it does not hangs and it runs QEMU but it does not display anything because by design there is nothing attached to the cmd
standard output and standard error.
I've tried to run it only with daemonize
but I got the error:
ERRO[0098] Unable to launch /usr/bin/qemu-system-x86_64: exit status 1
ERRO[0098] Unable to init server: Could not connect: Connection refused
gtk initialization failed
I've also tried to pass -vga none
but it did not work.
In qemu options, there is -pidfile
where it saves its pid of main qemu process.
Sometimes we need this pid, and applying cgroups or prlimits for resources limitations.
I would like to add an option to Config struct
so that user can configure it before LaunchCustomQemu()
~
There are two projects that make use of govmm:
However, Ciao is no longer being maintained (it's archived), hence we can now move govmm into the Kata Containers organisation!
The Kata Containers GitHub organisation is the logical "parent" for govmm now that there is only a single user of govmm; it makes the code and PRs more discoverable. It also makes contributions simpler.
govmm is used by both the 1.x Kata runtime and the new 2.0 runtime.
govmm and Kata Containers both use the same Apache-2.0 licence.
Moving this repo from https://github.com/intel/govmm to https://github.com/kata-containers/govmm can be done immediately by changing a setting in the govmm repo.
After the move, https://github.com/intel/govmm will automatically redirect to https://github.com/kata-containers/govmm.
This means nothing will "break" - URLs can be updated in a "lazy" fashion.
The output govmm when utilized with Kata is pretty verbose in some often-called-functions.
Let's remove the info print @ https://github.com/kata-containers/govmm/blob/master/qemu/qmp.go#L270-L272
For many usecases such as using userspace drivers (e.g: DPDK) to drive devices through VFIO, the presence of a virtual IOMMU device is needed.
Adding this issue to track the work needed to add such support.
Current versions that travis tests are 1.8, 1.9 and tip. Should consider adding more recent versions (and dropping old ones?)
in dimm support check func, arm64 is excluded. but arm64 does support memory dimm in qemu.
isDimmSupported(config *Config) bool { switch runtime.GOARCH { case "amd64", "386", "ppc64le": if config != nil && config.Machine.Type == MachineTypeMicrovm { // microvm does not support NUMA return false }
The easy to spot actions that should be changed are:
Reference: kata-containers/runtime#1155
ExecuteBlockdevAdd
and ExecuteBlockdevAddWithCache
both appear to be intended to create block devices in the guest which backend onto a block device in the host. That seems to be the way that Kata always uses it. However blockdevAddBaseArgs
, used by both those functions always uses the file
driver, which is only intended for use with regular file backends.
Use of the file
driver for host block devices was deprecated in qemu-3.0, and has been removed entirely in qemu-6.0 (commit 8d17adf34f5). We should be using the host_device
driver instead.
I'm implementing a https://godoc.org/github.com/intel/govmm/qemu#QMPLog logger. The API description mentions a log level, but it remains unclear what the values are that QMP uses internally.
Phrased differently, if I have 5 as my programs log level and compare against that, will QMP be silent, verbose, or log in excruciating detail everything that it does?
Problem description see kata-containers/runtime#1363
The ContextID of VSOCK should be a uint64
not a uint32
. That provokes an endianess issue in the kata-runtime. See kata-containers/runtime#947. Qemu requests an uint64
:
$ qemu-system-x86_64 -device vhost-vsock-pci,help
[...]
vhost-vsock-pci.guest-cid=uint64
Here I need some help to enable "-pflash" in govmm. thanks!
In some arches like arm64, "-pflash" is needed as UEFI volume. Specially, two pflash volumes are used in arm64, of which one is static for the UEFI firmware and eother one to store variables. So it's better to define pflash object as []string in qemu.go. Accordingly, a helper to append pflash to "QemuParams" is also needed.
https://godoc.org/github.com/intel/govmm/qemu#LaunchCustomQemu says:
The function will block until the launched qemu process exits.
This is a bit misleading, because it sounds like one has to call the function in goroutine to get continue working with the QEMU instance while it is running. But that's not true, because the function will return once QEMU is done with the virtual machine startup thanks to --daemonize
.
Some programs using this vendor may have their own children reapers which has reaped the
qemu's process before cmd.Wait to reap it. For this case, the cmd.Run/cmd.Wait
will get an error contains substring of "no child processes", thus it's needed
to ignore this error and deal it as success.
NUMA support is available for Power (ppc64le) architecture. Since this is disabled in govmm, it prevents using virtiofs with Kata for ppc64le architecture.
I'm raising this issue to track the changes for enabling NUMA for ppc64le architecture
When using an architecture/setup that does not have DIMM support, memory backends are ignored. Such setups can actually use memory backends provided the memory backend is specified as a -machine
parameter (instead of a -numa
option). This is required for enabling virtio-fs support for s390x in Kata. I have prepared a PR.
There is a max_ports option for virtio serial device in qemu. Qemu enables ioeventfd by default for virtio-serial-pci, if max_ports is 31 by default, virtio-serial-bus will add 2+2+30*2=64 virtqueues.
It will cost a lot of time overhead during boot time. It will be better that govmm provides max_ports for user to tun their qemu usecase
VirtioBlock device driver should be "virtio-blk-pci" instead of "virtio-blk".
MacVTap seems not set up properly.
When I create a NET device of type MacVtap in this way:
return qemu.NetDevice{
Type: qemu.MACVTAP,
ID: myobj.TAPIface.ID,
Driver: qemu.VirtioNetPCI,
IFName: myobj.TAPIface.Name,
MACAddress: myobj.MacAddr,
FDs: []*os.File{myobj.TAPIface.VMFds},
Script: "no",
DownScript: "no",
}
I get the following output:
-netdev tap,id=macvtap4,fds=3 -device driver=virtio-net-pci,netdev=macvtap4,mac=ce:ca:44:23:96:5f,disable-modern=false,mq=on,vectors=4,romfile=
Shouldn't I get also in the line the location of the tap device like 3<>/dev/tap5
? in this way seems the macvtap does not work
(replace this text with an explanation of what actually happened)
(replace this text with any extra information you think might be useful)
Which feature do you think can be improved?
virtio-mem-pci devices can be hotplugged only on the root bus.
How can it be improved?
Devices can't be hotplugged into the root bus of PCIe machines, eg Q35.
That means the feature can't be used on q35 machines.
Additional Information
Extend the API to optionally allow hotplugging memory devices to PCI bridges.
Enabling the support for Protected Execution Facility(PEF) which is he confidential computing technology on ppc64le.
iommu_platform is required for Secure Execution, but for some devices, GoVMM will omit it (my bad) or put an extra comma. PR is prepared.
Which feature do you think can be improved?
We should be able to Hotplug a block device to QEMU as read-only. The existing ExecuteBlockdevAdd
doesn't allow for specifying this option.
How can it be improved?
Add ability to specify the access permissions associated with a block device which is being added.
Quite a few open source projects have govmm as a dependency. It makes sense to include support for mainframes here to allow the users to have cross platform support
travis CI is failing with the following error:
Bad response status from coveralls: 422
{"message":"service_job_id (717167073) must be unique for Travis Jobs not supplying a Coveralls Repo Token","error":true}
The command "$GOPATH/bin/goveralls -v -service=travis-ci" exited with 1.
This PR broke s390x because they insert numa
and memory-backend-ram
as default behvior. However, this combination is not supported on s390x.
Travis CI is ridiculously slow, it takes too much time to run and sometimes it doesn't report the status, it gets stuck, see comment #134 (comment)
kata containers uses github actions to run some tests, they look stable and the CI run pretty fast, unfortunately only x86 is supported:
I propose to move x86 CI jobs to github actions and keep s390 jobs in travis
QEMU removed the -realtime
option, instead, -overcommit mem-lock=on|off
should be used. This is a trivial change to make in GoVMM, but the virtcontainers API 1.0 has the Realtime
bool.
Does this mean we need to update the virtcontainers API?
I'm working on the s390x support for kata
and govmm
. This issue tracks the modifications of the govmm code in order to be able to gradually add s390x support. These limitations or differences need to be taken into account:
mq
flag has no vectors flag. See commet in libvirtdisable-modern
option is not present for ccw devicesromfile
option not present for ccw devicesAdd support for the share-rw
flag for virtio-blk
.
check socket, thread and die, see #101 (comment)
should clarify https://github.com/intel/govmm/pull/24/files#diff-3f8ad3b51588f02283eda2d19c5ffd06R741
I assume that the default queues will be one if an argument is not specified to qemu. We should verify this and add a clarifying comment to the code, as it isn't clear in the qemu documentation.
@caoruidong can you please drive this?
In order to be able to hotplug and use the virtio-blk-ccw for kata. Each device needs to be identified by a device number. In this way, it's possible to determine which device number can be assigned to the hotplugged virtio-blk-ccw. This PR extends govmm to ba able to assign a devno to each device and it adds the virtio-blk-ccw device.
Main track issue for kata is kata-containers/runtime#1153
In order to support "sandbox" feature on qemu, I need to update the govmm code.
Param "-sandbox on" in qemu can introduce another protect layer on the host, to make the secure container more secure.
The default option is disable because this feature may introduce some performance cost, even though user can enable /proc/sys/net/core/bpf_jit_enable to reduce the impact.
Please refer to kata-containers/kata-containers#2266 for more info.
I wonder if vfio device can add x-pci-vendor-id and x-pci-device-id parameter pass to qemu.
Since some vendor id like 1ded can not be identified by virtio-pci driver, so I have to determine a specified vendor id pass to qemu
I was curious what blocks this package from compiling, on Windows.
$ go build -i -v -x
WORK=C:\Users\Worker\AppData\Local\Temp\go-build762776886
/C/Users/Worker/Desktop/govmm/qemu
mkdir -p $WORK\b001
cat >$WORK\b001\importcfg << 'EOF' # internalimport config
packagefile bufio=c:\go\pkg\windows_amd64\bufio.a
packagefile bytes=c:\go\pkg\windows_amd64\bytes.a
packagefile container/list=c:\go\pkg\windows_amd64\container\list.a
packagefile context=c:\go\pkg\windows_amd64\context.a
packagefile encoding/json=c:\go\pkg\windows_amd64\encoding\json.a
packagefile errors=c:\go\pkg\windows_amd64\errors.a
packagefile fmt=c:\go\pkg\windows_amd64\fmt.a
packagefile io=c:\go\pkg\windows_amd64\io.a
packagefile io/ioutil=c:\go\pkg\windows_amd64\io\ioutil.a
packagefile log=c:\go\pkg\windows_amd64\log.a
packagefile net=c:\go\pkg\windows_amd64\net.a
packagefile os=c:\go\pkg\windows_amd64\os.a
packagefile os/exec=c:\go\pkg\windows_amd64\os\exec.a
packagefile path=c:\go\pkg\windows_amd64\path.a
packagefile runtime=c:\go\pkg\windows_amd64\runtime.a
packagefile strconv=c:\go\pkg\windows_amd64\strconv.a
packagefile strings=c:\go\pkg\windows_amd64\strings.a
packagefile syscall=c:\go\pkg\windows_amd64\syscall.a
packagefile time=c:\go\pkg\windows_amd64\time.a
EOF
cd C:\Users\Worker\Desktop\govmm\qemu
"c:\go\pkg\tool\windows_amd64\compile.exe" -o "$WORK\b001\pkg.a" -trimpath "$WORK\b001=>" -p /C/Users/Worker/Desktop/govmm/qemu -complete -buildid dOlQOtmVqMsEWPaTfptx/dOlQOtmVqMsEWPaTfptx -goversion go1.15.2 -D /C/Users/Worker/Desktop/govmm/qemu -importcfg "$WORK\b001\importcfg" -pack -c=4 "C:\Users\Worker\Desktop\govmm\qemu\image.go" "C:\Users\Worker\Desktop\govmm\qemu\qemu.go" "C:\Users\Worker\Desktop\govmm\qemu\qmp.go"/C/Users/Worker/Desktop/govmm/qemu
.\qmp.go:1517:9: undefined: syscall.UnixRights
So my environment is like follows:
$ go version
go version go1.15.2 windows/amd64
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Worker\AppData\Local\go-build
set GOENV=C:\Users\Worker\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\Worker\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Worker\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Worker\AppData\Local\Temp\go-build111068182=/tmp/go-build -gno-record-gcc-switches
$ bash --version
bash --version
GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
If I would like to pass the flag -serial stdio
, would be possible to do that using the Config
object or would I need to use LaunchCustomQemu
func?
Looking over this file (qemu/qemu.go
, editorial note), there are a lot of functions (88, 18 of which are QemuConfig()
ones ;) and each seems to handle building up these params differently (either appending with a comma, then joining with a null string; or appending without a separator, then joining with a comma).
Originally posted by @jodh-intel in #179 (review)
Secure Execution, also known as Protected Virtualization in QEMU, is a confidential computing technology for s390x (IBM Z & LinuxONE). Let's enable it in GoVMM! PR is prepared.
Can you please provide usage with simple example in golang.
To hotplug IBM Adjunct Processor (AP) devices into a VM, functionality besides the existing ExecutePCIVFIODeviceAdd
is required, since the device_add
command is a little different. See also Issue #491 in the Kata Containers project.
I have prepared a patch for this which I intend to create a PR for.
/cc @alicefr
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.