rdkcentral / dobby Goto Github PK

From Dobby code :

    // the following restores the old skool RT scheduling behaviour, and it's
    // needed to ensure that containered apps can start in RT scheduling mode
    if (!mUtilities->writeTextFile("/proc/sys/kernel/sched_rt_runtime_us", "-1\n",
                                   O_TRUNC | O_WRONLY, 0))
    {
        AI_LOG_FATAL("failed to write to sched_rt_runtime_us file, you may have"
                     " issues starting containers");
    }

According to this RedHat and other documentation, this is a dangerous thing to do.
Setting the value to -1 means that real-time tasks may use up to 100% of CPU times. This is only adequate when the real-time tasks are well engineered and have no obvious caveats such as unbounded polling loops.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/real_time_throttling
https://access.redhat.com/solutions/1604133

Additional note:
In our RDK platforms, we have an AppArmor profile restricting write access inside /proc/sys/kernel/ except for shm* and core*.

Description

Describe the bug
When trying to start a image container, DobbyTool -vvv start, in the logs I'm receiving this message:

exec container process (missing dynamic library?) /usr/libexec/DobbyInit: No such file or directory

Looking in the host, the DobbyInit exists:

-rwxr-xr-x 1 root root 10052 Jan 31 12:28 /usr/libexec/DobbyInit

But inside the image, there is also a DobbyInit in the rootfs with 0 bytes:

-rwxrwxrwx 1 1000 1000 0 Feb 6 2024 rootfs/usr/libexec/DobbyInit

If including the Dobby log, please ensure Dobby has been started with the verbose flag (-v)

2024-02-01T20:41:12.653378Z: exec container process (missing dynamic library?) /usr/libexec/DobbyInit: No such file or directory
0000118804.415413 MIL: < M:Main.cpp F:main L:480 > Running hook poststop for container 'matter'
0000118804.416353 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'ipc' from '/usr/lib/plugins/dobby/libIpcPlugin.so.1'
0000118804.416605 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'logging' from '/usr/lib/plugins/dobby/libLoggingPlugin.so.1'
0000118804.416742 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'minidump' from '/usr/lib/plugins/dobby/libMinidumpPlugin.so.1'
0000118804.419458 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'networking' from '/usr/lib/plugins/dobby/libNetworkingPlugin.so.1'
0000118804.419625 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'oomcrash' from '/usr/lib/plugins/dobby/libOOMCrashPlugin.so.1'
0000118804.419788 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'storage' from '/usr/lib/plugins/dobby/libStoragePlugin.so.1'
0000118804.422024 NFO: < M:DobbyRdkPluginManager.cpp F:loadPlugins L:333 > Loaded plugin 'thunder' from '/usr/lib/plugins/dobby/libThunderPlugin.so.1'
0000118804.422106 NFO: < M:DobbyRdkPluginManager.cpp F:runPlugins L:912 > Running networking plugin
0000118804.422997 NFO: < M:DobbyRdkPluginManager.cpp F:runPlugins L:938 > Plugin networking has postStop hook run successfully
0000118804.423047 NFO: < M:DobbyRdkPluginManager.cpp F:runPlugins L:906 > Plugin logging has nothing to do at postStop
0000118804.423294 NFO: < M:Main.cpp F:main L:508 > Hook poststop completed

Platform/Environment

• What device are you running Dobby on:
• Crun version: crun version 1.7.2 commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
• Dobby version: Version: 3.6.3-e747921419aca764d3f758be2a3b2f39f3a024a0

Description

In L1 test, we have faced issue in pauseContainer use case.

Facing issue in cleanupContainersShutdown() API. If the DobbyContainer state is not DobbyContainer::State::Running, the while loop keeps looping because the mContainer's data is not erased, and it is not moving to next iterator.

Solution: we need to check for “paused” state in addition to the “running” in the while loop.

Expected Behaviour

cleanupContainersShutdown() function should erase the container and dobbymanger object should delete.

Actual Behaviour

the while loop keeps looping because the mContainer's data is not erased, and it is not moving to next iterator.

Steps to Reproduce

1. Create and start container(startContainerFromSpec)
2. pause the container (pauseContainer)
3. Delete the dobby manager object, so cleanupContainersShutdown() is calling from destructor.

Platform/Environment

Run the L1 test in github workflow

Description

Using a Raspberry Pi 3 RDK build, the networking plugin fails to bring up the dobby0 bridge device.

Visible with network types nat or none.

Workaround is to only use open network type to circumvent having to create a veth device.

Expected Behaviour

No error when launching container.

Actual Behaviour

The networking plugin fails, the container starts if the plugin is set to required.false, but the network set up does not work.

Logs:

0000000454.428183 ERR: < M:Netlink.cpp F:applyChangesToLink L:565 > failed to apply changes (10 - Operation not supported)

Steps to Reproduce

Start any container with networking type in the plugin set to none or nat.

Platform/Environment

• Raspberry Pi 3 Model B v1.2, RDK reference image
• Crun version: 0.14.1
• Dobby version: 3.0

Description

Using a Raspberry Pi 3 RDK build, the networking plugin fails to create a veth pair on container launch.

Visible with network types nat or none.

Workaround is to only use open network type to circumvent having to create a veth device.

Expected Behaviour

Container launches fine with no errors, creating a network of type none or nat.

Actual Behaviour

The logs show:

0000000571.171655 ERR: < M:Netlink.cpp F:createVeth L:1167 > failed to create veth pair ('veth0' : 'eth0') (12 - Object not found)
0000000571.171704 ERR: < M:NetworkSetup.cpp F:setupVeth L:715 > failed to create veth pair for container 'asd'
0000000571.171820 ERR: < M:NetworkingPlugin.cpp F:createRuntime L:209 > failed to setup virtual ethernet device
0000000571.171913 WRN: < M:DobbyRdkPluginManager.cpp F:runPlugins L:630 > Non-required plugin networking createRuntime hook has failed. Continuing running other plugins.

Container starts, but networking isn't working as expected, because the plugin didn't finalise setup.

Steps to Reproduce

Start any container with networking plugin type set as none or nat on the Raspberry Pi 3 Platco build.

Platform/Environment

• Raspberry Pi 3 Model B v1.2
• rdk-firebolt-mediaclient-image image, raspberrypi-cpc-mc machine, 2006_sprint branch.
• Crun version: 0.14.1
• Dobby version: 3.0 (master branch, commit 58ca105)

Description

In L1 test, Randomly test cases are failing

pthread_kill() is failing in stopRuncMonitorThread() which is calling from destructor.
Sometime runcMonitorThread() is starting late, with in the time, if dobbymanager object is deleted, pthread_kill() is failing because the thread is not yet started.

solution: Need to be add the semaphore handling to avoid this sync issue.

Expected Behaviour

pthread_kill() should not fail and destructor should happened without any issues

Actual Behaviour

pthread_kill() is failing and exit from the process

If including the Dobby log, please ensure Dobby has been started with the verbose flag (-v)

Steps to Reproduce

Run the L1 test in github workflow and randomly this issue will reproduce

Platform/Environment

• Run the L1 test in github workflow