shoenig / nomad-pledge-driver Goto Github PK

Nomad task driver capable of blocking unwanted syscall and filesystem access. Based on the pledge utility for Linux by Justine Tunney

License: Mozilla Public License 2.0

HCL 11.44% Go 87.55% Makefile 1.00%

golang linux nomad pledge task-driver isolation sandbox unveil

nomad-pledge-driver's Introduction

nomad-pledge-driver

nomad-pledge-driver is a Nomad task driver based on the pledge utility for Linux by Justine Tunney.

Security through SECCOMP sorcery

Features

Sandbox applications by restricting syscalls they are able to make (via promises)
Sandbox applications by allow-listing filepaths they are allowed to access (via unveil)
Sandbox applications by restricting resources using modern Linux cgroups (via cgroups v2)
Sandbox applications by namespace isolation using Linux namespaces (via nsenter and unshare)

Use cases

The nomad-pledge-driver is intended as a replacement for raw_exec. Sometimes there are those management tasks that just need to run as root and directly access the filesystem or perform privileged operations. While raw_exec provides no isolation, the pledge driver uses Landlock to restrict the files or directories the task is allowed to access. Specific groups of system calls are allow-listed, greatly reducing the attack surface of a mis- configured or compromised task.

Compatability

Use version 0.3 with Nomad 1.7 and higher
Use version 0.2 for Nomad 1.6 and below

Examples

The example below uses curl to fetch example.com, with the minimal set of promises to make a request.

More complex examples in the hack directory.

job "curl" {
  type        = "batch"

  group "group" {
    task "curl" {
      driver = "pledge"
      config {
        command  = "curl"
        args     = ["example.com"]
        promises = "stdio rpath inet dns sendfd"
        unveil   = ["r:${NOMAD_TASK_DIR}"]
      }
    }
  }
}

Building

The nomad-pledge-driver plugin is written in Go. It can be built using the normal Go toolchain steps, but the Makefile contains a dev target to make things easy. The compiled binary will appear in the output/ directory.

make dev

Installing

The plugin should be placed in the plugin_dir configured by the Nomad agent, per Nomad's documentation.

You'll also need the pledge executable (1.8 or higher) that powers the plugin sandboxing. Download the pledge executable from https://justine.lol/pledge/ and install it somewhere. The plugin configuration lets you specify where the path to the pledge executable.

sudo mkdir -p /opt/bin
curl -L -o /opt/bin/pledge-1.8.com https://justine.lol/pledge/pledge-1.8.com

👉 optional It is very convenient to bless the pledge executable with the cap_net_bind_service Linux capability. This will enable Nomad tasks using the pledge driver to bind to privileged ports (e.g. below 1024).

sudo setcap cap_net_bind_service+eip /opt/bin/pledge-1.8.com

The plugin will expose the driver.pledge.cap.net_bind attribute indicating whether the cap_net_bind_service capability has been set on the pledge-1.x.com executable.

Plugin Configuration

Currently there is only one configuration option for this plugin, which is to specify the path of the pledge executable.

plugin "nomad-pledge-driver" {
  config {
    pledge_executable = "/opt/bin/pledge-1.8.com"
  }
}

Note: in these examples the driver plugin is named pledge, and the utility executable is named pledge-1.8.com.

Task Configuration

Tasks need to specify which promises they require in order to run.

Tasks also need to unveil the filesystem paths needed to run.

For more information about which pledges are available and how this mechanism works, visit https://justine.lol/pledge/

If no user is specified for the task, the pledge plugin will use the user of the Nomad client by default. Like the raw_exec task driver, user cannot be set in hardened clusters according to the production guide.

command: The executable to run
args: The arguments to pass to executable
promises: The set of promises needed for the executable to run
unveil: The set of system filepaths to allow the task to access, and with what permission
importance: One of lowest, low, normal, high, highest (default is normal)

# see hack/http.hcl for complete python http.server example
# note that bridge mode also works, see hack/bridge.hcl

task "task" {
  driver = "pledge"
  user   = "nobody"
  config {
    command    = "python3"
    args       = ["-m", "http.server", "${NOMAD_PORT_http}", "--directory", "${NOMAD_TASK_DIR}"]
    promises   = "stdio rpath inet"
    unveil     = ["r:/etc/mime.types", "r:${NOMAD_TASK_DIR}"]
    importance = "low"
  }

  template {
    destination = "local/index.html"
    data        = <<EOH
<!doctype html>
<html>
  <title>example</title>
  <body><p>Hello, friend!</p></body>
</html>
EOH
  }
}

Troubleshooting

For help getting the plugin to work, see the TROUBLESHOOT doc. Otherwise feel free to file an issue!

Contributing

The nomad-pledge-driver plugin is currently under active development - anything may change at a moments notice!

hacking

The included Makefile includes helpful targets for hacking on the pledge plugin.

To simply compile, run make dev. The output will go into /tmp/plugins.

To start Nomad with the plugin, run make run. Under the hood this is using the hack/client.hcl Client config file, along with -dev mode defaults. You should be able to run jobs making use of pledge driver when launching Nomad this way.

There are example jobs in the hack/ directory.

License

The pledge task driver plugin is made open source under the MPL-2.0 license.

nomad-pledge-driver's People

Contributors

Stargazers

Watchers

Forkers

isgasho liunix61

nomad-pledge-driver's Issues

support for host PID/IPC namespacing

Similar to how the exec driver exposes pid_mode and ipc_mode so that folks can disable PID or IPC namespacing, so should the pledge driver.

bug: need to correctly set memory/memory_max from task resources

We aren't actually setting the memory limit(s) anywhere. Like the official drivers, if memory is set but memory_max is not, memory becomes a hard limit (set in the memory.max group interface).

If memory_max is set (which requires enabling oversubscription on the scheduler), memory_max becomes the hard limit (set in memory.max), and memory becomes the soft limit (set in memory.low).

At least, that's how podman works, and it seems fine. Would be interesting if we could do something with the memory.high interface which puts the cgroup under heavy reclaim pressure but doesn't trigger oom like .max.

e2e: flaky cases when trying to read from logs

I suspect the e2e flakiness is from not properly waiting for an allocation to become "complete" for batch jobs, or the deployment to be "complete" for service jobs, before reading the logs. In which case there are no longs, and the tests fail with empty strings.

handle new cgroup layout for nomad 1.7

Nomad 1.7 changes the way cgroups are managed, need to account for the new layout on the filesystem

idea: fingerprint unshare and nsenter paths

We should detect if unshare and/or nsenter are not finadable on $PATH, and set the plugin status to unhealthy if they are not found.

idea: make use of procctl() to establish a supervisor process

https://unix.stackexchange.com/questions/149319/new-parent-process-when-the-parent-process-dies/177361#177361

Nomad's reference task drivers spawn a go-plugin based "executor" supervisor process to which the driver plugin can re-attach on Client agent restarts. We can probably do something more clever here.

bubble error message when cgroup descriptor cannot be read

"failed to open cgroup for descriptor" is currently not exposing the underlying error for not being able to read the cgroup

idea: use CLONE_NEWCGROUP to fix racey cgroup assignment

Instead of using a supervisor like the comment suggests, we should be able to just fork/exec the subprocess into
the cgroup created for it. Looks like this will be available in an upcoming version of Go

golang/go#51246
golang/go@3204e62

// Ideally we would fork a trusted helper, enter the cgroup ourselves, then
// exec into the user subprocess. This is fine for now.
return e.isolate()

idea: do the assimilate step to avoid needing bash indirection

It might be possible to have the plugin do an "assimilate" step, where pledge generates arch-specific sandbox.so / .ape binaries. In this case we might no longer need to do the bash -c indirection to get around Go complaining about executable formats.

idea: set cap_net_bind_service on pledge binary

It helps to have cap_net_bind_service+eip set on the pledge helper executable, we can probably just have the plugin do that on startup. Or maybe just set a plugin attribute with the detected status.

set cpu bandwidth

Given the task config cpu or cores value, we should set the cpu.max cgroup to match the configuration. This task driver always enforces cpu resource limits (i.e. CFS).

use TMPDIR instead of HOME

The pledge binary searches TMPDIR before using HOME as a fallback; the dance we do with ensuring a HOME directory actually exists (i.e. for service users) should be unnecessary, as we can set TMPDIR to something reasonable ourselves.

idea: support for network bridge mode via pledge

Currently the pledge plugin really only works with network.mode = "host". Ideally we could make it work with network.mode = "bridge" as well. It's a bit complicated though due to how entering a network namespace works. In the other drivers (like exec, raw_exec, etc.) the driver launches a supervisor child process that watches over the actual user's Task process. When using bridge networking, that supervisor process takes on the responsibility of using unshare() to enter the allocation's network namespace. It is not possible to exec a process and have it directly use a given pre-existing namespace.

The pledge driver is designed to not need that intermediate supervisor process. The plugin spawns Task child processes and manages them directly, taking advantage of Linux features to reattach to orphan processes after a Nomad client / driver plugin restart. The problem is then we do not have an entrypoint for entering a namespace.

Except ... perhaps we could modify the pledge.com binary itself to support entering a [network] namespace. Doing so arguably fits in the purpose of the tool. Something like,

pledge.com -z "net:/var/run/netns/<name>"

If that feature exists, the plugin then simply instructs the pledge.com binary to enter the namespace created and managed by the Nomad client when the allocation is set to use network.mode = "bridge".

bug: using unveil when not supported should error somewhere

On my CentOS 9 box with 5.14.0-295.el9.x86_64 where unveil is not supported, submitting a job with an unveil configuration should probably fail somewhere, but it does not.