Tried doing this:

Under seLinuxOptions: you can specify a default, eg:

 seLinuxOptions:
   level: "s0:c123,c456"
   user: xxxx  #selinux username
   role: yyyy  #selinux role, not openshift roles
   type: zzz

I'm not sure what goes in xxxx and yyyy here.
but doing this:

# oc edit scc hostmount-anyuid

...
runAsUser:
  type: RunAsAny
seLinuxContext:
  seLinuxOptions:
    level: s0:c6,c1
    type: spc_t
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
...

doesn't work - fluentd doesn't start - AVCs like this:

type=AVC msg=audit(1457655809.603:929): avc:  denied  { entrypoint } for  pid=8879 comm="exe" path="/pod" dev="dm-1" ino=4718595 scontext=system_u:system_r:spc_t:s0:c1,c6 tcontext=system_u:object_r:svirt_sandbox_file_t:s0:c1,c6 tclass=file
        Was caused by:
                Missing type enforcement (TE) allow rule.

            You can use audit2allow to generate a loadable module to allow this access. 

from origin-aggregated-logging.

@rhatdan suggests this:

Save this file to /tmp

make -f /usr/share/selinux/devel/Makefile container_logger.pp
semodule -i container_logger.pp

docker run -ti -v /var/log:/var/log --security-opt label:type:container_logger_t fedora /bin/sh

How do I start the fluentd container with this argument?
Do I need to edit the scc hostmount-anyuid?

And you should be allowed to play with the logs in locked down mode.

If this works, I will write a blog on doing it.

Depending on what the app needs the policy could be a lot simpler, but I added full capabilities and full
networking.

container_logger.te

policy_module(container_logger, 1.5.0)

virt_sandbox_domain_template(container_logger)

logging_manage_all_logs(container_logger_t)

allow container_logger_t self:capability { chown dac_override fowner kill setgid setuid setpcap net_bind_service net_raw sys_chroot mknod audit_write setfcap };
dontaudit container_logger_t self:capability fsetid;
dontaudit container_logger_t self:capability2 block_suspend ;

kernel_read_network_state(container_logger_t)

allow container_logger_t self:capability { net_raw net_admin net_bind_service };

allow container_logger_t self:udp_socket create_socket_perms;
allow container_logger_t self:tcp_socket create_stream_socket_perms;
allow container_logger_t self:netlink_route_socket create_netlink_socket_perms;
allow container_logger_t self:packet_socket create_socket_perms;
allow container_logger_t self:socket create_socket_perms;
allow container_logger_t self:rawip_socket create_socket_perms;
allow container_logger_t self:netlink_kobject_uevent_socket create_socket_perms;

corenet_tcp_bind_generic_node(container_logger_t)
corenet_udp_bind_generic_node(container_logger_t)
corenet_raw_bind_generic_node(container_logger_t)
corenet_tcp_sendrecv_all_ports(container_logger_t)
corenet_udp_sendrecv_all_ports(container_logger_t)
corenet_udp_bind_all_ports(container_logger_t)
corenet_tcp_bind_all_ports(container_logger_t)
corenet_tcp_connect_all_ports(container_logger_t)

from origin-aggregated-logging.

to keep it tracked in the same place, this is the blog post written for this issue: https://www.projectatomic.io/blog/2016/03/selinux-and-docker-part-2/

from origin-aggregated-logging.

Unfortunately, the blog post is incomplete. To replicate the comment I just posted there:

Unfortunately, this isn't quite enough. It should be, as there's a docker_log_t type that inherits from var_log_t, and running logging_manage_all_logs(container_logger_t) gives us access to it. Unfortunately, Docker does not correctly set the log files to be of type docker_log_t, and they default to docker_var_lib_t. As a result, container_logger_t is blocked from accessing /var/lib/docker/containers/$ID/$ID-json.log. To work around this until Docker fixes the issue upstream, you can add the following line at the end of container_logger.te:

docker_read_lib_files(container_logger_t)

This makes things work in the interim, though it also provides wider access than would be ideal.

I'll be creating an issue upstream with Docker sometime tomorrow.

from origin-aggregated-logging.

opened bug https://bugzilla.redhat.com/show_bug.cgi?id=1322603

from origin-aggregated-logging.

Ultimately, we're probably going to need to use a different security context constraint specifically for the fluentd service account, as hostmount-anyuid requires its containers' SELinux context to be provided by Origin/Kubernetes, whereas we'll need to provide a custom one. Something like this should work:

allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: v1
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: 'container-logger provides all the features of the
      restricted SCC but allows host mounts and any UID by a pod, which runs as the
      container_logger_t SELinux type.  This is primarily used by the log aggregation
      system. WARNING: this SCC allows host file system access as any UID, including
      UID 0.  Grant with caution.'
  creationTimestamp: null
  name: container-logger
priority: null
requiredDropCapabilities: null
runAsUser:
  type: RunAsAny
seLinuxContext:
  seLinuxOptions:
    type: container_logger_t
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:logging:aggregated-logging-fluentd

Then we just need to make sure that the container_logger module is included with the Origin node install.

from origin-aggregated-logging.

This is, of course, dependent on either giving container_logger_t access to docker_var_lib_t files or having the issue with docker_log_t resolved in the relevant packages. I've made pull requests that fix them, and the ones relevant for Origin (on CentOS 7 and RHEL7, at least) are:

• projectatomic/docker#93
• containers/container-selinux#8

That being said, the former at least should not be merged until it is decided if its upstream counterpart (moby/moby#21673) will be merged.

from origin-aggregated-logging.

We're already providing custom roles for logging components, so adding a custom SCC is actually not much of a burden if that proves useful.

It seems like the main issue now is how to distribute the custom SELinux policy module. For RPM-based installs we can just add a policy RPM but we also have to consider containerized installs especially on Atomic where we can't add RPMs. Would it be fair to expect Ansible to install this module on all the nodes when configuring logging? At least until the upstream PRs mentioned here are shipped.

Or, as an alternative, does everything still work out if we just keep fluentd in the privileged SCC?

from origin-aggregated-logging.

@sosiouxme "Or, as an alternative, does everything still work out if we just keep fluentd in the privileged SCC?"

Yes, it works, it's just that "privileged" gives the fluentd container (and service account?) a lot more access than is needed.

from origin-aggregated-logging.

Yeah, it does. But that seems like a better experience for users until we can get the SELinux policy changes in a RHEL release.

from origin-aggregated-logging.

Sure, but as I stated above, your changes will have no effect on newly created containers. On on containers logs which have had restorecon run on them.

from origin-aggregated-logging.

That's where a new version of docker-selinux with containers/container-selinux#8 would come in, plus a fix to Docker so that new log files have the correct SELinux type to begin with. That work is being tracked in moby/moby#21673.

from origin-aggregated-logging.

The workaround of using privileged SCC for fluentd isn't solving the problem for me.

2016-04-06 08:45:37 +0000 [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos>
  2016-04-06 08:45:37 +0000 [error]: /opt/app-root/src/gems/fluentd-0.12.22/lib/fluent/plugin/in_tail.rb:97:in `initialize'
  2016-04-06 08:45:37 +0000 [error]: /opt/app-root/src/gems/fluentd-0.12.22/lib/fluent/plugin/in_tail.rb:97:in `open'
2016-04-06 08:45:37 +0000 [error]: /opt/app-root/src/gems/fluentd-0.12.22/lib/fluent/plugin/in_tail.rb:97:in `start'

workaround (didn't solve)
$ oc edit scc hostmount-anyuid

runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:openshift-infra:pv-recycler-controller

oc edit scc privileged

runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:openshift-infra:build-controller
- system:serviceaccount:management-infra:management-admin
- system:serviceaccount:default:router
- system:serviceaccount:default:registry
- system:serviceaccount:jenkins:default
- system:serviceaccount:logging:aggregated-logging-fluentd

Deleted the fluentd pods but stillt he same issue.
I recreated the logging project several times. The documentation is up to date about using privileged SCC for fluentd

from origin-aggregated-logging.

@lorenzvth7 please try it again. we recently merged a fix for this and pushed out a new deployer image. in addition to the SA being a member of the privileged SCC, the pod spec had to indicate that it's a privileged pod.

from origin-aggregated-logging.

@sosiouxme Thanks, it works!

from origin-aggregated-logging.