`/etc/sudoers` lost on virtual machine restart about for-azure HOT 25 CLOSED

17.05 will be released in May 2017 on the Edge Channel. The Edge channel releases are monthly releases. 17.06 in June 2017 will be the next Stable channel release (containing features/fixes from the previous monthly edge releases).

from for-azure.

Same problem here.

I also noticed that installing any extra package via apk, a lot of other packages get purged, including sudo, rendering the machine unable to work with :(

EDIT: I've just saw that reported on #10 !

from for-azure.

Thanks for reporting this issue. This is happening because the bash package shipped has become out of date is getting purged which is also causing sudo to get purged. A workaround for now: before invoking any sudo apk add operations, invoke sudo apk add sudo

from for-azure.

@ddebroy I remember sudo not being the only package getting purged out... are you sure just re-adding sudo will prevent the other packages being purged?

from for-azure.

@vovimayhem with sudo present, you should be able to install any other package you need through sudo apk add .... May I ask what packages you are trying to apk add? Note that you log into to the SSH server hosted in the docker4x/agent-azure container running on each node and anything that you add simply gets installed within that container. Not in the underlying minimal Moby Linux host OS. The expected workflow is: users/admin will spin up containers/services using the Docker tools with the packages needed (rather than apk add ... stuff into the system containers like agent-azure)

from for-azure.

I was just installing htop when I started noticing all of this. Running htop as a container instead of as an installed package - Got it! (Pretty much like RancherOS)

I'll give it a try!

from for-azure.

So are there any plans to resolve this to update the Bash package and prevent this from being purged after a reboot? The fact that we can't run Sudo after a reboot (note that we aren't installing anything into the container - we use Sudo to run various bash scripts such as Docker stack creation), is a big inconvenience.

from for-azure.

Yes we will have a fix for this in our upcoming release (17.05). Sorry about the inconvenience.

from for-azure.

Ok great! I noticed that 17.05 is marked down as a pre-release on your the Github Docker releases page....any ideas when this will actually be released? I also noticed version 17.04 was actually released at the start of April, but then stayed in the Edge Channel for Docker for Azure.....will 17.05 go into the stable channel when this is released, or will this stay in the Edge Channel for a period of time?

from for-azure.

Any news on when this will be released in June?

from for-azure.

We had a change in direction where we are refactoring the way we run the Agent container as well as the base OS environment based on LinuxKit. So this work has been postponed for a bit and will be incorporated as part of the larger refactoring. Sorry about the delay and inconvenience.

If you do want to run something as root, today you will need to spin up an independent container with the necessary tools/utilities there. You can configure this container to be privileged as well as bind mount arbitrary directories or persist data using local or cloudstor volumes (e.g. docker run --privileged -v /usr/bin/docker:/usr/bin/docker -v /var/run/docker.sock:/var/run/docker.sock -ti alpine /bin/sh). Please note that the environment that you log into through SSH is also an Alpine based container and not the base OS.

from for-azure.

17.06 was released at the end of June. I've just deployed Docker for Azure as a test, but it still looks like the problem with Sudo being purged after a reboot is around.

from for-azure.

Can confirm that the problem persists in Docker for Azure, is there any version contemplated where the aforementioned refactoring will take place?

from for-azure.

@marakame we have implemented a fix that should be out in the next edge release.

from for-azure.

Understood, I will test it as soon as I can and will report back. Thanks!

from for-azure.

Problem still persists on the Azure template.

from for-azure.

+1 This is preventing us from adopting this as our defacto deployment for docker onto Azure... any update on when this will be fixed?

from for-azure.

I'm confused as to which image is supposed to have the fix as well.

Over at https://docs.docker.com/docker-for-azure/ both the stable and the edge template currently deploy the docker-ce:1.0.15 VM image with docker 17.09.0-ce. Except they have a different "channel" variable that makes it into some tags.

The list of images currently is

az vm image list -o table --publisher docker --all
Offer                     Publisher    Sku                   Urn                                                      Version
------------------------  -----------  --------------------  -------------------------------------------------------  ---------
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.10                        1.0.10
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.11                        1.0.11
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.13                        1.0.13
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.14                        1.0.14
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.15                        1.0.15
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.6                         1.0.6
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.7                         1.0.7
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.8                         1.0.8
docker-ce                 docker       docker-ce             docker:docker-ce:docker-ce:1.0.9                         1.0.9
docker-ce-edge            docker       docker-ce-edge        docker:docker-ce-edge:docker-ce-edge:1.0.3               1.0.3
docker-ce-edge            docker       docker-ce-edge        docker:docker-ce-edge:docker-ce-edge:1.0.4               1.0.4
docker-datacenter-custom  docker       docker_datacenter     docker:docker-datacenter-custom:docker_datacenter:1.0.0  1.0.0
docker-datacenter-custom  docker       docker_datacenter     docker:docker-datacenter-custom:docker_datacenter:1.0.1  1.0.1
docker-datacenter-custom  docker       docker_datacenter     docker:docker-datacenter-custom:docker_datacenter:1.0.2  1.0.2
docker-datacenter-custom  docker       docker_datacenter     docker:docker-datacenter-custom:docker_datacenter:1.0.3  1.0.3
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.0                         1.0.0
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.1                         1.0.1
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.2                         1.0.2
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.5                         1.0.5
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.7                         1.0.7
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.8                         1.0.8
docker-ee                 docker       docker-ee             docker:docker-ee:docker-ee:1.0.9                         1.0.9
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.12.18                 1.12.18
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.12.20                 1.12.20
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.13.1                  1.13.1
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.13.2                  1.13.2
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.13.3                  1.13.3
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.13.4                  1.13.4
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.13.6                  1.13.6
docker4azure              docker       docker4azure          docker:docker4azure:docker4azure:1.13.7                  1.13.7
docker4azure-cs           docker       docker4azure-cs-1_12  docker:docker4azure-cs:docker4azure-cs-1_12:1.0.5        1.0.5
docker4azure-cs           docker       docker4azure-cs-1_1x  docker:docker4azure-cs:docker4azure-cs-1_1x:1.0.0        1.0.0

from for-azure.

Ok, so I've gone back to Docker for Azure to carry out some further testing following numerous stable releases since last time round. The latest test that I tried was to scale down the number of manager nodes from 3 to 1 (I understand the requirement to run at least 3 in production) and this appears to have completely killed the swarm.

I now receive on the single manager node which exists in the swarm:

swarm-manager000000:~$ docker node ls Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.

So even if I scale back up to 3 managers in the VMSS, the same happens and I can't access any swarm orchestration functions.

What's even more worrying is that once I've scale up the VMSS manager nodes from 1 to 3 again, it's apparently that the newly added nodes haven't even been joined to the original swarm (they were just lone nodes). I managed to create a new individual swarm on each of these nodes (docker swarm init).

Perhaps another issue needs to be opened for this? I guess it is different to the 'sudoers' being purged issue.

from for-azure.

@Jsmiiii your issue is not related to the original issue mentioned around /etc/sudoers. Since this is more of a discussion, please open a thread in https://forums.docker.com/c/docker-for-azure

When you scaled down from 3 managers to 1, the managers most likely lost quorum as you killed a majority of the managers in your swarm. At the moment, Azure does not provide any hooks during this scale down process that can allow us to gracefully make the managers leave the swarm. As a result you end up with a swarm in a bad state if a majority of managers dies/scaled down at the same time.

from for-azure.

Just got hit by a random restart today (may have been related to this cpu vul)
Anyway the problem is still here, Server Version: 17.09.0-ce.

Mainly I use SSH to deploy stacks via docker compose
The default version of docker compose is:

docker-compose version 1.15.0, build e12f3b9

Which doesnt allow some of the newer stack definition formats.
Previously I could install the latest docker compose by:

sudo curl -L https://github.com/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod 777 /usr/local/bin/docker-compose
docker-compose --version

Now that doesnt work:

curl: (23) Failed writing body (0 != 16360)

Converting to non-sudo:

mkdir -p /home/docker/bin
curl -L https://github.com/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` -o /home/docker/bin/docker-compose
chmod 777 /home/docker/bin/docker-compose
export PATH=$HOME/bin:$PATH
docker-compose --version

Which seems to work:

docker-compose version 1.17.1, build 6d101fb

from for-azure.

@djeeg The latest deployment should come with docker-compose installed out of the box. You could also check that the agent has the proper sudoer file mounted.

from for-azure.

Rebuilding my swarm, docker-compose still quite old

swarm-manager000000:~$ docker --version
Docker version 17.12.0-ce, build c97c6d6

swarm-manager000000:~$ docker-compose --version
docker-compose version 1.15.0, build e12f3b9

(which doesnt support compose format v3.4)
https://github.com/docker/compose/releases/tag/1.15.0

from for-azure.

@djeeg thanks for the ping - I've bumped it for the release of 17.12.1

from for-azure.

This problem is still present on all our nodes (currently 18.03.0-ce) and caused some troubleshooting obstacles. If you, like me, need to do some troubleshooting on your host here's a workaround you can use to restore sudo.

First create a ./sudoers file with the desired policy on your workstation. For example:

# See the man page for details on how to write a sudoers file.
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# User privilege specification
root    ALL=(ALL:ALL) ALL
docker    ALL=(ALL:ALL) NOPASSWD:ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:
#includedir /etc/sudoers.d

Next here is a ./restore-sudo.sh script to write it.

#!/bin/bash

HOST=${1:-"docker-host.example.com"}
PORT=${2:-"22"}
SSH="ssh docker@$HOST -p $PORT"

echo "Checking /etc/sudoers file exists..."
SUDOERS_EXISTS=`$SSH "[ -f /etc/sudoers ] && echo \"exists\""`

if [ -z "$SUDOERS_EXISTS" ]; then
	echo "It does not, writing using docker..."
	cat ./sudoers | $SSH "cat | docker run --rm -i -v /etc:/host-etc busybox sh -c \"cat > /host-etc/sudoers\""

	echo "Setting permissions using sudo..."
	$SSH "sudo chown root:root /etc/sudoers && sudo chmod 440 /etc/sudoers"

	echo "Done restoring sudo."
else
	echo "/etc/sudoers already exists."
	echo "Done restoring sudo."
fi

It assumes you will have SSH access to the machine on the docker user.
Writes the /etc/sudoers file using a docker volume mount to obtain write access.
Now that sudo is available, use that to set the correct permissions on the file.

from for-azure.