/build/compile.sh: line 24: cd: /app: Permission denied about buildstep HOT 31 CLOSED

That's definitely weird. I haven't run into that. Can anybody else
reproduce?

On Fri, Dec 19, 2014 at 10:45 AM, Nicolas [email protected] wrote:

Hi!

I get this error when building a Php app (Joomla CMS).
I've done some investigations and I really don't understand what is going
on.
It seems that after switching user with /usr/bin/setuidgid the user
doesn't have the permissions to read in /app

Here the result of my tests :

Before switching user (so as root) :

ls -hal /

remote: total 84K
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 .
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 ..
remote: -rwxr-xr-x 1 root root 0 Dec 19 16:20 .dockerenv
remote: -rwxr-xr-x 1 root root 0 Dec 19 16:20 .dockerinit
remote: drwxrwxrwx 32 u30655 u30655 4.0K Dec 19 16:20 app
remote: drwxr-xr-x 2 root root 4.0K Dec 17 17:40 bin
remote: drwxr-xr-x 2 root root 4.0K Apr 10 2014 boot
remote: drwxr-xr-x 3 root root 4.0K Dec 19 16:19 build
remote: drwxr-xr-x 3 u30655 u30655 4.0K Dec 19 13:55 cache
remote: drwxr-xr-x 5 root u30655 360 Dec 19 16:20 dev
remote: drwxr-xr-x 88 root root 4.0K Dec 19 16:20 etc
remote: drwxr-xr-x 2 root root 4.0K Apr 10 2014 home
remote: drwxr-xr-x 13 root root 4.0K Dec 17 17:40 lib
remote: drwxr-xr-x 2 root root 4.0K Dec 17 17:39 lib64
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:25 media
remote: drwxr-xr-x 2 root root 4.0K Apr 10 2014 mnt
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:25 opt
remote: dr-xr-xr-x 213 root root 0 Dec 19 16:20 proc
remote: drwx------ 2 root root 4.0K Nov 25 18:25 root
remote: drwxr-xr-x 7 root root 4.0K Dec 17 17:40 run
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:29 sbin
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:25 srv
remote: dr-xr-xr-x 13 root root 0 Dec 19 15:24 sys
remote: drwxrwxrwt 6 u30655 u30655 4.0K Dec 19 16:20 tmp
remote: drwxr-xr-x 16 root root 4.0K Nov 25 18:25 usr
remote: drwxr-xr-x 19 root root 4.0K Dec 19 16:20 var

ls -hal /app

remote: total 144K
remote: drwxrwxrwx 32 u30655 u30655 4.0K Dec 19 16:20 .
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 ..
remote: -rw-r--r-- 1 u30655 u30655 51 Dec 19 16:20 .env
remote: -rw-r--r-- 1 u30655 u30655 86 Dec 19 16:20 .gitignore
remote: -rw-r--r-- 1 u30655 u30655 3.1K Dec 19 16:20 .htaccess
remote: -rw-r--r-- 1 u30655 u30655 18K Dec 19 16:20 LICENSE.txt
remote: -rw-r--r-- 1 u30655 u30655 55 Dec 19 16:20 Procfile
remote: -rw-r--r-- 1 u30655 u30655 4.3K Dec 19 16:20 README.txt
remote: -rw-r--r-- 1 u30655 u30655 23 Dec 19 16:20 SCALE
remote: drwxr-xr-x 16 u30655 u30655 4.0K Dec 19 16:20 administrator
remote: drwxr-xr-x 2 u30655 u30655 4.0K Dec 19 16:20 bin
remote: drwxr-xr-x 2 u30655 u30655 4.0K Dec 19 16:20 cli
remote: drwxr-xr-x 32 u30655 u30655 4.0K Dec 19 16:20 components
remote: -rw-r--r-- 1 u30655 u30655 362 Dec 19 16:20 composer.json
remote: drwxr-xr-x 2 u30655 u30655 4.0K Dec 19 16:20 config
remote: -rw-r--r-- 1 u30655 u30655 2.5K Dec 19 16:20 configuration.php
remote: drwxr-xr-x 2 u30655 u30655 4.0K Dec 19 16:20 images
remote: drwxr-xr-x 2 u30655 u30655 4.0K Dec 19 16:20 includes
remote: -rw-r--r-- 1 u30655 u30655 1013 Dec 19 16:20 index.php
remote: drwxr-xr-x 6 u30655 u30655 4.0K Dec 19 16:20 jbox-sql
remote: -rw-r--r-- 1 u30655 u30655 1.9K Dec 19 16:20 joomla.xml
remote: drwxr-xr-x 8 u30655 u30655 4.0K Dec 19 16:20 language
remote: drwxr-xr-x 6 u30655 u30655 4.0K Dec 19 16:20 layouts
remote: drwxr-xr-x 24 u30655 u30655 4.0K Dec 19 16:20 libraries
remote: drwxr-xr-x 34 u30655 u30655 4.0K Dec 19 16:20 media
remote: drwxr-xr-x 54 u30655 u30655 4.0K Dec 19 16:20 modules
remote: drwxr-xr-x 26 u30655 u30655 4.0K Dec 19 16:20 plugins
remote: -rw-r--r-- 1 u30655 u30655 842 Dec 19 16:20 robots.txt
remote: -rw-r--r-- 1 u30655 u30655 108 Dec 19 16:20 tata
remote: drwxr-xr-x 8 u30655 u30655 4.0K Dec 19 16:20 templates
remote: -rw-r--r-- 1 u30655 u30655 1.4K Dec 19 16:20 toto

After user switch :

whoami :
remote: u30655

id :
remote: uid=30655(u30655) gid=30655(u30655) groups=30655(u30655)

ls -hal /

remote: total 84K
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 .
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 ..
remote: -rwxr-xr-x 1 root root 0 Dec 19 16:20 .dockerenv
remote: -rwxr-xr-x 1 root root 0 Dec 19 16:20 .dockerinit
remote: drwxrwxrwx 32 u30655 u30655 4.0K Dec 19 16:20 app
remote: drwxr-xr-x 2 root root 4.0K Dec 17 17:40 bin
remote: drwxr-xr-x 2 root root 4.0K Apr 10 2014 boot
remote: drwxr-xr-x 3 root root 4.0K Dec 19 16:19 build
remote: drwxr-xr-x 3 u30655 u30655 4.0K Dec 19 13:55 cache
remote: drwxr-xr-x 5 root u30655 360 Dec 19 16:20 dev
remote: drwxr-xr-x 88 root root 4.0K Dec 19 16:20 etc
remote: drwxr-xr-x 2 root root 4.0K Apr 10 2014 home
remote: drwxr-xr-x 13 root root 4.0K Dec 17 17:40 lib
remote: drwxr-xr-x 2 root root 4.0K Dec 17 17:39 lib64
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:25 media
remote: drwxr-xr-x 2 root root 4.0K Apr 10 2014 mnt
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:25 opt
remote: dr-xr-xr-x 214 root root 0 Dec 19 16:20 proc
remote: drwx------ 2 root root 4.0K Nov 25 18:25 root
remote: drwxr-xr-x 7 root root 4.0K Dec 17 17:40 run
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:29 sbin
remote: drwxr-xr-x 2 root root 4.0K Nov 25 18:25 srv
remote: dr-xr-xr-x 13 root root 0 Dec 19 15:24 sys
remote: drwxrwxrwt 6 u30655 u30655 4.0K Dec 19 16:20 tmp
remote: drwxr-xr-x 16 root root 4.0K Nov 25 18:25 usr
remote: drwxr-xr-x 19 root root 4.0K Dec 19 16:20 var

ls -hal /cache (owned by the generated user)

remote: total 12K
remote: drwxr-xr-x 3 u30655 u30655 4.0K Dec 19 13:55 .
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 ..
remote: drwxr-xr-x 3 u30655 u30655 4.0K Dec 19 14:01 php

ls -hal /tmp (owned by the generated user too)

remote: total 20K
remote: drwxrwxrwt 6 u30655 u30655 4.0K Dec 19 16:20 .
remote: drwxr-xr-x 51 root root 4.0K Dec 19 16:20 ..
remote: drwxr-xr-x 3 u30655 u30655 4.0K Dec 19 16:20 build
remote: drwxr-xr-x 26 u30655 u30655 4.0K Dec 19 16:20 buildpacks
remote: drwxr-xr-x 2 u30655 u30655 4.0K Dec 19 16:20 env

ls -hal /app

remote: ls: cannot open directory /app: Permission denied

What the hell is going on?
Thanks for your help!

—
Reply to this email directly or view it on GitHub
#127.

Jeff Lindsay
http://progrium.com

from buildstep.

Hrm. How are you invoking docker or is this a dokku deployment?

from buildstep.

What do you mean? Actually I'm invoking Docker on a Dokku deployment :)

from buildstep.

Cool. That's what I was looking for. Which version of dokku you are using?

from buildstep.

erf... a personal version written in Ruby

from buildstep.

Can you share the command you're using when you spin up docker to execute /build/compile.sh?

from buildstep.

Sure :

I'm using docker-api to talk with Docker :

image_name is : progrium/buildstep
command is : /build/builder
opts is :

{'Binds' => [ "#{cache_path}:/cache" ]}

docker_container = create(
  'Image' => image_name,
  'Cmd'   => [command]
)

# Launch command and stream output
docker_container.tap { |c|
  c.start(opts)
}.attach { |stream, chunk|
  yield chunk if block_given?
}

from buildstep.

and what might be in opts?

from buildstep.

I've updated the post.

from buildstep.

So my theory is there is a perms issue in/on the underlying /cache dir on the host OS.

from buildstep.

I don't get it... The problem is with /app not /cache.

from buildstep.

Oh crap. Yeah my bad. Just getting to my morning coffee...

from buildstep.

So, no other options are being passed to docker? Also, just so I can make sure I'm testing with the same env, what version of docker are you using? And which buildstep version?

from buildstep.

The latest :

Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

On Debian Wheezy with backported Kernel :
Linux services 3.16-0.bpo.2-amd64 #1 SMP Debian 3.16.3-2~bpo70+1 (2014-09-21) x86_64 GNU/Linux

And no, no other options.

from buildstep.

And which buildstep version?

The last one from Github.

from buildstep.

Hrm. I can't recreate this using dokku 0.3.10 and the latest buildstep image built from master.... I am, however on ubuntu 14.04. Not sure that is a big difference though. Does your setup work with a non-custom dokku? Also, just to be sure we're talking the same underlying fs, which docker filesystem module are you using? (aufs? btrfs?)

root@dokku:~/buildstep# uname -a
Linux dokku 3.16.0-28-generic #38-Ubuntu SMP Sat Dec 13 16:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root@dokku:~/buildstep# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:   trusty

from buildstep.

Docker host :

root@services:~# uname -a
Linux services 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt2-1~bpo70+1 (2014-12-08) x86_64 GNU/Linux

root@services:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.7 (wheezy)
Release:    7
Codename:   wheezy

I use aufs module.
By the way, Docker containers run with Debian Wheezy too.

from buildstep.

For sure. I was just pointing out a difference in our environments. As I said, not sure its a problematic one, though.

Some further troubleshooting steps as I'm unable to recreate the issue myself.

• Has this configuration (this app, this buildstep, and your custom dokku) ever deployed successfully?
• Is it possible to swap out the custom dokku for the standard one? Just to rule out any potential inconsistencies there.

from buildstep.

Has this configuration (this app, this buildstep, and your custom dokku) ever deployed successfully?

Without setuidgid yes, more than once.

Is it possible to swap out the custom dokku for the standard one? Just to rule out any potential inconsistencies there.

I can try.

from buildstep.

For clarification, when you say without setuidgid, do you mean you have a modified buildstep as well or just that you've use an earlier version of buildstep prior to setuidgid being added?

from buildstep.

do you mean you have a modified buildstep

yes but only to comment the line that calls setuidgid.

from buildstep.

@n-rodriguez can you post a docker info from the box you are seeing the problem. I suspect you are using AUFS as storage driver get bitten by a bug in AUFS. See my comment in #129 for a few more details.

from buildstep.

Containers: 11
Images: 184
Storage Driver: aufs
 Root Dir: /data/docker/aufs
 Dirs: 235
Execution Driver: native-0.2
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 6
Total Memory: 7.818 GiB
Name: services
ID: 4L3L:V2FB:UQ35:YEMB:DLRM:WWDF:3XFT:OLDU:KQGC:R657:7QEY:GJBB
WARNING: No memory limit support
WARNING: No swap limit support

from buildstep.

I suspect you are using AUFS as storage driver get bitten by a bug in AUFS. See my comment in #129 for a few more details.

It seems that it could be this one.

from buildstep.

@yabawock
We use aufs in dokku and can't repro this issue. Is there some other component at play here?

from buildstep.

AUFS is fine as long as you don't get layers with different permissions. Dokku with the latest buildstep image from DockerHub should be fine. Manual build from master branch as well, it's the same ...

directly calling /build/compile.sh could trigger the bug though

from buildstep.

My understanding was that @n-rodriguez was using dokku (albeit re-written in ruby) with the latest buildstep.

@n-rodriguez are you doing any sort of chown/chmod in the containers with your custom dokku?

from buildstep.

@n-rodriguez are you doing any sort of chown/chmod in the containers with your custom dokku?

Yes. Only once and it's a 644 chmod :

# Inject BUILD_ENV file if exists
env_file = File.join(container.repo_path, 'BUILD_ENV')
if File.exists?(env_file)
  logger.info "Adding environment variables to build environment", header: true
  copy_file_to_container(env_file, '/app/.env', '644')
end

from buildstep.

Any news/developments on this issue?

from buildstep.

I have to do more tests, I'll keep you posted.

from buildstep.

I eventually found it! It was a missing chmod ;)

In git_archive_all when you create the repository tarball, you do a chmod 755 of the temp directory... I missed this one.

Also this is a bit weird because if I remember well you do a chmod 755 of the app directory in builder script before calling setuidgid... Anyway, this is now working :)

Thank you all for you help and for those amazing tools :)

Edit :

Also this is a bit weird because if I remember well you do a chmod 755 of the app directory in builder script before calling setuidgid...

Nope, you don't. It was one of my tests (chmod 755 /app inside the container) but it doesn't work.

So the real issue was the missing chmod of the tarball :)

from buildstep.