Unable to create Yarn cluster due to permissions about dask-gateway HOT 5 CLOSED

Apologies for the delayed response here, I've been out for the last week (conference, then vacation). Dask-Gateway is written to start clusters for multiple users - as such it needs security permissions for whatever the backing system is. Currently this is non-optional, as we don't want to encourage insecure behavior. See https://gateway.dask.org/install-hadoop.html for full hadoop installation instructions. In particular, you need proxy-user permissions for the account running dask-gateway.

If you can provide a good argument for why we should make proxy-user permissions optional, I may be willing to make this support optional. In this case, all clusters would be run as the same user account, regardless of who's logged in.

from dask-gateway.

No problem.

I'm pretty sure I have proxy-user permissions for both the dask and hadoop users. Currently I'm running the dask gateway as the hadoop user. Is there a need for it to specifically be running as the dask user? And does that error look like the error I would get if I did not actually have proxy user permissions? In particular, I did specify hadoop.proxyuser.{user}.hosts and hadoop.proxyuser.{user}.groups for both hadoop and dask

from dask-gateway.

The username that dask-gateway-server runs as doesn't matter, what matters is that the account has proxy-user permissions for all users submitting clusters. Since you're running dask-gateway-server as the hadoop user and the server is successfully proxying you as benschreck, then this is working fine.

The failure here appears to be that user benschreck doesn't have permission to write to the /user directory on HDFS. Do you have a home directory on HDFS? We rely on the Java libraries to infer your home directory, perhaps if it's missing it falls back to /user which you don't have permissions for. On application submission a small bundle is uploaded to HDFS and stored in ~/.skein, this upload is what's failing.

from dask-gateway.

Got it, that makes sense. I'll dig deeper on hdfs perms

from dask-gateway.

Closing as stale. Feel free to reopen if the problem persists.

from dask-gateway.