Giter VIP home page Giter VIP logo

jupyter-cloud-scoped-creds's Introduction

jupyter-cloud-scoped-creds

When users log into JupyterHub on Kubernetes they can be given cloud provider credentials via a service account role. These permissions give access to resources like S3 buckets. Rather than making a bucket public, you can give authenticated users temporary to upload data files to object stores from any computer!

The goal of this server extension is to expose an API endpoint via a Jupyter Server Extension that makes it easy for JupyterHub users to get temporary credentials:

https://HUBURL/user/YOURNAME/api/cloudcreds/aws

Returns something that looks like:

{
  "Credentials": {
    "AccessKeyId": "ASIAXXXXXXXXXXXXXXXXX",
    "SecretAccessKey": "XXXXXXXXXXXXXXXXXX",
    "SessionToken": "XXXXXXXX+ihPNZdQzj7aRtD080V42+TrZ/TtMsXAIgDNqz41KPfvsCYC/GZDJ9tB9aTUPt2ceXOuJKCg+ZP98q8gQIZhADGgw3ODMzODA4NTk1MjIiDOmodQHqEpP4IM9nKyrPBKW/8E2CdEOMN8jdk1yRRKC6RpZh70ADc9wkQpavHV6BSR+DpSyJciz2yHH2TNWCPmt3xsFldUp5R8/znla7fQDhFs+dsTlZ6zxvV86OxFDf5qc8yxaVkEport2F0dSdxwyMWh6bJWsSNcZy/YZY6HPQUU8BzNAY8uUybTzgwg7QFM+5p4l45tl+CejaJxUyu/xa95U5er9isivexcD5yGg8NfouTWvwvMeGbZdj2wRez3DCEeafiHfBAPHiTr1LIBtWvPkAbCEa38bfkRpSkxGaMBfjEbjbpoDKvQxXZLMWUjZWK53EliM7+ON8NCLFHAh8ggFw4y9KIYEyNmrnQ4OkZAFHMVAtCPyPs+61jtiGSwlSCyZNZJk1FYOOWIUrvnnAqrDYOacFBELSVGiDAEoCEQP6ePveGO+FfAiSZy5Zrv2mkHYfIq8hJzXOeCLeIg7gAxnxn64jrO6WP97TofLm9Nt7LHpho4R7xsGoTYsbwjmvDhN4HZCsvHnMc37oZJ0rKvLgTbb50cHLfJ8VJVqZZWBkFtxN14y7f3Y/GSN8Dm40n3jrSGRbEwrS8uI5db235hJRfw0L9FQ3TJg+6l/iSAtO4WmCO+C8MoTOmpxEwy5ETUnYYhf3ACnlKP3nIr7gV1M9BNewY/RuGvtquZZ1ZbHcDVhSl0gSTjJJ7e4jLVgsAHPsLq3s6p34r/aj3ah92DDChP9iF1sPSgMFXvBIqUtWaI5k/3kqoMic1QT55dhPdTEr+iQf+c4DvI4Wr967m9cdY6O2Ui8j+XNFaTCUkeyWBjqaAVMO/oSBOTMk8mPRqsRKmQzUiC8enQSkzMCr1/V9z4+pz5spjkFzEhXQro46vLvma4OyD2dAANuJ/NVFUmQxrbzfnGb3uxDh+V3g+ugdUOiFmhwDV5eCpaUxCnMOzRs/ieVpyzUljmtKDeTivP30IiJBkst7bqzQ/P+LcRu3eIhQgiAlqnkpyl3pxxk5Kt906DYLWFvr8gjMbI4=",
    "Expiration": "2022-07-22T20:45:48Z"
  },
  "SubjectFromWebIdentityToken": "system:serviceaccount:prod:pangeo",
  "AssumedRoleUser": {
    "AssumedRoleId": "XXXXXXXXXXXXXX:jupyterhub-user-scottyhq",
    "Arn": "arn:aws:sts::XXXXXXXXX:assumed-role/eksctl-pangeo-binder-addon-iamserviceaccount-Role1-1FIHHPOY6UU2A/jupyterhub-user-scottyhq"
  },
  "Provider": "arn:aws:iam::XXXXXXXXX:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/ACA8E0F49907CBD2E6E3388B0448A911",
  "Audience": "sts.amazonaws.com"
}

How does it work?

Right now this only works for HUBs on AWS K8s that assign a service role for a user. The API endpoint just runs aws sts assume-role-with-web-identity https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role-with-web-identity.html

If you want to get credentials from another machine you can first go to https://HUBURL/user/YOURNAME/hub/token to get a token, then run:

curl https://HUBURL/api/cloudcreds/aws?token=0f5bf5fa97fe4ba0bb623226f0b33206

If you output the JSON returned to a file like /tmp/irp-cred.txt you can run the following commands in a terminal to set your credentials (requires jq):

export AWS_REGION="us-west-2"
export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")"
export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")"
export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")"

Roadmap

Developed quickly with Yuvi Panda at SciPy 2022!

Goal was to be as simple as possible, so no configuration options currently and it only works with AWS! But should be easy to extend to other Cloud providers

Install

Install this in the Docker Image defining the JupyterHub environment for users

pip install git+https://github.com/scottyhq/jupyter-cloud-scoped-creds.git

Local Development

pip install -e .
jupyter server
# go to http://localhost:8888/api/cloudcreds/aws

jupyter-cloud-scoped-creds's People

Contributors

scottyhq avatar

Watchers

 avatar  avatar

jupyter-cloud-scoped-creds's Issues

Boto3 alternative to AWS CLI

cmd = ['aws', 'sts', 'assume-role-with-web-identity',
'--role-arn', os.environ['AWS_ROLE_ARN'],
'--role-session-name', os.environ['JUPYTERHUB_CLIENT_ID'],
'--web-identity-token', f'file://{os.environ["AWS_WEB_IDENTITY_TOKEN_FILE"]}',
'--duration-seconds', '1000'
]
proc = await asyncio.create_subprocess_exec(

Just wanted to note this boto3 alternative https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts/client/assume_role_with_web_identity.html

# Get Temporary AWS Credentials on CryoCloud JupyterHub
client = boto3.client('sts')

with open(os.environ['AWS_WEB_IDENTITY_TOKEN_FILE']) as f:
    TOKEN = f.read()

response = client.assume_role_with_web_identity(
    RoleArn=os.environ['AWS_ROLE_ARN'],
    RoleSessionName=os.environ['JUPYTERHUB_CLIENT_ID'],
    WebIdentityToken=TOKEN,
    DurationSeconds=3600
)

ACCESS_KEY_ID = response['Credentials']['AccessKeyId']
SECRET_ACCESS_KEY_ID = response['Credentials']['SecretAccessKey']
SESSION_TOKEN = response['Credentials']['SessionToken']

Allow project to be adopted by the jupyterhub github organization?

@scottyhq @yuvipanda this project is a jupyter_server / notebook extension that is serving a purpose for jupyterhub in cloud environments (currently only AWS) where cloud provider specific credentials are provided to the user servers, allowing them to work against object storage for example.

I think this could be in scope to maintain within the jupyterhub github organization, and I would be a jupyterhub org member willing to champion an adoption to it help maintain this project long term from within the organization. This project is of direct relevance for hubs deployed by 2i2c.org where I work, but its also relevant for the broader jupyterhub community just like for example https://github.com/jupyterhub/gh-scoped-creds that has also been adopted from Yuvi to the jupyterhub organization.

Technical roadmap

  • Provide sphinx based documentation to cover
    • What the project is about
    • How it works
    • Security aspects
    • Admin user instructions (someone setting this up)
    • End user instructions (someone using this)
    • Changelog
  • Add support for GCP credentials
  • Add test of at least the extension setup
  • Setup CI/CD for autoformatting, tests, and release automation
  • Publish docs to readthedocs
  • Publish package to PyPI

Question

@scottyhq @yuvipanda are you okay with letting this be adopted by jupyterhub, and that I would go ahead and ask about if jupyterhub is willing to adopt this project in jupyterhub/team-compass?

Add a endpoint to emit the version etc for use in tests

It could be nice to have some endpoint that can be called, like /api/jupyter-cloud-creds (without /aws in the end) that emits version info for example.

One key benefit is that we would have a basic way to test the server extension that didn't involve using the aws or gcloud CLI's.

test on k8s deployment

so far this is was only tested locally with running jupyter server then going to http://127.0.0.1:8889/api/cloudcreds/aws which worked!

cc @yuvipanda

Rename to `jupyter-cloud-creds` ?

I'm considering a name chance along with #4.

Let's breakdown the current name jupyter-cloud-scoped-creds:

  1. jupyter because its exposing the creds through a jupyter server (jupyter_server or notebook)
    Perfect!
  2. cloud and creds because it relates to extracting cloud providers' credentials
    Perfect!
  3. scoped
    I guess the name jupyter-cloud-scoped-creds was inspired by gh-scoped-creds, but I think the scoped part doesn't translate to this project well. In gh-scoped-creds, the project enables a github users permissions provided to be reduced in scope to only a repository etc, but in this project we just expose all the available credentials to various cloud providers - they are not scoped to something specific by this project.

This project does ensre that the crendeitlas are temporary, and this project can the expiry time to some degree. I think adding temp or temporary to the name would be a bit much though, and that it could be sufficient to document this clearly and not include in the name.

Proposals

  1. A project name change to jupyter-cloud-creds (from jupyter-cloud-scoped-creds)
  2. An endpoint name change to api/jupyter-cloud-creds/aws (from api/cloudcreds/aws)
    If the endpoint name includes this projects name, it becomes easier for users to understand directly that this project provided the endpoint. Consider for example a server admin or similar sees request to api/jupyter-cloud-creds/aws in some logs, then it would be easier for them to find their way to this project and learn more than if it was api/cloudcreds/aws.

@scottyhq and @yuvipanda what do you think?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.