Giter VIP home page Giter VIP logo

fargatespawner's Introduction

fargatespawner

Spawns JupyterHub single user notebook servers in Docker containers running in AWS Fargate

Installation

pip install fargatespawner

Configuration

To configure JupyterHub to use FargateSpawner, you can add the following to your jupyterhub_config.py.

from fargatespawner import FargateSpawner
c.JupyterHub.spawner_class = FargateSpawner

You must also set the following settings on c.FargateSpawner in your jupyterhub_config.py. None of them are optional.

Setting Description Example
aws_region The AWS region in which the tasks are launched. 'eu-west-1'
aws_ecs_host The hostname of the AWS ECS API. Typically, this is of the form ecs.<aws-region>.amazonaws.com. 'ecs.eu-west-1.amazonaws.com'
notebook_port The port the notebook servers listen on. 8888
notebook_scheme The scheme used by the hub and proxy to connect to the notebook servers. At the time of writing 'https' will not work out of the box. However, users do not connect to the the notebook server directly, and does not, typically, allow incoming connections from the public internet. Instead, users connect to the proxy, which can be configured to listen on HTTPS independently of this setting. There is more information on setting up HTTPS for connections to the proxy at https://jupyterhub.readthedocs.io/en/stable/getting-started/security-basics.html. 'http'
get_run_task_args See below See below

The get_run_task_args argument must a callable that takes the spawner instance as a parameter, and returns a dictionary that is passed to the RunTask API call.

c.FargateSpawner.get_run_task_args = lambda spawner: {
    'cluster': 'jupyerhub-notebooks',
    'taskDefinition': 'jupyterhub-notebook:7',
    'overrides': {
        'taskRoleArn': 'arn:aws:iam::123456789012:role/notebook-task',
        'containerOverrides': [{
            'command': spawner.cmd + [f'--port={spawner.notebook_port}', '--config=notebook_config.py'],
            'environment': [
                {
                    'name': name,
                    'value': value,
                } for name, value in spawner.get_env().items()
            ],
            'name': 'jupyterhub-notebook',
        }],
    },
    'count': 1,
    'launchType': 'FARGATE',
    'networkConfiguration': {
        'awsvpcConfiguration': {
            'assignPublicIp': 'DISABLED',
            'securityGroups': ['sg-00026fc201a4e374b'],
            'subnets':  ['subnet-01fc5f15ac710c012'],
        },
    },
}

The parts of the return value that probably require change from the above example are:

  • cluster - The name of the ECS cluster into which the tasks are launched.

  • taskDefinition - The family:revision or full ARN of the task definition that runs the notebooks. Typically, this task definition would specify a docker image that builds on one from https://github.com/jupyter/docker-stacks.

  • taskRoleArn - The role the notebook tasks can assume. For example, in order for them to make requests to AWS, such as to use Jupyter S3 with role-based authentication.

  • securityGroups - The security groups associated with the Fargate task. They must allow communication to and from the hub/proxy. More information, such as the ports used, is at https://jupyterhub.readthedocs.io/en/stable/getting-started/networking-basics.html.

  • subnets - The list of possible subnets in which the task will start.

You must also, either, authenticate using a secret key, in which case you must have the following configuration

from fargatespawner import FargateSpawnerSecretAccessKeyAuthentication
c.FargateSpawner.authentication_class = FargateSpawnerSecretAccessKeyAuthentication

and the following settings on c.FargateSpawnerSecretAccessKeyAuthentication

Setting Description Example
aws_access_key_id The ID of the AWS access key used to sign the requests to the AWS ECS API. ommitted
aws_secret_access_key The secret part of the AWS access key used to sign the requests to the AWS ECS API . ommitted

or authenticate using a role in an ECS container, in which case you must have the following configuration

from fargatespawner import FargateSpawnerECSRoleAuthentication
c.FargateSpawner.authentication_class = FargateSpawnerECSRoleAuthentication

where FargateSpawnerECSRoleAuthentication does not have configurable options.

or authenticate using a IAM profile attached to EC2 instance, in which case you must have the following configuration:

from fargatespawner import FargateSpawnerEC2InstanceProfileAuthentication
c.FargateSpawner.authentication_class = FargateSpawnerEC2InstanceProfileAuthentication

Run-time dependencies

The spawner is deliberately written to not have any additional dependencies, beyond those that are required for JupyterHub.

Approximate minimum permissions

In order for the user to be able to start, monitor, and stop the tasks, they should have the below permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ecs:RunTask",
      "Resource": "arn:aws:ecs:<aws_region>:<aws_account_id>:task-definition/<task_family>:*",
      "Condition": {
        "ArnEquals": {
          "ecs:cluster": "arn:aws:ecs:<aws_region>:<aws_account_id>:cluster/<cluster_name>"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ecs:StopTask",
      "Resource": "arn:aws:ecs:<aws_region>:<aws_account_id>:task/*",
      "Condition": {
        "ArnEquals": {
          "ecs:cluster": "arn:aws:ecs:<aws_region>:<aws_account_id>:cluster/<cluster_name>"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ecs:DescribeTasks",
      "Resource": "arn:aws:ecs:<aws_region>:<aws_account_id>:task/*",
      "Condition": {
        "ArnEquals": {
          "ecs:cluster": "arn:aws:ecs:<aws_region>:<aws_account_id>:cluster/<cluster_name>"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::<aws_account_id>:role/<task-execution-role>",
        "arn:aws:iam::<aws_account_id>:role/<task-role>"
      ]
    }
  ]
}

fargatespawner's People

Contributors

jasoniamaunixadmin avatar michalc avatar zvymazal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fargatespawner's Issues

Question about `task-role` in documentation.

In the documentation the PassRole Action includes the resource arn:aws:iam::<aws_account_id>:role/<task-role>. What is this resource referring to? Our implementation is working without defining this resource.

    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::<aws_account_id>:role/<task-execution-role>",
        **"arn:aws:iam::<aws_account_id>:role/<task-role>"**
      ]
    }

Send environment from JupyterHub HTTP API

Hello! Just a quick question: is there any way to send variables from JupyterHub API request like POST /users/{name}/servers/{server_name} to the container. So for example if I set some kind of key in JSON body like { "key1":"value1"} would it be possible to use this variable inside the container at notebool?

Documentation issue

I am working on getting this setup for our current notebook setup. In the readme to use rolls for ECS auth this is what is written
from jupters3 import FargateSpawnerECSRoleAuthentication

Should that be
from fargatespawner import FargateSpawnerECSRoleAuthentication

I can do a PR if so.
Thanks

Support tags

It would be great to be able to configure tags. See PR #7 for an implementation.

Persisting User Input

Hey!

Will this spawner persist the user input inside the notbook?
Or is there a way to configure this?

Thanks!

Persistent Storage with EFS

Hello,
is there a way to configure fargateSpawner with EFS ?
I managed to get fargateSpawner running spawning Labs from a TaskDefinition using one EFS with an AccessPoint.
Resulting in all spawning Labs have the same storage dir.

I currently see two ways:
1.
Configuring Jupyterhub/FargateSpawner to redirect to a folder under the one EFS for example. (EFS is mounted on /home/joyvan)
/home/joyvan/user1 and /home/joyvan/user2 . The user1 or user2 dir must be created somewhere.
This would result in having one EFS for all and data accessible for everyone but users land in their home directory which is persistent. This is still okay for my solution.

Prettier:
2.
Configuring Jupyterhub/FargateSpawner to create an accessPoint for the known EFS for every User.
This will result in having one EFS but data is not accessible for everyone because of the accessPoints. But i need to set somewhere the new mounting configuration and it seems i am not able to do it with the c.FargateSpawner.get_run_task_args and the overrides.

   'cluster': cluster,
    'taskDefinition': runtaskTaskdefinitionArn,
    'count': 1,
    'overrides': {
        'containerOverrides': [{
....cutted

Depending its no even possible to override mount configs by the AWS RunTask or StartTask API.

The solutions seems to need more or less code implementations, which i really want to avoid.
Is there maybe already a solution?

Some help would be appreciated, thank you.

Redirect to '/lab' for the Spawner does not seem to work

Hi
I tried setting up the default_url on the Spawner to launch Jupyterlab by default but it does not seem to work. It ends up launching the Jupyter Notebook ('/user/username/tree')

c.Spawner.default_url = '/lab'

Is there another setting for fargatespawner that I should be using?
Thanks

How to set up JupyterHub in the first place

Dear @michalc,

sorry for the naive question: but how/where do I set up the JupyterHub in the first place? Does this assume that a JupyterHub (running inside a container?) is already running on AWS?

Thanks!

Best wishes,
-Filippo

set environment variable from options_form

I have tried to use a field from an options form to add an environment variable to a new notebook but cannot work out how

if I set the following i get an error, any pointers would be great, i also cant see how to add the value to an environment variable ourside the get_run_task_args
'environment': [
{
'name': 'newvar',
'value': spawner.user_options['key'][0]
},
{
'name': name,
'value': value,
} for name, value in spawner.get_env().items()
],

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.