Story

As someone using Duffy,
I want it to maintain pools of nodes,
so they can be quickly handed out when requests come in.

Acceptance Criteria

• A method exists to define pools in configuration
• Duffy can determine which pool an active nodes belongs to
• Duffy can tell how many nodes are available per pool to be handed out

Background, Thoughts

It could very well be that we can get by without storing e.g. OS type, version, (VM) flavour individually and use a simple string tag on nodes to describe the pool that a machine belongs to, and tenants would only ask for x machines of a node with a certain label. Because of the long-lived bare metal nodes, we need to add a field in nodes to store their architecture.

Story

As a an operator of Duffy,
I want to know if tables are present when trying to install the DB schema,
so that I don't think the schema is compatible with the code if it isn't.

Acceptance Criteria

• duffy setup-db will warn about tables being present rather than acting as if it did anything when it didn't

Background

MetaData.create_all() will check first if tables are present and then silently do nothing if that's the case.

@t0xic0der discovered an issue while reviewing #152: Namely that retiring a session didn't work through the API because a write lock held on the sqlite database file wasn't released. I examined the issue and found that a database session object from a previous (purposefully) failed request stuck around and wasn't cleaned up properly. To fix this, we need to refactor how DB sessions are handled to ensure we have one session per request which is cleaned up afterwards.

Story

As a CentOS CI admin,
I want to be able to keep records of tenants who are not active. Instead of deleting them completely, moving them to another table which signifies "Inactive".

Acceptance Criteria

• Active and Inactive tenants have a way to be identified

Story

As a contributor to Duffy,
I want CI jobs to be in place,
so my contributions are tested automatically when I submit them as PRs.

Acceptance Criteria

• A suitable framework from the alternatives (Jenkins, GH actions, …) has been chosen
• The basic structure to run CI jobs has been set up

Background

Potential candidates and options we can use:

• Jenkins
• dependabot
• LGTM

Story

As a Duffy developer,
I want access to the old database contents,
so I can make more educated decisions.

Acceptance Criteria

• A recent dump exists in a place where all team members can access it.

If a host was reserved through Duffy and then placed in the 'Reserved' state while the session is still active, the workers will still reclaim the node when the time limit for the session has expired.

We need either:

• A tool that finds active sessions tied to this host and marks them 'Reserved' as well, OR
• To modify the workers so machines can't transition out of the 'Reserved' state without manual intervention

Story

As a contributor to Duffy,
I want that it is tested against a number of Python versions,
so that I can be reasonably safe that it works in different environments.

Acceptance Criteria

• Tox is added as a dependency for development
• Tox is set up to execute tests under a variety of Python versions (3.6…3.9)

Story

As a user/contributor to Duffy,
I would like to know my rights while using or contributing to the software.

Acceptance criteria

• A LICENSE file with MIT license in the project root

The API key would be used to authenticate the normal tenants as well as the admin tenants for certain operations that require authorization.

Story

As an operator of Duffy,
I want to be able to configure parameters,
so I can be flexible in how I deploy and run it.

Acceptance Criteria

• A suitable configuration file format and location is chosen
• An example configuration file exists which demonstrates items that can be configured
• Code exists which loads the configuration file from its location and makes it available as an object for other code to consume

Background

At some point we need a way to point our service at its database, an OpenNebula cluster, etc. The configuration file format should be hierarchical to allow different levels of detail and grouping.

This (softly) depends on the CLI to be available (#55).

Stories

1. As an operator of Duffy,
   I want to apply updates,
   so I can make use of new features and fixes.

2. As a developer on Duffy,
   I want to be able to change the database models,
   so I don't have to think of everything up front.

Acceptance Criteria

• Alembic is added as a dependency and configured in Duffy

Oh noes:

nils@makake:~/src/fedora-infra/duffy (dev)> poetry install
Installing dependencies from lock file

No dependencies to install or update

  ValueError

  Invalid author string. Must be in the format: John Smith <[email protected]>

  at /usr/lib/python3.9/site-packages/poetry/core/packages/package.py:203 in _get_author
      199│ 
      200│         m = AUTHOR_REGEX.match(self._authors[0])
      201│ 
      202│         if m is None:
    → 203│             raise ValueError(
      204│                 "Invalid author string. Must be in the format: "
      205│                 "John Smith <[email protected]>"
      206│             )
      207│ 
nils@makake:~/src/fedora-infra/duffy (dev)>

This is part of #47.

Story

As a user of CentOS CI,

1. I want to be able to use VMs,
   because I don't want to waste resources.

2. I want to be able to use bare metal nodes,
   because I have requirements not met by VMs.

Acceptance Criteria

• An abstract base model for nodes exist
• Specific models for bare metal and VM nodes exist

Background

In the previous implementation, the Host model described bare metal nodes (specifically SeaMicro chassis), it mixed generic and specific information. We want to support different types, so need base classes modelling shared properties as well as child classes modelling the differences. Hierarchically, it could be e.g.:

• unspecified node -> bare metal -> bare metal in a sea micro chassis
• unspecified node -> VM/cloud node -> libvirt/kvm node

(No idea if we need that level of granularity.)

Expose API endpoint to add/edit/delete Users

• #77

Acceptance Criteria

Implement…

• a GET method to list items of the type
• a POST method to create new items of the type
• a PUT method to update items of the type
• a DELETE method to delete items of a type (only where applicable?)

Story

As an operator of Duffy,
I want to run Duffy from the command line,
to have a defined entry point for executing the service.

Acceptance Criteria

• A command line scripts exists which runs the web app service…
• …which is installed as an executable with poetry install.

Story

As a contributor to Duffy,
I want basic “Python project boilerplate” in place,
so I can actually develop something.

Acceptance Criteria

The repository is set up …:

• … with a top-level directory for the code and another for its tests
• … to use poetry for installing it and its dependencies (see below)
• … to use pytest and its cov plugin for testing and verifying coverage
• … so black assumes a line length of 100 characters
• … so .gitignore ignores some commonly generated files and directories
• … so pytest-black, pytest-flake8 and pytest-isort are dev deps and configured to be run with pytest

Background

Status: Everything is an empty void.

Background (serious)

These are the dependencies with which we want to build Duffy:

• fastapi as the web microframework
• sqlalchemy as the ORM encapsulating DB operations
• psycopg2 to use PostgreSQL
• pytest, pytest-cov for testing
• tox for running the tests, linting etc. in different environments

This needs #136.

Stories

As an admin of Duffy,
I want that the API authenticates its users,
so privileged operations are limited to admins.

As a normal tenant user of Duffy,
I want that the API authenticates requests,
so that only I can request resources on my behalf.

Acceptance Criteria

• A way exists to define an API key/token per tenant
• API endpoints check token validity and levels of access before performing work
• API endpoints return suitable status codes and error information if an activity that needs authorization is attempted with the wrong or a missing token

Background

• https://fastapi.tiangolo.com/tutorial/security/first-steps/

• #77

Acceptance Criteria

Implement…

• a GET method to list items of the type
• a POST method to create new items of the type
• a DELETE method to delete items of a type (only where applicable?)

Story

As a user of the Duffy API,
I want to have access to auto-generated online documentation,
so that Duffy understands my requests.

Acceptance Criteria

Tests exist verifying that

• the /openapi.json endpoint can be accessed and returns useful information
• both /docs and /redoc endpoints can be accessed and return valid HTML

Background

Problems with the /openapi.json endpoint can also be a sign for problems in the Pydantic models used in the API.

Configure a database with the necessary tables

• Populate the database
• Set up the database connection

Story

As a developer working on Duffy,
I want to know how the workflow to reserve nodes, use and release them is mapped to the API,
so I can implement it.

Acceptance Criteria

• A document exists which describes
  • the API endpoints involved in the nodes reservation workflow
  • and how a client will use them throughout the workflow
• A Pydantic model exists which let's users submit 'specifications' of the type and amount of nodes they need in a session
• The session endpoint
  • accepts such specifications in POST requests, attempting to allocate matching nodes for the session from those active and unused, and returns suitable results, status codes, errors if allocation fails.
  • lists allocated nodes in structures returned from the controller functions.
  • accepts PUT requests which lets normal tenants only set active to false, i.e. retire a session which will kick off the undeployment of the nodes etc.

Background

For the time being, this 👇 is the document mentioned above:

API workflow

Requesting nodes

API endpoints are implemented, and the central object node reservation revolves around is the 'session'. However, users shouldn't manipulate sessions and their related nodes individually, this work needs to be done by Duffy, users just need a way to tell Duffy what 'types' of nodes they need and how many (e.g. "2 bare metal, 1 large VM and 2 small VMs") when they request a session by POST-ing to /api/v1/sessions:

...
"nodes_specs": [
  {"quantity": 2, "type": "physical", "distro_type": "centos", "distro_version": "8"},
  {"quantity": 1, "type": "virtual", "flavour": "large", "distro_type": "fedora", "distro_version": "34"},
  {"quantity": 2, "type": "virtual", "flavour": "medium", "distro_type": "fedora", "distro_version": "34"},
],
...

Retiring a session

We could abuse the DELETE HTTP method on a particular session, but this is misleasing as no objects are deleted. The most "RESTful" thing to do is (I guess) to let users modify set a sessions active value to false, e.g.:

http --json PUT http://localhost:8080/api/v1/sessions/1234 active:=false
...

Listing allocated nodes

Currently, the structures returned from API endpoints are plain ORM objects. However, sessions and nodes are connected via intermediate SessionNode objects, which wraps a many-to-many intermediate table with additional fields. This needs to be suitably proxied into Session so a Pydantic model can serialize it to JSON like this:

"session": {
  "id": ...,
  "project": { ... },
  "nodes": [
    {"id": ..., "type": ..., "distro_type": ..., "distro_version": ...},
    ...
  ],
}

By default, when someone requests a node, duffy does a mysql query to find which nodes are available to be "given" to projects (https://github.com/CentOS/duffy/blob/master/duffy/api_v1/views.py#L62)
problem with that query is that it will always try the same node first, because of the "order by used_count"

So when that node is unreachable (the case we had today), duffy can't contextualize it (https://github.com/CentOS/duffy/blob/master/duffy/api_v1/views.py#L76) and so answer "Failed to allocate nodes" (https://github.com/CentOS/duffy/blob/master/duffy/api_v1/views.py#L90)

We should enhance (https://github.com/CentOS/duffy/blob/master/duffy/models/nodes.py#L56) to be sure that if we can't reach a node, it's just skipped and be marked to be reinstalled, so that it's removed from "Ready" status

So probably switching that line ? https://github.com/CentOS/duffy/blob/master/duffy/models/nodes.py#L72

Let's discuss this

Story

As a maintainer of Duffy,
I want that the dependencies used for testing are updated regularly,
so we can benefit from bugfixes and notice potential problems.

Acceptance Criteria

• Dependabot is configured in the repository to check for updated dependencies

Background

poetry install uses the dependency versions pinned in poetry.lock for local environments and CI tests. Dependabot checks for available updates and submits pull requests to update these versions.

Stories

1. As a contributor to Duffy,
   I want that branches are named descriptively,
   so that I make changes in the right place.

2. As a contributor to Duffy,
   I want that cloning the repository checks out the branch where development happens,
   so I make my contributions in the right place.

3. As a contributor to Duffy,
   I want stale branches to be clearly named as such,
   so I don't waste time with them.

Acceptance Criteria

• All existing branches are moved to a stale namespace, i.e. renamed to stale/<branch name>
• An empty dev branch is created
• The dev branch is made the default branch

Background

We want to do this from scratch, so all existing content should be moved out of the way. Besides that, master is a loaded term and doesn't really tell you the purpose of this branch.

Story

As a user of Duffy,
I want that my requests are answered quickly,
so that I don't feel ignored.

Acceptance Criteria

• A suitable task worker framework is integrated and can be used for long-running tasks.
  • Celery is set up to process tasks generated by the API frontend
  • Redis is set up to manage the queue(s) of tasks that need to processed

Background

There are a couple of things which need a longer time than what people expect an API call to take: provision a new node with an operating system (to be put into a pool for quick reuse), set up a provisioned host with SSH keys etc. before it can be handed out to tenants.

Story

As an operator of Duffy,
I want that it logs noteworthy things,
so I can later find out what happened.

Acceptance criteria

• Logging is set up when the app is started
• Logger objects are available in the different modules that need them

Background

Log state changes, actions, … so we can debug things better.

This depends on the CLI to be available (#55).

For people using the default configuration which is present in the repository, mistakenly or not, they would end up using an in-memory database. The application would start, no problem - but performing any path operation would result in referencing tables that do not exist, thereby causing an error and an exception message which is not very elegant.

(venv) [t0xic0der@fedorable duffy]$ duffy -c etc/duffy-example-config.yaml serve 

 * Starting Duffy...
 * Host address : 127.0.0.1
 * Port number  : 8080
 * Log level    : warning
 * Serving API docs on http://127.0.0.1:8080/docs
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-
.
.
<SOME MORE HUNDREDS OF BROWSER-CRASH-INDUCING LINES OF UNINDICATIVE EXCEPTION MESSAGE>

What do we do then?

Simple, we repurpose the code from here #132 to inspect the database that the config file points towards. If we do not find them the way we want them to - i.e. if it does not have the columns of the names and the types we are looking for, we do NOT start the application.

Story

As a user of Duffy,
I want to confine the set of nodes considered for my requests based on certain characteristics,
so that their peculiarities don't affect my work.

Acceptance Criteria

Users can limit the set of nodes considered for their requests…

• by a host name pattern
• …

Background

A user reported that their tests caused CPU lockups on nodes with Intel Xeon CPUs. See this pull request against the old code base: #25

Story

As a user of Duffy,
I want to be able to manage the various objects via the API,
because doing this with a DB CLI tool plain sucks.

Acceptance Criteria

API endpoints exist to add/edit/delete…

• Projects: #86
• Users: #80
• SSH Keys: #91
• Nodes: #89
• Sessions: #87

Background

Individual endpoints should provide a “RESTful” API, i.e. fulfil these acceptance criteria:

# Acceptance Criteria

Implement…
- [ ] a `GET` method to list items of the type
- [ ] a `POST` method to create new items of the type
- [ ] a `DELETE` method to delete items of a type (only where applicable?)

• #77

Acceptance Criteria

Implement…

• a GET method to list items of the type
• a POST method to create new items of the type
• a PUT method to update items of the type
• a DELETE method to delete items of a type (only where applicable?)

• #77

Acceptance Criteria

Implement…

• a GET method to list items of the type
• a POST method to create new items of the type
• a DELETE method to delete items of a type (only where applicable?)

With more specific information about the endpoints, what they do and their purpose.

How?

https://fastapi.tiangolo.com/tutorial/metadata/

Acceptance Criteria

• Functions listed for each model object
• App description and metadata entered

Refer #97 (review)

On raising a KeyboardInterrupt, supposedly the event loop abruptly exits out and shows an error. Catching the KeyboardInterrupt exception and treating it to close the event loop before destroying it, should do it.

Story

As a contributor to Duffy,
I want a skeleton application,
so I have some framework to code in.

Acceptance Criteria

• A URL structure exists for unversioned and versioned API endpoints
• A FastAPI app exists which serves a root API endpoint when run with uvicorn
• Document the method of setting up the application
• The new code is tested

Story

As a contributor to Duffy,
I want to be able to set the project up locally,
so I can actually develop something.

Acceptance Criteria

The repository is set up …:

• … with installation instructions for local development environment

Background

Status: We have the Poetry boilerplate.

Background (serious)

We use Poetry for dependency management so our instructions would hail from there.

Story

As a contributor to Duffy,
I want to know the ins and outs,
so my contributions go smooth.

Acceptance Criteria

A text file exists in the repository which documents:

• How to set up a development environment
• How to run the tests
• What requirements there are for contributions (coding style, refining commits, …)

Story

As an existing user of Duffy,
I want to be able to use the legacy API endpoints,
so that I don't have to update all my scripts immediately.

Acceptance Criteria

• Controller functions for the legacy API endpoints exist in a separate Python package.
• They accept the same arguments as the legacy API and map/translate requests where necessary, returning compatible results.
• These API endpoints authenticate using a key supplied as a GET parameter.

Background

The legacy API keys are mapped to tenants directly, this requires that they are stored in plain text, maybe we need to add a legacy_api_key field to accommodate. It would be nullable and we shouldn't expose it in the API, or allow tenants to be added (we'd just set these on import of the old project tenants). Idea: extend duffy.app.auth._req_tenant_factory so it lets you create dependencies that accept these keys, yet returning tenant objects into the controller functions.

This may be pretty large, feel free to repurpose into an epic and break out into individual issues.

Currently, tox is only configured to run tests with Python versions 3.6­…3.9 because we ran into weird unresolvable version issues with dependencies on Python 3.10.

Story

As an operator of Duffy,
I want to be able to store one SSH key per project,
to streamline shell access to CI nodes via e.g. the OpenShift console.

Acceptance Criteria

• User is gone
• Project has a field to store an SSH key
• Project has a field to store whether or not it is active/retired
• the ER diagram is updated accordingly

Being the absent-minded person that I am, I have started the Duffy server a lot many times without passing any configuration file as one of the command-line options. I usually faced this.

(venv) [t0xic0der@fedorable duffy]$ duffy serve

 * Starting Duffy...
 * Host address : 127.0.0.1
 * Port number  : 8080
 * Log level    : warning
 * Serving API docs on http://127.0.0.1:8080/docs
ERROR:duffy.cli:Configuration key missing or wrong: database

Pretty normal, elegant and descriptive error message, right?

When I try to do the same for setting up the database without passing any configuration file as one of the command-line options. I end up getting this.

(venv) [t0xic0der@fedorable duffy]$ duffy setup-db

Traceback (most recent call last):
  File "/home/t0xic0der/Projects/duffy/duffy/database/__init__.py", line 77, in get_sync_engine
    sync_config = deepcopy(config["database"]["sqlalchemy"]) or {}
KeyError: 'database'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/t0xic0der/Projects/duffy/venv/bin/duffy", line 5, in <module>
    cli()
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/t0xic0der/Projects/duffy/venv/lib64/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/t0xic0der/Projects/duffy/duffy/cli.py", line 67, in setup_db
    setup_db_schema()
  File "/home/t0xic0der/Projects/duffy/duffy/database/setup.py", line 18, in setup_db_schema
    engine = get_sync_engine()
  File "/home/t0xic0der/Projects/duffy/duffy/database/__init__.py", line 81, in get_sync_engine
    raise DuffyConfigurationError(
duffy.exceptions.DuffyConfigurationError: database

Not very elegant, right?

We would want to fix this.

• #77

Acceptance Criteria

Implement…

• a GET method to list items of the type
• a POST method to create new items of the type
• a DELETE method to delete items of a type (only where applicable?)

This is needed for #135.

Story

As a developer of Duffy,
I want that project and admin access are evaluated on the same basis,
so that the code dealing with it doesn’t get too complicated.

Acceptance Criteria

• ORM classes and associated tables, API endpoints and Python files related to projects are renamed to "tenant(s)", as applicable.
• The new Tenant ORM class (and following, the tenants table in the database) has a boolean column is_admin which would let requests authenticated by their token (yet to come, see #135) manipulate any object, not only those which are related to that tenant.

Background

I don't know why we started calling these projects in the first place. 😉

• #77
• This depends on #92
• It's possible that this should be an addition to the user endpoint, alternatively or in addition to a separate endpoint (discuss)

If 70 jobs come through and they all call /Node/fail, we're out 70 machines for 6 more hours. We should rate-limit this, and call done on their behalf if they exhaust the limit.

Story

As a Duffy developer,
I want an interactive shell
so I can experiment with database queries and objects.

Acceptance Criteria

A script or (sub)command exists which

• runs an interactive shell that
• sets up a database connection/session from specified configuration (see #95) and makes it available in the local namespace of the shell together with DB model classes

Background

See gearbox tgshell as an example (with much more features) for TurboGears-based web projects.

We have Ansible Playbooks that are not used immediately by the code but we need a way to fetch the same while installing Duffy via Poetry/DNF.

Ohai,

right now, the old Duffy API has a ver parameter in /Node/get, that can be used to select a node of a specific version (7, 8, etc) and if unset Duffy defaults to giving you a node with ver=7.

For our tests, it doesn't matter if we run them on 7 or 8 (as we deploy a couple of VMs ontop of the Duffy node we get), so it would be nice to be able to say "don't care" and get a free node from whichever pool is available right now.

Thanks!

Story

As an operator of Duffy,
I want that tenants are created with unique, random API keys,
to make this process simpler and more secure.

Acceptance Criteria

• When creating tenants, their API keys mustn't be supplied to the API, but they are chosen randomly and returned in the API response.

Background

To be able to securely identify and authenticate tenants for them to perform endpoint operations, we need to be able to uniquely identify them. If UUIDs are passed by the tenants, that adds more effort in the process of creation of the tenants and risks UUIDs being repeatedly used for multiple clients - thus sabotaging the unique identification via UUIDs, hence we would want to rework the model, the endpoint code and the tests to accommodate autogenerated and unique UUIDs. -- @t0xic0der

Random UUIDs (i.e. "version 4") are unique, for all intents and purposes: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions -- @nphilipp

Story

As a Duffy developer,
I want that various types of information are stored in a database,
because keeping them in a text file would be cumbersome.

Acceptance Criteria

Database models exist for:

• the nodes Duffy manages: #63
• the projects able to request nodes
• the users which contribute to projects
• the sessions for which nodes are reserved