We should provide a migration guide to allow developers easily identify which methods in the fasjson-client they will use instead of the method they are currently using in python-fedora.

The majority of apps use the following calls:

person_by_username
group_by_name
person_by_id
"/user/list"

PR here: fedora-infra/fasjson-client#14

See: #52

uses python-fedora to hit

/account
/user/list
/json/person_by_username

We want to write a REST API in Python following the OpenAPI standard. There are many web frameworks and libraries that can help us with that.

Here's what I think we should consider:

• Flask-RestPlus
• Connexion
• FastAPI -- not based on Flask but very promising

Acceptance Criteria

The project must:

• be maintained, with regular releases, a responding bug tracker and a strong developer community
• have a strong user community and be widely used
• be compatible with OpenAPI
• be reasonably fast

I haven't personally tried any of those frameworks but they look interesting. I think FastAPI is very much worth investigating, as it is based on solid libraries, has good references, and supports async. They have an interesting page listing other API projects that they took inspiration from.

This should be based on the investigation in #6.

Acceptance criteria:

• API works
• API supports pagination where it is relevant
• API follows a standard kind of schema
• Schema is published
• Documentation is available to describe how to use the API (including examples)

DoD:

• All AC are met
• Unit tests are written
• Coverage is 100% (or less if justified)
• Code passes the bandit security analyser
• Dependencies are checked for compatible licenses
• Code is formatted according to the Black tool

As a developer, I want an API endpoint to retrieve the open-api specification in JSON format.

Acceptance Criteria

• GET endpoint in /v1/
• Should return the specification content from the specs folder
• The specs folder should be included in the project distribution
• Should this endpoint be public (no authentication required)?

Definition of Done

Retrieve the project's open-api spec definition in a GET HTTP request.

Related Issue: #7

Example

from flask import Flask, Response

#subclasses the default flask.Response object
class AppResponse(Response):
	charset = 'utf-8'
	default_status = 200
	default_mimetype = 'application/json'

	@classmethod
	def from_result(cls, status_code, data=None):
		instance = cls(status=status_code)
		if data:
			instance.data = {
				'result': data
			}
		return instance

	@classmethod
	def from_error(cls, e):
		instance = cls(status=e.status_code)
		instance.data = json.dumps({
			'error': e.as_dict()
		})
		return instance

	@classmethod
	def force_type(cls, rv, environ=None):
		#forces json parsing for all responses
		if type(rv) != str:
			rv = json.dumps(rv)
		return super(AppResponse, cls).force_type(rv, environ)


app = Flask(__name__)
app.response_class = AppResponse


@app.route('/1')
def one():
	return AppResponse.from_result(status_code=200, data={'a': 1})

@app.route('/2')
def two():
	#works since the respone class is able to parse objects to jsonstring
	return {'a': 1}

Acceptance Criteria

• A response class that inherits from flask.Response
• A class that dumps objects into json strings with helper methods for result or error responses
• The class should be set as the response class for the flask app object

Definition of Done

The class being used as the flask response class.

As a Developer, I want to provide a health endpoint for monitoring tools

Acceptance Criteria

• A GET /v1/health endpoint that returns fasjson health status;
• It should return a JSON response, either result or error (as other methods);
• Add LDAP status if possible;
• Endpoint unit tests;
• Update open-api spec file(s) with the new method definition.

Definition of Done

A new GET API method, an updated open api spec and docs.

https://pagure.io/Fedora-Infra/packager-sync-bz
https://pagure.io/fedora-infra/toddlers/blob/master/f/toddlers/plugins/packager_bugzilla_sync.py

It uses person_by_username and group_by_name: https://pagure.io/fedora-infra/packager-sync-bz/blob/master/f/packager_sync_bz/fedora_account.py

Related Issue: #7

Acceptance Criteria

• A route that maps to /v1/groups - see OpenApi spec for details
• Returns a list strings that contains the CN for each group
• Pagination support if possible
• Unit tests using the flask test client: https://flask.palletsprojects.com/en/1.1.x/testing/
• Deal with expected errors such as 401, 403, 404, etc

Definition of Done

A valid REST API endpoint that follows the openapi spec and can be used from any HTTP client tool such as swagger, curl, etc.

See: #52

https://github.com/fedora-infra/pdc-updater/blob/9287defd89085105c7f125ee03fbdd6ee4bb33a4/pdcupdater/services.py#L86

queries /user/list/

Acceptance Criteria

• the list of agreement names is available either when getting a user directly or when getting a specific resource like /users/<username>/agreements/.

Definition of Done

• AC is met
• All tests pass
• Documentation is valid

Uses flask_fas_openid here: https://github.com/fedora-infra/nuancier/blob/master/nuancier/__init__.py

Related Issue: #7

Example:

from flask import Flask


app = Flask(__name__)


class AppError(Exception):
	def __init__(self, message=None, status_code=None, code=None, data=None):
		self.status_code = status_code
		self.message = message
		self.code = code
		self.data = None

	def as_dict(self):
		return {
			'code': f'{self.status_code}.{self.code}' if self.code else self.status_code,
			'message': self.message,
			'data': self.data
		}

	def as_json(self):
		return json.dumps(self.as_dict())


@app.route('/404')
def notfound():
	#raising AppError will call handle_error function
	raise AppError(message='dummy notfound error', status_code=404, code=100)

@app.route('/')
def root():
	x = []
	return {'value': x[50]} #raises IndexError


@app.errorhandler(AppError)
def handle_error(e):
	r = Response(status=e.status_code, mimetype='application/json')
	r.data = e.as_json()
	return r


#this is reached if an unregistered/unexpected exception is raised (IndexError in this case)
@app.errorhandler(InternalServerError)
def handle_error_500(e):
	original = getattr(e, 'original_exception', None)
	r = Response(status=500, mimetype='application/json')
	r.data = json.dumps({
		'error': {
			'code': '500.500',
			'message': 'unexpected internal error',
			'data': {
				'exception': str(original)
			}
		}
	})
	return r

Acceptance Criteria

Documentation: https://flask.palletsprojects.com/en/1.1.x/errorhandling/

• An error class to be used as de default base error exception in flask
• Other classes can inherit for this to specify specific behaviour
• This class needs to be registered as a flask error exception to be parsed as a valid response

Definition of Done

Flask app error responses being handled by the new exception class.

See: #52

Hacking on the project is currently a pretty involved thing, one needs a FreeIPA instance separate from the local machine (because FreeIPA automatically registers the host it's installed on as a client), one needs to install things to enable GSSAPI in Apache, etc.

It would be easier if there was a Vagrant setup (or equivalent) that a developer could use to make and test changes to the code.

For a Vagrant setup, the acceptance criteria would be :

• A developer just has to check out the code and run vagrant up to have a running VM with the project installed.
• There is no external dependency once the VM is running (no need to have a FreeIPA server installed somewhere)

DoD: all of the above are met.

Acceptance Criteria

• A search endpoint is available on FASJson
• Any value can be passed and it will match against multiple fields
• Multiple users can be returned for one search

Definition of Done

• all of the above are complete
• testing
• documentation

Related Issue: #7

Acceptance Criteria

• A route that maps to /v1/groups/$group_name/members - see OpenAPI spec for details
• Returns a list of user ids, logins or equivalent user info
• Pagination supported if possible
• Unit tests using the flask test client: https://flask.palletsprojects.com/en/1.1.x/testing/
• Deal with expected errors such as 401, 403, 404, etc

Defition of Done

A valid REST API endpoint that follows the OpenAPI spec and can be used from any HTTP client tool such as swagger, curl, etc.

Acceptance Criteria

• The project should follow a standard format
• The project should use a dependency manager
• The project should use tox to make managing and running common commands easier
• The dependencies should be clearly defined along with any other tools in a pyproject.toml file

Definition of Done

All of the above are complete
Other team member reviewed

Related Issue: #7

I think it is good a idea to structure the application (api resources and libs if needed) in a "versioned" way so we easily manage and support different API versions such as /v1/groups/.

This may require some kind of folder/project architecture re-factoring.

Acceptance Criteria

• The internal code API should manage versions where fit
• The Rest API routes should e versioned as well using the following scheme: /v1/...

Definition of Done

Project structure and http routing system using an API versioning scheme.

https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/batcave/files/retrieve-security-question.py

https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/batcave/files/sync-openshift-keys.py

There are multiple places within the ansible repo where configs will need to be updated also. Most apps have some config defined there.

Acceptance criteria:

• Every app defined in this epic needs to have its configuration updated
• Configuration should be uniform across all applications
• The PR should be merged with peer-review
• All files within the ansible repo which use the fas2 account system should also be updated.

Using the list of applications created in #9 and the python library created in #8, all applications need to be ported to use the library.

AC:

• A pull request with the migration to fasjson has been created in each app. Please add a link to those pull requests in comments here.
• The pull request follows the app's standards and is merged
• If possible, a configuration flag is available to use the old FAS API or the new fasjson API, to make deployment easier
• A new version of the app including the port is released
• The application is deployed in staging (after fasjson is deployed there obviously) and works properly

DoD: all of the above.

See: #52

uses python-fedora to call the person_by_username method

pull request sent here: https://pagure.io/sigul/pull-request/3

The current FAS has an read-only API that some of the Fedora applications use. These applications include zodbot and the sigul bridge.

In the new freeIPA / secuiritas world, such applications will interact with fasjson (https://github.com/fedora-infra/fasjson).

This ticket is to track the porting of these applications over to the new fasjson API

(description updated based on @relrod comment below -- https://github.com/fedora-infra/securitas/issues/56#issuecomment-581444027 )

Code lives at https://pagure.io/fedora-commops/geofp

It uses:

• group_members
• people_by_key
• group_by_name

It's a simple class to get Geographical information from the users account

elections sets up python-fedora here: https://pagure.io/elections/blob/develop/f/fedora_elections/__init__.py#_68

and uses it to make a few calls (mostly person_by_username) throughout the app.

This should be changed to use fasjson_client instead (based on a configuration setting) so that once redeployed it wil use the new backend.

Things to think about when designing the API:

• REST and HTTP links between results
• API version
• schema (food for thought: OpenAPI, CoreAPI, Swagger, etc)
• handling of paginated results
• writing a client should be easy

Acceptance criteria:

• The schema format has been decided
• We have a good vision of how fasjson's API should look like

DoD: all of the above

membership-map.py and membership-mwclient.py here: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/membership-map/build/templates both use python-fedora to communicate with FAS.

Related Issue: #7

Acceptance Criteria

• A route that maps to /v1/users/$username - see the OpenAPI spec for details
• Returns user info based on provided username
• Unit tests using the flask test client: https://flask.palletsprojects.com/en/1.1.x/testing/
• Deal with expected errors such as 401, 403, 404, etc

Definition of Done

A valid REST API endpoint that follows the OpenAPI spec and can be used from any HTTP client tool such as swagger, curl, etc.

Acceptance Criteria

• Check if the authentication header is set in the request (Authorization: Negotiate ...)
• If the header is not present the api should return a 401 error informing that the auth header is missing
• Should return authentication errors as early as possible
• Update the openapi spec if possible (probably as a "apiToken" type of auth)
• Use the header value which is an auth token to connect to the ldap server and handle operations
• Unit tests

Definition of Done

Authenticated API requests and ldap operations working.

Acceptance Criteria

• sphinx based documentation
• docsring everything (description, input, output, etc): http://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html
• open-api spec sphinx docs: https://github.com/sphinx-contrib/openapi

Definition of Done

Project and API documentation in HTML format

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv fasjson-MJrliQxI-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

[PackageNotFound]
Package requests-kerberos (0.12.0) not found.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

See: #52

https://pagure.io/fedora-badges has a dependency on python-fedora. Once python-fedora has been updated so too should this module.

As a developer, I'd like to have fasjson deployed alongside securitas in CommuniShift, in a CD manner.

It may be more complex than securitas because fasjson uses Kerberos / GSSAPI to authenticate.

Acceptance criteria:

• fasjson is deployed in CommuniShift
• the instance updates when code changes happen in fasjson
• it is linked to the same FreeIPA server as securitas

DoD: all of the above are met.

See: #52

https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/blob/develop/fedmsg_meta_fedora_infrastructure/fasshim.py it uses python-fedora to hit '/user/list'

https://pagure.io/fedscm-admin/blob/master/f/fedscm_admin/fas.py

closing as duplicate

CentOS uses x509 certificates to access build nodes, we need to provide a way for those users to get their certificates.

Acceptance Criteria

• Users have an API endpoint to list the certificates they own (possibly using the cert_find IPA command)
• Users have an API endpoint to get their signed certificate if they have one (possibly using the cert_show IPA command).
• Users have an API endpoint to upload their CSR to (possibly using the cert_request IPA command)
• Depending on whether the CSR endpoint returns signed certificates directly or not, users may need an API endpoint to check the status of their certificate signing request (possibly using the cert_status IPA command)
• Those endpoints are authenticated like the rest of FASJSON, through Kerberos
• A user only has access to the certs they are allowed to see. It's usually only theirs, but people with elevated permissions may be allowed to see more, so we shouldn't assume that a user can only have one certificate.

For example, there could be:

• a /certs/ namespace where certificates could be queried with ?users=<connected_user>, calling cert_find
• a /certs/<serial> endpoint where a certificate would be retrieved with cert_show.
• a /certs/request endpoint where CSRs could be uploaded. It could return the signed certificate in a JSON dict, that would also contain the URL to that certificate in the API as a hyperlink.
  Those are just suggestions. See fedora-infra/noggin#34 for the original request.

Definition of Done

• All AC are done
• There is an OpenAPI spec describing the endpoints
• Hyperlinks are used to link results to endpoints
• There is documentation
• Unit test coverage is 100%

Related Issue: #7

Acceptance Criteria

• A route that returns the user identity - see the OpenAPI definition
• It should deal with predictable errors such as 401, 403, etc
• Unit tests for the http rest method using the flask test client

Definition of Done

A valid REST API endpoint that follows the OpenAPI spec and can be used from any HTTP client tool such as swagger, curl, etc.

A good first step would be to look into the infra Ansible repo for the fedoraDummyUserPassword variable, it means that this app is using the dummy FAS account to make queries.

Another way would be to grep all our source codes for the fedoraproject.org/accounts URL.

Once we have the list we need to look at what those apps do in their FAS queries. Whether it's read-only or not, what endpoints they are querying, etc.

AC:

• There is a comprehensive list of apps that talk to FAS
• For each app the list details which endpoint / which query is made
• We mainly needs the apps that do read-only access but those who do read-write access should be flagged for later.
• The source code / repo and points of contacts are identified
• Code owners of those apps know that a code change will be necessary when the project is deployed, and they know that we are willing to do it or to assist them doing it.

DoD: All AC are met

Uses python-fedora to make basic FAS calls here: https://pagure.io/fedscm-admin/blob/master/f/fedscm_admin/fas.py

See: #52

LDAP queries are not bound as the connecting user but with the host's keytab.

$ klist -A
Ticket cache: KCM:1000
Default principal: [email protected]
Valid starting       Expires              Service principal
04/23/2020 09:40:04  04/24/2020 09:39:53  HTTP/[email protected]
04/23/2020 09:39:58  04/24/2020 09:39:53  krbtgt/[email protected]
04/23/2020 10:14:04  04/24/2020 09:39:53  HTTP/[email protected]
04/23/2020 13:13:05  04/24/2020 09:39:53  ldap/[email protected]
$ curl --negotiate -u : http://fasjson.example.test/fasjson/v1/me
{"result":{"info":{"cn":["services","accounts"],"dc":["example","test"],"krbprincipalname":"http/[email protected]"},"raw":"dn: krbprincipalname=http/[email protected],cn=services,cn=accounts,dc=example,dc=test"}}

FASJSON should return the DN of [email protected] and not the service.
Apache's mod_wsgi shows [email protected] in the log files and sets GSS_NAME to that value as well. It also sets KRB5CCNAME to FILE:/run/fasjson/ccaches/[email protected] in the request's environment.

The bind is done with:

self.conn.sasl_interactive_bind_s(
    "", ldap.sasl.sasl({}, "GSS-SPNEGO")
)

@tiran any idea what is going on?

the ansible playbook here: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/distgit/pagure/files/fas2.py creates the requests to contact FAS, these will need to be updated.

this possibly needs to be consolidated with python-fedora once that update happens.

the-new-hotness config.toml seems to use the old FAS dummy user account here: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/openshift-apps/the-new-hotness/templates/config.toml#n153

As a developer, I want a script that generates an open-api spec 2.0 file (aka swagger 2) from a open-api spec 3.0 file

Acceptance Criteria

• A script that takes a open-api spec 3.0 file as input and writes a swagger 2.0 file as output
• Should generate both json (app usage) and yaml (human readable) formats

Definition of Done

A swagger spec file being generated from a open-api spec 3.0 file.

Extra Notes

• This needs to be done because bravado, the open-api library we use in fasjson-client, do not support open-api spec 3.0
• This tool can be used to convert it to a swagger 2.0 file format: https://github.com/LucyBot-Inc/api-spec-converter
• swagger 2.0 does not support links so those need to be removed

Uses flask fas openID here: https://github.com/fedora-infra/mirrormanager2/blob/master/mirrormanager2/app.py#L92

A python library would be most useful, it would be based on the work in #7 and could take advantage of the schema that the API publishes.

AC:

• The library works and maps API objects to Python objects
• The library is published to PyPI
• Documentation exists to explain how to use the library

DoD:

• All AC are met
• Unit tests are written
• Coverage is 100% (unless properly justified)
• Code passes the bandit security analyser
• Dependencies are checked for compatible licenses
• Code is formatted according to the Black tool

Code lives at: https://pagure.io/fedora-infra/distgit-bugzilla-sync/

It uses:

• people_by_key
• group_by_name
• person_by_username

https://pagure.io/fedora-infra/distgit-bugzilla-sync/blob/master/f/distgit_bugzilla_sync/script.py

Each of the consumers below need to be updated to use fasjson instead of FAS. They are simple config changes to match the updates from their respective tickets.

• badges-awarder.py --> https://pagure.io/fedora-infra/ansible/blob/master/f/roles/badges/backend/templates/badges-awarder.py#n38
• fmn.consumer.py --> https://pagure.io/fedora-infra/ansible/blob/master/f/roles/notifs/backend/templates/fmn.consumer.py#n102
• fmn.web.py --> https://pagure.io/fedora-infra/ansible/blob/master/f/roles/notifs/frontend/templates/fmn.web.py#n17
• pdcupdater.py --> https://pagure.io/fedora-infra/ansible/blob/master/f/roles/pdc/backend/templates/pdcupdater.py#n34

A/C

• each config is updated to use fasjson
• the variables should be uniformly named
• the pull request is merged to the ansible repo

See: #52

https://github.com/fedora-infra/fmn/blob/abfc610bd443df2ae588f3d5ce4e1c689cd59c71/fmn/fmn_fasshim.py

same as fedmsg_meta_fedora_infrastructure #58

Acceptance Criteria

Creation of integration & unit tests
Coverage

Definition of Done

Tests to be completed identified and documented
Reviewed by team/sign off by team