microsoft / ogc-api-fast-features Goto Github PK

The application currently supports HTML and JSON/GeoJSON output encodings for collection metadata and item data.

Add support for additional output encodings. This could potentially include GeoPackage, GML or flatgeobuf to name a few.

Careful consideration will need to be given if introducing dependencies like GDAL OGR. There is the potential to add complexity and inefficiencies to the application. There is some discussion of this in the readme.

The existing CMD command references an incorrect path for the gunicorn conf file.

Update the path to reflect the working directory: CMD ["gunicorn", "-c", "/oaff/fastapi/gunicorn/gunicorn.conf.py", "oaff.fastapi.api.main:app", "--timeout", "185"]

SUPPORT.md has not been edited since this repository's creation.

Update the following in SUPPORT.md:

• Determine whether Customer Service & Support is required and update the file accordingly.
• Determine what is required in the following section and update accordingly:

  For help and questions about using this project, please REPO MAINTAINER: INSERT INSTRUCTIONS HERE FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER CHANNEL. WHERE WILL YOU HELP PEOPLE?.

Migrate existing code from private repo to new Open Source repo

Update oaff/testing/Dockerfile to benefit from a multistage build and decrease time taken for tests to run. Explanation from @cuttlefish in this PR:

The testing Dockerfile seems to inherit from the main image, then install a bunch of dependencies. That'll be pretty slow. All of the test requirements will need to be installed every time a single file changes in the base image.
A multistage build would be a lot faster, but I'm not sure how much time you have to put into this.

It is also worth considering a multistage build in /Dockerfile:

• The first stage would build the Python dependencies with the required OS dependencies
• The second stage would create an image containing the relevant built python dependencies, and runtime OS dependencies curl, but none of the dependencies from the first stage (libpq-dev, python3-pip, python3-psycopg2).

The information in the "Next Steps" has been used to create new issues and is therefore no longer relevant.

Remove it from README.md.

At present, PostgreSQL/PostGIS is the only supported input data source type.

Based on priorities, determine an alternate input data source and add support for it. This could include a text format like GeoJSON (potentially ndjson or GeoJSON Text Sequences flavors) or something entirely different.

There has also been some discussion around apache parquet and geo-arrow in relation to STAC in the past.

New sources can be added to feapi/app/feapi/app/data/sources. Note that references to feapi are likely to change to oaff in the future. This path will change to oaff/app/oaff/app/data/sources when that happens.

For example #26, #28

Workflows like .github/workflows/ci-test.yml do not appear in a pull request as a status check when a pull request is created from a branch of a forked repository. Perhaps GA settings need to be adjusted?

At present the API supports Part 1: Core) of the OGC API - Features specification.

Add support for Part 2: Coordinate Reference Systems by Reference.

The application currently depends on a Sparkgeo fork of pygeofilter. At present there is a manual step in requirements_dev.txt to include this dependency.

Sparkgeo contributed changes to pygeofilter which have now been accepted and incorporated into version 0.0.2 of the PyPI package: https://pypi.org/project/pygeofilter/.

Configure this package release as a dependency and remove the manual step.

The repository currently contains two redundant Azure DevOps Pipeline files:

• azure-develop-pipeline.yml
• azure-main-pipeline.yml

These files are no longer required as Github Actions will take their place. azure-develop-pipeline.yml has been replaced by .github/workflows/ci-tests.yml and azure-main-pipeline.yml will be replaced when #21 is resolved.

When API_ROOT_PATH is set to a value other than the default "", the OAFF UI and all associated URLs will be available at http://localhost:8008, however the OpenAPI OGC path will not be available.

To replicate this:

1. Set an API_ROOT_PATH value in .env-dev. e.g. API_ROOT_PATH=/oaff.
2. Start the API with scripts/server && scripts/demo_data.
3. Add http://localhost:8008/oaff to QGIS and attempt add a layer to a map or alternatively visit http://localhost:8008/oaff/ogc/openapi.json. There will be an Internal Server Error 500 response.
4. View the logs for the API service with docker compose logs API. The following error will be visible:

api_1  |   File "/opt/ogc-api-fast-features/./oaff/fastapi/api/openapi/openapi.py", line 35, in handler
api_1  |     definition["paths"]["/collections/{collection_id}/items"]["get"][
api_1  | KeyError: '/collections/{collection_id}/items'

The error is the result of the redirect path which is added to the definition in oaff/fastapi/api/routes/__init__.py if a custom ROOT_PATH is set. The redirect path contains the following (note that there is no parameters key):
{'get': {'summary': 'Redirect', 'operationId': 'redirect__get', 'responses': {'200': {'description': 'Successful Response', 'content': {'application/json': {'schema': {}}}}}}}

During data interrogation, temporal fields are identified and used to derive metadata to support temporal filtering in dat requests. TIMESTAMP WITH TIME ZONE, TIMESTAMP WITHOUT TIME ZONE, and DATE PostgreSQL types are currently supported.

PostgreSQL also supports a number of range-like temporal data types which are not currently supported (source):

• tsrange — Range of timestamp without time zone
• tstzrange — Range of timestamp with time zone
• daterange — Range of date

Add support for each of these temporal range types.

More information on temporal fields is included in the readme.

PostgreSQL/PostGIS tables are currently supported but views are not.

Add support for views.

The main obstacle to support is likely to be difficulties identifying a suitable unique field in each view, as a view is not required to have a Primary Key. It may be possible to provide this information via oaff.collections (formerly feapi.collections) or an alternate metadata entity within the oaff schema.

The application currently refers to the separation of concerns in the code as "backend" for business logic and "frontend" which deals with receiving requests and sending responses. To make this clearer and more consistent with other applications, the backend ("app") should be renamed to "core" and the frontend (fastapi) should be renamed to "application".

Update the application to publish an oaff PyPI package with two namespace packages/subpackages - oaff.core and oaff.application.

This approach is based on the approach taken by TiTiler which has a similar separation of concerns. It will make the terminology in this application and PyPI package clearer and more consistent with common practices elsewhere.

Another good approach which could be an alternative is stac-fastapi.

The working name for this application was "feapi" but with the repo being named "ogc-api-fast-features" it makes sense to remove these references to keep things consistent and avoid confusion. "ogc-api-fast-features" can be abbreviated to "oaff".

Replace referenced to "feapi" with "oaff" within the documentation/readme and the code.

Useful when the API is behind a proxy server like nginx with a stripped path prefix as described here.

Use root_path to configure the FastAPI application.
Potentially add an environment variable named APP_ROOT_PATH to control the root path. However this may not be in line with the APP/API naming conventions in this application at present.

Values for docs_url, openapi_url and redoc_url are currently being passed into the FastAPI constructor on line 27 of oaff/fastapi/api/main.py. Determine how to pass in root_path without affecting the behavior of the API or docs.

This API has been written with performance in mind, however quantitative benchmarking has not taken place against other OGC API - Features implementations.

Carry out benchmarking against a number of existing OGC API - Features implementations with identical data configurations.

This will establish a quantitative baseline and comparison with other implementations.

This is also discussed in the readme.

Support for this is required in:

• pre-commit hooks (.pre-commit-config.yaml)
• scripts/cibuild

Related to #5 and #26, which introduces mypy as a pre-commit hook with configuration defined in pyproject.toml. Currently this is set to ignore all errors but disabling this option and running mypy -p oaff or pre-commit run --all-files results in a large number of errors. See here for an example mypy output.

This issue is meant to track work required to resolve errors produced by mypy.

A question I'm not sure the answer to:
Is ignore_missing_imports=true appropriate here? even with mypy --install-types there are multiple packages without type stubs.

The current workflows (ci-test.yml and ci-build-push.yml) have some redundancy and can potentially be streamlined to avoid:

• overlapping event triggers in certain cases. ci-test.yml called on pull request and all push events, while ci-build-push.yml called on only pushes to main.
• tests are run by both, but in slightly different ways (ci-build-push.yml will call scripts/test with --no-cache arg)
• ci-build-push.yml will build an image more than once, can we instead reuse if tests pass?
• flak8 and mypy are called directly, can we instead use https://github.com/pre-commit/action to ensure outputs are consistent? (deprecated)

I suggest that

• Either consolidate into single workflow, or clearly separate their intended functionality e.g. tests run on pull requests and commits, while build & push are triggered only for tagged commits
• Add tag as an event trigger to support tagged releases on ghcr.io (and later to pypi per #16)

This action will build an image and publish a publicly accessible container to Github Container Registry.

At present the API supports Part 1: Core of the OGC API - Features specification.

Basic spatial and temporal filters are required as part 1 of the specification and these are provided by pygeofilter.

This issue is to add support for OGC API - Features - Part 3: Filtering and the Common Query Language (CQL). This part of the specification is still in draft as of November 2nd 2021, however it is nearing completion.

pygeofilter is developing support for Simple CQL as described by OGC API - Features - Part 3. This is expected to provide much of the functionality here but additional testing will be required to ensure the required functionality requirements are met. The CITE test coverage should also be expanded to cover part 3 of the specification.

Sparkgeo made contributions to pygeofilter to meet the requirements for OGC API - Features - Part 1 and the project's owners are aware of this project. Work with pygeofilter owners/developers to contribute to the project where required. Communication channels include the repo's discussions page and gitter.