This API provides the backend for the safers-dashboard-app.
PostGIS database is used for safers-dashboard-api.
Note that the POSTGRES_USER and POSTGRES_PASS must be set in ".env". This is a top-level configuration file used for Docker itself (as opposed to the server-level configuration file used by Django).
FusionAuth is used for authentication. This can be run locally for development (but not much else will work in safers because all the other components need to authenticate against tokens genreated from the deployed authentication server).
When run locally, the local PostGres database, "safers-auth-db", must also be run.
The local instance of FusionAuth is bootstrapped using kickstart. Kickstart will only run if the default API Key is unset. This means that even if you change the "auth/kickstart.json" file you will have to manually recreate the authentication database. The following command will do this: docker container ls | grep safers-auth | awk '{print $1}' | xargs docker rm --force.
When using the lcoal instance of FusionAuth, you must also populate "auth/.env" with the following environment variables:
FUSIONAUTH_ADMIN_EMAIL="[email protected]"
FUSIONAUTH_ADMIN_PASSWORD="password"
FUSIONAUTH_API_KEY
FUSIONAUTH_APPLICATION_ID
FUSIONAUTH_CLIENT_ID
FUSIONAUTH_CLIENT_SECRET
FUSIONAUTH_TENANT_ID
FUSIONAUTH_TENANT_NAME
FUSIONAUTH_URL
FUSIONAUTH_INTERNAL_URL
FUSIONAUTH_EXTERNAL_URL
FUSIONAUTH_REDIRECT_URL
(These should match the values in "server/.env" and "client/.env".)
If bootstrapping succeeeds, you should see the following sort of output:
--------------------------------------------------------------------------------------
----------------------------------- Kickstarting ? -----------------------------------
--------------------------------------------------------------------------------------
io.fusionauth.api.service.system.kickstart.KickstartRunner - Summary:
- Created API key ending in [...HSnO]
- Completed [POST] request to [/api/tenant/<tenant-id>]
- Completed [POST] request to [/api/application/<applications-id>]
- Completed [POST] request to [/api/user/registration/]
The FusionAuth Admin should be available at https://localhost:9011.
As mentioned above, though, all of this is probably superfluous since very little of the dashboard will actually work if not authenticated using the deployed instace of FusionAuth.
RabbitMQ
The RMQ app uses pika to monitor the SAFERS Message Queue. Upon receipt of a message with a registered routing_key the handlers defined in rmq.py#BINDING_KEYS are called.
The managemnt comamnd manage.py purge_camera_media should be run periodically to remove outdated camera_media objects.
In development this is done by a separate scheduler service that uses cron to run ./scheduler/scripts/purge_camera_media.development.sh. In deployment, this is done by heroku scheduler which runs ./scheduler/scripts/purge_camera_media.deployment.sh.
Django / Django-Rest-Framework
AWS S3
A standard configuration for VSCode is included w/ this repository at ".vscode/settings.json" and ".editorconfig". This ensures that linting, etc. works the same for all developers; It prevents local configuration discrepencies from cluttering up git diffs.
A standard set of extensions should also be used in VSCode by all developers. These can be loaded in one go by using the Astrosat VSCode Extensions Repository.
This project uses pdm. Unlike traditional python package and dependency managers, it doesn't create a complete virutal environment - instead it uses the existing python interpreter and isolates python packages in a local directory. Docker mounts that directory ("server/pypackages") into the container.
One annoyance about this is that the binaries in the venv (like yapf for instance) will use the python interpreter from the container; VSCode, which runs on the local filesystem, cannot access that interpreter. Therefore, those binaries should be re-installed using the global interpreter specified by pyenv. This is not ideal, but it works.
One other minor annoyance is that although pdm is build to support PEP-582 it will default to using standard virtual environments unless PEP-582 is enabled globally. Rather than impose that restriction on developers, this repository includes an empty "server/pypackages" directory which will force pdm to use PEP-582. More information can be found here: https://pdm.fming.dev/latest/usage/pep582/
To add a package use pdm add <package> or pdm add -dG <group> <package>.
The Django Server uses various environment variables as settings. Uers must provide them in "server/.env" or "server/.env.deployment" depending on the environment. Here are some examples:
- DJANGO_SECRET_KEY
- DATABASE_URL="postgis://safers_user:safers_pwd@safers-db:5432/safers_db"
- FUSIONAUTH_API_KEY
- FUSIONAUTH_APPLICATION_ID
- FUSIONAUTH_CLIENT_ID
- FUSIONAUTH_CLIENT_SECRET
- FUSIONAUTH_TENANT_ID
- FUSIONAUTH_TENANT_NAME
- FUSIONAUTH_URL="https://auth.safers-project.cloud"
- FUSION_AUTH_EXTERNAL_URL="https://auth.safers-project.cloud"
- FUSION_AUTH_INTERNAL_URL="https://auth.safers-project.cloud"
- DJANGO_RMQ_TRANSPORT="amqp"
- DJANGO_RMQ_HOST="bus.safers-project.cloud"
- DJANGO_RMQ_PORT=5674
- DJANGO_RMQ_EXCHANGE="safers.b2b"
- DJANGO_RMQ_VHOST
- DJANGO_RMQ_QUEUE
- DJANGO_RMQ_USERNAME
- DJANGO_RMQ_PASSWORD
- DJANGO_RMQ_APP_ID="dsh"
The server directory is mounted as a volume inside the docker container. In order to facilitate this, though, the "app" user in the container must have the same user id and group id as the local user. A ".env" file at the same level as the Dockerfile is used to store these values. The following command can generate that ".env" file:
echo -e "RUN_AS_UID=$(id -u)\nRUN_AS_GID=$(id -g)" >> .envFailure to do this may result in permissions errors when Django tries to copy static or media files or when the local user tries to modify files in the virtual environment.
This project uses pytest rather than the built-in Django testing framework to test the server code. This can be run locally via pdm run test. This will provide terminal output detailing each test run and providing a stack trace for any failing test. It will also produce a pretty HTML report for that test run. Previous test runs are archived in "server/archive" and the report allows you to compare current test results with archived tests.
The test are run in github whenever code is pushed. Again, this will provide both terminal output in the workflow trace and an HTML report as an artifact. The latter can be downloaded and viewed in the browser. At this time, reports genreated in CI will not include any archived test results.
In order to interact with the Django Admin you must first create a superuser:
docker-compose exec safers-server pdm run ./manage.py createsuperuser
You will be prompted for an email and password. It is recommended to use "[email protected]" and "password" during development.
Every domain that a Django Project is deployed to is associated with a Site instance. In orbison every Site has a SiteProfile associated with it.
It is a good idea to set the default site (the one that settings.SITE_ID references) to the development server by running docker-compose exec orbison-server pdm run ./manage.py update_site --domain localhost:8000 --name DEVELOPMENT.
Additionally, in order to ensure uniqueness among message ids in RabbitMQ, the SiteProfile contains a code attribute. This should be set in the Django Admin; just ensure the code is also registered with the deployed instance of RabbitMQ.
In order to show both operational and on-demand Data Layers, a DataType must be registerd. The current set of data_types is stored in a remote spreadsheet [datamappingform_imp.xlsx]. The admin action "Import DataTypes from CSV" can take that content and import it into the database; Note this is a brittle process, so backing up the database beforehand is recommended.
safers-dashboard-api is run locally via docker-compose. It is recommended, though not necessary, to run each service in a separate terminal pane so that each container's output is easily distinguished.
Prior to running make sure that you have followed the instructions in Local Settings and Permissions.
The following command will run a service: docker-compose up <service-name>
Having run the server, if you need to interact with it via the termnial you may use the following command: docker-compose exec safers-server pdm run ./manage.py <command>
The server should be available in the browser at http://localhost:8000.
Periodic backups of the database and the media files are taken by the scheduler. This utilises the "django-dbbackup" library.
The filename format is "safers-dashboard-api-<type>_<timestamp>" where "type" is one of "db" or "media".
The Django storages framework is used to store the backup files. In development these are stored in MEDIA_ROOT/backups; They can be accessed directly from the local filesystem. In deployment, these are stored in S3; They can listed using: aws --profile <profile name> s3 ls s3://<bucket-name>/backups/
and downloaded using: aws --profile <profile name> s3api get-object --bucket <bucket-name> --key backups/<backup-file> <backup-file>.
Localization is enabled in safers-dashboard-app. This localizes text used in the frontend, not the backend. The current set of localizations is stored in a remote spreadsheet [dashboard-translations.xlsx]. There are scripts in safers-dashboard-app which take that content and import it; Note this is a manual process.
Profiling is handled using *silk. Request info can be monitored by going to "http://localhost:8000/silk". All requests will provide high level timing information. Actual profiling (ie: tracking function calls rather than queries) can be enabled by setting SafersSettings.profile to True. Then the silk_profile decorator and/or context manager can be used on a per-view basis. Corresponding binary ".prof" files can be downloaded from w/in the silk GUI and then inspected using snakeviz.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent