Giter VIP home page Giter VIP logo

nl-covid19-notification-app-backend's Introduction

COVID-19 Notification App - Backend

Table of Contents

Introduction
External Documentation
Development and Contribution Process
Local Development Setup
License
Attribution
Unreleased

Introduction

This repository contains the backend code for the Dutch exposure notification app.

The backend code runs on .NET 6.

External Documentation

The Dutch exposure notification app uses the Google Apple Exposure Notification (GAEN) framework developed by Google and Apple as part of their effort to help combat the SARS-CoV-2 pandemic. Please find their documentation in one of the following 2 locations:

The Dutch exposure notification app is part of the group of EU countries using the European Federation Gateway Service (EFGS) for sharing their national exposure keys on a European level. Please find the EFGS code and documentation on GitHub:

Development and Contribution Process

The core team works on the repository in a private fork for reasons of compliance with existing processes, and will share its work as often as possible.

If you plan to make code changes, please feel free to open an issue where we can discuss your changes, before opening a pull request. This avoids possibly doing work that we might not be able to use due to various reasons (specific infrastructure demands, already working on, etc).

If you think the information contained in this README is incomplete or wrong, please feel free to directly open a pull request on the README.

If you have any other questions about the README or the information contained therein, please feel free to open an issue.

Local Development Setup

Before being able to run the projects contained in the backend solution, you will need to set up a database, and be able to connect to the HSM Signer Service test server.

Certificates

CoronaMelder has its files signed by the HSM Signer Service with a RSA certificate and an ECDSA certificate. The latter is a requirement set by Apple and Google.

Versions of these certificates for local testing can be found in the folder src/Crypto/Resources:

  • TestRSA.p12
  • TestECDSA.p12

Please note: these certificates are not production certificates.

The file password for TestRSA.p12 is Covid-19!; the password for TestECDSA.p12 is empty.

The file StaatDerNLChain-EV-Expires-2022-12-05.p7b can be ignored, as the local certificates are self-signed.

Database

This project assumes the presence of a PostgreSQL database.

After installing PostgreSQL, you can either create all the necessary databases and tables manually (using the SQL scripts contained in the databases directory), or run the DbProvision project to have everything generated automatically.

Projects

The codebase consists of the following projects, allowing you to locally set up a backend that contains Temporary Exposure Keys, Exposure Key Sets, a Manifest, and the various configuration JSON files that are representative of the actual backend.

DbProvision

A console application that removes and rebuilds the required databases; only used for development and debugging.
The nonuke-argument can be supplied to prevent removing any existing databases.
Additionally, several types of JSON files can be inserted into the database by means of passing one of the following arguments, followed by a path to the specific JSON file:

  • -a, for Appconfig.json
  • -r, for RiskCalculationParameters.json
  • -b, for ResourceBundle.json

GenTeks

A console application that generates Temporary Exposure Keys ('TEKs') and inserts them into the database. For development and testing purposes only.
Two arguments can be passed to the application: the amount of workflows (or 'buckets') and the amount of TEKs per workflow.
By default, 10 workflows with each 1000 TEKs are created, equivalent to passing the arguments 10 1000. These arguments can be found (and changed) in src/GenTeks/Properties/launchSettings.json

ForceTekAuth

A console application that authenticates all workflows in the database, equivalent to users contacting the GGD to publish their workflows, or self-authorising their workflows through coronatest.nl.

This project is for development and testing purposes only. No command-line arguments can be provided.

As the GenTeks project currently marks workflows as "authorised" when it creates them, there is strictly speaking no need to run the ForceTekAuth project anymore.

PublishContent

A console application that, equal to DbProvision, allows the insertion of various JSON files into the database.
The files can be inserted into the database by means of passing one of the following arguments, followed by a path to a JSON file that contains said content:

  • -a, for Appconfig.json
  • -r, for RiskCalculationParameters.json
  • -b, for Resourcebundle.json

SigTestFileCreator (deprecated)

Deprecated: the backend no longer self-signs content, so installing certificates is no longer necessary

A console application that is used to check if the private keys of the installed certificates can be accessed and used. For development and testing purposes only.

The program has one command-line argument: the path to a file that will be signed with the RSA and GAEN key.

ProtobufScrubber (deprecated)

Deprecated: the backend no longer self-signs content, and the X962PackagingFix.cs is no longer used

A console application that alters the GAEN signature of a signed file so that it can be verified by OpenSSL. The requirement of this program stems from a difference in writing the header bytes to the signature files. See X962PackagingFix.cs for more info.

The program has one command-line argument: the path to a zip with an export.sig file.

EfgsTestDataGenerator

A webservice that is used for automated end-to-end tests. It exposes one endpoint to mock the EFGS-service.

Content.WebApi

A webservice for downloading the various files by the clients. It exposes endpoints that allows the user to download the manifests, exposure keysets, appconfigs, resourcebundles and risk calculation parameters.

DailyCleanup

A console application that removes old data (i.e., older than 14 days) from the database, in order to comply with privacy regulations.

After the cleanup, it generates a set of statistics:

  • The total amount of TEKs
  • The amount of published TEKs
  • The total amount of Workflows
  • The total amount of Workflows with TEKs
  • The total amount of Workflows authorised

As the name suggests, the DailyCleanup is run every night at a set time.

EksEngine

A console application that takes all authorised workflows and downloaded TEKs from EFGS, bundles them into an EKS, signs it and places it in the databas. It then updates the manifest and prepares the TEKs from the authorised workflows for delivery to EFGS. Please find more elaborate documentation on the EksEngine in docs.

Icc.V2.WebApi

A webservice that exposes the PubTek-endpoint. This endpoint is used to authorise workflows with the GGD key. Its functionality is used by the GGD Portal and by coronatest.nl.

Icc.WebApp

This project contains 2 functionalities:

  1. the frontend code for the GGD Portal, utilizing Microsoft.AspNetCore.SpaServices.Extensions.
  2. a webservice that is used as a passthrough to the PubTek-endpoint described above for the GGD Portal, and that provides authorisation functionality for the GGD Portal.

Iks.Downloader

A console application that downloads TEKs from EFGS and stores them to be processed by the EKSEngine.

Iks.Uploader

A console application that takes the TEKs that the EKSEngine has prepared and uploads them to EFGS.

ManifestEngine

A console application that updates the manifest with the most recent content of the databases.

MobileAppApi.WebApi

The webservice that the clients use to request the creation of a Workflow (or 'bucket'), upload their TEKs and perform decoy uploads to preserve the privacy of the client.

License

This project is licensed under the EUPL license. See LICENSE for more information.

Attribution

Some parts of this application are inspired by the work on Private Tracer. You can find their license here.

Unreleased

The feature/dashboarddata branch contains the backend code changes for the (as of yet unreleased) in-app COVID-19 related statistics (similar to coronadashboard.rijksoverheid.nl).

The new functionalities added to the backend are:

  • fetching of data with the DashboardData.Downloader
  • making the downloaded data available for the apps through a new API endpoint v1/dashboard in Content.WebApi
  • removing old downloaded data with the DailyCleanup process

The status of this branch is "work in progress", and additional improvements are necessary to make it more robust.

For an extensive explanation of the dashboard functionality, please see one of the following pages for more information:

nl-covid19-notification-app-backend's People

Contributors

bartdenhoed avatar bdmcc-nl avatar capsert avatar casperboone avatar christian-marc avatar dirkx avatar glashelder1988 avatar hiddehs avatar hvisser avatar ijansch avatar mvdende82 avatar nicktencate avatar playtime222 avatar rkops-bd avatar rwiegmans avatar ryanbnl avatar skos001 avatar the8day avatar yorimyorimyorim avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nl-covid19-notification-app-backend's Issues

2020-08-19: [VWS/Covid19NotificationApp]

Describe the bug, issue or concern

The concern is the existence of the code block in https://github.com/minvws/nl-covid19-notification-app-backend/blob/master/Components/Icc/Models/TheIdentityHubClaimTypes.cs:


        internal const string AccessToken = "http://schemas.u2uconsult.com/ws/2014/03/identity/claims/accesstoken";
        internal const string AuthenticationStrength = "http://schemas.u2uconsult.com/ws/2016/08/identity/claims/authenticationstrength";
        internal const string ClientId = "http://schemas.u2uconsult.com/ws/2014/11/identity/claims/clientid";
        internal const string OldIdentityId = "http://schemas.u2uconsult.com/ws/2019/02/identity/claims/oldidentityid";
        internal const string DisplayName = "http://schemas.u2uconsult.com/ws/2014/04/identity/claims/displayname";
        internal const string EmailAddress = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress";
        internal const string EmailAddressVerified = "http://schemas.u2uconsult.com/ws/2017/02/identity/claims/emailaddressverified";
        internal const string GivenName = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname";
        internal const string IdentityProvider = "http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/identityprovider";
        internal const string LargePicture = "http://schemas.u2uconsult.com/ws/2014/04/identity/claims/largepicture";
        internal const string MediumPicture = "http://schemas.u2uconsult.com/ws/2014/04/identity/claims/mediumpicture";
        internal const string Scope = "http://schemas.u2uconsult.com/ws/2014/03/identity/claims/scope";
        internal const string SmallPicture = "http://schemas.u2uconsult.com/ws/2014/04/identity/claims/smallpicture";
        internal const string Surname = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname";
        internal static readonly string NameIdentifier = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier";

I'm assuming these are either not used or they are used to indicate URLs to third-parties that can be used to establish the validity of the schemes. However, I haven't looked into it. Reasons to remove this:

  • These are SEO links for Google to boost the http://u2uconsult.com site (this one as well :-) ).
  • These are http links rather than https links, so vulnerable for man in the middle attacks.
  • The corresponding server get hits from the system verifying their schemas. This leaks information on use.
  • Most of those links do not currently resolve.

Expected behavior

Fix. Https links to your own backend or remove.

Governance

Using docker-compose fails

Describe the bug, issue or concern

Currently docker-compose up --build fails on a clean clone due to a couple of errors in the Docker file.

This is running on Ubuntu 20.04.

To Reproduce

Steps to reproduce the behavior:

  1. Create a fresh clone of the repository:
git clone https://github.com/minvws/nl-covid19-notification-app-backend.git
  1. Enter the freshly created repository.
cd nl-covid19-notification-app-backend
  1. Start the docker compose process as described in the docs.
cd docker
docker-compose up --build
  1. Generates the following output and error.
$ docker-compose up --build
Building mobile_api
Step 1/23 : FROM mcr.microsoft.com/dotnet/core/sdk:3.1-alpine AS builder
 ---> 656495ef4e20
Step 2/23 : COPY . app/
 ---> Using cache
 ---> c642901eb776
Step 3/23 : WORKDIR app/
 ---> Using cache
 ---> 1ace1d52bf4c
Step 4/23 : COPY docker/development/appsettings.json .
 ---> Using cache
 ---> 88648b988891
Step 5/23 : RUN rm Components/Framework/WindowsIdentityStuff.cs
 ---> Running in d0d719382673
rm: can't remove 'Components/Framework/WindowsIdentityStuff.cs': No such file or directory
ERROR: Service 'mobile_api' failed to build: The command '/bin/sh -c rm Components/Framework/WindowsIdentityStuff.cs' returned a non-zero code: 1
  1. Fixing this error generates further output leading to another error:
MSBUILD : error MSB1009: Project file does not exist.
Switch: DbFillExampleContent/DbFillExampleContent.csproj
ERROR: Service 'mobile_api' failed to build: The command '/bin/sh -c dotnet publish DbFillExampleContent/DbFillExampleContent.csproj --no-self-contained  --configuration Release -o publish/Tools/DbFillExampleContent --version-suffix local' returned a non-zero code: 1

Expected behavior

The docker containers are successfully brought up.

Screenshots

n/a, see console output above

Additional context

I'll shortly submit a simple PR to address the issues.

Governance

docker build

Describe the bug, issue or concern

Docker local run fails

To Reproduce

Steps to reproduce the behavior:

  1. git clone repo
  2. cd docker
  3. docker-compose up --build
  4. Errors in Dockerfile: due deletion of Components/Framework/WindowsIdentityStuff.cs on initial run
    solved by using RUN rm -rf
    issues with the destination of csproj files and new names given to projects (CotentApi & MobileAppApi), path should be prefixed with src/

Missing projects:
DbProvision/DbProvision.csproj
DbFillExampleContent/DbFillExampleContent.csproj

Changed names of exe files in entrypoint in docker-compose.yml

Expected behavior

instruction from readme to run the project in docker should work without errors

Governance

Concern : In the unit-tests, run WireMock.Net on a random port instead of fixed port

Concern

I see that WireMock.Net is used to unit-test the TheIdentityHubService in the TheIdentityHubServiceTests.cs file.

However a fixed port is used:

This can cause issues when running these unit-tests on a build-server, there is not 100% guarantee that this port will be free on the OS.

My suggestion would be to start the WireMock.Net without a fixed port. Just like:

var server = WireMockServer.Start();

// Getting the random port on which the WireMock.Net server is running, can be done like:
var port = server.Ports[0];

// Or you can also get the full URL where the WireMock.Net server is running with:
var url = server.Urls[0];

However this means that setting up the TheIdentityHubOptions should be done in each unit-test because the port is not fixed anymore.

Another suggestion:
I see that in every test the WireMock.Net server is started and stopped, in that case using a local variable var server = instead of defining and reusing a private class variable like private WireMockServer _Server; would be a better option.

Another idea would be to start the WireMock.Net server once during start from the test-class and reset + add a new mapping in each test. This can be a bit quicker when running the tests.

Consider serving a security.txt file

First of all I would like to say that I really appreciate that this work is being open sourced.

I want to propose adding a security.txt file to be publicly served by this application.
security.txt is a proposed standard for listing responsible disclosure policies and contact information. Its purpose is to provide a standard location so security researchers can easily get in contact to report a security issue. The standard is currently in draft stage but is already in use by organizations like Google and GitHub.

This GitHub repository already has a security policy, but once this application is running in production a security researcher who has found a vulnerability might not be aware of the fact that this repository is publicly visible. They will have to search for out-of-band information in order to responsibly disclose security issues. As explained for example by security researcher Troy Hunt, who runs the popular data breach notification service Have I Been Pwned?, the security.txt file is one of the first things he will look for when he tries to contact an organization about a data breach.
Given the high public profile of this application and the focus on privacy and security I believe it is a good step to serve this file, and it comes at no real cost.

Based on the existing security policy published on GitHub, the contents of security.txt could look like this:

Contact: https://www.ncsc.nl/contact/kwetsbaarheid-melden
Contact: https://english.ncsc.nl/contact/reporting-a-vulnerability-cvd
Encryption: https://english.ncsc.nl/contact/pgp-key
Policy: https://github.com/minvws/nl-covid19-notification-app-backend/security/policy

This file should be served from /.well-known/security.txt (RFC8615). Other fields, such as Acknowledgments, Canonical and Preferred-Languages are also supported. Please refer to the draft standard or to https://securitytxt.org/ for the full specification.

I would be happy to create a pull request, but I'm not sure where to create this file in order for it to be served as a static file in the document root. If I'm understanding correctly the Docker configuration and ServerStandAlone directory are only used for development, and will not be used in production. Alternatively the file could be stored in the root of the repository (as suggested on https://securitytxt.org/) and be be copied to the right location during build-time. As I said, if someone can point me in the right direction I would happily create a pull request for this.

Scope of this project

What is this project doing?
I see it mentions .NET, MSSQL, as Node JS, Yarn and Angular CLI, shouldn't the 'frontend' for care workers be in another project?

My advise to make this more accessible for other developer ๐Ÿ‘€

  • Update README.md with folder overview plus short summary
  • Put README.md in every folder with the project/folder description what this thing is doing.

Decoy probability constant

Describe the bug, issue or concern

According to the design documentation the decoy probability should change depending on positive test numbers (and possibly install counts). However, it seems to be fixed to 0.00118, also on the production CDN.

Governance

Running in Docker on Linux fails

Describe the bug, issue or concern

When running the docker-compose install on Linux, executing the following command fails:

docker exec docker_content_api_1 EksEngine/EksEngine

This is running on Ubuntu 20.04.

To Reproduce

Steps to reproduce the behavior:

  1. Start/build the docker install on Linux.
docker-compose up --build
  1. Initialise the database.
docker exec docker_content_api_1 Tools/DbProvision/DbProvision
  1. Generate Teks.
docker exec docker_content_api_1 Tools/GenTeks/GenTeks
docker exec docker_content_api_1 Tools/ForceTekAuth/ForceTekAuth
  1. Attempt to initialise the Eks engine
docker exec docker_content_api_1 EksEngine/EksEngine
  1. Generates the following error.
[21:38:59 FTL] System.PlatformNotSupportedException: Windows Principal functionality is not supported on this platform.
   at System.Security.Principal.WindowsIdentity.GetCurrent()
   at NL.Rijksoverheid.ExposureNotification.BackEnd.Components.Framework.WindowsIdentityQueries.CurrentUserIsAdministrator() in /app/Components/Framework/WindowsIdentityQueries.cs:line 13
   at NL.Rijksoverheid.ExposureNotification.BackEnd.Components.ExposureKeySetsEngine.ExposureKeySetBatchJobMk3.Execute() in /app/Components/ExposureKeySetsEngine/ExposureKeySetBatchJobMk3.cs:line 78
   at NL.Rijksoverheid.ExposureNotification.BackEnd.EksEngine.Program.Start(IServiceProvider serviceProvider, String[] args) in /app/EksEngine/Program.cs:line 46
   at NL.Rijksoverheid.ExposureNotification.BackEnd.Components.ConsoleApps.ConsoleAppRunner.Execute(String[] args, Action`2 configure, Action`2 start) in /app/Components/ConsoleApps/ConsoleAppRunner.cs:line 41
  1. Notice that there are no exposureKeySet ids in the manifest.

Expected behavior

The EksEngine starts up successfully and the exposureKeySet ids are added to the manifest.

Screenshots

n/a.

Additional context

PR #27 highlights where the errors occur in the EtkEngine code. The problem stems from the fact that the user certificate store isn't available on Linux (I'm not sure whether this is a Linux issue or a Docker issue though). The PR works around the problem, but isn't really a fix because it results in dummy signatures being generated (it does, however, make clear where the issues are, I think).

Governance

Private key visible inside the sources

var privateKey = "MHcCAQEEIMrxoiydeg4MGFiC0yCrAW1iVcacKidg9ZCtyHEBe0wqoAoGCCqGSM49AwEHoUQDQgAE6VXlMAvHyd9xoOnsMujJLiptp9/dvXtVPtp9TgTa4GGJl07YxGqozgTRrSc6iVh6JG8ZStYgaIRDdjd9LMgKlQ==";

I don't think you want your private key visible like that. Best remove it quickly and regenerate.
It gives me the creeps to see stuff like this in a code base.

Also, I see you're not working with a process to handle security issues like this one.
Would be nice to have that in place too.

Developer level connection strings/credentials should be generated

Some files in this repository contain secrets. I don't think you want to expose sa db - passwords to the whole world.

For instance here and here. (You should also not add an appsettings.development.json file in source-control).

Make use of docker secrets to inject secrets in a container; do not expose connectionstrings via the git repository!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.