energinet-datahub / archived-geh-post-office Goto Github PK

Call blobstorage with url from subdomain - retrieve and parse data.
Send data back to Actor.

• Implementation
  ( - [ ] Convert data to CIM XML/ebiX)
• Architecture
• Test

Are the overall architecture correct and ready.

• Document. (1 day)

Vi forventer der kommer et idempodency check inden rammer PO. BLOCKED.

In some cases exceptions are thrown, and in other a bool, indicating success or failure, is returned.
This should be streamlined.
The bool result is returned all the way up to the DataAvailableInbox function and then discarded.

Undersøg problemet hvor der gemmes korrekt i idempotency men fejler ved DA save. Hvad kan der gøres, exceptions eller retry, deadletter ?

As Post Office I want to receive a notification about available data so that I can work with the data.

• Servicebus with a queue. (1 day)
• Azure function to handle messages on queue. (1 day)
• Test. (1 day)
• Apply Mediator

When a commit is done and code sent to repo, you need to document.

• Should we send new data or old data / has Dequeue() been called since last Peek()?

• Do we have data from previous Peek

• If yes return data

• If no send data request to sub domain

• Test

• Implementing use of Azure Table Storage

As a DB I want to be able to store data available-messages in a structured and efficient way.

• Architecture/design. (1 day)
• Create CosmosDB. (0.5 day)
• Test. (??)
• Documentation. (0.5 day)

Add content to domain road map (currently planned and perhaps future work if it makes sense).

If nothing is planned, use this text:
"No work planned in current program increment."

Currently the postoffice has a map of types -> container.
This map is defined in application settings through terraform.

We would like for this to be something that is generic, and can be reconfigured through a terraform created instance.

We need to setup the base solution for the post office to run from.
This includes

• dotnet solution
• Readme
• editoconfigs

Rename IDocumentStore to something that doesn't describe the underlying DB tech e.g. Repository

Add the following files in root with reference to main repo docs:

• COMMUNITY.md
• CONTRIBUTING.md
• SECURITY.md

Add docs folder in root and add the following files:

• code-of-conduct.md
• getting-started.md
• non-functional-requirements.md
• test.md

Move IDocumentStore to Energinet.DataHub.PostOffice.Domain to make it accessable from DomainServices etc.

• Request dataset from sub domain.

• Receive path to data.

• Implementation

• Session id

• Architecture

• Test

• On failure - handling and logging (DatasetNotFound, DataSetNotAvailable, InternalServiceError)

Is your feature request related to a problem? Please describe.

Other domains need to be able to communicate with the post office, and automatically through terraform fetch and use connection strings and such as variables.
Therefor we need to add the inbound queue + schema storage connection strings to the core keyvault

Look into MS' IConfiguration.

Create and register config objects in ServiceCollectionExtensions, and inject these where needed instead of using Environment.GetEnvironmentVariable() directly.

Suggested config objects:

• CosmosConfig
• ServiceBusConfig

Use beta package for StyleCop.Analyzers. It supports record types. After, all record types should have their property names updated to Pascal case.

Microsoft.CodeAnalysis.FxCopAnalyzers is deprecated and should be removed.

Look in https://github.com/Energinet-DataHub/geh-metering-point/blob/main/Directory.Build.props for inspiration.

The README file currently reflects the old architecture, and not the new one.
This should be updated to fit

We need to calculate the message priority before saving to Cosmos database. This should help us read importing messages first.
Calculation is domain logic.

We need an api that exposes an endpoint that gives outside domains the possibility to communicate with the post office.

To grpc or not to grpc

Remove if (dataAvailableContract is null) throw new InvalidOperationException("Cannot parse bytes to document.");. DataAvailable.Parser.ParseFrom never returns null.

Is your feature request related to a problem? Please describe.

We need to be able to support multiple outbound formats. EBIX, CIM and so on.
To do that we will expand the post office with the concept of schemas.

Describe the solution you'd like.

We will extend the function that peeks with the following functionallity.

1. It will fetch a bundle from cosmos
2. Take the type and version from the document, and combine that with the query parameter format
3. Combine those and look in the schemas storage for any schemas following the type type/format/version
4. Map the contents of messages into that format, and content type the query parameter content-type
5. Return the message

Change integration test for DataAvailable so we use a Cosmos Database and not Moq.
Test that a DataAvailable is save correctly to database.

Fields: uuid, recipient, messageType and origin are all nullable today.

• What if a DataAvailable request, from servicebus, fails in validation in PO ? should we notify sub-PO ?
• How do we completly upgrade to .net 5 with isolated functions ?
• Database - should we have 2 cosmos, one for dequeued items and one for active
• Peek Architechture
• How to test, do we used squadron? - cosmos ?
• Which format will data in BlobStorage have? Do sub domains save data as CIM XML/EbiX or do PO need to do this?
• How should we calculate priority and find the data we should request for at subdomain, when filtering data from dataavailable cosmos ? And how many records should we look up ?
• Skal vores GetMessageHandler returnere en "string" der repræsenterer indholdet, eller skal den returnere et Objekt der indeholder information omkring processen, altså om kaldet er gået godt ? Det er mere et spørgsmål om vi skal kaste exceptions når data f.eks. ikke er klart endnu. vi kunne sende en god besked tilbage også må vores function stå for at levere korrekt http fejl koder.
• Is recipient GLN or another supplier identification?
• Is messageType, pageSize supplied by caller when peeking?

Handle dequeue message from actor.
Mark saved data as dequeued.
Notify sub domain PO for dequeued data.

• Save message to database.

Following projects are missing Nullable attribute in .csproj: Energinet.DataHub.PostOffice.Inbound, Energinet.DataHub.PostOffice.Outbound

The PO solution has not been sufficiently covered with tests. Improve test coverage.

Current Behavior

Deprecated package WindowsAzure.Storage

Possible Solution

Use newer Azure.Storage.Blobs package

• Read available data-messages related to caller.
• Group UUIDs/data if possible.
• Test

• We need to set up Terraform.

• Github Actions

• DataAvailable function

• Peek function

• Cosmos db

• Servicebus DataAvailable

• Update project to .net5
  ...

Create the IaC needed for the post office.
This includes CD pipelines also.

Currently the documentation does not really have any perspective on HOW to get access to the queue (fetching the connection string from the keyvault).
This needs to be added.

• Recieve call from actor.
• Parse data to identify caller.
• Add validation to MediatR/input from market operator.
• Test.

We need to calculate the Priotity for DataAvailable before saving to DB.
Priority should be calculated with Domain, time .... in mind. This needs to be descripted by PO.

• Returns Blob Storage URI.
• Simulate return data from blob storage URI.
• Protobuf format