Add Unit Tests with Mocha and Chai

• update in README index
• update in docsify sidebar

TODO:

• util to remap type + externalId to core urn (https://developer.ard.de/core-api-v2-delivering-content#CoreID)
  • input: PermanentLivestream + externalId/222/555 => urn:ard:permanent-livestream:058e528e18310561
• util to remap core urn to pubsub-safe topic id/ label value (url encode)
  • input: urn:ard:permanent-livestream:058e528e18310561 =>
    urn%3Aard%3Apermanent-livestream%3A058e528e18310561
• remap externalIds in POST body

changes in post body:

{
  ...
  services: [
    { 
      type: "PermanentLivestream",
      crid: "externalId/222/555",
      publisherId: 123450
    }
  ],
  ...
}

remap in eventhub to:

{
  ...
  services: [
    {
      id: "urn:ard:permanent-livestream:058e528e18310561",
      publisherId: 123450
    }
  ],
  ...
}

First request of a new event will return "failed": 1 as response that the new topic has been created:

"statuses": {
  "published": 0,
  "blocked": 0,
  "failed": 1
 },

Second request of a new event will return "published": 1 as response that event has been published:

"statuses": {
  "published": 1,
  "blocked": 0,
  "failed": 0
 },

Currently updating node-crc to a version > 2.0.0 fails: magiclen/node-crc#8

Getting Clone found errors from openapi.yaml, ignoring the file with .jscpd.json fails:

• kucherenko/jscpd#496 (comment)

c.f. https://github.com/swrlab/ard-eventhub/pull/277/files

z.B. "externalId": "crid://swr.de/1234567" oder auch alternative IDs wie z.B.

• urn:ard:show:027708befb6bfe14
• crid://br.de/broadcastSeries/1235
• brid://[email protected]/editorialobject/1f11ad6d-e94b-4c9a-b303-dbf5f600cc04
• Bei übergreifenden SRs z.B. auch https://normdb.ivz.cn.ard.de/sendereihe/427
  – wobei die ARD Core Show ID (urn:ard:show:027708befb6bfe14) natürlich für die Audiothek am besten passen würde...

e.g. using npm-get-version-action in:

-> see internal Lab chat

curl -X 'POST' \
  'https://eventhub-ingest.ard.de/events/de.ard.eventhub.v1.feed.update' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "event": "de.ard.eventhub.v1.feed.update",
  "type": "feed",
  "url": "https://www.tagesschau.de/xml/rss2/",
  "lastBuildDate": "Mon, 20 Feb 2023 11:55:10 +0100",
  "modifiedAt": "2023-02-20T11:55:10+0100",
  "services": [
    {
      "type": "Publisher",
      "id": "urn:ard:publisher:58e1fa85b917c91c",
      "externalId": "10105",
      "title": "Tagesschau"
    }
  ],
  …
}'

TODOs / TBD:

• include new/changed item GUIDs?

We would have expected that we always get an identifier for connecting events with the same title. As both externalId and hfdbIds are optional fields, is there any mandatory alternative we can use for this purpose?

2023-03-22T15:41:29 (= local time) sent at 2023-03-22T15:41:36.956Z

Relates to #348

Noticed during upgrade procedure for version 1.5.0

{
	"data": {
		"statuses": {
			"blocked": 0,
			"failed": 0,
			"published": 1
		},
		"event": {
			// ...
			"created": "2023-03-22T15:41:36.956Z",
			"length": 230,
			"name": "de.ard.eventhub.v1.radio.track.playing",
			"start": "2023-03-22T15:41:29",
			"id": "6640889577340928",
			// ...
			"event": "de.ard.eventhub.v1.radio.track.playing",
			// ...
			"type": "music"
		}
	},
	"stage": "prod",
	"level": "notice",
	"nodeVersion": "v18.12.1",
	"source": "ingest/events/post",
	"serviceName": "eventhub-ingest",
	"version": "1.4.1"
}

Getting Clone found errors from openapi.yaml, ignoring the file with .jscpd.json fails:

• kucherenko/jscpd#496 (comment)

• On request POST /event verify passed in Service-IDs
• On Pub/Sub-Error topic doesn't exist ask ARD Core-API
• Implement ARD Core-API and get Service-IDs from Institution/Publishers
• Create Pub/Sub-Topics for verified Service-IDs
• Define Request-Answer (new topic created???)

Publisher-ID 10602 (prev 99024)

swrlab/swr-prettier-config#2

https://github.com/swrlab/ard-eventhub/blob/main/README.md?plain=1#L91

14.15 includes

• openssl/libssl1.1 Integer Overflow or Wraparound https://app.snyk.io/vuln/SNYK-ALPINE311-OPENSSL-1075738

Container Registry will be phased out, starting May 15, 2024. Please review the options below for how to upgrade your projects to Artifact Registry

We’re writing to let you know that Container Registry will be phased out starting May 15, 2024, and replaced by Artifact Registry, which offers a variety of new capabilities.

What do you need to know?

Starting May 15, 2024, we’ll limit the projects allowed to host images in Container Registry. The only projects allowed to host the images will be the ones in which:

1. The Container Registry API was enabled before May 15, 2024.
2. Images were pushed to Container Registry.
   For all other projects, once the discontinuation starts, images will be automatically hosted in Artifact Registry, and you will no longer have the option to host images in Container Registry. Container Registry is in maintenance and only receives critical legal or security fixes.

Changes that break functionality should trigger alerts in monitoring

Concerns

We should add some docs about the default configuration for subscriptions

• endpoint should always be available
• Message Retention
• Ack deadline

When subscribing to the different topics, we receive many different formats for the "start"-property in the "data"-Element. We would have expected to receive 1 format or at least patterns following the same ISO-Standard.
for example:
YouFM: 2022-06-15T08:36:33 (LocalDateTime)
MDR Sachsen: 2022-06-24 11:16:58 (LocalDateTime)
1Live: 2022-06-24T11:40:42.482+02:00 (OffsetDateTime)

Our expectation was to always receive an OffsetDateTime.

Create prettier version of docs using external tool (docsify?!) to display them via gh pages

https://docsify.js.org/#/

https://docsify.js.org/#/deploy?id=github-pages

Migrate and check its working

• dd-trace: 1.7.0 -> 2.7.0
• dotenv: 14.3.2 -> 16.0.0
• google-auth-library: 7.14.1 -> 8.0.2

Change source of package

• undici-wrapper frytg -> node-utils

Requires Rust-Toolkit in Docker

• node-crc: 1.3.2 -> 2.0.11

Requires ES6-Module

• ansi-regex: 5.0.1 -> 6.0.1
• strip-ansi: 6.0.1 -> 7.0.1

Lately the following error comes upon boot:

formats as an array is deprecated. Use object instead https://ajv.js.org/options.html#formats

Might have todo with express-openapi-validator update to v5.0.0

ToDo: Create a new topic (per env/ stage) for all published Events in Eventhub for easier collection by external ARD services

Name: to be discussed

de.ard.eventhub.test.subscription.01338cc4-1afa-4ab4-ab8e-cb2ab91e3af2
de.ard.eventhub.test.subscription.8cc6c0d8-4545-4b79-b78f-e07e9f7a0bfa

• Auth responses are missing in endpoints

  • /events
  • /topics

• Unify auth naming for

  • authentication
  • authorization

Add European Union Public License 1.2 to package.json/openapi docs

• Introduction to the EUPL licence
• license text
• EUPL-1.2 at SPDX
• overview at choosealicense.com
• 👍 GPL compatible

Full name
European Union Public License 1.2

Short identifier
EUPL-1.2

Clean up all expired/unused docker images from registry!

MDR currently sets in the fields externalId and playlistItemId the same identifier, playlistItemId is just prefixed by the station.

playlistItemId: MDR_4_457059C5
externalId: 457059C5

As a consequence, either the playlistItemId is not specific to the playout (connecting upcoming and playing event) or the externalId is not specific to the title (connecting events with the same title).

• music
• news
• live
• weather
• traffic
• audio
• commercial
• jingle

Eventhub is currently using a local ARD core dump, this should be migrated to actually calling the ARD Core API (via internal or public host)

Requirement: ARD Core API to be available publicly - this is required for non-cluster-internal CI runs (e.g. on Github)

• Is the field "id": "urn:ard:permanent-livestream:49267f7d67be180d" still valid in request body example of /events?
• Is it possible to alter eventV1ResBody scheme in /events (next) so request body example matches "event": "de.ard.eventhub.v1.radio.track.next" instead of "event": "de.ard.eventhub.v1.radio.track.playing"?
• Is it possible to alter eventV1ResBody scheme in /events response body examples to add fields like name and created?

Getting Clone found errors from openapi.yaml, ignoring the file with .jscpd.json fails:

• kucherenko/jscpd#496 (comment)

(plus more endpoints)

See https://validator.swagger.io/validator/debug?url=https%3A%2F%2Feventhub-ingest.ard.de%2Fopenapi%2Fopenapi.json