library-events-consumer

• Setup service that subscribes to Kafka stream and writes events to the database
• Log progress Info and Error events
• Log Metrics to measure perf

Ability to scale horizontally

Consider cloud infra re-deployment on helm changes

TBD

Service: api-library-events

Implement a service responsible for internal library data storage and usage:

• init database
• provide CRUD operations as REST/gRPC/GQL or other approach
  • write library events into db (called consumer)
  • read library events from db (called by UI layer, could be called by another Client API layer if needed)
• in future, consider switching to push model + websockets (might be a different service)
• log info events and errors
• metrics to measure perf

Considering ELK stack

• Expose access to internal services to log
• Expose access to client UI App to log

See if frontend + backend logging can be tied to a sessionId and being traceable end to end

Setup secret management tools to store all secrets
Candidates: Doppler or Vault

• Define tree hierarchy for secrets locations, e.g.

/{project}/{service}/{variable}/{env /w default}

e.g.

• /vortexia/lib-souce-kafka-producer/some-secret/default
• /vortexia/lib-souce-kafka-producer/some-secret/stage

Services to consider

• logger service: creds/auth
• database: creds
• producer: kafka auth? or might be an internal service
• consumer: internal service, or creds to access API service
• API service
  • db access

• basic react / next / etc app
• fetch events from API service /w polling
• show paginated list of events with auto-scroll / live feed

library-events-producer

• Setup a service that polls library events every X minutes and posts as events into Kafka messaging
• Events should be idempotent: kafka should dedupe
• Log Events and Errors
• Log Metrics to measure perf

Optional Extra

• Consider availability, should be able to run multiple pods, preferably avoiding duplicate events*