Collectors should be scheduled at the desired intervals, and support cron expressions.

• Create a new "run" command
• Accepts a directory of configuration files as CLI args
• Invokes each collector on initialization to ensure data is collected

The first version should be stateless and config file driven.
The config files will be loaded, scheduled and executed at the intervals specified.

The next version should introduce some sort of persistence and a REST interface on top of the scheduler. Persistence will also be necessary for oauth2 sources. The access tokens will need to be stored and mapped.

Right now an in memory duckdb state store is used for prometheus POC.

Signals collector should support a persistent state store.

Pass metric through a configurable serializer

It should default to JSON

Transformations will most likely be expanded. Right now the following transforms are supported internally:

• Adding Metric Name
• Adding metric type
• Adding metric tags
• Adding grain datetime

for _, m := range ms {
  for _, tFunc := range transforms {
     tFunc(m)
  }
}

• Project Goals
• Approach
• Getting Started
• Quickstart
• Examples

Many of the sinks need cleanup, such as kafka.

Make Close() part of the interface and call it on service shutdown.

The data has already been collected, make a configurable number of retry attempts with backoff and jitter.

Ability to collect partitioned data and sink it to target systems.

Examples include issueing S3 blob storage loads into a system like hive or clickhouse.

Right now each metric contains a time field containing the time the metric was generated. I'm guessing many output sources will be interested in the interval-aligned time.

Consider the following query that counts users by customer:

     SELECT 
        account as customer,
        COUNT(*) as value
      FROM 
        users
      GROUP BY
        account

Each row will produce a separate metric:

{
  "uuid": "55ef5b32-271f-496a-8cde-27924d7f67aa",
  "name": "core.users.total",
  "value": 3,
  "type": "COUNT",
  "tags": {
    "customer": "google",
    "env": "prod"
  },
  "time": "2023-12-23T20:53:57.150929Z"
}
{
  "uuid": "8fcaf7da-76c2-4152-b097-9e15bc47c8cf",
  "name": "core.users.total",
  "value": 2,
  "type": "COUNT",
  "tags": {
    "customer": "amazon",
    "env": "prod"
  },
  "time": "2023-12-23T20:53:57.150934Z"
}

Most of the time aggregating on time will work. Imagine that this collector runs 1x a day. While it's likely that all metrics will contain a time that is in the same day, it's not guaranteed.

An interval aligned time field can introduce a logical time bucket that all metrics in a data collection roll up to.

• Historic Tumbling Window -> Collects data across all historic fully complete windows. Provides windowing guarantees, i.e. counting by day, hour, week, etc.
• Tick - Invokes collector at point in time, great for running COUNTS like total users, and GAUGEs since it's self correcting. Captures a reading at a moment in time.
• Incremental - Invokes collector starting from last previous invocation until now. Fully captures all available data but no windowing guarantees.

• Load
• Aggregate
• Transform
• Transfer
• Extract

☕ - Latte turbo charges your data engineering teams by providing out of the box support for common data operations.

Each metric should contain a UUID to dedup in the case of retries.

https://github.com/uber-go/zap

Data durability approach for first version:

• Persistence
• Retry
• Resumption

• Add configurable Source Retry Option

The service should support restulf API operations. One of which should be the ability to query the the state store, especially in the case of the memory state store.

This should include a List() method on the statestore interface and expose this through an HTTP api

Ability to globally template the config file. While users can handle this on their end, it should reduce the barrier to entry to allow go template templating support.

Ability to collect entities and not metrics.

Entities are {}interfaces, or map[string]any types.

These have different rules for persistance as well, sometimes a "snapshot" will be created, where all entities are persisted to a single location (such as s3).

Mongo Source