Latte is a modern data engineering toolkit. Latte provides out of the box config-driven help to reduce the time and cost for many common data engineering tasks, including:
- Load: Load data into data sources for downstream analysis, such as kafka.
- Aggregate: Aggregate operational data at the source to produce business analytics.
- Transform: Transform data during processing.
- Transfer: Transfer data from one system into another maintaining the original state.
- Extract: Extract data from a variety of sources, including Mongo, Postgres and Prometheus.
Latte extracts and loads data from a variety of sources:
![Screenshot 2024-01-29 at 7 15 22 PM](https://private-user-images.githubusercontent.com/151242797/300639371-1f00acda-933d-4786-babe-b52b7378b5b0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1OTA4ODQsIm5iZiI6MTcyNDU5MDU4NCwicGF0aCI6Ii8xNTEyNDI3OTcvMzAwNjM5MzcxLTFmMDBhY2RhLTkzM2QtNDc4Ni1iYWJlLWI1MmI3Mzc4YjViMC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNVQxMjU2MjRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yYTZmMWI5ZDdhNGNhZjM0NGJjYTkxZjAxMGJmNWY4YjU2ZWE5YWViN2Q0NDEwMDA0ZmYzNTU2YTI0YmJjNjRkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9yLoc5xZGNHoQveA5K1mUug8VGoPE0-sJ0l-Kogw6gA)
Latte aims to:
- Reduce cost of analytic data calculation, storage and query.
- Increase data warehouse data fidelity.
- Reduce data lake sprawl.
- Reduce data transfer into and costs of querying datalakes and datawarehouses.
- Enable product engineers to write self-service analytics built on their operational data stores.
Latte is configured using simple yaml configuration files:
![Screenshot 2023-12-24 at 9 05 22 AM](https://private-user-images.githubusercontent.com/151242797/292674689-80a59dcc-4f5e-4dd3-95bb-b9402cb3e6e7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1OTA4ODQsIm5iZiI6MTcyNDU5MDU4NCwicGF0aCI6Ii8xNTEyNDI3OTcvMjkyNjc0Njg5LTgwYTU5ZGNjLTRmNWUtNGRkMy05NWJiLWI5NDAyY2IzZTZlNy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNVQxMjU2MjRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wNjQ3ZDdiYmIxNmIwZWExMDNmNGNiYzkzNmJmZjA0MjNlY2I1ZGE0MzUyZTc4MTA5ZGVlMjUxMDAyODE0ZGZkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.5qn_UqNEXtNiyJw7UxOXkB2koAuNO1ZO5z-Yg1S9Kew)
- Start docker dependencies
docker-compose -f dev/compose.yaml up -d
- Validate Configuration
go run cmd/main.go config validate --config=$(PWD)/dev/examples/postgres.kafka.stdout.yaml
VALID=true
- Invoke collector
go run cmd/main.go config invoke --config=$(PWD)/dev/examples/postgres.kafka.stdout.yaml
- Verify Kafka Output
docker exec -it kafka1 kafka-console-consumer --bootstrap-server=localhost:9092 --topic=latte --from-beginning | jq .
{
"uuid": "4fdbc492-7e76-4ae7-9e32-82da7463f374",
"name": "core.users.total",
"value": 3,
"type": "COUNT",
"tags": {
"customer": "google"
},
"timestamp": "2023-12-24T14:12:01.237538Z",
"grain_datetime": "2023-12-24T00:00:00Z"
}
{
"uuid": "cdc14916-15a4-4579-aa06-2dc65f442aba",
"name": "core.users.total",
"value": 2,
"type": "COUNT",
"tags": {
"customer": "amazon"
},
"timestamp": "2023-12-24T14:12:01.237543Z",
"grain_datetime": "2023-12-24T00:00:00Z"
}
- Start docker dependencies:
docker-compose -f dev/compose.with-collector.yaml up -d
- Validate config
docker-compose -f dev/compose.with-collector.yaml run latte config validate --config=/dev/config/postgres.kafka.stdout.yaml
Creating dev_latte_run ... done
/dev/config/postgres.kafka.stdout.yaml
VALID=true
- Invoke Collector
docker-compose -f dev/compose.with-collector.yaml run latte config invoke --config=/dev/config/postgres.kafka.stdout.yaml
Creating dev_latte_run ... done
{"level":"info","ts":1703725219.4218154,"caller":"collector/collector.go:165","msg":"collector.Invoke","id":"d20161ac-be75-4868-8794-04d7bfa7d9d3","name":"postgres.users.total.24h"}
{"uuid":"a540fb6c-1638-4109-a385-3b0afda6fa12","name":"core.users.total","value":3,"type":"COUNT","tags":{"customer":"google"},"timestamp":"2023-12-28T01:00:19.422545549Z","grain_datetime":"2023-12-28T00:00:00Z"}
{"uuid":"fdac2b3f-a053-4997-b1a3-1ce1c6ca89a4","name":"core.users.total","value":2,"type":"COUNT","tags":{"customer":"amazon"},"timestamp":"2023-12-28T01:00:19.422548216Z","grain_datetime":"2023-12-28T00:00:00Z"}
- Verify Kafka Output
docker exec -it kafka1 kafka-console-consumer --bootstrap-server=localhost:9092 --topic=latte --from-beginning
{"uuid":"a540fb6c-1638-4109-a385-3b0afda6fa12","name":"core.users.total","value":3,"type":"COUNT","tags":{"customer":"google"},"timestamp":"2023-12-28T01:00:19.422545549Z","grain_datetime":"2023-12-28T00:00:00Z"}
{"uuid":"fdac2b3f-a053-4997-b1a3-1ce1c6ca89a4","name":"core.users.total","value":2,"type":"COUNT","tags":{"customer":"amazon"},"timestamp":"2023-12-28T01:00:19.422548216Z","grain_datetime":"2023-12-28T00:00:00Z"}
- Run Collector as daemon
docker-compose -f dev/compose.with-collector.yaml run latte run -c=/dev/config
Creating dev_latte_run ... done
{"level":"info","ts":1703726983.41746,"caller":"cmd/run.go:52","msg":"loading configs","path":"/dev/config"}
{"level":"info","ts":1703726983.4632287,"caller":"cmd/run.go:71","msg":"initialized collectors","num_collectors":5}
{"level":"info","ts":1703726983.4632988,"caller":"service/service.go:27","msg":"run"}
{"level":"info","ts":1703726983.4633727,"caller":"collector/collector.go:165","msg":"collector.Invoke","id":"5ba44984-a8a3-42ac-a70d-85e11f808a6c","name":"postgres.users.total.24h"}
- Tail the audit log, Check Kafka, Verify Vector
tail -f dev/audit/latte.audit.log
Checkout the examples directory for configuration examples.
Additional documentation is available in the docs/ directory
Image by catalyststuff on Freepik