clymene-project / clymene Goto Github PK

the Clymene is time-series data and Logs collection platform for distributed systems.

Home Page: https://clymene-project.github.io

License: Apache License 2.0

Go 95.82% Dockerfile 0.21% Shell 0.67% Makefile 0.22% Batchfile 0.25% Lex 0.35% Yacc 1.67% CSS 0.08% JavaScript 0.15% HTML 0.59%

timeseries tsdb distributed-system metrics monitoring grpc elasticsearch opentsdb platform prometheus-agent

clymene's Introduction

Clymene

The Clymene is a time-series data and logs collection platform for distributed systems inspired by Prometheus and Jaeger. Time-series data and logs from various environments can be collected and stored in different types of databases. It can be configured in a variety of architectures. Choose the storage that users can use easily and build a monitoring system with dashboards that users can use easily. Also, Clymene Agent's time series data collection uses fewer resources than Prometheus' remote_write. Various time series data analysis functions will be added.

Architecture

The Clymene project is a platform for collecting time series data and log data. So there are two pipelines: timeseries and logs. All agents can collect data and store it directly in the database, Or, depending on the situation, you can use the Clymene components to configure a variety of architectures. It can be configured with an architecture that can be used with a lot of traffic. The Clymene Project provides various storage types, so choose a storage type you are familiar with. And build great monitoring systems using familiar dashboards.

Components

Pipeline 1. TimeSeries data collection

Clymene Agent(Getting Started)

dockerhub : bourbonkk/clymene-agent:v2.x.x
redhatQuay: quay.io/clymene/clymene-agent:v2.x.x

The Clymene-agent is service that collects time series data(does not use disks)

Service Discovery
- Prometheus's Service Discovery feature finds Metric collection endpoints.
scrape time series data
Time-series data transfer to gateway(gRPC) (Optional)
Time-series data transfer to kafka (Optional)
Time-series data insert to Database(supported DB) (Optional)

Clymene Ingester(Optional) (Getting Started)

dockerhub : bourbonkk/clymene-ingester:v2.x.x
redhatQuay: quay.io/clymene/clymene-ingester:v2.x.x

The Clymene ingester is an optional service responsible for insert time series data loaded on kafka into the database.

Kafka message consume
Time-series data insert to Database(supported DB) (Optional)

Clymene Gateway(Optional) (Getting Started)

dockerhub : bourbonkk/clymene-gateway:v2.x.x
redhatQuay: quay.io/clymene/clymene-gateway:v2.x.x

The Clymene Gateway is an optional service that can receive metric data from the another component through gRPC or HTTP communication.

gRPC, HTTP Service
Time-series data insert to Database(supported DB) (Optional)

Pipeline 2. Logs collection

Clymene Promtail(Getting Started)

dockerhub : bourbonkk/clymene-promtail:v2.x.x
redhatQuay: quay.io/clymene/clymene-promtail:v2.x.x

The Clymene-promtail customized loki's log collection agent for the Clymene project.

Service Discovery
log collection
log data transfer to gateway(gRPC or HTTP)
log data transfer to kafka
log data insert to Database(supported DB) (Optional)

Promtail Ingester(Optional) (Getting Started)

dockerhub : bourbonkk/promtail-ingester:v2.x.x
redhatQuay: quay.io/clymene/promtail-ingester:v2.x.x

Promtail ingester is an optional service responsible for insert log data loaded on kafka into the database.

Kafka message consume
Time-series data insert to Database(supported DB) (Optional)

Promtail Gateway(Optional) (Getting Started)

dockerhub : bourbonkk/promtail-gateway:v2.x.x
redhatQuay: quay.io/clymene/promtail-gateway:v2.x.x

The Promtail Gateway is an optional service that can receive log data from the another component through gRPC or HTTP communication.

gRPC, HTTP Service
Time-series data insert to Database(supported DB) (Optional)

How to work it on k8s

For example, see the k8s example

How to work it on docker-compose

For example, see the docker-compose example

Roadmap

AI/ML platform for clymene(https://github.com/Clymene-project/clymene-analyzer)
push-type agent(node-exporter, cadvisor, process-exporter, etc)
Add new pipeline(trace or Something - discussions)
eBPF trace agent(for v3.0.0 release)

Get in Touch

Have questions, suggestions, bug reports? Reach the project community via these channels:

GitHub issues and discussions

License

Apache License 2.0, see LICENSE.

Code Style

1. https://github.com/golang/go/wiki/CodeReviewComments

2. https://peter.bourgon.org/go-in-production/#formatting-and-style

supported

The Clymene project is being developed with the support of Open Source development license of Jetbrains'.

clymene's People

Contributors

Stargazers

Watchers

clymene's Issues

add kafka writer

eBPF trace agent architecture design

Design a trace agent using eBPF

clymene-agent using ebpf?

I'm studying ebpf technology, so should I collect all the metrics at the Node level using this?

grpc reporter

promtail - log data transfer to kafka(using ingester)

Cortex option. Separation.

Separate the prometheus option and the cortex option to block confusion

add opentsdb writer

add gRPC gateway

CrashLoopBackOff Issue

Hello.

First of all, thank you again for your modification that "loki.yaml" namespace problem.
I am very sorry that bother you again.

I am not familiar with k8s system however, trying to solve it so It will be very appreciate for your support.

Currently faced with CrashLoopBackOff Issues and having struggle with it.
Enveloped photos for detail.

Sincerely thank you for your efforts and I'll look forward to your reply.

Add service discovery

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

After saving data from gateway to kafka, iningester cannot be consumed

Describe the bug
A clear and concise description of what the bug is.
When the storage_type of gateway is set to kafka and storage_type of ingester is set to prometheus, abnormal operation occurs.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

ingester logs

Desktop (please complete the following information):

OS: [e.g. iOS] fedora
Browser [e.g. chrome, safari]
Version [e.g. 22] 36

Additional context
Add any other context about the problem here.

clymene-agent

kafka write
prometheus write
cortex write
elasticsearch write

clymene-ingester(Optional)

consume kafka message
write prometheus, cortex, elasticsearch

But it seems like I have misconfigured the app but could not find the solution.

So the issue is that when I configure the prometheus remote url the agent saids that the option does not exist.

Were there any updates?

below is my deployments.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: clymene-agent
  namespace: clymene
  labels:
    app: clymene-agent
spec:
  selector:
    matchLabels:
      app: clymene-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: clymene-agent
    spec:
      containers:
        - name: clymene-agent
          image: bourbonkk/clymene-agent:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 15691
            - containerPort: 15692
          args:
            - --config.file=/etc/clymene/clymene.yml
            - --prometheus.remote.url=http://prometheus.monitors.svc.cluster.local:9090/api/v1/write
            - --log-level=info
          env:
            - name: TS_STORAGE_TYPE
              value: kafka
          volumeMounts:
            - mountPath: /etc/clymene/
              name: config-volume
      volumes:
        - name: config-volume
          configMap:
            name: clymene-agent-config
      securityContext:
        runAsUser: 1000

And the error is

Error: unknown flag: --prometheus.remote.url

Is there something wrong with my yaml file? or did I miss something?

If you have time please let me know : )