parseablehq / collector Goto Github PK

View Code? Open in Web Editor NEW

16.0 16.0 3.0 127 KB

Kubernetes application log collector for Parseable server

Home Page: https://www.parseable.io/

License: GNU Affero General Public License v3.0

Dockerfile 1.13% Makefile 2.61% Go 89.53% Smarty 6.73%

golang kubernetes logging

collector's People

Contributors

Stargazers

Watchers

Forkers

nitisht adheipsingh pramine

collector's Issues

Add meta_labels for pod labels and send it in request

Support parseable style jwt auth

kubec should add jwt for each request sent to Parseable server.

Ensure kube-collector uses last log's timestamp on restart

This will need an API on Parseable server to fetch a logstream's last log

Debug 413 error code while sending request

Failed to push logs to parseable server

req: &{GET http://parseable.default.svc.cluster.local/api/v1/query HTTP/1.1 1 1 map[Authorization:[Basic cGFyc2VhYmxlOnBhcnNlYWJsZQ==] Content-Type:[application/json] User-Agent:[Parseable collector ( │
│ time="2022-08-13T12:37:20Z" level=error msg="unexpected status code: 405 while querying log data timestamp in stream: demoapp"

Write HTTP unit tests

Create HTTP interface

create http interface to support all CRUD operations.

Running Collector Locally

clean way to run collector locally

PodList should return []v1.Pod NOT v1.PodList

its better to calculate slice of pods at source rather than call the function and then optimise.

Configuration to pull logs from all namespaces and all pods in a namespace

In mem store will need a clean up with dead pods

Send Metadata to parseable in headers

curl --location --request POST 'https://demo.parseable.io/api/v1/logstream/frontend' \
--header 'X-P-META-Host: 10.116.0.3' \
--header 'X-P-META-Source: 10.244.0.147' \
--header 'X-P-META-ContainerName: log-generator' \
--header 'X-P-META-ContainerImage: mingrammer/flog' \
--header 'X-P-META-PodLabels: app=go-app,pod-template-hash=6c87bc9cc9' \
--header 'X-P-TAGS-AppName: go-app' \
--header 'X-P-TAGS-System: golang' \
--header 'Authorization: Basic password' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "id": 2,
        "host":"153.10.110.81", 
        "user-identifier":"-", 
        "datetime":"24/Jun/2022:14:12:15 +0000", 
        "method": "DELETE", 
        "request": "/virtual/drive", 
        "protocol":"HTTP/2.0", 
        "status":500, 
        "bytes":21969, 
        "referer": "http://www.seniordisintermediate.net/mesh/users"
    }
]

Ensure Parseable creds are always present in config before collector startup

Rename Labels to Tags while sending request

How to handle large line logs

Implement batch, if a size is greater X mb, write to file and upload.

Stop Channel and Sigterm for graceful shutdown of go routine

Add validation for stream name

if stream exists already response is 400
if stream name does have "-" or any character response is 400.
Add validation for string for stream name and exit 1.