axiomhq / axiom-syslog-proxy Goto Github PK

View Code? Open in Web Editor NEW

8.0 3.0 4.0 841 KB

A syslog push interface to Axiom.

Home Page: https://axiom.co

License: MIT License

Go 94.64% Dockerfile 0.54% Makefile 4.82%

syslog axiom

axiom-syslog-proxy's Introduction

Axiom unlocks observability at any scale.

Ingest with ease, store without limits: Axiom's next-generation datastore enables ingesting petabytes of data with ultimate efficiency. Ship logs from Kubernetes, AWS, Azure, Google Cloud, DigitalOcean, Nomad, and others.
Query everything, all the time: Whether DevOps, SecOps, or EverythingOps, query all your data no matter its age. No provisioning, no moving data from cold/archive to "hot", and no worrying about slow queries. All your data, all. the. time.
Powerful dashboards, for continuous observability: Build dashboards to collect related queries and present information that's quick and easy to digest for you and your team. Dashboards can be kept private or shared with others, and are the perfect way to bring together data from different sources.

For more information check out the official documentation and our community Discord.

Usage

💡 Setting up axiom-syslog-proxy for secure syslog? Think about using an endpoint! You can create one by visiting the Axiom endpoint settings.

There are multiple ways you can install the Axiom Syslog Proxy:

With Homebrew: brew install axiomhq/tap/axiom-syslog-proxy
Download the pre-built binary from the GitHub Releases
Using Go: go install github.com/axiomhq/axiom-syslog-proxy/cmd/axiom-syslog-proxy@latest
Use the Docker image: docker run axiomhq/axiom-syslog-proxy

Configuration

If you use the Axiom CLI, run eval $(axiom config export -f) to configure your environment variables.

Otherwise create an API token with the appropriate scopes in the Axiom API token settings and export it as AXIOM_TOKEN.

Export the dataset name to ingest into as AXIOM_DATASET. The dataset must exist prior to ingesting data into it.

License

Distributed under the MIT License.

axiom-syslog-proxy's People

Contributors

Stargazers

Watchers

Forkers

gordallott thecraftman smorimoto oboratav

axiom-syslog-proxy's Issues

Syslog server can't parse my file

Hi there!

I followed this instruction for install axiom syslog server: https://axiom.co/docs/send-data/syslog-proxy and I tested with echo -n "tcp message" | nc -w1 localhost 601 and everything is gone fine!

I setup in my /etc/syslog.conf I configured send logs into my syslog server

#maas config
module(load="imfile")
module(load="imtcp")
input(type="imtcp" port="601")
$InputFileName /var/snap/maas/common/log/rsyslog.log
$InputFileTag maas-logs
$InputFileStateFile maas-logs-state
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor

local7.* @localhost:601

But seens Axiom server can't parse my file:

docker logs f551b472a6f5

{"level":"info","ts":1710446128.4934351,"logger":"axiom-syslog-proxy","caller":"cmd/cmd.go:71","msg":"starting","release":"0.7.0","revision":"40b6431","build_date":"2023-08-16T11:44:20Z","build_user":"goreleaser","go_version":"go1.21.0"}
{"level":"info","ts":1710446128.4936218,"logger":"axiom-syslog-proxy","caller":"cmd/cmd.go:106","msg":"started"}
[19:55:28.49] info  axiom-syslog-proxy@logs/input udp.go:18 Started UDP server on udp:[::]:514
[19:55:28.49] info  axiom-syslog-proxy@logs/input tcp.go:23 Started TCP server on tcp:[::]:601
2024/03/14 20:07:34 Unable to parse log line: tcp message
2024/03/14 20:07:39 ingested 1 event(s)
2024/03/14 20:07:40 Unable to parse log line: udp message
2024/03/14 20:07:43 ingested 1 event(s)
2024/03/18 13:25:55 Unable to parse log line: tcp message
2024/03/18 13:25:58 ingested 1 event(s)

How can I handled with that?

[Feature request] Add support for organisation token as well

{
  "level": "info",
  "ts": 1691260032.7793918,
  "logger": "axiom-syslog-proxy",
  "caller": "cmd/cmd.go:71",
  "msg": "starting",
  "release": "0.4.1",
  "revision": "92d4c9b",
  "build_date": "2021-11-08T10:14:39Z",
  "build_user": "goreleaser",
  "go_version": "go1.17.2"
}

{
  "level": "error",
  "ts": 1691260032.7797287,
  "logger": "axiom-syslog-proxy",
  "caller": "cmd/cmd.go:94",
  "msg": "create axiom client",
  "error": "invalid access token"
}

{
  "level": "warn",
  "ts": 1691260032.7798183,
  "logger": "axiom-syslog-proxy",
  "caller": "cmd/cmd.go:60",
  "msg": "stopped"
}

Broken Go install command in README.md

The Go install command currently listed on the readme file (go install github.com/axiomhq/axiom-syslog-proxy/cmd/axiom-syslog-prox@latest) is broken:

The correct url should be github.com/axiomhq/axiom-syslog-proxy/cmd/axiom-syslog-proxy@latest.

JSON type null is unsupported

I am using Axiom Cloud, and I am attempting to stream logs from HashiCorp Vault with this syslog proxy and a socket audit log on Vault's end. I've configured the Vault resource with the following Terraform:

resource "vault_audit" "axiom" {
  options = {
    format      = "json"
    address     = "127.0.0.1:601"
    socket_type = "tcp"
  }
  type        = "socket"
  path        = "axiom"
  description = "Ingestion to Axiom"
  local       = false
}

When I perform actions in Vault, I get the following errors from this service:

[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.18] warn  [email protected] parse.go:237 JSON type null is unsupported
[03:23:57.19] warn  [email protected] parse.go:237 JSON type null is unsupported

Nothing appears on Axiom's end, however when I enable a prefix such as vault I continue getting the JSON error but I see the entries appear in Axiom (malformed due to the prefix).

How can I best troubleshoot why this service is unable to parse the JSON that Vault is sending? Why do the logs successfully send when a prefix string is prepended to the JSON blob, breaking the ability to index on fields?

axiomhq / axiom-syslog-proxy Goto Github PK

axiom-syslog-proxy's Introduction

Usage

Configuration

License

axiom-syslog-proxy's People

Contributors

Stargazers

Watchers

Forkers

axiom-syslog-proxy's Issues

Syslog server can't parse my file

[Feature request] Add support for organisation token as well

Broken Go install command in README.md

JSON type null is unsupported

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent