molu8bits / s3bucket_exporter Goto Github PK

S3 bucket exporter exposing metrics size and number of objects

License: MIT License

Go 96.66% Dockerfile 3.34%

s3 molu8bits s3bucket exporter s3-exporter s3bucket-exporter

s3bucket_exporter's Introduction

S3bucket Exporter

s3bucket_exporter collects informations about size and object list about all the buckets accessible by user. Was designed to work with ceph, but should work will all S3 compatible endpoints.

Badges

Getting started

Run from command-line:

./s3bucket_exporter [flags]

Run from command-line - example with minimal parameter list:

./s3bucket_exporter -s3_endpoint=192.168.0.1:7480 -s3_access_key=akces123 -s3_secret_key=secret123 -s3_name=MyS3Endpoint

Run as docker container - example for local s3-like buckets with ssl disabled:

docker run -p 9655:9655 -d -e LISTEN_PORT=:9655 -e S3_DISABLE_SSL=True -e S3_ENDPOINT=192.168.0.1:7480 -e S3_ACCESS_KEY=akces123 -e S3_SECRET_KEY=secret123 -e S3_NAME=MyS3Endpoint docker.io/molu8bits/s3bucket_exporter:0.3

Run from command-line - example for AWS /Please note that you need to have buckets only in one region. Otherwise script will fail with message "...BucketRegionError: incorrect region, the bucket is not in ..."/

./s3bucket-exporter -s3_endpoint https://s3.us-east-1.amazonaws.com -listen_port :9655 -s3_access_key ABCD12345678 -s3_secret_key mySecretKey -s3_name=My2S3Endpoint --s3_region=us-east-1 --s3_force_path_style=false --s3_disable_endpoint_host_prefix=true

The exporter supports two different configuration ways: command-line arguments take precedence over environment variables.

As for available flags and equivalent environment variables, here is a list:

environment variable	argument	description	default	example
S3_NAME	-s3_name	S3 configuration name, visible as a tag		MyS3Endpoint
S3_ENDPOINT	-s3_endpoint	S3 endpoint url with port
S3_ACCESS_KEY	-s3_access_key	S3 access_key (aws_access_key)		myAkcesKey
S3_SECRET_KEY	-s3_secret_key	S3 secret key (aws_secret_key)		mySecretKey
S3_REGION	-s3_region	S3 region name	default	"default" or "eu-west-1"
LISTEN_PORT	-listen_port	Exporter listen Port cluster	:9655	:9123
LOG_LEVEL	-log_level	Log level. Info or Debug	Info	Debug
S3_DISABLE_SSL	-s3_disable_ssl	If S3 endpoint is not secured by SSL set to True	False	True
S3_DISABLE_ENDPOINT_HOST_PREFIX	-s3_disable_endpoint_host_prefix	Disable endpoint host prefix	False	True
S3_FORCE_PATH_STYLE	-s3_force_path_style	Force use path style (bucketname not added to url)	True	False

Warning: For security reason is not advised to use credential from command line

Prometheus configuration example:

  - job_name: 's3bucket'
    static_configs:
    - targets: ['192.168.0.5:9655']

Grafana

Grafana dashboad ([resources/grafana-s3bucket-dashboard.json] (resources/grafana-s3bucket-dashboard.json)):

s3bucket_exporter's People

Contributors

Stargazers

Watchers

Forkers

ramp23 hipexbv worteks borovoykirill wkloucek chetansomkuwar lyqwaterway bjorns163 evelynmitchell

s3bucket_exporter's Issues

Multiple buckets

What if you want to monitor multiple buckets spread over multiple accounts. Is there a way to do this without having to depoly multiple Dockers?

caps for ceph account

Hello @molu8bits . you said in the readme that this project was initially made to work with Ceph. im currently trying to use it for it however i kept getting empty information from my setup:

level=debug msg="s3metrics read from s3_endpoint :{Test-bucket2 true 0 0 []}"

im wondering if you know what type of capabilities the user needs to have in order to get the right information about the buckets.

also, i dont know what should be the default values for this parameters to work with ceph:
S3_DISABLE_ENDPOINT_HOST_PREFIX
S3_FORCE_PATH_STYLE

kind regards

cannot make it work on S3 compatible storage

Hi,
I don't manage to make it work on a s3 compatible storage that is not AWS.
On my local computer on windows 10, docker desktop and Git Bash, I do :

docker run -p 9655:9655 -d -e LISTEN_PORT=:9655 -e S3_DISABLE_SSL=False -e S3_ENDPOINT=https://s3.waw2-1.cloudferro.com -e S3_ACCESS_KEY=xxxxxxxxx -e S3_SECRET_KEY=xxxxxxxxxxxxxxxx -e S3_NAME=s3_cloudferro -e S3_DISABLE_ENDPOINT_HOST_PREFIX=True -e LOG_LEVEL=Debug -e S3_FORCE_PATH_STYLE=True docker.io/molu8bits/s3bucket_exporter:1.0.2

Then I do:
curl -v http://127.0.0.1:9655/metrics

I get:

*   Trying 127.0.0.1:9655...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 9655 (#0)
> GET /metrics HTTP/1.1
> Host: 127.0.0.1:9655
> User-Agent: curl/7.84.0
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:07:56 --:--:--     0

In the logs I have:

time="2024-04-12T10:13:08Z" level=info msg="Beginning to serve on port:9655"
time="2024-04-12T10:13:08Z" level=info msg="s3 name 's3_wekeo' available at s3 endpoint 'https://s3.waw2-1.cloudferro.com' will be monitored"
time="2024-04-12T10:13:08Z" level=info msg="listenPort ::9655"

In the docker statistics, after one minute or so, the CPU usage increases and the network input increases by ~3 Mb/sec, reaching 500 Mb after like 2-3 minutes.

S3_NAME is not supposed to match with something, right ? It's just a string that I choose.
Do you see any mistake ?

Thank you

Counts only max 1000 objects per bucket

How to set the endpoint url for aws s3?

If I knew how I would submit a PR for the README because this is really hard to figure out for someone who doesn't program and just wants to monitor s3 buckets.

I've tried all sorts of combinations I found in various docs like this one and this one.

I get different errors depending on what url I try but here's on example using S3_ENDPOINT=https://bucket-name.eu-central-1.s3.amazonaws.com:443;

time="2021-05-11T21:34:15Z" level=info msg="Beginning to serve on port:9655"
time="2021-05-11T21:34:15Z" level=info msg="s3 name 'eu-central-1' available at s3 endpoint 'https://bucket-name.eu-central-1.s3.amazonaws.com:443' will be monitored"
time="2021-05-11T21:34:15Z" level=info msg="listenPort ::9655"
Connection to S3 endpoint failed : RequestError: send request failed
caused by: Get https://bucket-name.eu-central-1.s3.amazonaws.com:443/: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not bucket-name.eu-central-1.s3.amazonaws.com
panic: inconsistent label cardinality: expected 1 label values but got 0 in []string(nil)

goroutine 10 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
        /go/src/github.com/prometheus/client_golang/prometheus/value.go:107
main.S3Collector.Collect(0xc000088540)
        /build/main.go:84 +0xd3b
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
        /go/src/github.com/prometheus/client_golang/prometheus/registry.go:446 +0x19d
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
        /go/src/github.com/prometheus/client_golang/prometheus/registry.go:457 +0x571

Trying to do a docker-compuse up

While i have filled the docker-compuose.yml with wasabi endpoint, etc.
When i try to run it, i am given the following error:

ysername@servername:~/s3bucket_exporter-master$ sudo docker-compose up
ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.s3bucketexporter.environment.S3_DISABLE_SSL contains false, which is an invalid type, it should be a string, number, or a null