Tool performance-related issue about prometheus-nginxlog-exporter HOT 5 CLOSED

Each time the tool accesses the log, does it scan/parse the log file EACH time from the beginning? OR from the point it last accessed the log?

The log file is always opened at its end. If you point the exporter to a big log file, it will not parse the entire file, but instead start reading from the end.

What happens when Nginx is very busy and is writing a lot to the log?

The exporter works with a single Goroutine and should not consume more than one CPU core. If Nginx writes faster than the exporter can read, the metrics will lag behind. Apart from that, nothing bad should happen.

Have you encountered performance issues in the past? Were you able to run any sort of performance tests?

I am intending to use this tool in a container. What are the CPU/RAM resources recommended?

I am running this exporter in a production setup that receives about 11k reqs/min. In that setup, the exporter consumes about 2.4% of a CPU core, around 200 MiB of virtual memory and 38 MiB resident memory.

Hope that helps!

from prometheus-nginxlog-exporter.

from prometheus-nginxlog-exporter.

Hi there!

I have performance troubles with this module(1.7.1 version) while using it with Nginx(1.19) in docker-compose. I use Nginx docker image and a module docker image.

The module config file is very simple and works well. I use syslog, as u can see, but also I try the usual log file and it does not change anything.

Nginx config also is simple, also works well.

Prometheus sends GET requests to module each 120s.
The backend server get about 5-6 RPS from users. Not heavy load.

But module collects all Nginx logs and keep it in RAM and append new, then send all these logs(old and new) to Prometheus, but not delete already send data from his memory.

And after a few minutes I have smth like this:

After few hours\days i have this picture:

What I do wrong? Maybe some mistakes in config or smth like this?

Thanks!

from prometheus-nginxlog-exporter.

@AndreiDrang Looks like you're adding the response time value as label. This will result in the exporter creating a new time series for each request (only grouping those with absolutely identical response_time and upstream_*_time values).

Please drop the relabel rules for response_time and upstream_*_time. These are observations (and as such are processed by the exporter automatically as metric values) and make no sense to be exported as label.

from prometheus-nginxlog-exporter.

@martin-helmich Thanks! Everything looks better after relabels delete.

from prometheus-nginxlog-exporter.