Giter VIP home page Giter VIP logo

network_exporter's Introduction

network_exporter

Github Action Docker Pulls

ICMP & MTR & TCP Port & HTTP Get Prometheus exporter

This exporter gathers either ICMP, MTR, TCP Port or HTTP Get stats and exports them via HTTP for Prometheus consumption.

grafana

Features

  • IPv4 & IPv6 support
  • Configuration reloading (By interval or OS signal)
  • Dynamically Add or Remove targets without affecting the currently running tests
  • Automatic update of the target IP when the DNS resolution changes
  • Targets can be executed on all hosts or a list of specified ones probe
  • Extra labels when defining targets
  • Configurable logging levels and format (text or json)
  • Configurable DNS Server
  • Configurable Source IP per target source_ip (optional), The IP has to be configured on one of the instance's interfaces

Exported metrics

  • ping_up Exporter state
  • ping_targets Number of active targets
  • ping_status: Ping Status
  • ping_rtt_seconds{type=best}: Best round trip time in seconds
  • ping_rtt_seconds{type=worst}: Worst round trip time in seconds
  • ping_rtt_seconds{type=mean}: Mean round trip time in seconds
  • ping_rtt_seconds{type=sum}: Sum round trip time in seconds
  • ping_rtt_seconds{type=sd}: Squared deviation in seconds
  • ping_rtt_seconds{type=usd}: Standard deviation without correction in seconds
  • ping_rtt_seconds{type=csd}: Standard deviation with correction (Bessel's) in seconds
  • ping_rtt_seconds{type=range}: Range in seconds
  • ping_rtt_snt_count: Packet sent count total
  • ping_rtt_snt_fail_count: Packet sent fail count total
  • ping_rtt_snt_seconds: Packet sent time total in seconds
  • ping_loss_percent: Packet loss in percent

  • mtr_up Exporter state
  • mtr_targets Number of active targets
  • mtr_hops Number of route hops
  • mtr_rtt_seconds{type=last}: Last round trip time in seconds
  • mtr_rtt_seconds{type=best}: Best round trip time in seconds
  • mtr_rtt_seconds{type=worst}: Worst round trip time in seconds
  • mtr_rtt_seconds{type=mean}: Mean round trip time in seconds
  • mtr_rtt_seconds{type=sum}: Sum round trip time in seconds
  • mtr_rtt_seconds{type=sd}: Squared deviation in seconds
  • mtr_rtt_seconds{type=usd}: Standard deviation without correction in seconds
  • mtr_rtt_seconds{type=csd}: Standard deviation with correction (Bessel's) in seconds
  • mtr_rtt_seconds{type=range}: Range in seconds
  • mtr_rtt_seconds{type=loss}: Packet loss in percent
  • mtr_rtt_snt_count: Packet sent count total
  • mtr_rtt_snt_fail_count: Packet sent fail count total
  • mtr_rtt_snt_seconds: Packet sent time total in seconds

  • tcp_up Exporter state
  • tcp_targets Number of active targets
  • tcp_connection_status Connection Status
  • tcp_connection_seconds Connection time in seconds

  • http_get_up Exporter state
  • http_get_targets Number of active targets
  • http_get_status HTTP Status Code and Connection Status
  • http_get_content_bytes HTTP Get Content Size in bytes
  • http_get_seconds{type=DNSLookup}: DNSLookup connection drill down time in seconds
  • http_get_seconds{type=TCPConnection}: TCPConnection connection drill down time in seconds
  • http_get_seconds{type=TLSHandshake}: TLSHandshake connection drill down time in seconds
  • http_get_seconds{type=TLSEarliestCertExpiry}: TLSEarliestCertExpiry cert expiration time in epoch
  • http_get_seconds{type=TLSLastChainExpiry}: TLSLastChainExpiry cert expiration time in epoch
  • http_get_seconds{type=ServerProcessing}: ServerProcessing connection drill down time in seconds
  • http_get_seconds{type=ContentTransfer}: ContentTransfer connection drill down time in seconds
  • http_get_seconds{type=Total}: Total connection time in seconds

Each metric contains the below labels and additionally the ones added in the configuration file.

  • name (ALL: The target name)
  • target (ALL: The target defined Hostname or IP)
  • target_ip (ALL: The target resolved IP Address)
  • source_ip (ALL: The source IP Address)
  • port (TCP: The target TCP Port)
  • ttl (MTR: Time to live)
  • path (MTR: Traceroute IP)

Building and running the software

Prerequisites for Linux

The process must run with the necesary linux or docker previlages to be able to perform the necesary tests

apt update
apt install docker
apt install docker.io
touch network_exporter.yml

Local Build

$ goreleaser release --skip=publish --snapshot --clean
$ ls -l artifacts/network_exporter_*6?
# If you want to run it with a non root user
$ sudo setcap 'cap_net_raw,cap_net_admin+eip' artifacts/network_exporter_linux_amd64/network_exporter

Building with Docker

To run the network_exporter as a Docker container by builing your own image or using https://hub.docker.com/r/syepes/network_exporter

docker build -t syepes/network_exporter .
# Default mode
docker run --privileged --cap-add NET_ADMIN --cap-add NET_RAW -p 9427:9427 -v $PWD/network_exporter.yml:/app/cfg/network_exporter.yml:ro --name network_exporter syepes/network_exporter
# Debug level
docker run --privileged --cap-add NET_ADMIN --cap-add NET_RAW -p 9427:9427 -v $PWD/network_exporter.yml:/app/cfg/network_exporter.yml:ro --name network_exporter syepes/network_exporter /app/network_exporter --log.level=debug

Configuration

To see all available configuration flags:

./network_exporter -h

The configuration (YAML) is mainly separated into three sections Main, Protocols and Targets. The file network_exporter.yml can be either edited before building the docker container or changed it runtime.

# Main Config
conf:
  refresh: 15m
  nameserver: 192.168.0.1:53 # Optional
  nameserver_timeout: 250ms # Optional

# Specific Protocol settings
icmp:
  interval: 3s
  timeout: 1s
  count: 6

mtr:
  interval: 3s
  timeout: 500ms
  max-hops: 30
  count: 6

tcp:
  interval: 3s
  timeout: 1s

http_get:
  interval: 15m
  timeout: 5s

# Target list and settings
targets:
  - name: internal
    host: 192.168.0.1
    type: ICMP
    probe:
      - hostname1
      - hostname2
    labels:
      dc: home
      rack: a1
  - name: google-dns1
    host: 8.8.8.8
    type: ICMP
  - name: google-dns2
    host: 8.8.4.4
    type: MTR
  - name: cloudflare-dns
    host: 1.1.1.1
    type: ICMP+MTR
  - name: cloudflare-dns-https
    host: 1.1.1.1:443
    source_ip: 192.168.1.1
    type: TCP
  - name: download-file-64M
    host: http://test-debit.free.fr/65536.rnd
    type: HTTPGet
  - name: download-file-64M-proxy
    host: http://test-debit.free.fr/65536.rnd
    type: HTTPGet
    proxy: http://localhost:3128

Source IP

source_ip parameter will try to assign IP for request sent to specific target. This IP has to be configure on one of the interfaces of the OS. Supported for all types of the checks

  - name: server3.example.com:9427
    host: server3.example.com:9427
    type: TCP
    source_ip: 192.168.1.1

Note: Domain names are resolved (regularly) to their corresponding A and AAAA records (IPv4 and IPv6). By default if not configured, network_exporter uses the system resolver to translate domain names to IP addresses. You can also override the DNS resolver address by specifying the conf.nameserver configuration setting.

SRV records: If the host field of a target contains a SRV record with the format _<service>._<protocol>.<domain> it will be resolved, all it's A records will be added (dynamically) as separate targets with name and host of the this A record. Every field of the parent target with a SRV record will be inherited by sub targets except name and host

SRV record supported for ICMP/MTR/TCP target types. TCP SRV record specifcs:

  • Target type should be TCP and _protocol part in the SRV record should be _tcp as well
  • Port will be taken from the 3rd number, just before the hostname

TCP SRV example

_connectivity-check._tcp.example.com. 86400 IN SRV 10 5 80 server.example.com.
_connectivity-check._tcp.example.com. 86400 IN SRV 10 5 443 server2.example.com.
_connectivity-check._tcp.example.com. 86400 IN SRV 10 5 9247 server3.example.com.

ICMP SRV example

_connectivity-check._icmp.example.com. 86400 IN SRV 10 5 8 server.example.com.
_connectivity-check._icmp.example.com. 86400 IN SRV 10 5 8 server2.example.com.
_connectivity-check._icmp.example.com. 86400 IN SRV 10 5 8 server3.example.com.

Configuration reference

  - name: test-srv-record
    host: _connectivity-check._icmp.example.com
    type: ICMP
  - name: test-srv-record
    host: _connectivity-check._tcp_.example.com
    type: TCP

Will be resolved to 3 separate targets:

  - name: server.example.com
    host: server.example.com
    type: ICMP
  - name: server2.example.com
    host: server2.example.com
    type: ICMP
  - name: server3.example.com
    host: server3.example.com
    type: ICMP
  - name: server.example.com:80
    host: server.example.com:80
    type: TCP
  - name: server2.example.com:443
    host: server2.example.com:443
    type: TCP
  - name: server3.example.com:9427
    host: server3.example.com:9427
    type: TCP

Deployment

This deployment example will permit you to have as many Ping Stations as you need (LAN or WIFI) devices but at the same time decoupling the data collection from the storage and visualization. This docker compose will deploy and configure all the components plus setup Grafana with the Datasource and Dashboard

Deployment example

Deployment architecture

Contribute

If you have any idea for an improvement or find a bug do not hesitate in opening an issue, just simply fork and create a pull-request to help improve the exporter.

License

All content is distributed under the Apache 2.0 License Copyright © 2020-2023, Sebastian YEPES

network_exporter's People

Contributors

benjamins-81 avatar dependabot[bot] avatar mattmichal avatar mengskysama avatar paebersold-tyro avatar romainmou avatar syepes avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

network_exporter's Issues

The packet loss rate is displayed abnormally

Hi @syepes ,

This is not a issue, I am Just a little confused.

On ICMP part, The packet loss rate is not normal on Grafana. (But packet loss is OK on MTR)

I compared the data and found that ping_loss_percent is float from 0 to 1, MTR packet loss is float from 0 to 100.

I am not sure if it is better to format ping_loss_percent as float from 0 to 100?

Looking forward your reply.

ping_loss_percent{name="google",target="8.8.8.8"} 0.5
mtr_rtt_seconds{name="8.8.8.8",path="8.8.8.8",target="8.8.8.8",ttl="12",type="loss"} 33.333335876464844

image
image

The docker image of version 1.6.3 is invalid, it is still version 1.6.2.

@syepes Thank you very much for your timely update, but I found that using The docker image of version 1.6.3 is invalid, it is still version 1.6.2.

this is log :
kubectl logs -f -n kube-system network-exporter-b4bc8dd9b-bdvh5 ts=2022-11-04T15:52:01.558Z caller=main.go:54 level=info msg="Starting network_exporter" version=1.6.2 ts=2022-11-04T15:52:01.560Z caller=main.go:56 level=info msg="Loading config" ts=2022-11-04T15:52:01.560Z caller=main.go:58 level=error msg="Loading config" err="reading config file: open /network_exporter.yml: no such file or directory"

Add an option to add static labels

It would be great if we can add static labels for each target to enrich metadata:

for example:

targets:

  • name: internal
    host: 192.168.1.1
    type: ICMP
    labels:
    isp_name: ispname
    label_test_1: test1
  • name: google-dns1
    host: 8.8.8.8
    type: ICMP+MTR
    labels:
    isp_name: ispname
    label_test_2: test2

goroutine leak

goroutine leak with http respose timeout

because timeout option not incluce respose timeout

ref: https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/

(pprof) top
Showing nodes accounting for 5040, 99.94% of 5043 total
Dropped 59 nodes (cum <= 25)
Showing top 10 nodes out of 35
      flat  flat%   sum%        cum   cum%
      5040 99.94% 99.94%       5040 99.94%  runtime.gopark
         0     0% 99.94%       2435 48.28%  bufio.(*Reader).Peek
         0     0% 99.94%       2436 48.30%  bufio.(*Reader).fill
         0     0% 99.94%       2435 48.28%  bytes.(*Buffer).ReadFrom
         0     0% 99.94%       2435 48.28%  crypto/tls.(*Conn).Read
         0     0% 99.94%       2435 48.28%  crypto/tls.(*Conn).readFromUntil
         0     0% 99.94%       2435 48.28%  crypto/tls.(*Conn).readRecord (inline)
         0     0% 99.94%       2435 48.28%  crypto/tls.(*Conn).readRecordOrCCS
         0     0% 99.94%       2435 48.28%  crypto/tls.(*atLeastReader).Read
         0     0% 99.94%         49  0.97%  github.com/syepes/network_exporter/pkg/icmp.Icmp

Hello, syepes, ask questions

icmp:
interval: 3s
timeout: 1s
count: 6

How to understand interval and count in icmp parameter? How to write if I want to realize PING once per second?

Multiples SIGSEGV a day

I never had this kind of problem with the 1.6.2 version, but since I've upgraded to 1.6.4, I've sometimes 2 ou 3 time a day this kind of crash :

janv. 10 12:25:15 server network_exporter[1090]: panic: runtime error: invalid memory address or nil pointer dereference
janv. 10 12:25:15 server network_exporter[1090]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8ba4e1]
janv. 10 12:25:15 server network_exporter[1090]: goroutine 62333 [running]:
janv. 10 12:25:15 server network_exporter[1090]: github.com/syepes/network_exporter/pkg/tcp.Port({0xc00022e080, 0x7}, {0xc0001209c7, 0x7}, {0x0, 0x0}, {0xc00022e088, 0x3}, 0x6fc23ac00, 0x3b9aca00)
janv. 10 12:25:15 server network_exporter[1090]:         /home/runner/work/network_exporter/network_exporter/pkg/tcp/tcp.go:45 +0x441
janv. 10 12:25:15 server network_exporter[1090]: github.com/syepes/network_exporter/target.(*TCPPort).portCheck(0xc00013e6e0)
janv. 10 12:25:15 server network_exporter[1090]:         /home/runner/work/network_exporter/network_exporter/target/target_tcp.go:81 +0x65
janv. 10 12:25:15 server network_exporter[1090]: created by github.com/syepes/network_exporter/target.(*TCPPort).run
janv. 10 12:25:15 server network_exporter[1090]:         /home/runner/work/network_exporter/network_exporter/target/target_tcp.go:69 +0xa7

It's hard to reproduce it, I didn't find a special thing that trigger it.

With version 1.7.2 and above the nameserver_timeout field is not optional, without it hosts are not resolved

Hello,

With version 1.7.2 unless the (optional) config option 'nameserver_timeout' is set dns resolution will not work.

This is on Linux x86 (Suse Linux 12.5).

First test with 1.7.1 to show valid config/targets in test with nameserver_timeout commented out

# cat network_exporter.yaml
conf:
  # nameserver_timeout: 250ms # Optional

icmp:
  interval: 2s
  timeout: 1s
  count: 1
mtr:
  interval: 3s
  timeout: 500ms
  max-hops: 30
  count: 6
tcp:
  interval: 3s
  timeout: 1s
http_get:
  interval: 15m
  timeout: 5s

targets:
  - name: test
    host: jenkins-agent-5-sn.sn.tyrocorp.net
    type: ICMP
  - name: test2
    host: jenkins-agent-5-sn.sn.tyrocorp.net:22
    type: TCP

# ./network_exporter-1.7.1 --version
1.7.1

# ./network_exporter-1.7.1 --log.level=debug --config.file=network_exporter.yaml
ts=2023-10-12T05:19:58.658Z caller=main.go:56 level=info msg="Starting network_exporter" version=1.7.1
ts=2023-10-12T05:19:58.658Z caller=main.go:58 level=info msg="Loading config"
ts=2023-10-12T05:19:58.659Z caller=main.go:147 level=info msg="Configured default DNS resolver"
ts=2023-10-12T05:19:58.659Z caller=monitor_ping.go:59 level=debug type=ICMP func=AddTargets msg="Current Targets: 0, cfg: 1"
ts=2023-10-12T05:19:58.661Z caller=monitor_ping.go:80 level=debug type=ICMP func=AddTargets msg="targetName: [test 10.6.2.55]"
ts=2023-10-12T05:19:58.663Z caller=monitor_ping.go:111 level=info type=ICMP func=AddTargetDelayed msg="Adding Target: test 10.6.2.55 (jenkins-agent-5-sn.sn.tyrocorp.net/10.6.2.55) in 0s"
ts=2023-10-12T05:19:58.663Z caller=monitor_mtr.go:61 level=debug type=MTR func=AddTargets msg="Current Targets: 0, cfg: 0"
ts=2023-10-12T05:19:58.663Z caller=monitor_mtr.go:76 level=debug type=MTR func=AddTargets msg="targetName: []"
ts=2023-10-12T05:19:58.663Z caller=monitor_tcp.go:56 level=debug type=TCP func=AddTargets msg="Current Targets: 0, cfg: 1"
ts=2023-10-12T05:19:58.664Z caller=monitor_tcp.go:82 level=debug type=TCP func=AddTargets msg="targetName: [test2 10.6.2.55]"
ts=2023-10-12T05:19:58.668Z caller=monitor_tcp.go:128 level=info type=TCP func=AddTargetDelayed msg="Adding Target: test2 10.6.2.55 (jenkins-agent-5-sn.sn.tyrocorp.net/10.6.2.55:22) in 0s"
ts=2023-10-12T05:19:58.668Z caller=monitor_http.go:56 level=debug type=HTTPGet func=AddTargets msg="Current Targets: 0, cfg: 0"
ts=2023-10-12T05:19:58.668Z caller=monitor_http.go:71 level=debug type=HTTPGet func=AddTargets msg="targetName: []"
ts=2023-10-12T05:19:58.668Z caller=main.go:140 level=info msg="Starting ping exporter" version=1.7.1
ts=2023-10-12T05:19:58.668Z caller=main.go:141 level=info msg="Listening for /metrics on :9427"
ts=2023-10-12T05:20:01.669Z caller=target_tcp.go:95 level=debug type=TCP func=port msg="{\"success\":true,\"dest_address\":\"jenkins-agent-5-sn.sn.tyrocorp.net\",\"dest_ip\":\"10.6.2.55\",\"dest_port\":\"22\",\"src_ip\":\"10.6.2.54\",\"connection_time\":622569}"
ts=2023-10-12T05:20:02.665Z caller=target_ping.go:109 level=debug type=ICMP func=ping msg="{\"success\":true,\"dest_address\":\"jenkins-agent-5-sn.sn.tyrocorp.net\",\"dest_ip\":\"10.6.2.55\",\"drop_rate\":0,\"sum\":992409,\"best\":992409,\"avg\":992000,\"worst\":992409,\"sd\":0,\"usd\":0,\"csd\":-9223372036854775808,\"range\":0,\"snt_summary\":1,\"snt_fail_summary\":0,\"snt_time_summary\":0}"
ts=2023-10-12T05:20:04.665Z caller=target_ping.go:109 level=debug type=ICMP func=ping msg="{\"success\":true,\"dest_address\":\"jenkins-agent-5-sn.sn.tyrocorp.net\",\"dest_ip\":\"10.6.2.55\",\"drop_rate\":0,\"sum\":983380,\"best\":983380,\"avg\":983000,\"worst\":983380,\"sd\":0,\"usd\":0,\"csd\":-9223372036854775808,\"range\":0,\"snt_summary\":2,\"snt_fail_summary\":0,\"snt_time_summary\":0}"
ts=2023-10-12T05:20:04.669Z caller=target_tcp.go:95 level=debug type=TCP func=port msg="{\"success\":true,\"dest_address\":\"jenkins-agent-5-sn.sn.tyrocorp.net\",\"dest_ip\":\"10.6.2.55\",\"dest_port\":\"22\",\"src_ip\":\"10.6.2.54\",\"connection_time\":529819}"

Now test with 1.7.2

# ./network_exporter-1.7.2 --version
1.7.2

# ./network_exporter-1.7.2 --log.level=debug --config.file=network_exporter.yaml
ts=2023-10-12T05:21:16.639Z caller=main.go:56 level=info msg="Starting network_exporter" version=1.7.2
ts=2023-10-12T05:21:16.639Z caller=main.go:58 level=info msg="Loading config"
ts=2023-10-12T05:21:16.640Z caller=main.go:147 level=info msg="Configured default DNS resolver"
ts=2023-10-12T05:21:16.641Z caller=monitor_mtr.go:61 level=debug type=MTR func=AddTargets msg="Current Targets: 0, cfg: 0"
ts=2023-10-12T05:21:16.641Z caller=monitor_mtr.go:76 level=debug type=MTR func=AddTargets msg="targetName: []"
ts=2023-10-12T05:21:16.641Z caller=monitor_http.go:55 level=debug type=HTTPGet func=AddTargets msg="Current Targets: 0, cfg: 0"
ts=2023-10-12T05:21:16.641Z caller=monitor_ping.go:59 level=debug type=ICMP func=AddTargets msg="Current Targets: 0, cfg: 1"
ts=2023-10-12T05:21:16.641Z caller=monitor_http.go:70 level=debug type=HTTPGet func=AddTargets msg="targetName: []"
ts=2023-10-12T05:21:16.641Z caller=main.go:140 level=info msg="Starting ping exporter" version=1.7.2
ts=2023-10-12T05:21:16.641Z caller=main.go:141 level=info msg="Listening for /metrics on :9427"
ts=2023-10-12T05:21:16.641Z caller=monitor_tcp.go:56 level=debug type=TCP func=AddTargets msg="Current Targets: 0, cfg: 1"
ts=2023-10-12T05:21:16.641Z caller=monitor_ping.go:71 level=warn type=ICMP func=AddTargets msg="Skipping resolve target: jenkins-agent-5-sn.sn.tyrocorp.net" err="resolving target: lookup jenkins-agent-5-sn.sn.tyrocorp.net: i/o timeout"
ts=2023-10-12T05:21:16.641Z caller=monitor_ping.go:80 level=debug type=ICMP func=AddTargets msg="targetName: []"
ts=2023-10-12T05:21:16.641Z caller=monitor_tcp.go:73 level=warn type=TCP func=AddTargets msg="Skipping resolve target: jenkins-agent-5-sn.sn.tyrocorp.net:22" err="resolving target: lookup jenkins-agent-5-sn.sn.tyrocorp.net: i/o timeout"
ts=2023-10-12T05:21:16.641Z caller=monitor_tcp.go:82 level=debug type=TCP func=AddTargets msg="targetName: []"

Update config and add in nameserver_timeout option

# head -5 network_exporter.yaml
conf:
  nameserver_timeout: 250ms # Optional

icmp:
  interval: 2s

# ./network_exporter-1.7.2 --log.level=debug --config.file=network_exporter.yaml
ts=2023-10-12T05:22:11.067Z caller=main.go:56 level=info msg="Starting network_exporter" version=1.7.2
ts=2023-10-12T05:22:11.067Z caller=main.go:58 level=info msg="Loading config"
ts=2023-10-12T05:22:11.068Z caller=main.go:147 level=info msg="Configured default DNS resolver"
ts=2023-10-12T05:22:11.068Z caller=main.go:140 level=info msg="Starting ping exporter" version=1.7.2
ts=2023-10-12T05:22:11.069Z caller=main.go:141 level=info msg="Listening for /metrics on :9427"
ts=2023-10-12T05:22:11.069Z caller=monitor_ping.go:59 level=debug type=ICMP func=AddTargets msg="Current Targets: 0, cfg: 1"
ts=2023-10-12T05:22:11.070Z caller=monitor_mtr.go:61 level=debug type=MTR func=AddTargets msg="Current Targets: 0, cfg: 0"
ts=2023-10-12T05:22:11.070Z caller=monitor_mtr.go:76 level=debug type=MTR func=AddTargets msg="targetName: []"
ts=2023-10-12T05:22:11.070Z caller=monitor_tcp.go:56 level=debug type=TCP func=AddTargets msg="Current Targets: 0, cfg: 1"
ts=2023-10-12T05:22:11.070Z caller=monitor_http.go:55 level=debug type=HTTPGet func=AddTargets msg="Current Targets: 0, cfg: 0"
ts=2023-10-12T05:22:11.070Z caller=monitor_http.go:70 level=debug type=HTTPGet func=AddTargets msg="targetName: []"
ts=2023-10-12T05:22:11.071Z caller=monitor_tcp.go:82 level=debug type=TCP func=AddTargets msg="targetName: [test2 10.6.2.55]"
ts=2023-10-12T05:22:11.071Z caller=monitor_ping.go:80 level=debug type=ICMP func=AddTargets msg="targetName: [test 10.6.2.55]"
ts=2023-10-12T05:22:11.073Z caller=monitor_ping.go:111 level=info type=ICMP func=AddTargetDelayed msg="Adding Target: test 10.6.2.55 (jenkins-agent-5-sn.sn.tyrocorp.net/10.6.2.55) in 0s"
ts=2023-10-12T05:22:11.075Z caller=monitor_tcp.go:128 level=info type=TCP func=AddTargetDelayed msg="Adding Target: test2 10.6.2.55 (jenkins-agent-5-sn.sn.tyrocorp.net/10.6.2.55:22) in 0s"
ts=2023-10-12T05:22:14.076Z caller=target_tcp.go:95 level=debug type=TCP func=port msg="{\"success\":true,\"dest_address\":\"jenkins-agent-5-sn.sn.tyrocorp.net\",\"dest_ip\":\"10.6.2.55\",\"dest_port\":\"22\",\"src_ip\":\"10.6.2.54\",\"connection_time\":702514}"
ts=2023-10-12T05:22:15.078Z caller=target_ping.go:109 level=debug type=ICMP func=ping msg="{\"success\":true,\"dest_address\":\"jenkins-agent-5-sn.sn.tyrocorp.net\",\"dest_ip\":\"10.6.2.55\",\"drop_rate\":0,\"sum\":1782875,\"best\":1782875,\"avg\":1782000,\"worst\":1782875,\"sd\":0,\"usd\":0,\"csd\":-9223372036854775808,\"range\":0,\"snt_summary\":1,\"snt_fail_summary\":0,\"snt_time_summary\":0}"

HTTPGet checks using a proxy do not reuse network connection

Hello,
we have found that using a proxy for HTTPGet requests results in every check generating a new network connection (and old connections are never closed). Our sample config (with names/ips changed)

---
http_get:
 interval: 15s
 timeout: 5s
targets:
  - name: useproxy
    host: https://bob.server.com/ready
    type: HTTPGet
    proxy: http://proxy:3128
  - name: noproxy
    host: http://1.2.3.4:9100/metrics
    type: HTTPGet

For target 'useproxy' ever check generates a new connection so after a period of time we have hundreds for established connections, and we see keepalive traffic (an ACK of length 0) on each connection every interval period.

For target 'noproxy' we see one connection that is kept for the entire duration and see the normal traffic from the http GET every interval.

Initially I thought this may have been an issue in pkg/http/http.go that the function HTTPGetProxy does not do the boilerplate

defer resp.Body.Close()

as suggested in https://pkg.go.dev/net/http#pkg-overview. However adding this did not change the behaviour. Also HTTPGet works just fine without it.

Setting the Close field on the HTTP Request in HTTPGetProxy ala

req.Close = true

fixes the issue as expected.

However I am still puzzled on the difference in behaviour as the two functions HTTPGetProxy and HTTPGet are almost identical. Is this behaviour seen by others?

For reference the client system is Sles12sp5 linux x86 (4.12.14-122.136) and the proxy is running Apache 2.4.51 and using mod_proxy.

Thanks for your time.

packet loss for ipv6 destinations

I'm having problems when I perform mtr to destinations that are ipv6. Packet losses appear at hops along the way. when i stop the networ_exporter service the losses also stop. What can I do to resolve?
image
image
image

Exporter returns last successful result when the target fails, doesn't return a failure

Scenario:

  1. An HTTPGet target is configured that goes through a proxy.
  2. The target succeeds initially.
  3. The proxy goes down.

Expected result:
When getting metrics, the exporter should return a failure for the HTTPGet target.

Actual result:
When getting metrics, the exporter returns the last successful result.

It seems it is caused by the following lines:

Note that though only the HTTPGet target is tested, this seems to be an issue of all the targets.

Grafana sort display by name

Grafana version 10.2
Can fix the number of ttl completed with double digits? (only 1~9 is single)
like this image
image

It's can show the next hop step by step
Thank you.

Windows Connection on wait

Hi,
if the network exporter is running, than the monitored destinations would stack as "Waiting" and "SYN" shown in netstat.
It looks like that he catched them, so I see now changes.

No info about hop address in MTR metric

I use network_exporer v1.7.6.
For example try to probe mtr google.com. And i cant see in metric addresses of intermediate hops.
If i correctly understand this info must be in path label

# TYPE mtr_hops gauge
mtr_hops{name="google",target="142.250.147.100"} 29
# HELP mtr_rtt_seconds Round Trip Time in seconds
# TYPE mtr_rtt_seconds gauge
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="best"} 8.7974e-05
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="csd"} -9.223372036854776e+09
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="last"} 8.7974e-05
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="loss"} 0.8333333333333334
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="mean"} 8.7e-05
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="range"} 0
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="sd"} 0
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="sum"} 8.7974e-05
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="usd"} 0
mtr_rtt_seconds{name="google",path="172.17.0.1",target="142.250.147.100",ttl="1",type="worst"} 8.7974e-05
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="best"} 0
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="csd"} 0
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="last"} 0
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="loss"} 1
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="mean"} 0
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="range"} 0
mtr_rtt_seconds{name="google",path="unknown",target="142.250.147.100",ttl="10",type="sd"} 0
...

If I run local mtr google.com i can see host ip for each hop.

Panels display issues

Hi syepes,

It's me again...

I'm looking forward to deploying network_exporter in the production env as my leader was very satisfied with the monitoring stack. But I found something I am not able to deal with. so that I post it here.

  1. It's not an issue. I would like to sort the items by ttl. I looked up a lot of information but couldn't find a way to do it.
    image

  2. Even if the node is unreachable, the panel still shows it.
    In the time range shown below, hops 3-7 areunreachable, but they keep the same value like a straight line until they became reachable. this issue is not relevant to network_exporter as I checked network_exporter metircs were OK. Could it be caused by Victoriametrics?
    image

  3. Invalid nodes will still be displayed.
    As the panel shown below, we can see the hop 11 keeps updating. As I understand, currently the hop 11 should be "unknown" as the MTR shown below.
    image
    image

Support for testing HTTP PUT and POST

Hello,

I believe it would be nice to have a way to submit pseudo-random buffer (or user-specified static file) to user-specified URLs and measure same set parameters as for GET, plus http_put_seconds{type=ContentTransfer} (keeping http_get_seconds as is for compatibility reasons). WDYT?

Config question

Is it possible to configure also the Source IP + Port?
Now he only monitored the destination IP + Port.

The name of taegets contains an underscore, and an error is reported in the network_exporter log.

The name of taegets contains an underscore, and an error is reported in the network_exporter log.

network_exporter config file:
111

this is log:
`
-- Logs begin at Wed 2023-10-25 22:52:11 CST. --
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.106Z caller=monitor_mtr.go:101 level=info type=MTR func=AddTargetDelayed msg="Adding Target: cloudflare-dns1 (vpnprod.ilanni.com) in 0s"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.357Z caller=monitor_mtr.go:87 level=warn type=MTR func=AddTargets msg="Skipping target: vpnprod.ilanni.com" err="resolving target: lookup vpnprod.ilanni.com: i/o timeout"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.357Z caller=monitor_ping.go:71 level=warn type=ICMP func=AddTargets msg="Skipping resolve target: vpnprod.ilanni.com" err="resolving target: lookup vpnprod.ilanni.com: i/o timeout"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.357Z caller=monitor_ping.go:111 level=info type=ICMP func=AddTargetDelayed msg="Adding Target: cloudflare-dns0_1 119.8.16.11 (119.8.16.11/119.8.16.11) in 0s"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.357Z caller=monitor_tcp.go:73 level=warn type=TCP func=AddTargets msg="Skipping resolve target: vpnprod.ilanni.com:443" err="resolving target: lookup vpnprod.ilanni.com: i/o timeout"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.357Z caller=monitor_tcp.go:128 level=info type=TCP func=AddTargetDelayed msg="Adding Target: cloudflare-dns-https 19.8.16.10 (19.8.16.10/19.8.16.10:443) in 0s"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.607Z caller=monitor_ping.go:87 level=warn type=ICMP func=AddTargets msg="Skipping resolve target: vpnprod.ilanni.com" err="resolving target: lookup vpnprod.ilanni.com: i/o timeout"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.607Z caller=monitor_ping.go:111 level=info type=ICMP func=AddTargetDelayed msg="Adding Target: cloudflare-dns 19.8.16.10 (19.8.16.10/19.8.16.10) in 0s"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.607Z caller=monitor_tcp.go:94 level=warn type=TCP func=AddTargets msg="Skipping resolve target: cloudflare-dns-https1" err="resolving target: lookup vpnprod.ilanni.com: i/o timeout"
Oct 31 10:14:52 qxt-gw-03 network_exporter[20571]: ts=2023-10-31T02:14:52.858Z caller=monitor_ping.go:87 level=warn type=ICMP func=AddTargets msg="Skipping resolve target: vpnprod.ilanni.com" err="resolving target: lookup vpnprod.ilanni.com: i/o timeout"

`

new module?

Hi!

I want add module like in blackbox_exporter
for example some like this:

http_2xx_test:
  prober: http
  timeout: 5s
  http:
    valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
    valid_status_codes: [200]  # Defaults to 2xx
    method: GET
    no_follow_redirects: false
    fail_if_ssl: false
    fail_if_not_ssl: false
    fail_if_body_matches_regexp:
      - "Could not connect to database"
    fail_if_body_not_matches_regexp:
     - "Database OK"
    tls_config:
        insecure_skip_verify: false

How to do it in proper way?

hope: The configure file same name in targets supports multiple ip probes

hope: The configure file same name in targets supports multiple ip probes, and can support such ip probes:

targets:

  • name: idc_ 1
    host: 183.100.0.1
    host: 183.100.0.2
    host: 183.100.0.3
    host: 183.100.0.4
    type: ICMP
  • name: idc_ 2
    host: 204.152.0.1
    host: 204.152.0.2
    host: 204.152.0.3
    host: 204.152.0.4
    type: ICMP

thanks!~~

docker-compose setcap error

Hello,

Running the docker-compose deployment, I'm getting the following error both as root and non-root. Please help me resolve this issue.

ls
cfg data docker-compose.yml

docker-compose up
Creating network "deploy_data" with driver "bridge"
Creating volume "deploy_grafanadata" with default driver
Creating station1_agt ...
Creating station1_exporter ... error
Creating central_grafana ...
Creating station1_agt ... done

Creating central_grafana ... done
/proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown

ERROR: for station1_exporter Cannot start service station1_exporter: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ip_unprivileged_port_start: open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown
ERROR: Encountered errors while bringing up the project.

kubernetes mount network-exporter.yml file problem

Thank you very much for having such an exporter. I plan to deploy the exporter to kubernetes, but I found that the network-exporter.yml file cannot be mapped to the pod through configmap, mainly because the file is mounted to the / root directory by default, may I ask this Is it possible to modify to other directory instead of root directory

deployment config:
`---
apiVersion: v1
kind: ConfigMap
metadata:
name: network-exporter-configmap
namespace: kube-system
labels:
app.kubernetes.io/instance: network-exporter
app.kubernetes.io/name: network-exporter
data:
network-exporter.yml: |
icmp:
interval: 3s
timeout: 1s
count: 6

  mtr:
    interval: 3s
    timeout: 500ms
    max-hops: 30
    count: 6
  
  tcp:
    interval: 3s
    timeout: 1s
  
  http_get:
    interval: 15m
    timeout: 5s
  
  targets:
    - name: test_mtr
      host: 192.168.0.26
      type: MTR
    - name: test_tcp
      host: 192.168.0.26:1792
      type: TCP

apiVersion: v1
kind: Service
metadata:
name: network-exporter-svc
namespace: kube-system
labels:
app.kubernetes.io/instance: network-exporter
app.kubernetes.io/name: network-exporter
spec:
type: NodePort
ports:

  • name: tcp-9427
    port: 1080
    protocol: TCP
    selector:
    app.kubernetes.io/instance: network-exporter
    app.kubernetes.io/name: network-exporter

apiVersion: apps/v1
kind: Deployment
metadata:
name: network-exporter
namespace: kube-system
labels:
app.kubernetes.io/instance: network-exporter
app.kubernetes.io/name: network-exporter
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: network-exporter
app.kubernetes.io/name: network-exporter
template:
metadata:
labels:
app.kubernetes.io/instance: network-exporter
app.kubernetes.io/name: network-exporter
spec:
containers:
- name: network-exporter
image: syepes/network_exporter:1.6.2
imagePullPolicy: IfNotPresent
args:
- --config.file=/tmp/network-exporter.yml
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
securityContext:
runAsUser: 0
volumeMounts:
- name: network-exporter-conf
mountPath: /tmp/
readOnly: true
volumes:
- name: network-exporter-conf
configMap:
name: network-exporter-configmap
items:
- key: network-exporter.yml
path: network-exporter.yml`

this is log:
1s Normal SuccessfulCreate pod/network-exporter-7c87b6f7c6-dwn4p Created container network-exporter 1s Warning FailedStart pod/network-exporter-7c87b6f7c6-dwn4p Error: failed to start container "network-exporter": Error response from daemon: OCI runtime create failed: container_linux.go:318: starting container process caused "exec: \"--config.file=/tmp/network-exporter.yml\": stat --config.file=/tmp/network-exporter.yml: no such file or directory": unknown 0s Warning BackOffStart pod/network-exporter-7c87b6f7c6-dwn4p the failed container exited with ExitCode: 127 0s Warning BackOffStart pod/network-exporter-7c87b6f7c6-dwn4p Back-off restarting failed container

Helm Chart for Network exporter

Hi,

I believe having a helm chart to deploy the exporter in kubernetes would be a nice thing to do. Let me know if i can pick this up.

Only one MTR probe is working although multiple MTR targets are configured

Hi,

I've configured multiple MTR targets:
conf:
refresh: 15m
nameserver: 192.168.1.1:53

icmp:
interval: 3s
timeout: 1s
count: 10

mtr:
interval: 3s
timeout: 500ms
max-hops: 30
count: 3

tcp:
interval: 3s
timeout: 1s

http_get:
interval: 15m
timeout: 5s

targets:

  • name: google-dns1
    host: 8.8.8.8
    type: ICMP
  • name: google-dns2
    host: 8.8.4.4
    type: MTR

However, in the exporter, i can see only one MTR. also MTR count metric is 1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.