xrootd-monitoring-collector's Introduction

XRootD Monitoring Collector

XRootD monitoring collector for ingesting monitoring data from the XRootD server, aggregating it into one monitoring record per file transfer, and sending a resulting JSON-formatted record into a AMQP-based message bus.

Configuration

The DetailedCollector needs the a configuration in order to connect to the AMQP message bus.

[AMQP]

# Host information
url = amqps://username:[email protected]

# Exchange to write to
exchange = xrd.detailed

This file is named connection.conf and should be in the Collectors directory or deployed with docker volumes, as shown below.

Deployment

The Detailed collector is available from DockerHub: https://hub.docker.com/repository/docker/opensciencegrid/xrootd-monitoring-collector

You can deploy this monitoring collector with docker-compose, or your favorite container orchestration engine. It will need to receive UDP packets from XRootD servers.

Here is an example docker-compose.yml file for the Detailed Collector:

version: '3.2'
services:
  detailed_collector:
    image: "opensciencegrid/xrootd-monitoring-collector"
    volumes:
      - ./connection.conf:/configs/connection.conf
    ports:
      - "9930:9930/udp"
      - "8000:8000/tcp"
    restart: always

Monitoring

The collector exports a Prometheus compatible interface on port 8000 that gives information about the internal state of the collector, including:

Number of packets received
Number of WLCG or StashCache file transfers sent to the AMQP message bus.
Errors while processing the monitoring packets

xrootd-monitoring-collector's People

Contributors

Stargazers

Watchers

xrootd-monitoring-collector's Issues

Need to add write bytes

Currently write bytes are not sent to WLCG.
I think that needs to be sent too.
AlsoI I think this may not be the issue with this repo, but I think
write bytes are all zero.

Propagate the raw appinfo into it's own attribute

Not all files are collected

I installed the collector and Florida xrootd (4.9.1-1) was configured to report the detailed monitoring
to the installed collector.
But number of closed files per day is too small to be true:
(base) [coldhead@oo SummaryCollector]$ wc -l wlcg_record.log.2019-04-30_20
36220 wlcg_record.log.2019-04-30_20
Here wlcg_record.log.2019-04-30_20 is a one day worth of list of closed files collected using
xrootd-monitoring-collector (XMC).
I was expecting at least 100K closed files based on the xrootd log file record.
From the XMC log, I see a lot of these:
2019-05-06 23:33:26,146 - 29472 - DetailedCollector - ERROR - File close record from unknown UserID=0, SID=1556882878#128.227.221.226#39878
TypeError: cannot unpack non-iterable NoneType object
2019-05-06 23:33:26,146 - 29472 - DetailedCollector - ERROR - file to close not found. fileID: 257523, serverID: 1556882878#128.227.221.226#39878. close=fileClose(rectype=0, recFlag=2, recSize=80, fileID=257523, read=332, readv=19501343, write=0, ops=ops(read=1, readv=6, write=0, rsMin=3, rsMax=69, rsegs=159, rdMin=332, rdMax=332, rvMin=307447, rvMax=8943104, wrMin=0, wrMax=0))

Did I misconfigure something?

Recommend Projects