pierky / pmacct-to-elasticsearch Goto Github PK

A python script to send pmacct's output to ElasticSearch

License: MIT License

Shell 12.33% Python 87.67%

pmacct elasticsearch bgp netflow sflow nms accounting traffic analysis

pmacct-to-elasticsearch's Introduction

pmacct-to-elasticsearch

pmacct-to-elasticsearch is a python script designed to read output from pmacct daemons, to process it and to store it into ElasticSearch. It works with both memory and print plugins and, optionally, it can perform manipulations on data (such as to add fields on the basis of other values).

pmacct daemons collect IP accounting data and process them with their plugins;
data are stored into in-memory-tables (memory plugins), JSON or CSV files (print plugins);
crontab jobs (memory plugins) or trigger scripts (print plugins) are invoked to execute pmacct-to-elasticsearch;
pmacct's output records are finally processed by pmacct-to-elasticsearch, which reads them from stdin (memory plugins) or directly from file.

Optionally, some data transformations can be configured, to allow pmacct-to-elasticsearch to add or remove fields to/from the output documents that are sent to ElasticSearch for indexing. These additional fields may be useful to enhance graphs and reports legibility, or to add a further level of aggregation or filtering.

Installation

Install the program using pip:

pip install pmacct-to-elasticsearch

Then clone the repository and run the ./install script to setup your system:

cd /usr/local/src/
git clone https://github.com/pierky/pmacct-to-elasticsearch.git
cd pmacct-to-elasticsearch/
./install

Configuration

Please refer to the CONFIGURATION.md file. The TRANSFORMATIONS.md file contains details about data transformations configuration.

A simple tutorial on pmacct integration with ElasticSearch/Kibana using pmacct-to-elasticsearch can be found at http://blog.pierky.com/integration-of-pmacct-with-elasticsearch-and-kibana.

Future work

Add support of more pmacct output formats (Apache Avro, ...).

Author

Pier Carlo Chiodi - https://pierky.com/

Blog: https://blog.pierky.com/ Twitter: @pierky

pmacct-to-elasticsearch's People

Contributors

Stargazers

Watchers

Forkers

timohund lassizci junpei-yoshino benroeder verzulli acoul coocla mihalysz ammoam kvitex omar-zrd ivosandoval tuxnotes

pmacct-to-elasticsearch's Issues

Tab and import errors on running pmacct

I've tried running pmacct and get the following errors.

I fixed the first error by removing all tabs in the python files in the pierky/p2es/ directory
I fixed the importation errors by adding more specific importations to all the files in the pierky/p2es/ directory

I will submit a pull request that fixes this shortly.

[root@tsv009 tmp]# pmacct -l -p /var/spool/pmacct/netflow.pipe -s -O json -e | pmacct-to-elasticsearch netflow
  File "/usr/local/bin/pmacct-to-elasticsearch", line 226
    sys.stderr.write("Can't setup logging to stderr.")
                                                     ^
TabError: inconsistent use of tabs and spaces in indentation
[root@tsv009 tmp]#

[root@tsv009 p2es]# pmacct -l -p /var/spool/pmacct/netflow.pipe -s -O json -e | pmacct-to-elasticsearch netflow
Traceback (most recent call last):
  File "/usr/local/bin/pmacct-to-elasticsearch", line 22, in <module>
    from pierky.p2es.readers import JSONReader, CSVReader
  File "/usr/lib/python3.5/site-packages/pierky/p2es/readers.py", line 10, in <module>
    from transformations import *
ImportError: No module named 'transformations'

"chmox" shoud be "chmod" on line 98 of install script

Title says it all really.

json parse is wrong for pmacct

example:
/home/pstaszewski/pmacct-to-elasticsearch# pmacct -l -p /tmp/epix_mem.pipe -s -O json -e | pmacct-to-elasticsearch e_mem
2022-12-20 00:18:56,105 ERROR Error loading configuration from /etc/p2es/epix_mem.conf
Traceback (most recent call last):
File "/usr/local/bin/pmacct-to-elasticsearch", line 398, in main
new_cfg = json.load(f)
File "/usr/lib/python3.8/json/init.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 4 column 1 (char 69)

Or for json
/usr/local/bin/pmacct-to-elasticsearch e_print
2022-12-20 00:26:46,363 ERROR Error loading configuration from /etc/p2es/epix_print.conf
Traceback (most recent call last):
File "/usr/local/bin/pmacct-to-elasticsearch", line 398, in main
new_cfg = json.load(f)
File "/usr/lib/python3.8/json/init.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 4 column 5 (char 73)

And sfacctd config
! Elastic
aggregate[e_mem]: peer_src_ip, vlan, src_as, dst_as, in_iface, out_iface, src_net, dst_net, src_mask, dst_mask, flows
imt_path[e_mem]: /tmp/e_mem.pipe
aggregate[e_print]: peer_src_ip, vlan, src_as, dst_as, in_iface, out_iface, src_net, dst_net, src_mask, dst_mask, flows
print_output_file[e_print]: /data/e_print.json
print_output[e_print]: json

ES 6 - error occurred while creating index

Hi, just leaving a quick note here, I upgraded my cluster to ES 6.2.2 yesterday and found out that the index creation function was failing to create the index with "error unknown", after a quick review of the code and ES output I fixed it by changing the content type on the http header to 'application/json'. I didn't have the time to confirm if there are any side-effects but the data seems to be flowing to ES just fine.

Here's the http function for reference.

# /p2es/es.py
def http(CONFIG, url, method="GET", data=None):
    auth = None
    if CONFIG['ES_AuthType'] != 'none':
        if CONFIG['ES_AuthType'] == 'basic':
            auth = HTTPBasicAuth(CONFIG['ES_UserName'], CONFIG['ES_Password'])
        elif CONFIG['ES_AuthType'] == 'digest':
            auth = HTTPDigestAuth(CONFIG['ES_UserName'], CONFIG['ES_Password'])
        else:
            raise P2ESError(
                'Unexpected authentication type: {}'.format(CONFIG['ES_AuthType'])
            )

    headers = {'Content-Type': 'application/json'} # <<<<<<<

    if method == "GET":
        return requests.get(url, auth=auth, headers=headers)
    elif method == "POST":
        return requests.post(url, auth=auth, data=data, headers=headers)
    elif method == "PUT":
        return requests.put(url, auth=auth, data=data, headers=headers)
    elif method == "HEAD":
        return requests.head(url, auth=auth, headers=headers)
    else:
        raise Exception("Method unknown: {}".format(method))

Failure when running elasticsearch v.7.x

I'm having an issue with inserting an index when using elasticsearch v7.x, specifically elasticsearch-7.3.2-1.x86_64

It seems to be unrelated to the pmacct installation and solely related to the version of elasticsearch as I'm not relying on any pmacct output when getting this message. Please verify if possible.

Details

Prior to diverting netflow traffic to pmacct-to-elasticsearch using this command:

*/5 *	* * *     root  pmacct -l -p /var/spool/pmacct/netflow.pipe -s -O json -e | pmacct-to-elasticsearch netflow

I get the following error:

2019-09-27 21:37:12,549 ERROR Error while creating index netflow-2019-09-27: An error occurred while creating index netflow-2019-09-27 from template /etc/p2es/new-index-template.json: error unknown

It is specifically being caused by the pmacct-to-elasticsearch netflow pipe. When I curl the expected url I get the following error.

[root@tsv009 tmp]# curl http://localhost:9200/netflow-`date +%F`/socket/_search?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [netflow-2019-09-27]",
        "resource.type" : "index_or_alias",
        "resource.id" : "netflow-2019-09-27",
        "index_uuid" : "_na_",
        "index" : "netflow-2019-09-27"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [netflow-2019-09-27]",
    "resource.type" : "index_or_alias",
    "resource.id" : "netflow-2019-09-27",
    "index_uuid" : "_na_",
    "index" : "netflow-2019-09-27"
  },
  "status" : 404
}
[root@tsv009 tmp]

The new-index-template.json file I'm using is the one provided by your latest code:

{
  "mappings": {
    "_default_": {
      "_all" : { "enabled" : false },

      "_source" : { "enabled" : true },

      "dynamic_templates": [
        {
          "string_template" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping": { "type": "keyword", "index": true }
          }
        }
      ],

      "properties": {
        "@timestamp": { "type": "date" },
        "bytes": { "type": "long" },
        "packets": { "type": "long" },
        "flows": { "type": "long" }
      }
    }
  }
}

Multiple ES servers

Hi Pier,

Is it possible to specify multiple ES_URL for failover ?

Thanks you,

Andrew

/usr/local/bin/pmacct-to-elasticsearch: Permission denied

Hi Guys,

When I try to run sfacctd -f /some/conf/file with a trigger I get

/etc/p2es/triggers/plugin2: 21: /etc/p2es/triggers/plugin2: /usr/local/bin/pmacct-to-elasticsearch: Permission denied

Config file

plugins: memory[plugin1], print[plugin2]

print_output_file[plugin2]: /var/lib/pmacct/plugin2.json
print_output[plugin2]: json
print_trigger_exec[plugin2]: /etc/p2es/triggers/plugin2
print_refresh_time[plugin2]: 60
aggregate[plugin2]: proto,src_port

Trigger

{
"ES_URL": "http://10.255.28.199:9200",
"ES_IndexName": "example-%Y-%m-%d",
"ES_Type": "source_port",
"InputFile": "/var/lib/pmacct/plugin2.json",
}

Any help would be greatly appreciated.

Thanks

Adam

pmacct-to-elasticsearch 0.3.1 PIP module has extra tab

The tab on line 226 prevents the pip module from running. When it's removed, the module works correctly.

# pip3 list | grep -i elastic
pmacct-to-elasticsearch 
# pmacct-to-elasticsearch netflow
  File "/usr/local/bin/pmacct-to-elasticsearch", line 226
    sys.stderr.write("Can't setup logging to stderr.")
                                                     ^
TabError: inconsistent use of tabs and spaces in indentation

Getting 401 Unauthorized access when posting data to ES !

Hi, I'm trying to post the data to ES using this pmacct-to-elasticsearch option. But getting "401 Unauthorized access" . Since the basic authentication enabled in my ES setup.

Does pmacct-to-elasticsearch have an option to include the authentication in the configuration file?

I couldn't find any option from the configuration guide : https://github.com/pierky/pmacct-to-elasticsearch/blob/master/CONFIGURATION.md

Doc: print versus memory

In CONFIGURATION.md it says:

For "print" plugins, a crontab job is needed to run pmacct client and to redirect its output to pmacct-to-elasticsearch; for "memory" plugins the pmacct daemon can directly execute pmacct-to-elasticsearch.

However that is contradicted by the examples which follow. They show a cronjob querying the memory plugin (pmacct -p /var/spool/pmacct/my_mem.pipe), and the print plugin using print_trigger_exec to run p2es.

Has the first paragraph got print and memory the wrong way round?