raychen05 / elasticsearch-log-processor Goto Github PK

This is a way to visualise Elasticsearch logs (not slow logs) in Kibana. See the following screenshot for what it looks like:

• Elasticsearch 5.x or higher
• Kibana 5.x or higher
• Logstash 5.x or higher

Load the Elasticsearch mapping template in elasticsearch/elasticsearch-template.json:

curl -XPUT <eshost>:<esport>/_template/es-logs -d @elasticsearch/elasticsearch-template-5x.json

Replace <eshost> and <esport> to your Elasticsearch ip/host and port combo.

Import the Kibana dashboard/visualisations in kibana/export.json

There is a helper script called run.sh in the root of this repo that can assist with performing one-off indexing of logs via Logstash and Elasticsearch.

Usage is simple:

./run.sh -h https://myes.com:9200 -u user -p password path/to/log

Where:

• -h is the URL (including the port) for your Elasticsearch endpoint
• -u and -p are the username and password to access your Elasticsearch endpoint (optional).

The remaining command-line arguments are treated as log files. Standard shell wildcard/globbing applies. Need to process a bunch of logs in different directories? Try:

find -type f -name \*.log | xargs ~/git/elasticsearch-log-processor/run.sh -u user -p password

If you add a -t option the script will load the Elasticsearch template shipped in this repo into Elasticsearch for you before performing any indexing with Logstash.

Create an output configuration in the logsash/conf.d directory appropriate for your Elasticsearch server. Ensure the index pattern is elasticsearch-logs-5-%{YYYY.MM.DD}. Then simply pipe your log files to logstash using this configuration file:

/path/to/bin/logstash --config logstash/conf.d < /path/to/elasticsearch.log
# or
cat /path/to/logs/*.log | /path/to/bin/logstash --config logstash

Most of the time you'll probably want to add new Logstash filters to parse various components not already handled here. You may also need to update the dynamic template mapping used by the index if you add new fields or import/export a new dashboard/visualisation from/to Kibana.

The mapping file is located in the elasticsearch directory in the file elasticsearch-template.json. After editing, you'll need to update the mapping in your Elasticsearch cluster with the curl command above and reindex any log files.

All of the filters for a base component of logging in Elasticsearch go into a seperate logstash/conf.d/50-filter-<component>.conf Logstash configuration file. So for example, filters for index.shard and index.fielddata log messages go into a 50-filter-index.conf Logstash configuration file.

The following standard input, filter and output configuration files also exist:

• 10-input-stdin.conf: for reading stdin.
• 40-filter-base.conf: base filter configuration to parse timestamp, log level and component.
• 45-filter-exceptions.conf: parses multiline java stack traces.
• 90-output-dots.conf: outputs a . to the console for each log line processed.

The dashboards and visualisations shown in the screenshot are in the kibana directory in the export.json file.