this fork is intended to make a dockerized version of ElastiFlow

there are two containers:

1. elastiflow - contains the whole ELK stack with the logstash service configured to process a netflow v5|9
2. elastic.slave - a secondary elasticsearch node

both docker containers are build up from the sebp/elk docker image

to run services just issue a command:

docker-compose up --build

all the information below left as is from the parent repository

ElastiFlow™ provides network flow data collection and visualization using the Elastic Stack. As of version 2.0.0 it supports Netflow v5/v9, sFlow and IPFIX flow types (1.x versions support only Netflow v5/v9).

I was inspired to create ElastiFlow™ following the overwhelmingly positive feedback received to an article I posted on Linkedin... WTFlow?! Are you really still paying for commercial solutions to collect and analyze network flow data?

ElastiFlow™ is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. Refer to the following compatibility chart to choose a release of ElastiFlow™ that is compatible with the version of the Elastic Stack you are using.

NOTE: The instructions that follow are for ElastiFlow™ 2.x.

Currently there is no specific configuration required for Elasticsearch. As long as Kibana and Logstash can talk to your Elasticsearch cluster you should be ready to go. At high ingest rates (>10K flows/s), or for data redundancy and high availability, a multi-node cluster is recommended.

To use ElastiFlow™ you will need to install the community supported sFlow codec for Logtsash. It is also recommended that you always use the latest version of Netflow codec. This can achieved by running the following commands:

$ LS_HOME/bin/logstash-plugin install logstash-codec-sflow
Validating logstash-codec-sflow
Installing logstash-codec-sflow

$ LS_HOME/bin/logstash-plugin update logstash-codec-netflow
Updating logstash-codec-netflow
Updated logstash-codec-netflow 3.10.0 to 3.11.2

There are four sets of configuration files provided within the logstash/elastiflow folder:

logstash
+-- elastiflow
    |-- conf.d  (contains the logstash pipeline)
    |-- dictionaries (yaml files used to enrich raw flow data)
    |-- geoipdbs  (contains GeoIP databases)
    +-- templates  (contains index templates)

Copy the elastiflow directory to the location of your Logstash configuration files (e.g. on RedHat/CentOS this would be /etc/logstash/elastiflow.

Rather than directly editing the pipeline configuration files for your environment, environment variables can be used. The supported environment variables are:

The files profile.d/elastiflow.sh and logstash.service.d/elastiflow.conf are provided to help you setup the environment variables. For example, if you will be running Logstash using systemd, simply complete following steps...

1. Edit the provided logstash.service.d/elastiflow.conf for your environment and needs.
2. Copy the file to /etc/systemd/system/logstash.service.d/elastiflow.conf
3. Run systemctl daemon-reload to pickup the new systemd configuration files
4. Start Logstash by running systemctl start logstash

WARNING: DNS resolution of IP addresses to hostnames is controlled by the ELASTIFLOW_RESOLVE_IP2HOST environment variable. Within the pipeline caching is disabled for the dns filter as this causes performance issues due to the blocking nature of the cache lookups. For the best DNS performance it is recommended to use a local dnsmasq process to handle caching and to forward any unresolved lookups to upstream DNS servers. This is the reason that ELASTIFLOW_NAMESERVER defaults to 127.0.0.1. When receiving very high rates of flow records, leaving DNS resolution disabled will ensure the best performance.

Logstash can be configured to load the ElastiFlow™ pipeline using one of the following methods:

• From the command line using the path.config option.

$ LS_HOME/bin/logstash --path.config=/etc/logstash/elastiflow/conf.d

• By setting path.config in logstash.yml

path.config: /etc/logstash/elastiflow/conf.d

• If using Logstash 6.0 or above, ElastiFlow™ can also be started ba adding it to pipelines.yml

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d"

As of Kibana 5.6 an API (yet undocumented) is available to import and export Index Patterns. The JSON file which contains the Index Pattern configuration is kibana/elastiflow.index_pattern-json. To setup the elastiflow-* Index Pattern run the following command:

curl -X POST -u USERNAME:PASSWORD http://KIBANASERVER:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/PATH/TO/elastiflow.index_pattern.json

Finally the vizualizations and dashboards can be loaded into Kibana by importing the elastiflow.dachboards.json file from within Kibana. This is done from the Management - > Saved Objects page.

The following dashboards are provided.

NOTE: The dashboards are optimized for a monitor resolution of 1920x1080.

There are separate Top-N dashboards for Top Talkers, Top Services and Top Conversations.

There are separate Geo Loacation dashboards for Client/Server and Source/Destination perspectives.

Provides a view of traffic to and from Autonomous Systems (public IP ranges)

This product includes GeoLite data created by MaxMind, available from (http://www.maxmind.com)