Need to upgrade Kibana to 4.2 and create bats test for status to be green

See https://www.elastic.co/guide/en/kibana/current/status.html

Setting module parameters should occur with a .conf file in /etc/modprobe.d as detailed in the RHEL documentation.

/etc/pf_ring/pf_ring.conf doesn't actually do anything unless you use the PF_RING init script. This isn't necessary unless you're using the ZeroCopy commercial drivers, as it requires a daemon for management.

Apparently, the install fails with Bro (full transcript attached).

rock_install_fail_20160119.txt

User Story: As a power user, I want a web UI to manage advanced Elasticsearch queries so that I can access raw data and aid in development of new features.

Sense is now a Kibana plugin. This should be added to the download / offline install procedures.

https://www.elastic.co/guide/en/sense/current/installing.html

Need a script to configure kibana to load indexes upon starting of kibana

After the first Bro log rollover, the unified2 reader seems to stop picking up snort alerts.

Temp workaround is (possibly) to add a "-k" to the curl syntax. Cert handling/generation upstream may need adjusted to permanently fix.

For prettier, sexier graphs, we should install timelion, which eats timeseries data for breakfast (lunch is zebras).

For connected systems, the way to do this is as follows:

./bin/kibana plugin -i kibana/timelion

Need to research offline install methods.

It appears that pf_ring is preventing the monitor interface from starting (or staying up). The install appeared to work properly, all the other services are working.

Troubleshooting:

• reboot
• restart interface
• manually stop/start pf_ring
• cursed
• stopped ROCK via rock_stop
• started ROCK via rock_start
• "Basic Troubleshooting" from rocknsm.io

Output from rock_status

[root@simplerockbuild ~]# rock_status
 ✗ Check each monitor interface is live
   (in test file /usr/local/bin/rock_status, line 24)
     `[ $packets -gt 0 ]' failed
   eno33554960 had 0 packets in 5 secs.
 ✓ Check for interface errors
 ✓ Check monitor interface for tx packets
 ✓ Check PF_RING settings
 ✓ Check that broctl is running
 ✓ Check for bro-detected packet loss
 ✓ Check that zookeeper is running
 ✓ Check that zookeeper is listening
 ✓ Check that client can connect to zookeeper
 ✓ Check that kafka is running
 ✓ Check that kafka is connected to zookeeper
 ✓ Check that logstash is running
 ✓ Check that elasticsearch is running
 ✓ Check that kibana is running

14 tests, 1 failure

systemctl log
journal log

Simple Documentation, no change requested.

Decided to do a little tinkering with SimpleRock from a Windows 7 Machine with Virtualbox 5.0.12 and ran into a slight problem with the Virtualbox provider configs for --hostonlyadapter2. From the virtualbox logs I found this useful error message:

00:00:00.492329 VMSetError: Nonexistent host networking interface, name 'vboxnet0'

Switching vboxnet0 with the actual host only adapter name pulled from ipconfig /all did the trick.

Create a systemd service that runs idstool to generate JSON events for snort. This then needs to be ingested by logstash as is currently done with suricata eve.json

See https://blog.jasonish.org/2014/04/16/snort-logstash-elastic-search-and-kibana/

This should be relatively easy to implement with Logstash aggregating, but I've had a couple people ask lately for an ability to setup a clustered mode similar to that of Security Onion. Just dropping here for idea recording.

plugin:marvel Marvel version 2.2.0 is not supported with Kibana 4.3.1
error on current build of the ROCK

Installation of SimpleRock failed with non-vagrant VM method. Used:

• Fusion Pro
• CentOS 7.2 min.
• 4 cores
• 8GB RAM

Beginning of installation failure as follows:

+++++++++++++++++++++++++++++++++++++++++++

* execute[start_bro] action run

Error executing action `run` on resource 'execute[start_bro]'

Mixlib::ShellOut::ShellCommandFailed

+++++++++++++++++++++++++++++++++++++++

Attached is the full transcript:

SimpleRock_Snort-Build_install_failure.txt

With the exception of wget, opsode runs a pretty nice, and well maintained base set of boxes for various hypervisors.

Here's the kickstart.
https://github.com/chef/bento/blob/master/http%2Fcentos-7.2%2Fks.cfg

For suricata eve.json, the project provides a recommended configuration

/data/suricata/*.log /data/suricata/*.json /data/suricata/unified2*.alert
{
    rotate 3
    missingok
    nocompress
    create
    sharedscripts
    postrotate
            /bin/kill -HUP $(cat /var/run/suricata.pid)
    endscript
}

Users can benefit from being able to generate alerting conditions based on frequency, volume, and content based (keyword and diff) searches that are scheduled to run periodically and when they meet a defined condition they report to another application (like a chat client, email, tracker, etc...). It looks like Yelp has built a nice framework for this in ElastAlert that could be a suitable candidate for inclusion in SimpleROCK.

Help us Obi-wan @dcode, you're our only hope!

Maybe something like:

• Add prompt to install ansible if it is not found (check other base requirements?)
• Host configuration (sensor interfaces, hostname, data dir, NTP, etc)
• Offline vs. Online
• Advanced (feature selection: turn off kibana, for example; other things)
• Run playbook (yes/no); indicate that config file is located at /etc/rocknsm/config.yml

From Derek:
How To Update the Ethernet Monitoring Interface AFTER Installation
Modify the following:

1. /opt/bro/etc/node.cfg
2. /sbin/ifup-local
3. The interface script (ifcfg-foo)
4. The steno config

Update Wiki (or readme) with enabling tcpreplay procedure for testing:

vi /etc/pf_ring/pf_ring.conf

change enable_tx_capture to "1"

Restart system

• I am not sure if there is a way to do this without a restart

I need to be able to specify the Ethernet monitoring interface. Short-term fix is to manually modify the Chef code to specify the specific interface that monitoring needs to take place on, but not sure how to do that.

By default, Kafka dumps topic data to /tmp/kafka-logs. This is....not exactly good. Should consider moving to /data/kafka to be consistent and make it easier to allocate data space.

This is set in /opt/kafka/config/server.properties.

Propose change:

log.dirs=/tmp/kafka-logs

to

log.dirs=/data/kafka

The bro-cut command is not located in $PATH

echo $PATH output currently shows
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/kafka/bin:/root/bin

bro-cut works by using /opt/bro/bin/bro-cut currently.

CentOS-CR.repo doesn't exist until yum update -y && yum upgrade -y, might want to add this to the prep or the recipe.

Also I think yum install -y kernel-headers kernel-devel && reboot should be part of the prep or part of the recipe.

Super low priority request but I noticed that a couple of the diagrams under https://github.com/CyberAnalyticDevTeam/SimpleRock/tree/gh-pages/images are using outdated logos for Elasticsearch, Logstash, and Kibana. The most recent product logos are available at https://www.elastic.co/brand. Thanks!

After installing SimpleRock per the instructions on Github and trying to execute rock_start, I get error "command not found." Error codes at the end of installation may give a clue. Screen shot attached.

Heads-up. Checked out the Snort setup for SimpleRock with a friend and it looks like the new build with Snort has a misconfiguration with Share Object Rules in snort.conf and pulledpork.

http://blog.talosintel.com/2009/01/using-vrt-certified-shared-object-rules.html

More info on Shared Object rules:

http://blog.snort.org/2011/02/snort-shared-object-rules.html
However, some may be finding it difficult to use the rules, so let me point you to a couple guides. One guide is here, on Snort.org, at the bottom of the "platform" list. The VRT also has a blog post that can help you install the Shared Object rules.

But, by far, the easiest way to use Shared Object rules reliably is through the configuration and use of a tool called PulledPork, which JJ Cummings of Sourcefire is the primary author of. After the configuration of PulledPork, the tool will generate the Shared Object rule stubs for you, and place everything in the correct directories for ease of use. This is amongst the many features of PulledPork (including flowbit dependency resolving) which are useful.

RockIssuesHelp.txt

Upon setup, would like to have the option to monitor 2-4 Ethernet interfaces instead of the default single interface. How would this be done after installation?

To free space on ROCK without without permanently losing data, include or document a method to export indexes from elasticsearch. Should also be able to import previously exported data.

Added bonus: This would allow Kibana dashboards to be exported and imported.

Some helpful items:

• Number of indexes/shards
• Number of records
• Free space in /data
• Kafka offset

The duration field is being indexed and valued, but as an individual "string" such as 0.0000986, 0.0000987, etc.

This is crippling because you can't visualize strings over time due to obvious reasons.

Side Note: Using Offline ISO Installation Build.

I'm having an issue running Vagrant.

SimpleRock$ vagrant up
/opt/vagrant/embedded/gems/gems/vagrant-1.8.1/lib/vagrant/pre-rubygems.rb:31: warning: Insecure world writable dir /opt in PATH, mode 040777
/opt/vagrant/embedded/gems/gems/bundler-1.10.6/lib/bundler/shared_helpers.rb:78: warning: Insecure world writable dir /opt in PATH, mode 040777
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'relativkreativ/centos-7-minimal' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 5601 (guest) => 5601 (host) (adapter 1)
    default: 9200 (guest) => 9200 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'poweroff' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.

I opened VirtualBox and tried to start it manually and it said that there was a network interface issue.

Nonexistent host networking interface, name 'vboxnet0' (VERR_INTERNAL_ERROR).
Result Code: 
NS_ERROR_FAILURE (0x80004005)
Component: 
ConsoleWrap
Interface: 
IConsole {872da645-4a9b-1727-bee2-5585105b9eed}

I find it helpful to create a symlink from the "normal" directories to the modified directories. This reduces unfamiliarity to new users so they can just get the job done.

Need option for optimized settings for higher-speed monitoring interface if used for sniffing - 10GB Ethernet.
If this is best done/tweaked AFTER installation, then we should document in the Wiki.