kurtado / quick-elk Goto Github PK

View Code? Open in Web Editor NEW

71.0 14.0 31.0 3.69 MB

Install and run an ELK stack on your laptop, lickety-split!

Shell 100.00%

quick-elk's Introduction

quick-elk

Install and run an ELK stack on your laptop, lickety-split!

Now featuring Kibana 4... Hope you enjoy it... Play around a bit and let me know of any issues. Thanks!

Quick Install

Simply clone this repo and run install.sh. It will automatically download versions of Elasticsearch, Logstash and Kibana and install them in a local directory. (Kibana will be installed as an Elasticsearch plugin).

The install script will also ingest a dataset containing 2012 NFL play-by-play data, plus install a very basic Kibana dashboard (containing only one row + panel).

To re-ingest the NFL data, run this command (or similar, depending on path, etc):

cat 2012_nfl_pbp_data.csv| ./logstash-1.4.2/bin/logstash -f nfl.conf

Apache Logs

This bundle also includes a configuration allowing easy ingestion of Apache webserver logs into Elasticsearch via Logstash. Simply place a file called logs in the directory and run the following command:

./logstash-1.4.2/bin/logstash -f apachelog.conf < logs

Note that the logs file must be in the Apache Combined log format for this to run out-of-the-box.

Twitter Streaming API

Another interesting out-of-the-box demo is possible using the twitter.conf Logstash configuration file. You'll need to acquire your Twitter credentials by visiting http://dev.twitter.com/ and creating an application via the "My Applications" link. Substitute your twitter oAuth credentials for the placeholders in the twitter.conf file, and run:

./logstash-1.4.2/bin/logstash -f twitter.conf

This will populate an Elasticsearch index called "tweets" with tweet data. To change the twitter keywords, simply modify the keywords array in the configuration file.

quick-elk's People

Contributors

Stargazers

Watchers

quick-elk's Issues

Failing to run `kibana-4.0.1-darwin-x64/bin/kibana`

lsoave@basenode:~/elk$ sh -x  kibana-4.0.1-darwin-x64/bin/kibana
+ SCRIPT=kibana-4.0.1-darwin-x64/bin/kibana
+ [ -h kibana-4.0.1-darwin-x64/bin/kibana ]
+ dirname kibana-4.0.1-darwin-x64/bin/kibana
+ DIR=kibana-4.0.1-darwin-x64/bin/..
+ NODE=kibana-4.0.1-darwin-x64/bin/../node/bin/node
+ SERVER=kibana-4.0.1-darwin-x64/bin/../src/bin/kibana.js
+ CONFIG_PATH=kibana-4.0.1-darwin-x64/bin/../config/kibana.yml NODE_ENV=production exec kibana-4.0.1-darwin-x64/bin/../node/bin/node kibana-4.0.1-darwin-x64/bin/../src/bin/kibana.js
kibana-4.0.1-darwin-x64/bin/../node/bin/node: 1: kibana-4.0.1-darwin-x64/bin/../node/bin/node: ����������: not found
kibana-4.0.1-darwin-x64/bin/../node/bin/node: 2: kibana-4.0.1-darwin-x64/bin/../node/bin/node: Syntax error: ")" unexpected

Marvel and elasticsearch-kopf are not in the locations specified by the script

As subject. I couldn't find obvious alternative download locations.

Their absence seems to cause the whole thing to fail, since I've had the stack running using this script several times in the past.

$ ./install.sh
Installing ELK stack into /home/pgw22/elk
Logstash already Downloaded
Elasticsearch already Downloaded
Downloaded... Now installing
Unpacking logstash
Unpacking Elasticsearch
Unpacking nfl dataset
Installing Marvel latest
-> Installing elasticsearch/marvel/latest...
Trying http://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip...
Trying http://search.maven.org/remotecontent?filepath=elasticsearch/marvel/latest/marvel-latest.zip...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/elasticsearch/marvel/latest/marvel-latest.zip...
Trying https://github.com/elasticsearch/marvel/archive/latest.zip...
Trying https://github.com/elasticsearch/marvel/archive/master.zip...
Failed to install elasticsearch/marvel/latest, reason: failed to download out of all possible locations..., use --verbose to get detailed information
Kibana already Downloaded
Kibana already installed
Installing kopf latest
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip...
Failed to install lmenezes/elasticsearch-kopf, reason: failed to download out of all possible locations..., use --verbose to get detailed information
Starting Elasticsearch to run in the background.
loading nfl data using logstash
Using milestone 2 filter plugin 'csv'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
Using milestone 1 codec plugin 'dots'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable: [503] <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Temporarily Unavailable</title>
</head><body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

  __raise_transport_error at /home/pgw22/elk/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/base.rb:132
          perform_request at /home/pgw22/elk/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/base.rb:227
          perform_request at /home/pgw22/elk/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:20
          perform_request at /home/pgw22/elk/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/client.rb:102
          perform_request at /home/pgw22/elk/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/namespace/common.rb:21
             get_template at /home/pgw22/elk/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/actions/indices/get_template.rb:24
         template_exists? at /home/pgw22/elk/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:132
         template_install at /home/pgw22/elk/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:21
                 register at /home/pgw22/elk/logstash-1.4.2/lib/logstash/outputs/elasticsearch.rb:259
                     each at org/jruby/RubyArray.java:1613
             outputworker at /home/pgw22/elk/logstash-1.4.2/lib/logstash/pipeline.rb:220
            start_outputs at /home/pgw22/elk/logstash-1.4.2/lib/logstash/pipeline.rb:152
Now browse to:
http://localhost:9200/_plugin/marvel
or
http://localhost:9200/_plugin/kopf
or
http://localhost:9200/_plugin/kibana
or
http://localhost:9200/_plugin/kibana/index.html#/dashboard/file/week-by-week.json

Missing packages to run install.sh

be sure to have openjdk-7-jre and unzip packages installed prior to launch the install script

Ubuntu 14.04 Linux Install

So I had an issue trying to fight to get this to install on a new ubuntu 14.04 install

I modified "install.sh" line 11
KIBANA_OS=linux-x64

was originally =darwin-x64

and I finally got kibana to start on port 5601

Just wanted to make someone aware, for those that don't use Mac's :)

Twitter stream error- A plugin had an unrecoverable error. Will restart this plugin.

Updated twitter.conf with correct credentials. However when I run /elk/logstash-1.4.2/bin/logstash -f twitter.conf, I get below error. I truncated the output a bit for brevity. So it looks like twitter is responding with JSON that the plugin is not able to parse? Am I missing anything?

Using milestone 1 input plugin 'twitter'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}

.....A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Twitter consumer_key=>"xxxx", oauth_token=>"xxxx", keywords=>["nfl", "football", "touchdown"]>
  Error: unexpected token at '{"created_at":"Mon Nov 09 03:17:23 +0000 2015","id":663555956903686145,"id_str":"663555956903686145","text":"RT <I purposefully truncated here>"

jruby complete jar not accessible eventhough present in the required folder

installation script when run on cygwin completed with errors : one of them specific to logstash and reference to jruby complete " Unable to access jarfile /home/name/elk/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar"

Is there a fix for this case.

Questions: How to make twitter.conf index as "tweets-2015.04.19"

Thanks for the quick-elk for twitter.conf
However, the reality is that the size of tweets index will grow up very soon , therefore, it seems to be better to separate the index by date . Is it possible to set in the twitter.conf or need to configure somewhere ?

Unable to install properly

I get the following error when I try to install, ES and Logstash run properly but kibana crashes

./kibana-4.0.1-darwin-x64/bin/../node/bin/node: 1: ./kibana-4.0.1-darwin-x64/bin/../node/bin/node: ??????: not found
./kibana-4.0.1-darwin-x64/bin/../node/bin/node: 2: ./kibana-4.0.1-darwin-x64/bin/../node/bin/node: Syntax error: ")" unexpected
^C
[1]+ Exit 2 ./kibana-4.0.1-darwin-x64/bin/kibana

I am running this on an ubuntu 14.04 64 bit machine,