marco-lancini / docker_offensive_elk Goto Github PK

View Code? Open in Web Editor NEW

136.0 136.0 34.0 811 KB

Elasticsearch for Offensive Security

License: Other

Shell 2.80% Python 83.36% Dockerfile 13.84%

docker docker-compose elasticsearch elk-stack kibana nmap pentesting

docker_offensive_elk's Introduction

💼 I am a Principal Security Engineer with experience building security functions at Fortune500, pre-IPO companies, and exponential growth startups. Currently, I own efforts to secure GitLab's SaaS cloud infrastructure and container technologies.
✍🏻 I curate CloudSecList, a newsletter that keeps thousands of security professionals informed about current happenings and news related to the security of cloud-native technologies, and CloudSecDocs, a website collecting and sharing my technical notes and knowledge on cloud-native technologies, security, technical leadership, and engineering culture.
📚 I blog about cloud security, strategy, and technical leadership, and I'm writing The CloudSec Engineer, a book on how to enter, establish yourself, and thrive in the cloud security industry as an individual contributor.
🔑 I also run a boutique security advisory firm, and I am an advisor and angel investor of a few early-stage startups.
🛡 I am a member of the AWS Community Builders program, part of the committee tasked with creating the Certified Kubernetes Security Specialist (CKS) Certification, and a mentor in the Lead the Future program, a non-profit that helps young Italian talents pursue a career in the STEM field.
💬 You can find additional information about me at: marcolancini.it

How to Reach Me

Tech Stack

Certifications

Accolades

👉 Subscribe to CloudSecList

docker_offensive_elk's People

Contributors

Stargazers

Watchers

docker_offensive_elk's Issues

Help needed to setup the project

Every version i try non of them working
the problem is
from Kibana Server not ready and cannot connect to elastic search

i did everything like tutorial said but, non of them is working why??

for elasticsearch py need to be 7.1.0

Change extensions/ingestor/requirements.txt to
elasticsearch==7.1.0

Elasticsearch not running on fresh install

I cloned the repo, then docker-compose-up -d (no need to create _data, as it exists). The errors below about Elasticsearch occurred. I don't know what is going on, or how to fix.

System info:

wfcody@hunt-bison:~/docker_offensive_elk$ uname -a
Linux hunt-bison 4.15.0-38-generic #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
wfcody@hunt-bison:~/docker_offensive_elk$ docker-compose -v
docker-compose version 1.21.2, build a133471
wfcody@hunt-bison:~/docker_offensive_elk$ docker -v
Docker version 18.09.0, build 4d60db4

Errors:

wfcody@hunt-bison:~/docker_offensive_elk$docker-compose up -d
Creating elk_elasticsearch ... done
Creating elk_logstash      ... done
Creating elk_kibana        ... done
Creating elk_ingestor      ... done
wfcody@hunt-bison:~/docker_offensive_elk$ curl -XPUT 'localhost:9200/nmap-vuln-to-es'
curl: (7) Failed to connect to localhost port 9200: Connection refused
wfcody@hunt-bison:~/docker_offensive_elk$ netstat -antp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:6012          0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.176.169:22       192.168.184.240:2614    ESTABLISHED -
tcp        0     36 192.168.176.169:22       192.168.184.240:1795    ESTABLISHED -
tcp        0      0 192.168.176.169:22       192.168.184.240:32705   ESTABLISHED -
tcp        0      0 192.168.176.169:22       192.168.184.240:32706   ESTABLISHED -
tcp        0      0 192.168.176.169:22       192.168.184.240:1796    ESTABLISHED -
tcp        0      0 192.168.176.169:22       192.168.184.240:2613    ESTABLISHED -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 ::1:6010                :::*                    LISTEN      -
tcp6       0      0 ::1:6011                :::*                    LISTEN      -
tcp6       0      0 ::1:6012                :::*                    LISTEN      -
tcp6       0      0 :::5601                 :::*                    LISTEN      -
tcp6       0      0 :::5000                 :::*                    LISTEN      -
wfcody@hunt-bison:~/docker_offensive_elk$ docker-compose down
Stopping elk_logstash      ... done
Stopping elk_kibana        ... done
Stopping elk_elasticsearch ... done
Removing elk_ingestor      ... done
Removing elk_logstash      ... done
Removing elk_kibana        ... done
Removing elk_elasticsearch ... done
Removing network docker_offensive_elk_elk
wfcody@hunt-bison:~/docker_offensive_elk$ docker ps -aq
wfcody@hunt-bison:~/docker_offensive_elk$ docker-compose up
Creating network "docker_offensive_elk_elk" with driver "bridge"
Creating elk_elasticsearch ... done
Creating elk_logstash      ... done
Creating elk_kibana        ... done
Creating elk_ingestor      ... done
Attaching to elk_elasticsearch, elk_logstash, elk_ingestor, elk_kibana
elk_elasticsearch | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elk_elasticsearch | [2018-12-05T20:31:29,687][INFO ][o.e.n.Node               ] [] initializing ...
elk_elasticsearch | [2018-12-05T20:31:29,698][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
elk_elasticsearch | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch | Caused by: java.lang.IllegalStateException: Failed to create node environment
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:273) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     ... 6 more
elk_elasticsearch | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
elk_elasticsearch |     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elk_elasticsearch |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elk_elasticsearch |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elk_elasticsearch |     at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?]
elk_elasticsearch |     at java.nio.file.Files.createDirectory(Files.java:682) ~[?:?]
elk_elasticsearch |     at java.nio.file.Files.createAndCheckIsDirectory(Files.java:789) ~[?:?]
elk_elasticsearch |     at java.nio.file.Files.createDirectories(Files.java:775) ~[?:?]
elk_elasticsearch |     at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     ... 6 more
elk_ingestor exited with code 0
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:33Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:33Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.23.0.2:9200"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:34Z","tags":["status","plugin:[email protected]","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_elasticsearch exited with code 1
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:36Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
elk_kibana       | {"type":"log","@timestamp":"2018-12-05T20:31:36Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
elk_elasticsearch | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elk_elasticsearch | [2018-12-05T20:31:29,687][INFO ][o.e.n.Node               ] [] initializing ...
elk_elasticsearch | [2018-12-05T20:31:29,698][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
elk_elasticsearch | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch | Caused by: java.lang.IllegalStateException: Failed to create node environment
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:273) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     ... 6 more
elk_elasticsearch | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
elk_elasticsearch |     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elk_elasticsearch |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elk_elasticsearch |     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elk_elasticsearch |     at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?]
elk_elasticsearch |     at java.nio.file.Files.createDirectory(Files.java:682) ~[?:?]
elk_elasticsearch |     at java.nio.file.Files.createAndCheckIsDirectory(Files.java:789) ~[?:?]
elk_elasticsearch |     at java.nio.file.Files.createDirectories(Files.java:775) ~[?:?]
elk_elasticsearch |     at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
elk_elasticsearch |     ... 6 more
elk_elasticsearch | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

can't create index: missing authentication token for REST request

I'm working through the instructions & hit a snag. I'm at "create an index", run curl & get this error back:

$ curl -XPUT 'localhost:9200/nmap-vuln-to-es'
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/nmap-vuln-to-es]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/nmap-vuln-to-es]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Nmap Index not showing on management console

Having an issue with the index that was created using the command
curl -XPUT 'localhost:9200/nmap-vuln-to-es'
not showing up in the management console.
Checking the indices via curl gave this result
curl -X GET 'localhost:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open nmap-vuln-to-es OUn7TnHkQ42bhXwstSjAuw 5 1 0 0 1.2kb 1.2kb

Yet pulling up the web interface I am presented with the following.

This is on a fresh install of Docker running on MacOS 10.14.3 I'm new to Docker and ELK so I'm a little stumped

Deprecated method in VulntoES.py

In the line 90
self.es.index(index=self.index_name, body=json.dumps(to_upload))

from the ingestor logs return some error related with the deprecation of the index body used. Apparently I was able to solve this problem changing body for document index
self.es.index(index=self.index_name, document=json.dumps(to_upload))

By the way I still wasn't able to run correctly this docker-compose but I wanted to log this problem I found.

Couldn't find any Elasticsearch data

Hi,

I've created successfully the index
Invoke-WebRequest -Method PUT -Uri "http://localhost:9200/nmap-vuln-to-es"

In Kibana, it's never detected. Do you have any idea why ?

Elastiksearch container is started. I have the feeling that both containers can't speak to each other. Please note that I'm running it on Windows + Visual Studio code.

Thank you.

Update VulntoES.py to remove depricated methods

Running ingestor gives the following warning:
/opt/VulntoES/VulntoES.py:48: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

Updating code to remove soon to be depricated method requested.

Add Visualizations to import directly

Visualization

Hello!

I wanted to use your dashboard, but it appears that you didn't include any of the visualizations.

Would you be able to upload those?

Parser issue

Hello Marco,

Appreciate the efforts you have put in here and thanks for sharing it with the community.

A quick observation based on some testing I did.

Scenario 1
running "nmap -T4 -A -p 1-1000 -oX 1401.xml 192.168.2.140 " the xml file contains

      1401.xml:<address addr="192.168.2.140" addrtype="ipv4"/>

Scenario 2
running " nmap -A -O -oX 1402.xml 192.168.2.140" the xml file contains

     1402.xml:<address addr="192.168.2.140" addrtype="ipv4"/>
     1402.xml:<address addr="xx:xx:xx:xx:xx:xx" addrtype="mac" vendor="Super Micro Computer"/> ---- this line is additional

Note the difference of an additional line with address addr in scenario2. This is where your VulntoES.py script uses the second entry and populates Elasticsearch with the following entry

protocol:
tcp
scanner:
nmap
service:
ssh
ip:
xx:xx:xx:xx:xx:xx -- mac address
product_version:
7.6
port:
22
state:
open
scripts.ssh-hostkey:
2048 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
time:
May 29th 2019, 10:39:06.000
product_name:
OpenSSH
_id:
iNE_BGsB7OQXLf0qdKjo
_type:
vuln
_index:
nmap-vuln-to-es
_score:
1

It applies this to each subsequent port entry for that host within the file.

Will appreciate if you can please take a look into it and update your script accordingly.

Thanks

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.