honeynet / beeswarm Goto Github PK

Extend the Beekeeper web interface to include generation of Hive/Feeder configurations.
It would be best if all configuration could be done with the web interface. The work process could be something like:

1. Open webinterface
2. Click "Create Hive"
3. (enters some options)
4. Click save
5. Download beeswarm to the machine (vm) which act as hive. (pip install beswarm)
6. Starts beeswarm in hive mode with a configuration url (something like beeswarm --hive --config=http://beekeeper/ws/getconfig?token=abc123 where token would be a unique identifier
7. (hive would now configure itself using data fra beekeeper)

This very same procedure would go for feeder deployment.

2013-05-11 01:58:13,478 (beeswarm.hive.hive) Hive running - see log file (hive.log) for attack events.
2013-05-11 01:59:48,778 (beeswarm.hive.models.session) pop3 authentication attempt from 127.0.0.1. [username:test, password:test] (dda8bf2a-9f86-4598-bc7c-d6351139ae50)
Traceback (most recent call last):
  File "/home/czardoz/env_vir/bee/local/lib/python2.7/site-packages/gevent-0.13.8-py2.7-linux-i686.egg/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/czardoz/git/beeswarm/beeswarm/hive/capabilities/pop3.py", line 68, in handle_session
    return_value = func_to_call(session, gsocket, msg)
  File "/home/czardoz/git/beeswarm/beeswarm/hive/capabilities/pop3.py", line 202, in cmd_list
    index = int(argument) - 1
ValueError: invalid literal for int() with base 10: ''
<Greenlet at 0xa1cc1bc: <bound method pop3.handle_session of <beeswarm.hive.capabilities.pop3.pop3 object at 0xa1c86ac>>(<HiveSocket at 0xa1ce3ac fileno=18 sock=127.0.0.1:, ('127.0.0.1', 40363))> failed with ValueError

2013-05-11 01:59:48,779 (beeswarm.hive.helpers.streamserver) Unhandled "invalid literal for int() with base 10: ''" exception caused a greenlet to crash: <Greenlet at 0xa1cc1bc: <bound method pop3.handle_session of <beeswarm.hive.capabilities.pop3.pop3 object at 0xa1c86ac>>(<HiveSocket at 0xa1ce3ac fileno=18 sock=127.0.0.1:, ('127.0.0.1', 40363))>

Steps to reproduce:

1. Run the Hive:

$ beeswarm -hi -v

1. Run this script:

import poplib
M = poplib.POP3('localhost', 110)
M.user('test')
M.pass_('test')
numMessages = len(M.list()[1])
for i in range(numMessages):
    for j in M.retr(i+1)[1]:
        print j

After the first connection to the ftp capability all other capabilities hangs forever. I suspect that this needs to be fixed in the pyftpdlib.ioloop.

Secure communications between feeder/hive and beekeeper. Preferably using certificates.

Currently Sessions (hive/attackers) and Honeybees (feeder) entities are written to the database without correlation.

All Session which are Honeybees needs to be marked as such, to identify this one would have to compare data from the two entities.

Some notes on classification:

• Honeybee (which succeeded) and matching Session
  • Everything ok! Delete Session.
• Honeybee (which succeeded) but no matching Session
  • Potential MiTM attack (attacker mimics server).
• Session with no Honeybee
  • Potential brute force attack.
• Session with matching username/password in honeybee.
  • Potential MiTM attack (attacker interceptes and reuse credentials).

This errors occurs when unit testing, but triggers no test error. (failure after assert?)

(beeswarm)Johnnys-iMac:beeswarm jkv$ nosetests
........Traceback (most recent call last):
  File "/Users/jkv/virtualenvs/beeswarm/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/jkv/repos/beeswarm/hive/capabilities/http.py", line 96, in handle_session
    options=self._options)
  File "/Users/jkv/repos/beeswarm/hive/capabilities/http.py", line 41, in __init__
    BaseHTTPRequestHandler.__init__(self, request, client_address, server)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 639, in __init__
    self.handle()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 343, in handle
    self.handle_one_request()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 331, in handle_one_request
    method()
  File "/Users/jkv/repos/beeswarm/hive/capabilities/http.py", line 68, in do_GET
    self.send_html('base.html')
  File "/Users/jkv/repos/beeswarm/hive/capabilities/http.py", line 73, in send_html
    file = self.vfs.open(filename)
  File "/Users/jkv/virtualenvs/beeswarm/lib/python2.7/site-packages/fs/wrapfs/__init__.py", line 33, in wrapper
    return func(self,*args,**kwds)
  File "/Users/jkv/virtualenvs/beeswarm/lib/python2.7/site-packages/fs/wrapfs/__init__.py", line 150, in open
    f = self.wrapped_fs.open(self._encode(path), wmode, **kwargs)
  File "/Users/jkv/virtualenvs/beeswarm/lib/python2.7/site-packages/fs/errors.py", line 229, in wrapper
    return func(self,*args,**kwds)
  File "/Users/jkv/virtualenvs/beeswarm/lib/python2.7/site-packages/fs/osfs/__init__.py", line 211, in open
    return open(sys_path, mode, kwargs.get("buffering", -1))
ResourceNotFoundError: Resource not found: /base.html
<Greenlet at 0x1032d9f50: <bound method http.handle_session of <hive.capabilities.http.http object at 0x102c42b50>>(<socket at 0x102c56a10 fileno=12 sock=127.0.0.1:80, ('127.0.0.1', 63623))> failed with ResourceNotFoundError

.............
----------------------------------------------------------------------
Ran 21 tests in 1.868s

OK

Some logon attempts does not get logged because the attacker does not conform to protocol standards (surprise, surprise).
A hard timeout must be enforced on all protocol to ensure the session is ended in a timely manner.

• Login at Hive
• Download random file

The pop3 and imap capabilities needs a stable source of mail.
Would be spiffy to hook into the spamcan hpfeeds channel to extract this information.

If hpfeeds is down the hpfeed's greenlet will crash. This must be fixed.

2013-04-02 02:20:04,210 (hive.consumer.loggers.hpfeed) Connecting to feed broker at hpfriends.honeycloud.net:20000
2013-04-02 02:20:04,251 (hive.consumer.loggers.hpfeed) Connected to hpfeed broker.
Traceback (most recent call last):
  File "/home/jkv/virtenv_bee/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/jkv/beeswarm/hive/consumer/consumer.py", line 52, in start
    active_loggers = self.get_loggers()
  File "/home/jkv/beeswarm/hive/consumer/consumer.py", line 89, in get_loggers
    hive_logger = l()
  File "/home/jkv/beeswarm/hive/consumer/loggers/hpfeed.py", line 78, in __init__
    self.connect()
  File "/home/jkv/beeswarm/hive/consumer/loggers/hpfeed.py", line 111, in connect
    self.broker_read()
  File "/home/jkv/beeswarm/hive/consumer/loggers/hpfeed.py", line 82, in broker_read
    data = self.socket.recv(1024)
  File "/home/jkv/virtenv_bee/local/lib/python2.7/site-packages/gevent/socket.py", line 432, in recv
    wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
  File "/home/jkv/virtenv_bee/local/lib/python2.7/site-packages/gevent/socket.py", line 169, in wait_read
    switch_result = get_hub().switch()
  File "/home/jkv/virtenv_bee/local/lib/python2.7/site-packages/gevent/hub.py", line 164, in switch
    return greenlet.switch(self)
timeout: timed out
<Greenlet at 0x8c2d25c: <bound method Consumer.start of <hive.consumer.consumer.Consumer instance at 0x8c28f6c>>> failed with timeout

(In general the consumer needs to be hardened)
Example of a crash:

Traceback (most recent call last):
  File "/home/jkv/virtenv_bee/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/jkv/beeswarm/hive/consumer/consumer.py", line 63, in start
    log.log(session)
  File "/home/jkv/beeswarm/hive/consumer/loggers/hpfeed.py", line 127, in log
    data = json.dumps(session.to_dict(), default=self.json_default)
  File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid continuation byte
<Greenlet at 0x8ac798c: <bound method Consumer.start of <hive.consumer.consumer.Consumer instance at 0x8ae06cc>>> failed with UnicodeDecodeError

This work package includes work on all capabilities and honeybees to make them look more legit and interesting. See this diagram for a overview of the type of interactivity each honeybee should have.

In summary, this work item includes the following work:

• Hive
  • All capabilities must have enough functionality to look/feel legit at first looks.
  • The pop3 capability should generate (or fetch) a stable source of mails. (spam?)
• Feeder
  • All type A (see diagram) honeybees must be intelligent enough to react dynamically to the capability they connect to. We do NOT want to script this - basically we want a protocol specific Sense->Decide->Act loop for every type of honeybee. Remember that the honeybee might be subject to a MITM attack - which which essentially means that the feeder might end up communicating directly with the attacker.
  • All type B honeybees are pretty autonomous in the real world and it should be pretty easy to replicate this to the honeybees. One isssue is that we need to provide a stable and realistic mail source for the pop3 and smtp capabilities.

It is expected that at least 2 test cases for each honeybee are shipped with this work package.

A critical part of the whole system is being able to differentiate honeybee activity from malicious activity. One part of doing this is to keep exact timings.

At regular intervals (every few hours) we need to check timings using ntp (tip: https://pypi.python.org/pypi/ntplib/). If the timings are too far off (+/- 5 seconds?) we crash the application. We don't want to implement all kinds of time adjustments mechanisms so crashing is just fine.
This functionality should take the following options from the config file:

• Enabled/Disabled (default: disabled)
• Poll interval in hours (default: 5)
• ntp pool (default: pool.ntp.org)

This issue is assigned to Google Summer Of Code student Aniket Panse (@czardoz).

The user (feeder or attacker) should be able to interact with the hive capability in the following ways:

• List files (ls).
• Navigate directories (cd, pwd)
• Print files (cat)
• Extract various system information (uname, uptime, date, etc.)
• The welcome message must be realistic (including last login ip, free disc space, ip address, etc). See this for an example.

It would be beneficial if the implementation could be used for the telnet capability also.

It would properly be worthwhile to investigate how this is implemented in Kippo, we might be able to reuse (with proper attribution of course) key parts of the kippo implementation.

It is expected that at least 5 test cases are shipped with this work package.

This issue is assigned to Google Summer Of Code student Aniket Panse (@czardoz).

The Beekeeper web application needs the following:

• All /sessions pages must be paginated and sortable. Default sort would be descending sort on timestamp.
• The database must be searchable (ip, protocol, time, etc).
• Functionality to export data in a easily parsable format (xml or json).
• The webui needs to be protected by a login/password. Password can only be changed with shell access. (KISS!)
• Various forms needs to be created/improved:
  • Create new hive
  • Create new feeder
• The design (html/css) in general needs some serious hipsterness.
• A overview of feeders/hive clients should be possible.
  • The overview should be displayed as a list (WATable?)
  • The list should display last activity from the client (hive/feeder), how many sessions that have been transmitted to/from the client, name of client, data of creation?, etc.
  • It should be possible to delete and add hive/feeders from the overview.
  • Small red X on the right to delete
  • Maybe green "Add" buttons at the bottom...

It is expected that every single route has test cases.

It would be beneficial if the following options were to be added to the ftp capability:

banner = Microsoft FTP Server
max_attempts = 3

Existing code already transfers the [cap_ftp] configuration section to the ftp capability. What is left to be done is modification of the BeeswarmFTPServer class.

After four failed logon attempts the telnet capability issues a "logged in" and starts accepting commands. As per the telnetsrvlib a exceptions has to be thrown in the authCallback - returning False is not enough.

It leaves a very cryptic error:

2013-06-02 17:22:12,446 (root) Caught exception: (-1, '2\x00\x00\x000\x00\x00\x00 \x00\x00\x00M\x00\x00\x00i\x00\x00\x00c\x00\x00\x00r\x00\x00\x00o\x00\x00\x00s\x00\x00\x00o\x00\x00\x00f\x00\x00\x00t\x00\x00\x00 \x00\x00\x00E\x00\x00\x00S\x00\x00\x00M\x00\x00\x00T\x00\x00\x00P\x00\x00\x00 \x00\x00\x00M\x00\x00\x00A\x00\x00\x00I\x00\x00\x00L\x00\x00\x00 \x00\x00\x00s\x00\x00\x00e\x00\x00\x00r\x00\x00\x00v\x00\x00\x00i\x00\x00\x00c\x00\x00\x00e\x00\x00\x00 \x00\x00\x00r\x00\x00\x00e\x00\x00\x00a\x00\x00\x00d\x00\x00\x00y\x00\x00\x00\r\x00\x00\x00') (<class 'smtplib.SMTPConnectError'>)

Only basic functionality, it would be nice to catch:

• Login and password
• Various RDP metrics which can be used to fingerprint attacker / attackers tool

There seems to be some related work being done in the Amun project.

• Implement webclient on Hive/Feeder to facilitate data transfer. (Requests is recommended as client library)
• Implement receiving webservice on Hive.

All traffic must use https and be authenticated - preferably using certificates.

Currently we do not know which passwords the attacker uses when trying to brute-force the VNC capability. To gain insight into this we need to have a background task which performs dictionary based attacks on the challenge/response pairs. The wordlist should be made from commonly available wordlists, but also passwords collected by other Hive capabilities should be used to build a dynamic wordlist.

The following data is available for each VNC authentication attempt:

vnc authentication attempt from X.Y.Z.A. [challenge:"\xa1\xcd\xed\x89u\xcb!k.\xdd\x86%\xa8'\xea\xbc", response:'\xf4\xd0\xc7JM\x94\xda\xba\x15:M\xd0/\xb1\x92;'] (94b706e5-d527-46ee-a77a-63d5613c38ed)
vnc authentication attempt from X.Y.Z.A. [challenge:"\x0b\xe0\xb1\x0b\xbb\xf9\xb6J\xe6'\xe6@\xd6}s6", response:'D\xd2\xa8\xc7\xf2\x80bx\xf0~C\x00\xf2\xaa\xa4\xcc'] (0b501903-77b4-4c97-863c-020869acfb1b)

Instead of closing the socket from the Session, can we somehow pass a message (or call a close_session() method from the capability which created the session? Since the gsocket is actually being used from the capability, it makes sense to also close it from there, instead of closing it from the session

Standard lib already has a basic smtpd implementation, which needs to be extended with smtp auth.

• Login at Hive
• Send mail to someone. (who?) We might consider setting up fake mail acccounts

The telnet (and ssh) library used by Hive has a limited command capability, it would be interesting to replace this capability with the pseudo-shell used in Kippo.

I get this output after running nosetests. Not sure what the TypeErrors are, or what's causing them.

czardoz [~/git/beeswarm] -> nosetests                                                                                                                                                  ±[master]
......................
----------------------------------------------------------------------
Ran 22 tests in 3.300s

OK
Exception TypeError: 'must be type, not None' in <bound method OSFS.__del__ of <fs.osfs.OSFS object at 0x9df408c>> ignored
Exception TypeError: 'must be type, not None' in <bound method OSFS.__del__ of <fs.osfs.OSFS object at 0x9df072c>> ignored
czardoz [~/git/beeswarm] ->                                                                                                                                                            ±[master]

The web application should use SSL.

• The keys and certificate should be created on first startup.
• The certificate fingerprint should be stored in the clients configuration file.
• Clients (feeders/hives) should refuse to transmit data if they detect a invalid certificate.

Submission of data from clients should be password protected

• When creating a new client (hive/feeder) the corresponding credentials should also be created.
• When deleting a new client (hive/feeder) the corresponding credentials should also be deleted.
• Username would be client ID
• Password would be generated, preferably a UUID.

http://www.bro-ids.org/documentation/components/broccoli-python/README.html

Implement persistence layer on Beekeper.

Ref this TODO

The following error is recorded when attackers connects to the ssh capability:

2013-05-28 16:35:05,798 (beeswarm.hive.capabilities.ssh) Unexpected end of ssh session: not a valid RSA private key file. (2f93b046-b194-4bbb-8ae4-2300a0917080)
2013-05-28 16:35:05,836 (beeswarm.hive.consumer.consumer) Removed ssh connection from xxx.yyy.zzz.aaa. (2f93b046-b194-4bbb-8ae4-2300a0917080)

This issue is assigned to Google Summer Of Code student Aniket Panse (@czardoz).

The following work needs to be done:

• Implement POP3s

It is expected that at least 1 test cases are shipped with this work package.

The values must be added to the BeeSession class

Feeders should keep an inventory of public keys for all hive instances. If a public key mismatches the session would be classified as a MiTM attack.
This relates to #49.

Feeder should drop privileges after initialisation(port binding, config file writing, etc).