Hi everyone,

Currently we have an syslog to local logstash prototype available as an output. We however utilize ArcSight and would love to be able to receive either the 'logstash' syslog or a CEF formatted one. CEF syntax can be found here.

Creating such an prototype would support the ArcSight community out there!

Regards,
Forseti

The AAA API to change password and admin users should double check the current user password if the authentication comes from the session cookie.

Create a miner for the URLs http://www.malwaredomainlist.com/mdlcsv.php instead of using only the IP addresses.

Add support of specifying a jmespath query inside a parameter to the feed to filter indicators.

Miner for AWS IP ranges, available here:
https://ip-ranges.amazonaws.com/ip-ranges.json

Starting with support for operator OR.

Hello,
When node of type "Output" is enabled, in "Config" Tab its position is shown as "Processor". However, when the node is disabled, its position correctly matches the type - "Output"

Could be mined using DNS TXT queries. Ref: https://support.google.com/a/answer/60764?hl=en

At a first look the counters of dagpusher don't sum up. Double check.

Current implementation of TAXII miner uses MITRE python library and loads the full response in memory. This doesn't scale too well and should be changed into an event-based STIX parser.

A new miner should be added to support ingesting indicators from CIF (http://csirtgadgets.org).

Following exception is thrown when indicator contains non ASCII characters:

Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 358, in _callback
m(**params)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 494, in update
value=fltvalue
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 920, in filtered_update
self._add_indicator(now, indicator, value)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 842, in _add_indicator
indicator
UnicodeEncodeError: 'ascii' codec can't encode characters in position 77-80: ordinal not in range(128)

Currently JSON miner retrieves the whole JSON feed before parsing it. This is too memory intensive. We should try switching to an streaming JSON parser like this one: http://lloyd.github.io/yajl/

Prototypes should be added in a library in the local directory.

Version: 0.9.18

How to reproduce:

• create a syslog miner node
• create a rule in syslog miner with no fields
• check minemeld-engine.log

Exception is thrown:

2016-07-22T11:14:03 (5015)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/chassis.py", line 90, in configure
    config=ftconfig.get('config', {})
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 25, in factory
    config=config
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 371, in __init__
    super(SyslogMiner, self).__init__(name, chassis, config)
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 194, in __init__
    self.configure()
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 410, in configure
    self._load_side_config()
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 491, in _load_side_config
    cf = self._compile_rule(fname, f)
  File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 466, in _compile_rule
    result['fields'] = [fld for fld in fields if type(fld) == str]
TypeError: 'NoneType' object is not iterable

Currently Syslog processor doesn't export full details of matching session. The processor should be enhanced to export full details to an external logstash instance for history logging.

To use blueprints and better code separation.

Background
Users would like to share indicators with platforms supporting STIX/TAXII. This is currently not possible.

Requirements

• create new output node for TAXII Data Feed
• implement TAXII protocol in via Flask
• add support for basic authentication, where users are not MineMeld admins

Right now URLs are handled using a naive aggregator, where aggregation and whitelisting are based on string matching.
A URL specific aggregator should:

• support whitelist with wildcards/regexs
• handle indicators with wildcards
• handle different URLs formats (with or without protocol, ...)
• match on the host part of the URL should be case insensitive

Currently there is no way to modify field contents inside CSV Miners, like it is possible with the plain text Miner. That would be useful.

Like IPv4 aggregator but for IPv6 :-)

This to improve compatibility with TAXII clients not supporting IP ranges.

This to avoid issues when some product are capitalised. It happens from time to time.

See https://github.com/kennethreitz/requests/issues/2807

For persistent feeds.

Miner for Autofocus export lists and tags

Error messages in TAXII endpoints are too generic, they should be more specific to facilitate troubleshooting.

Nodes should be identified by an id independent from the name. This would permit node renaming and avoid name collision.

Currently syslogMatcher supports IPs and domains indicators, this to track support for matching URLs.

Currently DagPusher supports only non-persistent registered-ips. We should support also persistent registered-ips:

• it should be possible to specify the default value of persistent flag via prototype
• it should be possible to override the default value inside the device list

It should be possible to upload CSV files via the API and translate them into the format processed by the local Miners.

Exception handling domain indicators in taxii.DataFeed. Trace:

2016-09-12T17:09:19 (3329)amqp._callback ERROR: Exception in handling update on topic domainAggregator with params {u'source': u'domainAggregator', u'indicator': u'bestinghana.com', u'value': {u'confidence': 70, u'share_level': u'red', u'sources': [u'autofocusMMDemoEL'], u'autofocus_label': u'mmdemo', u'first_seen': 1473370831665L, u'type': u'domain', u'last_seen': 1473370831665L}}
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 344, in _callback
    m(**params)
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 122, in _counter
    f(self, *args, **kwargs)
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 453, in update
    value=fltvalue
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 122, in _counter
    f(self, *args, **kwargs)
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 838, in filtered_update
    self._add_indicator(now, id_, indicator, value)
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 777, in _add_indicator
    type_mapper['mapper'](oid, indicator, value)
  File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 608, in _stix_domain_observable
    type_="FQDN"
TypeError: __init__() got an unexpected keyword argument 'type_'

• branch factor should be more aggressive, most of the IPv4 indicators are unicast. Keep unicast in a dedicated table ? Would help in open-close interval
• move ST to native extension

There is a JSON file listing all the RedHat and Akami IPs that can be used as an additional check for RedHat updates. We should create a Miner to parse the JSON file.

Refs:
https://access.redhat.com/solutions/65300
https://access.redhat.com/articles/1525183
https://access.redhat.com/sites/default/files/attachments/cdn-ranges-2015-07-14.zip

Could be mined using DNS TXT queries. Ref: https://cloud.google.com/compute/docs/faq#networking

unverified, but should be checked.

• move code to native extension
• implement tri-gram index for faster searching of RegExs (see Russ Cox page)

Currently the historic period polled during the first request of the TAXII Miner is fixed to 1 hour. This should be made configurable.

TAXII Discovery and collection management services use the X-Server header to build the URL, this should be fixed.

Requested by msabena:
https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Microsoft-Azure-Datacenter-IP-Ranges/m-p/78681#U78681

Feed URL:
http://www.microsoft.com/EN-US/DOWNLOAD/confirmation.aspx?id=41653

$ find . -name '*.py' -exec grep -s recev x {} ;
./minemeld-core/minemeld/ft/base.py: LOG.error("update recevied from checkpointed source")
./minemeld-core/minemeld/ft/base.py: raise AssertionError("update recevied from checkpointed source")
./minemeld-core/minemeld/ft/base.py: LOG.error("withdraw recevied from checkpointed source")
./minemeld-core/minemeld/ft/base.py: raise AssertionError("withdraw recevied from checkpointed source")

An output & miner nodes should be added to push and retrieve indicators to MISP (http://www.misp-project.org/). Suggested by @sn8doc.

Currently the User-Agent header generated by the base poller Miners doesn't contain any reference to MineMeld. Adding a MineMeld specific string with version would help the feed sources to have a sense of the number of accesses performed via MineMeld.