paloaltonetworks / minemeld Goto Github PK

View Code? Open in Web Editor NEW

375.0 74.0 63.0 4 KB

Main MineMeld documentation repo

minemeld's Introduction

MineMeld

An extensible indicator processing framework

Getting Started

The best place for getting started with MineMeld Open Source project is the Wiki

Source Code

There are 3 main repos for the MineMeld source code:

the engine https://github.com/PaloAltoNetworks/minemeld-core
the Web UI https://github.com/PaloAltoNetworks/minemeld-webui
the prototypes library https://github.com/PaloAltoNetworks/minemeld-node-prototypes

Community

You can meet the MineMeld community:

in the MineMeld section of the Palo Alto Networks live community http://go.paloaltonetworks.com/minemeld
on the #minemeld channel of the pan-community Slack team (https://pan-community.net/)

You don't need to be a Palo Alto Networks customer to join the communities !

minemeld's People

Contributors

Stargazers

Watchers

minemeld's Issues

Provide 16.04 minemeld VM/cloud image

title says it all

Read Timeout for all o365 API Miners

For the last few days every one of my O365 miners have been erroring out with "HTTPSConnectionPool(host="endpoints.office365.com", port="443"): Read Timed out (read timeout=20)"

Not sure why this is happening but I can access the API from elsewhere within my network. Wondering if it's perhaps not cycling GUIDs on the API Calls?

Engine log is as below:
2020-01-04T09:54:38 (31903)basepoller._actor_loop INFO: wl_o365-api_any - command: 1578131678131 poll 2020-01-04T09:54:38 (31903)basepoller._polling_loop INFO: Polling wl_o365-api_any 2020-01-04T09:54:58 (31903)basepoller._poll ERROR: Exception in polling loop for wl_o365-api_any: HTTPSConnectionPool(host='endpoints.office.com', port=443): Read timed out. (read timeout=20) Traceback (most recent call last): File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 732, in _poll performed = self._polling_loop() File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 579, in _polling_loop iterator = self._build_iterator(now) File "/opt/minemeld/engine/core/minemeld/ft/o365.py", line 373, in _build_iterator latest_version = self._check_version() File "/opt/minemeld/engine/core/minemeld/ft/o365.py", line 288, in _check_version **rkwargs File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 524, in request resp = self.send(prep, **send_kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 637, in send r = adapter.send(request, **kwargs) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) ReadTimeout: HTTPSConnectionPool(host='endpoints.office.com', port=443): Read timed out. (read timeout=20)

Add 'required' field to o365-api miner

It would be very helpful to have the miners be able to select o365 lists based on whether or not MS says the URL or IP addresses are required.

"id": 53,
"serviceArea": "Common",
"serviceAreaDisplayName": "Microsoft 365 Common and Office Online",
"urls": [],
"tcpPorts": "443",
"expressRoute": false,
"category": "Default",
"required": true

feodotracker.domainblocklist broken

feodotracker.domainblocklist is pulling an HTML page, and not properly parsing the data. An example of the output is attached.

OUTPUT_DOMAIN_onboard_feodo.txt

IOC's not making it to Output node

Recently I have noticed that some IOC's are not being classified into an Output miner.
The logs show that the IOC is going through the motions and should eventually be added to the HighConfRed Output node but it continues to DROP_UPDATE. Please see the below log for a particularly dangerous IOC. Any help is appreciated.

Error retrieving minemeld metrics: Internal Server Error

Just cloned the master branch of this project on a CentOS 7 machine, followed the steps to install it on that OS, restored some data from a previous version we were handling and each time I refresh the Dashboard I get a few Python exceptions:

[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] RPC sent to @mbus:master:rpc for method status
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /status/minemeld?_=1557994429 HTTP/1.0" 200 8245 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:49 WEST] [19313] [INFO] redis connection pool: in use: 0 available: 1
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /extensions?_=1557994429 HTTP/1.0" 200 19 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 1 available: 5
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /aaa/users/current?_=1557994429 HTTP/1.0" 200 71 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
127.0.0.1 - - [16/May/2019:09:13:49 +0100] "GET /supervisor?_=1557994429 HTTP/1.0" 200 594 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 1 available: 5
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 2 available: 4
[2019-05-16 09:13:49 WEST] [19313] [DEBUG] redis session connection pool: in use: 3 available: 3
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /status/info?_=1557994429 HTTP/1.0" 200 65 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [ERROR] Exception on /metrics/minemeld/miners [GET]
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 125, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 135, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 127, in get_node_type_metrics
    v = _fetch_metric(cc, m, cf=cf, dt=dt, r=resolution, type_=type_)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 62, in _fetch_metric
    cc.flush(identifier='minemeld/%s/%s' % (metric, type_))
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 81, in flush
    cmd
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /metrics/minemeld/miners?_=1557994429&dt=86400&r=1800 HTTP/1.0" 500 291 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /config/running?_=1557994429 HTTP/1.0" 200 4732 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /prototype?_=1557994429 HTTP/1.0" 200 261753 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 1 available: 5
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 2 available: 4
[2019-05-16 09:13:50 WEST] [19313] [ERROR] Exception on /metrics/minemeld [GET]
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 125, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 135, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 170, in get_global_metrics
    v = _fetch_metric(cc, m, cf=cf, dt=dt, r=resolution, type_=type_)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 62, in _fetch_metric
    cc.flush(identifier='minemeld/%s/%s' % (metric, type_))
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 81, in flush
    cmd
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /metrics/minemeld?_=1557994429&dt=86400&r=1800 HTTP/1.0" 500 291 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [ERROR] Exception on /metrics/minemeld/outputs [GET]
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 125, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/aaa.py", line 135, in decorated_view
    return f(*args, **kwargs)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 127, in get_node_type_metrics
    v = _fetch_metric(cc, m, cf=cf, dt=dt, r=resolution, type_=type_)
  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 62, in _fetch_metric
    cc.flush(identifier='minemeld/%s/%s' % (metric, type_))
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 81, in flush
    cmd
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/opt/minemeld/engine/core/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
127.0.0.1 - - [16/May/2019:09:13:50 +0100] "GET /metrics/minemeld/outputs?_=1557994429&dt=86400&r=1800 HTTP/1.0" 500 291 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:13:50 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
127.0.0.1 - - [16/May/2019:09:13:51 +0100] "GET /prototype?_=1557994430 HTTP/1.0" 200 261753 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
[2019-05-16 09:14:00 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
[2019-05-16 09:14:00 WEST] [19313] [DEBUG] RPC sent to @mbus:master:rpc for method status
127.0.0.1 - - [16/May/2019:09:14:00 +0100] "GET /status/minemeld?_=1557994294 HTTP/1.0" 200 8245 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"
[2019-05-16 09:14:01 WEST] [19313] [DEBUG] redis session connection pool: in use: 0 available: 6
127.0.0.1 - - [16/May/2019:09:14:01 +0100] "GET /supervisor?_=1557994295 HTTP/1.0" 200 594 "https://minemeld.domain.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"```

Error at task npm install on RHEL

Hi all,

I'm installing minemeld-ansible on Red Hat Enterprise Linux server release 7.2 (maipo).
When i run this command:

sudo ansible-playbook -K -i 127.0.0.1, local.yml

I got this error:

TASK [minemeld : npm install] *********************************************************************************************************************************************************************************
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872" && echo ansible-tmp-1570795128.56-188516051705872="echo /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-2546_xCcAW/tmpW0MG6u TO /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872/ /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'PATH=/opt/minemeld/www/venv/lib/node_modules/.bin:/opt/minemeld/www/venv/bin:/opt/minemeld/www/webui/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin NODE_VIRTUAL_ENV=/opt/minemeld/www/venv npm_config_prefix=/opt/minemeld/www/venv NPM_CONFIG_PREFIX=/opt/minemeld/www/venv NODE_PATH=/opt/minemeld/www/venv/lib/node_modules /usr/bin/python /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1570795128.56-188516051705872/ > /dev/null 2>&1 && sleep 0'
fatal: [127.0.0.1]: FAILED! => {
"changed": true,
"cmd": [
"npm",
"install"
],
"delta": "0:01:04.638760",
"end": "2019-10-11 13:59:53.309982",
"invocation": {
"module_args": {
"_raw_params": "npm install",
"_uses_shell": false,
"argv": null,
"chdir": "/opt/minemeld/www/webui",
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"stdin_add_newline": true,
"strip_empty_ends": true,
"warn": true
}
},
"msg": "non-zero return code",
"rc": 228,
"start": "2019-10-11 13:58:48.671222",
"stderr": "npm WARN deprecated [email protected]: We don't recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/\nnpm WARN deprecated [email protected]: Typings is deprecated in favor of NPM @types -- see README for more information\nnpm WARN deprecated [email protected]: wrench.js is deprecated! You should check out fs-extra (https://github.com/jprichardson/node-fs-extra) for any operations you were using wrench for. Thanks for all the usage over the years.\nnpm WARN deprecated [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5\nnpm WARN deprecated [email protected]: Please use gulp-clean-css\nnpm WARN deprecated [email protected]: The Node Security Platform service is shutting down 9/30 - https://blog.npmjs.org/post/175511531085/the-node-security-platform-service-is-shutting\nnpm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).\nnpm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).\nnpm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).\nnpm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).\nnpm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).\nnpm WARN deprecated [email protected]: Use uuid module instead\nnpm WARN deprecated [email protected]: This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.\nnpm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130\nnpm WARN deprecated [email protected]: catastrophic backtracking in regexes could potentially lead to REDOS attack, upgrade to 2.17.2 as soon as possible\nnpm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).\nnpm WARN deprecated [email protected]: This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: express 2.x series is deprecated\nnpm WARN deprecated [email protected]: connect 1.x series is deprecated\nnpm WARN deprecated [email protected]: Old versions of Formidable are not compatible with the current Node.js; Upgrade to 1.2.0 or later\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: Please use the native JSON object instead of JSON 3\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js\nnpm WARN deprecated [email protected]: This module relies on Node.js's internals and will break at some point. Do not use it, and update to [email protected].\nnpm WARN deprecated [email protected]: Security vulnerability surrounding event-stream, do NOT use.\nnpm WARN deprecated [email protected]: Removed event-stream from gulp-header\nnpm WARN deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.\nnpm WARN deprecated [email protected]: CircularJSON is in maintenance only, flatted is its successor.\nnpm WARN deprecated [email protected]: Use uuid module instead\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm\nnpm WARN deprecated [email protected]: Use agent option with popsicle directly\nnpm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js\nnpm ERR! tar.unpack untar error /root/.npm/rxjs/5.5.12/package.tgz\nnpm ERR! tar.unpack untar error /root/.npm/bower/1.8.8/package.tgz\nnpm ERR! tar.unpack untar error /root/.npm/rxjs/5.5.12/package.tgz\nnpm ERR! tar.unpack untar error /root/.npm/bower/1.8.8/package.tgz\nnpm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):\nnpm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})\nnpm WARN [email protected] No repository field.\nnpm WARN [email protected] No license field.\nnpm ERR! Linux 3.10.0-327.36.3.el7.x86_64\nnpm ERR! argv "/opt/minemeld/www/venv/bin/node" "/opt/minemeld/www/venv/bin/npm" "install"\nnpm ERR! node v6.14.4\nnpm ERR! npm v3.10.10\nnpm ERR! code ENOSPC\nnpm ERR! errno -28\nnpm ERR! syscall write\n\nnpm ERR! nospc ENOSPC: no space left on device, write\nnpm ERR! nospc This is most likely not a problem with npm itself\nnpm ERR! nospc and is related to insufficient space on your system.\n\nnpm ERR! Please include the following file with any support request:\nnpm ERR! /opt/minemeld/www/webui/npm-debug.log",
"stderr_lines": [
"npm WARN deprecated [email protected]: We don't recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/",
"npm WARN deprecated [email protected]: Typings is deprecated in favor of NPM @types -- see README for more information",
"npm WARN deprecated [email protected]: wrench.js is deprecated! You should check out fs-extra (https://github.com/jprichardson/node-fs-extra) for any operations you were using wrench for. Thanks for all the usage over the years.",
"npm WARN deprecated [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5",
"npm WARN deprecated [email protected]: Please use gulp-clean-css",
"npm WARN deprecated [email protected]: The Node Security Platform service is shutting down 9/30 - https://blog.npmjs.org/post/175511531085/the-node-security-platform-service-is-shutting",
"npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
"npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
"npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
"npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
"npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
"npm WARN deprecated [email protected]: Use uuid module instead",
"npm WARN deprecated [email protected]: This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.",
"npm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130",
"npm WARN deprecated [email protected]: catastrophic backtracking in regexes could potentially lead to REDOS attack, upgrade to 2.17.2 as soon as possible",
"npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
"npm WARN deprecated [email protected]: This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.",
"npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue",
"npm WARN deprecated [email protected]: express 2.x series is deprecated",
"npm WARN deprecated [email protected]: connect 1.x series is deprecated",
"npm WARN deprecated [email protected]: Old versions of Formidable are not compatible with the current Node.js; Upgrade to 1.2.0 or later",
"npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue",
"npm WARN deprecated [email protected]: Please use the native JSON object instead of JSON 3",
"npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue",
"npm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js",
"npm WARN deprecated [email protected]: This module relies on Node.js's internals and will break at some point. Do not use it, and update to [email protected].",
"npm WARN deprecated [email protected]: Security vulnerability surrounding event-stream, do NOT use.",
"npm WARN deprecated [email protected]: Removed event-stream from gulp-header",
"npm WARN deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.",
"npm WARN deprecated [email protected]: CircularJSON is in maintenance only, flatted is its successor.",
"npm WARN deprecated [email protected]: Use uuid module instead",
"npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue",
"npm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm",
"npm WARN deprecated [email protected]: Use agent option with popsicle directly",
"npm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js",
"npm ERR! tar.unpack untar error /root/.npm/rxjs/5.5.12/package.tgz",
"npm ERR! tar.unpack untar error /root/.npm/bower/1.8.8/package.tgz",
"npm ERR! tar.unpack untar error /root/.npm/rxjs/5.5.12/package.tgz",
"npm ERR! tar.unpack untar error /root/.npm/bower/1.8.8/package.tgz",
"npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):",
"npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})",
"npm WARN [email protected] No repository field.",
"npm WARN [email protected] No license field.",
"npm ERR! Linux 3.10.0-327.36.3.el7.x86_64",
"npm ERR! argv "/opt/minemeld/www/venv/bin/node" "/opt/minemeld/www/venv/bin/npm" "install"",
"npm ERR! node v6.14.4",
"npm ERR! npm v3.10.10",
"npm ERR! code ENOSPC",
"npm ERR! errno -28",
"npm ERR! syscall write",
"",
"npm ERR! nospc ENOSPC: no space left on device, write",
"npm ERR! nospc This is most likely not a problem with npm itself",
"npm ERR! nospc and is related to insufficient space on your system.",
"",
"npm ERR! Please include the following file with any support request:",
"npm ERR! /opt/minemeld/www/webui/npm-debug.log"
],

Someone could help me ?
thanks

Fatal error

HI ,

After activated this extension , Minemeld engine cannot start. #

Minemeld Taxii Server

Hello guys.
I am a newbie user of Minemeld.
My scope is to create a a Taxii server like output node in Minemeld.
I created it from prototype stdlib.taxiiDataFeed.
This node works correctly. I see new indicator in its from a custom chain (simple_miners-->default_aggegator-->My taxii node)
Now from an external Taxii client I retrieve the service discovery and the collection but I do not know the server inbox. This permits to upload the Stix data in external server Saxii.
Can I do with Minemeld? How I add this service discovery?

Many thanks for your help

MineMeld withdrawing indicators when used by multiple users

Let’s say you have two users connected to MineMeld.

The first user opens one miner to add an indicator to it and left the browser opened. At the same time, a second user opens the same miner and left the browser opened. So, they are seeing the same indicators.
Now, let’s say that the first user adds an indicator to the miner, but the second user does not refresh the page and adds an indicator just after the first user.

In this situation, MineMeld will withdraw the first indicator added by the first user because the browser of the last user (second user) didn’t list that first indicator. Similar behavior if you delete an indicator (minemeld will add the indicator again).

I am now adding 200+ indicators to the miner again because of this issue. Looks like Minemeld use the data from the browser to compare the data on the server. Everything that is new is dropped (withdraw) and everything that was deleted is added again based on the data from you browser.

Cloud notification service is not available

Hi,

I installed minemeld on rhel 7 via the playbook. Installation was completed but minemeld has issues when starting.

I've set a proxy in environment variables, in /etc/default/minemeld and in minemeld.service. Curl returns a json as expected.

Any idea what might be causing this?

Thanks

First_Seen Error

I am getting a weird error when using a miner to pull in Talos and Firehol threat feeds. It worked fine in my test environment, but in my actual deployment after the initial retrieval anytime the node runs it gets an error. When I hover over this all it states is 'first_seen.'

This traffic goes through the Palo Alto's and I have turned off URL filtering/SSL Decryption from the Minemeld server but this didn't help. The traffic is not going through a Proxy.

Need help creating a miner

I'm trying to set up connectivity to an MISP instance. I've tried using the minemeld MISP connector, but due to an open ticket not being resolved and not being able to query things from a relative date, I'm trying to come up with alternative ways to pull data from an MISP instance to minemeld.

Is there a way I can modify one of the existing miners or nodes to where it passes the header:
Authorization: [api key here]
in its HTTP/HTTPS request?

Minemeld Starting sudden issue

I think this is probably wasted breath as it seems minemeld isn't being supported for free anymore.

However, recently my miners starting acting up. So I was going to restart the minemeld service via CLI. That didn't go to well as I received this error.

Someone@MineMeld:~$ sudo service minemeld start

Starting: minemeld /opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
return pkg_resources.EntryPoint.parse("x="+spec).load(False)

Might be something with a recent update, I am running Ubuntu LTS 16.0.4 as the build recommends. Any help would be appreciated, as I have these dynamic lists deployed to 80+ firewalls.

not able to install ssl certificate from local CA on minemeld

I am trying to install SSL certificate on Minemeld but each time I tried it fails. I am using our organization CA. Could you please guide me. When I install certificate and restart services URL stop working.

Accounting feature - logging user activity

We would like to use a stdlib.listIPv4Generic miner, editing this static list via web gui.
It would be nice to have log on user activity (add/remove indicators to/from a node, create/delete nodes, creating users, purge logs, etc).
For us, it would be pretty important to be able to track user activity regarding indicators (we need to be able to find out who and when blocked an IP address).

Python3 support?

Python 2.7 is End of life and many distributions will be dropping it (if they haven't already)

Is there a plan to update minemeld to allow Python3? As it stands on Ubuntu 18.04 fully patched, minemeld-ansible doesn't work because of issues with ujson (which itself hasn't been updated in 3 years)

Anything newer than 18.04 will not have Python 2.7 by default (checked on 19.10 and 20.04 pre-release.)

Invalid Element Name Error

Hi there,

I use Minemeld to get FS ISAC data and it was working perfectly for the last few months. However I checked today and noticed that there was an error in polling.

The error I get (Screenshot attached) is StartTag: invalid element name, line 33896, column 2 (line 33896)

A quick search on google states this error is usually related .yml files but is there a way to pinpoint what file exactly and the reason for the change since no one has touched any files.

Documentation: Wrong link location

In this page https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner, the MineMeld Engine Architecture link is wrong (actually is https://github.com/PaloAltoNetworks/minemeld-core/docs/architecture.rst). The correct link seems to be https://github.com/PaloAltoNetworks/minemeld-core/blob/master/docs/architecture.rst

Update minemeld repo to support Ubuntu 16.04

Following the instructions here: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-14-04/ta-p/98454

...On Ubuntu 16.04 results in sadness:

root@minemeld:~# apt-get update && apt-get install -y minemeld rsyslog-minemeld rsyslog-mmnormalize
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://minemeld-updates.panw.io/ubuntu trusty-minemeld InRelease [2,428 B]
Get:6 http://minemeld-updates.panw.io/ubuntu trusty-minemeld/main amd64 Packages [3,278 B]
Fetched 5,706 B in 0s (9,575 B/s)
Reading package lists... Done
W: http://minemeld-updates.panw.io/ubuntu/dists/trusty-minemeld/InRelease: Signature by key 234D91524C5B20C49AD9C9B978FDEF917B630999 uses weak digest algorithm (SHA1)
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
 minemeld : Depends: librabbitmq1 but it is not installable
            Depends: libsnappy1 but it is not installable
            Depends: libleveldb1 but it is not installable
 rsyslog-mmnormalize : Depends: rsyslog (= 8.17.0-0adiscon2trusty1) but 8.16.0-1ubuntu3 is to be installed
E: Unable to correct problems, you have held broken packages.
root@minemeld:~# apt-get install librabbitmq1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package librabbitmq1 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Could you supply a 16.04 repo soon, please?

Report as Output in MineMeld [Question]

Hello,

First of all, I have to say that I am a newbie to the field and to MineMeld. Thus it is possible that my understanding of the subject is flawed or that my terminology is crooked.

I want to know if there is a prototype in MineMeld for an output node with reporting capabilities. That is, is it possible to generate a document/report in MineMeld for human use, for humans to consult and investigate?

Thanks in advance,
Galip

not applying 'whitelist' to CIF requests

When performing a CIF query, minemeld appears to just be pulling everything with an otype and confidence.
https://github.com/PaloAltoNetworks/minemeld-core/blob/master/minemeld/ft/cif.py#L104

Minemeld should utilize the --feed option in the CIFSDK which incorporates a whitelist on the CIF server before sending the results.
https://github.com/csirtgadgets/cif-sdk-py/blob/master/cifsdk/client.py#L455

We know in CIFv2 it's not perfect, but right now we've heard complaints from a handful of customers about a lot of items that should be filtered as whitelisted are making it through and being placed in block lists on their firewalls.

We've done some abstracting and improved the whitelisting in CIFv3. Another suggestion would be to try to replicate what we did in client.py for cif.py https://github.com/csirtgadgets/bearded-avenger-sdk-py/blob/master/cifsdk/client/client.py#L222 and build a cifv3.py client with #222 for when we start rolling out v3.

FATAL Exited too quickly - minemeld-engine & minemeld-web

Hello,
I'm trying to install a new instance of Minemeld on Ubuntu 14 using instruction https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-14-04/ta-p/98454

and I couldn't log in. I found that the minemeld-engine and minemeld-web couldn't start. Below is some log. Any suggestion? Thanks!

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
minemeld-engine FATAL Exited too quickly (process log may have details)
minemeld-supervisord-listener RUNNING pid 8868, uptime 0:34:30
minemeld-traced RUNNING pid 8870, uptime 0:34:30
minemeld-web FATAL Exited too quickly (process log may have details)

Below is the minemeld-engine.log log:

AttributeError: 'module' object has no attribute 'get_installed_distributions'
2018-05-08T15:31:58 (8893)launcher.main INFO: Starting mm-run.py version 0.9.44
2018-05-08T15:31:58 (8893)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-run", line 11, in
sys.exit(main())
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 218, in main
config = minemeld.run.config.load_config(args.config)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 567, in load_config
return _load_config_from_dir(config_path)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 382, in _load_config_from_dir
ccvalid, cconfig = _load_and_validate_config_from_file(ccpath)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 281, in _load_and_validate_config_from_file
valid = resolve_prototypes(config)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 477, in resolve_prototypes
prototypes_entrypoints = minemeld.loader.map(minemeld.loader.MM_PROTOTYPES_ENTRYPOINT)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 105, in map
_initialize_entry_point_group(entrypoint_group)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 55, in _initialize_entry_point_group
installed = _installed_versions()
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 29, in _installed_versions
installed_dists = pip.get_installed_distributions()
AttributeError: 'module' object has no attribute 'get_installed_distributions'
2018-05-08T15:32:01 (8918)launcher.main INFO: Starting mm-run.py version 0.9.44
2018-05-08T15:32:01 (8918)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-run", line 11, in
sys.exit(main())
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 218, in main
config = minemeld.run.config.load_config(args.config)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 567, in load_config
return _load_config_from_dir(config_path)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 382, in _load_config_from_dir
ccvalid, cconfig = _load_and_validate_config_from_file(ccpath)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 281, in _load_and_validate_config_from_file
valid = resolve_prototypes(config)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 477, in resolve_prototypes
prototypes_entrypoints = minemeld.loader.map(minemeld.loader.MM_PROTOTYPES_ENTRYPOINT)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 105, in map
_initialize_entry_point_group(entrypoint_group)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 55, in _initialize_entry_point_group
installed = _installed_versions()
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 29, in _installed_versions
installed_dists = pip.get_installed_distributions()
AttributeError: 'module' object has no attribute 'get_installed_distributions'
2018-05-08T15:32:05 (8947)launcher.main INFO: Starting mm-run.py version 0.9.44
2018-05-08T15:32:05 (8947)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-run", line 11, in
sys.exit(main())
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 218, in main
config = minemeld.run.config.load_config(args.config)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 567, in load_config
return _load_config_from_dir(config_path)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 382, in _load_config_from_dir
ccvalid, cconfig = _load_and_validate_config_from_file(ccpath)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 281, in _load_and_validate_config_from_file
valid = resolve_prototypes(config)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/run/config.py", line 477, in resolve_prototypes
prototypes_entrypoints = minemeld.loader.map(minemeld.loader.MM_PROTOTYPES_ENTRYPOINT)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 105, in map
_initialize_entry_point_group(entrypoint_group)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 55, in _initialize_entry_point_group
installed = _installed_versions()
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/loader.py", line 29, in _installed_versions
installed_dists = pip.get_installed_distributions()
AttributeError: 'module' object has no attribute 'get_installed_distributions'

No Module Named datetime

Hi all,

I have updated ubuntu server from version 14.04.05 to 16.04.06.

When i run the follow command :

$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
i got the follow error:

noModulNamedDateTime.png

Someone could help me?

Thanks

tab expansion generates invalid JSON

I'm processing a STIX feed with taxiing.node.Miner linked to a minemeld.redis.RedisSet.
Stix feed contain string with tab characters in it. Tab char is escaped with "\t".
When get values from the output node via web service, the escaped tab char is expanded generating an invalid json message.

STIX message
{"stix_title":"FQDN: mydomain.com\t" }

output JSON message

{"indicator":"mydomain.com ","value":{"stix_title":"FQDN: mydomain.com\t" <SNIP> }

in the indicator string "mydomain.com " the tab is expanded generating an invalid JSON message

RHEL - /bin/sh ./configure Permission Denied

Hi all,

I'm installing minemeld-ansible on Redhat 7.
When i run this command:

sudo ansible-playbook -K -i 127.0.0.1, local.yml
i got this error:
`PLAY [minemeld playbook] **************************************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [infrastructure : debug] *********************************************************************************************************************************************************************************
ok: [127.0.0.1] => {
"msg": "Loading vars for RedHat 7.2"
}

TASK [infrastructure : include_vars] **************************************************************************************************************************************************************************
ok: [127.0.0.1] => (item=/root/minemeld-ansible/roles/infrastructure/vars/RedHat-7.yml)

TASK [infrastructure : create minemeld group] *****************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [infrastructure : include task based on distribution] ****************************************************************************************************************************************************
included: /root/minemeld-ansible/roles/infrastructure/tasks/RedHat-7.yml for 127.0.0.1

TASK [infrastructure : enable epel] ***************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [infrastructure : enable RedHat RHEL 7 Optional repo (AWS)] **********************************************************************************************************************************************
changed: [127.0.0.1]

TASK [infrastructure : enable RedHat RHEL 7 Optional repo] ****************************************************************************************************************************************************
changed: [127.0.0.1]

TASK [infrastructure : Check for custom python] ***************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [infrastructure : Extract Python 2.7.15] *****************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [infrastructure : Configure Python 2.7.15] ***************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "./configure --prefix=/usr/local", "delta": "0:00:00.004001", "end": "2019-10-07 19:42:36.961017", "msg": "non-zero return code", "rc": 126, "start": "2019-10-07 19:42:36.957016", "stderr": "/bin/sh: ./configure: Permission denied", "stderr_lines": ["/bin/sh: ./configure: Permission denied"], "stdout": "", "stdout_lines": []}

PLAY RECAP ****************************************************************************************************************************************************************************************************
127.0.0.1 : ok=10 changed=2 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0`

I have root privileges and i have disabled SELinux, so why does it happen?
Thanks.

Problem installing MineMeld on Ubuntu 16.04 using Ansible

Hi,

I encountered the below error at npm install stage. Below is the error.

TASK [minemeld : npm install] ***************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["npm", "install"], "delta": "0:01:53.179532", "end": "2017-09-28 16:20:48.060202", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-09-28 16:18:54.880670", "stderr": "npm WARN package.json [email protected] No repository field.\nnpm WARN deprecated [email protected]: Please use gulp-clean-css\nnpm WARN deprecated [email protected]: ..psst! While Bower is maintained, we recommend Yarn and Webpack for new front-end projects! Yarn's advantage is security and reliability, and Webpack's is support for both CommonJS and AMD projects. Currently there's no migration path, but please help to create it: https://github.com/bower/bower/issues/2467\nnpm WARN deprecated [email protected]: wrench.js is deprecated! You should check out fs-extra (https://github.com/jprichardson/node-fs-extra) for any operations you were using wrench for. Thanks for all the usage over the years.\nnpm WARN deprecated [email protected]: Typings is deprecated in favor of NPM @types -- see README for more information\nnpm WARN peerDependencies The peer dependency jasmine-core@* included from karma-jasmine will no\nnpm WARN peerDependencies longer be automatically installed to fulfill the peerDependency \nnpm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.\nnpm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.\nnpm WARN deprecated [email protected]: Use uuid module instead\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.\nnpm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">= 4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN deprecated [email protected]: Use uuid module instead\nnpm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN optional dep failed, continuing [email protected]\nnpm WARN optional dep failed, continuing [email protected]\nnpm WARN optional dep failed, continuing [email protected]\nnpm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">= 4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4.5.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\nnpm WARN deprecated [email protected]: express 2.x series is deprecated\nnpm WARN deprecated [email protected]: connect 1.x series is deprecated\nnpm WARN engine [email protected]: wanted: {"node":">=4.0"} (current: {"node":"0.12.2","npm":"2.7.4"})\n/opt/minemeld/www/webui/node_modules/gulp-sass/node_modules/node-sass/node_modules/request/node_modules/hawk/node_modules/boom/lib/index.js:5\nconst Hoek = require('hoek');\n^^^^^\nSyntaxError: Use of const in strict mode.\n at exports.runInThisContext (vm.js:73:16)\n at Module._compile (module.js:443:25)\n at Object.Module._extensions..js (module.js:478:10)\n at Module.load (module.js:355:32)\n at Function.Module._load (module.js:310:12)\n at Module.require (module.js:365:17)\n at require (module.js:384:17)\n at Object. (/opt/minemeld/www/webui/node_modules/gulp-sass/node_modules/node-sass/node_modules/request/node_modules/hawk/lib/index.js:5:33)\n at Module._compile (module.js:460:26)\n at Object.Module._extensions..js (module.js:478:10)\ngyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/0.12.2"\ngyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/utf-8-validate/.node-gyp"\ngyp ERR! build error \ngyp ERR! stack Error: not found: make\ngyp ERR! stack at F (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:40:28)\ngyp ERR! stack at E (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:43:29)\ngyp ERR! stack at /opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:54:16\ngyp ERR! stack at FSReqWrap.oncomplete (fs.js:95:15)\ngyp ERR! System Linux 4.4.0-62-generic\ngyp ERR! command "node" "/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"\ngyp ERR! cwd /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/utf-8-validate\ngyp ERR! node -v v0.12.2\ngyp ERR! node-gyp -v v1.0.3\ngyp ERR! not ok \ngyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/0.12.2"\ngyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/bufferutil/.node-gyp"\ngyp ERR! build error \ngyp ERR! stack Error: not found: make\ngyp ERR! stack at F (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:40:28)\ngyp ERR! stack at E (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:43:29)\ngyp ERR! stack at /opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:54:16\ngyp ERR! stack at FSReqWrap.oncomplete (fs.js:95:15)\ngyp ERR! System Linux 4.4.0-62-generic\ngyp ERR! command "node" "/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"\ngyp ERR! cwd /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/bufferutil\ngyp ERR! node -v v0.12.2\ngyp ERR! node-gyp -v v1.0.3\ngyp ERR! not ok \nnpm WARN optional dep failed, continuing [email protected]\nnpm WARN optional dep failed, continuing [email protected]\nnpm ERR! Linux 4.4.0-62-generic\nnpm ERR! argv "/opt/minemeld/www/venv/bin/node" "/opt/minemeld/www/venv/bin/npm" "install"\nnpm ERR! node v0.12.2\nnpm ERR! npm v2.7.4\nnpm ERR! code ELIFECYCLE\n\nnpm ERR! [email protected] install: node scripts/install.js\nnpm ERR! Exit status 1\nnpm ERR! \nnpm ERR! Failed at the [email protected] install script 'node scripts/install.js'.\nnpm ERR! This is most likely a problem with the node-sass package,\nnpm ERR! not with npm itself.\nnpm ERR! Tell the author that this fails on your system:\nnpm ERR! node scripts/install.js\nnpm ERR! You can get their info via:\nnpm ERR! npm owner ls node-sass\nnpm ERR! There is likely additional logging output above.\n\nnpm ERR! Please include the following file with any support request:\nnpm ERR! /opt/minemeld/www/webui/npm-debug.log", "stderr_lines": ["npm WARN package.json [email protected] No repository field.", "npm WARN deprecated [email protected]: Please use gulp-clean-css", "npm WARN deprecated [email protected]: ..psst! While Bower is maintained, we recommend Yarn and Webpack for new front-end projects! Yarn's advantage is security and reliability, and Webpack's is support for both CommonJS and AMD projects. Currently there's no migration path, but please help to create it: https://github.com/bower/bower/issues/2467", "npm WARN deprecated [email protected]: wrench.js is deprecated! You should check out fs-extra (https://github.com/jprichardson/node-fs-extra) for any operations you were using wrench for. Thanks for all the usage over the years.", "npm WARN deprecated [email protected]: Typings is deprecated in favor of NPM @types -- see README for more information", "npm WARN peerDependencies The peer dependency jasmine-core@* included from karma-jasmine will no", "npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency ", "npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.", "npm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm", "npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue", "npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.", "npm WARN deprecated [email protected]: Use uuid module instead", "npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue", "npm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130", "npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue", "npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.", "npm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">= 4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue", "npm WARN deprecated [email protected]: Use uuid module instead", "npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN optional dep failed, continuing [email protected]", "npm WARN optional dep failed, continuing [email protected]", "npm WARN optional dep failed, continuing [email protected]", "npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">= 4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4.5.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue", "npm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN engine [email protected]: wanted: {"node":">=4.0.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "npm WARN deprecated [email protected]: express 2.x series is deprecated", "npm WARN deprecated [email protected]: connect 1.x series is deprecated", "npm WARN engine [email protected]: wanted: {"node":">=4.0"} (current: {"node":"0.12.2","npm":"2.7.4"})", "/opt/minemeld/www/webui/node_modules/gulp-sass/node_modules/node-sass/node_modules/request/node_modules/hawk/node_modules/boom/lib/index.js:5", "const Hoek = require('hoek');", "^^^^^", "SyntaxError: Use of const in strict mode.", " at exports.runInThisContext (vm.js:73:16)", " at Module._compile (module.js:443:25)", " at Object.Module._extensions..js (module.js:478:10)", " at Module.load (module.js:355:32)", " at Function.Module._load (module.js:310:12)", " at Module.require (module.js:365:17)", " at require (module.js:384:17)", " at Object. (/opt/minemeld/www/webui/node_modules/gulp-sass/node_modules/node-sass/node_modules/request/node_modules/hawk/lib/index.js:5:33)", " at Module._compile (module.js:460:26)", " at Object.Module._extensions..js (module.js:478:10)", "gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/0.12.2"", "gyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/utf-8-validate/.node-gyp"", "gyp ERR! build error ", "gyp ERR! stack Error: not found: make", "gyp ERR! stack at F (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:40:28)", "gyp ERR! stack at E (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:43:29)", "gyp ERR! stack at /opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:54:16", "gyp ERR! stack at FSReqWrap.oncomplete (fs.js:95:15)", "gyp ERR! System Linux 4.4.0-62-generic", "gyp ERR! command "node" "/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"", "gyp ERR! cwd /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/utf-8-validate", "gyp ERR! node -v v0.12.2", "gyp ERR! node-gyp -v v1.0.3", "gyp ERR! not ok ", "gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/0.12.2"", "gyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/bufferutil/.node-gyp"", "gyp ERR! build error ", "gyp ERR! stack Error: not found: make", "gyp ERR! stack at F (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:40:28)", "gyp ERR! stack at E (/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:43:29)", "gyp ERR! stack at /opt/minemeld/www/venv/lib/node_modules/npm/node_modules/which/which.js:54:16", "gyp ERR! stack at FSReqWrap.oncomplete (fs.js:95:15)", "gyp ERR! System Linux 4.4.0-62-generic", "gyp ERR! command "node" "/opt/minemeld/www/venv/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"", "gyp ERR! cwd /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/bufferutil", "gyp ERR! node -v v0.12.2", "gyp ERR! node-gyp -v v1.0.3", "gyp ERR! not ok ", "npm WARN optional dep failed, continuing [email protected]", "npm WARN optional dep failed, continuing [email protected]", "npm ERR! Linux 4.4.0-62-generic", "npm ERR! argv "/opt/minemeld/www/venv/bin/node" "/opt/minemeld/www/venv/bin/npm" "install"", "npm ERR! node v0.12.2", "npm ERR! npm v2.7.4", "npm ERR! code ELIFECYCLE", "", "npm ERR! [email protected] install: node scripts/install.js", "npm ERR! Exit status 1", "npm ERR! ", "npm ERR! Failed at the [email protected] install script 'node scripts/install.js'.", "npm ERR! This is most likely a problem with the node-sass package,", "npm ERR! not with npm itself.", "npm ERR! Tell the author that this fails on your system:", "npm ERR! node scripts/install.js", "npm ERR! You can get their info via:", "npm ERR! npm owner ls node-sass", "npm ERR! There is likely additional logging output above.", "", "npm ERR! Please include the following file with any support request:", "npm ERR! /opt/minemeld/www/webui/npm-debug.log"], "stdout": "\n> [email protected] install /opt/minemeld/www/webui/node_modules/awesome-typescript-loader/node_modules/deasync\n> node ./build.js\n\nlinux-x64-node-0.12 exists; testing\nBinary is fine; exiting\n\n> [email protected] install /opt/minemeld/www/webui/node_modules/phantomjs\n> node install.js\n\nPhantomJS not found on PATH\nDownloading https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2\nSaving to /opt/minemeld/www/webui/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2\nReceiving...\n\nReceived 12854K total.\nExtracting tar contents (via spawned process)\nRemoving /opt/minemeld/www/webui/node_modules/phantomjs/lib/phantom\nCopying extracted folder /opt/minemeld/www/webui/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2-extract-1506586829476/phantomjs-1.9.8-linux-x86_64 -> /opt/minemeld/www/webui/node_modules/phantomjs/lib/phantom\nWriting location.js file\nDone. Phantomjs binary available at /opt/minemeld/www/webui/node_modules/phantomjs/lib/phantom/bin/phantomjs\n\n> [email protected] install /opt/minemeld/www/webui/node_modules/gulp-sass/node_modules/node-sass\n> node scripts/install.js\n\n\n> [email protected] install /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/utf-8-validate\n> node-gyp rebuild\n\n\n> [email protected] install /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/bufferutil\n> node-gyp rebuild", "stdout_lines": ["", "> [email protected] install /opt/minemeld/www/webui/node_modules/awesome-typescript-loader/node_modules/deasync", "> node ./build.js", "", "linux-x64-node-0.12 exists; testing", "Binary is fine; exiting", "", "> [email protected] install /opt/minemeld/www/webui/node_modules/phantomjs", "> node install.js", "", "PhantomJS not found on PATH", "Downloading https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2", "Saving to /opt/minemeld/www/webui/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2", "Receiving...", "", "Received 12854K total.", "Extracting tar contents (via spawned process)", "Removing /opt/minemeld/www/webui/node_modules/phantomjs/lib/phantom", "Copying extracted folder /opt/minemeld/www/webui/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2-extract-1506586829476/phantomjs-1.9.8-linux-x86_64 -> /opt/minemeld/www/webui/node_modules/phantomjs/lib/phantom", "Writing location.js file", "Done. Phantomjs binary available at /opt/minemeld/www/webui/node_modules/phantomjs/lib/phantom/bin/phantomjs", "", "> [email protected] install /opt/minemeld/www/webui/node_modules/gulp-sass/node_modules/node-sass", "> node scripts/install.js", "", "", "> [email protected] install /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/utf-8-validate", "> node-gyp rebuild", "", "", "> [email protected] install /opt/minemeld/www/webui/node_modules/gulp-protractor/node_modules/protractor/node_modules/selenium-webdriver/node_modules/ws/node_modules/bufferutil", "> node-gyp rebuild"]}

Document how to install from source

For both RPM and DEB-based distros, if you would be so kind.

NPM problem with the installation in CentOS7 from the playbook

Im using CentOS7 and i am trying to install the master version from the source code.
The task that install NPM says:

TASK [minemeld : npm install] ******************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["npm", "install"], "delta": "0:16:23.745860", "end": "2018-06-01 10:58:53.285418", "msg": "non-zero return code", "rc": 1, "start": "2018-06-01 10:42:29.539558", "stderr": "npm WARN package.json [email protected] No repository field.\nnpm WARN package.json [email protected] No license field.\nnpm WARN deprecated [email protected]: Please use gulp-clean-css\nnpm WARN deprecated [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5\nnpm WARN deprecated [email protected]: ...psst! Your project can stop working at any moment because its dependencies can change. Prevent this by migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from-bower/\nnpm WARN deprecated [email protected]: Typings is deprecated in favor of NPM @types -- see README for more information\nnpm WARN deprecated [email protected]: wrench.js is deprecated! You should check out fs-extra (https://github.com/jprichardson/node-fs-extra) for any operations you were using wrench for. Thanks for all the usage over the years.\nnpm WARN peerDependencies The peer dependency jasmine-core@* included from karma-jasmine will no\nnpm WARN peerDependencies longer be automatically installed to fulfill the peerDependency \nnpm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.\nnpm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm\nnpm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js\nnpm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js\nnpm ERR! fetch failed https://registry.npmjs.org/node-sass/-/node-sass-3.13.1.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.97.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/osenv/-/osenv-0.1.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/read-installed/-/read-installed-3.1.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.95.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/npmconf/-/npmconf-1.1.5.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.95.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/string.prototype.endswith/-/string.prototype.endswith-0.2.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.97.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/vinyl-fs/-/vinyl-fs-0.3.14.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443\nnpm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue\nnpm ERR! fetch failed https://registry.npmjs.org/bufferstreams/-/bufferstreams-1.0.1.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/glob/-/glob-5.0.15.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.96.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/glob/-/glob-7.1.2.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/vinyl/-/vinyl-2.1.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/readable-stream/-/readable-stream-1.0.34.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.95.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/vinyl/-/vinyl-0.4.6.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.96.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/graceful-fs/-/graceful-fs-3.0.11.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/strip-bom/-/strip-bom-1.0.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.97.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/globby/-/globby-2.1.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443\nnpm ERR! fetch failed https://registry.npmjs.org/globby/-/globby-2.1.0.tgz\nnpm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443\nnpm ERR! Linux 3.10.0-862.3.2.el7.x86_64\nnpm ERR! argv "/opt/minemeld/www/venv/bin/node" "/opt/minemeld/www/venv/bin/npm" "install"\nnpm ERR! node v4.2.6\nnpm ERR! npm v2.14.12\nnpm ERR! code ETIMEDOUT\nnpm ERR! errno ETIMEDOUT\nnpm ERR! syscall connect\n\nnpm ERR! network connect ETIMEDOUT 104.18.94.96:443\nnpm ERR! network This is most likely not a problem with npm itself\nnpm ERR! network and is related to network connectivity.\nnpm ERR! network In most cases you are behind a proxy or have bad network settings.\nnpm ERR! network \nnpm ERR! network If you are behind a proxy, please make sure that the\nnpm ERR! network 'proxy' config is set properly. See: 'npm help config'\n\nnpm ERR! Please include the following file with any support request:\nnpm ERR! /opt/minemeld/www/webui/npm-debug.log", "stderr_lines": ["npm WARN package.json [email protected] No repository field.", "npm WARN package.json [email protected] No license field.", "npm WARN deprecated [email protected]: Please use gulp-clean-css", "npm WARN deprecated [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5", "npm WARN deprecated [email protected]: ...psst! Your project can stop working at any moment because its dependencies can change. Prevent this by migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from-bower/", "npm WARN deprecated [email protected]: Typings is deprecated in favor of NPM @types -- see README for more information", "npm WARN deprecated [email protected]: wrench.js is deprecated! You should check out fs-extra (https://github.com/jprichardson/node-fs-extra) for any operations you were using wrench for. Thanks for all the usage over the years.", "npm WARN peerDependencies The peer dependency jasmine-core@* included from karma-jasmine will no", "npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency ", "npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.", "npm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm", "npm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js", "npm WARN deprecated [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js", "npm ERR! fetch failed https://registry.npmjs.org/node-sass/-/node-sass-3.13.1.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.97.96:443", "npm ERR! fetch failed https://registry.npmjs.org/osenv/-/osenv-0.1.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443", "npm ERR! fetch failed https://registry.npmjs.org/read-installed/-/read-installed-3.1.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.95.96:443", "npm ERR! fetch failed https://registry.npmjs.org/npmconf/-/npmconf-1.1.5.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.95.96:443", "npm ERR! fetch failed https://registry.npmjs.org/string.prototype.endswith/-/string.prototype.endswith-0.2.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.97.96:443", "npm ERR! fetch failed https://registry.npmjs.org/vinyl-fs/-/vinyl-fs-0.3.14.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443", "npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue", "npm ERR! fetch failed https://registry.npmjs.org/bufferstreams/-/bufferstreams-1.0.1.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443", "npm ERR! fetch failed https://registry.npmjs.org/glob/-/glob-5.0.15.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.96.96:443", "npm ERR! fetch failed https://registry.npmjs.org/glob/-/glob-7.1.2.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443", "npm ERR! fetch failed https://registry.npmjs.org/vinyl/-/vinyl-2.1.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443", "npm ERR! fetch failed https://registry.npmjs.org/readable-stream/-/readable-stream-1.0.34.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.95.96:443", "npm ERR! fetch failed https://registry.npmjs.org/vinyl/-/vinyl-0.4.6.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.96.96:443", "npm ERR! fetch failed https://registry.npmjs.org/graceful-fs/-/graceful-fs-3.0.11.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443", "npm ERR! fetch failed https://registry.npmjs.org/strip-bom/-/strip-bom-1.0.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.97.96:443", "npm ERR! fetch failed https://registry.npmjs.org/globby/-/globby-2.1.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.94.96:443", "npm ERR! fetch failed https://registry.npmjs.org/globby/-/globby-2.1.0.tgz", "npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 104.18.98.96:443", "npm ERR! Linux 3.10.0-862.3.2.el7.x86_64", "npm ERR! argv "/opt/minemeld/www/venv/bin/node" "/opt/minemeld/www/venv/bin/npm" "install"", "npm ERR! node v4.2.6", "npm ERR! npm v2.14.12", "npm ERR! code ETIMEDOUT", "npm ERR! errno ETIMEDOUT", "npm ERR! syscall connect", "", "npm ERR! network connect ETIMEDOUT 104.18.94.96:443", "npm ERR! network This is most likely not a problem with npm itself", "npm ERR! network and is related to network connectivity.", "npm ERR! network In most cases you are behind a proxy or have bad network settings.", "npm ERR! network ", "npm ERR! network If you are behind a proxy, please make sure that the", "npm ERR! network 'proxy' config is set properly. See: 'npm help config'", "", "npm ERR! Please include the following file with any support request:", "npm ERR! /opt/minemeld/www/webui/npm-debug.log"], "stdout": "", "stdout_lines": []}

Please help.

Typo in path for supervisorctl command

Luigi,

Please update the Wiki under Building the Core VM step 10 which currently reads:

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

To read instead:

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

The supervisor directory is actually right under /opt/minemeld, not /opt/minemeld/local/. This will help users that aren't familiar with the path troubleshoot the error they will receive if they run that command as is:

Error: could not find config file /opt/minemeld/local/supervisor/config/supervisord.conf
For help, use /opt/minemeld/engine/current/bin/supervisorctl -h

Thanks!
Nasir

DROP_UPDATE

Hi,

I've just created a new node and I'm seeing events such us: DROP_UPDATE on aggregator type.

My miner has all domain list but the aggregator has this meessge with "drop_update", do you know it??? Why???

Thanks a lot

Feeds for single IPs instead of ranges

Hello,

Does anyone have any suggestion creating output IP feeds that only contain lists of individual IP addresses instead of IP ranges.

For example:
What I need:
123.10.10.1
123.10.10.2
124.11.11.6

Instead of:
123.10.10.1-123.10.10.2
124.11.11.6-124.11.11.6

The challenge is, some of our tools (Cisco TID) only ingests single IPs and not ranges.

Any advice is appreciated!

Thanks!

Provide documentation on Outputing to remote logstash (if possible)

Thank you

Missing Content-Length header

Hi,

I have a question. Is Minemeld API sending the Content-Length header along with dynamic content (feeds)?
I mean, I,ve tried to retrieve the feed´s content length coming from upstream server (app_server), but I have not been successful.

For example, for the feed: https://192.168.1.1/feeds/inboundfeedhc, I understand that the content-length will be stored in the Nginx variable ** $upstream_response_length**

But when I test this, response headers show: Content-Length: 0

minemeld isn't applying 'tags' filter to CIF requests

In minemeld under the CIF miner I see options for tags, but the actual client doesn't appear to utilize the tags for filtering in the CIF request.
https://github.com/PaloAltoNetworks/minemeld-core/blob/master/minemeld/ft/cif.py#L155

The tags should be sent with the request to filter, and reduce the time/size of the results.

Authorization feature - multiple user types on web gui

We would like to use a stdlib.listIPv4Generic miner, editing this static list via web gui.
It would be nice, if I could create special users for NOC/SOC operators who are permitted to add/remove addresses to/from the miner, but modifying the node structure or creating new users or purge logs is forbidden.
This feature would require two user roles/types: administrator (who can do anything), and operators (who can only add/delete indicators to/from a specific node).

Duplicate IOCs in Logstash node

Hello guys,
I have configured a miner in Minemeld to collect IOC's from MISP.
The configuration used is below:

age_out
default: null
sudden_death: true
interval: 10800
source_name: misp

Every three hours the polling sends IOC's to my logstash node. In SIEM I see same IOC with same tag,attribute and misp uid but with different timestamp (a few milliseconds of difference).
There are duplicate event for me.
I suppose the issue is in configuration of miner node.
Could you help me to set it correctly to avoid this behavior?

Many thanks

Performance problem when stdlib.listIPv4Generic node contains a large number of addresses

We would like to use a stdlib.listIPv4Generic miner with more than 25000 addresses. These addresses has been imported from a texfile using minemeld-sync.py.
My problem is that it takes around 20 seconds to list the indicators (500 pages, 50 indicators per page). During the process minemeld generates the paginator component, one CPU core is used at 100%, but other cores are not used.
When I add a new IP, after ~10 seconds I get a green message saying "Changes saved". When I refresh the indicator list, I can find the recently added address. During the add process one core of the CPU works at 100% for several sconds. So at only one address it works, just a bit slow.
When I add 3-4 IPs one-by-one using the web gui (my typing speed is around 3 seconds per IP), I get a red message: "Error adding indicator: Timeout". In this case recently added addresses has not got in the list.

Under "How to write a simple miner", the link to Minemeld launcher does not work

Link to the page here:

https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner

O365 Miner Not Retrieving IOCs

Just noticed this happened recently, but the office365.any miner isn't retrieving any IOCs. This was working without issue for almost a year. The office365.o365 miner doesn't retrieve anything either. Did MS change something on their end that the miner hasn't been updated for?

CONFIGURE AUTHENTICATION BY LDAP/TACACS+

Hi all,

It's possible configure LDAP/TACACS+ on Minemeld login?
Thanks.

save IOC from minemeld as file

Hello Team,

Can some one help me with Logstash output documentation. I need to save the IOC collected from minemeld into either JSON or CSV or YAML file so that the file can be used by Logstash using translate filter for comparing with elasticsearch data.

So i tried Logstash output to collect the TCP input from minemeld and save it as file. but my logstash output node in minemeld is with zero indicators though my aggregator has so many indicators.

kindly help me if there is any other way to save IOC from minemeld as file.

I tried this logstash config but no success. dont know what i am missing. I might be missing something from minemeld or logstash end. kind help is much appreciated.

#logstash configuration:
input {
tcp {
port => 5514
}
}

output {
file {
path =>/etc/logstash/minemeld-output.txt
}
}

Wrong link location

Hi,
the MineMeld section of the Palo Alto Networks live community link wrote in the README.md is wrong (actual link is https://live.paloaltonetworks.com/MineMeld), it gives the following message: The page you are trying to access was not found. Please check your URL for typos and try again.
Maybe the correct page is this? https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld

Best regards,
Giorgio

Miners from complex STIX data

Hello,
I configured correctly miner (TAXII-SERVER)-->aggregator (file,ip,md5 and so on)-->output node (TAXII DATA FEED).
This structure works without problem if the STIX is a sample STIX but if I have a STIX with Observable_Composition Operator and several observable _id with several cybox:Properties xsi:type and cybox:Related_Object id Minemeld does not download indicators.
Also I do not have any error during the poll into my TAXII Server, I download in fact all STIX package.
Thanks for your help

Add route to minemeld UI

Hi,

I want to add a route to navbar in minemeld UI . Which all files are need to be modified i also want to write a function for that route which will run some lines of code and render a template.

Thanks in advance.

Kubernetes incompatibilites and other issues

Hey y'all - I've been asked to spin up this application inside a kubernetes cluster, but I'm looking at the docker image and it seems to be odd because it runs at least 10 different daemons(!) within the container, several of which log to files(?), and that isn't dockeresque or a modern microservice architecture design in the slightest.

I've been digging through the code and documentation, and it's not obvious to me if or how the system would run mindmeld/syslog/cron(?)/redis/collectd as microsevices, with, perhaps, environmental variables used to configure their locations. I've also been unable to glean what the individual components requirements are in terms of IPC.

The largest "blocker" for me is the inclusion of an NGINX daemon that serves 301 redirects to https, which makes it hard to run behind the Cluster Ingress which would (ideally) assume the ssl offloading role.

Is it possible to publish stand-alone, documented, docker images for the Core and UI applications which have the ability to set the locations and ports for redis, syslog-ng, syslog-forwarder, and collectd (if required?) via environmental variables or is there some reason they all need to be on the same host/in the same container?

For people interested in running it outside of Kubernetes, a docker-compose.yml file would suffice.

how to install 0.9.50 directly?

I have an issue when I install the minemeld.
Setting up minemeld (0.9.11-3build1) ... Traceback (most recent call last): File "/usr/sbin/minemeld-auto-update", line 15, in <module> import grp ImportError: No module named grp

how to fix this issue, or how I can install the latest version 0.9.50 directly? Thanks!

taxii._decode_object_properties ERROR

Hi all,

during the IoC import from Taxii Feed (FS-ISAC) to Minemeld through a Miner Taxii client (minemeld.ft.taxii.TaxiiClient) i've found some issue in minemeld-engine.log:

2018-05-12T15:45:05 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:05 (50697)taxii._poll_collection INFO: node-Prova - TAXII Content contains observables but no indicators
2018-05-12T15:45:05 (50697)taxii._incremental_poll_collection INFO: node-Prova - polling datetime.datetime(2018, 4, 4, 14, 41, 55, tzinfo=) to datetime.datetime(2018, 4, 5, 14, 41, 55, tzinfo=)
2018-05-12T15:45:07 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:07 (50697)taxii._poll_collection INFO: node-Prova - TAXII Content contains observables but no indicators
2018-05-12T15:45:07 (50697)taxii._incremental_poll_collection INFO: node-Prova - polling datetime.datetime(2018, 4, 5, 14, 41, 55, tzinfo=) to datetime.datetime(2018, 4, 6, 14, 41, 55, tzinfo=)
2018-05-12T15:45:10 (50697)taxii._decode_object_properties ERROR: node-Prova - no email address listed
2018-05-12T15:45:10 (50697)taxii._decode_object_properties ERROR: node-Prova - unknown type MutexObjectType {'xsi:type': 'MutexObjectType', 'name': 'Global\C::Users:User01:AppData:Local:Microsoft:Windows:Explorer:thumbcache_1024.db!dfMaintainer'}
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:41 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:42 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]
2018-05-12T15:45:42 (50697)taxii._decode_object_properties ERROR: node-Prova - Unknown IP version: [email protected]

As you can see above the Miner succeed to get all IoC (sha256,md5, IP etc..) exept the e-mail.

My Minemeld version is: 0.9.44

Could you please help me?

Thank you in advance!
Regards,
R.

How to debug a Miner - without committing

Hello MineMeld Community,

is there any easy way to quickly test if a miner works or not?
Its really annoying to delete the miner (and the configs) everytime I need to "test" my miner.

How are you "developing" all your miners?

Kind regards

ConnectionError: Too many connections

Hi,

I'm using outputs feeds from minemeld in my Firewall and I have seen the next:

If I try to retrieve an output with 142 or less indicators, it works. The Firewall shows the entries from minemeld output node.
If I try to retrieve 150 or more indicators from an minemeld output, it doesn't works. The entries in the Firewall are empty.

The logs in/opt/minemeld/log/minemeld-web.log show this:

**Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/minemeld/flask/events.py", line 28, in _retry_wrap
self._listen()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/minemeld/flask/events.py", line 37, in _listen
pubsub = self.SR.pubsub(ignore_subscribe_messages=True)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/client.py", line 563, in pubsub
return PubSub(self.connection_pool, **kwargs)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/client.py", line 2079, in init
conn = connection_pool.get_connection('pubsub', shard_hint)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/connection.py", line 897, in get_connection
connection = self.make_connection()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/connection.py", line 904, in make_connection
raise ConnectionError("Too many connections")
ConnectionError: Too many connections
[2019-06-12 20:31:39 UTC] [12421] [ERROR] Exception in event listener
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/minemeld/flask/events.py", line 28, in _retry_wrap
self._listen()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/minemeld/flask/events.py", line 37, in _listen
pubsub = self.SR.pubsub(ignore_subscribe_messages=True)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/client.py", line 563, in pubsub
return PubSub(self.connection_pool, kwargs)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/client.py", line 2079, in init
conn = connection_pool.get_connection('pubsub', shard_hint)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/connection.py", line 897, in get_connection
connection = self.make_connection()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/redis/connection.py", line 904, in make_connection
raise ConnectionError("Too many connections")
ConnectionError: Too many connections
[2019-06-12 20:31:39 UTC] [12421] [ERROR] Exception in event listener
Traceback (most recent call last):

Could someone help me?

Thanks in advance!

paloaltonetworks / minemeld Goto Github PK

minemeld's Introduction

MineMeld

Getting Started

Source Code

Community

minemeld's People

Contributors

Stargazers

Watchers

Forkers

minemeld's Issues

Recommend Projects

Recommend Topics

Recommend Org