dynamiteai / dynamite-nsm Goto Github PK

I install just ES like this, failed in passwd?
`[root@venus dynamite_nsm]# dynamite elasticsearch install --es-heap-size=5 --verbose
2020-07-09 20:35:43 DYNAMITE_CMD INFO | LOG LEVEL: 10
2020-07-09 20:35:43 DYNAMITE_CMD INFO | LOG FILE: /var/log/dynamite/dynamite-09-07-2020.log
[?] Enter the password for logging into ElasticSearch:
[?] Confirm Password:
2020-07-09 20:35:49 ELASTICSEARCH ERROR | Could not resolve ES_PATH_CONF environment variable. Is ElasticSearch installed?
/tmp/dynamite/install_cache/elasticsearch-7.2.0.tar.gz
/tmp/dynamite/install_cache/java-11.0.2.tar.gz
useradd: user 'dynamite' already exists
2020-07-09 20:35:59 ELASTICSEARCH INFO | Creating ElasticSearch installation, configuration, and logging directories.
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/config/elasticsearch.yml -> /etc/dynamite/elasticsearch/
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/config/jvm.options -> /etc/dynamite/elasticsearch/
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/config/log4j2.properties -> /etc/dynamite/elasticsearch/
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/bin/ -> /opt/dynamite/elasticsearch/bin/
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/lib/ -> /opt/dynamite/elasticsearch/lib/
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/logs/ -> /opt/dynamite/elasticsearch/logs/
2020-07-09 20:35:59 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/modules/ -> /opt/dynamite/elasticsearch/modules/
2020-07-09 20:36:00 ELASTICSEARCH DEBUG | Copying /tmp/dynamite/install_cache/elasticsearch-7.2.0/plugins/ -> /opt/dynamite/elasticsearch/plugins/
2020-07-09 20:36:00 ELASTICSEARCH INFO | Updating ElasticSearch default configuration path [/etc/dynamite/elasticsearch/]
2020-07-09 20:36:00 ELASTICSEARCH INFO | Updating ElasticSearch default home path [/opt/dynamite/elasticsearch/]
2020-07-09 20:36:00 ELASTICSEARCH INFO | Overwriting default configuration.
2020-07-09 20:36:00 ELASTICSEARCH INFO | Setting up JVM default heap settings [5GB]
2020-07-09 20:36:00 ELASTICSEARCH INFO | Setting up Max File Handles [65535] VM Max Map Count [262144]
vm.max_map_count = 262144
fs.file-max = 65535
fs.inotify.max_user_watches = 524288
vm.max_map_count = 262144
fs.file-max = 65535
2020-07-09 20:36:00 ELASTICSEARCH INFO | Creating certificate keystore.
2020-07-09 20:36:17 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-07-09 20:36:21 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-07-09 20:36:24 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-07-09 20:36:27 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-07-09 20:36:30 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-07-09 20:36:33 ELASTICSEARCH INFO | [Attempt: 6] Starting ElasticSearch on PID [19952]
2020-07-09 20:36:33 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-07-09 20:36:38 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-07-09 20:36:43 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-07-09 20:36:48 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-07-09 20:36:53 ELASTICSEARCH INFO | ElasticSearch API is up.
2020-07-09 20:36:58 ELASTICSEARCH INFO | Bootstrapping passwords.
2020-07-09 20:37:11 ELASTICSEARCH DEBUG | Updating password for apm_system
2020-07-09 20:37:11 ELASTICSEARCH DEBUG | Fallback; encoding bytes to utf-8 charset before building b64 auth package.
2020-07-09 20:37:11 ELASTICSEARCH DEBUG | Fallback; encoding bytes to utf-8 charset making request.
2020-07-09 20:37:11 ELASTICSEARCH ERROR | General exception while resetting Elasticsearch password.
2020-07-09 20:37:11 ELASTICSEARCH DEBUG | General exception while resetting Elasticsearch password; Remote end closed connection without response
2020-07-09 20:37:11 ELASTICSEARCH ERROR | Failed to bootstrap password.
2020-07-09 20:37:11 ELASTICSEARCH ERROR | General error occurred while attempting to bootstrap ElasticSearch passwords.
2020-07-09 20:37:11 ELASTICSEARCH DEBUG | General error occurred while attempting to bootstrap ElasticSearch passwords An error occurred while attempting to install component: An error occurred while installing elasticsearch: Failed to bootstrap password.
2020-07-09 20:37:11 DYNAMITE_CMD CRITICAL | Fatal error. Exiting.
2020-07-09 20:37:11 DYNAMITE_CMD DEBUG | Fatal error. Exiting; An error occurred while attempting to install component: An error occurred while installing elasticsearch: General error occurred while attempting to bootstrap ElasticSearch passwords An error occurred while attempting to install component: An error occurred while installing elasticsearch: Failed to bootstrap password.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/config.py", line 310, in set_user_password
urlopen(url_request)
File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.6/urllib/request.py", line 524, in open
req = meth(req)
File "/usr/lib64/python3.6/urllib/request.py", line 1248, in do_request
raise TypeError(msg)
TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/config.py", line 313, in _set_user_password
urlopen(url_request, data=json.dumps({'password': password}).encode('utf-8'))
File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib64/python3.6/urllib/request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib64/python3.6/urllib/request.py", line 1321, in do_open
r = h.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/install.py", line 353, in setup_passwords
setup_from_bootstrap(bootstrap_p_res[0].decode())
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/install.py", line 285, in setup_from_bootstrap
es_pass_config.set_all_passwords(new_password=self.password)
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/config.py", line 392, in set_all_passwords
self.set_apm_system_password(new_password)
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/config.py", line 338, in set_apm_system_password
self._set_user_password('apm_system', new_password)
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/config.py", line 329, in _set_user_password
"General exception while resetting Elasticsearch password; {}".format(e))
dynamite_nsm.exceptions.ResetPasswordError: An error occurred while attempting to reset password: General exception while resetting Elasticsearch password; Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/install.py", line 358, in setup_passwords
raise elastic_exceptions.InstallElasticsearchError("Failed to bootstrap password.")
dynamite_nsm.services.elasticsearch.exceptions.InstallElasticsearchError: An error occurred while attempting to install component: An error occurred while installing elasticsearch: Failed to bootstrap password.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/dynamite", line 88, in
ElasticsearchCommandlineComponent(args)
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/components/elasticsearch/component.py", line 103, in init
self.execute_install_strategy()
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/components/base/component.py", line 63, in execute_strategy_function
self.execute_strategy(self.component_name, strategy)
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/components/base/component.py", line 82, in execute_strategy
strategy.execute_strategy()
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/components/base/execution_strategy.py", line 51, in execute_strategy
func(**args)
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/install.py", line 416, in install_elasticsearch
es_installer.setup_elasticsearch()
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/install.py", line 266, in setup_elasticsearch
self.setup_passwords()
File "/usr/local/lib/python3.6/site-packages/dynamite_nsm/services/elasticsearch/install.py", line 364, in setup_passwords
"General error occurred while attempting to bootstrap ElasticSearch passwords {}".format(e))
dynamite_nsm.services.elasticsearch.exceptions.InstallElasticsearchError: An error occurred while attempting to install component: An error occurred while installing elasticsearch: General error occurred while attempting to bootstrap ElasticSearch passwords An error occurred while attempting to install component: An error occurred while installing elasticsearch: Failed to bootstrap password.
`

dynamite_nsm.services.*.config modules have no base implementation. They are all over the place, even though they rely on many of the same underlying concepts for config manipulation.

We will be moving towards base YAML, INI, class abstractions.

• #83 Create Service Object Primative Classes
• Create YAML Base config class
• Create INI Base config class
• Filebeat ConfigManager (YAML)
• Suricata ConfigManager (YAML)
• Zeek Node.cfg (INI)
• ElasticSearch ConfigManager (YAML)
• Kibana ConfigManager (YAML)
• Logstash ConfigManager (YAML)

Hello,
How can we use network flow as sflow for example ?
we have Ovs switch with sflow configure on it, how send and receice into dynamite ?
Regards,
Nicolas

Description: Agent installation fails due to zeek-af_packet-plugin not being able to find the correct kernel headers during compilation. This happens when running in a container (VM in my case under QubesOS), where the incorrect kernel may be returned by uname -r.

Error:

Could NOT find KernelHeaders (missing: KERNELHEADERS_ROOT_DIR)

Need a way to specify kernel path for ./configure and maybe option to compile using zeek-af_packet-plugin master branch (not sure if current release is compatible with Zeek 4.0?).

Expected Result: zeek-af_packet-plugin compiles correctly, allowing agent installation to complete without errors.

Steps to Reproduce:

1. Install the dynamite python package in a container or VM.
2. Attempt to install the agent on the same instance using sudo dynamite agent install --inspect-interfaces eth0 --targets https://dynamite-monitor:9200
3. All steps succeed, except the compilation of zeek-af_packet-plugin

This is related to:

• zeek/zeek-af_packet-plugin#24
• zeek/zeek-af_packet-plugin#29
• zeek/package-manager#98.

Btw, I was able to successfully manually compile the plugin from git source using these changes. However, the agent install command does not pickup that it is installed on the system and still tried to compile it from source itself, which errors. Is there any good workaround for this issue?

Evaluate available CI Frameworks and determine which is the best fit for DynamiteNSM.

• TravisCI
• CircleCI
• Github Matrices
• More

Fully exercise all I/O operations available to services.suricata.config.ConfigManager

• #70 - Create Service Config Base Classes
• Implement unittest based tests to fully exercise all I/O operations.

It is becoming obvious that additional abstraction is needed to manipulate the somewhat convoluted ConfigManager system.

Developers for example should be able to modify a configuration in the following way, rather than the current way of manipulating complex (undocumented) dictionaries.

config = ConfigManager('/opt/dynamite/filebeat')
config.kafka_targets.username = 'admin'
config.kafka_targets.enabled = True
config.elasticsearch_targets.enabled = False
config.commit()

• Filebeat
• Zeek
• Suricata
• ElasticSearch
• Logstash
• Kibana

When i stop logstash, but the process is ruuning, i need to 'kill -9 pid'

root@earth:/home# dynamite logstash stop
2020-08-07 19:32:22 DYNAMITE_CMD INFO | LOG LEVEL: 20
2020-08-07 19:32:22 DYNAMITE_CMD INFO | LOG FILE: /var/log/dynamite/dynamite-07-08-2020.log
2020-08-07 19:32:22 LOGSTASH INFO | Attempting to stop LogStash [19124]
2020-08-07 19:32:22 LOGSTASH ERROR | An error occurred while attempting to stop LogStash.
{
"PID": 19124,
"RUNNING": false,
"USER": "dynamite",
"LOGS": "/var/log/dynamite/logstash/logstash-plain.log"
}
root@earth:/home# dynamite logstash status
2020-08-07 19:32:30 DYNAMITE_CMD INFO | LOG LEVEL: 20
2020-08-07 19:32:30 DYNAMITE_CMD INFO | LOG FILE: /var/log/dynamite/dynamite-07-08-2020.log
{
"PID": 19124,
"RUNNING": false,
"USER": "dynamite",
"LOGS": "/var/log/dynamite/logstash/logstash-plain.log"
}
root@earth:/home# ps aux |grep logstash
dynamite 19122 463 28.3 14824464 4654416 ? Sl 19:30 10:30 /usr/lib/jvm/jdk-11.0.2//bin/java -cp /opt/dynamite/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/opt/dynamite/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/opt/dynamite/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/opt/dynamite/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/opt/dynamite/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/guava-22.0.jar:/opt/dynamite/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/jackson-annotations-2.9.8.jar:/opt/dynamite/logstash/logstash-core/lib/jars/jackson-core-2.9.8.jar:/opt/dynamite/logstash/logstash-core/lib/jars/jackson-databind-2.9.8.jar:/opt/dynamite/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.8.jar:/opt/dynamite/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/opt/dynamite/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/opt/dynamite/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/opt/dynamite/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/opt/dynamite/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/logstash-core.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/opt/dynamite/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/opt/dynamite/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings=/etc/dynamite/logstash/
root 19484 0.0 0.0 17668 732 pts/2 S+ 19:32 0:00 grep --color=auto logstash
root@earth:/home# kill -9 19122
root@earth:/home# ps aux |grep logstash
root 19486 0.0 0.0 17668 2880 pts/2 S+ 19:32 0:00 grep --color=auto logstash

According to doc at https://dynamite-nsm.readthedocs.io/en/0.7.2/getting_started/agent_setup/
I have to execute: dynamite agent-dependencies install
But it throws option not found. Any problems if we don't install agent-dependencies?
I install with pip and version is 0.7.2
Thanks

ElastiFlow and Synesis are amazing tools, but maintaining compatibility with them has introduced significant technical debt into DynamiteNSM.

We will be migrating away from these LogStash pipelines and instead handle normalization on the agent component itself, allowing for more flexible downstream integration.

NetFlow, IPFix, Zeek, and Suricata will be supported via ECS going forward.

As of 0.8.0 the agent component supports

https://www.elastic.co/guide/en/ecs/current/ecs-using-ecs.html

• #72 - Migrate to Open Distro
• Standarize Default Index Names (Templates/Patterns)
• Remove ElastiFlow Dashboards
• Remove Synesis Dashboards

With the transition away from Python2 we can leverage Python3's typing module. Explicitly defining types for all methods' parameters and return types moves us towards a more readable and testable code.

• Type-hints across all service modules
• Type-hints across all component modules
• Type-hints across utilities.py

Create a list of pros and cons based on dashboards available within the latest version of Security Onion.

What works? What can be improved upon?

Strip out any legacy code related to Python2. We are no longer going to support it.

I have install java11.0.8 before, but it will reinstall java when i install monitor.

Hi,

This tool is amazing.
I'm wondering that is it possible to import Pcap files for analysis rather than deploy an agent?

Affects: Version 0.6.4

When Logstash is ES_PASSWD is never passed through, and authentication fails.

Hi,
I've read with interest https://github.com/DynamiteAI/dynamite-nsm/wiki/Agent-Deployment-Considerations and was wondering if you had succes stories to share related to agent install on 40Gb cards + any tweaks required on high speed hw in general.

Thanks!

Wherever possible on native fileIO, and remove subprocess calls invoking shell=True

• audit utilities.py
• audit service (install.InstallManager) classes

I have a virtual machine, 16GB mem and 16 vCpus
i want to install monitor in this, and another vm deploy agent, Unfortunately，i always fail to install monitor , the log as below.

[?] Enter the password for logging into ElasticSearch: develop
[?] Confirm Password: develop

root@earth:~# dynamite monitor install --es-heap-size=5 --ls-heap-size=3 --ls-listen-addr 192.168.11.44 --kb-listen-addr 192.168.11.44 --kb-listen-port 5601
2020-08-07 02:09:17 DYNAMITE_CMD INFO | LOG LEVEL: 20
2020-08-07 02:09:17 DYNAMITE_CMD INFO | LOG FILE: /var/log/dynamite/dynamite-07-08-2020.log
[?] Enter the password for logging into ElasticSearch:
[?] Confirm Password:
2020-08-07 02:09:20 ELASTICSEARCH ERROR | Could not resolve ES_PATH_CONF environment variable. Is ElasticSearch installed?
2020-08-07 02:09:20 LOGSTASH ERROR | Could not resolve LS_PATH_CONF environment variable. Is Logstash installed?
2020-08-07 02:09:20 ELASTICSEARCH ERROR | Could not resolve ES_PATH_CONF environment variable. Is ElasticSearch installed?
2020-08-06 18:09:22 DOWNLOAD_MANAGER | 2.55 MB/s |#############################| (elasticsearch-7.2.0.tar.gz) Time: 0:02:11
2020-08-06 18:11:40 DOWNLOAD_MANAGER | 1.36 MB/s |#####################################| (java-11.0.2.tar.gz) Time: 0:02:18
useradd: user 'dynamite' already exists
2020-08-07 02:14:01 ELASTICSEARCH INFO | Creating ElasticSearch installation, configuration, and logging directories.
2020-08-07 02:14:02 ELASTICSEARCH INFO | Updating ElasticSearch default configuration path [/etc/dynamite/elasticsearch/]
2020-08-07 02:14:02 ELASTICSEARCH INFO | Updating ElasticSearch default home path [/opt/dynamite/elasticsearch/]
2020-08-07 02:14:02 ELASTICSEARCH INFO | Overwriting default configuration.
2020-08-07 02:14:02 ELASTICSEARCH INFO | Setting up JVM default heap settings [5GB]
2020-08-07 02:14:02 ELASTICSEARCH INFO | Setting up Max File Handles [65535] VM Max Map Count [262144]
2020-08-07 02:14:02 ELASTICSEARCH INFO | Creating certificate keystore.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020-08-07 02:14:10 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-08-07 02:14:13 ELASTICSEARCH WARNING | An issue occurred while attempting to start.
2020-08-07 02:14:16 ELASTICSEARCH INFO | [Attempt: 3] Starting ElasticSearch on PID [1995]
2020-08-07 02:14:16 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-08-07 02:14:21 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-08-07 02:14:26 ELASTICSEARCH INFO | ElasticSearch API is up.
2020-08-07 02:14:31 ELASTICSEARCH INFO | Bootstrapping passwords.
2020-08-07 02:14:34 ELASTICSEARCH ERROR | General exception while resetting Elasticsearch password.
2020-08-07 02:14:34 ELASTICSEARCH ERROR | Failed to bootstrap password.
2020-08-07 02:14:34 ELASTICSEARCH ERROR | General error occurred while attempting to bootstrap ElasticSearch passwords.
2020-08-07 02:14:34 DYNAMITE_CMD CRITICAL | Fatal error. Add --verbose flag for details. Exiting.

I try it again:
2020-08-07 02:30:21 KIBANA INFO | Updating Kibana default configuration path [/etc/dynamite/kibana/]
2020-08-07 02:30:21 KIBANA INFO | Updating Kibana default home path [/opt/dynamite/kibana/]
2020-08-07 02:30:21 KIBANA INFO | Updating Kibana default log path [/var/log/dynamite/kibana/]
2020-08-07 02:30:21 KIBANA INFO | Overwriting default configuration.
2020-08-07 02:30:21 KIBANA INFO | Installing Kibana Dashboards
2020-08-07 02:30:21 KIBANA INFO | Waiting for ElasticSearch to become accessible.
2020-08-07 02:30:21 KIBANA INFO | Starting ElasticSearch.
2020-08-07 02:30:21 ELASTICSEARCH INFO | ElasticSearch is already running on PID [1995]
2020-08-07 02:30:21 KIBANA INFO | ElasticSearch API is up.
2020-08-07 02:30:21 KIBANA INFO | Sleeping for 5 seconds, while ElasticSearch API finishes booting.
2020-08-07 02:30:26 KIBANA INFO | Optimizing Kibana Libraries.
2020-08-07 02:30:49 KIBANA INFO | Starting Kibana.
2020-08-07 02:30:54 KIBANA INFO | [Attempt: 1] Starting Kibana on PID [3342]
2020-08-07 02:30:54 KIBANA INFO | Kibana API is up.
2020-08-07 02:30:54 KIBANA INFO | Sleeping for 10 seconds, while Kibana API finishes booting.
2020-08-07 02:31:05 KIBANA INFO | [Attempt 1] Attempting to install dashboards/visualizations.
[-] Failed to create ElastiFlow objects - [Note: Unnecessary use of -X or --request, POST is already inferred.

• Uses proxy env variable http_proxy == 'http://192.168.11.70:808'
• Trying 192.168.11.70:808...
• TCP_NODELAY set
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.11.70 (192.168.11.70) port 808 (#0)
• Server auth using Basic with user 'elastic'

POST http://192.168.11.44:5601/api/saved_objects/_import HTTP/1.1
Host: 192.168.11.44:5601
Authorization: Basic ZWxhc3RpYzpkZXZlbG9w
User-Agent: curl/7.68.0
Accept: /
Proxy-Connection: Keep-Alive
kbn-xsrf: true
Content-Length: 3578604
Content-Type: multipart/form-data; boundary=------------------------dc0a62586229f500
Expect: 100-continue

• Done waiting for 100-continue
  0 3494k 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0} [65536 bytes data]
• We are completely uploaded and fine
• Mark bundle as not supporting multiuse
  < HTTP/1.1 100 Continue
• Mark bundle as not supporting multiuse
  < HTTP/1.1 503 Service Unavailable
  < retry-after: 30
  < content-type: text/html; charset=utf-8
  < cache-control: no-cache
  < content-length: 30
  < connection: close
  < Date: Thu, 06 Aug 2020 18:31:05 GMT
  <
  { [30 bytes data]
  100 3494k 100 30 100 3494k 22 2627k 0:00:01 0:00:01 --:--:-- 2627k
• Closing connection 0

===============================================================

1. if i fail to install monitor, the install process will download tar.gz again and again, can you check md5 to avoid repeated downloads?

We need to identify the roles that dynamited will fill for the open-source project.

Hi, there is a logstash warning and the data are not indexed in elaticsearch, the warning is:
[WARN ][logstash.filters.split ] Only String and Array types are splittable. field:zeek is of type = NilClass
any suggestions?

Since the Open Distro project is archived (opendistroforelasticsearch-1.13.3-linux-x64.tar.gz can't be downloaded any more) and replaced by OpenSerach the installer does not work.

Fully exercise all I/O operations available to services.kibana.config.ConfigManager

• #70 - Create Service Config Base Classes
• Implement unittest based tests to fully exercise all I/O operations.

Hi, there are a bug, when I install monitor, it always waiting for ElasticSearch API to become accessible, my OS is ubuntu18.04, and the memory is 8G.

OpenDistro provides a fully open-source alternative to ElasticSearch and Kibana. Furthermore, it provides an excellent alternative to XPack authentication and anomaly detection modules.

The monitor will install OpenDistro ElasticSearch and Kibana going forward.

• Migrate to Filebeat-OSS
• Monitor component to install OpenDistro ElasticSearch and setup default account
• Monitor component to install OpenDistro Kibana (without any dashboards)

Determine if DynamiteNSM can leverage OpenDistro's Alerting Module

Fully exercise all I/O operations available to services.zeek.config.ConfigManager

• #70 - Create Service Config Base Classes
• Implement unittest based tests to fully exercise all I/O operations.

Determine if it is feasible to programmatically generate Kibana visualizations and dashboards built around a pre-defined set of fields.

Fully exercise all I/O operations available to services.filebeat.config.ConfigManager

• #70 - Create Service Config Base Classes
• Implement unittest based tests to fully exercise all I/O operations.

With the migration to ECS standard we can expect relative consistency across many fields.

• Identify dashboards which can be derived from these common fields
• Identify dashboards that are log specific (E.G protocols)
• Lessons learned from other NSM projects (Security Onion #73)/ElastiFlow/Synesis

Determine how OpenDistro's Anomaly Detection Module can be leveraged.

Currently, only Kafka and Logstash outputs are supported by the command-line utility. Setup should allow the user to select from any of Filebeats supported options.

• Remove the need to define output connector at installation time (Default to logstash)
• Add configuration options to configure all supported connectors (Kafka, ElasticSearch, Logstash, and Redis) within TUI.

DynamiteNSM Docs desperately need a refresh.

The codebase is too large at this point not to fully automate this process.

Fully exercise all I/O operations available to services.logstash.config.ConfigManager

• #70 - Create Service Config Base Classes
• Implement unittest based tests to fully exercise all I/O operations.

Currently, the monitor set's up a 1-node ElasticSearch cluster. However, users may wish to scale this up in the future. We should account for this workflow.

• Create from commandline
• Create from configuration
• How is authentication and SSL (node-node encryption handled?

Notices that events stop on the monitor after a few days,

I found this error in the logstash log file /var/log/dynamite/logstash/logstash-plain.log

[2020-04-21T14:14:51,111][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"event-flows-2020.04.21", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x42ca4ce>], :response=>{"index"=>{"_index"=>"event-flows-2020.04.21", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}}}} [2020-04-21T14:14:51,115][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dns-events-2020.04.21", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x4e962452>], :response=>{"index"=>{"_index"=>"dns-events-2020.04.21", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}}}} [2020-04-21T14:14:51,115][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dns-events-2020.04.21", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x549c373b>], :response=>{"index"=>{"_index"=>"dns-events-2020.04.21", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}}}}

System is very simple one agent that sends to one monitor, enough disk space, only one Elasticsearch node.

I notice in the templates shards is set to 3 and replicas 1

What is the best cost effective way to address this, Increase the ES nodes or to reduce the shards and replicas.

what about a curator scrip to delete old data and scheduled script to merge the indices

I have a virtual machine, 16GB memory and 16 vCpus, OS(ubuntu 20)
Just install elasticsearch always failed, i found that it is related to the security certificate setting and password changing.

When I ignore this error and start directly, I find that I cannot log in with my account password (elastic/mypasswd)

root@earth:~# dynamite elasticsearch install
2020-08-07 17:40:28 DYNAMITE_CMD INFO | LOG LEVEL: 20
2020-08-07 17:40:28 DYNAMITE_CMD INFO | LOG FILE: /var/log/dynamite/dynamite-07-08-2020.log
[?] Enter the password for logging into ElasticSearch:
[?] Confirm Password:
2020-08-07 17:40:46 ELASTICSEARCH ERROR | Could not resolve ES_PATH_CONF environment variable. Is ElasticSearch installed?
2020-08-07 09:41:19 DOWNLOAD_MANAGER | 2.34 MB/s |###################################| (elasticsearch-7.2.0.tar.gz) Time: 0:02:24
2020-08-07 09:43:49 DOWNLOAD_MANAGER | 1.74 MB/s |###########################################| (java-11.0.2.tar.gz) Time: 0:01:48
useradd: user 'dynamite' already exists
2020-08-07 17:45:40 ELASTICSEARCH INFO | Creating ElasticSearch installation, configuration, and logging directories.
2020-08-07 17:45:41 ELASTICSEARCH INFO | Updating ElasticSearch default configuration path [/etc/dynamite/elasticsearch/]
2020-08-07 17:45:41 ELASTICSEARCH INFO | Updating ElasticSearch default home path [/opt/dynamite/elasticsearch/]
2020-08-07 17:45:41 ELASTICSEARCH INFO | Overwriting default configuration.
2020-08-07 17:45:41 ELASTICSEARCH INFO | Setting up JVM default heap settings [4GB]
2020-08-07 17:45:41 ELASTICSEARCH INFO | Setting up Max File Handles [65535] VM Max Map Count [262144]
2020-08-07 17:45:41 ELASTICSEARCH INFO | Creating certificate keystore.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020-08-07 17:45:49 ELASTICSEARCH INFO | [Attempt: 1] Starting ElasticSearch on PID [8357]
2020-08-07 17:45:54 ELASTICSEARCH INFO | [Attempt: 2] Starting ElasticSearch on PID [9176]
2020-08-07 17:45:54 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-08-07 17:45:59 ELASTICSEARCH INFO | Waiting for ElasticSearch API to become accessible.
2020-08-07 17:46:04 ELASTICSEARCH INFO | ElasticSearch API is up.
2020-08-07 17:46:09 ELASTICSEARCH INFO | Bootstrapping passwords.
2020-08-07 17:46:12 ELASTICSEARCH ERROR | General exception while resetting Elasticsearch password.
2020-08-07 17:46:12 ELASTICSEARCH ERROR | Failed to bootstrap password.
2020-08-07 17:46:12 ELASTICSEARCH ERROR | General error occurred while attempting to bootstrap ElasticSearch passwords.
2020-08-07 17:46:12 DYNAMITE_CMD CRITICAL | Fatal error. Add --verbose flag for details. Exiting.

Cool project. Anyway to get it to accept netflow data from pfsense and other hosts?

The newest suricata is version 5.0, do you have a plan to update?

• #70 - Create Service Config Base Classes
• Implement unittest based tests to fully exercise all I/O operations.