From Malcolm created by mmguero: cisagov#14

Investigate integrating other scanners into Malcolm:

• yara
• rita
• snort
• packetbeat
• suricata

If/when I decide to move on any of these, I'll log a separate issue.

From Malcolm created by mmguero: cisagov#11

There is scripts/wipe.sh but it would be nice to have a GUI-ish way for wiping data on the fly. Maybe tie this in to Moloch's ES Indices tab, implement something to multi-select indices or pattern-select indices and delete everything with one click?

From Malcolm created by mmguero: cisagov#7

see https://wiki.wireshark.org/SampleCaptures and search for c05-http-reply-r1.pcap.

This is malformed data, but it is discarded by Elasticsearch because it is too long. is this something to be concerned about? Maybe, maybe not. i just wanted to document it to see if we want to do something about it.

there is a truncate filter:

However, we don't want to do this on all fields as it would be expensive.

From Malcolm created by mmguero: cisagov#70

https://opendistro.github.io/for-elasticsearch/

This looks like it might be a good choice for the default ES images used for Malcolm

Arkime v3 has been released.

https://arkime.com/v3release

Moved from cisagov#168 by @dajohn78

🐛 Summary

When placing a large pcap (50GB +) in the Malcolm upload folder it doesn't seem to get processed. Testing the same procedure with a 1GB pcap works just fine.

To reproduce

Steps to reproduce the behavior:

1. Place a large pcap in the 'upload' folder. (Copy or via SFTP)
2. The process pcap-monitor_1 renames (moves) the pcap to the 'processed' folder
3. Nothing happens... elasticsearch_1 doesn't create or updates the index. No zeek logs created either. Size of elasticsearch index stays the same.

Expected behavior

Creating or updating the index, creation of Zeek logs, etc.

I noticed that the clamav database wasn't being updated during docker image build. This bug is to track that fix.

From Malcolm created by mmguero: cisagov#15

The documentation should be improved to show how to use signed certificates (vs. self-signed). Use something like Let's Encrypt as an example.

Also I need to research what this looks like between a remote beats forwarder and Malcolm. I've done it with the Malcolm web interface certs (via nginx config) but I haven't set it up in beats/logstash yet.

In preseed_vmware.cfg

     .                                         \
     150% 150% 150% linux-swap                 \
       $defaultignore{ }                       \
       $lvmok{ }                               \
       in_vg { main } lv_name{ swap }          \
       method{ swap }                          \
       format{ }                               \
     .     

For large sizes of RAM this is stupid, as it will create an enormous swap partition. I need to set the maximum correctly.

In Malcolm v2.0.5 we switched from GCC toolchain to LLVM toolchain. At this point, the ldap-analyzer stopped working correctly in that the ldap.log file is not being generated for write operations.

It turns out that if zeek and/or the plugin (not sure if it's one or the other) is compiled with a GCC toolchain it "works," but with an LLVM toolchain it doesn't.

I've distilled down a reproduction environment here: https://github.com/mmguero-dev/misc-debug/tree/main/zeek-ldap-analyzer

With the Alerting plugin in open distro for elasticsearch, it appears that using SSL/TLS with an SMTP server that uses a self-signed certificate (or any certificate with a custom CA) will not work.

It appears that such a CA can be imported by running a command such as:

keytool -importcert -file foobar.crt -alias "foobar" -keystore /usr/share/elasticearch/jdk/lib/security/cacerts -keypass changeit -storepass changeit -noprompt

I have not tested this yet on the elasticsearch container, but I believe this is how it should be done. We are already importing from ./nginx/ca-trust CA root certificates for LDAP server validation, I don't think there would be any problem with using that same folder to contain this certificate and use the contents for both purposes.

Greetings,

when building hedgehog, there are no obvious errors during the generation of the ISO, however when installing it on a host, it attempts to install Debian, and then fails with a blank screen (no video output) after installing grub. The resulting install does not boot at all.

ISO built on a host using Ubuntu 18.0.4, vagrant 2.2.10, and VirtualBox 5.2.42.

From Malcolm created by mmguero: cisagov#26

Currently Malcolm's tested manually by me on a per-change basis. As the project matures, I need to look into implementing some kind of test framework that can be run overnight or something to ensure builds and functionality don't break without my knowing it.

From Malcolm created by mmguero: cisagov#41

I need to investigate what happens if the ES snapshot repository gets full, what to do if that happens.

This outlines the new "best guess" feature for identifying potential ICS protocols.

There are many, many ICS (industrial control systems)/OT protocols and Malcolm parses a handful of them. A lot of them, particularly the more obscure or proprietary ones, are unlikely to ever be supported with a full parser. But it would be nice to identify more of them even without a full parser.

This feature involves a mapping/lookup file (in the Malcolm source under zeek/config/guess_ics_map.txt) and a zeek script (zeek/config/guess.zeek) that hooks on Zeek's connection close event and looks up the protocol (e.g., tcp or udp) and destination port and/or source port to make a "best guess" at whether a connection belongs to one of those protocols based on those values alone. Of course, this could mean a much higher false positive rate than usual, so these logs (which get written into bestguess.log) are only shown in their own dashboard (Best Guess under the ICS section of the dashboards navigation pane) with a disclaimer that they might have false positives. Values such as IP addresses, ports, or UID can be used to pivot to other dashboards to investigate further.

These are categorized by vendor, where possible.

As it's not likely that all users of Malcolm will want this enabled, the environment variable ZEEK_DISABLE_BEST_GUESS_ICS in docker-compose.yml is set to true by default, meaning the bestguess.log file won't be written. Clearing this value out (setting it to empty, '') will enable it.

some change to the npm http-server package changed its location from /usr/bin to /usr/local/bin, causing the process to break as used in Malcolm. Need to reflect path in supervisord.conf config file in the kibana-helper image.

From Malcolm created by mmguero: cisagov#5

investigate logstash pipeline.workers setting (for better Logstash performance?)

nMustaki/debinterface v3.5.0 (at nMustaki/debinterface@772e978) has broken my configure-interfaces.py script which has worked for like 2 years.

Now I'm getting an error about ifup not getting a valid interface.

For a temporary fix I've reverted back to the previous release (v3.4.0) but I need to see what's up.

From Malcolm created by mmguero: cisagov#40

I ran across this the other day testing:

x-common-upload-variables: &common-upload-variables
  AUTO_TAG : 'false'

Setting AUTO_TAG to false in docker-compose.yml not only disables auto tagging, but also tags manually added via the upload web interface. That's not ideal.

This issue is for tracking the transition from ElasticSearch to the open source fork, OpenSearch.

Malcolm will be switching to the OpenSearch project as the basis of its search and analytics capabilities, mainly for two reasons:

1. Elastic.co's decision to no longer release Elasticsearch and Kibana under an open source license
2. Capabilities available under OpenSearch (and previously under Open Distro for Elasticsearch) that are only available with paid "premium" Elastic.co subscriptions (machine learning anomaly detection, alerting, reporting, etc.)

Historical context:

• Elastic announces license change

  • Amazon NOT OK
  • Doubling Down on Open
  • Doubling Down on Open, Part II
  • Elastic License v2
  • FAQ on 2021 License Change
    • Does this mean that Elasticsearch and Kibana are no longer Open Source? Yes. Neither the Elastic License nor SSPL have been approved by the OSI, so to prevent confusion, we no longer refer to Elasticsearch or Kibana as open source.
  • old "open source" tier ("Apache 2.0: Now and always" 🙄) goes away
  • The SSPL is not an open source license

• OpenSearch fork:

  • Stepping up for a truly open source Elasticsearch
  • Introducing OpenSearch
  • Truly Doubling Down on Open Source @ logz.io
  • FAQ

• Third-party blogs, etc.

  • Elasticsearch does not belong to Elastic
  • Elasticearch and Kibana are now business risks
  • Is Elasticsearch no longer open source software?
  • The Implications of Elasticsearch and Kibana License Change
  • Let's talk about the Elastic license change
  • Elastic is going closed-source. Where does that leave MSSPs?

From Malcolm created by mmguero: cisagov#152

See issue #14

Capa would be an interesting file scanner for PE (portable executable) files.

• docker-compose pull
• ./start

moloch Loading session packets No response for a long time

log

moloch_1         | Unsupported pcap file /data/pcap/processed/AUTOZEEK,AUTOCARVEall,test.pcapng link type 4294967295
moloch_1         | Sun, 01 Nov 2020 11:15:47 GMT aaa GET /moloch/session/201101-ywDxm2VLF9xDzIk-nIkIAf5a/packets?base=last&line=false&image=false&gzip=false&ts=false&decode=%7B%7D&packets=200&showFrames=false&showSrc=true&showDst=true - - bytes - ms
nginx-proxy_1    | 172.19.0.1 - aaa [01/Nov/2020:11:15:47 +0000] "GET /moloch/session/201101-ywDxm2VLF9xDzIk-nIkIAf5a/packets?base=last&line=false&image=false&gzip=false&ts=false&decode=%7B%7D&packets=200&showFrames=false&showSrc=true&showDst=true HTTP/1.1" 504 167 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
nginx-proxy_1    | 172.19.0.1 - - [01/Nov/2020:11:15:47 +0000] "GET /favicon.ico HTTP/1.1" 401 179 "https://localhost/moloch/session/201101-ywDxm2VLF9xDzIk-nIkIAf5a/packets?base=last&line=false&image=false&gzip=false&ts=false&decode=%7B%7D&packets=200&showFrames=false&showSrc=true&showDst=true" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"

A bug was introduced in kibana-create-moloch-sessions-index.sh where if index state management wasn't enabled, the zeek_template.json index wouldn't get created, leading to index pattern session errors in kibana.

the title says it all, as alternative though follow the instructions manually in the repo to change limits and sysctl.conf

also the script runs without root or sudo permissions

From Malcolm created by mmguero: cisagov#4

PCAP files with colon (:) in the name don't process correctly when put into pcap/upload folder. they are moved, but not autozeek-d or scanned by moloch-capture.

Tracking updating elastic components from 7.6.2 to 7.9.2.

The unstable/development branch these commits are being done on before integration into the main branch can be found at https://github.com/mmguero-dev/Malcolm/tree/topic/elastic_new

• elasticsearch updated and runs
• logstash updated and runs (starts up faster actually)
• filebeat (Malcolm) updated and runs
• kibana updated and runs
• kibana comments plugin builds/installs/works
• kibana swimlane plugin
• kibana sankey plugin
• kibana elastalert plugin
• kibana drilldown plugin
• test elastalert
• test moloch
• protologbeat ("heatbeat") builds now, have not tested on Hedgehog yet
• metricbeat tested on hedgehog
• auditbeat tested on hedgehog
• filebeat tested on hedgehog
• filebeat (syslog) tested on hedgehog

It appears that for (at least some) files.log entries, the FUID is getting set for both zeek.uid and zeek.fuid. It would be more useful to put the connection ID(s) in zeek.uid.

I'm implementing a change so that port 9200 isn't exposed automatically in docker-compose.yml, unless prompted to do so during configuration of install.py --configure.

moved from cisagov#167 by @colt-cmd

🐛 Summary

I currently have Malcolm installed on RHEL 8.4 and I'm trying to run the script to make a Malcolm.iso file. I have Vagrant and VirtualBox installed. Whenever I run the ./malcolm-iso/build_via_vagrant.sh -f script I get an connection to 127,0.0.1 closed by remote host along with all the CPUs saying stuck.

Install.py is supposed to prompt to change the ownership of the extracted directory to a non-root user. This was not occurring correctly.

From Malcolm created by timyardley: cisagov#144

Malcom Team,

The following add-ons may be useful to you in analyzing ICS protocols in Malcolm:

ARMORE "Bro" Scripts

Second Generation

Third Generation

Happy to elaborate on these if it isn't clear from the code.

Tim

From Malcolm created by mmguero: cisagov#125

The idea is that we assign a severity rating to logs (all logs? some logs?)

So, imagine 1 - not severe at all (blue or green), 5 - super severe (red)

in Logstash enrichment we'd do stuff like:

cleartext password - 5
connection to naughty country - 5
certain notices - 5
insecure or old versions of protocols - 4
file transfers of certain mime types - 3
connection within subnet - 1
connection to other subnet - 2
connection to outside world - 3
etc.

Of course those are just examples. I'd need to hammer out a real list.

Then in some of the dashboards, we can have "number of red events" "number of green events" etc.

From Malcolm created by mmguero: cisagov#14

Investigate integrating other scanners into Malcolm:

• yara
• rita
• snort
• packetbeat
• suricata

If/when I decide to move on any of these, I'll log a separate issue.

From Malcolm created by mmguero: cisagov#65

Some data in the database exists in two fields: for example, zeek_gquic.user_agent and quic.useragent.

The Kibana dashboards right now generally use the zeek versions. It would be better to rework them to use the moloch fields, as it would allow more data to be visualized in kibana (moloch sessions vs. just zeek logs).

created from cisagov#171

🐛 Summary

What's wrong? Please be specific.
LDAP Bind credentials in this file are readable by anyone. Can we put some permissions on the file when it gets created in the nginx entrypoint?

-rw-r--r-- /var/lib/docker/overlay2/*****/diff/etc/nginx/nginx_ldap_rt.conf

To reproduce

Steps to reproduce the behavior:

1. Standard build with LDAP
2. Grep overlay2 for bind credentials

Expected behavior

Readable only by user running nginx

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

From Malcolm created by mmguero: cisagov#16

In some cases it will make more sense for people to use their own elasticsearch OpenSearch deployment rather than Malcolm's dockerized one. For example, in order to do a larger scale-out implementation with multiple data notes, etc.

I'm going to trying things out for this in a personal branch dedicated to this topic. Specifying the connection parameters (IP/port) should be pretty easy once things are normalized into a single source of environment variables in the compose file. I think the trick will be how to specify authentication information for all of the clients. This will include:

• arkime
• logstash
• dashboards
• dashboards-helper
• pcap-monitor
• api

should cover:

• Zeek:
  • adding/enabling new zeek plugins
  • tweaking local.zeek
• Logstash
  • parsing new Zeek log files
  • new enrichments
  • completely new data sources/log file types
  • how the dynamic pipelines stuff works
• Moloch
  • adding new fields to WISE
• Kibana
  • adding new dashboards
• PCAP
  • adding a new PCAP processor
• file carving
  • adding a new carved file scanner

I'll add more to this list if I think about it.

From Malcolm created by mmguero: cisagov#10

Research ES to Splunk export/searching capabilities

Possibilities:

• Forwarding from ES via Logstash syslog
  • https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html
  • https://www.elastic.co/guide/en/logstash/current/plugins-outputs-syslog.html
• Searching ES databases with Splunk
  • https://github.com/brunotm/elasticsplunk
  • https://splunkbase.splunk.com/app/4175/#/details
  • https://github.com/hvandenb/splunk-elasticsearch

From Malcolm created by robefernandez: cisagov#113

Congratulations for the project, it's really useful and easy to setup in just minutes using the scripts and docker compose.

I've just deployed the solution for testing it so I'm actually a newbie and I have to spend more time to discover all the features but I have a question that will be decisive to continue using it or not by the moment:
Does it have asset inventory capabilities to list all the devices on the network?

I set to true the property LOGSTASH_OUI_LOOKUP (Logstash will map MAC addresses to vendors for all source and destination MAC addresses when analyzing Zeek logs).

Is there any dashboard or any place where we can obtain a list of the network devices?

Best regards.

Tracking updating zeek to 3.2.2

From Malcolm created by mmguero: cisagov#121

It might be cool to look at grassmarlin's fingerprint database and see if there's anything there we could to do identify more kinds of ICS devices. At what point would this fingerprinting be done?

@ObsidianKnife suggested better ways to fine-tune node.cfg for performance as it's created by zeekdeploy.sh on the hedgehog. See cisagov#158

I'm going to for now add the following to control_vars.conf:

export ZEEK_PIN_CPUS_LOGGER=
export ZEEK_PIN_CPUS_MANAGER=
export ZEEK_PIN_CPUS_PROXY=
# zeekdeploy.sh will also check and use (if present):
#   ZEEK_PIN_CPUS_WORKER_1 .. ZEEK_PIN_CPUS_WORKER_n
# where n is the number of capture interfaces

in addition, this existed in there previously:

export ZEEK_LB_PROCS=1
export ZEEK_LB_METHOD=custom

These variables will be used in creating node.cfg to add optional (only created if defined) pin_cpus sections for logger, manager, and proxy, and for each worker (1 .. n where n is the number of capture interfaces). Additionally, for the workers' lb_procs values, I will use the following order of preference, if they exist:

1. ZEEK_LB_PROCS_WORKER_1 .. ZEEK_LB_PROCS_WORKER_n
2. the number of pinned CPUs in ZEEK_PIN_CPUS_WORKER_1 .. ZEEK_PIN_CPUS_WORKER_n
3. the value in ZEEK_LB_PROCS (defaults to 1)

From Malcolm created by cyamal1b4: cisagov#131

Greetings, so I've been up and down in the docs for Zeek and understand decently well how to create Intel in Zeek, however, that is largely aligned with a normal deployment of Zeek. I have loaded the required policies and pointed zeek (in local.zeek) to my local bro formatted file with an indicator. I tested everything in the TryZeek page with the same pcap, local.zeek changes, as well as the same bro intel file. However, when I replicate I cannot get Malcolm (under the Kibana-Zeek Intel or Notice dashboards) to pick up on my Intel hit. I have noticed some differences in the deployment with Malcolm so I figured it would be best to ask the developer directly. Thanks for your support and contribution!!

I noticed that the LDAP plugin was broken for some of my PCAPs and did a git bisect on it. The commit zeek/spicy-analyzers@6896d49 works fine, bit the commit zeek/spicy-analyzers@a44bc7c gives this error message for ldap.spicy:393

1538420480.118397 analyzer error in /devel/github/mmguero-dev/spicy-analyzers/analyzer/protocol/ldap/ldap.spicy, line 393: unset optional value
1538420480.202406 analyzer error in /devel/github/mmguero-dev/spicy-analyzers/analyzer/protocol/ldap/ldap.spicy, line 393: unset optional value

I'm discussing it with the zeek folks in slack, but i'm thinking there's a bug in the switch-level parse-from (see added, I believe, with zeek/spicy@0895997 )

This is the PCAP I used to discover the issue.

For now I have reverted the commit (zeek/spicy-analyzers@a44bc7c) in my fork (mmguero-dev/spicy-analyzers@54bbd2d73f70b897902d399001246bc0c559f3d2) and am switching Malcolm's build instructions to build from that while we figure out why it broke upstream. That should be a temporary fix, anyway.

There's an issue where the logstash.keystore file is getting created and then mounted with different permissions, preventing logstash from starting up if the UID is not the default (1000).

This is a regression and a show-stopper if you're not using the default UID. I need to fix it before the next release.

From Malcolm created by mmguero: cisagov#79

see https://www.elastic.co/blog/introducing-the-elastic-common-schema
and https://github.com/elastic/ecs/tree/master/schemas

Eventually Moloch is going to do this as well, so we might as well get a head start.

Now that Vagrant has open sourced its vmware provider (https://www.vagrantup.com/docs/providers/vmware), It should be possible to build the ISOs using vagrant with VMware as an alternative to virtualbox. It would be nice to support both.

From Malcolm created by yagely80: cisagov#140

Ive tried to use the debug alerter, but i cant figure out where can i view these alerts.. they do not appear in the console in which malcolm is dumping all the logs

From Malcolm created by mmguero: cisagov#149

My zeek_carve_*.py scripts in shared/bin aren't bad, but I just became aware of Strelka which might scale better for a file scanning solution.

Zeek-extracted files can be preserved/"quarantined" based on scanning results, but there's not a real convenient way to get at those files.

I've added optional environment variables for a new feature:

• EXTRACTED_FILE_HTTP_SERVER_ENABLE – if set to true, the directory containing Zeek-extracted files will be served over HTTP at ./extracted-files/ (e.g., https://localhost/extracted-files/ if you are connecting locally)

• EXTRACTED_FILE_HTTP_SERVER_ENCRYPT – if set to true, those Zeek-extracted files will be AES-256-CBC-encrypted in an openssl enc-compatible format (e.g., openssl enc -aes-256-cbc -d -in example.exe.encrypted -out example.exe)

• EXTRACTED_FILE_HTTP_SERVER_KEY – specifies the AES-256-CBC decryption password for encrypted Zeek-extracted files; used in conjunction with EXTRACTED_FILE_HTTP_SERVER_ENCRYPT

The encryption is more for safety's sake than anything (as the files may contain live malware). It's a very no-frills HTTP server. It's disabled by default.