molu8bits / modsecurity-parser Goto Github PK
View Code? Open in Web Editor NEWmodsecurity audit log analyser and parser
License: Apache License 2.0
modsecurity audit log analyser and parser
License: Apache License 2.0
Hi! I am using ModSecurity 3.0.9 + Nginx and I get this error:
ValueError: time data '19/May/2023:00:21:35 +0500' does not match format '%d/%b/%Y:%H:%M:%S %z'
It seems like the mask is completely suitable, but still something is wrong.
I'll attach part of the log.
1.txt
Haven't seen the log files for previous version but ModSecurity 3.0 Audit logs take the following pattern:
---PAbyO0H9---A--
This is different from the expected patter in the modsecurity-parser.py
# modsec_patterns
a_pattern = re.compile('^--\w{6,10}-A--$')
z_pattern = re.compile('^--\w{6,10}-Z--$')
Is it possible to generate rsyslog compatible log file with json format so that I can send that to any remote log server like kibana, graylog ?
On windows, log file can be handled through nxlog, On Linux (Ubuntu 22.04 ) rsyslog handles log file and can easily be forwared to graylog server.
The log file format can be json, for example, each log file seperated by blank line.
{ "time": "15/Jan/2024:00:01:32 +0530", "transaction_id": "16573246656640753946", "remote_address": "152.32.153.53",
"request": {
"request_line": "GET / HTTP/1.1",
"headers": {
"Host": "192.168.5.99",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0"
}
}
"response": "None",
"audit_data": {
"messages": [
"Message: Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host. [file \"C:\\/Program Files/ModSecurity IIS/owasp_crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"810\"] [id \"920350\"] [rev \"2\"] [msg \"Host header is a numeric IP address\"] [data \"192.168.5.99\"] [severity \"WARNING\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"9\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST\"] [tag \"WASCTC/WASC-21\"] [tag \"OWASP_TOP_10/A7\"] [tag \"PCI/6.5.10\"]"]
}
}
{ "time": "15/Jan/2024:00:01:35 +0530", "transaction_id": "16573246656640753979"}
I've used modsecurity_parser.py to analyse a 27gb modsec_audit.log file with over 4.5million modsec events spread over a total of 171 million rows.
the issue is that it's giving no insight on the rules that are passed and so i get no intel when running modsec on haproxy enterprise in the Dectection only mode. Is there a way this can be enabled?. Thanks for a great tool!
If matplotlib version is 3.3.1 you get the following deprecation warning:
modsecurity-parser.py:454: MatplotlibDeprecationWarning: normalize=None does not normalize if the sum is less than 1 but this behavior is deprecated since 3.3 until two minor releases later. After the deprecation period the default value will be normalize=True. To prevent normalization pass normalize=False
patches, texts, autotexts = plt.pie(intercepted_cnt_top10.values(), autopct='%1.1f%%', shadow=True, startangle=90, radius=1.0)
I've read that you're using an older version of that python lib, but in case you're interested in supporting newer versions you can update your code
I get the following error when I try to analyse my modsec audit log:
Traceback (most recent call last):
File "modsecurity-parser.py", line 588, in <module>
json_modsec_entry = modsecLog2Info(modsec_entry)
File "modsecurity-parser.py", line 499, in modsecLog2Info
modsec_f_headers = dict(map(lambda s: [s, '-'] if len(s.split(': ')) == 1 else s.split(': '), modsec_f[1:-1]))
ValueError: dictionary update sequence element #8 has length 3; 2 is required
Can't install matplotlib==3.3.2 with python 3.11. PIP thows error.
pip3 install matplotlib works but it thows this error,
File "C:\Users\Administrator\Downloads\modsecurity-parser-master\modsecurity_parser.py", line 864, in <module>
output_with_graphs = modsec_view_graphs(modsec_entries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\modsecurity-parser-master\modsecurity_parser.py", line 565, in modsec_view_graphs
plt.subplot(ax21)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\pyplot.py", line 1323, in subplot
key = SubplotSpec._from_subplot_args(fig, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\gridspec.py", line 573, in _from_subplot_args
raise ValueError(
ValueError: Single argument to subplot must be a three-digit integer, not <Axes: >
Edit: Solved uninstalling Python 3.11 and installing Python 3.8.11
hello, i am trying to see what your script does, but i am now getting:
TypeError: 'set' object does not support indexing
Traceback (most recent call last):
File "modsecurity-parser.py", line 683, in <module>
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "modsecurity-parser.py", line 434, in modsecViewGraphs
ex.plot(ax=ax1, kind='bar', title=plot_title, stacked=True, color={'purple', 'red'}, fontsize=7, rot=45)
File "/usr/lib/python3/dist-packages/pandas/tools/plotting.py", line 3671, in __call__
sort_columns=sort_columns, **kwds)
File "/usr/lib/python3/dist-packages/pandas/tools/plotting.py", line 2556, in plot_frame
**kwds)
File "/usr/lib/python3/dist-packages/pandas/tools/plotting.py", line 2384, in _plot
plot_obj.generate()
File "/usr/lib/python3/dist-packages/pandas/tools/plotting.py", line 987, in generate
self._make_plot()
File "/usr/lib/python3/dist-packages/pandas/tools/plotting.py", line 1890, in _make_plot
kwds['color'] = colors[i % ncolors]
TypeError: 'set' object does not support indexing
i am not 100% sure that i installed all required python3.5 modules on my ubuntu16.04 (if you know exact names of them per apt installer it would be nice)
apt-get install python3-openpyxl python3-numexpr python3-pandas python3-matplotlib
thank you for any info, stan
$ diff -N0u run.sh.orig run.sh
--- run.sh.orig 2024-01-27 15:15:45.961138946 +0100
+++ run.sh 2024-01-27 15:16:03.588852800 +0100
@@ -26,2 +26,2 @@
Hi,
I tried to use it on ModSecurity Version 3.
inputFileName : /var/log/modsec_audit.log
---------- modsec_audit events processed: 249 ----------
---------- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID : 0 ----------
modsecurity-parser.py:437: MatplotlibDeprecationWarning: Passing non-integers as three-element position specification is deprecated since 3.3 and will be removed two minor releases later.
plt.subplot(ax21)
Traceback (most recent call last):
File "modsecurity-parser.py", line 689, in
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "modsecurity-parser.py", line 437, in modsecViewGraphs
plt.subplot(ax21)
File "/usr/local/lib/python3.8/dist-packages/matplotlib/pyplot.py", line 1272, in subplot
key = SubplotSpec._from_subplot_args(fig, args)
File "/usr/local/lib/python3.8/dist-packages/matplotlib/gridspec.py", line 632, in _from_subplot_args
raise ValueError(
ValueError: Single argument to subplot must be a three-digit integer, not AxesSubplot(0.125,0.420962;0.149038x0.148077)
Hy,
've compile libmodsecurity3 (3.0.9) and apache-connector.
How resolve it?
----- modsec_audit events processed: 68 -----
----- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID: 0 -----
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Team,
Excellent tool. For some reason when I want to generate the image or the excel gives me an error.
Is it a compatibility issue or does the log have something difficult to partial?
What is the level of debugging desired by the tool?
command line : python3 modsecurity-parser.py -x 1.xls -g 1.png --jsonaudit --jsononeperline --version3 -f modsec_audit.log
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses 'remote_address'
Exception in TOP 20 rule hits 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Traceback (most recent call last):
File "modsecurity-parser.py", line 689, in
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "modsecurity-parser.py", line 359, in modsecViewGraphs
event_times1 = np_event_time_action[:, 0]
regards
First time clone, install pre-requisites and run. It simply throws a stack trace:
[shaund@peregrine modsecurity-parser]$ python3 modsecurity_parser.py -f /home/shaund/tmp/mod_sec/modsec_audit.log
input_filename: /home/shaund/tmp/mod_sec/modsec_audit.log
----- modsec_audit events processed: 990 -----
----- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID: 0 -----
Traceback (most recent call last):
File "/home/shaund/Software/modsecurity-parser/modsecurity_parser.py", line 864, in
output_with_graphs = modsec_view_graphs(modsec_entries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shaund/Software/modsecurity-parser/modsecurity_parser.py", line 565, in modsec_view_graphs
plt.subplot(ax21)
File "/usr/lib64/python3.11/site-packages/matplotlib/pyplot.py", line 1323, in subplot
key = SubplotSpec._from_subplot_args(fig, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/site-packages/matplotlib/gridspec.py", line 575, in _from_subplot_args
raise ValueError(
ValueError: Single argument to subplot must be a three-digit integer, not <Axes: >
Hi,
I just updated to the most recent version on my server (running Ubuntu 18.04 LTS). Unfortunatelly I get the following error:
Traceback (most recent call last):
File "/opt/modsecurity-parser-master/modsecurity-parser.py", line 689, in
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "/opt/modsecurity-parser-master/modsecurity-parser.py", line 454, in modsecViewGraphs
patches, texts, autotexts = plt.pie(intercepted_cnt_top10.values(), autopct='%1.1f%%', shadow=True, startangle=90, radius=1.0, normalize=True)
TypeError: pie() got an unexpected keyword argument 'normalize'
Any ideas?
hi,
i am facing issue that when i am running python3 modsecurity-parser.py -f /var/log/modsec_audit.log i am getting below error. please help to resolve the same and attached is the sample logs which i am getting
inputFileName : /var/log/modsec_audit.log
No modsecurity events found in the specified file
Hi , What is the perfect Parts string for your test?
This is needed to config more feauters e.g. Choice file outputs.... or max events...or anything..
Hi Guys,
run this script, met issue(as title), can help? thanks
my modsecurity(3.0.9) logs format:
{"transaction":{"client_ip":"10.200.101.16","time_stamp":"Thu May 11 02:13:58 2023","server_id":"6c63a629cf8ef75665cbe6abb55daaf9d4fa7b2b","client_port":33042,"host_ip":"172.22.0.2","host_port":80,"unique_id":"168377123884.428748","request":{"method":"GET","http_version":1.1,"uri":"/pub/","headers":{"Connection":"Keep-Alive","Host":"10.200.101.18","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"}},"response":{"body":"<!--\n\n Copyright © 2016-2023 The Thingsboard Authors\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n-->\n<!doctype html>\n<html lang=\"en\" style=\"width: 100%; height: 100%;\">\n<head>\n <meta charset=\"utf-8\">\n <title>ThingsBoard</title>\n <base href=\"/\">\n\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n <link rel=\"icon\" type=\"image/x-icon\" href=\"thingsboard.ico\">\n <link rel=\"preload\" href=\"assets/fonts/MaterialIcons-Regular.ttf\" as=\"font\" type=\"font/ttf\" crossorigin=\"anonymous\"/>\n <link rel=\"stylesheet\" href=\"assets/fonts/material-icons.css\"/>\n <style type=\"text/css\">\n\n body, html {\n height: 100%;\n overflow: hidden;\n background-color: #eee;\n }\n\n .tb-loading-spinner {\n margin: auto;\n z-index: 1;\n position: absolute;\n top: 0;\n bottom: 0;\n left: 0;\n right: 0;\n width: 136px;\n height: 30px;\n text-align: center;\n }\n\n .tb-loading-spinner > div {\n width: 30px;\n height: 30px;\n margin-right: 10px;\n background-color: rgb(43,160,199);\n\n border-radius: 100%;\n display: inline-block;\n -webkit-animation: tb-bouncedelay 1.4s infinite ease-in-out both;\n -moz-animation: tb-bouncedelay 1.4s infinite ease-in-out both;\n animation: tb-bouncedelay 1.4s infinite ease-in-out both;\n }\n\n .tb-loading-spinner .tb-bounce1 {\n -webkit-animation-delay: -0.32s;\n -moz-animation-delay: -0.32s;\n animation-delay: -0.32s;\n }\n\n .tb-loading-spinner .tb-bounce2 {\n -webkit-animation-delay: -0.16s;\n -moz-animation-delay: -0.16s;\n animation-delay: -0.16s;\n }\n\n @-webkit-keyframes tb-bouncedelay {\n 0%, 80%, 100% { -webkit-transform: scale(0) }\n 40% { -webkit-transform: scale(1.0) }\n }\n\n @-moz-keyframes tb-bouncedelay {\n 0%, 80%, 100% { -moz-transform: scale(0) }\n 40% { -moz-transform: scale(1.0) }\n }\n\n @keyframes tb-bouncedelay {\n 0%, 80%, 100% {\n -webkit-transform: scale(0);\n -moz-transform: scale(0);\n transform: scale(0);\n } 40% {\n -webkit-transform: scale(1.0);\n -moz-transform: scale(1.0);\n transform: scale(1.0);\n }\n }\n\n </style>\n<link rel=\"stylesheet\" href=\"styles.10895964a4a3aa21d65a.css\"></head>\n<body class=\"tb-default\">\n <tb-root></tb-root>\n <div id=\"tb-loading-spinner\" class=\"tb-loading-spinner\">\n <div class=\"tb-bounce1\"></div>\n <div class=\"tb-bounce2\"></div>\n <div class=\"tb-bounce3\"></div>\n </div>\n<script src=\"runtime.286f6982886cb90bbe7a.js\" defer></script><script src=\"polyfills.e2023dc347cde42f7c8d.js\" defer></script><script src=\"scripts.d93c5ee41f6da54bd100.js\" defer></script><script src=\"vendor.3f3611f892c51888617d.js\" defer></script><script src=\"main.ed39576ce9947da26638.js\" defer></script></body>\n</html>\n","http_code":200,"headers":{"Accept-Ranges":"bytes","Vary":"Origin","Vary":"Access-Control-Request-Method","Vary":"Access-Control-Request-Headers","Connection":"keep-alive","Last-Modified":"Tue, 07 Feb 2023 14:18:35 GMT","Last-Modified":"Tue, 07 Feb 2023 14:18:35 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Content-Type":"text/html;charset=UTF-8","Content-Length":"3345","Date":"Thu, 11 May 2023 02:13:58 GMT","Server":"nginx/1.22.1","X-Content-Type-Options":"nosniff","X-Content-Type-Options":"nosniff","X-XSS-Protection":"1; mode=block","Pragma":"no-cache","Content-Language":"en","Expires":"0","X-Frame-Options":"SAMEORIGIN"}},"producer":{"modsecurity":"ModSecurity v3.0.9 (Linux)","connector":"ModSecurity-nginx v1.0.3","secrules_engine":"Enabled","components":["OWASP_CRS/4.0.0-rc1\""]},"messages":[{"message":"Host header is a numeric IP address","details":{"match":"Matched \"Operator `Rx' with parameter `(?:^([\\d.]+|\\[[\\da-f:]+\\]|[\\da-f:]+)(:[\\d]+)?$)' against variable `REQUEST_HEADERS:Host' (Value: `10.200.101.18' )","reference":"o0,13o0,13v48,13","ruleId":"920350","file":"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf","lineNumber":"744","data":"10.200.101.18","severity":"4","ver":"OWASP_CRS/4.0.0-rc1","rev":"","tags":["modsecurity","application-multi","language-multi","platform-multi","attack-protocol","paranoia-level/1","OWASP_CRS","capec/1000/210/272","PCI/6.5.10"],"maturity":"0","accuracy":"0"}}]}}
Hi,
the .json output is not valid, tested on https://jsonlint.com/?code= on sample file modsec_audit2_2018-06-11_20-16-27.json report the following problem:
Error: Parse error on line 61: ...de": "ENABLED" }} { "transaction": { ---------------------^ Expecting 'EOF', '}', ',', ']', got '{'
Hello =)
I noticed that if difference in time zone is specified with "--" the program gives an error.
python3 modsecurity-parser.py -g LOL.png -f modsec_audit.log inputFileName : modsec_audit.log ---------- modsec_audit events processed: 9577 ---------- ---------- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID : 0 ---------- Traceback (most recent call last): File "modsecurity-parser.py", line 689, in <module> outputWithGraphs = modsecViewGraphs(modsec_entries) File "modsecurity-parser.py", line 360, in modsecViewGraphs event_times = list(map(lambda x: datetime.strptime(x, LOG_TIMESTAMP_FORMAT).replace(tzinfo=None), event_times1)) File "modsecurity-parser.py", line 360, in <lambda> event_times = list(map(lambda x: datetime.strptime(x, LOG_TIMESTAMP_FORMAT).replace(tzinfo=None), event_times1)) File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime tt, fraction = _strptime(data_string, format) File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime (data_string, format)) ValueError: time data '03/Mar/2020:08:52:27 --0500' does not match format '%d/%b/%Y:%H:%M:%S %z'
Not sure if this something wrong with my settings. However quick work-around is to replace the --0500 to +0500 in each line for time and everything looks good.
Also if there are around 10000 events the program exists with the killed error for me.
python3 modsecurity-parser.py -x auditlog_1 -f prefix_aa inputFileName : prefix_aa ---------- modsec_audit events processed: 12430 ---------- ---------- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID : 0 ---------- Killed
I did split the audit log, and this resolves the issue. Suspect this is related to the 1gb of ram on my virtual machine =)
This is just FYI, not sure this is something you will be interested in correcting.
Thanks a lot anyway, this program saved me a lot of time by helping me to analyze the mod_security audit logs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.