jeffxue / web-log-parser Goto Github PK

An open source analysis web log tool

Python 35.82% HTML 64.18%

web-log-parser's Introduction

web-log-parser

web-log-parser is an open source analysis web log tool, developed in python language, with flexible log format configuration

中文说明文档

Example

Instructions

store the logs files in the data directory ./data
modify config.ini
install requirements pip install -r requirements.txt
start the analysis in bin dircd ./bin && python start.py
check html result in result dir result/index.html

Configuration

Format

log-pattern: regular expression for parsing logs
log-format: log content format, corresponding log-pattern match. The following configuration items are supported by default.：
- ip： request IP
- real_ip： user real IP. If this option is configured, ip configuration will be ignored, and real_ip will be used for statistics.
- datetime： request access time
- url： request url
- method： request method
- protocol： request protocol
- cost: request cost

Filter

support_method: supported request method, otherwise no statistics
is_with_parameters: enable flag for statistics url parameter，By default, the parameters in the URL will be converted, such as? Key = 123 when converting to key = {key}
urls_most_number: the maximum number of URLs separately counted
custom_parameters: configure special parameter key-value pairs, separated by commas, when configured t = {timeStamp}, the value of t key will be replaced with a fixed {timeStamp}
fixed_parameter_keys: configure special parameter key values, separated by commas, when configured key = 123, will not be replaced with key = {key}
ignore_urls: urls will be ignore
static-file: file suffixes will be ignore

Report

language: the language use in the report
- chinese
- english
second_line_flag: enable flags for PV curve graph per second
cost_time_flag: enable flags for response time interval proportion map and response time distribution map

GoAccess

goaccess_flag: whether to get GoAccess analysis results, set to 1: Get, 0: Do not get (requires GoAccess installed)
time-format: time format, for GoAccess
date-format: date format, for GoAccess
goaccess-log-format: log format，for GoAccess

web-log-parser's People

Contributors

Stargazers

Watchers

Forkers

sdgdsffdsfff yinwer81 mypythontest fralychen michaelwayneliu popmedd zbaolong shentao17792 haochenxiao666 logan0709 v1cker f8syw5v harry1080 xuacker 5up3rc bernytt lovebair2022 kevinkaiqian c0010 vsery able-fan junjun2712 basecloudwaf sry309 zhanglei x0james liu0tufang 8589 ith4cker jasonlo-tw jjsmida shuoshuoc luckxiao520 xiaoq1z 5icorgi hacker-one nsdown luandaoluan antime difa12321 miumiug arryboom cnwpdb jimsonzhang darknight-666 squallgold xiaoyaodashao mia0sh3n silencezxh a290013130 ch3ngxuyuan llxxwww jiandandan pckey lp008 billdegithub jeromeji northolympiclibrarysystem 5l1v3r1 xalxl cwxsss happyqingye a0xpg ly8008love luoam jojoli123 lv1y2 sandyddlong swiftzl blockchainguard xiaofengtongxue warren-jace taomingming weiwhy pyking icysun lightning-zb jingqiangliu daveqqq xianglinli-soc zerone0x01 wslxy rocker9527 yemoli kangd1w2 sherlockzm want2live233 gwazbc pingxcpost bahaahassanieh cc-ship luozaicc bettydong0806 lw320yao yufeixuan sjhgithub fireskyno1 jzmrexu1s you920 akiyamaliet

web-log-parser's Issues

运行报错

2022-06-08 14:07:15.899174 Start parse file : access_http.log
Traceback (most recent call last):
File "/opt/log_analyse/web-log-parser-master/bin/start.py", line 381, in main
parse_log_file(target_file, log_format)
File "/opt/log_analyse/web-log-parser-master/bin/start.py", line 232, in parse_log_file
response_avg = int(pv / len(set(times)))
ZeroDivisionError: division by zero

最大处理消息时间变成了IP

如题

encoding

C:\Users\Shane\source\repos\web-log-parser\bin>python start.py
Traceback (most recent call last):
File "start.py", line 12, in
from config import config
File "C:\Users\Shane\source\repos\web-log-parser\bin\config.py", line 54, in
config = Config('../conf/config.ini')
File "C:\Users\Shane\source\repos\web-log-parser\bin\config.py", line 11, in init
all_config.readfp(cfg_file)
File "C:\Python38\lib\configparser.py", line 763, in readfp
self.read_file(fp, source=filename)
File "C:\Python38\lib\configparser.py", line 718, in read_file
self._read(f, source)
File "C:\Python38\lib\configparser.py", line 1017, in _read
for lineno, line in enumerate(fp, start=1):
File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 493: character maps to

尝试分析nginx日志遇到错误

大神您好，尝试使用您的日志分析器，把access.log放到data下运行 python start.py遇到下列错误
[root@localhost bin]# python start.py
2021-04-30 15:59:00.200930 Start parse file : access.log
Traceback (most recent call last):
File "/home/web-log-parser-master/bin/start.py", line 381, in main
parse_log_file(target_file, log_format)
File "/home/web-log-parser-master/bin/start.py", line 232, in parse_log_file
response_avg = int(pv / len(set(times)))
ZeroDivisionError: division by zero

问一下是哪里没有设置呢？谢谢您。

能否增加一些攻击日志分析功能？

某些URL解析出现问题

245 /app/index.jsp&client=weixin&deviceId=&userId=&companyName=%E6%98%93%E8%AE%A2%E8%B4%A7 GET HTTP/1.0

253 /app/index.jsp?client={client}&companyName={companyName} GET &from=groupmessage&isappinstalled=0

254 /app/index.jsp?client={client}&companyName={companyName} GET +
255 /app/index.jsp?client={client}&companyName={companyName} GET [