userjack6880 / open-report-parser Goto Github PK

Installing the listed dependencies, especially libfile-mimeinfo-perl, on Ubuntu, will pull in 170 MB of dependencies, including Wayland and X11 desktop libraries. Is this really required for the DMARC parser? Or is there an alternative?

# apt-get install libfile-mimeinfo-perl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  adwaita-icon-theme at-spi2-core dconf-gsettings-backend dconf-service gsettings-desktop-schemas gtk-update-icon-cache humanity-icon-theme
  libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libcairo-gobject2 libcolord2 libdconf1 libepoxy0 libfile-basedir-perl
  libfile-desktopentry-perl libgdk-pixbuf-2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgtk-3-0 libgtk-3-bin libgtk-3-common libgtkd-3-0
  libipc-system-simple-perl libllvm11 libphobos2-ldc-shared98 librsvg2-2 librsvg2-common libvte-2.91-0 libvte-2.91-common libvted-3-0
  libwayland-client0 libwayland-cursor0 libwayland-egl1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon0
  libxrandr2 libxtst6 session-migration tilix tilix-common ubuntu-mono x11-common
Suggested packages:
  colord gvfs librsvg2-bin python-nautilus
The following NEW packages will be installed:
  adwaita-icon-theme at-spi2-core dconf-gsettings-backend dconf-service gsettings-desktop-schemas gtk-update-icon-cache humanity-icon-theme
  libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libcairo-gobject2 libcolord2 libdconf1 libepoxy0 libfile-basedir-perl
  libfile-desktopentry-perl libfile-mimeinfo-perl libgdk-pixbuf-2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgtk-3-0 libgtk-3-bin
  libgtk-3-common libgtkd-3-0 libipc-system-simple-perl libllvm11 libphobos2-ldc-shared98 librsvg2-2 librsvg2-common libvte-2.91-0
  libvte-2.91-common libvted-3-0 libwayland-client0 libwayland-cursor0 libwayland-egl1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6
  libxinerama1 libxkbcommon0 libxrandr2 libxtst6 session-migration tilix tilix-common ubuntu-mono x11-common
0 upgraded, 50 newly installed, 0 to remove and 22 not upgraded.
Need to get 36.9 MB of archives.
After this operation, 170 MB of additional disk space will be used.

Per request of a specific client - Oauth2 support to work with M365 Modern Auth in Exchange.

Hi I tried to use m365 modern Auth but it didn't connect I use this command -i and -d flag and I created azure app registration for imap. also I used auth2 uri and client ID and my debug info is below

Can you help me? I think it will help with all questions.

--- DEBUG ---
  Open Report Parser
  Version 0 Alpha 5
-------------
Open Report Parser DEBUG ENABLED
-- Script Options --

Report Source:   0
(0: IMAP, 1: Message, 2: XML, 3: MBOX, 4: ZIP, 5: JSON)
Show Processed:   0
Delete Reports:   0
Delete Failed:    0
Replace Reports:  0
DMARC Only:       1
(0: DMARC\TLS, 1: DMARC Only, -1: TLS Only)

-- Database Options --

DB Type:          mysql
DB Name:          dmarc
DB User:          dmarc
DB Host/Port:     localhost:3306
DB TX Support:    1

Max XML Size:     500000
Max JSON Size:    500000
Compress XML:     0
Compress JSON:    0

-- IMAP Options --

IMAP Server:      ps.outlook.com
IMAP Port:        143
TLS:              1
SSL:              0
TLS Verify:       1
IMAP User:        [email protected]
IMAP Ignore Err:  0
IMAP Auth:        oauth2
Oauth2 URI:       xxxxxxxxxxxxxxxxxxxxx
OAuth2 Client ID: xxxxxxxxxxxxxxxxxxxxx
DMARC Folders:
   Reports:       Inbox
   Processed:     Inbox.Processed
TLS Folders:
   Reports:       tls
   Processed:     tls.Processed
----



--- DEBUG ---
  use tls with verify servercert.
-------------


--- DEBUG ---
  connection to ps.outlook.com with Ssl => 0, User => [email protected], Ignoresizeerrors => 0
-------------
Started at Tue Nov 21 15:59:21 2023
Using Mail::IMAPClient version 3.42 on perl 5.032001
Connecting with IO::Socket::IP PeerAddr ps.outlook.com PeerPort 143 Proto tcp Timeout 600 Debug 1 SSL_verify_mode 1
Connected to ps.outlook.com
Read:   * OK The Microsoft Exchange IMAP4 service is ready. [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==]
Sending: 1 STARTTLS
Sent 12 bytes
Read:   1 OK Begin TLS negotiation now.


--- DEBUG ---
  using oauth2
-------------
no token found, requesting
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at lib/OAuth.pm line 172.

Originally posted by @Firat-Gulec in #9 (comment)

DBI::mysql requires MySQLv8 since perl5-dbi/DBD-mysql@7e1ce07 which can be annoying when you use MariaDB (or another MySQL version).

It would be nice if the DBI type can support MariaDB?

I did a quick test, it seems dbx_mysql.pl can stay unchanged but the report-parser.pl script needs some refactor.

Per Issue #112 on the pre-fork version, it's been pointed out that Duplicate ReportIDs are possible. It is possible to fix, and a solution is provided, though the implementation is not quite what we need. Adapt the solution to this project and include a check to automatically update the index in both Maria and PostgreSQL (index checks would be different for the two versions). Perhaps introduce an update script instead?

We recently moved over from the techsneeze parser/reporting interface due to consistent errors when parsing reports.

We are continuing to get report parse errors.

I have included examples below. I have asked this as a question as it may be a problem on our end so any advice, tips, or suggestions to resolve are appreciated in advance. Many thanks!

Open Report Parser: wp.pl: 1688599419.913278356: source_ip is empty. Skipped.
Open Report Parser: wp.pl: 1688599419.913278356: Cannot add records to rptrecord. Rolling back DB transaction.
Open Report Parser: Skipping IMAP message with UID #51454 due to database errors.
Open Report Parser: Failed to ungzip ZIP file (</tmp/msg-32518-2.dat>)! 
Open Report Parser: The IMAP message with UID #51471 does not seem to contain a valid DMARC report. Skipped.
Open Report Parser: Failed to ungzip ZIP file (</tmp/msg-32518-3.dat>)! 
Open Report Parser: The IMAP message with UID #51539 does not seem to contain a valid DMARC report. Skipped.

Describe the bug
If you have the tls report being served by an IPv6 server, then the following lines don't work:
elsif($nip = inet_pton(AF_INET6, $ip)) {
$ipval = $dbx{to_hex_string}{$nip}; <== this one
$iptype = "ip6";
}

I'm not a Perl developer at all, but should it be parenthesis instead of brackets around $nip ?

The error message that is sent is the following:
Not a HASH reference at /root/Open-Report-Parser/report-parser.pl line 1651.

To Reproduce
See above

Expected behavior
This should just work out of the box

Screenshots

Software Version
[X] Version 0 Alpha 5
[] Version 0 Alpha 4
[] Version 0 Alpha 3
[] Version 0 Alpha 2

Server (please complete the following information):

• Ubuntu LTS 20.04
• Perl Version 5.34.0
• MariaDB 10.6.16

I started to use the dmarc analyser and am happy. I am wondering what is this tls report parsing for? I have not found any information about it. Can anyone enlighten me?

libperl-json is missing in depencies for debian/ubuntu

Describe the enhancement request

PostgreSQL functionality is assumed, but needs to be tested.

Describe the bug
I ran the script for the 1st time and it produced the following error:
Use of uninitialized value in concatenation (.) or string at ./report-parser.pl line 448.

Line 448 is this:
printInfo("Processing ". $imap->message_count($imapdmarcfolder)." messages in folder <$imapdmarcfolder>.") if $dmarc_only >= 0;
It's 1 line off of the original script because I added this line in the header since I'm running it in cPanel:
use cPanelUserConfig;

My question is: can this error happen because the mailbox is still empty (because it currently is)? Or should I look for other causes of this error?

Software Version
[] Version 0 Alpha 5

Server (please complete the following information):

• Server OS: CloudLinux v8.9.0
• Perl Version: 5.26.3
• SQL Variant and Version: MariaDB 10.6.17

Hi,
we are trying to setup Open-DMARC-Analyzer with Open-Report-Parser.
When the Report Parser loads the DMARC Reports from our Exchange 2019, we get following error with hint to use IgnoreSizeErrors option, but we dont know how to enable or to set:

Read:   * 1 FETCH (RFC822.SIZE 19140 UID 133)
Read:   9 OK FETCH completed.
ERROR: message_string() expected 19140 bytes but received 5817 you may need the IgnoreSizeErrors option at /usr/share/perl5/Mail/IMAPClient.pm line 891.
        Mail::IMAPClient::message_string(Mail::IMAPClient=HASH(0x5650208d6108), 133) called at ./report-parser.pl line 618
        main::processIMAP(Mail::IMAPClient=HASH(0x5650208d6108), 0, "Inbox/DMARC-RUA", "Inbox/DMARC-RUA/gelesen", "Inbox/DMARC-RUA/Fehler", 0) called at ./report-parser.pl line 456

----------------------------------------------------------------
Processing IMAP message with UID #133
----------------------------------------------------------------
Type: 1
Use of uninitialized value $filecontent in concatenation (.) or string at ./report-parser.pl line 703.
FileContent:
MSG: IMAP message with UID #133
----------------------------------------------------------------


--- DEBUG ---
  getting XML from message
-------------
Use of uninitialized value $message in substitution (s///) at ./report-parser.pl line 761.
parse_data: No data passed at ./report-parser.pl line 766.

Thanks Marc

Posit your question!
I am trying to get this to work with an account for our Google Workspace account. I am not very experienced with OAuth2. I have a Client ID from another app, but I am not sure what URI to use. I have made an educated guess that it is

https://accounts.google.com/o/oauth2/v2/auth

When I try to connect, I get this error:

no token found, requesting
Wide character in subroutine entry at lib/OAuth.pm line 172.

Any suggestions?

So, I have the basic functions working by reading the report files from a directory. Now I want to automate reading the files and insert the data into the database. I have so far been executing ./report-parser.pl directly from the directory it is installed into.

When setting up cron, I assumed I would be able to use the fully qualified path to report-parser.pl. But when I execute it, it complains that it cannot find OAuth.pm. I note that OAuth.pm is actually located in the /lib subdirectory.

Do I need to installed OAuth via cpan or something so that it's findable from anywhere on the sytem?

Describe the bug
When running report-parser.pl on a debian 12 base image I got an error with missing JSON in @INC

To Reproduce
Steps to reproduce the behavior:

1. create a docker container from php:8.3-apache (debian 12 based) and copy open-report-parser with dependencies listed in the README.md
2. add a configuration file and run report-parser.pl -i --info

Expected behavior
No error message and report-parser able to parse emails

Software Version
[X] Version 0 Alpha 5
[] Version 0 Alpha 4
[] Version 0 Alpha 3
[] Version 0 Alpha 2

Server (please complete the following information):

• Server OS: debian 12
• Perl Version: 5.36.0
• SQL Variant and Version: MariaDB 10.11 (not revelant)

Additional context

You can fix the problem adding libjson-perl in the dependencies! You only need to patch README.md

Having a docker image would be useful to easily set this up and get it running periodically.

Hi,

I'm trying to connect to the imap server using oauth2 but i receive de following message, i'm using postgres 13.11:

  Open Report Parser
  Version 0 Alpha 5
-------------
Open Report Parser DEBUG ENABLED
-- Script Options --

Report Source:   0
(0: IMAP, 1: Message, 2: XML, 3: MBOX, 4: ZIP, 5: JSON)
Show Processed:   0
Delete Reports:   0
Delete Failed:    0
Replace Reports:  0
DMARC Only:       1
(0: DMARC\TLS, 1: DMARC Only, -1: TLS Only)

-- Database Options --

DB Type:          postgres
DB Name:          dmarc
DB User:          dmarc
DB Host/Port:     localhost:5432
DB TX Support:    1

Max XML Size:     50000
Max JSON Size:    50000
Compress XML:     0
Compress JSON:    0

-- IMAP Options --

IMAP Server:      imap.gmail.com
IMAP Port:        993
TLS:              0
SSL:              1
TLS Verify:       0
IMAP User:        [email protected]
IMAP Ignore Err:  0
IMAP Auth:        oauth2
Oauth2 URI:       https://accounts.google.com/o/oauth2/v2/auth
OAuth2 Client ID: xxxxxxx
DMARC Folders:
   Reports:       inbox
TLS Folders:
   Reports:       tls
----



--- DEBUG ---
  using ssl without verify servercert.
-------------


--- DEBUG ---
  connection to imap.gmail.com with Ssl => 1, User => [email protected], Ignoresizeerrors => 0
-------------
Started at Wed Mar 13 14:49:35 2024
Using Mail::IMAPClient version 3.43 on perl 5.026003
Connecting with IO::Socket::SSL PeerAddr imap.gmail.com PeerPort 993 Proto tcp Timeout 600 Debug 1 SSL_verify_mode 0
Connected to imap.gmail.com
Read:   * OK Gimap ready for requests from xxx.xxx.xxx.xxx xxxxxxxxxx


--- DEBUG ---
  using oauth2
-------------
DBD::Pg::st execute failed: ERROR:  relation "oauth" does not exist
LINE 1: ...ken, UNIX_TIMESTAMP(expire) AS expire, valid FROM oauth WHER...
                                                             ^ at lib/OAuth.pm line 63.
DBD::Pg::st fetchrow_hashref failed: no statement executing at lib/OAuth.pm line 64.
no token found, requesting
Wide character in subroutine entry at lib/OAuth.pm line 172.

Any suggestion?

Part of changing the name to simply "Open Report Parser" is to explore the possibility of incorporating TLS reports into this script. See the feature request "[Feature Request]TLS Reporting #74" from Open DMARC Analyzer.

Describe the bug
The script can not connect to a database because of a wrong mysql.sock location

To Reproduce
I was using bitnami lamp which has mariadb installed by default and it has a different location for mysql.sock.
Start the script without changing it manually from report.parser.pl

Expected behavior
Script should have ran without any problems.

Software Version
[X] Version 0 Alpha 5
[] Version 0 Alpha 4
[] Version 0 Alpha 3
[] Version 0 Alpha 2

Server (please complete the following information):

• Debian GNU/Linux 11 (bullseye) x86_64 AWS bitnami lamp
• Perl Version [e.g. 5.32
• MariaDB 11.2.3

Additional context
I fixed it by adding changing line 367 in report-parser.pl:

Could be nice to have some kind of option to change that in the config.

Also changing MySQL my.cnf did not fix this issue.

OK, so I am switching to reading XML files directly rather than from emails. (I have a semi automated system for doing this which I will document once I have it polished). But I can't get any of my reports to be parsed. All come up with the above error.

I've added the -d flag for more detail.

`baccount@ip-xxx-xxx-xxx-xxx:~/domains/dmarc.baccount.org/public_html/Open-Report-Parser$ ./report-parser.pl -x -d ../../dmarcreports/appstate.xml

--- DEBUG ---
Open Report Parser
Version 0 Alpha 5

Open Report Parser DEBUG ENABLED
-- Script Options --

Report Source: 2
(0: IMAP, 1: Message, 2: XML, 3: MBOX, 4: ZIP, 5: JSON)
Show Processed: 0
Delete Reports: 0
Delete Failed: 0
Replace Reports: 0
DMARC Only: 1
(0: DMARC\TLS, 1: DMARC Only, -1: TLS Only)

-- Database Options --

DB Type: mysql
DB Name: dmarc
DB User: baccount
DB Host/Port: localhost:3306
DB TX Support: 1

Max XML Size: 50000
Max JSON Size: 50000
Compress XML: 0
Compress JSON: 0

-- IMAP Options --

IMAP Server: smtp.gmail.com
IMAP Port: 993
TLS: 0
SSL: 1
TLS Verify: 0
IMAP User: [email protected]
IMAP Ignore Err: 0
IMAP Auth: oauth2
Oauth2 URI: https://accounts.google.com/o/oauth2/v2/auth
OAuth2 Client ID:
DMARC Folders:
Reports: dmarc
TLS Folders:
Reports: tls

Processing xml file <../../dmarcreports/appstate.xml>

Type: 2
FileContent: ?Tێ? }?W?C_?6?W?%??'?!A?N??!??d??)̙?s?(TȞ?+???Fo????????R??g}?F?????????w.D?9?r?????????Ƚx?r????<5??=?5???i?,ÁW0O?ڴd-QU%??u??????J????q???Ac?ynγp?z8kL?X?rlOZfd郑??G?ԧ5????m? uv??sS??p?U??,
?m???4?
?>?W=2O?|????K?r?X?ts??ʉ?H??9?????
?e??Z?;y?v????P??.C?˨??A2Z???|p?Äiɥ?"??٠cڿlb???R???k?Y??Tr?V??kN?<?3?%?????_?]?K??v|?,{|ۉ
AF?Vc*?á%?(­0Ӫ??f/ܔO???79OV??/9@D???x-?~?????5?=?Źq?????٘??{?*??.?SI?yMi?????vK??
MSG: xml file <../../dmarcreports/appstate.xml>

--- DEBUG ---
getting XML from string

Open Report Parser: The xml file <../../dmarcreports/appstate.xml> does not seem to contain a valid DMARC report. Skipped.

--- DEBUG ---
Open Report Parser: Processed 1 messages(s).`

Obviously there's a lot of garbage showing in the FileContent, but it doesn't appear that way when I view it manually. And if I open it in a text editor it shows it's encoded for UTF-8 so I don't think it's an encoding error.

Ideas?