Since I updated Perl from version 5.28 to 5.30 I am getting this error (on FreeBSD 11.3 x64):

The XML found in ZIP file (temp. location: </tmp/msg-93794-1.zip>) does not seem to be valid XML! The IMAP message with UID #413 does not seem to contain a valid DMARC report. Skipped.

I reinstalled perl 5.30,the xml-simple and PerlIO-gzip to no avail.
Running with -d does not give any extra information.

$ ./dmarcts-report-parser.pl -i
...
create tables...
...
Use of uninitialized value in concatenation (.) or string at ./dmarcts-report-parser.pl line 233.

I cannot get the script to connect succesfully to imap.gmail.com:993. I've edited the config file, have configured $imapssl = '1' and I have set $debug = 1 but I only get the following output:

$ ./dmarcts-report-parser.pl -d -i
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 361.
IMAP Failure: Unable to connect to imap.gmail.com:993:  at ./dmarcts-report-parser.pl line 217.

Shouldn't debug mode be a bit more explicit than this?

I'm running on a freshly installed CentOS 7.3 Virtual Machine. I have changed the setting on the GSuite account to "allow less secure apps".

I can connect ok manually from the VM using openssl s_client...

$ openssl s_client -connect imap.gmail.com:993 -crlf
CONNECTED(00000003)
..<snip>..
* OK Gimap ready for requests from 1.2.3.4 b9mb58885590wrb
a1 LOGIN [email protected] password_here
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
a1 OK [email protected] authenticated (Success)
a2 SELECT inbox
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing \*)] Flags permitted.
* OK [UIDVALIDITY 1] UIDs valid.
* 8 EXISTS
* 0 RECENT
* OK [UIDNEXT 13] Predicted next UID.
* OK [HIGHESTMODSEQ 2171]
a2 OK [READ-WRITE] inbox selected. (Success)
a2 LOGOUT
* BYE LOGOUT Requested
a2 OK 73 good day (Success)
read:errno=0

Thanks in advance for any help you can provide

As the title says, if the "source ip" is an IPv6 address is does not get parsed and does not show up in the GUI.

Platform: Centos 7
Mariadb 10.2 and 10.3

Running into this issue.

/opt/dmarc/dmarcts-report-parser.pl -i -d
Adding missing table <rptrecord> to the database.
CREATE TABLE rptrecord (
id int(10) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
serial int(10) unsigned NOT NULL,
ip int(10) unsigned,
ip6 binary(16),
rcount int(10) unsigned NOT NULL,
disposition enum('none','quarantine','reject'),
reason varchar(255),
dkimdomain varchar(255),
dkimresult enum('none','pass','fail','neutral','policy','temperror','permerror'),
spfdomain varchar(255),
spfresult enum('none','neutral','pass','fail','softfail','temperror','permerror','unknown'),
spf_align enum('fail','pass','unknown') NOT NULL,
dkim_align enum('fail','pass','unknown') NOT NULL,
identifier_hfrom varchar(255),
PRIMARY KEY (id), KEY serial (serial,ip), KEY serial6 (serial,ip6)) ;
DBD::mysql::db do failed: Multiple primary key defined at /opt/dmarc/dmarcts-report-parser.pl line 840.

root@ip-172-31-41-213:~/dmarcts-report-parser# ./dmarcts-report-parser.pl -d -i
use tls with verify servercert.
connection to imap.gmail.com:993 with Ssl => 1, User => [email protected], Ignoresizeerrors => 1
Started at Sat Dec 9 15:18:09 2017
Using Mail::IMAPClient version 3.38 on perl 5.022001
Connecting with IO::Socket::SSL PeerAddr imap.gmail.com:993 PeerPort 993 Proto tcp Timeout 600 Debug 1
Connected to imap.gmail.com:993
Read: * OK Gimap ready for requests from 34.21.56.21 p111mb294215025wrb
Sending: 1 STARTTLS
Sent 12 bytes
Read: 1 BAD Unknown command p111mb294215025wrb
ERROR: 1 BAD Unknown command p111mb294215025wrb at /usr/share/perl5/Mail/IMAPClient.pm line 1365.
Mail::IMAPClient::ANON("1 BAD Unknown command p111mb294215025wrb\x{d}\x{a}") called at /usr/share/perl5/Mail/IMAPClient.pm line 1401
Mail::IMAPClient::_get_response(Mail::IMAPClient=HASH(0x20f81b0), 1, undef) called at /usr/share/perl5/Mail/IMAPClient.pm line 1327
Mail::IMAPClient::_imap_command_do(Mail::IMAPClient=HASH(0x20f81b0), "STARTTLS") called at /usr/share/perl5/Mail/IMAPClient.pm line 1226
Mail::IMAPClient::_imap_command(Mail::IMAPClient=HASH(0x20f81b0), "STARTTLS") called at /usr/share/perl5/Mail/IMAPClient.pm line 436
Mail::IMAPClient::starttls(Mail::IMAPClient=HASH(0x20f81b0)) called at /usr/share/perl5/Mail/IMAPClient.pm line 413
Mail::IMAPClient::Socket(Mail::IMAPClient=HASH(0x20f81b0), IO::Socket::SSL=GLOB(0x20f8678)) called at /usr/share/perl5/Mail/IMAPClient.pm line 366
Mail::IMAPClient::connect(Mail::IMAPClient=HASH(0x20f81b0)) called at /usr/share/perl5/Mail/IMAPClient.pm line 314
Mail::IMAPClient::new("Mail::IMAPClient", "Server", "imap.gmail.com:993", "Ssl", 1, "Starttls", ARRAY(0x20f8240), "User", "[email protected]", ...) called at ./dmarcts-report-parser.pl line 234
ERROR: 1 BAD Unknown command p111mb294215025wrb at /usr/share/perl5/Mail/IMAPClient.pm line 1275.
Mail::IMAPClient::_imap_command(Mail::IMAPClient=HASH(0x20f81b0), "STARTTLS") called at /usr/share/perl5/Mail/IMAPClient.pm line 436
Mail::IMAPClient::starttls(Mail::IMAPClient=HASH(0x20f81b0)) called at /usr/share/perl5/Mail/IMAPClient.pm line 413
Mail::IMAPClient::Socket(Mail::IMAPClient=HASH(0x20f81b0), IO::Socket::SSL=GLOB(0x20f8678)) called at /usr/share/perl5/Mail/IMAPClient.pm line 366
Mail::IMAPClient::connect(Mail::IMAPClient=HASH(0x20f81b0)) called at /usr/share/perl5/Mail/IMAPClient.pm line 314
Mail::IMAPClient::new("Mail::IMAPClient", "Server", "imap.gmail.com:993", "Ssl", 1, "Starttls", ARRAY(0x20f8240), "User", "[email protected]", ...) called at ./dmarcts-report-parser.pl line 234
IMAP Failure: 1 BAD Unknown command p111mb294215025wrb at ./dmarcts-report-parser.pl line 234.

Im having issues on first run with it creating the tables. It was able to create rptrecord, but nothing else.

Adding missing table <report> to the database.
DBD::mysql::db do failed: Can't create table `dmar_db`.`report` (errno: 140 "Wrong create options") at ./dmarcts-report-parser.pl line 1025.

Wonder if anyone could provide me an sql dump of the db schema where i could try to manually import it to get more details information on where its failing. Trying to piece it together from the perl source is kind of a mess.

I have followed the instructions in the guide on how to set up the report parser and I'm getting this error.

parse_data: No data passed at ./dmarcts-report-parser.pl line 439.

Looks as though the pearl program has trouble recognizing the XML files that are stored in a ZIP file in my IMAP server email inbox.

Anyone know of / can think of a fix for this problem.

Much appreciated.

Domain seznam.cz generates report emails with text and an attachment with content-type 'multipart/related'.
The script outputs Could not find an embedded ZIP!, as it does not process all parts.
I suppose we just need to change lc $mtype eq "multipart/mixed" to lc $mtype eq "multipart/mixed" or lc $mtype eq "multipart/related", but I can not test it as I currently have no such report emails (outlook converted all of them to "multipart/mixed")

If the DB queries fail but leave the entry in the 'report' table, the subsequent run will move it to the $imapmovefolder as if it's a duplicate.

See pull request #88

This record with both dkim signatures from example.org (fail) and example.com (pass) leads to dkim: pass in db, I guess it should be fail

  <record>
    <row>
      <source_ip>1.2.3.4</source_ip>
      <count>1</count>
      <policy_evaluated>
        <disposition>quarantine</disposition>
        <dkim>fail</dkim>
        <spf>fail</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>example.org</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>example.com</domain>
        <result>pass</result>
        <selector>dkim</selector>
      </dkim>
      <dkim>
        <domain>example.org</domain>
        <result>fail</result>
        <selector>mail</selector>
      </dkim>
      <spf>
        <domain>example.com</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>

Hey,

I got an error when processing.

DBD::mysql::db do failed: Index column size too large. The maximum column size is 767 bytes. at ./dmarcts-report-parser.pl line 861.

as advised i added the bellow part to the mysql config.

innodb_large_prefix	= on
innodb_file_format	= barracuda
innodb_file_per_table	= true

please advise.
Mark

Hello,
I think it would be super amazing if the dmarc report parser could also parse the new TLS-RPT reports.

https://tools.ietf.org/html/rfc8460

Hi,

I am fairly new to DMARC and perl, so when this appear I feel totally lost.
The .conf file looks as this:

$debug = 0;
$delete_reports = 0;

$dbname = 'dmarc_mots_local';
$dbuser = 'dmarc_mots_local';
$dbpass = '4gj47ryO';
$dbhost = 'localhost'; # Set the hostname if we can't connect to the local socket.

$imapserver = 'mailserver:993';
$imapuser = 'user';
$imappass = 'password';
$imapssl = '1'; # If set to 1, remember to change server port to 993.
$imaptls = '1'; # Enabled as the default and best-practice.
$imapreadfolder = 'Inbox';

$imapmovefolder = 'Inbox.processed';

$maxsize_xml = 50000;
$compress_xml = 0;
$delete_failed = 0;

The error message I get when I try to run the .pl file:

IMAP Failure: 1 BAD Command received in Invalid state. at ./dmarcts-report-parser.pl line 185.

Warning introduced by the ipv6 commit (c0bce69)

Constant subroutine main::AF_INET6 redefined at ...
Prototype mismatch main::AF_INET6 () vs none at ...

In the current code, you inlcude everything from Socket and just (inet_ntop inet_pton AF_INET6) from Socket6.

If you remove that restriction and do

use Socket;
use Socket6;

The warning goes away. I HAVE ABSOLUTLY NO IDEA about WHY this works, and what really is up with socket and socket6. I just want to bring it to your attention.

Hello,

Any way to run the script as cron job?

I am having theese issues when running ./dmarcts-report-parser.pl -i

Already have emailsrvr.com a71202c5-d108-40bf-a818-6252fc01fb16, skipped
Could not create IMAP folder: Inbox.processed.
Error on moving (copy and delete) processed IMAP message: Could not COPY message to IMAP folder: <Inbox.processed>!
Messsage will not be moved/deleted. [11 NO [CANNOT] Character not allowed in mailbox name: '.']

No logs are generated on the mailserver,
I am quite clueless here :)

Hi,

First of all, thanks for the script. Nice to have all my reports in a DB. I finally got round to adding a DMARC entry in my DNS and have started getting reports. One issue I have is about 50% of them are not .zip files but .gz, which isn't supported by the script. Any chance we could see this in the next version?

Some of the domains that send .gz are:

• fastmail.com
• iledefrance.fr
• chu-dijon.fr
• linkedin.com
• qq.com
• smtp99.wallonie.be
• comcast.net
• ...

Hi,
I got the following when I am trying to run the script. The config file is populated with correct information

root@dmarc-parser:~/dmarcts-report-parser# ./dmarcts-report-parser.pl -i 
Use of uninitialized value $dbname in concatenation (.) or string at ./dmarcts-report-parser.pl line 193.
Use of uninitialized value $dbhost in concatenation (.) or string at ./dmarcts-report-parser.pl line 193.
DBD::mysql::db selectall_arrayref failed: No database selected at ./dmarcts-report-parser.pl line 778.
Can't use an undefined value as an ARRAY reference at ./dmarcts-report-parser.pl line 778.

Hi there,

I've been processing a load of reports and the script chokes on a malformed comcast report with empty auth_results element:

        <auth_results>
        </auth_results>

I'm guessing this contravenes the DMARC rfc.

The script returns the following before exiting:

Can't use string ("") as a HASH ref while "strict refs" in use at ./dmarcts-report-parser.pl line 575.

Maybe the script can be fixed to skip such entries instead of exiting.

Hello,

libmail-imapclient-perl libmime-tools-perl libxml-simple-perl
libclass-dbi-mysql-perl libio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl

Is not avabel for Centos 7.1?

For example...
{{{

dmarcts-report-parser -x /dmarc/2019/11/08/1573224396.M356010P333496.net14-hivelocity,S=5425,W=5540:2,a

Unable to recognise encoding of this document at /usr/local/share/perl/5.26.1/XML/SAX/PurePerl/EncodingDetect.pm line 100.
The xml file </dmarc/2019/11/08/1573224396.M356010P333496.net14-hivelocity,S=5425,W=5540:2,a> does not seem to contain a valid DMARC report. Skipped.
}}}
Likely this is because this tool can't actually parse real DMARC reports, which normally arrive as base64 encoded messages.

This means there's an extra step which must be done, which is... missing from docs...

If a file is an email message, the message must be decoded into the actual .xml file, rather than just be a mail file.

Hi, i'm decided to try your utility.
Configured:
egrep "dbname|dbhost" dmarcts-report-parser.conf
$dbname = 'dmarcreports';
$dbhost = 'localhost:3306'; # Set the hostname if we can't connect to the local socket.
The values of variables $dbuser and $dbpass are actual.

But I get an error:
./dmarcts-report-parser.pl -i
Use of uninitialized value $dbname in concatenation (.) or string at ./dmarcts-report-parser.pl line 193.
Use of uninitialized value $dbhost in concatenation (.) or string at ./dmarcts-report-parser.pl line 193.
DBI connect('database=;host=','',...) failed: Access denied for user 'john'@'localhost' (using password: NO) at ./dmarcts-report-parser.pl line 193.
Cannot connect to database

what's wrong? 8)

I receive dmarc reports in the inbox. what do I set $imapreadfolder to ?

I recently installed your project. but I realized it doesn't load the SPF Alignment field.
Is it a bug or doesn't it detect the tag?

https://mxtoolbox.com/DmarcReportAnalyzer.aspx?id=F-990c6c3d-e57e-4d7e-a462-ddf683a53ae1

yahoo.com!laurinzbabuin.com!1586563200!1586649599.xml.zip

I had also to install libmail-mbox-messageparser-perl in Debian to make it run.

Sometimes the parser thinks a message does not have a report:

----------------------------------------------------------------
Processing IMAP message with UID #17
----------------------------------------------------------------
Subject: Report Domain: afvalonline.nl Submitter: zeeland.nl Report-ID: <2018.3.21.3204>
MimeType: multipart/mixed
This is a multipart attachment
Skipped an unknown attachment
Skipped an unknown attachment
Could not find an embedded ZIP! The IMAP message with UID #17 does not seem to contain a valid DMARC report. Skipped.
Moving (copy and delete) processed IMAP message file to IMAP folder: Inbox.processed

This is because of a formatting error in the mail:

----=e6b1e617-42ae-4760-ac67-8328714cd151
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

This is a DMARC report generated by Trustwave SEG.

----=e6b1e617-42ae-4760-ac67-8328714cd151
ContentType: application/gzip;
	name="zeeland.nl!afvalonline.nl!1521020356!1521631697!3204.xml.gz"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="zeeland.nl!afvalonline.nl!1521020356!1521631697!3204.xml.gz"

ContentType must be Content-Type...

I humby suggest the following patch:

root@mx-2:/home/dmarcreport# diff -u dmarcts-report-parser dmarcts-report-parser.new
--- dmarcts-report-parser	2018-05-30 23:40:54.348653837 +0200
+++ dmarcts-report-parser.new	2018-05-30 23:39:26.860653839 +0200
@@ -436,6 +436,11 @@
 # itself is not checked to be a valid DMARC report.
 sub getXMLFromMessage {
 	my $message = $_[0];
+
+        # fixup type in trustwave SEG mails
+        $message =~ s/ContentType:/Content-Type:/;

 	my $parser = new MIME::Parser;
 	$parser->output_dir("/tmp");
@@ -499,7 +504,7 @@
 			} else {
 				# Skip the attachment otherwise.
 				if ($debug) {
-					print "Skipped an unknown attachment \n";
+					print "Skipped an unknown attachment (".lc $part->mime_type.")\n";
 				}
 				next; # of parts
 			}

Reports seen in the wild sometime lack actual 'auth_results' in the XML
which leads to

Can't use string ("") as a HASH ref while "strict refs" in use

See pull request #89

./dmarcts-report-parser.pl Can't locate Mail/Mbox/MessageParser.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./dmarcts-report-parser.pl line 64. BEGIN failed--compilation aborted at ./dmarcts-report-parser.pl line 64.
Using CentOS 7

Package perl-Mail-IMAPClient-3.37-1.el7.noarch already installed and latest version
Package perl-MIME-tools-5.505-1.el7.noarch already installed and latest version
Package perl-XML-Simple-2.20-5.el7.noarch already installed and latest version
Package perl-DBI-1.627-4.el7.x86_64 already installed and latest version
Package perl-Socket6-0.23-15.el7.x86_64 already installed and latest version
Package perl-PerlIO-gzip-0.19-1.el7.x86_64 already installed and latest version
Package perl-DBD-MySQL-4.023-5.el7.x86_64 already installed and latest version
Package unzip-6.0-16.el7.x86_64 already installed and latest version
Nothing to do

I'm missing some perl stuff?

Dear All,

My aim is to automate dmarc report processing by parsing reports from an exchangeserver based mailbox. Unfortunately, this does fail. I am able to parse unzipped xml files from a folder but neither zipped reports from a folder nor from IMAP.

On Centos 7, the debug report does contain two noticeable errors per message:

Could not find an embedded ZIP! The IMAP message with UID #29 does not seem to contain a valid DMARC report. Skipped.

ERROR: message_string() expected 1791 bytes but received 3546 you may need the IgnoreSizeErrors option at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 869

On Debian stretch, only the second error does occur.

Moving the messages to "Inbox.processed" does work on Centos but not on Debian?

I am certainly not a perl expert.

Could you please be so kind as to point me to possible next steps?

Regards,

Michael Schefczyk

How to parse Forensic report with this script? I got bunch of emails in Not Processed folder?

Hi, I am thankful for this great little tool. Helped a lot.
Since yesterday I noticed my imap emails were not being read from my routine script so when I run it manually today I notice following messages.

DBD::mysql::db do failed: Multiple primary key defined at /root/dmarcts-report-parser/dmarcts-report-parser.pl line 983.
DBD::mysql::db do failed: Incorrect datetime value: '1969-12-31 18:00:00' for column 'mindate' at row 1 at /root/dmarcts-report-parser/dmarcts-report-parser.pl line 770.
Cannot add report to database (Incorrect datetime value: '1969-12-31 18:00:00' for column 'mindate' at row 1). Skipped.
Skipping IMAP message with UID #10605 due to database errors.
DBD::mysql::db do failed: Incorrect datetime value: '1969-12-31 18:00:00' for column 'mindate' at row 1 at /root/dmarcts-report-parser/dmarcts-report-parser.pl line 770.
Cannot add report to database (Incorrect datetime value: '1969-12-31 18:00:00' for column 'mindate' at row 1). Skipped.
Skipping IMAP message with UID #10611 due to database errors.
Not a HASH reference at /root/dmarcts-report-parser/dmarcts-report-parser.pl line 720.

What could be causing these messages all of a sudden? Thank you,
UG

Hi @techsneeze

it looks like the release tags for the repository have gone (and possibly have been turned into branches?).

Could please create the release tags anew so that it becomes possible to track new upstream releases of this project?

Thanks,
Mike (maintaining dmarcts-report-parser in Debian)

Hi,

while setting up Rspamd with SPD/DKIM/DMARC on my own mail server recently, I today stumbled over this project. I have deployed it right a away and its awesome as it is simple to setup. Nice work!

Would you mind me to upload this package to Debian? If not, then I'd like to ask you for a release of the current code base (if it is releasable).

Thanks+Greets
Mike (aka sunweaver at debian.org)

Suddenly, from the 15th of Mars all reports is unable to parse due to this error message.

I tried do redeploy the script by removing it and download it from github again.

Hi everyone,
I have a strange issue today.
I had to re-install the server so I did chose to install CentOS 7 instead of the previously installed CentOS 6 and I am having a strange issue.

I did restore the DMARC Parser from a backup and now the script cannot "see" the configuration file which is in the same folder with the same name as before the new installation.

I have checked all the pre-requisites and all the packages are already there.

Here some details:

>yum install perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI  perl-Socket6 perl-PerlIO-gzip perl-DBD-MySQL unzip perl-Mail-Mbox-MessageParser
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.crazynetwork.it
 * epel: mirror.23media.de
 * extras: mirror.crazynetwork.it
 * rpmforge: mirror.de.leaseweb.net
 * updates: mirror.crazynetwork.it
Package perl-Mail-IMAPClient-3.37-1.el7.noarch already installed and latest version
Package perl-MIME-tools-5.505-1.el7.noarch already installed and latest version
Package perl-XML-Simple-2.20-5.el7.noarch already installed and latest version
Package perl-DBI-1.627-4.el7.x86_64 already installed and latest version
Package perl-Socket6-0.23-15.el7.x86_64 already installed and latest version
Package perl-PerlIO-gzip-0.19-1.el7.x86_64 already installed and latest version
Package perl-DBD-MySQL-4.023-5.el7.x86_64 already installed and latest version
Package unzip-6.0-16.el7.x86_64 already installed and latest version
Package perl-Mail-Mbox-MessageParser-1.5002-15.el7.noarch already installed and latest version
Nothing to do

>ls -l
total 36
-rw------- 1 dmarc dmarc  1973 Feb  3 14:22 dmarcts-report-parser.conf
-rwx------ 1 dmarc dmarc 29414 Feb  2 13:28 dmarcts-report-parser.pl
>./dmarcts-report-parser.pl

 Usage:
    ./dmarcts-report-parser.pl [OPTIONS] [PATH]

 This script needs a configuration file called <dmarcts-report-parser.conf> in
 the current working directory, which defines a database server with credentials
 and (if used) an IMAP server with credentials.

 Additionaly, one of the following source options must be provided:
        -i : Read reports from messages on IMAP server as defined in the
             config file.
        -m : Read reports from mbox file(s) provided in PATH.
        -e : Read reports from MIME email file(s) provided in PATH.
        -x : Read reports from xml file(s) provided in PATH.

 The following optional options are allowed:
        -d : Print debug info.
        -r : Replace existing reports rather than skipping them.
  --delete : Delete processed message files (the XML is stored in the
             database for later reference).

Please provide a source option (-i, -x, -m or -e).

>head -10 dmarcts-report-parser.conf
################################################################################
### configuration ##############################################################
################################################################################

# If IMAP access is not used, config options starting with $imap do not need to
# be set and are ignored.

$debug = 0;
$delete_reports = 0;

So, packages are there, the script doesn't seems to be able to find the config, the config is there and data are inside the config.

Do you have any idea on what I can do more? :D I also thought to pass the configuration file to the command line with a full path but I don't see any option that allow me to do that.

Thanks
Best regards
Andrea

Hello,

The DMARC Reports from does not show up in the DB? is that a bug?

DBD::mysql::db do failed: Specified key was too long; max key length is 767 bytes at ./dmarcts-report-parser.pl line 816. Use of uninitialized value in concatenation (.) or string at ./dmarcts-report-parser.pl line 233.

Full output: https://pastebin.com/2hLCL3eR

Hi there,
I am using this Parser for a few months now and it works perfectly, today I wanted to switch on the SSL but I am having some issue with it, once I enable SSL and change the IMAP Server port to 993 the script doesn't work and return this error:

connection to mail.myserver.tld:993 with Ssl => 1, User => [email protected], Ignoresizeerrors => 1
Started at Fri Feb 2 13:33:53 2018
Using Mail::IMAPClient version 3.37 on perl 5.016003
Connecting with IO::Socket::SSL PeerAddr mail.myserver.tld:993 PeerPort 993 Proto tcp Timeout 600 Debug 1

Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
together with SSL_ca_file|SSL_ca_path for verification.
If you really don't want to verify the certificate and keep the
connection open to Man-In-The-Middle attacks please set
SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.

at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 361.
ERROR: Unable to connect to mail.myserver.tld:993: at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 370.
Mail::IMAPClient::connect('Mail::IMAPClient=HASH(0x1c993f8)') called at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 313
Mail::IMAPClient::new('Mail::IMAPClient', 'Server', 'mail.myserver.tld:993', 'Ssl', 1, 'Starttls', undef, 'User', '[email protected]', ...) called at /home/dmarc/DMARC-Parser/dmarcts-report-parser.pl line 234
IMAP Failure: Unable to connect to mail.myserver.tld:993: at /home/dmarc/DMARC-Parser/dmarcts-report-parser.pl line 234.

While if I use TLS on port 993 It just hangs on the connection:
use tls with verify servercert.
connection to mail.myserver.tld:993 with Ssl => 0, User => [email protected], Ignoresizeerrors => 1
Started at Fri Feb 2 13:36:45 2018
Using Mail::IMAPClient version 3.37 on perl 5.016003
Connecting with IO::Socket::INET PeerAddr mail.myserver.tld:993 PeerPort 143 Proto tcp Timeout 600 Debug 1
Connected to mail.myserver.tld:993

Disabling TLS and SSL while using port 143 works perfectly.

Any suggestion?

Thanks

Best regards
Andrea

Hi,

is there a switch to ignore an invalid ssl cert?
I get this error:

IMAP Failure: Unable to start TLS: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at ./dmarcts-report-parser.pl line 154.

regards

Leaving aside that MyISAM is an outdated engine it also prevents usage on Percona Cluster:

DBD::mysql::db do failed: Percona-XtraDB-Cluster prohibits use of DML command on a table (email.rptrecord) that resides in non-transactional storage engine with pxc_strict_mode = ENFORCING or MASTER at ./dmarcts-report-parser line 720.

After upgrading dmarcts-report-parser, it fails with:

DBD::mysql::db do failed: Incorrect table definition; there can be only one auto column and it must be defined as a key at /home/users/dmarc/dmarcts-report-parser/dmarcts-report-parser.pl line 860.

when it tries to do ALTER TABLE rptrecord ADD id int(10) unsigned NOT NULL AUTO_INCREMENT FIRST; statement. SHOW CREATE TABLE says:

CREATE TABLE rptrecord (
serial int(10) unsigned NOT NULL,
ip int(10) unsigned DEFAULT NULL,
ip6 binary(16) DEFAULT NULL,
rcount int(10) unsigned NOT NULL,
disposition enum('none','quarantine','reject') DEFAULT NULL,
reason varchar(255) DEFAULT NULL,
dkimdomain varchar(255) DEFAULT NULL,
dkimresult enum('none','pass','fail','neutral','policy','temperror','permerror') DEFAULT NULL,
spfdomain varchar(255) DEFAULT NULL,
spfresult enum('none','neutral','pass','fail','softfail','temperror','permerror','unknown') DEFAULT NULL,
spf_align enum('fail','pass','unknown') NOT NULL,
dkim_align enum('fail','pass','unknown') NOT NULL,
identifier_hfrom varchar(255) DEFAULT NULL,
KEY serial (serial,ip),
KEY serial6 (serial,ip6)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

mysql-server is 5.5.60-0+deb8u1

Is see that not all SPF information from the XML is saved into the database.
The field are empty in the database and in the script I don't see any value getting set as value for these field.

I have modified my version of the program, so that also these SPF fields that are empty are filled with the correct information.

I will create a "Pull requests" for this.

When running the script on CentOS 7, you'll see the following error:

*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 454.

Hello,
Some attached reports comes in tar.gz, some in xml.gz format some in plain txt file
proximus.be!gemoss.lv!1506937803!1507024204.xml.gz
the import script should be able to import them, or at least skip them, because when it meets the message with one of those attached, it stops, and cant process further messages.

Hi,

I have a problem running the script after a few months of using this.
The system is giving the following error with debugging:

[root@dmarc dmarcts-report-parser]# ./dmarcts-report-parser.pl -d
There are 207 messages in the Inbox folder.

The Current Message UID is: 169

Subject: MimeType: multipart/report
/tmp/msg-2609-1.txt
/tmp/msg-2609-2.msg
Use of uninitialized value in print at ./dmarcts-report-parser.pl line 463.

Can't call method "purge" on unblessed reference at ./dmarcts-report-parser.pl line 466.

Hello,

If anyone's interested I have forked and changed a postgres version at https://github.com/grinapo/dmarcts-report-parser

Almost identical except:

• some cleanup and db generalisations
• removed automated table generation (little value for lot of time for me)

Feel free to pull back ideas if interested. :)

Also do whatever you please with this non-issue.

Thanks for the code!

Hi I had a zip file skipped for some reason

I see this in debug:

MimeType: multipart/mixed
This is a multipart attachment
Skipped an unknown attachment (text/plain)
Skipped an unknown attachment (application/x-gzip)
Could not find an embedded ZIP! The IMAP message with UID #133 does not seem to contain a valid DMARC report. Skipped.

Does this make sense ?

Hi techsneeze

i have some issues with the script. my DB is working and i can connect with my credentials.

mysql -h localhost -u dmarc -p dmarc

if i run the script like this

./dmarcts-report-parser.pl -x /tmp/dmarc/emailsrvr.com\!domain.com\!1531094400\!1531180800\!5074c731-3737-4597-8977-5af1717fb902.xml

i receive the following error:

DBI connect('database=dmarc;host=localhost','dmarc',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at ./dmarcts-report-parser.pl line 220.

this is the content of my dmarcts-report-parser.conf

################################################################################
### configuration ##############################################################
################################################################################

# If IMAP access is not used, config options starting with $imap do not need to
# be set and are ignored.

$debug = 1;
$delete_reports = 0;

$dbname = 'dmarc';
$dbuser = 'dmarc';
$dbpass = 'my_pass';
$dbhost = 'localhost'; # Set the hostname if we can't connect to the local socket.

$imapserver       = 'mail.example.com:143';
$imapuser         = 'dmarcreports';
$imappass         = 'xxx';
$imapssl          = '0';        # If set to 1, remember to change server port to 993 and disable imaptls.
$imaptls          = '1';        # Enabled as the default and best-practice.
$tlsverify        = '1';        # Enable verify server cert as the default and best-practice.
$imapignoreerror  = 0;          # set it to 1 if you see an "ERROR: message_string() 
                                # expected 119613 bytes but received 81873 you may 
                                # need the IgnoreSizeErrors option" because of malfunction
                                # imap server as MS Exchange 2007, ...
$imapreadfolder   = 'Inbox';

# If $imapmovefolder is set, processed IMAP messages will be moved (overruled by
# the --delete option!)
$imapmovefolder = 'Inbox.processed';

# maximum size of XML files to store in database, long files can cause transaction aborts
$maxsize_xml = 50000;
# store XML as base64 encopded gzip in database (save space, harder usable)
$compress_xml = 0;

# if there was an error during file processing (message does not contain XML or ZIP parts, 
# or a database error) the parser reports an error and does not delete the file, even if 
# delete_reports is set (or --delete is given). Deletion can be enforced by delete_failed, 
# however not for database errors.
$delete_failed = 0;

do you have any hints for me, where the misconfig could be?

thx

Hi there,

Been trying the IMAP feature but the script is having troubles with the emails I feed it.

In debug mode, I get errors such as the following:

The Current Message UID is: 45
--------------------------------
Subject: MimeType: text/plain
Could not find an embedded ZIP in <IMAP message with UID #45>. Skipped.
Moving (copy and delete) processed IMAP message file to IMAP folder: Mail server/DMARC/processed

Please find a slightly redacted version of a example email that fails. Seems to fail quite early on at the MIME::Parser stage as the subject isn't extracted.

Report Domain: unil.ch Submitter: ComUE Report-ID: [email protected]