——-

• About the TA-mailclient
• Release notes
• Support and resources

• Hardware and software requirements
• Installation steps
• Configure TA-mailclient
  • Parameters

• Key concepts
• Data types
• Troubleshooting
• Upgrade
• Copyright & License

The TA-mailclient add-on fetches emails for Splunk to index from mailboxes using either POP3 or IMAP, with or without SSL.

The modular input also stores takes the password from inputs.conf in plain text, and replaces it with a place holder, while storing it encrypted within Splunk. This is built using the Splunk SDK for Python, should work on any Splunk installation with Python available including SHC. Passwords should also get replicated between search heard peer members.

This only fetches emails form the 'inbox' folder. A future upgrade might include support for additional mailbox directories.

By default, it fetches up to 25 new emails at every run. Be sure to set the interval to run this as frequently as required.

It supports all 'text/*' content types and several well known scripts (.bat, .js, .sh) detailed below:

'application/xml'
'application/xhtml'
'application/x-sh'
'application/x-csh',
'application/javascript'
'application/bat'
'application/x-bat'
'application/x-msdos-program'
'application/textedit'

Images, videos and executables are not indexed.

Includes:

• Splunk SDK for Python (1.6.2)
• mail_lib - supports the calculation of vincenty distances which is used by default
  • pop_utils.py - functions used to retrieve mails via POP3 using the built-in poplib library
  • imap_utils.py - functions used to retrieve mails via IMAP using the built-in imaplib library
  • constants.py - A number of constants / defaults used throughout the mail_lib module.
  • mail_common.py - Shared functions used to parse emails and attachments
  • exceptions raised by functions used in the mail_lib module.

Version 1.1.2 of the TA-mailclient is compatible with:

The administrator is responsible for setting the sourcetype to whatever is desired, as well as extracting CIM fields for the sourcetype. This app already includes several extractions for different parts of the message that can be reused.

This app will not work on a universal forwarder, as it requires Python which comes with an HF or a full Splunk install.

Note: Travis CI includes tests for both secure and insecure versions of POP3 / IMAP.

TA-mailclient includes the following new features:

• Fixed unicode conversion of emails following contributions from Francois Lacombe on GitHub
  • Also added static mail preamble for line break. Event breaking configuration may not be required since the modular input writes individual events separately, but it's always a good idea.
• Added delineations and extractions to multipart content
• Removed interval from inputs.conf.spec
• Upgraded Splunk SDK to 1.6.2
• Added additional test cases on Travis CI to test that functionality works
• Fix loading local exceptions
• modularized storage/password functions to make them reusable and simpler
• Also fixed exception handling when dealing with storage/password
• Fixed type casting for boolean parameters (is_secure, include_headers)
• Rewrote sections of mail_common
• Fixed bool parameter for mail port validation for insecure protocols

Currently no known issues in version 1.1.2 of TA-mailclient. This is currently tested against the latest version of Splunk Enterprise. Issues can be reported and tracked on Github at this time.

This uses the inbuilt poplib and imaplib that comes with Python by default.

Contributions on github are welcome and will be incorporated into the main release. Current contributors are listed in AUTHORS.md.

Questions and answers

Access questions and answers specific to the TA-mailclient at (https://answers.splunk.com/).

Support

This Splunk support add-on is community / developer supported.

Questions asked on Splunk answers will be answered either by the community of users or by the developer when available. All support questions should include the version of Splunk and OS.

You can also contact the developer directly via Splunkbase. Feedback and feature requests can also be sent via Splunkbase.

Issues can also be submitted at the TA-mailclient repo via on Github

Future release will support

1. Support for configuration of mail limits in inputs.conf
2. Recursive option to read all folders inside Inbox, and not just emails within inbox.
3. Support indexing mails from additional folders in a mailbox

Note : This has not been tested against an exhaustive list of mail servers, so I'll welcome the feedback.

Also, feel free to send me a list of well known servers that you 're using this with without problems.

• v0.5.1

  • encoding corrections
  • deduplicate Date and MessageId from indexed headers
  • correction of MessageID extraction
  • changed the separator to a predefined one instead of Date and MessageID
  • activated and changed label for unsupported attachment

• v0.5.0

  • Fixed UTF-8 encoding of mails before indexing. (Supporting Gmail and others)

• v0.4.9

  • Changed encoding to support reading gmail.

• v0.4.8

  • removed error introduced in v0.4.7

• v0.4.7

  • Removed password field validation to allow users have complex or easy passwords however long
  • Handled all mail exceptions

• v0.4.6

  • Fixed bug.
  • Fixed header inclusion

• v0.4.5

  • Fixed bug. Removed line which caused v0.4.4 to fail
  • Fixed header inclusion

• v0.4.4

  • Updated app to ignore case of file attachment extension

• v0.4.3

  • Made extensions case insensitive
  • Added support for indexing .docx extensions
  • Generalised Mail.save_password() to allow reuse of code when writing other modular inputs.
  • Optimized python import statements
  • Fixed deleting of mails in poplib which was broken in 0.4

• v0.4.2

  • Added support for indexing mail headers

• v0.4.1

  • Fixed bug with 0.4.0
  • Made updates to fix unneeded else statement which introduced bug in 0.4.0.

• v0.4

  • Added support for decoding unicode characters in other languages or and removing the unicode identifier in the header.
  • Improved support for indexing some file types even if the content-type is not set correctly. (as with Microsoft sending some files as binaries instead of text)
  • Added fundamental code to support indexing of attachment as a configurable option in future release by the user.
  • Added multiple field extractions for the email header and file attachments.
  • Introduced a bug which was corrected in 0.4.1 Faulty version

Note: filename and filecontent are multi-valve fields.

• v0.3

  • Adds support for mailbox cleanup options

• v0.2

  • Adds support for base64 encoded emails.

TA-mailclient supports the following server platforms in the versions supported by Splunk Enterprise:

• Linux
• Windows

The app was developed to be platform agnostic, but tests are mostly run on Linix.

Please contact the developer with issues running this on Windows. See the Splunk documentation for hardware requirements for running a heavy forwarder.

To function properly, TA-mailclient has no external requirements but needs to be installed on a full Splunk install which provides python and the required libraries (poplib and imaplib).

Because this add-on runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.

Download the TA-mailclient at one of the following locaitons:

• Splunkbase
• Github

To install and configure this app on your supported standalone platform, do one of the following:

• Install on a standalone Splunk Enterprise install via the GUI. See Link
• Extract the technology add-on to $SPLUNK_HOME/etc/apps/ and restart Splunk

Install to search head - (Standalone or Search head cluster)

• Install the support add-on located at TA-mailclient/appserver/SA-mailclient.tgz on the search head. If using search head cluster, install the SA-mailclient.tgz via a search head deployer.

Install to indexers

• No App needs to be installed on indexers

Install to forwarders

• Follow the steps to install the TA-mailclient on a heavy forwarder. More instructions available at the following URL

• Configure an email input by going to the setup page or configuring inputs.conf.

For Splunk cloud installations, install TA-mailclient on a heavy forwarder that has been configured to forward events to your Splunk Cloud instance.

You can work with Splunk Support on installing the Support add-on on Splunk Cloud.

This app adds a mail:// modular input and supports a variety of parameters in inputs.conf.

[mail://[email protected]]
interval = 600
is_secure = 1
mailserver = imap.domain.com
password = mypassword
protocol = IMAP|POP3
disabled = 0
mailbox_cleanup = delete

Once the input is read, the password gets replaced and shows as 'encrypted'. As such, the password for the mailbox must not be set to 'encrypted'.

The input can be edited if the password needs to be updated, and the password stored in a password storage endpoint would get updated automatically. Passwords are never stored in clear text.

A different sourcetype can be specified for each input, thus making it possible to have different sourcetypes for every mailbox. Mailbox cleanup is also managed automatically, and emails are deleted once it has been indexed.

mailserver - This is a mandatory field and should be the hostname or IP address for the mail server or client access server with support for retrieving emails via POP3 or IMAP

protocol - This must be set to either POP3 or IMAP

is_secure - This should be set to 1 if emails should be retrieved using the protocol selected over SSL.

password - Passwords must be set for every account, or the input will get disabled.

mailbox_cleanup = This indicates if every email should be deleted as it is read, or delayed until the next interval. Setting this to readonly prevents mails from being deleted.

The default is readonly. Supported options are: delayed|delete|readonly

interval - This should be configured to run as frequent as required to retreive emails. This modular input retrieves up to 20 emails at each run. A future release to this input might allow the limit to be configured as a parameter to the modular input.

This modular input supports multiple instances, and each input runs at separate intervals.

include_headers - This determines if email headers should be included.

Once an email is indexed, it will not be re-indexed except the checkpoint directory is emptied. This can be achieved by running the following command:

splunk clean inputdata mail

Logs can be found by searching Splunk internal logs

index=_internal sourcetype=splunkd (component=ModularInputs OR component=ExecProcessor) mail.py

Additional logging can be enabled by turning on debug logging for ExecProcessor and ModInputs. set the logging level of the ExecProcessor to Debug

/opt/splunk/bin/splunk set log-level ExecProcessor -level DEBUG /opt/splunk/bin/splunk set log-level ModInputs -level DEBUG

You can find additional ways to enable debug logging on here.