xme / pastemon Goto Github PK

Introduction
------------

pastemon.pl is a script which runs in the background as a daemon and monitors pastebin.com for
interesting content (based on regular expressions). Found information is sent to syslog

The script can also generate (CEF events).

More information is available here: 
http://blog.rootshell.be/2012/01/17/monitoring-pastebin-com-within-your-siem/

v1.14 - 2012/10/31
------------------
- [FEATURE] Added SQLite DB support to store pasties details
  (some fields must still be implemented)

v1.13 - 2012/10/24
------------------
- [CONTRIBUTION] Added support for multiple SMTP recipients (email addresses separared by commas)
  Contribution from [email protected]
- [CONTRIBUTION] Added a new macro-% to specify the site name in the dump function.
  '%S' will be replaced by the site name. Example: '%S/%Y/%M' => 'pastebin.com/2012/10'. 

v1.12 - 2012/09/20
------------------
- [BUGFIX] Fixed FuzzyMatch() which was broken with gziped pasties.
- [FEATURE] Email notification: The subject is now appended with the <description> field(s)
  corresponding to the matched regex(es). This allows a better view of received emails as well
  as filtering them.
- [BUGFIX] Fixed FuzzyMatch() to detect properly duplicate pasties (fixed regex).

v1.11 - 2012/09/14
------------------
- [FEATURE] Added support for nopaste.me.
- [FEATURE] Added support for pastesite.com.
- [FEATURE] Added configurable sleep delays per pastie website.

v1.10 - 2012/08/01
------------------
- [FEATURE] If not configuration file is specified, pastemon.pl tries to load /etc/pastemon.conf
  by default.

- [FEATURE] pastemon.pl uses specific Perl modules like WordPress::XMLRPC or Text::JaroWinkler.
  The script now handles properly environment without those modules. It's not required to comment
  them in the code. If a module is missing, related configuration is automatically disabled.

- [FEATURE] Added optional compression (via IO::Compress::Gzip) of dumped pasties.
  In configuration file:

	<core>
		<compress-pasties>yes</compress-pasties>
	</core>

v1.9 - 2012/07/23
-----------------
- [FEATURE] pastemon can now follow (search for regex) URLs detected in pasties. This is
  configured via the main configuration file:

	<urls>
		<follow>yes</follow>
		<matching>(bit\.ly)</matching>
	</urls>

- [FEATURE] The regex.conf format changed to an XML format.
  Examples:
	<regex>
		<search>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}</search>
		<count>10</count>
		<description>IP Address</description>
	</regex>

- [FEATURE] A minimum number of regex occurences can be defined to notify
  (<count> tag in the XML file)

- [FEATURE] HTTP requests are now using now a random User-Agent. 

- [BUGFIX] Optimized the detection of already processed pasties. This reduces the amount of HTTP 
  requests send to the website.

v1.8 - 2012/06/25
-----------------
- [FEATURE] Adder for pastie.org! 
- [FEATURE] Added multi-thread support (1 thread per website monitored)
- [FEATURE] Added substitution macro in the dump directory. Support macros are:
	%Y - replace with the current year
	%M - replace with the current month
	%D - replace with the current day
  Directory is automatically created.
  Example: /home/user/pastemon/%Y/%M/%D

- [FEATURE] Added a new configuration directive:
	<dump-all>yes|1</dump-all>
  This feature enables a dump of *ALL* pastie wheter they match a regex or not.
  This is similar to a mirror mode
  WARNING: Huge disk space might be required by this feature!
- [BUGFIX] Test if the provided SMTP server (for mail notifications) is available
  (Thanks to @manuelsubredu for the patch)
- [BUGFIX] Fixed an issue in createBlogPost() which caused an unexpected process exit.

v1.7 - 2012/05/11
-----------------
- Added support for "included" regular expressions
- Fixed in bug in getRegexDesc()
- Added support for comments ('#') in the regex configuration file
- Moved configuration parameters from command line switches to an XML file
- Added matching regex description in dump files 
- Added SMTP notifications
- Added distance check to detect duplicate pasties (using Jaro-Winkler algorithm)

v1.6 - 2012/02/21
-----------------
- Added a detection of "slow down" messages returned by Pastebin (add a small pause)
- Added support for Wordpress XMLRPC 
- Added support for random proxies
- Some bug fixes

v1.5 - 2012/02/19
-----------------
- Fixed the regex to grab pasties from the archive page. (HTML code changed)

v1.4 - 2012/02/15
-----------------
- Fixed a bug with CEF events: custome fields start at 1 not 0! (Thanks to Heiko Hansen for the report)
- Notify the presence of a proxy variable (HTTP_PROXY) 

v1.3 - 2012/01/26
-----------------
- Added a '--pidfile=file' configuration switch to specify an alternative location for the PID file.
  This allows the script to be executed with a non-root account.
- Added a '--sample=x' configuration to display a sample a data matching a regular expression. 'x' is 
  the number of bytes displayed before and after the matching string. This is useful to estimate the
  value of the pastie.  Example:
  Found in http://pastebin.com/raw.php?i=Q8pQRHKW : belgium (2 times) | Sample: g(0) ""\n  [32] => string(11) "Belgium(32)"\n  [31] => string(14) "Ne

v1.2 - 2012/01/21
-----------------
- Fixed a bug affecting the case sensitivity search
- New feature: an exception can be associated to a regular expression in the configuration file.
  The syntax is: "regex1 _EXCLUDE_ regex2". This could prevent some false positive matches.

v1.1 - 2012/01/20
-----------------
- Added a '--dump' configuration switch to save matching pasties in a directory.
  This is to keep the pasties posted with an expiration date (example: for later review)

v1.0 - 2012/01/18
-----------------
Initial release