wvanbergen / request-log-analyzer Goto Github PK

A --silent option that skips printing the marketing blurb and output feedback would be great for when you need to run this from cron.

I realize that you could simply ignore STDOUT/STDERROR with redirection but I'd rather the script not print anything so that one is able to catch real errors that may arise from running it from cron.

It would be nice to have an --amount command line switch to toggle the number of hits returned in the output, be it HTML, plain text, or yaml, with the ability to specify a particular number or unlimited.

hey :-)

sometimes i get the following error, when parsing large log files (about 7-24 MB). sometimes means i cannot reproduce the error, but it seems to occur when doing things in parallel ...

    C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:188:in `log': SQLite3::SQLException: SQL logic error or missing database: INSERT INTO "requests" ("first_lineno", "last_lineno") VALUES(53906, 53910) (ActiveRecord::StatementInvalid) 
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite_adapter.rb:132:in `execute'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite_adapter.rb:372:in `catch_schema_changes'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite_adapter.rb:132:in `execute'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:171:in `insert_sql'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite_adapter.rb:146:in `insert_sql'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:44:in `insert_without_query_dirty'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/query_cache.rb:18:in `insert'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:2733:in `create_without_callbacks'  ... 27 levels...
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/request-log-analyzer-1.6.4/bin/../lib/request_log_analyzer/controller.rb:320:in `run!'
    from C:/Data/tools/InstantRails/ruby/lib/ruby/gems/1.8/gems/request-log-analyzer-1.6.4/bin/request-log-analyzer:120
    from C:/Data/tools/InstantRails/ruby/bin/request-log-analyzer:19:in `load'
    from C:/Data/tools/InstantRails/ruby/bin/request-log-analyzer:19

(i working on a windows system, with instantrails, ruby 1.8.6 and rails 2.2.2, sqlite-ruby 1.2.1 and sqlite version 3.5.4)

thank you

Like parsing from back to front to determine starting position if after was given and file bigger than x bytes.

Is there anyway to do continuous logging of *.logs such that after an inital parse of a log into a sqlite3 database, RLA would monitor the log and append the changes since the last parse date into the database?

That way old logs even after it gets rotated, will still be recorded into the database.

Before parsing requests into an existing database, check if the database schema is compatible with the file format. Currently, it will not insert all information or crash if the database schema is different from the file format.

Introduce parallelization to speed up parsing. Some ideas:

• Try the regexps for every line type in parallel.
• Parse multiple lines at once.

Allow every command line option to be set using a configuration file. This file should probably be in YAML format and should be provided with the --config option.

I run (in my project):

request-log-analyzer log/development.log

Outputs nothing of value, but I see warning:

...
Warnings: teaser_check_failed: 312, no_current_request: 302
...

It worked for me before I updated the gem yesterday.

Ditch our current custom command line options parser for a standard parser using the optparse library. After this is finished, create a wiki page with all command line options.

Allow specifying the line prefix for the file format. This may speed up parsing the file because the parser only has to check the beginning of the line for a match.

• file_format.line_prefix = :any: allow any prefix (current behavior)
• file_format.line_prefix = 123: Every line has a prefix of 123 characters
• file_format.line_prefix = /line prefix regexp/: Every line starts with the given regexp.

The file_format's regular expressions will be adjusted accordingly. This option may be supplied to r-l-a using a command line switch like --line-prefix

I've noticed a large increase in memory usage when parsing large line entries in a log file. It does not appear to be a fault of RLA but an issue with IO#gets.

It appears that as the line is read in chunks and written to a buffer, memory usage grows. I've seen cases of greater than 500 MB memory usage when parsing a log file with 5 MB line entries.

While it isn't good practice for a log file to contain these large lines, we've seen it several times in Rails apps that handle file uploads. The biggest issue is when RLA is run in the background - it could exhaust the memory usage on a server without the user being aware that the process is running.

To reproduce:

Create a log file with a 5 MB line entry:

ruby -e 'puts "one"; puts "t" * 1024 * 1024 * 5' > big_data.txt

Run RLA and observe the memory usage:

request-log-analyzer big_data.txt

I've observed this on Ruby 1.8.6 (haven't tested on other releases).

I'm looking at a possible fix: When reading the contents of a file, ignore chunks that don't have a line separator. If RLA reads the file in 1024 segments, it can be reasonably sure that the request path will not be this long.

Just wanted to make sure you are aware. I'll submit a pull request when I have something to demo.

Is it a good idea to start a mailing list so people can get help using r-l-a?

Allow returning the reports in an easy-to-parse format, like YAML. This allows easily comparing different reports with automated tools.

I can parse a log file successfully, but when I attempt to save the results into a sqlite file, I get the error described below. Does this file need to already exist? Help text says the RLA will create it for me, so I have not created the database file before running the command below.

$ rails --version
Rails 2.2.2
$ ruby --version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.6.0]

$ request-log-analyzer --database web2.db log/ipsm_dev_oracle.log
Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version 1.2.1
Website: http://railsdoctors.com

/opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:326:in retrieve_connection': ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished) from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_specification.rb:121:inretrieve_connection'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_specification.rb:113:in connection' from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/migration.rb:346:inmethod_missing'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/migration.rb:326:in say_with_time' from /opt/software/wxweb/opt/lib/ruby/1.8/benchmark.rb:293:inmeasure'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/migration.rb:326:in say_with_time' from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/migration.rb:342:inmethod_missing'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/../lib/request_log_analyzer/aggregator/database.rb:141:in create_request_table_and_class' from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/../lib/request_log_analyzer/aggregator/database.rb:166:increate_database_schema!'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/../lib/request_log_analyzer/aggregator/database.rb:27:in prepare' from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/../lib/request_log_analyzer/controller.rb:193:inrun!'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/../lib/request_log_analyzer/controller.rb:193:in each' from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/../lib/request_log_analyzer/controller.rb:193:inrun!'
from /opt/software/wxweb/opt/lib/ruby/gems/1.8/gems/wvanbergen-request-log-analyzer-1.2.3/bin/request-log-analyzer:93
from /opt/software/wxweb/opt/bin/request-log-analyzer:19:in `load'
from /opt/software/wxweb/opt/bin/request-log-analyzer:19

Design and build a tool that can easily run queries on a request database that has been created with r-l-a.

As request log analyzer can very easily be used to find any errors that occured in say the last five munites, whould a munin extention be a good addition?

For one-pass algorithms to do this, see http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

We always forget to update the version constant. The version in the .gemspec file is always correct, so we could use the value specified there.

To speeds up parsing when you're not interested in the report, but only want a request database. This can boost performance and requires significantly less memory when there are a lot of request categories.

Add tests to ensure the database console is working as expected.

A tracker to keep track of the sources that were parsed. The same information is already stored when using the database inserter. Using the --dump functionality to store this information after a parsing run, this can be used to implement resuming.

how do i import data to an existing database?

i get an error saying that the index of table XY already exists ... guessing that r-l-a is trying to create the database schema again!?

thanks for support & keep up the good work!

Allow parsing compressed files. This can be auto-detected using the filename extension, e.g. .gz.

Maybe it is possible to decompress the file in small parts, in order to keep the memory usage down. Otherwise, the files can be decompressed into a temporary file.

To ensure that everything works to mail reports, including the --mail command line option.

Add a new output class that creates a PDF report.

I'm running an analysis of a syslog file and I get this crash.

/var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/line_definition.rb:57:in =~': broken UTF-8 string (ArgumentError) ] ETA: 25:56:27 from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/line_definition.rb:57:inmatches'
from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/file_format.rb:268:in block in parse_line' from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/file_format.rb:267:ineach'
from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/file_format.rb:267:in parse_line' from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/source/log_parser.rb:163:inparse_line'
from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/source/log_parser.rb:151:in parse_io' from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/source/log_parser.rb:112:inblock in parse_file'
from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/source/log_parser.rb:112:in open' from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/source/log_parser.rb:112:inparse_file'
from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/source/log_parser.rb:62:in each_request' from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/lib/request_log_analyzer/controller.rb:320:inrun!'
from /var/lib/gems/1.9.0/gems/request-log-analyzer-1.6.1/bin/request-log-analyzer:120:in <top (required)>' from /var/lib/gems/1.9.0/bin/request-log-analyzer:19:inload'
from /var/lib/gems/1.9.0/bin/request-log-analyzer:19:in `

Add a SQL index checker to see whether indexes are set for the most popular queries, if the log file contains SQL queries.

For inspiration, see: http://github.com/samdanavia/ambitious_query_indexer

when I use this set of options on a rack formatted log file :
request-log-analyzer --format rack log_file.log --output html

the html report is printed but a exception is finally thrown :

/Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/tracker/traffic.rb:92:in report': undefined methodcolorize' for #RequestLogAnalyzer::Output::HTML:0x531178 (NoMethodError)
from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/aggregator/summarizer.rb:128:in report' from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/aggregator/summarizer.rb:128:ineach'
from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/aggregator/summarizer.rb:128:in report' from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/controller.rb:327:inrun!'
from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/controller.rb:327:in each' from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/../lib/request_log_analyzer/controller.rb:327:inrun!'
from /Library/Ruby/Gems/1.8/gems/request-log-analyzer-1.5.0/bin/request-log-analyzer:116
from /usr/bin/request-log-analyzer:19:in `load'
from /usr/bin/request-log-analyzer:19

Allow output classes to override the output of a tracker to allow for more beautiful reports.

The line numbers that are used while parsing relate to the current file, but there is no way of knowing what file the line was in when using multiple files.

Allow other kinds of DB connections besides MySQL, for example by allowing a YAML configuration file with DB connection parameters.

http://askmonty.org/wiki/MariaDB

See: http://narnach.blogspot.com/2009/06/upgraded-to-ruby-191.html

Use the records in the sources table in the database to skip the first part of a file if that was already parsed before.

We could improve integration with a Rails project. This tickets contains some ideas to accomplish this:

Settings

• Request database connection in config/database.yml (see also #10). This way, it could even be accessed from the rails application itself :-)
• Store settings in config/rla.yml
• Store log filename pattern per environment in settings file (e.g. production.*.log.gz)

Rake tasks

• rla:report Run rla on and return ASCII report in terminal
• rla:report:html Run rla and create a HTML report
• rla:report:yaml Run rla and create raw YAML report (se #13)
• rla:database Parse request log and store requests in database
• rla:database:reset Clear the request database

Other ideas

• Archive reports (with date in filename) on a default location so comparisons are possible (e.g. /doc/performance)
• Run r-l-a as cronjob?

Make a screencast to demonstrate the tool.

Add --skip-bytes option to skip the first n bytes of a file. This makes it easier to not parse duplicate data.

This may be useful when used with a cronjob to do periodical analysis on an application's logs. Adding an example cronjob would be nice as well for reference.

Currently, the source_id field in the warnings table will always be filled with NULL.

When I pass the "--after" option to RLA v1.6.2 for a Delayed Job file, the following exception is returned:

/usr/lib/ruby/gems/1.8/gems/request-log-analyzer-1.6.2/bin/../lib/request_log_analyzer/filter/timespan.rb:35:in `<=': comparison of Fixnum with nil failed (ArgumentError)

Full trace: http://gist.github.com/292222

I believe this may be because the Delayed Job log does not have timestamps (at least v 1.7.0, the most recent version).

Sample:

• [JOB] aquiring lock on MassUpdate#copy_from_file
• [JOB] MassUpdate#copy_from_file completed after 1.1547

If this is the case, I don't think there is anything RLA can do. If so, let me know and I'll add info the Delayed Job format wiki entry.

Describe the pipeline, the related classes and how to extend functionality in the project’s wiki.

One other minor issue I've had with RLA is wishing there was an API that more closely mirrored the command-line invocation. For example, I wish I could use it like a library sort of like:

result = StringIO.new
RequestLogAnalyzer.analyze( log, :after  => start_time,
                              :output => "HTML",
                              :file   => result )

This way, the DBMS does not have to update every index over and over when inserting new records

A tracker, similar to the duration tracker, to keep track of the total amount of traffic. Apache access logs and Amazon S3 access log contain fields about the number of bytes sent, which can be analyzed by such a tracker.

minor finding:

line_definition :print do |line| does not work ... maybe names can be escaped?

(using rla 1.6.2)