I've noticed a large increase in memory usage when parsing large line entries in a log file. It does not appear to be a fault of RLA but an issue with IO#gets.
It appears that as the line is read in chunks and written to a buffer, memory usage grows. I've seen cases of greater than 500 MB memory usage when parsing a log file with 5 MB line entries.
While it isn't good practice for a log file to contain these large lines, we've seen it several times in Rails apps that handle file uploads. The biggest issue is when RLA is run in the background - it could exhaust the memory usage on a server without the user being aware that the process is running.
To reproduce:
Create a log file with a 5 MB line entry:
ruby -e 'puts "one"; puts "t" * 1024 * 1024 * 5' > big_data.txt
Run RLA and observe the memory usage:
request-log-analyzer big_data.txt
I've observed this on Ruby 1.8.6 (haven't tested on other releases).
I'm looking at a possible fix: When reading the contents of a file, ignore chunks that don't have a line separator. If RLA reads the file in 1024 segments, it can be reasonably sure that the request path will not be this long.
Just wanted to make sure you are aware. I'll submit a pull request when I have something to demo.