daxbert / rds_slow_query_log_examiner Goto Github PK

Let's get fingerprinting into a module, shoot for a more pythonic approach.

And hey... maybe actually have some tests.

See error below

2018-08-13 15:05:43,121 - rds_slow_query_log_examiner - INFO - Key: ..._1534197600000_1534197265000
2018-08-13 15:05:51,791 - rds_slow_query_log_examiner - INFO - Event Count: 0
2018-08-13 15:05:51,805 - rds_slow_query_log_examiner - INFO - Budget Left: 20000
[2018-08-13 15:05:51,813] ERROR in app: Exception on /us-west-2/stream/data/arn:aws:logs:us-west-2:.../slowquery:log-stream:.../ [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/apps/rds_slow_query_log_examiner/www/app.py", line 210, in stream_page
    logEntries = getLogEntries(stream['logGroup'], stream['logStreamName'], start_timestamp, end_timestamp)
  File "/apps/rds_slow_query_log_examiner/www/app.py", line 343, in getLogEntries
    logger.info('TOTAL Queries Parsed: {}'.format(logEntries['METRICS']['TOTAL_QUERY_COUNT']))
KeyError: 'METRICS'

adding .ps1 files for Powershell on Windows.

This should allow Windows 8+ users who also have Docker installed on Windows to work with this project

Show queries parsed, aggregate time, aggregate rows, etc for the entire time window parsed.

Here's an example where we really should have just collapsed the VALUES to VALUES ( {VALUE_LIST} )

INSERT INTO table ( ... ) VALUES (####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####,{STRING},{STRING},{STRING},{STRING}),(####... );

Set start date, clear end date, click submit.

[2018-08-02 10:23:22,636] ERROR in app: Exception on /us-west-2/stream/data/... [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functionsrule.endpoint
File "/apps/rds_slow_query_log_examiner/www/app.py", line 208, in stream_page
endDateTimestamp = int(datetime.datetime.strptime(endDateString, "%Y/%m/%d %H:%M:%S").timestamp() * 1000)
File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime
(data_string, format))
ValueError: time data '' does not match format '%Y/%m/%d %H:%M:%S'

Let's make this suck less...

Need to get a test framework in place

• we should be able to run tests after all commits
• do we auto-merge to master with success?
• how to mock AWS?
• enhance existing DockerHub integration? or another web build pipeline?

Right now the credentials are expected to be provided by the command line via environment variables.

Add feature to support credentials via the web UI.

Considerations:

• Session Management
• Layout changed to support login
• SSL ( while this is intended to be used locally, however dumb crap happens, insure SSL when localhost is not in use )

Not sure why but sometimes the AWS API is returning "storedBytes" as 0 when the CW logs actually have events.

Change the template test from storedBytes to something else... TBD

In short, if the user fails to set the environment variables, or sets them incorrectly, the app blows up with a 500 error.

Handle this in a web friendly way.

All metrics are currently aggregated over the given query's fingerprint/hash. Provide a slowest / worst view for those queries within a given fingerprint/hash.

LIkely an issue with the tied cache on disk. Need to solve for locking, or figure out a less lame cache.

Get docker hub docs here too

Fix typos.

Help folks understand how to contribute