obsrvbl-oss / flowlogs-reader Goto Github PK

Hi,

Thank you for this tool, very useful!

A couple of suggestion:

1. Using aggregate action header and date are not aligned
2. Is possible to change the protocol number with actual protocol name according: https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml
3. Using the command line tool, would be nice to read data from a file and/or a variable or directly from a pipeline (for performing analysis on exported data)

Thanks

This is not an issue. Just want to know your opinion, which is the best - CloudWatch logs vs S3 logs in terms of performance using this library for same infrastructure?

On occasion the pager returned by get_paginator('filter_log_events') will raise PaginationError, noting The same next token was received twice.

One hopes that this will be eventually fixed in botocore, and one hopes that this actually signals the end of the stream.

This issue is to catch the exception and terminate without raising an exception.

Trying to use filter pattern with S3FlowLogsReader Class in python and it returns error

FILTER_PATTERN = (
    '[version="2", account_id, interface_id, srcaddr, dstaddr, '
    'srcport, dstport, protocol="17", packets, bytes, '
    'start, end, action, log_status="OK", instance_id]'
)

reader = S3FlowLogsReader('xyz-vpc-flow-logs', include_regions=['us-east-1'], filter_pattern=FILTER_PATTERN)
records = list(reader)

for record in records:
    print (record)

Error is as follows

Traceback (most recent call last):
  File "flowlogs-read1.py", line 14, in <module>
    reader = S3FlowLogsReader('xyz-vpc-flow-logs', include_regions=['us-east-1'], filter_pattern=FILTER_PATTERN)
  File "/usr/local/lib/python2.7/site-packages/flowlogs_reader/flowlogs_reader.py", line 266, in __init__
    super(S3FlowLogsReader, self).__init__('s3', **kwargs)
TypeError: __init__() got an unexpected keyword argument 'filter_pattern'

The flow_logs_reader hangs while trying to retrieve large result dataset.

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 377, in _make_request httplib_response = conn.getresponse(buffering=True) TypeError: getresponse() got an unexpected keyword argument 'buffering'

It would be very helpful if you could tag releases as well. This would enable distributions who want to run your tests to fetch the package from GitHub instead of PyPI, where the tests are excluded.

It looks like that this was done in the past.

Thanks

The AWS region name is currently hardcoded to us-east-1, but it should use Boto's default behavior (e.g. using the default region configured in ~/.aws/config):

https://github.com/obsrvbl/flowlogs-reader/blob/master/flowlogs_reader/flowlogs_reader.py#L128