dynatrace-oss / dynatrace-aws-s3-log-forwarder Goto Github PK

Is your feature request related to a problem? Please describe.
When a user has multiple Dynatrace tenants configured as destinations and there's an exception POST'ing log data, the exception doesn't include the tenant ID and so the user requires setting up DEBUG logging to see the tenant id.

Describe the solution you'd like
Add tenant id to Dynatrace exceptions. Example log message:

[ERROR]	2022-11-28T08:07:43.892Z a2957f67-2ce9-53d0-8af8-2499428d3a23 Usable space limit reached. Exhausted retry attempts...
[ERROR] 2022-11-28T08:07:43.892Z a2957f67-2ce9-53d0-8af8-2499428d3a23 Error processing message 550b0e17-3d3b-4b93-9cfe-af39bab741b4 

Is your feature request related to a problem? Please describe.
Data that we would like to sent to logs may contain a couple of JSON keys from a larger pool (beyond 2000). Though we could list them all in attribute_extraction_jmespath_expression this would be harder to maintain and may seriously affect performance of ingest, when scanning though the attribute_extraction_jmespath_expression mapping.

Describe the solution you'd like
We would like to define a simple rule where all top level JSON keys are mapped to attributes based on given pattern.
For example:
attribute_extraction_regexp_expression: "myattr_$1": "(.+)"
would map all top level keys to myattr_* attributes.
Since there is a limit on number of attributes created, we should also enable a way to exclude some top level keys from mapping, e.g.,
attribute_extraction_regexp_expression: "exclude!": "(skip|these|keys)" "myattr_$1": "(.+)"

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency jinja2 from 3.1.3 to 3.1.4
• chore(deps): update public.ecr.aws/lambda/python docker tag to v3.9.2024.04.17.17
• chore(deps): update dependency aws-lambda-powertools from 2.33.0 to 2.37.0
• chore(deps): update dependency responses from 0.24.1 to 0.25.0
• chore(deps): update dependency moto from 4.2.14 to 5.0.6
• chore(deps): update dependency regex from 2023.12.25 to 2024.4.28
• chore(deps): update dependency urllib3 from 1.26.18 to <3
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Is your feature request related to a problem? Please describe.
If the NotificationsEmail parameter of the SAM template is left empty, the template fails to deploy as the SNS subscription is part of the AWS::SNS::Topic resource. It would be better to make this parameter optional.

Describe the solution you'd like
Customers should be free to subscribe an e-mail address to the SNS Alerts notifications topic or not. We should create a separate AWS::SNS::Subscription resource that's only created if NotificationsEmail is not empty. Also, we should add the SNS topic as a Cloudformation export, so customers can subscribe the endpoints they want on a separate CloudFormation template (e.g. Service Now, PagerDuty...).

At the moment, the configuration of log forwarding rules and custom log processing rules (if defined) is deployed together with the log forwarding code so any configuration changes require re-deployment. We need to decouple the configuration from the log forwarder code.

Is your feature request related to a problem? Please describe.
Our files which we would like to load as logs do not have extension, while some of them may be gziped.

Describe the solution you'd like
The easiest approach would be to depend on content-encoding.

Describe alternatives you've considered
Potentially we could look into magic bytes - but that would require operations on streams which would be ugly and slow processing down.
Another alternative would be to define different log_format, e.g., json_stream_compressed - but that would mean too many changes in the code.
Also introducing processing rules name convention, e.g., something_gziped ... sounds even worse.

Is your feature request related to a problem? Please describe.
The dynatrace-aws-s3-log-forwarder currently supports parsing log entries within a JSON array, but some logs like VPC Resolver Query Logs are a stream of JSON objects.

Describe the solution you'd like
Add a processing rule for JSON-stream logs to be forwarded to Dynatrace.

Additional context
Example logs for VPC Resolver Query logs:

{"version":"1.100000","account_id":"012345678910","region":"us-east-1","vpc_id":"vpc-0123456789abcdef","query_timestamp":"2022-11-02T08:13:55Z","query_name":"ec2-1-2-3-4.compute-1.amazonaws.com.","query_type":"A","query_class":"IN","rcode":"NOERROR","answers":[{"Rdata":"1.2.3.4","Type":"A","Class":"IN"}],"srcaddr":"172.31.28.35","srcport":"43314","transport":"UDP","srcids":{"instance":"i-0123456789abcdef0"}}
{"version":"1.100000","account_id":"012345678910","region":"us-east-1","vpc_id":"vpc-0123456789abcdef","query_timestamp":"2022-11-02T08:13:55Z","query_name":"ec2-1-2-3-4.compute-1.amazonaws.com.","query_type":"AAAA","query_class":"IN","rcode":"NOERROR","answers":[],"srcaddr":"172.31.28.35","srcport":"45490","transport":"UDP","srcids":{"instance":"i-0123456789abcdef0"}}
{"version":"1.100000","account_id":"012345678910","region":"us-east-1","vpc_id":"vpc-0123456789abcdef","query_timestamp":"2022-11-02T08:14:20Z","query_name":"ec2-1-2-3-4.compute-1.amazonaws.com.","query_type":"A","query_class":"IN","rcode":"NOERROR","answers":[{"Rdata":"1.2.3.4","Type":"A","Class":"IN"}],"srcaddr":"172.31.28.35","srcport":"52635","transport":"UDP","srcids":{"instance":"i-0123456789abcdef0"}}
{"version":"1.100000","account_id":"012345678910","region":"us-east-1","vpc_id":"vpc-0123456789abcdef","query_timestamp":"2022-11-02T08:14:20Z","query_name":"ec2-1-2-3-4.compute-1.amazonaws.com.","query_type":"AAAA","query_class":"IN","rcode":"NOERROR","answers":[],"srcaddr":"172.31.28.35","srcport":"43134","transport":"UDP","srcids":{"instance":"i-0123456789abcdef0"}}
{"version":"1.100000","account_id":"012345678910","region":"us-east-1","vpc_id":"vpc-0123456789abcdef","query_timestamp":"2022-11-02T08:14:30Z","query_name":"ec2-1-2-3-4.compute-1.amazonaws.com.","query_type":"A","query_class":"IN","rcode":"NOERROR","answers":[{"Rdata":"1.2.3.4","Type":"A","Class":"IN"}],"srcaddr":"172.31.28.35","srcport":"58485","transport":"UDP","srcids":{"instance":"i-0123456789abcdef0"}}

Hi,

coming from the recent blog article (https://www.dynatrace.com/news/blog/accelerate-your-cloud-journey-with-dynatrace-observability-for-aws-s3-logs/) and reading "or Log Monitoring Classic" I assumed the forwarder can also be used for managed DT instances.

But reading the dynatrace.py it seems only API URIs from SaaS live instances are supported.
A Managed instance would produce a POST API string like "companydomain/api/v2/logs/ingest" which would obviously not work I guess.

Adding this as Feature request as I am not sure if that was in the scope of the forwarder, but I think adding support for Environment ActiveGate URIs would not be a lot of work.

Best regards,
Johannes

Currently, users need to manually replicate Lambda container images from the ECR public repository to a private ECR repository in the region where the dynatrace-aws-s3-log-forwarder is deployed before they can deploy it via CloudFormation. Users also need to manually replicate images as new versions are released before they can be deployed.

We could simplify deployment and management of versions by automating image replication with CodeBuild (e.g. https://github.com/aws-samples/amazon-ecr-replication-for-pub-container-images)