The CloudFormation Resource Provider Package For Amazon CloudWatch Logs
Home Page: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html
License: Apache License 2.0
The Handler Contract states that we should fail and return NotFound on Delete, and then the Client should ignore this error. This is to provide consistency across different implementations
The Property KmsKeyId is misleading as its looking for an Arn not a Key Id. This also is a break to naming conventions in cloudformation
The optional Unit property of a MetricFilter can be specified when manually creating the filter, but there is no way to specify it in CloudFormation for the MetricFilter.
I found that Terraform also supports this attribute in their CloudWatch Log Metric Filter resource and am wondering if CloudFormation would support doing the same. Otherwise, if this attribute is not a priority, could I get some clarification as to why?
I'm trying to get the LogGroup provider working locally, and created a test event to do that (see https://github.com/ikben/aws-cloudformation-resource-providers-logs/blob/try-to-get-things-working/aws-logs-loggroup/sam-tests/delete.json but with the credentials filled).
However, when I run sam local invoke TestEntrypoint --event sam-tests/delete.json I get the following error (there was an earlier error because the handler in template.yaml was wrong, but I fixed that):
{"status":"FAILED","errorCode":"InternalFailure","message":"Credentials must not be null.","callbackDelaySeconds":0}
I am probably doing something wrong, but I can't find what it is. Does the provider need it's own credentials outside of the ones supplied in the event?
Full output:
~$ sam local invoke TestEntrypoint --event sam-tests/delete.json
Invoking software.amazon.logs.loggroup.HandlerWrapper::testEntrypoint (java8)
Decompressing /Users/ben/src/vendor/aws-cloudformation-resource-providers-logs/aws-logs-loggroup/target/aws-logs-loggroup-handler-1.0-SNAPSHOT.jar
Fetching lambci/lambda:java8 Docker container image......
Mounting /private/var/folders/pq/cvfgj3_s0c565_sc9rwywd0w0000gn/T/tmpoh2fn5y6 as /var/task:ro,delegated inside runtime container
START RequestId: 9211bbbd-82c0-1911-5360-628b0a13acdf Version: $LATEST
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[DELETE] invoking handler...Failed to execute remote function: {Credentials must not be null.}java.lang.NullPointerException: Credentials must not be null.
at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:117)
at software.amazon.awssdk.auth.signer.params.Aws4SignerParams.<init>(Aws4SignerParams.java:42)
at software.amazon.awssdk.auth.signer.params.Aws4SignerParams$BuilderImpl.build(Aws4SignerParams.java:206)
at software.amazon.awssdk.auth.signer.internal.BaseAws4Signer.sign(BaseAws4Signer.java:34)
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.signRequest(SigningStage.java:63)
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:49)
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:114)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:87)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:63)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:81)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:61)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:198)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:122)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:148)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:102)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.deleteLogGroup(DefaultCloudWatchLogsClient.java:603)
at software.amazon.cloudformation.proxy.AmazonWebServicesClientProxy.injectCredentialsAndInvokeV2(AmazonWebServicesClientProxy.java:328)
at software.amazon.logs.loggroup.DeleteHandler.handleRequest(DeleteHandler.java:22)
at software.amazon.logs.loggroup.DeleteHandler.handleRequest(DeleteHandler.java:11)
at software.amazon.logs.loggroup.HandlerWrapper.invokeHandler(HandlerWrapper.java:76)
at software.amazon.logs.loggroup.HandlerWrapper.testEntrypoint(HandlerWrapper.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:354)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:906)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:341)
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:63)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:119)
END RequestId: 9211bbbd-82c0-1911-5360-628b0a13acdf
REPORT RequestId: 9211bbbd-82c0-1911-5360-628b0a13acdf Init Duration: 2852.11 ms Duration: 2140.07 ms Billed Duration: 2200 ms Memory Size: 128 MB Max Memory Used: 78 MB
{"status":"FAILED","errorCode":"InternalFailure","message":"Credentials must not be null.","callbackDelaySeconds":0}
The MetricFilter has much higher latency, it seems the call chain make an unnecessary stabilization, but this resource needs no stabilization.
The default stabilization delay is 5 seconds, we should remove such 5 seconds unnecessary latency.
I should be able to hard code a dimension (or use Fn::Sub) to specify a Dimension value, rather than always pull it from a log line.
Use cases:
This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)
Currently if you try do do this you get:
metricName: `MemoryUsed`,
metricValue: "$some_value",
dimensions: {
FunctionName: `${node.ref}`
},
Resource handler returned message "invalid request provided: AWS::Logs::MetricFilter. Invalid metric transformation: dimension values must be valid selector.
So instead I need to inject the lambda name into the metric name - This makes looking for the metric and reporting on it a miserable experience, but to get it to works I have to do this :
metricName: `MemoryUsed-${node.ref}`,
metricValue: "$some_value",
AWS::Logs::ResourcePolicy
As with most other resource policies and identity policies, you're able to define these in YAML within a CloudFormation template. However, with the AWS::Logs::ResourcePolicy resource, you have to define the JSON string within your CloudFormation, which is less ideal. For instance, this example resource will fail with an error message Properties validation failed for resource MyLogGroupPolicy with message: #/PolicyDocument: expected type: String, found: JSONObject:
MyLogGroupPolicy:
Type: AWS::Logs::ResourcePolicy
Properties:
PolicyName: MyLogGroupPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: someservice.amazonaws.com
Action: logs:PutLogEvents
Resource: !GetAtt MyLogGroup.ArnResource spec: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-resourcepolicy.html
AWS::IAM::Policy spec showing the definition of a policy in YAML: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-policy.html
If we look in the cloudformation docs for the log group resource, we see these allowed values in the retentionInDays field:
The number of days to retain the log events in the specified log group. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, and 3653.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html
However when I try to use value 2557 in a cloudformation template, I get an error:
Model validation failed (#/RetentionInDays: #: only 1 subschema matches out of 2)
#/RetentionInDays: failed validation constraint for keyword [enum] (#/RetentionInDays)
If we look in the source to see the actual schema, we see (for example) that 2557 does not appear in the list.
When I use the AWS console I can successfully choose 2557 (or rather, seven years). So it seems the schema in this repository is missing some entries.
Catching and rethrowing as CloudFormation specific exceptions, will improve retries. Ideally there should be no uncaught exceptions from the client.
There are more exceptions that can be thrown
Originally posted by @miparnisari in #27
The AWS::Logs::SubscriptionFilter Resources sets all its Properties as requiring replacement, making it impossible to update resources with 2 Subscription Filters.
Log Groups support a maximum of 2 Subscription Filters per Log Group. When you have 2 Filters attached to Log Group, and attempt to modify one of them, CloudFormation attempts to create a temporary 3rd Filter before deleting the existing one. As it's only possible to have 2 LogGroups, the 3rd Filter fails to create.
The PutSubscriptionFilter API supports updates (https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutSubscriptionFilter.html).
It would be useful if some of the Properties of a SubscriptionFilter could be updated without requiring the entire resource to be replaced. This would allow updates to filters when there are 2 filters attached to a LogGroup.
We are sucessfully creating/updating log groups in the handlers, but #21 removed a read call that set the arn in the returned model, so the models returned by the create and read calls differ, as so update & read. Found during contract tests:
assert {'Arn': 'arn:...onInDays': 60} == {'LogGroupName...onInDays': 60}
E Omitting 2 identical items, use -vv to show
E Left contains 1 more item:
E {'Arn': 'arn:aws:logs:us-west-2:123456789012:log-group:name*'}
E Full diff:
E + {'LogGroupName': 'name', 'RetentionInDays': 60}
E - {'Arn': 'arn:aws:logs:us-west-2:123456789012:log-group:name*',
E - 'LogGroupName': 'name,...
Need to either add back in the read call at the end of the handler or construct this arn in the create handler, although having to maintain logic for contstructing arns seems like overkill.
The following announcement makes dimensions available for metric filters.
However, it appears to be missing when specifying the resource through CloudFormation. Getting the following error on stack events console.
Encountered unsupported property Dimensions
The Python SDK documentation correctly shows this as available, but missing in CloudFormation docs for CloudWatchLogs.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.put_metric_filter
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html
Current properties according to the schema:
But according to the service API, CreateLogGroup also accepts:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.