AWS::Logs::MetricFilter does not accept dimensions about aws-cloudformation-resource-providers-logs HOT 31 CLOSED

Deployment beginning shortly. I will update here once completed.

from aws-cloudformation-resource-providers-logs.

@waifa Any updates on this? 🙏

from aws-cloudformation-resource-providers-logs.

We have been working on issues related to deployment and traffic migration. Stay tuned to update which I should be able to provide by End of July

from aws-cloudformation-resource-providers-logs.

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

from aws-cloudformation-resource-providers-logs.

@waifa Hey there! Curious if there are any updates on this? Thanks!

from aws-cloudformation-resource-providers-logs.

The dimension value needs to be a value matched in the filterPattern. Jyoti's problem was fixed by changing the value of the dimension in his template to "$.AZ".

Can you support arbitrary values for dimension name? In my case the dimension name is not present in the log event and I need to specify it manually for the filter.

from aws-cloudformation-resource-providers-logs.

The deployment from February failed unfortunately. We are currently working on a fix, thanks for your patience!

from aws-cloudformation-resource-providers-logs.

AWS::Logs::MetricFilter now accepts dimensions!

from aws-cloudformation-resource-providers-logs.

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

@alancnet I agree with you and this is also something I raised internally, which is also pending prioritization.

from aws-cloudformation-resource-providers-logs.

Working on this feature request. Support should be available shortly.

from aws-cloudformation-resource-providers-logs.

@waifa can you advise on the status of this?

from aws-cloudformation-resource-providers-logs.

This is a duplicate of aws-cloudformation/cloudformation-coverage-roadmap#419

from aws-cloudformation-resource-providers-logs.

Any updates?

from aws-cloudformation-resource-providers-logs.

This seems to be live (at least defined) in the CloudFormation specification for us-east-1: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

from aws-cloudformation-resource-providers-logs.

This seems to be live (at least defined) in the CloudFormation specification for us-east-1: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

I just checked in eu-central-1: It's not live as of now. However, it's possible to use the Unit property which is not yet covered in the CloudFormation documentation.

from aws-cloudformation-resource-providers-logs.

The documentation description for the Dimensions property is a little weird, but I've managed to successfully use it in ap-northeast-1

from aws-cloudformation-resource-providers-logs.

@KirtiKunalShah You mentioned there might be an update around this time. Anything you can share with us on current progress?

from aws-cloudformation-resource-providers-logs.

@waifa Can you help with a sample of how dimensions work? Looks like this is available in us-west-2 and i can consume it.
However, i get this error in Cfn Resource handler returned message: "Invalid request provided: AWS::Logs::MetricFilter" (RequestToken: abc41535-8432-1c18-e4ad-45438e510ff2, HandlerErrorCode: InvalidRequest)

Can you help with an example ?

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "CloudFormation template configuring AWS EKS Dataplane AZ metrics log groups",
    "Resources": {
        "KubeApiServerAzHealthzAPAusiw1az1MetricFilter1": {
            "Type": "AWS::Logs::MetricFilter",
            "Properties": {
                "LogGroupName": "abc",
                "FilterPattern": "{ $.xyz < 0.9 && $.AZ = usiw1-az1 }",
                "MetricTransformations": [{
                    "MetricValue": "1",
                    "MetricNamespace": "something",
                    "MetricName": "something",
                    "Dimensions": [{
                      "Key": "ab",
                      "Value": "de"
                    }]
                }],
            }
        }
    }
}

from aws-cloudformation-resource-providers-logs.

The dimension value needs to be a value matched in the filterPattern. Jyoti's problem was fixed by changing the value of the dimension in his template to "$.AZ".

from aws-cloudformation-resource-providers-logs.

Announcement: Dimensions and units are available in almost all regions. There are a select few regions that still need some work to make this feature available(like ap-northeast-3)!

I will close this issue once it is available in all regions.

from aws-cloudformation-resource-providers-logs.

Announcement: Dimensions and units are available in almost all regions. There are a select few regions that still need some work to make this feature available(like ap-northeast-3)!

As of now, it is not available in eu-central-1:

CREATE_FAILED - Encountered unsupported property Dimensions

from aws-cloudformation-resource-providers-logs.

It used to work in eu-central-1

Now our stacks no longer deploy:

12:40:53 | CREATE_FAILED | AWS::Logs::MetricFilter | MetricFilter
Encountered unsupported property Dimensions

from aws-cloudformation-resource-providers-logs.

Sincere apologies from AWS for this inconsistency. A mistake on our end caused the feature to be rolled back in eu-central-1 for a week and a couple other regions for a short period of time.

from aws-cloudformation-resource-providers-logs.

@waifa Do you know if it's on the roadmap to include a dimension value that is not in the filterPattern? For example, I want the Lambda function name to be a dimension but it's often not in the log and thus is not extracted by the filterPattern.

from aws-cloudformation-resource-providers-logs.

@msambol

@waifa Do you know if it's on the roadmap to include a dimension value that is not in the filterPattern? For example, I want the Lambda function name to be a dimension but it's often not in log and thus is not extracted by the filterPattern.

I raised this issue internally and trying to get it prioritized, but it's not currently on any roadmap.

from aws-cloudformation-resource-providers-logs.

Are you guys 100% sure that this is working in eu-central-1?

Resource handler returned message: "Invalid request provided: AWS::Logs::MetricFilter.
The specified filter pattern does not support dimensions
(Service: CloudWatchLogs, Status Code: 400, Request ID: a2271e5d-6464-4bfc-84db-XXXXXXXX)"
(RequestToken: 7b3cce6f-d6f0-8718-beb2-XXXXXXXX, HandlerErrorCode: InvalidRequest)

Example:

  ECSLogGroupFilter:
    Type: AWS::Logs::MetricFilter
    Properties: 
      FilterPattern: "?ERROR ?WARN"
      LogGroupName: !Ref ECSLogGroup
      MetricTransformations: 
        - MetricNamespace: "MetricNamespace"
          MetricName: "MetricName"
          Dimensions:
            - Key: "ClusterName"
              Value: !Ref ECSCluster
            - Key: "ServiceName"
              Value: !GetAtt ECSService.Name
          MetricValue: 1
          Unit: "Count"

This is working, by the way:

  ECSLogGroupFilter:
    Type: AWS::Logs::MetricFilter
    Properties: 
      FilterPattern: "?ERROR ?WARN"
      LogGroupName: !Ref ECSLogGroup
      MetricTransformations: 
        - MetricNamespace: "MetricNamespace"
          MetricName: "MetricName"
          MetricValue: 1
          Unit: "Count"

from aws-cloudformation-resource-providers-logs.

@r-heimann

Are you guys 100% sure that this is working in eu-central-1?

Yes, it should be working in eu-central-1.

I believe your Metric Filter is returning an error because you are using a text-based filter pattern (as opposed to space-delimited or JSON). Text-based filter patterns do not support dimensions.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html#logs-metric-filters-dimensions:~:text=publish%20dimensions%20with%20metrics%20that%20JSON%20and%20space%2Ddelimited%20log%20events%20generate

from aws-cloudformation-resource-providers-logs.

Yes, it should be working in eu-central-1.

I believe your Metric Filter is returning an error because you are using a text-based filter pattern (as opposed to space-delimited or JSON). Text-based filter patterns do not support dimensions.

I see, i have no idea why such a limitation exists in the first place but i'll try to get it working using your link.

from aws-cloudformation-resource-providers-logs.

I raised this issue internally and trying to get it prioritized, but it's not currently on any roadmap.

@gordonpn it would be great to include this feature on roadmap for the use cases it can support :)

from aws-cloudformation-resource-providers-logs.

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

Raised this issue since this issue is not solved still.

from aws-cloudformation-resource-providers-logs.

Any updates on this?

from aws-cloudformation-resource-providers-logs.