Giter VIP home page Giter VIP logo

cloudwatch-logs-customize-alarms's Introduction

Cloudwatch Logs Customize alarms

Package cloudwatch-logs-customize-alarms

Copyright 2016- Amazon.com, Inc. or its affiliates. All Rights Reserved.

Introduction

When you get an alarm, you want enough information to decide whether it needs immediate attention or not. You also want to customize the alarm text to operational needs. The CloudWatch Logs Customize Alarms is a Lambda function that helps in reading the logs from CloudWatch Logs during an alarm and send a customized email through SES.

Flow of Events

Flow of events

Setup Overview

Lambda function is written in Node.js. We do have a dependency on the latest aws sdk which includes the metrics to logs feature. Hence we create a deployment package. You can create a new Lambda function, and copy the code in index.js from this repository to your function. See 'Configurable parameters' section below.

Pre-requisite

  • CloudWatch Logs has a Log group with a metric filter.
  • A CloudWatch Alarm is created to trigger when that metric exceeds a threshold.
  • SES domain is created and verified

Triggers

  • The Lambda function is triggered on a SNS event.
  • You need to provide the SNS topic.

Authorization

Since there is a need here for various AWS services making calls to each other, appropriate authorization is required. This takes the form of configuring an IAM role, to which various authorization policies are attached. This role will be assumed by the Lambda function when running. The below two permissions are required:

1.CloudWatch Logs permits Lambda to call describeMetricFilters and filterLogEvents api. Note that we have given full CloudWatch Logs access, but it is recommended that you only give permissions to call specific api's.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

2.SES permits Lambda to send a customized email. Note that we have given full SES access, but it is recommended that you only give permissions to send an email.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ses:*"
      ],
      "Resource": "*"
    }
  ]
}

Lambda Function

Configurable parameters:

  1. Destination - The destination email address where email needs to be send.
  2. Source - The source email address sending the email.

For more information visit the SES documentation here

Instructions:

  • Handler: The name of the main code file. In this example, we have used index as the name of the handler.
  • When the metric (from the logs metric filters) reaches a threshold an alarm is triggered.
  • Once Alarm invokes the SNS topic, Lambda function is invoked and it reads the metricName and metricNamespace from the alarm.
  • It then calls describeMetricFilters to get the filterPattern.
  • Then Lambda calls filterLogEvents to get the relevant logs.
  • SES uses those filtered logs and additional customizations to send an email.

Lambda Configuration

This Lambda function was created with runtime Node.js 4.3. It has been tested with 128 MB and 3 seconds timeout. No VPC was used. You can change the configuration based on your testing.

Getting started

  1. Download the zip file located at dist/customize-alarms.
  2. Unzip the file. You will see an index.js file and node_modules folder. Index.js is the Lambda function and node_mdoules contain the specific version of AWS SDK.
  3. Open index.js in editor of your choice and add the information as specified in the 'Configuration parameters'.
  4. Once done with the changes in Lambda, zip the node.js file and node_modules folder.
  5. Upload the zip file to the Lambda function.
  6. For ease of use, the index.js is also copied in the main directory here.

Known Limitation

The Lambda function has the following limitation:

  • The nextToken in filterLogEvents api call is ignored. For production use, the lambda function should be modified to query filterLogEvents again with the nextToken received from the previous call. We need to continue querying until we get a null nextToken. More information on the api can be found here.

License Summary

This sample code is made available under the MIT license.

cloudwatch-logs-customize-alarms's People

Contributors

hyandell avatar jamesiri avatar jpeddicord avatar nikunjnaws avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cloudwatch-logs-customize-alarms's Issues

AWS python SDK getting invoked more than once with scheduled cloudwatch event

This issue is with respect to another issue I had logged a few days ago [https://github.com//issues/292].

I am invoking a lambda from another lambda using python SDK. Both the lambdas belong to the same VPC. The trigger lambda only contains a python script that invokes the second lambda (loader_development) and the code for that is as under:

from __future__ import print_function

import json
import logging
import os
from urllib2 import urlopen,Request,HTTPError
import boto3
import botocore.config

logger = logging.getLogger()
logger.setLevel(logging.INFO)
region = os.environ['AWS_REGION']
cfg = botocore.config.Config(retries={'max_attempts': 0})
def lambda_handler(event, context):
    logger.info('starting loading')
    invokeLambda = boto3.client('lambda', region_name = 'us-east-1', config=cfg)  
    request = {'resource':'/bucketstatus/latest','path':'/bucketstatus/latest','httpMethod':'GET', 'requestContext': {'requestId': context.aws_request_id,'identity': {'userArn': context.invoked_function_arn}}}
    invoke_response = invokeLambda.invoke(FunctionName='loader_development',
                                           InvocationType='RequestResponse',
                                           Payload=json.dumps(request))
    print(invoke_response['Payload'].read())

logger.info('Process Complete')

But, "starting loading" is printing thrice so I think this script is getting invoked thrice. Previously only the loader_development lambda was being retried for which I set the value of max retry attempts to 0. (FYI: I have a cloudwatch event enabled for my trigger cron(0/30 7-9 * * ? *) so does the cloudwatch event also have a retry mechanism?) How do I disable this cloudwatch event from triggering my script more than once?

P.S. This code works if I don't have a scheduled cloudwatch event.

SMTP Option

I would love to see an SMTP option so this could be used in places where SES is not easily accessible like GovCloud.

describeMetricFilters() returns empty response


import * as AWS from 'aws-sdk';
var cwl = new AWS.CloudWatchLogs();

....


   var requestParams = {
      metricName: message.Trigger.MetricName,
      metricNamespace: message.Trigger.Namespace
    };
    console.log('describeMetricFilters', requestParams);
    const logEvents = await cwl.describeMetricFilters(requestParams, function(err, data) {
        if(err) {
          console.log('Error is:', err);
        } else {
          console.log('Metric Filter data is:', data);
          getLogsAndSendEmail(message, data);
        }
    }).promise();
    console.log("logEvents...", logEvents);

Log in CloudWatch
INFO Metric Filter data is: { metricFilters: [] }

CloudWatchLogs filterLogEvents empty events

I am trying to get CloudWatchLogs filterLogEvents by sending parameters. some times it work and sends me the exact logs in that time, but for the different lambda function it returns empty events with next token.

parameters = {

'logGroupName' : metricFilter.logGroupName, 'filterPattern' : metricFilter.filterPattern ? metricFilter.filterPattern : "", 'startTime' : timestamp - offset, 'endTime' : timestamp };

i am retrying by sending parameters with next token but its still getting empty events. Does anyone has idea about it?

{
"events": [],
"searchedLogStreams": [],
"nextToken": "long text"
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.