It is possible this function could be wrapped (hopefully accidentally) by the Python agent. The log ingestion function should make sure the agent is shutdown if detected.

Current developer documentation (e.g. Developer.pm) references pipenv, which the project appears to have moved to Poetry. This is confusing to new developers working on the project. I managed to figure that out, but I have no idea how many other parts of this document are incorrect.

When HTTP POST request causes errors other than aiohttp.ClientResponseError, such as aiohttp.TimeOutError, the function doesn't retry and outputs the following error trace and Log entry sent message.

[ERROR] TimeoutError
Traceback (most recent call last):
  File "/var/task/function.py", line 510, in lambda_handler
    asyncio.run(_send_log_entry(log_entry, context))
  File "/var/lang/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/var/lang/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/var/task/function.py", line 222, in _send_log_entry
    return await asyncio.gather(*requests)
  File "/var/task/function.py", line 228, in _send_payload
    status, url = await http_post(
  File "/var/task/function.py", line 135, in http_post
    resp = await session.post(url, data=data, headers=headers)
  File "/var/task/aiohttp/client.py", line 466, in _request
    await resp.start(conn)
  File "/var/task/aiohttp/client_reqrep.py", line 803, in start
    self._continue = None
  File "/var/task/aiohttp/helpers.py", line 603, in __exit__
    raise asyncio.TimeoutError from None

Here is the whole log for this invocation. It looks the request succeeded for the log endpoint but failed for the infra endpoint.

START RequestId: ***(masked)*** Version: $LATEST
Log entry sent. Response code: 202. url: https://log-api.newrelic.com/log/v1
[ERROR] TimeoutError
END RequestId: ***(masked)***
REPORT RequestId: ***(masked)***	Duration: 3821.20 ms	Billed Duration: 3822 ms	Memory Size: 128 MB	Max Memory Used: 77 MB	

When session.post raises an error such as aiohttp.TimeOutError, except block doesn't handle this error. It means retry logic doesn't work.

The http_post method raises an error and this error isn't handled anywhere and goes to out of the function. It means lambda invocation fails and the function will be retried if the retry is enabled.

Could you add more except blocks for such errors as TimeOutError?

Hello,

Logs sent to NR via this lambda are not related to any specific APM. Is there a way for this lambda to set the appropriate entity.guid, entity.guids, and entity.name fields in the log JSON so that logs may be related/tied to a specific APM? (I believe these fields would accomplish that linking?)

(I understand using an infrastructure agent typically accomplishes this, though we have reasons for shipping logs separately)

Question.

Hi there, not sure if this is the correct place to ask this, but, I'd like to know how should be the accurate way to set a specific version to the source property in the configuration of the module.

for example I'ld like point to the version 2.8.0 which uses python 3.9.
When I set the source property pointing to "github.com/newrelic/aws-log-ingestion", it takes the latest version of the module which uses with python 3.11.

module "newrelic_log_ingestion" {
  source             = "github.com/newrelic/aws-log-ingestion"
  nr_license_key     = "{{YOUR_LICENSE_KEY}}"
}

Thanks in advance.

im trying to add some custom data to our lambda monitoring and its not working.

the event itself is reflecting in the lambda console but the custom events are not appearing in insights.
this is a pic of the customevent property in the logs of the event object.

I use two regions AWS.

• main: ap-notheast-1
• secondary: us-east-1

How do I deploy log-ingestion to each region by Terraform ?

my terraform

I specified providers in the module "newrelic_log_ingestion_virginia" , but warnings occur when executing the plan.
Also, in executing apply, referenced to the main region.

provider "newrelic" {
  account_id = var.newrelic_account_id
  api_key    = var.newrelic_api_key
  region     = "US"
}
provider "aws" {
  region = "ap-northeast-1"
}
provider "aws" {
  region = "us-east-1"
  alias  = "virginia"
}
module "newrelic_log_ingestion" {
  source                       = "github.com/newrelic/aws-log-ingestion"
  nr_license_key               = var.newrelic_license_key
}
module "newrelic_log_ingestion_virginia" {
  source                       = "github.com/newrelic/aws-log-ingestion"
  nr_license_key               = var.newrelic_license_key
  providers = {
    aws = aws.virginia
  }
}

I think to add required_providers to aws-log-ingestion/terraform.tf.

• For example, like this

  terraform {
    required_version = ">= 0.14, < 2.0"
  +  required_providers {
  +    aws = {
  +      source = "hashicorp/aws"
  +    }
    }
  }

Is there another way?

The documentation still says "AWS Serverless Application that sends log data from CloudWatch Logs and S3 to New Relic Infrastructure - Cloud Integrations. "

But when I hooked it up to S3 events, I'm getting an error message in the lambda log.

Is there a different solution for S3?

The external dependencies required by NR logs ingestion lambda are only supported in python 3.7 (pep517, aio packages etc). I have a Jenkins pipeline using alpine image with Python 3.8. This will cause Jenkins to fail creating correct deployable package for AWS Lambda, and lambda would eventually fail.

Hi there, I went to deploy this stack today and got the following error:

(thrown when creating the ConfigRole resource)

Policy arn:aws:iam::aws:policy/service-role/AWSConfigRole does not exist or is not attachable. (Service: AmazonIdentityManagement; Status Code: 404; Error Code: NoSuchEntity; Request ID: 7a7836dd-0c87-4dca-85cb-6e94da045748; Proxy: null)

This probably caused the second error:

(thrown when creating the DeliveryChannel resource)

Configuration recorder is not available to put delivery channel. (Service: AmazonConfig; Status Code: 400; Error Code: NoAvailableConfigurationRecorderException; Request ID: 0c9fa708-2bc4-43bf-a0e5-9f64290c1004; Proxy: null)

I confirmed that the ConfigRole resource is indeed trying to access

Resources:
  # ...
  ConfigRole:
    Type: AWS::IAM::Role
    Condition: ShouldCreateConfig
    Properties: 
      AssumeRolePolicyDocument: 
        Version: 2012-10-17
        Statement: 
          - Effect: Allow
            Principal:
              Service:
              - config.amazonaws.com 
            Action: 
              - 'sts:AssumeRole'
      Path: /service-role/
      ManagedPolicyArns: 
        - arn:aws:iam::aws:policy/service-role/AWSConfigRole
      Policies:
        - PolicyName: ConfigService-S3Access
          PolicyDocument: 
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: 
                  - 's3:PutObject'
                  - 's3:PutObjectAcl'
                Resource: !Join ['', ['arn:aws:s3:::', !Ref ConfigS3Bucket, '/AWSLogs/', !Ref AWS::AccountId, '/*']]
      Description: Role to allow Config Service communicate with Delivery Channel
      RoleName: !Join ['-', ['ConfigRole', !Select [0, !Split ['-', !Select [2, !Split ['/', !Ref 'AWS::StackId' ]]]]]]

I noticed that the managed policy referenced in the policy does not exist in our AWS account. Could it be because our account is part of an AWS Organization?

As a workaround, I'll copy the YAML for this stack directly into the CDK app we use to deploy, and simply change the policy name to the AWS Organizations version.

The value is passed as a variable and possibly stored in the terraform state. This is a lambda after all and it can read from the secret manager instead.

Desired Behavior

Have secret_arn as input that can be used instead of license key value.

Possible Solution

Use ARN when available, and fallback to secret value input.

Wasn't sure how to issue you a README.md PR. Provisioned this with AWS CDK v2 using TypeScript

import * as cdk from 'aws-cdk-lib';
import { aws_sam, aws_iam } from 'aws-cdk-lib';

new aws_iam.Role(scope, `ServiceRoleNewRelicLogIngestionLambda`, {
  assumedBy: new aws_iam.ServicePrincipal(`lambda.amazonaws.com`),
  roleName: `NewRelicLogIngestionRole`,
  managedPolicies: [
    aws_iam.ManagedPolicy.fromAwsManagedPolicyName(`service-role/AWSLambdaBasicExecutionRole`),
  ],
})

const samApp:aws_sam.CfnApplication = new aws_sam.CfnApplication(scope,`LambdaNewRelicLogIngestion`, {
  location: {
    applicationId   : `arn:aws:serverlessrepo:us-east-1:463657938898:applications/NewRelic-log-ingestion`,
    semanticVersion : `2.6.4`
  },
  parameters: {
    `DebugLoggingEnabled`               : `False`,
    `FunctionRole`                      : `NewRelicLogIngestionRole`,
    `LogIngestionFunctionArnExportName` : 'LogIngestionFunctionArn',
    `MemorySize`                        : `128`,
    `NRInfraLogging`                    : `True`,
    `NRLicenseKey`                      : `1a2b3c4d5e6f7g8h9i10j11k12l13m14no15p16q`,
    `NRLoggingEnabled`                  : `True`,
    `Timeout`                           : `30`
  }
})

When logs are shipped via the Extension, logs show up. I believe it is due to the addition of the faas.execution attribute, but it would be good to confirm what the requirements are for the query on the Logs UI page.
https://github.com/newrelic/newrelic-lambda-extension/blob/dd3683d3bfe7473db35fa3dec24b159b009e6505/telemetry/request.go#L33-L45

The log ingestion function is not currently setting faas.execution.

The result is that logs are not displayed on the Logs page in the Lambda UI.

I am using the terraform module for newrelic log ingestion and it deploys fine to the first region, but when trying to deploy to a second region, I get the following error:

Error: failed creating IAM Role (newrelic-log-ingestion): EntityAlreadyExists: Role with name newrelic-log-ingestion already exists

I have tried changing the service_name to include the region, but then I get this error:

Error: unable to load "temp/newrelic-log-ingestion.zip": open temp/newrelic-log-ingestion.zip: no such file or directory

Even changing the service_name, lambda_image_name, and lambda_archive properties to match, it still complains that it cannot find the .zip file.

What needs to be done to allow multi-region deployment through terraform?

Customers environment is an internal app platform (running on AWS) with the policy that all resources need to be placed and run within a VPC. Hence there is the need to deploy the lambda function (manual install) into a VPC.

Currently, the config settings available do not allow to specify VPC settings.

In addition to specify VPC settings, their requirement is also that all outbound communication goes through a PROXY.
So the ability to specify PROXY settings is related and required as well - please let me know if I should create a separate issue for this.

IMHO src/requirements.txt should be version controlled OR build.sh should by part of the build process

Step to reproduce:

• add/update dependency
• git push
• trigger a new deployment using terraform.tf in CICD pipeline
• null_resource builds docker image -> copy src/requirement.txt

This will always be the version of requirements.txt currently included in the repo. As the file is ignored by git (.gitignore). The script build.sh is never called.

Hey friends,

I tested this lambda function recently and it seems that the API is no longer supported by New Relic.
The service at https://infra-api.newrelic.com/integrations/aws returns either 400 or 500, even by using a valid license key. When I try with an invalid key i get: Invalid license key.

Do you know it this endpoint is still operational?
Would you have by chance some valid payload i could use to test this endpoint by curling it?
Or else, do you have a documentation about this REST API? I could not find it anywhere on New Relic portal.

Thanks a lot in advance
Julio Avero

I have installed aws-log-ingestion and it is forwarding my logs from CloudWatch to New Relic. So, I have that part working. My issue is that, even when I log valid JSON, New Relic does not parse it because CloudWatch adds on a prefix to the logged string.

So, if I log a JSON string from my AWS Lambda, I get this in New Relic logs 2020-11-03T02:52:20.873Z ca92741f-d3bd-595a-adda-2c051ba23d5a INFO {<JSON Content>} and the JSON isn't parsed.

Is there a way to get around this?

Many thanks for this project!

FinTech customers have some common restrictions:

• Cannot use code from AWS Public Serverless Repo
• Cannot run SAM templates that reference containers sourced from public repos

Would be helpful if as part of the release artifacts; New Relic staged the Lambda Deployment Zip package on github releases.

Hi,

Please consider adding support for cross-stack CloudFormation imports & exports.

Use case:
In order to subscribe the CloudWatch LogGroup to a lambda - the DestinationArn must be provided. If an existing stack exports this value - then the ARN can be imported from the consumer’s CF code directly, by referencing the export via its Name.

Adding support:
Given the provided CF template.yaml, enabling the support could be enabled by adding the following Outputs section:

Outputs:
  LogIngestionFunctionArn:
    Condition: NoRole
    Description: Log ingestion lambda function ARN
    Value: !GetAtt NewRelicLogIngestionFunction.Arn
    Export:
      Name: !Sub "${AWS::StackName}-LogIngestionFunctionArn"
  LogIngestionFunctionNoCapArn:
    Condition: NoCap
    Description: Log ingestion capless lambda function ARN
    Value: !GetAtt NewRelicLogIngestionFunctionNoCap.Arn
    Export:
      Name: !Sub "${AWS::StackName}-LogIngestionFunctionArn"

Thank you

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

• Fail
  • ❌ readme-starts-with-community-plus-header
  • ❌ readme-contains-link-to-security-policy
  • ❌ readme-contains-forum-topic
• Warning
  • ⚠️ third-party-notices-file-exists
• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ code-of-conduct-should-not-exist-here

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

• README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
  • 🔨 Suggested Fix: prepend the latest code snippet found at https://github.com/newrelic/opensource-website/wiki/Open-Source-Category-Snippets#code-snippet-2 to file

❌ readme-contains-link-to-security-policy #

Doesn't contain a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

❌ readme-contains-forum-topic #

Doesn't contain a link to the appropriate forum.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate forum.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

Warning #

Passed #

Hello,

The documentation for this says:

NR_TAGS
Specify tags to be added to all log events.

Added how? Does it actually mean attributes rather than tags?

(And is there equivalent functionality for the Kinesis Firehose integration? What, if anything, do the "Parameters" that can be configured for the integration do?)

The change here, which adds AWS::StackId suffix to the lambda name (#88) breaks several of the commands in the newrelic-lambda-cli. See: newrelic/newrelic-lambda-cli#240 for details.

-----------------------------------------------------
  on .terraform/modules/newrelic_log_ingestion/terraform.tf line 113, in resource "aws_iam_role" "lambda_role":
 113:   count = var.function_role == null ? 1 : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

Usage:

module "newrelic_log_ingestion" {
  source             = "github.com/newrelic/aws-log-ingestion"
  nr_license_key     = data.aws_secretsmanager_secret_version.newrelic_api_key_secret.secret_string
  nr_logging_enabled = true
  function_role      = aws_iam_role.newrelic_lambda_role.arn
}

It looks like there might be a bug in https://github.com/newrelic/aws-log-ingestion/blob/master/src/function.py when reading s3.

The stack trace looks like:

Traceback (most recent call last):
File "/var/task/function.py", line 496, in lambda_handler
_send_log_entry(log_line, context)
File "/var/task/function.py", line 203, in _send_log_entry
entry_type = _get_entry_type(log_entry)
File "/var/task/function.py", line 309, in _get_entry_type
if '"logGroup":"/aws/vpc/flow-logs"' in log_entry:
TypeError: a bytes-like object is required, not 'str'

Steps to reproduce:

• Hook up the lambda to an S3 bucket (I'm not sure it matters which events you choose to trigger on, as long as there's data in the bucket)
• Add a file to the bucket
• Check the output of this Lambda

this is the error received when installing from the aws serverless application repository

In case of aws-log-ingestion lambda function errors, the logs won't be shipped into New Relic.

Lambda function has an error handling feature and we can store logs in other places.

https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#invocation-async-errors

It's good to add error handling feature into deployment template.

When Attempting to use version 2.4.0 of the NR SAM template I have recieved an error that the permissions boundary is not set.

Looking at the cloudformation template code has the following set in the main function
PermissionsBoundary: !Ref PermissionsBoundary

Work will be needed on ensuring that people can use the SAM template in new acconts