aws-powertools / powertools-lambda-dotnet Goto Github PK

Fix issue with content for bug report template.

Expected Behaviour

We have a test project which references functions defined in their own projects/dlls. There seems to be issues with the static instances within the Metrics class.

We have forked the project and provided a sample program which causes a null reference exception.

The if (_instance != null) return; check means that the following lines do not run:

_instance = this;
_powertoolsConfigurations = powertoolsConfigurations;
_raiseOnEmptyMetrics = raiseOnEmptyMetrics;
_context = InitializeContext(nameSpace, service, null);

https://github.com/awslabs/aws-lambda-powertools-dotnet/blob/develop/libraries/src/AWS.Lambda.Powertools.Metrics/Metrics.cs#L61

I think that getting rid of the AspectInjector would simplify this project a lot. If the focus was on providing a library that does logging / metrics without all the magic we can choose how we want to use it ourselves.

I think the use of statics instances within the library makes thread safety a bit of a concern too.

Current Behaviour

N/A

Code snippet

https://github.com/jakejscott/aws-lambda-powertools-dotnet/blob/bug/metrics-issues/libraries/examples/AWS.Lambda.Powertools.Examples.MetricsFunction/Program.cs

Possible Solution

No response

Steps to Reproduce

https://github.com/jakejscott/aws-lambda-powertools-dotnet/blob/bug/metrics-issues/libraries/examples/AWS.Lambda.Powertools.Examples.MetricsFunction/Program.cs

AWS Lambda Powertools for .NET version

latest

AWS Lambda function runtime

dotnet6

Debugging logs

No response

Is this related to an existing feature request or issue?

No

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

When using an AWS Lambda function to process a batch of items from either Amazon SQS, Amazon Kinesis Data Streams or Amazon DynamoDB Streams, and the function fails during processing of any item within the batch, then the entire batch will be returned to the queue or stream, by default. This means that already successfully processed items risk being processed multiple times, which increases Lambda function invocation count, costs and the overall time it takes to consume items from one of these event sources.

By adding the Batch Processing Utility, we can bring support for simple utilization of the AWS Lambda function response type ReportBatchItemFailures where partial batch item failures can be reported, and thereby help reducing the number of items that are being re-processed. The utility will automatically monitor the processing of each item within a batch, and report which items failed to be processed. This enable developers to focus on writing business logic while benefiting from the Batch Processing Utility automatically doing the reporting of partial failures within a batch.

Use case

There are multiple use cases for adding the Batch Processing Utility to Lambda Powertools for .NET:

1. Enable developers to get "plug-n-play" support for automated reporting of partial failures when consuming batches from SQS Queues, Kinesis Data Streams and DynamoDB Streams.
2. By reporting partial failures, we can reduce the number of items being re-processed, leading to fewer Lambda function invocations, reduced cost, and faster consumption of events.
3. Improve feature parity across the different AWS Lambda Powertools projects.

Proposal

Key features include:

• Reduce the number of items being re-processed by automatically reporting partial failures when processing batches from Amazon SQS Queues, Amazon Kinesis Data Streams and Amazon DynamoDB Streams.
• Batch items are processed in isolation. One item failing processing will not cause the Lambda function to immediately fail.
• Ease of use (simple to setup and configure).
• Extensibility of the batch processing logic (i.e. by decorating the batch processor and/or deriving from it). This provides extension points for handling per-item failure/success, hooks for running code before and after a batch has been processed, as well as an option for decorating the per-item batch processing logic. All of this very similar to the extension points currently available in AWS Lambda Powertools for Python.
• Support for enabling and configuring parallel processing of items in a batch (to utilize multiple cores if available). This of course means that focus will also be on providing a thread-safe utility.

Out of scope

Integration with the Idempotency Utility
Even though reporting partial failures will reduce the number of items being re-processed, it cannot be guaranteed that successfully processed items are only processed once. To guard against this, it makes sense to look at the Idempotency Utility and see if we can provide simple utilization of that utility within the Batch Processing Utility. However, since the Idempotency Utility is not available yet, the use of this utility is considered out of scope for now.

Potential challenges

Event Source Data Classes
When processing a batch event, we should provide an object that describes the specific event, i.e., an SQSEvent. In Lambda Powertools for Python we have an Event Source Data Classes utility containing classes for different types of events, including SQSEvent, KinesisStreamEvent and DynamoDBStreamEvent. In the the AWS Lambda for .NET project we have classes defined for each of these events:

• Amazon.Lambda.SQSEvents (github, nuget)
• Amazon.Lambda.KinesisEvents (github, nuget)
• Amazon.Lambda.DynamoDBEvents (github, nuget)

While there are benefits to using the event types that are already defined and maintained (instead of having to replicate and maintain these in this project as well), it also means that there will be a dependency on another project / team which to some degree goes against "keeping it lean" and minimizing external dependencies (as per the AWS Lambda Powertools Tenets).

Required Configuration of Event Source Mapping
The utility will require that ReportBatchItemFailures is included in the FunctionResponseTypes list of the event source mapping configuration. We need to be clear on this in the documentation for AWS Lambda Powertools, as this could otherwise turn into a source of confusion.

Dependencies and Integrations

Dependencies
Depending on what direction we decide to go in terms of Event Source Data Classes (see Potential Challenges above), we might take a dependency on:

• Amazon.Lambda.SQSEvents (github, nuget)
• Amazon.Lambda.KinesisEvents (github, nuget)
• Amazon.Lambda.DynamoDBEvents (github, nuget)

Alternative solutions

Today, developers can implement support for reporting batch item failures by following the AWS Lambda documentation on "Reporting batch item failures" for each of the following event sources:

Amazon SQS
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting

Amazon Kinesis Data Streams
https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html#services-kinesis-batchfailurereporting

Amazon DynamoDB Streams
https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html#services-ddb-batchfailurereporting

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java, TypeScript

What were you initially searching for in the docs?
nothing specific. Just browsing through the API docs and looks like there are some classes that are exposed but not everything. Looking into the code looks like it's only generating docs for public classes and methods - which is fine - as long as it is coherent.

Example: IXrayRecorder is generated for Tracing.Internal namespace but the actual implementation is not.

Is this related to an existing part of the documentation? Please share a link
https://sturdy-umbrella-97433f53.pages.github.io/api/api/AWS.Lambda.Powertools.Tracing.Internal.IXRayRecorder.html is an example but there might be others.

Describe how we could make it clearer
Make it coherent across utilities.

If you have a proposed update, please share it here
Decide whether to expose only classes and APIs to be used by end customers (developers) or expose everything since the code is all open source either way.

Create API docs
Generate API docs via build

What were you initially searching for in the docs?

Update the tracer_utility_showcase.png so it aligns with our tracing example. The one that is there now is for a python function

Is this related to an existing part of the documentation? Please share a link

Describe how we could make it clearer

If you have a proposed update, please share it here

Expected Behaviour

Log events in CloudWatch with debug log level.

Current Behaviour

Log events are showing info log level despite setting the following environment variable in the Lambda fn.

POWERTOOLS_LOG_LEVEL: debug

Code snippet

Here is the code snippet from Function.cs file.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

using Amazon.Lambda.Core;
using AWS.Lambda.Powertools.Logging;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace lambda_dotnet_core;

public class Function
{
    
    /// <summary>
    /// A simple function that takes a string and does a ToUpper
    /// </summary>
    /// <param name="input"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    [Logging]
    public string FunctionHandler(string input, ILambdaContext context)
    {
        Logger.LogInformation("Welcome to AWS Lambda Powertools for .NET");
        return input.ToUpper();
    }
}

Steps to Reproduce

1.) Create an empty .NET 6 based lambda function using the templates provided by Amazon.Lambda.Templates NuGet package.
2.) Use the code shared above in Function.cs file.
2.) Add the following nuget package - AWS.Lambda.Powertools.Logging
3.) Deploy and invoke the function a few times.
4.) You will notice the Log events with Log level "Info"
5.) Now, add the following environment variables in the Lambda fn. either manually using AWS console or AWS CLI command.
POWERTOOLS_LOG_LEVEL: debug
POWERTOOLS_SERVICE_NAME: <give any meaningful name>

6.) Invoke the function again a few times. You will again see the log events being logged with "Info" log level.

AWS Lambda Powertools for .NET version

latest

AWS Lambda function runtime

dotnet6

What were you searching in the docs?

ATM the extra keys samples do not show correct implementation. Shows Logger.AppendKeys method not Logger.AppendKey.

Is this related to an existing documentation section?

https://awslabs.github.io/aws-lambda-powertools-dotnet/core/logging/#extra-keys

How can we improve?

Add `Logger.AppendKey` sample
Add `Logger.LogInformation(extra keys...)` sample
Add anonymous type to log entire object payload

Got a suggestion in mind?

No response

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

Syncing main branch with latest develop branch

After introducing the new secure_workflow.yml, CI is failing as it should for unpinned untrusted GH actions workflows

We should pin to the latest release git sha

Error: https://github.com/awslabs/aws-lambda-powertools-dotnet/actions/runs/2727347410

Expected Behaviour

Some Exceptions have types on them that are not serializable by System.Text.Json for example 'System.Type' and 'System.IntPtr'

See related issue here:
dotnet/runtime#43026

Not sure of the best way to handle this. I think libraries like Serilog implement their own custom JSON serialization.

Current Behaviour

When I log certain exceptions System.NotSupportedException is thrown

Code snippet

[Fact]
public void ReproduceIssue()
{
    var myEx = new MyException
    {
        Handle = (IntPtr)1
    };

    var value = JsonSerializer.Serialize(myEx);

    // System.NotSupportedException: Serialization and deserialization of 'System.IntPtr' instances are not supported. Path: $.Handle.
}

public class MyException : Exception
{
    public IntPtr Handle { get; set; }
}

Possible Solution

Possibly implement a JsonConverter which can serialize exceptions (Message, StackTrace)

Steps to Reproduce

[Fact]
public void ReproduceIssue()
{
    var myEx = new MyException
    {
        Handle = (IntPtr)1
    };

    var value = JsonSerializer.Serialize(myEx);

    // System.NotSupportedException: Serialization and deserialization of 'System.IntPtr' instances are not supported. Path: $.Handle.
}

public class MyException : Exception
{
    public IntPtr Handle { get; set; }
}

AWS Lambda Powertools for .NET version

latest

AWS Lambda function runtime

dotnet6

Debugging logs

No response

Documentation review and enhancements prior to alpha release

What were you initially searching for in the docs?

1. The api reference documentation looks completely different for the main docs.
2. The AWS logo looks pixelated
3. I am missing a link to go back to the main docs.

Is this related to an existing part of the documentation? Please share a link
Yes. https://sturdy-umbrella-97433f53.pages.github.io/api/

Describe how we could make it clearer

1. Update the template to use the same colours as the main docs.
2. Replace logo with un-pixelated version.
3. Add link on the navbar to go back to main docs.

Bump version for initial release

What were you searching in the docs?

I was reading the examples README to understand what they are for.

Is this related to an existing documentation section?

https://github.com/awslabs/aws-lambda-powertools-dotnet/tree/develop/examples/Metrics

How can we improve?

Despite it being a metrics example, the title mentions "Logging". It's just a simple typo.

Got a suggestion in mind?

Fix "Logging" to "Metrics"

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

Update package README with package description and example

What were you searching in the docs?

In the documentation and readme file there are some links to samples and the blog post which are not accurate.

Is this related to an existing documentation section?

https://awslabs.github.io/aws-lambda-powertools-dotnet/

How can we improve?

fix the broken links

Got a suggestion in mind?

No response

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

Due to gh-pages being changed during the workflow, you can only run one workflow at a runtime. We need to use concurrency group feature to prevent CI failure due to git being out of sync.

Is this related to an existing feature request or issue?

No

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

The idempotency package that provides a simple solution to convert your Idempotent functions into idempotent operations which are safe to retry.

The developer will annotate the Lambda function with Idempotent attribute and the package will take care of making the function idempotent by:

• Extracting unique identifier to represent the event
• Search the persistence layer for the extracted identifier
  • if response exists in the persistence layer; return the response.
  • if response doesn't exist in the persistence layer; execute the function, save result in persistence layer and return the response.

Use case

Idempotency is a very useful design characteristic of any system. It enables safely retrying requests without accidentally performing the same operation twice.

Networks are unreliable which leads to situations where the Lambda Function receives duplicate events. Idempotency enables overcoming this unreliable environment.

It is also mentioned on Stream processing section of the AWS Well Architected Framework - Serverless Lens that "Duplicated records may occur, and you must use both retries and idempotency within your application
for both consumers and producers"

Please note that Idempotency module is already implemented for Lambda Powertools for Python and Java.

Proposal

Getting Started

Developer can quickly start by configuring Idempotency and using it with the Idempotent attribute on your Lambda function.

public class CoolFunction
{
    public CoolFunction()
    {
        Idempotency.Configure(builder => builder.UseDynamoDb("idempotency_table"));
    }
    
    [Idempotent]
    public Task<string> FunctionHandler(string input, ILambdaContext context)
    {
        return Task.FromResult(input.ToUpper());
    }
}

The package comes with a built-in DynamoDB persistence store. The table has to be created before using Idempotency

Advanced Configurations

Idempotency behavior can be further configured with IdempotencyOptions using a builder:

Idempotency.Configure(builder =>
            builder
                .WithOptions(optionsBuilder =>
                    optionsBuilder.WithEventKeyJmesPath("id")
                        .WithPayloadValidationJmesPath("paymentId")
                        .WithThrowOnNoIdempotencyKey(true)
                        .WithExpiration(TimeSpan.FromMinutes(1))
                        .WithUseLocalCache(true)
                        .WithHashFunction("MD5"))
                .UseDynamoDb("idempotency_table"));

These are the available options for further configuration:

When using DynamoDB as a persistence layer, you can alter the attribute names by passing these parameters when initializing the persistence layer:

Use the builder to customize the table structure:

Idempotency.Configure(builder =>
            builder
                .UseDynamoDb(storeBuilder => storeBuilder
                    .WithTableName("TABLE_NAME")
                    .WithKeyAttr("idempotency_key")
                    .WithExpiryAttr("expires_at")
                    .WithStatusAttr("current_status")
                    .WithDataAttr("result_data")
                    .WithValidationAttr("validation_key")));

Here is the description of each attribute

Disabling the idempotency utility
When testing your code, you may wish to disable the idempotency logic altogether and focus on testing your business logic. To do this, you can set the environment variable POWERTOOLS_IDEMPOTENCY_DISABLED to true.

Out of scope

Implementing other persistence layers such as Redis, MySQL,...

Potential challenges

• This solution will have increase the execution time of Lambda functions. Every execution would require accessing the persistence layer.

• When using the built-in DynamoDB persistence store, the function's responses must be smaller than 400KB.

• The solution supports functions that has return type Task<T> because the AmazonDynamoDBClient supports only async methods for .NET.

Dependencies and Integrations

The implementation will leverage the AspectInjector package to build the Idempotent attribute.

Alternative solutions

No other designs considered at the moment.

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java, TypeScript

What were you initially searching for in the docs?
Why AppendKey on the Logger does not add my key to the output.

Is this related to an existing part of the documentation? Please share a link
Yes. https://sturdy-umbrella-97433f53.pages.github.io/core/logging/#appending-additional-keys

Describe how we could make it clearer
You should explain that AppendKey should be the first thing to setup to avoid keys not being added to the output. Also, if message is a reserved keyword it must be stated in the docs.

If you have a proposed update, please share it here

• Be clear on the docs what's the behaviour for AppendKey to avoid developers spend a lot of time on troubleshooting things that an implementation detail.

When using untrusted GitHub Actions, we should ensure the commit sha is used to prevent malicious code injection or secret exposure.

Branch commit misalignment. Resolve conflict before merge

What were you searching in the docs?

Just reading the docs.

Is this related to an existing documentation section?

No response

How can we improve?

* Fix some of the grammar errors.
* Update the links to the example projects to point to the source, not just the readme file.

Got a suggestion in mind?

I'll create a PR in a bit to fix this issue.

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

Implement semver versioning strategy and for nuget deployment

Expected Behaviour

Dockerfile in PowertoolsLoggingExample .NET solution has the following lines:

# Build and Copy artifacts depending on build mode. RUN mkdir -p build_artifacts RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then dotnet lambda package --configuration Debug; else dotnet lambda package --configuration Release; fi RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then cp -r /build/bin/Debug/net6.0/publish/* /build/build_artifacts; else cp -r /build/bin/Release/net6.0/publish/* /build/build_artifacts; fi

I think it should be (possibly) SAM_BUILD_MODE = "release" for the second if statement?

Current Behaviour

# Build and Copy artifacts depending on build mode. RUN mkdir -p build_artifacts RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then dotnet lambda package --configuration Debug; else dotnet lambda package --configuration Release; fi RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then cp -r /build/bin/Debug/net6.0/publish/* /build/build_artifacts; else cp -r /build/bin/Release/net6.0/publish/* /build/build_artifacts; fi

Code snippet

`# Build and Copy artifacts depending on build mode.
RUN mkdir -p build_artifacts
RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then dotnet lambda package --configuration Debug; else dotnet lambda package --configuration Release; fi
RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then cp -r /build/bin/Debug/net6.0/publish/* /build/build_artifacts; else cp -r /build/bin/Release/net6.0/publish/* /build/build_artifacts; fi
`

Possible Solution

# Build and Copy artifacts depending on build mode. RUN mkdir -p build_artifacts RUN if [ "$SAM_BUILD_MODE" = "debug" ]; then dotnet lambda package --configuration Debug; else dotnet lambda package --configuration Release; fi RUN if [ "$SAM_BUILD_MODE" = "release" ]; then cp -r /build/bin/Debug/net6.0/publish/* /build/build_artifacts; else cp -r /build/bin/Release/net6.0/publish/* /build/build_artifacts; fi

Steps to Reproduce

• Clone the current repository
• Open the PowertoolsLoggingExample solution in Visual Studio
• Examine the Dockerfile

AWS Lambda Powertools for .NET version

latest

AWS Lambda function runtime

dotnet6

Debugging logs

No response

Use case

It is currently difficult to test functions that depend on the static Logger, as we cannot provide an alternate or mock implementation.

Additionally, if we want to use the functionality that it offers for existing applications, we would have to replace all existing references to an implementation for ILogger with the static Logger instance.

Solution/User Experience

Instead of having to use the new Logger static class to perform logging, we would prefer to be able to inject it as an implementation of ILogger and set it up through dependency injection.

This would be the idiomatic approach to supply a logger implementation in C#/.NET

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

Update package README with package description and example

What were you searching in the docs?

Adding metadata code sample incorrectly rendering

Is this related to an existing documentation section?

https://awslabs.github.io/aws-lambda-powertools-dotnet/core/metrics/#adding-metadata

How can we improve?

Fix tabs

Got a suggestion in mind?

No response

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

What were you initially searching for in the docs?
How to define logger output casing with attributes.

Is this related to an existing part of the documentation? Please share a link
https://sturdy-umbrella-97433f53.pages.github.io/core/logging/#configure-log-output-casing

Describe how we could make it clearer
Explain how to set the logger output casing through attributes and not only environment variables.

If you have a proposed update, please share it here
Update documentation by adding an example.

Is this related to an existing feature request or issue?

No

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

The parameters utility provides high-level functions to retrieve one or multiple parameter values from AWS Systems Manager Parameter Store, AWS Secrets Manager, AWS AppConfig, Amazon DynamoDB, or bring your own.

Use case

Features parity across Powertools languages.

Proposal

Key features

• Retrieve one or multiple parameters from the underlying provider
• Cache parameter values for a given amount of time (defaults to 5 seconds)
• Transform parameter values from JSON or base 64 encoded strings
• Bring Your Own Parameter Store Provider

Out of scope

Anything which is not mentioned in the Summary and Proposal sections.

Potential challenges

TBD

Dependencies and Integrations

Dependencies to relevant AWS SDK NuGet Packages

Alternative solutions

Implement separately, and directly call relevant SDK APIs

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java, TypeScript

Metrics documentation

What were you searching in the docs?

I saw that Slack will be replaced by Discord https://discord.gg/B8zZKbbyET.

Is this related to an existing documentation section?

How can we improve?

Replace all Slack inviation links with the links to Discord 

Got a suggestion in mind?

No response

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

Is your feature request related to a problem? Please describe.
To keep it consistent with other AWS Powertools runtimes and allow search across runtimes, logging keys should be snake case instead of camel case.

Describe the solution you'd like

• Allow logging case to be configurable through Lambda function environment variables
• Output logging data using the configured case
• If not defined, use snake case as the default

Update nuget references to Version="0.0.1-preview.1" for sample projects

Expected Behaviour

release_label_on_merge failing

https://github.com/awslabs/aws-lambda-powertools-dotnet/actions/runs/3036132511/jobs/4887086940

Current Behaviour

release_label_on_merge failing

https://github.com/awslabs/aws-lambda-powertools-dotnet/actions/runs/3036132511/jobs/4887086940

Code snippet

NA

Possible Solution

No response

Steps to Reproduce

NA

AWS Lambda Powertools for .NET version

latest

AWS Lambda function runtime

dotnet6

Debugging logs

No response

Remove the dependency to the Amazon.Lambda.Core library and the ILambdaContext interface

Update package README with package description and example

What were you searching in the docs?

I was searching whether the Logger implements the Microsoft.Extensions.Logging.ILogger interface.

Is this related to an existing documentation section?

https://awslabs.github.io/aws-lambda-powertools-dotnet/api/

How can we improve?

I've found that there is a `Create` method for the `AWS.Lambda.Powertools.Logging` which returns a `Microsoft.Extensions.Logging.ILogger`. Perhaps this could be documented somewhere.


    public static ILogger Create(string categoryName);
    public static ILogger Create<T>();


Another topic that may be interesting is whether this Logger could be used outside of Lambda. From the point of portability, some may use this in a Lambda Function, some in a container, which is not related to Lambda at all. Some may DI the Logger to an `ILogger` interface. Is this recommended or not?

I'm not a fan of DIing Loggers, but that relates to the topic of whether this Logger is capable to run outside of Lambda functions.

Got a suggestion in mind?

No response

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

https://github.com/aws-samples/cookiecutter-aws-sam-dotnet/

Is your feature request related to a problem? Please describe.
Currently the .NET runtime does not support adding extra keys to a specific logging. Here is the Python implementation -> https://awslabs.github.io/aws-lambda-powertools-python/latest/core/logger/#extra-parameter

Describe the solution you'd like
I would like to be able to add extra fields to logging and have examples on documentation.