We have the Lambda function to send emails working. 🎉
Now we want to create an SNS topic so that our Lambda can receive SES related notifications.
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/configure-sns-notifications.html

But what happens when with the information in the SNS Notification?
In the past we have stored this data in S3 or DynamoDB but both those data stores are cumbersome to query and have per-request charges for saving/getting data. So we need a way of getting the SNS notification info back to our PostgreSQL (main) database.

Our ultimate objective with email is to have a real-time dashboard where we can visualise the aggregate deliverability stats and specific stats for a given person using our App (while preserving their anonymity/privacy).

We still want our Lambda function to receive and process the SNS messages,
and then forward the formatted info back to Phoenix App via REST API endpoint.

Todo

In order to use the same tables/schema as the MVP which has detailed explanations
see: https://github.com/dwyl/app-mvp-phoenix#schema
We are borrowing the actual migration files:

• Create tags table by reusing 20191113100513_create_tags.exs migration from the MVP:
  https://github.com/dwyl/app-mvp-phoenix/blob/master/priv/repo/migrations/20191113100513_create_tags.exs
• Create status table by reusing 20191113100912_create_status.exs from MVP: https://github.com/dwyl/app-mvp-phoenix/blob/master/priv/repo/migrations/20191113100912_create_status.exs
• Create people table by reusing 20191113100920_create_people.exs migration from the MVP:
  https://github.com/dwyl/app-mvp-phoenix/blob/master/priv/repo/migrations/20191113100920_create_people.exs
• Add person_id to Tags using 20191113114340_add_person_id_to_tag.exs from MVP:
  https://github.com/dwyl/app-mvp-phoenix/blob/master/priv/repo/migrations/20191113114340_add_person_id_to_tag.exs
• Add person_id to Status using 20191113141229_add_person_id_to_status.exs form MVP:
  https://github.com/dwyl/app-mvp-phoenix/blob/master/priv/repo/migrations/20191113141229_add_person_id_to_status.exs

This way we can run the Email and Auth Apps in several ways:

1. both apps use the same database (our preferred option during the first 100 people "Beta" phase)
2. Auth and Email can be run separately
3. Email can be run as an Umbrella App to Auth
4. We can safely combine/merge all the apps into one App for simplicity.

Next

• Create sent table/schema with the following fields:
  • person_id (references person.id) - the person the email was sent to.
  • template - the template that was used to send the email to the person.
  • message_id and request_id fields to store SES metadata.
  • status_id (references status.id) - the status of the email e.g: 6:sent, 7:delivered, 8:bounce, etc. This allows us to have unlimited statuses without tons of duplication of text.

  Note: the "status" does not need to be a single word, it can be for example:
  "bounce: addressee unknown"

Then

• Create /api endpoint that:
  • requires a valid JWT in order to allow data in
  • receives data in a standardised form

It's not possible to load the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in AWS lambda see: dwyl/aws-lambda-deploy#58 (comment)
Which is why we are seeing the AccessDenied error #10 😞
There appears to be a way forward:
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-send-email-ses

Todo:

• Create the IAM policy and attach it to the LambdaExecRole
• Test if that works!

This is a potential features "roadmap" in (descending) order of importance.

• Email templates should be in a separate repo
  from the "main" application in any (all) our projects.
  Non-technical team members should be able
  to update the template (using their "basic" HTML + handlebars knowledge)
  and should be able to save a draft of the changes to GitHub
  (without having to run the "CI" for an entire "web app" project!)
• Send Sample Email to myself to check layout/design and confirm
  the flow is working.
• Extend the Deployment Tool
  to automatically configure the API Gateway
  see: dwyl/learn-aws-lambda#62 (comment)
  (for now I am doing it manually...)
  see: https://github.com/dwyl/aws-lambda-deploy
• Consider Using DynamoDB for truly "Serverless" email?
  (this could/should be implemented by someone other than the "core" dwyl team
  as I have no "appetite" for the DynamoDB pricing model... but it's something
  someone else might consider adding)
  Not using DynamoDB. It has a horrible query API. Using PostgreSQL instead!
  see: https://github.com/dwyl/email/

If you are keen on a particular feature or want to propose other ideas, please comment below! 👍

https://thehustle.co have a pretty good intro/welcome email:

Have you seen other examples?

These are the steps we originally defined:

Each time an email is sent using the aws-ses-lambda 5 things happen:

1. Pre-Send Checks

• Is the email address valid. (basic checks to avoid wasting money)
• Check our records to see if the email address we are attempting to
  send to has bounced in the past.

2. Send Email

• Send the email using AWS SES and keep a note of the
  unique ID confirming the email was sent.

3. Log Email Sent

• Log all detail of the sent email to the Database

4. Query SES Bounce/Spam Info Service

• Check which emails have bounced

5. Report Stats/Summary in Dashboard

• Update the Database with the send/receipt stats
  so that they can be viewed on the team's Email Dashboard!

Frequently Asked Questions (FAQ)

Separate Lambda Functions or One Lambda with Independent Functions?

From experience making distinct functions separate lambda functions,
just increases our cost/complexity without any discernible benefit.

Having separate lambdas when we know that all functionality is executed
each time

aws-ses-bounce-checker periodically retrieves stats on outbound emails from SES.
These include bounce rates and whether an email has been opened.

We need to take a fresh look at these and determine which features are still needed.

while building the functionality for aws-ses-lambda I've been working off my imagination ... 💡
Obviously that doesn't make it very easy for other people to understand it. 🙄

Todo

• Make a nice version of this:

The main App does not do anything with Email as that is is not it's core function.
It delegates all email sending and monitoring activity to the aws-ses-lambda.

┌─────┐ send  ┌────────────────┐
| App | ----->| aws-ses-lambda |┐ the Lambda function Sends email 
└─────┘ email └────────────────┘| and handles SNS notifications
  ┌───────┐    SNS Notification | for bounce events.
  | Email | <───────────────────┘
  | Stats |
  └───────┘  The Email Stats App aggregates and visualises email stats.
             This allows us to be more data-driven in our communications.
             And understand exactly who is engaged with the app.

In order to debug the data we are receiving in the Lambda event,
we need to save a copy of the event to S3 so that we can view it after the Lambda has finished execution.
(yes, we can do this by hunting through the cloudwatch logs, but it's so much easier if we can just open a file and view it immediately rather than scrolling for ages)

Todo

• Save event object as JSON in a file on S3.

https://moonmail.io/

Our mission is to create an SNS topic to "listen" for Bounce emails
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/configure-sns-notifications.html

Hi,

I was just wondering how AWS is invoicing SES mails sent from Lambda functions?
SES has two types of invoices:

1. If the mail is originating from EC2 instances the first 62.000 mails are free
2. In other cases you pay $0.10 for a 1000 mails
   So, under which of the 2 cases fall mails send from Lambda functions

Regards

Paul

There are a couple of options for invoking a Lambda function from Elixir:

• https://github.com/bbc/elixir-invoke-lambda - sadly the BBC code is tightly coupled to their infra. So it's not much use to anyone else ... but it's a reference and it's good to know they're using Elixir! 🎉
• https://github.com/ex-aws/ex_aws_lambda

Todo

• Setup a test Elixir project: https://github.com/dwyl/elixir-invoke-lambda-example
• attempt to invoke the AWS SES Lambda function
  • arn:aws:lambda:eu-west-1:110797354241:function:aws-ses-lambda-v1

AWS SES is great for sending email but does not have great UI for checking the stats of our sent emails. Thankfully, there's an API we can use to lookup the data and display it to our "admin" people so they are aware of the success/failure (bounce) rate. 👍

While attempting to send SNS notification data to the Phoenix App https://phemail.herokuapp.com
we are seeing the following error:

problem with request: Hostname/IP does not match certificate's altnames: Host: https. is not in the cert's altnames: DNS:*.herokuapp.com, DNS:herokuapp.com

Reading request/request#1777 we discover
that we have the option to ignore this error by adding the option:

rejectUnauthorized: false,

https://stackoverflow.com/questions/36588963/node-js-requests-with-rejectunauthorized-as-false

https://www.gkogan.co/blog/dont-design-emails/

In order to deploy and invoke a Lambda function and send email from that function,
we need to define an AWS Identity and Access Management (IAM) Role with the required permissions.

Todo

• Take screenshots of all steps in the process while setting up an IAM Role
• name: lambda-user
• Permissions:
  • AWSLambdaFullAccess
  • AmazonSESFullAccess
  • AmazonS3FullAccess
• No tags.

One of the only "down sides" of AWS Lambda is latency.
It can take up to 700ms for a "cold" Lambda function to startup.
The median cold-start is around 400ms
Once the Lambda has been invoked and is cached, it only takes 50ms to respond to subsequent requests.
What this means in practice is that someone registering for our App they will wait up to 1 second for an email to be sent.

Todo

• Create a "no op" execution branch in index.js so the lambda function can be invoked without any operation or output. See: https://en.wikipedia.org/wiki/NOP_(code)
  • return event as data

Attempting to run the Lambda function on Lambda getting the following error:

{
  "errorType": "AccessDenied",
  "errorMessage": "User `arn:aws:sts::123456789:assumed-role/LambdaExecRole/aws-ses-lambda-v1' is not authorized to perform `ses:SendEmail' on resource `arn:aws:ses:eu-west-1:123456789:identity/[email protected]'",
  "trace": [
    "AccessDenied: User `arn:aws:sts::123456789:assumed-role/LambdaExecRole/aws-ses-lambda-v1' is not authorized to perform `ses:SendEmail' on resource `arn:aws:ses:eu-west-1:123456789:identity/[email protected]'",
    "    at Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/query.js:50:29)",
    "    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
    "    at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
    "    at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14)",
    "    at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)",
    "    at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)",
    "    at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10",
    "    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)",
    "    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:685:12)",
    "    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"
  ]
}

Investigating now ... 🔍