openclimatefix / ocf-infrastructure Goto Github PK

Create a PV consumer that runs every 5 mins on ECS

can copy forecast ECS module to start with

• terrafrom stack
• setup terraform cloud
• setup aws account
• setup auth0
• setup consumer secrets

• set up s3 buckets

• Add capacitys to pv and gsp systems

• add new FE

• Check everything works?

• switch over webaddress

• [ ]

• ec2 bastion

Detailed Description

Setup database for saving predictions to

• terraform setup, start with something small t3small
• secret for password
• IAM read password secret
• define simple datamodel

Context

For nowcasting service
https://github.com/openclimatefix/nowcasting_infrastructure/issues/6 needs this database

Detailed Description

Automatically remove 30 day old data from s3 bucket.

Context

We don't want to store the data that long. This is just for prediction, not for training

Possible Implementation

I think this can be done with settings on the s3 bucket

We use Cloudflare to manage our DNS records (and a few other things).
Currently we do that by hand, via their dashboard.

There is however a Terraform Provider for Cloudflare.
We should setup Terraform to work with our Cloudflare setup, so that we can have this part of our infrastructure as code as well.

Detailed Description

Would be good to have a test that checks if the a file has been made once the satellite consumer has run in the local stack.
The file will be in the docker volume 'sat_data' in a folder called 'data'

Context

good to test

Possible Implementation

unsure how to check docker volumes using python / pytest

Detailed Description

Use terraform to run nowcasting API on AWS elastic beanstalk

version of docker file to use
tf: elastic beanstalk
tf: IAM independent role
put inside a VPC -
stack_name, so mulitple of these can be set up
tf: ec2: t2.medium

Context

Nowcasting API
Good to use terraform, so it can easily be rebuilt

For nowcasting project

Ready to upgrade nowcasting_api to 0.1.20

release notes - openclimatefix/uk-pv-national-gsp-api@0.0.18...0.1.20

the biggest change is - openclimatefix/uk-pv-national-gsp-api#69, so just want to check with @flowirtz before doing this upgrade

Detailed Description

Create database for pv data
secret password
iam policy to read password

similar to forecast database

Detailed Description

Dont load latest image, load docker tagged versions. These versions should be set at the environment level.

This would be for

• nwp: code
• forecast: code
• API might be harder to do right now

Versions could go in development vars and passed through

Context

Good to load tagged version, incase a new latest version comes out which is broken

Could include:

• AWS Config
• AWS Security Hub
• moving all security rules to a nowcasting-security workspace in TF
• ...?

Detailed Description

Create satellite app that runs in ecs every 5 mins
create iam and roles

Possible Implementation

Copy the NWP consumer, just change docker image and

Detailed Description

• Create AWS task to run docker file from MetOfficeDataHub
• use terraform to create AWS task

first need to finsih

• #2
• #4
• openclimatefix/MetOfficeDataHub#4

Context

Possible Implementation

Detailed Description

Add terraform code for creating

• s3 bucket. The bucket should be private.
• iam policy to read to the bucket
• iam policy to write to the bucket

Context

Might have to delete old bucket in s3 to make this work

Detailed Description

Run dagster locally and see what it can do. Would be interesting to see

• how it kicks off jobs - can it kick off AWS tasks on ECS?
• how to make jobs dependent on each other?
• how it deals with error messages
• what informatino does the user get?
• how can this be deployed not locally

This will be useful for nowcasting project

Change nwp data to only keep 1 week of data

Context

Good to save costs

Possible Implementation

This can be done in the s3 lifetime rules

Detailed Description

Add method that local user can logging to the VPC in the cloud.

Context

This is useful for debugging

See the data vis service keeps crashing, could be due to memory

Detailed Description

Add GSP Consumer to terraform code.
Should have two scheduled tasks

• run every 30 mins, collecting todays data
• run once a day at say 9, collecting yesterdays data

Possible Implementation

Can mainly copy PVConsumer code

We want to have an internal status dashboard for Nowcasting that let's us see the service health at a glance.

Architecture

To not reinvent the wheel, we can try to use Prometheus for that.
There is a hosted Prometheus service on AWS that takes care of storing and aggregating the metrics (backend). We will still need a Prometheus instance to scrape the metrics, but that could live as a docker container in our cluster.

There are helper libraries for almost all major languages, so we can use those libraries to expose metrics from our individual services. Once all our metrics are in Prometheus, we can make use of existing tooling like the Alertmanager for alerting and Grafana for visualising these metrics.

  graph TD;
      A[Data Source 1]-->B[Prometheus Ingester on Docker];
      E[Data Source 2]-->B[Prometheus Ingester on Docker];
      B[Prometheus Ingester on Docker]-->C[Prometheus Managed Service on AWS];
      C[Prometheus Managed Service on AWS]-->D[Grafana];

Further reading

• Blog: Using Amazon Managed Service for Prometheus to monitor EC2 environments
• Best Practices: Metric and label naming
• Example: Next.js Metric Exposer

make elastic beanstalk bucket private

code is here

Detailed Description

Would be good to use latetst postgres version -> 14.1

Possible Implementation

change code here

The PV Consumer database seems to be stalled

https://eu-west-1.console.aws.amazon.com/rds/home?region=eu-west-1#performance-insights-v20206:/resourceId/db-MQLKRIDRWEM4P2M3733A3CLK6I/resourceName/pv-development/startTime/1647760961296/endTime/1647847361296

Create Satellite consumer s3 bucket
add read and write iam policies

perhaps following on from https://github.com/openclimatefix/nowcasting_infrastructure/blob/main/terraform/modules/s3/s3.tf

Describe the bug
Currently CI local stack tests fail. Would be good to run local stack locally with logs and run tests

To Reproduce
See CI

Expected behavior
tests to pass

Additional context
Hard to see logs in CI

Currently status dashboard is on faragate,

Context

For continuous services I think faragate is more expensive that ECS, but we should check this,

Possible Implementation

Move to ecs

Detailed Description

Add route53 i.e an internet address to elastic beanstalk application.

Context

Useful for people to access the api, and not give them a funny elastic beanstalk address.
Might have to ask @flowirtz for help

Detailed Description

Give data vis module access to nwp and satellite s3 buckets

Context

So data vis can plot latest data

Possible Implementation

similar to how forecast module has access to these buckets. Needs IAM read access

Currently National results are in, but not GSP when it gets trigger.
Might be worthing triggering it a few minutes later, as well

Adjust cron job
to 9,11,39 and 41

Complete if:
See if data is available on data-vis app after 11 past?

n order to save when the last data was pulled, we need to give the nwp consumer access to the forecast database

Update s3 buckets

More details to come

Detailed Description

Add auth env to API
need to add

• AUTH0_DOMAIN
• AUTH0_API_AUDIENCE

Context

This is how the api can be authenticated

Possible Implementation

add values to AWS secret and then load them in as secrets to API

Detailed Description

• Use terraform to set up simple ECS cluster. Perhaps start with 1 t3small machine and we can always change it easily.
• no auto scalling

Context

need to setup NWP consumer

Detailed Description

Use 'terraform cloud' as a backend for terraform deployment

IAM policy to read pv output.org api key

Detailed Description

Try using kubernetic cluster

Context

This make it easier to use airflow

Possible Implementation

https://learn.hashicorp.com/tutorials/terraform/eks

Remove unnecessary nowcasting strings from elastic beanstalk application

for api and data vis

Detailed Description

Let the database expand its storage automatically. A upper limit could be set of 500 MB.

Context

The less things we have to maintain the better

Possible Implementation

code could add using this

Detailed Description

• Move all security to a separate module. This makes it easy to review and check we know whats going on

Context

Security is hard, best to get a very simple view of it, so we know whats going on

Detailed Description

Add tags to all (if possible) components
The tags could be

• 'nowcasting'
• give the enviornment
• give the module

Context

Useful to filter on tags and see what is going on

Add local GSP consumer to local stack

Detailed Description

Should pull todays and yesterday data

Describe the bug
As per #49, our current status dashboard test setup consists of an AWS-managed service for prometheus and a prometheus ingestion container running on Docker (ECS). The Ingester uses the managed-service as a "remote backend", meaning it just fetches metrics data and sends it to the managed-service.

There seems to be a permissions issue, as the ECS container keeps logging the following error message:

ts=2022-03-01T17:34:55.724Z
caller=dedupe.go:112
component=remote
level=error
remote_name=f3f5a2
url=https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-dcdefd46-7af3-483d-a430-24811887ff36/api/v1/remote_write
msg="non-recoverable error while sending metadata"
count=29
err="server returned HTTP status 403 Forbidden: {\"Message\":\"User: arn:aws:sts::008129123253:assumed-role/statusdash-iam-role/b7997edbd1a342ef8c3d8d57cd70a6e2 is not authorized to perform: aps:RemoteWrite on resource: arn:aws:aps:eu-west-1:008129123253:workspace/ws-dcdefd46-7af3-483d-a430-24811887ff36 with an explicit deny in an identity-based policy\"}"

The violating role statusdash-iam-role should already have this permission though, according to
https://github.com/openclimatefix/nowcasting_infrastructure/blob/16e71e768bf9802ec1d25188609a5306872b59e4/terraform/modules/statusdash/prometheus.tf#L86-L107

In order to save when the last data was pulled, we need to give the satellite consumer access to the forecast database

Detailed Description

Make sure PV conusmer doesnt go and get all 1000 metadata, just use 10 in the test file

Context

this is cause a rate limit effect on production service

Possible Implementation

set FILENAME to
filename = os.path.dirname(pvconsumer.file) + "/data/pv_systems_10.csv"

The current locally stack test is failing

test docker is here

Need to

• Make sure PV Consumer is limits to say 10 PV systems, this is stops our API limit being reached
• update to latests versions
• Add satellite - #59
• adjust wait time for all things to run - here

Detailed Description

Setup up simple VPC
use terraform

Context

for Nowcasting project

Detailed Description

Add an ec2 bastion that we can use as a baston.
Add ssh keys for me, @JackKelly and @jacobbieker on this machine. These could be loaded from AWS secret
This means we could set up ssh tunneling ont this machine

Context

Good to have a backdoor into our system

Might be nice to have an variable, if to have this or not. As most of the time we wont need this.

Detailed Description

• Create s3 bucket for models
• iam read policy
• add policy to forecast role

Context

forecats module should be able to download s3 items form this bucket

We want to have a shared load balancer for our API that doesn't change with each deployment but rather is persistent.

Current Situation
The load balancer gets recreated (with a different URL) on every deploy.

What we want
One persistent load balancer that always points at the most recent deployment(s).

How?
This guide in the AWS docs explains how to create a shared load balancer.

Basically:

1. Create a standalone Application Load Balancer (ALB)
2. Reference that standalone ALB in the EB environment config
3. Done!

Would maybe also be good to export "aws_lb.dns_name" so we can easily find it for adding it as a CNAME to Cloudflare.

Add auotmatic backups to rds database

Perhaps start with backup every weeks on s3

two databases are here - https://github.com/openclimatefix/ocf-infrastructure/blob/main/terraform/modules/storage/database-pair/rds.tf

one more is here - https://github.com/openclimatefix/ocf-infrastructure/blob/main/terraform/modules/storage/postgres/rds.tf