Instead, it is set to our unsecure value.

https://github.com/posthog/deployment/blob/master/terraform/digitalocean/single_node/docker-compose.do.yml#L27

I tried updating a previously deployed cloudformation stack via their update button to bump database instance and worker sizes.

This failed during updating TaskDefinition step with the following error:

Invalid request provided: Create TaskDefinition: No Fargate configuration exists for given values. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: XXX; Proxy: null)

Will investigate at a later date.

We are using posthog:latest on a few images here, which, while making things easier for us (we don't have to be updating version numbers), means a user might get an unstable deployment (since our latest tag is a build from the HEAD in master).

Wondering if this is something we should talk about @macobo

Hello,

I would like an enhancement to the Cloudformation template to allow ALB to provide a re-direct value for port 80, see below:

PublicLoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- TargetGroupArn: !Ref 'DummyTargetGroupPublic'
Type: 'forward'
LoadBalancerArn: !Ref 'PublicLoadBalancer'
Port: 80
Protocol: HTTP

Link: https://github.com/PostHog/deployment/blob/master/aws/cloudformation/ecs/posthog.yaml

Currently the created DB size can be rather small - we can/should make it configurable for smooth updates.

Benefit: Users don't need to open any docs/command line when creating the stack. Need to make sure it stays the same as stack gets updated.

Blog on how to do this: https://www.itonaut.com/2018/01/03/generate-passwords-in-aws-cloudformation-template/

After #21 lands, there are several linting errors in posthog.yaml:

⟩ cfn-lint aws/cloudformation/ecs/posthog.yaml                                                                                                                                                             14:31:14
W2001 Parameter SentryDSN not used.
aws/cloudformation/ecs/posthog.yaml:146:3

E3012 Property Resources/EcsSecurityGroupIngressFromPublicALB/Properties/IpProtocol should be of type String
aws/cloudformation/ecs/posthog.yaml:359:7

E3012 Property Resources/EcsSecurityGroupIngressFromPrivateALB/Properties/IpProtocol should be of type String
aws/cloudformation/ecs/posthog.yaml:366:7

E3012 Property Resources/EcsSecurityGroupIngressFromSelf/Properties/IpProtocol should be of type String
aws/cloudformation/ecs/posthog.yaml:373:7

E3012 Property Resources/PublicLoadBalancerSG/Properties/SecurityGroupIngress/0/IpProtocol should be of type String
aws/cloudformation/ecs/posthog.yaml:397:13

W3005 Obsolete DependsOn on resource (PublicLoadBalancer), dependency already enforced by a "Ref" at Resources/PublicLoadBalancerListener/Properties/LoadBalancerArn/Ref
aws/cloudformation/ecs/posthog.yaml:430:9

E3012 Property Resources/PrivateLoadBalancerIngressFromECS/Properties/IpProtocol should be of type String
aws/cloudformation/ecs/posthog.yaml:452:7

W3005 Obsolete DependsOn on resource (PrivateLoadBalancer), dependency already enforced by a "Ref" at Resources/PrivateLoadBalancerListener/Properties/LoadBalancerArn/Ref
aws/cloudformation/ecs/posthog.yaml:485:9

W3011 Both UpdateReplacePolicy and DeletionPolicy are needed to protect Resources/PosthogDB from deletion
aws/cloudformation/ecs/posthog.yaml:696:3

W2501 Password shouldn't be hardcoded for Resources/PosthogDB/Properties/MasterUserPassword
aws/cloudformation/ecs/posthog.yaml:707:7

@fuziontech Which of these should we action? The Sentry, Password and Policy warnings seem especially juicy.

Own thoughts:

1. Add SentryDSN to be used as an environment variable (as intended)
2. Make DB user/password part of the variables, use current values as defaults
3. Add UpdateReplacePolicy and DeletionPolicy are needed to protect Resources/PosthogDB from deletion

After this we could also automate this linting via github actions.

The SITE_URL environment variable is necessary for emails and some toolbar features to work correctly. When launching an instance with the CloudFormation template, we don't guide the user through setting up this environment variable.

I'm self-hosting a AWS stack, and it went down for a day without me noticing until hitting the node.

It would be nice if I could have received an email about these issues.

Relevant documentation: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-cloudformation-template-to-create-cloudwatch-alarms.html

Recently I launched a CF template to set up the infrastructure for a customer. Just after the resources were deployed, I received a few alarms in the SNS topic. The alarms were cleared pretty soon, presumably just due to a transition state on the launch of the resources. While this is not critical, as the alarms were cleared soon, it's not a great experience that the first emails you get after the deployment are error notices.

Alarms triggered:

• Posthog-ElastiCacheMemoryFullAlarm. Threshold Crossed: 1 datapoint [4.65436672E8] was less than the threshold (6.4E8). Redis memory over the last 10 minutes is too low, your instance might not be ingesting events
• Posthog-RDSDiskFullAlarm. Threshold Crossed: 1 datapoint [1.015771136E10] was not less than the threshold (1.0E9)

The Cloudformation template currently creates a VPC with a hard-coded CIDR of10.0.0.0/16 which is the default CIDR block used by most AWS VPCs.

Unfortunately, this means that you can't create a peering connection between the Posthog VPC and another VPC that was created using these defaults, so inter-VPC communication requires the more expensive Transit gateway.

I don't think there's a way to change this for an existing deployment, but it would prevent a few headaches for future deployments if Posthog selected a less common CIDR block for it's VPC.

When deploying with the CF template we generate a SECRET_KEY and also store it under password in AWS Secrets Manager. However, in the task definition we set a plain text version as an environment variable instead of making use of the secret version.

Currently logs from ECS from our cloudformation setup are not reaching cloudwatch - it only seems to contain logs only for server boot.

Currently, install instructions point to old file, which runs into problems with e.g. CacheNodeType.

Ideally we could also automate this?

It's a little worrying to see 500's when you start the cloud formation stack and the first thing you see is a 500. If you didn't know any better you might kill the stack before waiting for webpack to finish. Would be nice to have a 'tidying things up' or 'cleanup up the dust' landing page while webpack finishes.

Among the parameters specified in deployment/aws/cloudformation/ecs/posthog.yml are EmailUserSSL and EmailUserTLS. In the container definition I see Ref! 'EmailUseTLS' and Ref! 'EmailUseSSL'.

When deploying the script I get ValueError: Ref! 'EmailUseSSL' is an invalid value for EMAIL_USE_SSL, expected boolean and such.

I assume these are typos?

The DATABASE_URL environment variable for the ECS Task definition uses hard coded values for username and password, as a result changing the RDS username or password results in a 503.

- Name: DATABASE_URL
  Value: !Join ['', ['postgres://', !Ref 'RDSMasterUser', ':', !Ref 'RDSMasterUserPassword', '@', !GetAtt [PosthogDB, Endpoint.Address], ':', !GetAtt [PosthogDB, Endpoint.Port], '/posthog']]

Should do the trick for both.

This came up in posthog users chat: https://posthogusers.slack.com/archives/CT7HXDEG3/p1605868648007300

Currently you can't connect to the RDS instance for posthog from outside the VPC. This might however be needed to connect to Metabase/other BI or do some other hackery.

We could add an Parameter to our template to expose RDS to the world. Just need to make sure the user isn't using the default password.

Related to PostHog/posthog.com#1167 & this user report we definitely need to add some protections against accidental data deletion. Some suggestions:

• Add database protection by default to the RDS instance.
• Set a default PITR rolling backup for something low like 2-4 days. This will ensure there's something to rollback on just in case something happens and it won't add too much infrastructure expense.
• We need a way to update the CF stack without recreating the DB. If this is definitely not possible for now, can we add a warning for when you're trying to recreate your stack?