Auto create partitions about serverless-website-analytics HOT 3 CLOSED

I've only briefly looked into the code, but here is a suggestion that may or may not make sense.

I see you have 3 partitions: site, year, and month.

For year and month, you can use partition projection. There is even a specific type for dates. Then you don't need to update partitions, Athena will know that there may exist year partitions, let's say from 2023 to 2099. And month partitions from 1 to 12.

site is more tricky. But it's also less common change that can be done manually. With dynamic partitions, you can use enum type, if possible sites are given in Swa config (I'm not sure about it). Otherwise, I see there is also an "injected type", but I did not use it and don't know exactly how it works.

Again, great job with this project!

from serverless-website-analytics.

That is a great idea! Completely slipped my mind. I will explore this, thanks for the suggestion.

Regarding site that can easily be managed via a Custom Resource in the CDK. So it will then create/update/delete the site partition whenever it gets added, so a once-off. Then the date partitions are dynamic which means no more partition creating 🙂. Definitely going to look into this one. Thanks again.

from serverless-website-analytics.

Released in v.0.0.5, thanks @m-radzikowski! I could delete a lot of code 😃

from serverless-website-analytics.