Comments (1)
Released in v1.2.0 #45
This PR contains more details about implementation. Also, see docs/CONTRIBUTING.md
The lambda function runs at 1 hour past midnight UTC. At this point, the Firehose is done writing for the previous day and no new data will be written. The function is idempotent and will delete the raw files after doing the rollup only if it is successful.
During the test, it does an incredible job at rolling up.
BEFORE:
S3 Objects (388)
Input rows: 650
Input bytes: 181.34 KB
Output rows: 148
Output bytes: 37.37 KBAFTER:
S3 Objects (4)
Input rows: 148
Input bytes: 33.73 KB
Output rows: 148
Output bytes: 37.26 KBThe important part is the almost 100x reduction in S3 file count. This will decrease the amount of files that Athena has to scan by magnitudes, which means big savings to gain from this rollup.
from serverless-website-analytics.
Related Issues (20)
- Clicking on a referrer should have a context menu instead HOT 1
- Outbound link tracking HOT 1
- Add performance tracking
- Logo and favicon
- Email reports HOT 1
- Auto analyize traffic
- Add option to view by hour/day HOT 4
- Document query your own data HOT 1
- Uptime monitoring
- Add time filtering
- Explore using AWS Timestream instead of Athena HOT 2
- Auto vacuum if X amount of records have been ingested for that day
- Insights
- Experiment Glue Table Statistics HOT 1
- S3 Express One Zone
- Anomaly Detection HOT 1
- Udpate Anomaly Detection Algorithm so that it better detects when an anomaly is over HOT 6
- Basic real time view
- Use LLRT runtime if possible
- Bug: wrong A record for Cognito domain if using a subdomain HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from serverless-website-analytics.