Comments (5)
1yuv
commented on July 3, 2024
1
If we delete the data, it means we can't reuse the informaiton already available for other purposes. If you want to ensure that reporting is not impacted because of the huge dataset, we can drop the old data in historical tables and keep reporting data for 6 months only. Here's how we are doing that for impact monitoring.
This way, we can keep the scraped data for longer term, yet keep current reporting data small and fast.
from cht-app-monitoring-data-ingest.
eljhkrr
commented on July 3, 2024
Thanks @yrimal, added as agenda item for upcoming meeting
from cht-app-monitoring-data-ingest.
derickl
commented on July 3, 2024
If the target table is well formed, you wouldn't have an issue with a proper indexing strategy
from cht-app-monitoring-data-ingest.
kennsippell
commented on July 3, 2024
This is how many docs we have. We have been monitoring for about a year. It is about 50MB of data.
Should we just ignore this issue for now?
| doctype | count |
|---|---|
| klipfolio_datasource | 8,944 |
| monitoring | 12,127 |
| error | 113 |
| couch2pg | ~5k |
from cht-app-monitoring-data-ingest.
1yuv
commented on July 3, 2024
Should we just ignore this issue for now?
I agree, this sounds like a nominal data.
from cht-app-monitoring-data-ingest.
Related Issues (20)
- Replication performance - How long is replication taking HOT 1
- History of deployments to production instance HOT 6
- Gateway availability HOT 2
- Replication - remove first value from docs replicated per day HOT 1
- Display detailed stack traces from feedback docs HOT 3
- Document how to get info on denied replications HOT 1
- Update refresh script to capture performance data on how long each view takes to refresh HOT 1
- Devices tab not pulling data on insufficient disk space & chrome version HOT 3
- Add companion table for form completion widget by username HOT 2
- New Metric - # of Successful Initial Replications HOT 1
- CHT version upgrade history HOT 4
- Update views to reuse couch2pg telemetry views HOT 1
- Add contact creation to engagement metric
- System should have single criteria for projects to be monitoring
- Alert on number of task documents on the user's device > 3000 or 5000. HOT 1
- Deprecate ingest functionality available in prometheus
- Data is duplicated in watchdog grafana panels
- migrate existing SQL queries away from deprecated exporter
- SQL optimizations
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cht-app-monitoring-data-ingest.