noppakorn / ddc-dashboard-scraping Goto Github PK

• Daily dashboard link scraping takes a long time (Use selenium)
• Will try to scrape later

The issue

• Testing data provided in the dashboard is displayed as a 14-day average value
  

Goal

• It would be nice if we can know the real value

Possible Solution

• There is a dataset which contains testing data but it is updated on a weekly basis which is not ideal for tracking
• If we can use the historical data to prove that the average value is calculated from the same dataset we can pin point the real value from the average value

• This is because the scraping pipeline check if dashboard date match today's date and throw an error if the date does not match
• Maybe for create a special CLI argument for testing pipeline that just fetch the data and not check for date