albertosantagostino / itaqa-air-quality-aggregator Goto Github PK

Branch: crawler/veneto

• Implement the data crawler from ARPA Veneto

See #5 (comment)

Without knowing how many days data is retained and available in the various ARPA websites, a snapshot of the air quality situation in the various regions should be made (if possible pre-processed data dumps, if not possible even screenshots at the data pages) to collect pollution data at the beginning of March before its possible expiration

• Abruzzo
• Basilicata
• Calabria
• Campania
• Emilia-Romagna
• Friuli-Venezia Giulia
• Lazio
• Liguria (past data always available, wait for crawler)
• Lombardia
• Marche
• Molise
• Piemonte
• Puglia
• Sardegna
• Sicilia
• Toscana
• Trentino-Alto Adige
• Umbria
• Valle d'Aosta
• Veneto

This is to make visualization consistent

A decision on the database framework to use is needed, to be done after a technical analysis of different possible solutions

If needed:

• Strip it, keep only needed parts, evaluate when is updated (performance analysis)
  If not needed:
• Remove it, refactor all the places that use the information to get it directly

Evaluate the advantages when encapsulating Encapsulate AQS lists within dedicated objects of class AirQualityStationCollection (?)
Refactor on a separate experimental branch

Possible useful methods
get_overall_info(): Print a table containing info on all the included AQS:

This is now a necessity and something to do soon, before other things are implemented

Plot ideas

• Lombardia (all stations) in time for each pollutant. Vertical bar to indicate lockdown due to quarantine in Italy

Ciao!

La tua repository è stata selezionata dalla nostra redazione per il format "FOSS Explorer" di Schrodinger Hat, una rubrica del nostro talk show riguardo il software development e il mondo open source italiano, e verrà mostrata nella puntata di Martedì 26 Maggio in diretta sul canale Twitch e YouTube alle ore 13.00.

Il tuo lavoro ci è piaciuto molto e speriamo che questo possa aiutare a fornire visibilità al tuo lavoro ;)
Se per te va bene, la includeremo nella nostra top15 bisettimanale!

A presto!

• Implement the data crawler from ARPA Lombardia

When one or more AQS are dumped to msgpack, create also a (yaml?) file with the same name containing info on amount of data, time range...

Most likely in msgpack, to quickly save/reload large AQS lists

Due to changes in data format, the crawler needs to be rewritten

• Store locally "final" data (all complete data of past years), without redownloading it each time
• Refactor crawler to improve performance

Branch: crawler/emiliaromagna

• Implement the data crawler from ARPA Emilia-Romagna