doreapp / prix-carburants Goto Github PK

On github-pages website, add a javascript script that fetches the data from data branch and display very basic information.

Depends on #12 and #10.

Initialize the project

• Create a python module
• Fetch basic data from https://www.prix-carburants.gouv.fr/
• Include a README explaining how to use the library
• Add automation (CI/CD)

Add all the data available in Open Data to data branch.

Needs #19

Create source code for a github-pages website using Jekyll.

It must include:

• A very basic webpage
• Documentation about how to install, build and display the website (on linux only)

Add missing information in current parsing:

• opening and closing hours
• several prices with the same fuel, but at different times (for data on several days)
• out of order of some fuels
• closing days
• services

• Create a branch data in this repository
• Create a command to save the data generated (thanks to #8) in this branch

The command may be a Makefile target or a python script.

It can be done either using git tools or GitHub REST API.

Depends on #8.

Should provide a static analysis of

• [MUST] Javascript
• [MAY] HTML, Markdown

To show data on github pages we do not need to have all the data from every day save in a dedicated branch.
We only need to have relevant data in gh-pages/assets/data folder.

Therefore, remove actions that fetches frequently data and save it in data branch & remove the branch

Add an action that runs when the website sources are edited and that deploy the source code to gh-pages branch of this repository.

Depends on #12.

Some errors occurs during the update of the data (as https://github.com/Doreapp/prix-carburants/actions/runs/4476648671/jobs/7867225579 and https://github.com/Doreapp/prix-carburants/actions/runs/4486219971/jobs/7888499757 can showcase).

This issue is meant to improve the logging system, so that such errors are understood (even though not catched yet)

For now, we consider that department codes are "01" to "95". We can imagine adding a 96th department, starting at 0, removing a code or adding a non-numeric code, such as 2A for Corsica.

The list of departments to use should be dynamically found from the data

• In the map info layer, use french:

• Also rename "Hover a state" inof

Rather than using

python -m prixcarburants download instantaneous

We would use

python -m prixcarburants download now

Once #10 is done, create an action to automatically and frequently (e.g. every 3 hours) fetch new data from Open Data and save it in the data branch.

Save the data as latest.

This new data should come from instantaneous data source, and contains only meaningful values, i.e.:

• id
• address (city, address, postcode)
• fuels
  • id
  • price if available
  • reason of unavailability (out of order, no service)
• pairs fuel_id / fuel_name

Parse the data fetch from open data

Javascript files are under /assets/javascript/. They are not found on the deployed website. I think they should be under assets/js/ so that GitHub pages accept them.

Some of the continuous integration workflow are failing.

403

• https://github.com/Doreapp/prix-carburants/actions/runs/4971715332/jobs/8896440793
• https://github.com/Doreapp/prix-carburants/actions/runs/4967689998/jobs/8889832900

Given those jobs, the error is coming from a 403 HTTP error:

403 Client Error: rate limit exceeded for url: https://api.github.com/repos/Doreapp/prix-carburants/contents/assets/data/metrics.json?ref=gh-pages

502

Some other jobs (i.e. https://github.com/Doreapp/prix-carburants/actions/runs/4942342860/jobs/8835770321) are failing due to

502 Server Error: Bad Gateway for url: https://api.github.com/repos/Doreapp/prix-carburants/contents/assets/data/latest.json

https://github.com/Doreapp/prix-carburants/actions/runs/4942342860/jobs/8835770321

The request should retry when failing on CI, multiple time with a time delta between each.

Display a Map showing every sale point prices of a given fuel type

Disable fuel selector while data is still loading, and display sliders

From fetched and parsed zip file, create a file in a web format (such as json or yaml).

Should be able to use it like:

$ python -m prixcarburants download day
Downloaded in .tmp/PrixCarburants_jour.xml
$ python -m prixcarburants transform .tmp/PrixCarburants_jour.xml
Data saved in .tmp/PrixCarburants_jour.json

Depends on #7.

• Using python script, extract average prices per department (2 first digits of the postcode) an fuel type into metrics
• On website, display a Map view with prices, using a library such as plotly.js (see this example)

Add -l / --level parameter, that takes one argument: the logging level.
Choices are : debug, info, warning and error

Create design files about the direction of the app.

It should contain goals and possible approaches.

Be able to list sale points from latest data

From latest data saved every hour, extracts metrics and display them in github-pages website.

Metrics are:

• average prices by type (gazole, GLPc, E10...)

This metrics must be saved in /_data/ directory in gh-pages branch, and be used to display the data using jekyll templating

Rather than having a dictionary like prices[department][fuel_type], have prices[fuel_type][department].

Needed as we more often split by fuel_type rather than by department.

Moreover, rather than using numbers as key, transform the prices and fuel_names objects in arrays:

{
  "prices" : [
    {
      "44": 2.2
      ..
    }, {
      "44": 1.1
      ..
    }
  ],
  "fuel_names": [...]
}

During long execution, such as file downloading of parsing, display progress bars

Add https://donnees.roulez-eco.fr/opendata/jour URL as the source for daily data.

Goal:

• Be able to fetch data related to the current day

Fix error from https://github.com/Doreapp/prix-carburants/runs/6677547566?check_suite_focus=true#step:6:25

To download JS libraries (leaflet for now), we are currently using a makefile target that uses curl commands.. Improve this process.