Giter VIP home page Giter VIP logo

watchdoge_apientreprise's Introduction

README Build Status Maintainability Test Coverage

Build Status (develop branch)

Watchdoge is a watchdog for API Entreprise, the monitoring done by Watchdoge is available on the Dashboard

Watchdoge run on Ruby 2.6.2

Elasticsearch

The API uses Elasticsearch API, only authorised IP can make requests to Elasticsearch API, add yours for development purpose.

sudo ufw allow from <your.ip> to any port 9200 proto tcp
sudo ufw delete allow from <your.ip> to any port 9200 proto tcp

You can test your ELK queries in Kibana.

Configuration

Database

The database user should exists, it uses PostgreSQL (cf. config/database.yml). Execute the following command line to create the default user:

sudo -u postgres -i
cd /path/to/watchdoge
psql -f db/init.sql

And

bundle exec rails db:create:all
bundle exec rails db:migrate RAILS_ENV=<desired_env>

Development environment

Run:

rake dev:init

It will init the development environement (defaults config files & database)

Add new API to test

Add a block in app/data/endpoints.yml following this schema (have a look to existing and test with rake) :

- uname: <unique name>
  name: <display name>
  api_name: <apie or sirene>
  provider: <provider name only relevant for apie>
  api_version: <no more used because v1 is deprecated>
  ping_period: <period between pings in minutes>
  http_path: <http path following the base url>
  http_query: <json of http query params if needed>

Rake tasks

You will need to fill config/secrets.yml with valid jwt for each RAILS_ENV you'd like to run the tasks.

Run the main job in a rake task and get debug info (specify RAILS_ENV):

rake watch:all

Can run only one endpoint (specify RAILS_ENV):

rake watch_v2:one apie_2_attestations_sociales_acoss

Can run only one API :

bundle exec rake watch:apie
bundle exec rake watch:sirene
bundle exec rake watch:rna
bundle exec rake watch:data.gouv.fr

Deployment

Run:

mina deploy domain=<MY IP> to=<development|production>

Dependencies

Needs PostgreSQL installed, certifcates (and IP whitelist) for API Entreprise and it's providers

Services

Watchdoge use whenever to perform periodic pings. Mina updates crontab on deployment.

Tests

Run:

bundle exec rspec

Very long tests :

bundle exec rspec --tag very_long_test

watchdoge_apientreprise's People

Contributors

am-beta avatar brindu avatar dependabot[bot] avatar haelle avatar hugolepetit avatar kheops2713 avatar samuelfaure avatar skelz0r avatar teleboas avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

watchdoge_apientreprise's Issues

Monitoring de l'ADEME

Ajouter le monitoring de l'endpoint RGE de l'ADEME.

Jusqu'ici j'ai pris le premier siret venu qui fonctionnait pour les devs (34812725900052), pour le monitoring il peut être mieux de choisir une boite précise (une "grosse" dans le domaine de quelconques travaux pour laquelle il y a peu de chance que la ou les certifications expirent) mais je n'ai pas spécialement d'idée.

L'annuaire en ligne des certifications RGE est disponible ici pour trouver un bon siret.

Upgrade to Sidekiq 6

Currently we cannot upgrade to Sidekiq 6 because it requires Redis >4 and Ubuntu 16.04 does not have it natively.

To do when server upgrade is done.

Insee v3 endpoints monitoring

Les tests ne passent plus car différence entre la réalité et les specs sur les endpoints insee v3

rspec ./spec/aggregators/dashboard/ping_history_aggregator_spec.rb:24 # Dashboard::PingHistoryAggregator with valid input size should eq 27
rspec ./spec/services/dashboard/current_status_service_spec.rb:26 # Dashboard::CurrentStatusService results contains specifics endpoints names and sub_names
rspec ./spec/services/dashboard/current_status_service_spec.rb:20 # Dashboard::CurrentStatusService results size should equal 27

Check response format

Check response format in ping :

  • if API expected JSON
  • if front expected HTML

actually if it's a 200 it's OK even if it's not an JSON API response...

Identifier et remonter les indisponibilités des fournisseurs

Objectifs :

  • identifier les indisponibilités des fournisseurs
  • lever des alertes auprès d'eux
  • proposer ce service d'alerte aux clients d'API Entreprise (dans un second temps imo)

Besoin de contacts auprès de nos différents fournisseurs à qui remonter les alertes.

Requêter Elasticsearch en HTTPS

Même si toutes les données demandées ne sont pas forcément privées, le TLS assure leur intégrité et évite d'éventuelles injections.

HTTPS n'est pas encore disponible côté Elasticsearch mais lorsque ça sera le cas ça serait bien de remplacer les URL en http://kibana… par https://kibana….

Je prends note de ça ici pour ne pas oublier d'ici là.

[CallResult] Sanitize parameters & rename class

Renaming

CallCharacteristics to CallResults
CallCharacteristics.to_json => CallResults.as_json...

Refactoring

better naming extract_parameters to sanitize_parameters + extact_parameters

Duplicate entry in 'Temps réel'

When updating the parameter in the database, it fails to aggregate correctly.

To agregate correctly use : ELK route.keyword parameter.

Warning : using ELK controller.keyword would make some API disappear (liasses_fiscales_dgfip has many action on the same controller)

Full CI

Remettre :

  • Travis
  • CodeClimate
  • Déploiement auto

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.