ec-jrc / cbm Goto Github PK

Checks by Monitoring

License: BSD 3-Clause "New" or "Revised" License

Jupyter Notebook 12.30% Dockerfile 0.07% Python 48.39% HTML 35.04% R 4.17% TeX 0.03%

cbm's Introduction

Repository of the JRC-CbM for the CAP

This repository provides access to the Checks by Monitoring (CbM) project code and technical documentation developed by the Joint Research Centre (JRC) to support Member States (MS). Here the Guidance and Tools for CAP (GTCAP) JRC group shares a CbM system layout and code examples to demonstrate how Paying Agencies (PA) can process and use Sentinel data to check aid applications for the common agricultural policy (CAP). The JRC CbM is based on a cloud infrastructure solution that is modular and built exclusively on open source components.

The technical description of the system components and the instruction on how to set up and use a CbM system are available in the JRC CbM TECHNICAL DOCUMENTATION.
A general overview of the system architecture and scope is provided in the JRC CbM GENERAL DOCUMENTATION.

JRC CbM ARCHITECTURE is made of two layers. The first is the BACKEND SERVER that provides the end-points to retrieve data and includes the physical infrastructure and the routines that generates the information used by the analysts. The second is the FRONTEND COMPONENT that is manipulated by the user and provides access to the data generated by the backend through standard Application Programming Interface (API) and the functions and tools to analyse the data.

In this repository we share the code developed for both the backend and the frontend, structured according to three levels:

Setting up the infrastructure (target users: system administrators)
Develop analytical functionalities (target users: analysts)
Apply the functionalities (target users: final users)

Structure of the repository

The scripts and documentation are organized in the following sections/subfolders:

api: Modules to build a RESTful API for CbM
cbm: CbM Python library
docker: Docker image files
docs: Documentation pages with example codes
ipynb: Jupyter Notebook examples
scripts: Python scripts for signal processing (marker detection, parcel extraction routines and time series calendar view)
tests: Test scripts for generic functionalities

General requisites to set up the CbM system

In the framework of the Outreach project, an operational cloud infrastructure (based on CreoDIAS) has been created by GTCAP with the backend component developed and managed by JRC and offered to Member States to experiment the functionalities of the system. Here, MS can use dedicated API to explore and analyse Sentinel data extracted for their declared parcels using Python and Jupiter Notebooks.

A PA/MS that wants to developed and managed its own CbM system must have as a prerequisite:

Computing resources
Copernicus Analysis Ready Data (CARD) (Sentinel 1 and 2 data for the study area)
Agricultural parcel data (typically, declared parcels from the Land Parcel Identification System (LPIS) and the Geospatial Aid Application (GSAA))

The first two requisites can be achieved using one of the five Copernicus Data and Information Access Services (DIAS) available (CREODIAS, WEKEO, SOBLOO, MUNDI, ONDA).

Deployment of the CbM system

There are several steps to set up the core components for CbM that require different types of technical expertise.

Setup server applications

Docker (containerization system)
PostgreSQL database with PostGIS extension
Jupyter (interactive analysis and visualization environment)
Restful API (intermediate layer to access and use Copernicus data and the database)

Adding data to the database
- Parcels data
- CARD Metadata and other setting data
Process Sentinel data to derive relevant information
- Parcel stats extraction routines
- Machine learning algorithms
- Analytical routines (e.g. markers detection)
Analyzing and reporting
- Using Jupyter Notebooks to explore and analyze data
- Generate reports and other outputs to classify aid applications

Contributing

This repository is open for contribution, particularly from PA experts. Please read "Creating a pull request" for details on the process for submitting pull requests.

Authors

Guido Lemoine - glemoine62
Konstantinos Anastasakis - konanast

See also the list of contributors who participated in this project.

License

This project is licensed under the 3-Clause BSD license - see the LICENSE file for details.

cbm's People

Contributors

Stargazers

Watchers

cbm's Issues

scipy.signal.filtfilt only returns nans if input contains nans

from scipy import signal
import numpy as np
B = np.array([1.06465635e-11, 1.38405326e-10, 8.30431953e-10, 3.04491716e-09,
       7.61229290e-09, 1.37021272e-08, 1.82695030e-08, 1.82695030e-08,
       1.37021272e-08, 7.61229290e-09, 3.04491716e-09, 8.30431953e-10,
       1.38405326e-10, 1.06465635e-11])
A = np.array([ 1.00000000e+00, -1.03940772e+01,  5.00808900e+01, -1.48071732e+02,
        2.99671908e+02, -4.38305805e+02,  4.76569467e+02, -3.89960803e+02,
        2.40104193e+02, -1.09852658e+02,  3.62966803e+01, -8.20079127e+00,
        1.13548913e+00, -7.27601400e-02])
values = np.array([         np.nan,          np.nan,          np.nan,          np.nan,
                np.nan,          np.nan,          np.nan,          np.nan,
                np.nan,          np.nan,          np.nan,          np.nan,
                np.nan,          np.nan,          np.nan,          np.nan,
                np.nan,          np.nan,          np.nan,          np.nan,
                np.nan, 307.35974121, 298.75695583, 290.15417045,
       281.55138506, 272.94859968, 264.3458143 , 255.74302891,
       247.14024353, 263.82520549, 280.51016744, 297.19512939,
       294.65074851, 292.10636763, 289.56198675, 287.01760587,
       284.47322499, 281.92884411, 279.38446322, 276.84008234,
       274.29570146, 271.75132058, 269.2069397 , 266.66255882,
       264.11817793, 261.57379705, 259.02941617, 256.48503529,
       253.94065441, 251.39627353, 248.85189264, 246.30751176,
       243.76313088, 241.21875   , 244.45207596, 247.68540192,
       250.91872787, 254.15205383, 257.38537979, 260.61870575,
       263.85203171, 267.08535767, 266.35974808, 265.63413849,
       264.9085289 , 264.18291931, 263.45730972, 262.73170013,
       262.00609055, 261.28048096, 260.55487137, 259.82926178,
       259.10365219, 258.3780426 , 257.65243301, 256.92682343,
       256.20121384, 255.47560425, 254.74999466, 254.02438507,
       253.29877548, 252.57316589, 255.13048096, 257.68779602,
       260.24511108, 262.80242615, 265.35974121, 272.71584473,
       280.07194824, 287.42805176, 294.78415527, 302.14025879,
       308.94220947, 315.74416016, 322.54611084, 329.34806152,
       336.15001221, 342.95196289, 349.75391357, 356.55586426,
       363.35781494, 370.15976563, 376.96171631, 383.76366699,
       390.56561768, 397.36756836, 404.16951904, 410.97146973,
       417.77342041, 424.57537109, 431.37732178, 438.17927246,
       444.98122314, 451.78317383, 458.58512451, 465.3870752 ,
       472.18902588, 476.42438965, 480.65975342, 484.89511719,
       489.13048096, 493.36584473, 497.6012085 , 501.83657227,
       506.07193604, 510.3072998 , 514.54266357, 525.10781583,
       535.67296808, 546.23812034, 556.80327259, 567.36842485,
       577.9335771 , 588.49872936, 599.06388161, 609.62903387,
       620.19418612, 630.75933838, 641.32449063, 651.88964289,
       662.45479514, 673.0199474 , 683.58509965, 694.15025191,
       704.71540416, 715.28055642, 725.84570867, 736.41086093,
       746.97601318, 726.80515544, 706.63429769, 686.46343994,
       693.94025879, 701.41707764, 708.89389648, 716.37071533,
       723.84753418, 709.91827393, 695.98901367, 682.05975342,
       668.13049316, 654.20123291, 640.74268799, 627.28414307,
       613.82559814, 600.36705322, 586.9085083 , 569.68502808,
       552.46154785, 549.76888483, 547.07622181, 544.38355879,
       541.69089577, 538.99823275, 536.30556974, 533.61290672,
       530.9202437 , 528.22758068, 525.53491766, 522.84225464,
       520.14959162, 517.4569286 , 514.76426558, 512.07160256,
       509.37893954, 506.68627652, 503.9936135 , 501.30095048,
       498.60828746, 495.91562445, 493.22296143, 490.53029841,
       487.83763539, 485.14497237, 482.45230935, 479.75964633,
       477.06698331, 474.37432029, 471.68165727, 468.98899425,
       466.29633123, 463.60366821, 481.52440186, 499.4451355 ,
       517.36586914, 535.28660278, 553.20733643, 563.47075195,
       573.73416748, 583.99758301, 594.26099854, 604.52441406,
       615.4834246 , 626.44243513, 637.40144566, 648.36045619,
       659.31946673, 670.27847726, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779, 681.23748779, 681.23748779,
       681.23748779, 681.23748779])
signal.filtfilt(B, A, values)

Output:

array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])

Error in the .format of the cbm/scripts/extraction/postgisS2Extract.py

In the script "cbm/scripts/extraction/postgisS2Extract.py", the query parcelcountsql on line 127 has 7 "{}" and the .format on line 137 has 9 words. The code of incurs.execute should look like this:

incurs.execute(parcelcountsql.format(
    dbconfig['tables']['parcel_table'],
    dbconfig['tables']['aoi_table'], srid, srid,
    dbconfig['args']['aoi_field'], dbconfig['args']['name'],oid))

problem installing setup.py

Am i alone with this issue?

Traceback (most recent call last):
  File "setup.py", line 13, in <module>
    long_description = fh.read()
  File "...\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 159: character maps to <undefined>

ec-jrc / cbm Goto Github PK

cbm's Introduction

Repository of the JRC-CbM for the CAP

Structure of the repository

General requisites to set up the CbM system

Deployment of the CbM system

Contributing

Authors

License

cbm's People

Contributors

Stargazers

Watchers

Forkers

cbm's Issues

scipy.signal.filtfilt only returns nans if input contains nans

Error in the .format of the cbm/scripts/extraction/postgisS2Extract.py

problem installing setup.py

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent