Giter VIP home page Giter VIP logo

delta-producer-dump-file-publisher's Introduction

delta-producer-dump-file-publisher

Service that produces and publishes a dump file of the current state of deltas. It publishes according to the DCAT v2 standard. Api is provided by mu-resource. Files are hosted by mu-file.

Installation

docker-compose.yml

Per cache-graph to manage, you will need an instance of the service, suppose here it is related to leidinggevenden. To add the service to your mu.semte.ch stack, add the following snippet to docker-compose.yml:

Add configuration

A config.json needs to be present under /config/config.json

{
  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/leidinggevenden" : {
    "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/LeidinggevendenCacheGraphDump",
    "targetGraph": "http://redpencil.data.gift/id/deltas/producer/loket-leidinggevenden-producer",
    "fileBaseName": "dump-leidinggevenden",
    "targetDcatGraph": "optional graph where the information of the data set should reside",
    "targetFilesGraph": "optional graph where the meta data of the ttl should reside"
  }
}
services:
  delta-producer-dump-file-publisher-leiddingevenden:
    image: lblod/delta-producer-dump-file-publisher:x.x.x
    volumes:
      - ./config/delta-producer/dump-file-publisher:/config
      - ./data/files:/share

Wire the deltas

This service works by receiving deltas from the delta-notifier. It should be configured as such :

  {
    match: {
      predicate: {
        type: 'uri',
        value: 'http://www.w3.org/ns/adms#status'
      },
      object: {
        type: 'uri',
        value: 'http://redpencil.data.gift/id/concept/JobStatus/scheduled'
      }
    },
    callback: {
      url: 'http://delta-producer-dump-file-publisher-leiddingevenden/delta',
      method: 'POST'
    },
    options: {
      resourceFormat: 'v0.0.1',
      gracePeriod: 1000,
      ignoreFromSelf: true
    }
  }

Environment variables

Provided environment variables by the service. These can be added in within the docker declaration.

Name Description Default
SERVICE_NAME The name of the service delta-producer-dump-file-publisher
EXPORT_TTL_BATCH_SIZE Size of the batched queries 1000
DUMP_FILE_CREATION_TASK_OPERATION Uri of the dump task http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation
FILES_GRAPH Graph to store the file (can be set per job too) http://mu.semte.ch/graphs/public
DCAT_DATASET_GRAPH Graph to store the dcat dataset (can be set per job too) http://mu.semte.ch/graphs/public
RELATIVE_FILE_PATH relative path to store the files under delta-producer-dumps
JOBS_GRAPH graph where the jobs resides http://mu.semte.ch/graphs/system/jobs
GRAPH_TO_DUMP Graph to dump in file
EXPORT_FILE_BASE_NAME Base name of the dump file
DUMP_FILE_CREATION_JOB_OPERATION Uri of the dump task's job
DCAT_DATASET_SUBJECT The dct:subject URI of the dataset

API

Will be served by mu-resource. See e.g. https://github.com/kanselarij-vlaanderen/app-themis/blob/master/config/resources/dcat.json for the api definition.

Development

For a more detailed look in how to develop a microservices based on the mu-javascript-template, we would recommend reading "Developing with the template"

Developing in the mu.semte.ch stack

Paste the following snip-it in your docker-compose.override.yml:

delta-producer-dump-file-publisher:
  image: semtech/mu-javascript-template:1.4.0
  environment:
    NODE_ENV: "development"
  volumes:
    - /absolute/path/to/your/sources/:/app/

TODO

  • period clean up of deprecated datasets
  • eventually add dcat:Dataset to dcat:Catalog
  • upate logic to remove the need for one instance per cache graph to maintain
  • add as a result container a link to the data set uri. So we can click through in jobs dashboard

Experimental features

config.json

In case the publication graph is residing in a different endpoint as the default database, you can add

{
  "publicationGraphEndpoint": "http://different/endpoint"
}

All other queries will keep using the default database.

delta-producer-dump-file-publisher's People

Contributors

cecemel avatar claire-lovisa avatar madnificent avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.