hobbit-project / ontology Goto Github PK

The ontology used by the HOBBIT platform

Makefile 100.00%

ontology's Introduction

ontology

This repository contains the HOBBIT ontology definition in the ontology.ttl file. The ontology is used to describe the benchmarking of systems including:

Benchmarks with their parameters, features and KPIs
Systems and their features
Experiments in which a single system is benchmarked with a certain benchmark
Results of an experiment
Challenges and their single challenge tasks as well as
Additional information that is needed to rerun the same experiment again, e.g., the hardware on which the experiment has been carried out

Additionally, this repository contains the definition of errors that can occur during the benchmarking of a system inside the HOBBIT platform. The errors are defined in the errors.ttl file.

Example is provided in the example/example.ttl file.

ontology's People

Contributors

Stargazers

Watchers

Forkers

mjovanovik

ontology's Issues

Wrong hobbit prefix

The prefix in the ontology file is hobbit: <http://w3id.org/hobbit/experiments#> . but should be hobbit: <http://w3id.org/hobbit/vocab#> .

We should add provenance data to the experiment results. I think that it would be even possible to define some of the HOBBIT classes as subclasses of the PROV-O classes to make the usage of PROV-O easier.

https://www.w3.org/TR/prov-o/

hobbit:KPI subclasses should define ordering for their instances

We need to be able to correctly sort KPI values to implement leaderboards.

At least ascending and descending sorting orders are needed.

Default ordering may be specified in hobbit:KPI class.

Properties that have type `date` actually used as `dateTime`

Check if it's true for all such properties and reflect that in ontology.

Main KPI or list of KPIs in challenge tasks

We need to be able to specify main KPI for challenge tasks, to implement leaderboards.

Proposed name: hobbit:hasMainKPI.

Another possibility is to specify a list of KPIs instead of that for tie breaks and just use it instead - we probably would need such a list anyway.

hobbit:visible property is missing

Used on challenge entities.

Introduce an Executable class (or similar)

To improve the reusability of the ontology, it is important to keep the domain and range of properties open to other use cases. Properties like the parameters or the hobbit:imageName property are restricted to the union of system instances and benchmarks. However, they actually could be useful for every container that can be executed. Hence, a super class of system instances and benchmarks should be introduced (e.g., Executable) that represents a more generic class of resources that simply can be executed and may need some parameters.

Regularly executable challenges

new property for period of challenge execution, maybe hobbit:executionPeriod
final date of challenge execution, maybe hobbit:executionEndDate (currently executionDate exists for start date)