Giter VIP home page Giter VIP logo

sensapp's Introduction

SensApp

SensApp is an open-source sensor data platform developed by SINTEF for research purposes. It manages time-series data from a multitude of sensors.

It enables the handling of small time series data of the edge efficiently to large-scale big data digital twins.

Features

  • Flexible Time Series DataBase Storage: Supports various time-series databases such as SQLite, PostgreSQL (with optional TimeScaleDB plugin), DuckDB, and ClickHouse, with the potential to extend support to other databases in the future.
  • Data Lake Storage: Supports Parquet files over S3 compatible object stores for long-term time-series data storage.
  • Multiple Data Ingestion Protocols: Easy data ingestion via HTTP REST API, MQTT, AMQP, KAFKA, OPCUA, and NATS.
  • Compatibility with Existing Pipelines: Offers Prometheus Remote Write and InfluxDB line format support for seamless integration into existing sensor data pipelines.
  • Data formats: Supports various data formats like JSON, CSV, Parquet, or SenML.

Architecture

SensApp should be stateless and scale from the edge to big data. The message queue software and the database software solve the complex problems. SensApp is a simple adapter between.

  • SensApp supports simple deployments without requiring a message queue and only an embedded SQLite database.
  • SensApp supports medium deployments with a single message broker and a PostgreSQL database.
  • For larger deployments, SensApp advises a distributed message queue, an automatic load balancer for the SensApp instances, and a ClickHouse cluster.

Check the ARCHITECTURE.md file for more details.

Built With Rust™️

SensApp is developed using Rust, a language known for its performance, memory safety, and annoying borrow checker. SensApp used to be written in Scala, but the new author prefers Rust.

Not only the language, it's also the extensive high quality open-source ecosystem that makes Rust a great choice for SensApp:

  • Tokio asynchronous runtime
  • Serde serialization framework
  • Axum web framework
  • SQLx database driver
  • Polars data frame library
  • nom parser combinator library
  • and many more…

Contributing

We appreciate your interest in contributing to SensApp! Contributing is as simple as submitting an issue or a merge/pull request. Please read the CONTRIBUTING.md file for more details.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

The SensApp software is provided "as is," with no warranties, and the creators of SensApp are not liable for any damages that may arise from its use.

You may not want to use it in production (yet)

SensApp is currently under development. It is not yet ready for production.

Acknowledgments

We thank the historical authors of SensApp who created the first version a decade ago.

SensApp is developed by SINTEF (Digital division, Sustainable Communication Technologies department, Smart Data research group).

It is made possible thanks to the research and development of many research projects, founded notably by the European Commission and the Norwegian Research Council.

We also thank the open-source community for all the tools they create and maintain that allow SensApp to exist.

sensapp's People

Contributors

bitdeli-chef avatar brice-morin avatar ffleurey avatar fungiboletus avatar jnain avatar mosser avatar nicolasferry avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sensapp's Issues

Improve error reporting

"If provided, 'units' must be defined as a IANA unit code" is not a very friendly message... it would be good to tell which unit is responsible for the problem...

Bridge CoAP ~> HTTP

- bridge between ThingML and SenML
- Generation of CoAP server to transfer data to SensApp

Services to clone and to flush raw data

When using raw sensors we often want to save a subset of the data to a new sensor and delete the rest of the data.
To do that we need 2 services:

  • A copy service which allow copying the data from a sensor to a new sensor. We should be able to specify as parameters a start and an end date.
  • A delete service which allow removing all the data from a sensor between a start and an end date.

In the admin these services can be used to have "clear data" and a "clone sensor" buttons for every sensor. A more elaborate version could allow specifying the start and end dates but this is not really needed in the admin.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.