Giter VIP home page Giter VIP logo

zenoh-flow's Introduction

Eclipse Zenoh-Flow

Join the chat at https://gitter.im/atolab/zenoh-flow

Zenoh-Flow provides a zenoh-based dataflow programming framework for computations that span from the cloud to the device.

⚠️ This software is still in alpha status and should not be used in production. Breaking changes are likely to happen and the API is not stable.


Description

Zenoh-Flow allow users to declare a dataflow graph, via a YAML file, and use tags to express location affinity and requirements for the operators that makeup the graph. When deploying the dataflow graph, Zenoh-Flow automatically deals with distribution by linking remote operators through zenoh.

A dataflow is composed of set of nodes: sources — producing data, operators — computing over the data, and sinks — consuming the resulting data. These nodes are dynamically loaded at runtime.

Remote source, operators, and sinks leverage zenoh to communicate in a transparent manner. In other terms, the dataflow the dafalow graph retails location transparency and could be deployed in different ways depending on specific needs.

Zenoh-Flow provides several working examples that illustrate how to define operators, sources and sinks as well as how to declaratively define they dataflow graph by means of a YAML file.


How to build it

Install Cargo and Rust. Zenoh Flow can be successfully compiled with Rust stable (>= 1.5.1), so no special configuration is required — except for certain examples.

To build Zenoh-Flow, just type the following command after having followed the previous instructions:

$ cargo build --release

How to run

Assuming that the previous steps completed successfully, you'll find the Zenoh-Flow runtime under target/release/runtime. This executable expects the following arguments:

  • the path of the dataflow graph to execute: --graph-file zenoh-flow-examples/graphs/fizz_buzz_pipeline.yaml,
  • a name for the runtime: --runtime foo.

The graph describes the different nodes composing the dataflow. Although mandatory, the name of the runtime is used to "deploy" the graph on different "runtime instances" (see the related examples).


Creating your nodes

Assuming that the build steps completed successfully, you'll be able to use the cargo zenoh-flow subcommand to create a boilerplate for your nodes. First let's ensure to have the cargo-zenoh-flow binary in the Cargo path.

$ ln -s $(pwd)/target/release/cargo-zenoh-flow ~/.cargo/bin/

Then you can create your own node with:

$ cd ~
$ cargo zenoh-flow new myoperator

By default cargo zenoh-flow generates the template for an operator. In order to create a source or a sink you need to add either --kind source or --kind sink.
The Cargo.toml will contain metadata information (eg. the inputs/outputs) used during the build process to generate the descriptor.

More information about the cargo zenoh-flow can be obtained using cargo zenoh-flow --help.
You can now modify the src/lib.rs file with your business logic and update the Cargo.toml according to the inputs/outputs that you need.

Once you are done you can build it:

$ cargo zenoh-flow build

It will provide you the path of the descriptor for the new node, that can be used inside a flow descriptor.


Examples

Examples can be found in our example repository.

zenoh-flow's People

Contributors

gabrik avatar haianos avatar j-loudet avatar jenoch avatar kydos avatar mallets avatar olivierhecart avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.