dream-lab / echo Goto Github PK

The resource_updater loop is being started as a new Thread. However, there is no way to stop this thread which continue to persist and execute even after the main thread is killed. This needs to be handled.

Needs to include IP of master and RD and mqtt broker etc.

An AppManager object is created whenever a new Application is received by the server. The AppManager remains in memory for as long as the application is alive in the framework. During deployment, the AppManager invokes the AppDeployer interface which creates a few Map objects that is used during deployment. The same Maps are later reused whenever the Application needs to be redeployed or stopped.
It would be a useful to persist these objects in permanent storage so that the master service can reattain its previous state if it is restarted for any reason.

This would be script that pulls the required lxc container, initializes it, starts up the container and registers the device with the resource Directory.
Currently there is only a tar ball of the container which needs to be manually installed in the edge device.

The NiFi Client used by the cloud master (and platform service) is currently written in python and being interfaced by the java service using jython. porting to java would make the interfacing more seamless and easier to extend.

After deployment, when there are multiple input ports are available in the remote Nifi instance, the remote process group is not able to identify which input port to choose. This decision is made at the master side but not enforced properly at the platform side. The assumption taken is only a single input port is available. Commit d477f25 addresses this issue and selects the proper input port.
A similar handling is needed at the output port as well.

It needs to include the MQTT broker IP, the Kafka IP, the RD IP etc.

I was trying to build echo_master_service but it fails due to a failure in building the module json2pojo. The json2pojo module uses jsonschema2pojo-maven-plugin plugin to create pojo classes from the schema located under json2pojo/src/main/resources/schema. This directory contains two files : dataflow_input and wire. The dataflow_input file has the following content

{
"$schema": "http://json-schema.org/schema#",
"#id": "http://dream-lab.in/dataflow_input.json",
"type": "object",
"properties": {
"processors": {
"type": "array",
"minItems": 1,
"items": {"$ref": "processor"},
"uniqueItems": true
},
"wiring": {
"type": "array",
"minItems": 1,
"items": {
"$ref": "wire"
},
"uniqueItems": true
},
"inputstreams": {
"type": "array",
"minItems": 1,
"items": {
"type": "string"
},
"uniqueItems": true
},
"QOS": {
"type": "number"
}
}
}

wherein the processors field has a reference to processor but there exists no file of that name while the same works for wiring as there is a file of name wire. Am i getting this wrong or this is only related to the missing file ?

The rebalance works perfect for the first time. After the first rebalance, invoking rebalance again will leave the canvas with elements which should be removed from the canvas such as processors which are moved to some new device (this processor is created on the new device but should be removed from the old device), input ports ,rpgs etc. This is happening because to remove the processor, first need to stop it and stop its connected elements such as ports and rpgs. During rebalance call, the various maps which holds the mapping of elements to a particular device are updated. The implementation is not correct which keeps some unused elements there as well. Since after one rebalance this mapping is messed up , we get a correct view of the rebalanced configuration but again rebalancing will show various should-be-removed elements there.
This can be fixed by properly updating the wiring maps of individual devices wherein currently we are only updating them with new wirings if they have no wirings previously. This assumption is wrong and we should append the new wirings with the wirings which are left after stopping and removing various assets in this new configuration. (Fixed on new branch and will be merged later with master).

Test the MQTT broker from the cloud.

Currently the Service is not very robust. It can handle only one requests thread at a time and doesn't seem to have a good enough fault tolerance. Need to either refactor the code or replace the service with an already existing java REST server.

This is related to rebalance in case when we are running in a configuration with some number of devices and switch to a configuration where not all of these devices are used. The reason for this is when we are done with assets stop and removal and go for reconnecting new assets, while starting the processors (using 'start_processor') , there is no check while sending 'start_processor' message to the the device's topic (UUID of the device) whether there is any ControlMethod added to the ControlDatagram. As a result, sending a blank message to the mqtt topic on which the device is listening will result in no response from the platform service running on the device. The check of having one or more ControlMethods in the sending ControlDatagram resolves this issue. However the proper solution to such scenario is before starting to reconnect the assets, check which all devices are present from the previous configuration that contains no processors, rpgs and ports. These devices can then be removed from all the included maps.
(Changes will be a made to a different branch and later will be merged with the master).

During rebalance phase, new wirings are created between devices (via ports and rpgs). However the data structure which maintains these wirings is not properly updated. As a result, calling rebalance again will have extra elements on each device which should have been removed. This fix is necessary to fix the issue #15

Platform service has dependencies like paho that need to be pip installed before it is run.

This could be added in the bootstrap.py or user needs to explicitly install it.

When a rebalance is called on a submitted dag, the input ports are created but not enabled. This doesn't happen for the output ports. Looking into the python side (platform service) , the output port is enabled as soon as it is created. However this is not the case for input port as well kafka port. For both of these, separate message must be sent ('enable_input_port' and 'enable_kafka_port' respectively) to the listening platform service on the device where these ports are placed. Sending these messages during rebalance seems to fix the issue. The api can be modified to keep a unified behaviour.

Rebalance an already existing dataflow via a REST API. Only the diff of the dataflows are moved.