smartfog / fogflow Goto Github PK

FogFlow is a standard-based IoT fog computing framework that supports serverless computing and edge computing with advanced programming models

Home Page: https://fogflow.readthedocs.io

License: BSD 3-Clause "New" or "Revised" License

Go 100.00%

serverless-functions edge-computing fog-computing fiware iot ngsi-ld ngsi function-as-a-service serverless-computing programming-model

fogflow's Introduction

FogFlow

FogFlow is an IoT edge computing framework to automatically orchestrate dynamic data processing flows over cloud and edges driven by context, including system context on the available system resources from all layers, data context on the registered metadata of all available data entities, and also usage context on the expected QoS defined by users.

This project is part of FIWARE. For more information check the FIWARE Catalogue entry for Processing.

このドキュメントは日本語でもご覧いただけます。

📚 Documentation	🎓 Academy	🐳 Docker Hub	🎯 Roadmap

Content

Background
Installation
Tutorial
API
Testing
Quality Assurance
Roadmap
Publications
Talks

Background

FogFlow is a standard-based data processing framework for service providers to easily program and manage IoT services over cloud and edges. Below are the motivation, functionalities, and benefits of FogFlow.

Why do we need FogFlow?
- the cost of a cloud-only solution is too high to run a large scale IoT system with >1000 geo-distributed devices
- many IoT services require fast response time, such as <10ms end-to-end latency
- service providers are facing huge complexity and cost to fast design and deploy their IoT services in a cloud-edge environment
- business demands are changing fast over time and service providers need to try out and release any new services over their shared cloud-edge infrastructure at a fast speed
- lack of programming model to fast design and deploy IoT services over geo-distributed ICT infrastructure
- lack of interoperability and openness to share and reuse data and dervied results across various applications
What does FogFlow provide?
- efficient programming model: programming a service is like building lego blocks
- dynamic service orchestration: launching necessary data processing only when it is required
- optimized task deployment: assigning tasks between cloud and edges based on the locality of producers and consumers - scalable context management: allowing flexible information exchanging (both topic-based and scope-based) between producers and consumers
How can customers benefit from FogFlow?
- fast time-to-market when realizing and releasing new services over the shared, geo-distributed ICT infrastructure
- reduced operation cost and management complexity when operating variou services
- being able to provide services that require low latency and fast response time

Installation

The instructions to install FogFlow can be found in the Installation Guide

API

APIs and examples of their usage can be found here

Testing

For performing a basic end-to-end test, you can follow the detailed instructions here.

Quality Assurance

This project is part of FIWARE and has been rated as follows:

Version Tested:
Documentation:
Responsiveness:
FIWARE Testing:

Publications

B. Cheng, G. Solmaz, F. Cirillo, E. Kovacs, K. Terasawa and A. Kitazawa, “FogFlow: Easy Programming of IoT Services Over Cloud and Edges for Smart Cities,” in IEEE Internet of Things Journal, vol. 5, no. 2, pp. 696-707, April 2018, doi: 10.1109/JIOT.2017.2747214. IoT Journal, 2020 Best Paper Award Runner-Up
Cheng, Bin, Jonathan Fuerst, Gurkan Solmaz, and Takuya Sanada. "Fog function: Serverless fog computing for data intensive iot services." In 2019 IEEE International Conference on Services Computing (SCC), pp. 28-35. IEEE, 2019. IEEE SCC, 2019 Best Paper Award

Talks

fogflow's People

Contributors

Stargazers

Watchers

fogflow's Issues

Build does not work

PLEASE update your docker files, so the build process is contained within the docker containers.
You can find examples here:
https://docs.docker.com/develop/develop-images/multistage-build/

It is very easy.

[MUST] Every GE must have training materials

At the moment this enabler has no Tutorial and no link to the Academy and does not fulfil the minimum Training section requirements. There are no FogFlow training materials to be found in the Academy

The suggestion is to create a simple FogFlow to Orion tutorial to provide a series of simple hello world connections for simple processing - this needs to include

One docker compose file for base config
A piece of code to enable Fog computing
An indication on how to make minimal measure arrive (e.g. publishing an MQTT topic or equivalent)
Proof that the data arrives in Orion.

Presence of training is a MUST requirement

go get fail

$ go get
# fogflow/broker
./thinBroker.go:48:22: multiple-value uuid.NewV4() in single-value context
./thinBroker.go:534:21: multiple-value uuid.NewV4() in single-value context

$ go get
# fogflow/discovery
./fastDiscovery.go:45:31: multiple-value uuid.NewV4() in single-value context
./fastDiscovery.go:205:21: multiple-value uuid.NewV4() in single-value context

$ go get
# fogflow/master
./taskGeneration.go:49:50: multiple-value uuid.NewV4() in single-value context

I think a change of specifications of https://github.com/satori/go.uuid is the cause.

Create manual for NGSIv2 support

Update FogFlow manual for NGSIv2 support
New interfaces implemented for Fiware GE's:

Quantum leap
Wirecloud
Orion

update the scripts

please check all dockerfiles and also docker-compose.yml. After adding the new features, they are not updated accordingly.

Error to download operator docker image.

I'm trying to test the multi-node FogFlow environment for "USE CASE ON ANOMALY DETECTION".
multi-node means Cloud node , Edge and device node are different node.

We have used raspberry pi for Edges and devices.Our environment built a Cloud Node, two Edge Nodes, and three PowerPanel devices. the environment is same as #32

We found a problem that operator container in the Edge fail to launch. The Edge's worker log is as bellows.

$ docker logs edgeworker
INFO: 2019/07/11 01:29:34 annouce myself to the nearby broker
2019/07/11 01:29:35 [*] Waiting for messages. To exit press CTRL+C
INFO: 2019/07/11 01:38:01 execute task  AnomalyDetector.c7cf3f9c-86ef-44d7-baae-d7539915e466  with operation
INFO: 2019/07/11 01:38:01 task configuration {ID:AnomalyDetector.c7cf3f9c-86ef-44d7-baae-d7539915e466 ServiceName:anomaly-detection TaskType: TaskName:AnomalyDetector FunctionCode: DockerImage: WorkerID:Worker.2.3 IsExclusive:false PriorityLevel:50 Status: Inputs:[{Type:PowerPanel Streams:[Stream.PowerPanel.03] URLs:map[]} {Type:Rule Streams:[] URLs:map[]}] Outputs:[{Type:Anomaly StreamID:Stream.Anomaly.03 Annotations:[]}]}
INFO: 2019/07/11 01:38:01 task runnable =  true
INFO: 2019/07/11 01:38:01 to execute Task  AnomalyDetector.c7cf3f9c-86ef-44d7-baae-d7539915e466  to perform Operation
INFO: 2019/07/11 01:38:01 operator image  does not exist locallyoptions : {Username: Password: Email: ServerAddress: IdentityToken: RegistryToken:}options : {Repository: Tag:latest Registry: OutputStream:<nil> RawJSONStream:falseInactivityTimeout:0s Context:<nil>}
ERROR: 2019/07/11 01:38:01 executor.go:137: failed to pull this image , error no such image
ERROR: 2019/07/11 01:38:01 executor.go:282: failed to fetch the image

I think Edge's worker can't know operator docker image name.

If you have seen this error, Please teach me How to solve or Could you let me how to debug .

Entity data cannot be separated for security.

Hi,We try to run Fogflow using "UseCase on Anomaly Detection".
When I get entity data from Fog Flow Broker, all entity data are Accessible, I think It contains security risks.
What is the way for separating entity data like Orion by Fiware-Service and Fiware-ServicePath?

In persistence storage Designer delays in data creation

Report not generating

testing report is not generating while testing ngsiv2

NGSI-LD Support addition in FogFlow

FogFlow should understand the NGSI-LD format and response in NGSI-LD format.

firewall issue

Currently, the FogFlow edge node requires to have a public IP address to be accessible by the FogFlow cloud node. In the actual deployment environment, the FogFlow edge node is very often deployed behind the company firewall via NAT, we need to find a way to support this scenario. One way to address this is to find a proxy for such kind of edge nodes. For example, assign the FogFlow cloud broker to be the proxy for the FogFlow brokers at this type of edge nodes.

NGSI10/contextEntities API has disappeared.

Is it right to use NGSI10/entity API instead?

python-client to interact with FogFlow

Currently, FogFlow provides a nice graphical user interface for users to create operator, fog function, service topology. It is nice for beginners to use them, but there is also a need to provide a client to interact with FogFlow programmatically, especially a python-based client library to use FogFlow in a python development environment.

Data isn't sometimes obtained from Operator

In anomaly detection demo, data isn't sometimes obtained from 'Stream.Anomaly.all.*' in Operator. It was possible to get strange Operator ID 'Stream.Anomaly.02.all' from log of several FogFlow components. Data was obtained right in case of 'Stream.Anomaly.all.02'. Could you check the log?

Master

INFO: 2018/02/21 06:42:42 condition to select streams : map[type:Rule]
INFO: 2018/02/21 06:42:42 returned streams : [0xc4200f67e0]
INFO: 2018/02/21 06:42:42 ==returned stream ID : Stream.Rule.01
INFO: 2018/02/21 06:42:42 ==returned stream URL : 
INFO: 2018/02/21 06:42:42 add new type of input streams {Type:Rule Streams:[Stream.Rule.01] URLs:map[]}
INFO: 2018/02/21 06:42:42 add new type of input streams {Type:Anomaly Streams:[Stream.Anomaly.02.all] URLs:map[]}

Discovery

INFO: 2018/02/21 06:42:43 registration:{ID:Task.AnomalyDetector.dc0bd1a9-ab40-4aaa-95ee-ff25db39b166 Type:Task IsPattern:false}
INFO: 2018/02/21 06:42:43 registration:{ID:Stream.Anomaly.02.all Type:Anomaly IsPattern:false}
DEBUG: 2018/02/21 06:42:43 repository.go:319: SELECT entity_tab.eid, entity_tab.type, entity_tab.ispattern, entity_tab.providerurl FROM entity_tab  WHERE entity_tab.eid = 'Stream.Anomaly.02.all' 
INFO: 2018/02/21 06:42:43 registration:{ID:Task.Counter.551e50e8-86f4-4205-9718-7cf506f0efab Type:Task IsPattern:false}
INFO: 2018/02/21 06:42:48 registration:{ID:Stream.Stat.all Type:Stat IsPattern:false}
DEBUG: 2018/02/21 06:42:56 repository.go:319: SELECT entity_tab.eid, entity_tab.type, entity_tab.ispattern, entity_tab.providerurl FROM entity_tab  WHERE entity_tab.eid = 'Stream.Anomaly.all.01' 
DEBUG: 2018/02/21 06:42:59 repository.go:319: SELECT entity_tab.eid, entity_tab.type, entity_tab.ispattern, entity_tab.providerurl FROM entity_tab  WHERE entity_tab.eid = 'Stream.Anomaly.all.02'

Broker

INFO: 2018/02/21 06:42:43 NEW subscription: {[{Stream.Anomaly.02.all Anomaly false}] [] http://192.168.0.108:46583  { []} []  {false false true http://192.168.0.108:8070/ngsi10 []}}
{"entities":[{"id":"Stream.Anomaly.02.all","type":"Anomaly"}],"reference":"http://192.168.0.108:8070/ngsi10/notifyContextAvailability","restriction":{"attributeExpression":""}}
{"subscribeId":"1beee86f-763c-4bbe-8afb-26af2f385d81","errorCode":{"code":200,"reasonPhrase":"OK"}}
INFO: 2018/02/21 06:42:43 CREATE  subID  ae8bd32f-550f-488d-9a81-98be1d5a74eb
INFO: 2018/02/21 06:42:43 ===>  {Stream.Anomaly.02.all Anomaly false}  ,  ae8bd32f-550f-488d-9a81-98be1d5a74eb
INFO: 2018/02/21 06:42:43 http://192.168.0.108:8070/ngsi10 ,  http://192.168.0.108:8070/ngsi10
INFO: 2018/02/21 06:42:43 TO ngsi10 subscription,  ae8bd32f-550f-488d-9a81-98be1d5a74eb
INFO: 2018/02/21 06:42:43 entity list: [{ID:Stream.Anomaly.02.all Type:Anomaly IsPattern:false}]
send discovery : 
http://192.168.0.108:8071/ngsi9
{Entities:[{ID:Stream.Anomaly.all.01 Type: IsPattern:false}] Attributes:[] Restriction:{AttributeExpression: Scopes:[]}}

Worker

INFO: 2018/02/21 06:42:42 execute task  Counter.551e50e8-86f4-4205-9718-7cf506f0efab  with operation counter
INFO: 2018/02/21 06:42:42 task configuration {ID:Counter.551e50e8-86f4-4205-9718-7cf506f0efab ServiceName:anomaly-detection TaskType: TaskName:Counter FunctionCode: DockerImage:counter WorkerID:SysComponent.Worker.22 IsExclusive:false PriorityLevel:100 Status: Inputs:[{Type:Anomaly Streams:[Stream.Anomaly.02.all] URLs:map[]}] Outputs:[{Type:Stat StreamID:Stream.Stat.all Annotations:[]}]}
INFO: 2018/02/21 06:42:42 task runnable =  true
INFO: 2018/02/21 06:42:42 to execute Task  Counter.551e50e8-86f4-4205-9718-7cf506f0efab  to perform Operation  counter
INFO: 2018/02/21 06:42:42 execute task  AnomalyDetector.dc0bd1a9-ab40-4aaa-95ee-ff25db39b166  with operation anomaly
INFO: 2018/02/21 06:42:42 task configuration {ID:AnomalyDetector.dc0bd1a9-ab40-4aaa-95ee-ff25db39b166 ServiceName:anomaly-detection TaskType: TaskName:AnomalyDetector FunctionCode: DockerImage:anomaly WorkerID:SysComponent.Worker.22 IsExclusive:false PriorityLevel:100 Status: Inputs:[{Type:PowerPanel Streams:[Stream.PowerPanel.02] URLs:map[]} {Type:Rule Streams:[Stream.Rule.01] URLs:map[]}] Outputs:[{Type:Anomaly StreamID:Stream.Anomaly.02.all Annotations:[]}]}
INFO: 2018/02/21 06:42:42 task runnable =  true
INFO: 2018/02/21 06:42:42 to execute Task  AnomalyDetector.dc0bd1a9-ab40-4aaa-95ee-ff25db39b166  to perform Operation  anomaly
INFO: 2018/02/21 06:42:42  task AnomalyDetector.dc0bd1a9-ab40-4aaa-95ee-ff25db39b166  started within container = 6f2b7bbd65b0c4f35469b6ba732d6cbf95ffde604638c73f2ae2fe0b11f10ff4
INFO: 2018/02/21 06:42:42 configure the task with [map[command:CONNECT_BROKER brokerURL:http://192.168.0.108:8070/ngsi10] map[command:SET_REFERENCE url:http://192.168.0.108:44384] map[command:SET_OUTPUTS type:Anomaly id:Stream.Anomaly.02.all]], via port 44384
INFO: 2018/02/21 06:42:42 http://192.168.0.108:44384/admin
INFO: 2018/02/21 06:42:42  task Counter.551e50e8-86f4-4205-9718-7cf506f0efab  started within container = 1221be96b67993f2b6e85ad789027f01e55d255c2520b50e5c05ace3dcb61e99
INFO: 2018/02/21 06:42:42 configure the task with [map[command:CONNECT_BROKER brokerURL:http://192.168.0.108:8070/ngsi10] map[command:SET_REFERENCE url:http://192.168.0.108:46583] map[command:SET_OUTPUTS type:Stat id:Stream.Stat.all]], via port 46583
INFO: 2018/02/21 06:42:42 http://192.168.0.108:46583/admin
response Status: 200 OK

Subscrption-Id is not present in notification of NGSI-V2 Feature

subscription-id should be present in notification of NGSI-V2, same is present in NGSI-V1 notification.
It can be utilized for automation test cases.

[MUST] Show versioned documentation

On the Fogflow ReadtheDocs project, no versioned documentation is displayed

Goto https://readthedocs.org/projects/fogflow/versions/
Ensure that active checkmark is ticked against relevant release tags

[SHOULD] run tests on CI - test suite missing

A test suite is currently missing from the public codebase for this GE, and the tests should be run on CI. The standard GO testing framework is suceeding because no tests are present.

$ go test -v ./...
?   	github.com/smartfog/fogflow/master	[no test files]
The command "go test -v ./..." exited with 0.

Running tests on CI is a TSC SHOULD requirement

Can the Tasks that make up a service topology run on different architectures?

Can the Tasks that make up a service topology run on different architectures?

We have a fog infrastructure formed by three Works (one Raspberry Pi 4, one notebook, and a virtual machine). We created a service topology with two Tasks. We registered two Operators, only one of which is for ARM architecture. However when we fire an Intent the FogFlow orchestrator tries to run both Tasks on the Worker located in Raspberry. Obviously the docker image for X86 architecture does not run properly in Raspberry.

small changes to the FogFlow tutorial

Some FogFlow users from The Polytechnic University of Milan shared with me their feedback in terms of FogFlow tutorial. What they like to achieve is to develop a new FogFlow application for their own use cases, but they are confused and overwhelmed by the tutorial, because they do not have a clear idea how to design and implement their new FogFlow application.

To address this type of confusion, one way is to provide a separated "Developer Guide" section in the FogFlow tutorial to explain how a new FogFlow application can be designed and implemented with a concrete example. We can move "intent-based programming" from the current "user guide" into a new "developer guide". The existing "user guide" will be more like a "system operator guide".

let's discuss how we improve the FogFlow tutorial for the developers perspective.

Manual and Dashboards for System Monitoring

Integration has been completed and basic functionality has been achieved but user friendly dashboard should be created and how user can monitor the solution, dashboards should be created.

Create manual for system Monitoring and Create dashboard for following functionality:

Memory and CPU monitoring
Message received and sent
Health of nodes edges and clouds
also explore if we can configure Alert in case of edge or cloud get unhealthy

Sending Requirement Object

Hi,

I think it might be a good idea to put a sample Requirement object in the docs. I have set up the Fogflow on one server and another worker (broker, worker) on another VM. The setup is fine and two brokers are visible in the System Management of the designer.

I try to send a requirement to the broker to start the flow. This is the requirement object I send:

POST http://MY_BROKER_IN_THE_CLOUD:8080/ngsi10/updateContext
BODY:
{
	"contextElements":[
           {
                "entityId": {
                    "id": "Requirement.LTU",
                    "type": "Requirement",
                    "isPattern": false
                }, 
                "attributes":[{
                	"output":   {"type":"string", "value":"Stat"},
                	"scheduler":{"type":"string", "value":"closest_first"}
                }],
                "metadata":{
                	"topology":{"type":"string", "value":"anomaly-detection"}
                }

            }
		],
	"updateAction": "UPDATE"
}

however, on the logs I see that the attributes are not transmitted properly. see the log:

broker_1        | DEBUG: 2019/03/01 08:20:58 utils.go:41: {"subscriptionId":"31750544-afbc-47cd-b6d7-4d180aff7924","originator":"","contextResponses":[{"contextElement":{"entityId":{"id":"Requirement.LTU","type":"Requirement"},"attributes":[{"name":"","contextValue":null}]},"statusCode":{"code":200,"reasonPhrase":"OK"}}]}

Seems some problem with getting attributes. That leads to a panic in the master as well:

master_1        | DEBUG: 2019/03/01 08:20:58 master.go:212: NGSI10 NOTIFY  31750544-afbc-47cd-b6d7-4d180aff7924  ,  Requirement
master_1        | INFO: 2019/03/01 08:20:58 =================handle requirement update=================
master_1        | INFO: 2019/03/01 08:20:58 read the requirement entity
master_1        | interface conversion: interface {} is nil, not string
master_1        | goroutine 982 [running]:
master_1        | runtime/debug.Stack(0xc000239800, 0x777600, 0xc000068f30)
master_1        | 	/usr/local/Cellar/go/1.11.4/libexec/src/runtime/debug/stack.go:24 +0xa7
master_1        | github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1.1(0xab0eb0, 0x842800, 0xc000068e70)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/recover.go:41 +0x6e
master_1        | panic(0x777600, 0xc000068f30)
master_1        | 	/usr/local/Cellar/go/1.11.4/libexec/src/runtime/panic.go:513 +0x1b9
master_1        | main.(*TopologyMgr).handleRequirementUpdate(0xc00001e870, 0xc000186000, 0x1, 0x4, 0xc000092180, 0x24)
master_1        | 	/Users/cheng/work/go/src/github.com/smartfog/fogflow/master/topologyMgr.go:95 +0x7f7
master_1        | main.(*Master).onReceiveContextNotify(0xc000073ba0, 0xc00009c140)
master_1        | 	/Users/cheng/work/go/src/github.com/smartfog/fogflow/master/master.go:222 +0x34e
master_1        | main.(*Master).onReceiveContextNotify-fm(0xc00009c140)
master_1        | 	/Users/cheng/work/go/src/github.com/smartfog/fogflow/master/master.go:97 +0x34
master_1        | github.com/smartfog/fogflow/common/ngsi.(*NGSIAgent).handleNotifyContext(0xc00000a980, 0x842800, 0xc000068e70, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/smartfog/fogflow/common/ngsi/ngsiagent.go:51 +0x171
master_1        | github.com/smartfog/fogflow/common/ngsi.(*NGSIAgent).handleNotifyContext-fm(0x842800, 0xc000068e70, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/smartfog/fogflow/common/ngsi/ngsiagent.go:24 +0x48
master_1        | github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x842800, 0xc000068e70, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/router.go:56 +0x9f
master_1        | github.com/ant0ine/go-json-rest/rest.(*RecoverMiddleware).MiddlewareFunc.func1(0x842800, 0xc000068e70, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/recover.go:57 +0x8a
master_1        | github.com/ant0ine/go-json-rest/rest.(*PoweredByMiddleware).MiddlewareFunc.func1(0x842800, 0xc000068e70, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/powered_by.go:26 +0xae
master_1        | github.com/ant0ine/go-json-rest/rest.(*RecorderMiddleware).MiddlewareFunc.func1(0x842840, 0xc00000a0c0, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/recorder.go:22 +0x9a
master_1        | github.com/ant0ine/go-json-rest/rest.(*TimerMiddleware).MiddlewareFunc.func1(0x842840, 0xc00000a0c0, 0xc00000a0a0)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/timer.go:20 +0xfc
master_1        | github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x842300, 0xc0000fe380, 0xc00013c400)
master_1        | 	/Users/cheng/work/go/src/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0xd7
master_1        | net/http.HandlerFunc.ServeHTTP(0xc00001c960, 0x842300, 0xc0000fe380, 0xc00013c400)
master_1        | 	/usr/local/Cellar/go/1.11.4/libexec/src/net/http/server.go:1964 +0x44
master_1        | net/http.serverHandler.ServeHTTP(0xc00016f860, 0x842300, 0xc0000fe380, 0xc00013c400)
master_1        | 	/usr/local/Cellar/go/1.11.4/libexec/src/net/http/server.go:2741 +0xab
master_1        | net/http.(*conn).serve(0xc000084b40, 0x842580, 0xc0000560c0)
master_1        | 	/usr/local/Cellar/go/1.11.4/libexec/src/net/http/server.go:1847 +0x646
master_1        | created by net/http.(*Server).Serve
master_1        | 	/usr/local/Cellar/go/1.11.4/libexec/src/net/http/server.go:2851 +0x2f5

I think it is a good idea that master validate the objects before using them.
My speculation is that there is some problem with my Requirement. What do you think?

it will be nice to move FogFlow from NGSI v1 to NGSI v2

Since most of the other FIWARE GE now only support v2, it will be nice for FogFlow to support v2 as well. In this case, it will be easy to integrate FogFlow with IoTAgent for device integration, with WireCloud for visualization, and with QuantumLeap for saving historical data. With all these together, we can form a minimal edge computing oriented IoT platform. How do you thin? I think this does not require too much change and we can easily support it.

@fogflow users, how do you think? Please let me know your opinion. Do you also expect to see this feature in FogFlow?

Cannot find documentation on how to configure "config.json" for setting Edge Node.

Hi, I am building a multi-node FogFlow using Raspberry Pi.
It is necessary to set "config.json" in the setting of Edge Node, but the description was not found in the document for most setting items. (e.g. site_id, start_actual_task, capacity and so on)
Is there a detailed documentation about config.son? If not, can I create a document?
I think it is important to maintain documentation so that beginners can use FogFlow.

Cannot download Docker Image of the latest code from Docker Hub

Hi,
I tried building with this document(https://fogflow.readthedocs.io/en/latest/edge.html#start-both-edge-iot-broker-and-fogflow-worker), but the latest image downloaded from Docker Hub doesn't seem to be the latest code on GitHub.
Is there a document with instructions for building the latest image on my local machine?
Is it possible to update the Docker Hub image in conjunction with GitHub Commit?

How to fix a Task to a given Worker?

How to fix the execution of a Task in a given Worker?
For example, if an accelerator hardware attached to Worker.03 and a particular Task must be executed on it.

persistent storage of all defined operators, service topologies, and fog functions

right now all initialized operators, service topologies, and fog functions are saved as NGSI entities, but they are not saved into a persistent storage file system or database. There could be three solutions to this problem:

save them into some database, such as Postgresql
save them into some global key-value storage system, such as consul
save them as NGSI entities but we need to change FogFlow broker to save NGSI entities persistently (by providing some additional parameter to do this)

Personally, I think option 2) could the best choice. let's discuss about this next time.

Data cannot be restored if FogFlow Broker down.

Hi,We try to run Fogflow using "UseCase on Anomaly Detection".
I think that Fog Flow Broker manages system and user information in-memory. What is the best way to recover if a process goes down and data is lost?

reconstruct the current FogFlow tutorial

since we have achieved lots of new features in FogFlow, it is necessary to take some time to re-organize the tutorial and make it easier to follow.

Any idea or suggestion?

Wreckage of a container of stopped Operator is left.

Worker doesn't seem to remove the container after stopping it.
Is it right not to remove a container?

error to communicate with rabbitmq Channel: Exception (504) Reason: "channel/connection is not open"

I trying to test the multi-node FogFlow environment. I built a Cloud Node, two Edge Nodes, and three PowerPanel devices.
But worker container in the Edge Node1 stopped in about a minute.

$ docker ps -a
CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS                     PORTS                    NAMES                                                                                                  
8d784c698380        fogflow/worker:arm   "/worker"           5 minutes ago       Exited (2) 4 minutes ago                            edgeworker                                                                                             
e98b3fdf2fac        fogflow/broker:arm   "/broker"           5 minutes ago       Up 5 minutes               0.0.0.0:8080->8080/tcp   edgebroker

And it output the following error.

$ docker logs edgeworker
INFO: 2019/07/11 04:31:56 annouce myself to the nearby broker                                                         

2019/07/11 04:31:58 [*] Waiting for messages. To exit press CTRL+C                                                    
Going to retry launching the edge node. Error: Received an empty message.error to communicate with rabbitmq Channel: Exception (504) Reason: "channel/connection is not open"                                                               
                                                                                                                      
panic: runtime error: invalid memory address or nil pointer dereference                                               
[signal SIGSEGV: segmentation violation code=0x1 addr=0x34 pc=0x2f8dc0]                                               
                                                                                                                      
goroutine 35 [running]:                                                                                               
github.com/streadway/amqp.(*Channel).Close(0x0, 0x0, 0x0)                                                             
        /Users/cheng/work/go/src/github.com/streadway/amqp/channel.go:426 +0x24                                       
github.com/smartfog/fogflow/common/communicator.(*Communicator).Publish(0xfcc280, 0x103e000, 0x445e38, 0xc5e078)      
        /Users/cheng/work/go/src/github.com/smartfog/fogflow/common/communicator/communicator.go:134 +0x2a4           
main.(*Worker).heartbeat(0xc9a580)                                                                                    
        /Users/cheng/work/go/src/github.com/smartfog/fogflow/worker/worker.go:210 +0xb8                               
main.(*Worker).onTimer(0xc9a580)                                                                                      
        /Users/cheng/work/go/src/github.com/smartfog/fogflow/worker/worker.go:205 +0x1c                               
main.(*Worker).Start.func2(0xc9a580)                                                                                  
        /Users/cheng/work/go/src/github.com/smartfog/fogflow/worker/worker.go:111 +0x38                               
created by main.(*Worker).Start                                                                                       
        /Users/cheng/work/go/src/github.com/smartfog/fogflow/worker/worker.go:108 +0x5cc

The other worker is working fine.
This error seems to be a connection error with RabbitMQ, RabbitMQ is working fine.

The configuration of Edge Node1 is below.

{                                       
    "coreservice_ip": "192.168.28.10",  
    "external_hostip": "192.168.28.230",
    "internal_hostip": "172.17.0.1",    
    "physical_location":{               
        "longitude": 127.6784711,       
        "latitude": 26.2157166,         
        "section": "01",                
        "district": "02",               
        "city": "Naha"                  
    },                                  
    "logical_location": {               
        "node_no": 1,                   
        "site_no": 2,                   
        "layer_no": 2,                  
        "parent_site_no": 1             
    },                                  
    "logging":{                         
        "info":"stdout",                
        "error":"stderr",               
            "protocol": "stdout",       
            "debug": "stdout"           
    },                                  
    "discovery": {                      
        "port": 443                     
    },                                  
    "broker": {                         
        "port": 8080,                   
        "websocket": 8090               
    },                                  
    "worker": {                         
        "container_autoremove": false   
    },                                  
    "rabbitmq": {                       
        "port": 5672,                   
        "username": "admin",
        "password":"mypass"
    }
}

The configuration of Edge Node2 is below.

{
    "coreservice_ip": "192.168.28.10",
    "external_hostip": "192.168.28.231",
    "internal_hostip": "172.17.0.1",
    "physical_location":{
        "longitude": 127.6903532,
        "latitude": 26.2422305,
        "section": "02",
        "district": "03",
        "city": "Urasoe"
    },
    "logical_location": {
        "node_no": 2,
        "site_no": 3,
        "layer_no": 2,
        "parent_site_no": 1
    },
    "logging":{
        "info":"stdout",
        "error":"stderr",
            "protocol": "stdout",
            "debug": "stdout"
    },
    "discovery": {
        "port": 443
    },
    "broker": {
        "port": 8080,
        "websocket": 8090
    },
    "worker": {
        "container_autoremove": false
    },
    "rabbitmq": {
        "port": 5672,
        "username": "admin",
        "password":"mypass"
    }
}

If this is a known issue, Is there a solution?

the scripts to run a FogFlow edge node on both normal linux node and ARM-based node

The provided scripts to start a FogFlow edge node must be further checked and corrected: 1) the IoT-agent is not launched right now; 2) the cadvisor is not used

system monitoring

it will be quite useful to deploy Grafana and Prometheus to monitor the resource and tasks in the entire system.

provide a learning-based algorithm to make orchestration decisions to dynamically configure data processing pipelines

In the current intent-based programming model, the algorithm to ensure the requested QoS is still missing. The idea is to separate this part of decision-making out from the core logic of Topology Master and then to be able to plug in a learning-based algorithm (which is called dynamic orchestrator) to do this type of decisions.

Possible SQL injection

Hi!

Looking at the Discovery code (mainly here: https://github.com/smartfog/fogflow/blob/master/discovery/repository.go) I can see that Sprintf's are used to create the commands that are then executed against the DB.
This open's up the possibility for SQL injections. I haven't tried to attack the Discovery component but I confident that with a carefully chosen entityID (for example) one can do significant damage and/or get some data to be leaked.
This happens because the inputs are not sanitized. I will fork the repo and I'll create a pull request with a possible solution.

problem with the function postNotifyContext

the latest version can deal with v2 for subscribe-update-notify, but it introduced some problem with v1. please fix the problem with the flag (IsOrionBroker bool). This is an urgent problem to be fixed soon.

[SHOULD] run lint on CI

.travis is missing a before_script check to ensure the files are linted - i.e. go lint

Adding a lint check is a SHOULD requirement from the TSC

[MUST] Docker Images not clearly linked and available

There is a mismatch between the docker images produced in the repository and the listings elsewhere

According to DockerHub, Fogflow produces 16 images.
According to the README only 1 image fogflow/discovery is badged for pulls.
According to the FIWARE Catalogue no docker images are listed

Looking in the docker/README, 5 images are listed.

It is unclear which of these is correct.

If you look at Cygnus, you will see that all Docker Images are mentioned and listed in the README

prepare the release with the southbound integration

Since the southbound integration is ready, it will be helpful to make one release. Please consider to name the release version properly.

Correct roadmap link in readme doc

It seems that fogflow roadmap link is not correct https://github.com/smartfog/fogflow/blob/master/roadmap.md in https://github.com/smartfog/fogflow. Fix it

URL of API of IoT Discovery is 404.

https://app.swaggerhub.com/apis/fogflow/discovery/1.0.0

Is this the one which isn't exhibited?

All automated test cases of FogFlow need to be automatically executed before new feature addition.

. it will help to validate the previous delivered features are not corrupted due to addition of new features. it will enhance the continuous integration of FogFlow. .travis.yml should execute all the automated test cases of FogFlow as a part of CI.

Issues dynamically adding Edge Nodes

Hi,

We are testing to increase the edge node of FogFlow in the tutorial environment.
First, we built an environment with one cloud node and one edge node, launched the Operators by define Requirements.
After that, when we added a new edge node and defined the Requirement again, Anomaly events were double-counted in Cloud Node. I guess this is because there are unnecessary operators left.
Resources of edge nodes are consumed by unwanted operators. Shouldn't unnecessary operators be removed?
It is necessary to be able to change the node configuration. Is FogFlow currently supporting the dynamic addition of Edge nodes? How can I add edge nodes it possible to fix it?

[Operator placement before adding an edge node]
- Cloud node
  - Count Operator
- Edge Node1
  - Anomaly Operator
[Actual placement of Operator after adding an edge node]
- Cloud node
  - Count Operator
  - Count Operator
- Edge Node1
  - Anomaly Operator
  - Anomaly Operator
- Edge Node2
  - Anomaly Operator
[Expected placement of operators after adding an Edge node]
- Cloud node
  - Count Operator
- Edge Node1
  - Anomaly Operator
- Edge Node2
  - Anomaly Operator

Broker: Notify not effective

Using the object structure from this url to send a notify request.

By using the example to notify, an error is thrown. Reason: under the attributes the value type is string but the contextValue is an empty array. It should be a simple string. I am not sure that the swaggerhub documentation is in compliance with which version, because the swagger test is performed without error.

However, after fixing it, my fog function (the subscriber) would not receive the data. Any idea on how to check the subscriptions while the broker is running?

My fog function is trigger and is ready. However to send data to it, I have to issue ngsi10 notify. I checked subscriptions and my fog function is subscribed to temperature sensor.
My notify request is as follows:

{
  "subscriptionId": "ss",
  "originator": "0.0.0.0",
  "contextResponses": [
    {
      "contextElement": {
        "entityId": {
          "id": "Device.Temperature.temp.*",
          "type": "Temperature",
          "isPattern": false
        },
        "attributes": [
          {
            "name": "temp",
            "type": "integer",
            "contextValue": 44
          }
        ],
        "attributeDomainName": "a city"
      },
      "statusCode": {
        "code": 0,
        "reasonPhrase": "the reason",
        "details": "somedetail"
      }
    }
  ]
}

Contribution Guidelines for FogFlow

Define the community contribution guidelines for Fogflow like other GE's
e.g
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/contribution_guidelines.md
https://fiware-sth-comet.readthedocs.io/en/latest/contribution-guidelines/

[MUST] Sematic Versioned Docker Images Missing

Looking at Docker Hub, there are no semantically versioned tags on the docker images. This is required for stability.

e.g. fogflow/discover:2.0 rather than fogflow/discover:latest

MUST requirement from TSC

Discovery exits because of segmentation violation

Hi!
I am Running the components as mentioned in the tutorial. After submitting a fog function, discovery exits. Here is the log before termination:

DEBUG: 2019/02/19 22:49:10 repository.go:320: SELECT entity_tab.eid, entity_tab.type, entity_tab.ispattern, entity_tab.providerurl FROM entity_tab  WHERE entity_tab.type like 'Topology'
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x72729c]

goroutine 459 [running]:
main.(*FastDiscovery).sendNotify(0xc000048280, 0xc000148570, 0x24, 0xc00005ac80, 0x37, 0xc0003bb440, 0x7fc2d9, 0x6)
	/Users/cheng/work/go/src/github.com/smartfog/fogflow/discovery/fastDiscovery.go:294 +0x51c
main.(*FastDiscovery).handleSubscrieCtxAvailability(0xc000048280, 0xc000176750)
	/Users/cheng/work/go/src/github.com/smartfog/fogflow/discovery/fastDiscovery.go:242 +0xfc
created by main.(*FastDiscovery).SubscribeContextAvailability
	/Users/cheng/work/go/src/github.com/smartfog/fogflow/discovery/fastDiscovery.go:235 +0x21f

Tell me what kind of information is needed to provide so that you reproduce the error.

[SHOULD] Double tag FIWARE Release

Please add an additional tag to 72d9eea to indicate that it was included as part of FIWARE 7.5.1 - you can see a similar examples here:

IoT Agent JSON

Also double tag 1760f1c to indicate that it was included as part of FIWARE 7.5 and FIWARE 7.4

Change the current design of FogFlow designer

As per current design of the FogFlow designer, a user can interact with the web browser and whenever any ngsi entity (operator, topology, fog function) created on web browser, web browser sends the created entity to the broker.
So this design will be changed, In new design any entity that is created on web browser will send to designer first and then designer will send it to broker.

prepare the step-by-step FogFlow tutorial

we need to prepare a step-by-step tutorial for fogflow based on the the examples as below.

https://github.com/FIWARE/tutorials.Step-by-Step

Add FogFlow to tour guide

As FogFlow moves to Full GE membership, it needs a section in the FIWARE Tour Guide - see FIWARE-Ops/docs.TourGuide#81

A tour guide entry should be a brief getting started, typically a single page describing usage based on the assumption that the product is up and running. There should be no docker-compose set-up and it can be assumed that the product is already connected with a context broker instance.

Good examples with an appropriate level of detail which follow the template can be found here:

It doesn't need to be long - it is effectively a teaser - a simple demonstration of the sort of thing FogFlow can do so that readers can move on to the full documentation if interested.

Please fork https://github.com/Fiware/docs.TourGuide and raise a PR adding your files under ./docs/processing/fogflow