Since tensorflow-lite-dev package for debian is built without -fPIC, so below error occurs when making debian package.

• Error Code

[ 21%] Linking CXX shared library libtensor_filter_tflitecore.so
cd /home/again4you/nnstreamer_work/jy_review/nnstreamer/build/gst/tensor_filter && /usr/bin/cmake -E cmake_link_script CMakeFiles/tensor_filter_tflitecore.dir/link.txt --verbose=1
/usr/bin/x86_64-linux-gnu-g++  -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2   -Wall -Werror -fPIC -g -std=c++11  -Wl,-Bsymbolic-functions -Wl,-z,relro -shared -Wl,-soname,libtensor_filter_tflitecore.so -o libtensor_filter_tflitecore.so CMakeFiles/tensor_filter_tflitecore.dir/tensor_filter_tensorflow_lite_core.cc.o -lgstcontroller-1.0 -lgstvideo-1.0 -lgstaudio-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 ../../libcommon.a -ltensorflow-lite -lgstcontroller-1.0 -lgstvideo-1.0 -lgstaudio-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libtensorflow-lite.a(error_reporter.o): relocation R_X86_64_32S against `_ZN6tflite14StderrReporter6ReportEPKcP13__va_list_tag' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libtensorflow-lite.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
gst/tensor_filter/CMakeFiles/tensor_filter_tflitecore.dir/build.make:98: recipe for target 'gst/tensor_filter/libtensor_filter_tflitecore.so' failed
make[3]: *** [gst/tensor_filter/libtensor_filter_tflitecore.so] Error 1

Add common internal functions for tensor_load filter.
It appears that these can be shared with tensor_save later as well.

This prepares #322

Signed-off-by: MyungJoo Ham [email protected]

Self evaluation:

1. Build test: [X]Passed [ ]Failed [*]Skipped
2. Run test: [ ]Passed [ ]Failed [X]Skipped

Issue Description

May need Fast RCNN Model for object detection. The output of this RCNN is other/tensors.

• Input : input image
• Output : detection_boxes, detection_scores, detection_classes, num_detections

However, it is somewhat difficult to find pre-translated tflite model. According to the tflite manual, we could use toco to translate flatbuffer. It is worth to try. ( It is much better if we could find the proper tflite model from internet.)

Support audio/x-raw with converter.

Let's hope that caffe2 users are not going to be mad enough to modify library itself for their own models.

We need example application (as a test case as well) that modifies tensor_* related pipelines in run-time.

We need to ensure that it's possible with our plugins.

1. Please add URLs to get all the files required by the examples (i.e., *.tflite, labels.txt)

• If possible, we may use github.com/nnsuite/testcase/... ( as long as the files are not touched by us, we do not need to worry administrative problems.)

2. /dev as an example installation path is not good. Some people will really use it and possible break the system. Let's use something not harmful such as ~/lib/ or ~/test/

The requirement is to allow a developer with a clean system (a newly installed Ubuntu) to build, install, and run the example.

We "know" that gstreamer supports different modes for synchronization and queue handling.

However, soon, we will need deep understandings in this area.

Some people (at least two! :) ) should be familiar enough with this topic so that he can let us know how to:

1. construct stream pipeline with queues or inter-element connections dropping frames whenever "next" frame comes while the "previous" frame is still being processed.
2. at tensor_mux or tensor_merge, use the most recent tensor frame for each sink-pad, dropping any "obsolete" frame and keeping "recent" frame EVEN if it is already processed and sent away to next element.

It was successful to extract labels from video stream data.
Since the processing term takes almost 0.3sec for each tensor, however, we need to decide the policy about it.
I attached the part of log data.

...

** Message: Invoke() is finished: 312.764000
** Message: Invoke() is finished: 311.526000
** Message: Invoke() is finished: 311.302000
** Message: Invoke() is finished: 311.696000
** Message: Invoke() is finished: 313.468000

...

You can check this log with #328

Additional requirement: be ready for models with heavily modified caffe libraries. (how are we going to let models bring their own custmoized libraries?) This is really a hell-hole, but I'm sure we will be required to support such.

Implement save/load of tensors in the format of other/tensorformat (.tnsr file)

Not for per-PR at this moment, but for CD basis.

1. Required: GUI-based stream design (e.g., pipeviz)
2. Required: tensor_* must be supported (or could be tensor_*'s faults... though...)
3. Required: save/load stream design.
4. Highly recommended: code generation from the GUI-drawn pipelines
5. Highly recommended: supports both Linux and Windows.

We meed to migrate our in-house CI system to here so that we can actively work in github.com instead of the internal repo. Until then, we will use github.com as a mirror.

CC: @sewon-oh @leemgs

For DEB packaging with launchpad.net, we may use https://help.launchpad.net/API/Webhooks
For RPM packaging with build.opensuse.org, we may use https://openbuildservice.org/2013/11/22/source-update-via_token/

For Per-PR checks, we may migrate to AWS/Azure. During migration, we may use (slow but free) babot.net if really needed :)

Additional requirement: be ready for models with heavily modified caffe libraries. (how are we going to let models bring their own custmoized libraries?) This is really a hell-hole, but I'm sure we will be required to support such.

We need to configure gst plugin install paths per SW platform. (ubuntu, tizen, smp).
We need to let users use nnstreamer without configuring gst-plugin-paths.

While CI system is being migrated to github.com, we also need our dev history migrated to here.

1. Required: GUI-based stream design (e.g., pipeviz)
2. Required: tensor_* must be supported (or could be tensor_*'s faults... though...)
3. Required: save/load stream design.
4. Highly recommended: code generation from the GUI-drawn pipelines
5. Highly recommended: supports both Linux and Windows.

We need to prepare a VM instance to run CI service for NNstreamer (http://github.com/nnsuite/nnstreamer).

1. added audio format
2. code refactoring

• add macro for debug message
• use common functions to set tensor and media caps

Needs to check later : test with debug message (set silent=false)

Signed-off-by: Jaeyun Jung [email protected]

This is going to be a show case of "multi-modal" interaction.
Prerequisite: tensor_merge ( #351 )

Note that in addition to the following suggestions, we need additional elements for the visualization.

Suggestion 1.

cam1 -- preprocessors -- tensor_converter --+-- tensor_merge -- tensor_filter -- tensor_sink
cam2 -- preprocessors -- tensor_converter --+

Suggestion 2.

cam1 -- preprocessors -- tensor_converter -- tensor_filter_1 --+-- tensor_merge -- tensor_sink
cam2 -- preprocessors -- tensor_converter -- tensor_filter_2 --+

Suggestion 3.

cam1 -- preprocessors -- tensor_converter ----- tensor_filter_1 --+-- tensor_merge -- tensor_sink_1
cam2 -- preprocessors -- tensor_converter --+-- tensor_filter_2 --+
                                            +-- tensor_filter_3 -- tensor_sink_2

Find somewhere appropriate to store example NN models (e.g., ./tflite_model/mobilenet_v1_1.0_224_quant.tflite in our WIki)

Later, we will need to let CI system to load such NN models to test the code. (at least with "smoke test")

FOr this tflite file, we may need to specify the URL in the wiki and/or in some script?

This is going to be a show case of "multi-modal" interaction.
Prerequisite: tensor_merge ( #351 )

Note that in addition to the following suggestions, we need additional elements for the visualization.

Suggestion 1.

cam1 -- preprocessors -- tensor_converter --+-- tensor_merge -- tensor_filter -- tensor_sink
cam2 -- preprocessors -- tensor_converter --+

Suggestion 2.

cam1 -- preprocessors -- tensor_converter -- tensor_filter_1 --+-- tensor_merge -- tensor_sink
cam2 -- preprocessors -- tensor_converter -- tensor_filter_2 --+

Suggestion 3.

cam1 -- preprocessors -- tensor_converter ----- tensor_filter_1 --+-- tensor_merge -- tensor_sink_1
cam2 -- preprocessors -- tensor_converter --+-- tensor_filter_2 --+
                                            +-- tensor_filter_3 -- tensor_sink_2

Let's hope that caffe2 users are not going to be mad enough to modify library itself for their own models.

For instance, http://suprem.sec.samsung.net/confluence/display/STAR/NN+Streamer.

Requirement:

source generation or in-the-middle filtering framerate throtting.

E.g.,

FILESRC (fast) --> QUEUE (leaky=2. # frame = 1) --> FILTER (slow) --> SINK

What if we do not lose and do not overfill queues at the same time?

Then, FIELSRC needs to feed QUEUE only when it's empty.

How do we do this?

HW: Any ARM board (RPI3, Odroid, ...)
SW: Any OS + TF-Lite with NNRuntime backend + NNStreamer + Example NNStreamer App.

CC: @sj925-lee @seanshpark @chunseok-lee

An ROS node that sends ROS topic with frames from tensor_sink?

Or maybe we can provide tensor_sink_ros or tensor_to_ros plugin.

Reference: http://wiki.ros.org/gscam

What does "rank" mean in other/tensors?

Unlike other/tensor, we have multiple tensors in other/tensors.
How "rank" is defined in other/tensors?

If this means the RANK LIMIT, please document so in the code where other/tensors is defined (tensor_common.h)

ps. The range of num_tensors is [1, 16], not [1, 65535]. Right?

We "know" that gstreamer supports different modes for synchronization and queue handling.

However, soon, we will need deep understandings in this area.

Some people (at least two! :) ) should be familiar enough with this topic so that he can let us know how to:

1. construct stream pipeline with queues or inter-element connections dropping frames whenever "next" frame comes while the "previous" frame is still being processed.
2. at tensor_mux or tensor_merge, use the most recent tensor frame for each sink-pad, dropping any "obsolete" frame and keeping "recent" frame EVEN if it is already processed and sent away to next element.

Refer: https://help.github.com/articles/adding-a-code-of-conduct-to-your-project/

We need to configure gst plugin install paths per SW platform. (ubuntu, tizen, smp).
We need to let users use nnstreamer without configuring gst-plugin-paths.

This is to brag potential performance advantages of NNStreamer.

CASE 1

The two implementations to be compared:

NNStreamer:

[USB-CAM 1920x1080 or higher] --> [Video Scaler (videoconverter) 640x480 or lower] --> [tf_lite model, "A"] --> [tensor_sink] --> [APP print out]

ROS:

[USB-CAM 1920x1080 or higher] --> [Video Scaler (not sure if this needs to be a node) 640x480 or lower] --> [tf_lite model, "A"] --> [APP print out]

[APP Print out] can be replaced by tee + tensor_dec (if tensor_dec is ready for bounding-boxes or text-overlay) + video_mixer + video_sink, which is much better!!!

It is highly recommended to use useful, but lightweight NN model. (e.g., MTCNN or MobileNet).

We may need to find a way to emphasize the performance effect,

CASE 2

With this case, we have "linear NN model", Nx, whose input and output tensor size is huge (e.g., 1920x1080x3) while it has almost no real processing in it (extremely light model. probably output == input). For proper comparison, we may need to add artifical load in it (to simulate an optimized lightweight model)

The proposed execution is:

[USE_CAM] --> [Nx 1] --> [Nx 2] --> [Nx 3] --> [tf_lite mode, "A" (real useful)] --> [tensor_sink]

For demonstration purpose we may add video_mixer and text/box overlay.

Performance Measurements

Execution Scenario A:

• Run as fast as possible in a system. Get the average FPS.
• Note that this could be misleading if USB-CAM overshoots frames and the stream / node configuration is not well implemented.

Execution Scenario B: (This is required as well!!!)

• Run at a slower fixed rate (e.g., 1 per 10 second). Count the number of memory transactions with PMU
• For this we need someone who's familiar with PMU/Perf/... (At least one of @wook16-song @sangjung-woo @geunsik-lim is familiar with this.)

Ease of App Implementation

Write concise test app code. Make it as short as possible but readable.
Write a single-line bash command as well. (as an extreme comparison case)

Other Things to Consider

Performance profiling (execution time, cpu time, memory transactions) of ROS nodes might be troublesome or tricky. You need to count ALL relevant processes. For NNStreamer/Gstreamer, you don't need to worry this as it's a single process program.

We need example application (as a test case as well) that modifies tensor_* related pipelines in run-time.

We need to ensure that it's possible with our plugins.

https://download.tizen.org/live/home:/prerelease:/Tizen:/Unified:/submit:/tizen:/20180729.232051/standard/x86_64/

Support many popular NN models with decoder.

• image recognition (labeling)
• bounding boxes
• bounding boxes + image recognition (labeling)
• segmentation
• ...

It was successful to extract labels from video stream data.
Since the processing term takes almost 0.3sec for each tensor, however, we need to decide the policy about it.
I attached the part of log data.

...

** Message: Invoke() is finished: 312.764000
** Message: Invoke() is finished: 311.526000
** Message: Invoke() is finished: 311.302000
** Message: Invoke() is finished: 311.696000
** Message: Invoke() is finished: 313.468000

...

You can check this log with #328

Issue Description

Tensor Mux & Demux should sync by the frame rate accordingly.

                                ------------------> fixed frame rate
tensor 1 -->    +-----------------+      X     |    3   |    X    |
tensor 2 -->    |  tensor mux     |      1     |    X   |    5    |
tensor 3 -->    +-----------------+      2     |    4   |    6    |
                                         |          |         |
                                        \/         \/        \/
                                       none      tensors   tensors  
                                                 [ 3,1,4 ] [ 3,5,6 ]

The code related with caps negotiation still has obsolete concepts.
The usage of input/outputCapNegotiated should be corrected.

Issue Description

Tensor Mux & Demux should sync by the frame rate accordingly.

                                ------------------> fixed frame rate
tensor 1 -->    +-----------------+      X     |    3   |    X    |
tensor 2 -->    |  tensor mux     |      1     |    X   |    5    |
tensor 3 -->    +-----------------+      2     |    4   |    6    |
                                         |          |         |
                                        \/         \/        \/
                                       none      tensors   tensors  
                                                 [ 3,1,4 ] [ 3,5,6 ]

Implement tensor_merge.

Unlike tensor_mux, this creates a "larger" single tensor from multiple "smaller" tensors.

• mux creates other/tensors from multiple instances of other/tensor
• merge creates other/tensor (no s!!!) from multple instance of other/tensor

mux is useful for creating a single stream-line with different types of tensors. (e.g., video and audio). merge is useful for creating a single tensor from tensors that may placed in a same plane or with same dimensions (e.g., expanding colorspaces by merging RGB video with IR video, adding more channels to same dimensions by merging images of stereo-cameras, stiching RGB videos from mutliple cameras directing different angles.)

Implement save/load of tensors in the format of other/tensorformat (.tnsr file)

Many plugins ignore "filter" parameter of transform_caps callback.
We should not ignore this.

An ROS node that sends ROS topic with frames from tensor_sink?

Or maybe we can provide tensor_sink_ros or tensor_to_ros plugin.

Reference: http://wiki.ros.org/gscam

If we add tensor_converter (other tensor_* filters are not tested) into a pipeviz ( https://github.com/virinext/pipeviz ) gstreamer pipeline, we soon get:

mzx@kohaku:/source/AutoDrv/pipeviz$ ./pipeviz  --gst-plugin-path ../NNStreamer/build/gst/tensor_converter/:../NNStreamer/build/gst/tensor_filter/:../NNStreamer/build/gst/tensor_decoder/
failed to get the current screen resources
QXcbConnection: XCB error: 170 (Unknown), sequence: 163, resource id: 90, major code: 146 (Unknown), minor code: 20
Segmentation fault (core dumped)
mzx@kohaku:/source/AutoDrv/pipeviz$ 

This is a decoder plugin. This cannot inherit transform class; we cannot get srcpadcaps from sinkpadcaps. We need to read the frame contents to identify srcpadcaps.

If we add tensor_converter (other tensor_* filters are not tested) into a pipeviz ( https://github.com/virinext/pipeviz ) gstreamer pipeline, we soon get:

mzx@kohaku:/source/AutoDrv/pipeviz$ ./pipeviz  --gst-plugin-path ../NNStreamer/build/gst/tensor_converter/:../NNStreamer/build/gst/tensor_filter/:../NNStreamer/build/gst/tensor_decoder/
failed to get the current screen resources
QXcbConnection: XCB error: 170 (Unknown), sequence: 163, resource id: 90, major code: 146 (Unknown), minor code: 20
Segmentation fault (core dumped)
mzx@kohaku:/source/AutoDrv/pipeviz$ 

Implement tensor_merge.

Unlike tensor_mux, this creates a "larger" single tensor from multiple "smaller" tensors.

• mux creates other/tensors from multiple instances of other/tensor
• merge creates other/tensor (no s!!!) from multple instance of other/tensor

mux is useful for creating a single stream-line with different types of tensors. (e.g., video and audio). merge is useful for creating a single tensor from tensors that may placed in a same plane or with same dimensions (e.g., expanding colorspaces by merging RGB video with IR video, adding more channels to same dimensions by merging images of stereo-cameras, stiching RGB videos from mutliple cameras directing different angles.)

Find somewhere appropriate to store example NN models (e.g., ./tflite_model/mobilenet_v1_1.0_224_quant.tflite in our WIki)

Later, we will need to let CI system to load such NN models to test the code. (at least with "smoke test")

FOr this tflite file, we may need to specify the URL in the wiki and/or in some script?

Support audio/x-raw with converter.