zkmkarlsruhe / ofxtensorflow2 Goto Github PK

I posted it already in the OF forum. Here is a Yolo v4 example: https://forum.openframeworks.cc/t/yolo-v4-example-with-ofxtensorflow2/40040

The ofxTF2::ThreadedModel inherits the load() function from ofxTF2::Model which is a simple, blocking load. On slower systems, this can be relatively expensive as it will generally happen on the main thread.

We could look into adding asynchronous model loading on the background thread, possibly by an additional loadAsync() function.

Add support for optimized libtensorflow builds for:

• macOS arm64 (M1)
• iOS (TensorFlow Light) possibly leave for later

References:

• Accelerating TensorFlow Performance on Mac
• Apple macOS Tensorflow Github repo
• TensorFlow iOS quickstart

I just came across this post, 3D Pose Detection with MediaPipe BlazePose GHUM and TensorFlow.js.
Do you have plans to make an example on how to use this library. I think it's TF.js pose-detection API ?

As always thank you for your efforts to make tensor flow accessible to the OF community.

Hello!!

Im playing with this to help me to understand how tensorflow c pi works.

I have a centernet model that has one input and 4 outputs

The given SavedModel SignatureDef contains the following input(s):
  inputs['input_tensor'] tensor_info:
      dtype: DT_UINT8
      shape: (1, -1, -1, 3)
      name: serving_default_input_tensor:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['detection_boxes'] tensor_info:
      dtype: DT_FLOAT
      shape: (1, 100, 4)
      name: StatefulPartitionedCall:0
  outputs['detection_classes'] tensor_info:
      dtype: DT_FLOAT
      shape: (1, 100)
      name: StatefulPartitionedCall:1
  outputs['detection_scores'] tensor_info:
      dtype: DT_FLOAT
      shape: (1, 100)
      name: StatefulPartitionedCall:2
  outputs['num_detections'] tensor_info:
      dtype: DT_FLOAT
      shape: (1)
      name: StatefulPartitionedCall:3
Method name is: tensorflow/serving/predict

I load the model and set input and output names

    model.load("model");


    vector<string> inputs{"serving_default_input_tensor:0"};
    vector<string> outputs{"StatefulPartitionedCall:0",
                            "StatefulPartitionedCall:1",
                            "StatefulPartitionedCall:2",
                            "StatefulPartitionedCall:3"};

    model.setup(inputs,outputs);

Add one dimension to input to follow model requeriments

    input = cppflow::expand_dims(input, 0);
    auto output = model.runModel(input);

And now how i acces to the output data? I tried this but dont work, i expected that the return of runModel was a vector of vectors.... i tried reading the cppflow doc but no luck

    auto detection_boxes = output[0];
    auto detection_classes = output[1];
    auto detection_scores = output[2];
    auto num_detections = output[3];

Thanks!

I trained a new model for neural style using the python files in your repo, that create a model is pb format instead of checkpoints as original repo.

But when i try to load the model in the of code i get a black screen and a msg on console
[warning] ofxTensorFlow2: shape mismatch at dimension 0 for lhs [480, 640, 3] and rhs [640, 480, 3]

I opened the model with Netron and i see a input called "serving_default_input_1" with size float32[?,640,480,3] so i don't understand whats happening

Can you give me some clue?

I'm trying to compile in MacOsX example_basic, but I'm having this error:
No member named 'exchange' in namespace 'std', in context.h line 78.

inline context::context(context&& ctx) noexcept : tfe_context(std::exchange(ctx.tfe_context, nullptr)) { }
inline context& context::operator=(context&& ctx) noexcept { tfe_context = std::exchange(ctx.tfe_context, tfe_context); return *this; }

I might be missing a step of the installation?

I had this addon working successfully with OF 11.2 on a previous macOS system.
now I am on a M1 with macOS 12.6.8 and Xcode 14.1.0 and get these 2 errors:

CodeSign /Applications/of_v0.12.0_osx_release/addons/ofxTensorFlow2/example_video_matting/bin/example_video_mattingDebug.app/Contents/MacOS/libtensorflow_framework.dylib (in target 'example_video_matting' from project 'example_video_matting')
    cd /Applications/of_v0.12.0_osx_release/addons/ofxTensorFlow2/example_video_matting
    
    Signing Identity:     "-"
    
    /usr/bin/codesign --force --sign - --deep --timestamp\=none --preserve-metadata\=identifier,entitlements,flags --generate-entitlement-der /Applications/of_v0.12.0_osx_release/addons/ofxTensorFlow2/example_video_matting/bin/example_video_mattingDebug.app/Contents/MacOS/libtensorflow_framework.dylib

/Applications/of_v0.12.0_osx_release/addons/ofxTensorFlow2/example_video_matting/bin/example_video_mattingDebug.app/Contents/MacOS/libtensorflow_framework.dylib: main executable failed strict validation
Command CodeSign failed with a nonzero exit code

The .dylib is in the right place

and I think it has the right permissions

and then the main issue I guess is the Signing Identity: "-" thing?

the same issue is mentioned but for a different project.
openframeworks/openFrameworks#6705

I am a bit lost.

other default OF examples compile and run just fine.

As noted by @golanlevin, the configure_xcode.sh configure script appears to break Xcode projects. Whoops.

It comes down to how the build script is stored as an escaped string internally in the .pbxproj file... it does not match how the script contents are displayed in the Xcode editor, so bad assumption on my part.

Hi friends,
I'm having M1-architecture-related linking problems ("Undefined Symbol" etc.) similar to those described in Issues #15 and #3. I have tried the posted fixes without luck, and also (possibly) discovered a problem with one of the scripts. My configuration is:

• Hardware: MacBook Pro (14-inch, 2021) with Apple M1 Pro.
• Software: OSX 12.6, XCode 14.0.1 (14A400).
• Building within a fresh copy of of_v0.11.2_osx_release.

I have followed the posted instructions as follows:

• I did the downloads: I cloned https://github.com/zkmkarlsruhe/ofxTensorFlow2.git to OFROOT/addons; I pulled & checked-out cppflow to ofxTensorFlow2/libs/cppflow (using git submodule update --init --recursive); and I downloaded the models using ./scripts/download_example_models.sh.
• I'm pretty certain that I'm using c++14. I updated CLANG_CXX_LANGUAGE_STANDARD = c++14 suchly in CoreOF.xcconfig. I also updated PLATFORM_CXXFLAGS += -std=c++14 suchly at line 142 of config.osx.default.mk.
• I used download_tensorflow.sh to install libtensorflow.2.8.0 in ofxTensorFlow2/libs/tensorflow/. As expected, these x86_64 libraries didn't work; so, I then replaced the contents of the ofxTensorFlow2/libs/tensorflow/ directory with the "unofficial" arm64 materials from https://github.com/vodianyk/libtensorflow-cpu-darwin-arm64.
• I created the project example_video_matting.xcodeproj using the Project generator
• In the Build Phases for this project's target, in the 2nd Run Script, I added the following before the final echo line: $OF_PATH/addons/ofxTensorFlow2/scripts/macos_install_libs.sh "$TARGET_BUILD_DIR/$PRODUCT_NAME.app";
• As recommended, I set Build Settings->Excluded Architectures to "x86_64" for the Project. (For good measure, I also tried every combination of arm64/x86_64 libraries with excluding each architecture, since there is seemingly contradictory advice about which architectures to exclude here and here)
• I cleaned the build folder before trying to (re)build.

Unfortunately, I'm still getting Undefined Symbols for all of the Tensorflow stuff:

I feel like I'm close, but I'm out of ideas for how to get further. Thanks so much for any help.

Side Note 1: The ofxTensorFlow2 script, configure_xcode.sh, appears to be corrupting XCode 14 projects. After I run this script, .xcodeproj files no longer open, and XCode reports the error: "The project ‘example_video_matting’ is damaged and cannot be opened due to a parse error. Examine the project file for invalid edits or unresolved source control conflicts."

Side Note 2: FWIW, TensorFlow.org is now offering version 2.9.1 for download (though, there is no indication that it works for arm64).

i'm using 060_hair_segmentation model which is download from https://github.com/PINTO0309/PINTO_model_zoo/tree/main/060_hair_segmentation/10_tftrt
got those errors during the loading

SavedModel load for tags { serve }; Status: fail: Not found: Op type not registered 'CreateTRTResourceHandle' in binary running on SLDEV. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.. Took 175474 microseconds.

any ideas? since i'm not familiar with those model conversion stuff,any hints would be appreciated!

I managed to train images in format 256x256 and tried with format 512x512
in config.py

IMG_WIDTH = 512
IMG_HEIGHT = 512

got errors the following errors

Epoch 1 going on....
  0%|                                                                                    | 0/40 [00:00<?, ?it/s]2022-06-26 00:18:30.959669: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
  0%|                                                                                    | 0/40 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "main.py", line 96, in <module>
    checkpoint, checkpoint_prefix
  File "C:\Users\plesk\Documents\coding\of_v0.11.2_vs2017_release\addons\ofxTensorFlow2\example_pix2pix\python\src\training.py", line 143, in train
    fit(train_dataset, test_dataset, EPOCHS)
  File "C:\Users\plesk\Documents\coding\of_v0.11.2_vs2017_release\addons\ofxTensorFlow2\example_pix2pix\python\src\training.py", line 130, in fit
    for input_image, target in tqdm(train_ds):
  File "C:\Users\plesk\.conda\envs\inpaintEnv37\lib\site-packages\tqdm\std.py", line 1167, in __iter__
    for obj in iterable:
  File "C:\Users\plesk\.conda\envs\inpaintEnv37\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 761, in __next__
    return self._next_internal()
  File "C:\Users\plesk\.conda\envs\inpaintEnv37\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 747, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "C:\Users\plesk\.conda\envs\inpaintEnv37\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2727, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "C:\Users\plesk\.conda\envs\inpaintEnv37\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Need value.shape >= size, got ] [2 286 286 3] [2 512 512 3]
         [[{{node StatefulPartitionedCall/random_crop/Assert/Assert}}]] [Op:IteratorGetNext]

Is it possible to train larger sizes?
Is it possible training rectangular format images?

Thank you so so much for making this.
It hopefully removes the need for use JS to run a lot of the models and send their info to OF over OSC. As the awesome Lingdong had done here: https://github.com/LingDong-/PoseOSC

Do you have advice on how to make other models like PoseNet or the mediapipe handpose from tensorflow.js work with your addon ?

Again, thanks so much for making this.
Stephan.

Hi.
Thanks for make this addon available to us.

I am on macOS 10.15.7, OF 0.11.2m Xcode 12.4

I followed the macOS install steps and get a bunch of compile errors related to arm64

Would you know why?
Should I rather try OF 0.11.0 ?
Should I be on a more recent macOS?

Undefined symbols for architecture arm64:
  "_TF_NewSessionOptions", referenced from:
      cppflow::model::model(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in ofxTF2Model.o
  "_TF_NewBufferFromString", referenced from:
      cppflow::model::model(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in ofxTF2Model.o
  "_TF_NewGraph", referenced from:
      cppflow::model::model(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in ofxTF2Model.o
  "_TF_DeleteSessionOptions", referenced from:
      cppflow::model::model(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in ofxTF2Model.o
  "_TF_GraphNextOperation", referenced from:
      cppflow::model::get_operations() const in ofxTF2Model.o
  "_TF_GraphGetTensorNumDims", referenced from:
      cppflow::model::get_operation_shape(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const in ofxTF2Model.o
  "_TF_OperationName", referenced from:
      cppflow::model::get_operations() const in ofxTF2Model.o
  "_TF_GraphOperationByName", referenced from:
      cppflow::model::operator()(std::__1::vector<std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, cppflow::tensor>, std::__1::allocator<std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, cppflow::tensor> > >, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >) in ofxTF2Model.o
      cppflow::model::get_operation_shape(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const in ofxTF2Model.o
  "_TF_SessionRun", referenced from:

This addon was initially developed on Linux and macOS, but has not been tested on Windows. The layout, scripting, and project support files are set up, more or less, but we haven't been able to verify yet.

If you are a Windows developer, please let us know if this addon is working as expected in:

• Visual Studio, 64-bit
• Msys2, 64-bit

This was not easy for me (to find and understand a pretrained colorization model).
With this pretrained model I got it working (had to convert it to a saved model):
https://github.com/EnigmAI/Artistia/blob/main/ImageColorization/trained_models_v2/U-Net-epoch-100-loss-0.006095.hdf5
It tried it with this model before, but it seems they use two models together (do not really understand it yet): https://github.com/pvitoria/ChromaGAN

I converted the model like this (with Python):

from tensorflow.keras.models import Model, load_model
model = load_model(MODEL_FULLPATH)
model.save(MODEL_FULLPATH_MINUS_EXTENSION)

Anyway, here is the example: https://github.com/Jonathhhan/ofxTensorFlow2/tree/example_video_and_image_colorization/example_video_and_image_colorization

GPU training using main.py with pix2pix is not working on my setup. The CPU training and the oF examples are working.

First I was missing .dll such as cudart64_101.dll or cusolver64_10.dll , i added the .dll from other folder or changed the name as suggested here https://stackoverflow.com/questions/65608713/tensorflow-gpu-could-not-load-dynamic-library-cusolver64-10-dll-dlerror-cuso
Ultimately it started training but blocking on Epoch 1 then, epoch 1 stays at 0% and the training continue. After training the resulting predicated image output is a full black images.

my setup:
Windows 11
RTX 3090 driver 516.01
conda environement with python 3.7 and the packages version from requirments.txt
CUDA 11.7 and CUDNN 8.4.1

I managed to train a model for pix2pix, however the output files are looking different than the included edge2shoe example.

I don't have assets and variables folder and no .pb file. I would appreciate some more guidance for how to format my trained model so I can use it with the example.

I am on macOS 12.6.8 with Xcode 141.0. and OF 12
I had this issue. I did add the needed line "$OF_PATH"/addons/ofxTensorFlow2/scripts/macos_install_libs.sh "$TARGET_BUILD_DIR/$PRODUCT_NAME.app"; to the 2nd run script but it did not work.

I had to add a new run script field by clicking on the +

and then add the line as required.

roymacdonald also suggested this option:

in the addons addon_config.mk file

add the following to the osx section

ADDON_LIBS  = libs/tensorflow/lib/osx/libtensorflow_framework.2.8.0.dylib
	ADDON_LIBS  += libs/tensorflow/lib/osx/libtensorflow_framework.2.dylib
	ADDON_LIBS  += libs/tensorflow/lib/osx/libtensorflow_framework.dylib
	ADDON_LIBS  += libs/tensorflow/lib/osx/libtensorflow.2.8.0.dylib
	ADDON_LIBS  += libs/tensorflow/lib/osx/libtensorflow.2.dylib
	ADDON_LIBS  += libs/tensorflow/lib/osx/libtensorflow.dylib
	ADDON_LDFLAGS = -Xlinker -rpath -Xlinker @executable_path

make sure there is a tab at the begining of each line

This was the most difficult for me to understand. But now it works quite well, I guess:
https://github.com/Jonathhhan/ofxTensorFlow2/tree/example_chatbot/example_chatbot
https://forum.openframeworks.cc/t/ofxtensorflow2-chatbot-almost-solved/40090/9

I wonder if somebody use a word2vec model with ofxTensorflow2?
I found a model and tried to use it, but without success: https://tfhub.dev/google/Wiki-words-250-with-normalization/2
For now I use ofxWord2vec https://github.com/perevalovds/ofxWord2Vec for finding subtitle similarities (with a pretrained model) and I would like to do that with ofxTensorflow2 too...
https://github.com/Jonathhhan/ofEmscriptenExamples/tree/main/montageautomat_threaded_2

Hello!!

What do you think to add support to use frozen graph? That way you can use a pb created in tf1, so is more compatible.... i'm doing that and works ok. Old cppflow had that option

I try to run the arbitrary-image-stylization-v1-256 model: https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2
I can load the model without any errors, but it chrashes if I try to run the model (without an error message).
This is how I try to run it:

	if(!model.load("magenta")) {
		std::exit(EXIT_FAILURE);
	}

	model.setup({ {"serving_default_placeholder:0"} ,{"serving_default_placeholder_1:0"} }, { {"StatefulPartitionedCall:0"} });

	auto input_1 = cppflow::decode_jpeg(cppflow::read_file(std::string(ofToDataPath("example_1.jpg"))));
	input_1 = cppflow::cast(input_1, TF_UINT8, TF_FLOAT);
	input_1 = cppflow::expand_dims(input_1, 0);
	auto input_2 = cppflow::decode_jpeg(cppflow::read_file(std::string(ofToDataPath("example_2.jpg"))));
	input_2 = cppflow::cast(input_2, TF_UINT8, TF_FLOAT);
	input_2 = cppflow::expand_dims(input_2, 0);
	std::vector<cppflow::tensor> vectorOfInputTensors = {
	 input_1, input_2
	};
	auto output = model.runModel(vectorOfInputTensors);

Any idea what I am doing wrong (or if the model runs at all with ofxTensorflow2)? Thank you.

I get the following access rights issue when following the Quick Start guide. Do I need to fork the add-on first?

stephanschulz@Stephans-Komputer ofxTensorFlow2 % git submodule update --init --recursive
Cloning into '/Applications/of_v20220530_osx_release/addons/ofxTensorFlow2/libs/cppflow'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:serizba/cppflow.git' into submodule path '/Applications/of_v20220530_osx_release/addons/ofxTensorFlow2/libs/cppflow' failed
Failed to clone 'libs/cppflow'. Retry scheduled
Cloning into '/Applications/of_v20220530_osx_release/addons/ofxTensorFlow2/libs/cppflow'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:serizba/cppflow.git' into submodule path '/Applications/of_v20220530_osx_release/addons/ofxTensorFlow2/libs/cppflow' failed
Failed to clone 'libs/cppflow' a second time, aborting

I am on macOS 12.6.3 with OF of_v20220530_osx_release on an M1.

I cloned the repo. Had trouble with the submodules so downloaded the cpp stuff manually.
Then ran these scripts
./scripts/download_tensorflow.sh
./scripts/download_example_models.sh
which placed things where they belong.

Then used PG on example_yolo_v4
Closed Xcode project.
Ran scripts/configure_xcode.sh example_yolo_v4
And then get the below error.

Would anyone know why?
Thx a bunch.

dyld[66435]: Library not loaded: '@rpath/libtensorflow_framework.2.dylib'
  Referenced from: '/Applications/of_v20220530_osx_release/addons/ofxTensorFlow2/example_yolo_v4/bin/example_yolo_v4.app/Contents/MacOS/example_yolo_v4'
  
  Reason: tried: '/Applications/of_v20220530_osx_release/addons/ofxTensorFlow2/example_yolo_v4/bin/libtensorflow_framework.2.dylib' (no such file), 
  '/usr/lib/system/introspection/libtensorflow_framework.2.dylib' (no such file), 
  '/usr/local/lib/libtensorflow_framework.2.dylib' (no such file), 
  '/usr/lib/libtensorflow_framework.2.dylib' (no such file)

Hello

Im trying to convert some neural style models im using on python. I have those model in checkpoints

If i understand correctly (sure is not the case) i need to run the checkpoint2SavedModel.py to set the input layer size to expected size and get the model in savedModel format

Problem is that i get a very extrange result and sure you have more experience than me on those so maybe you can give me some clues on what im doing wrong

Thanks!

I wonder if it is possible to implement stable diffusion or any other text-to-image network (dall-e for example).
https://github.com/CompVis/stable-diffusion
I guess, the complicated part is the pre-/postprocessing and converting the model to a saved_model...
Any hints are very welcome...

I am on macOS 10.15.7 with OF 0.11.0 and Xcode 12.4
I successfully compiled and ran all provided examples except example_keyword_spotting
Not sure why or how to fix it.
I don't have an immediate need for it but thought you might like to know.

/Applications/of_v0.11.0_osx_release/addons/ofxTensorFlow2/src/ofxTensorFlow2Utils.h:119:14: Non-constant-expression cannot be narrowed from type 'size_t' (aka 'unsigned long') to 'long long' in initializer list

Thanks,
Stephan.

The script for downloading the models does not work:
mv: cannot stat 'download?path=%2F&files=model_basics.zip': No such file or directory
It is still possible with the direct link: https://cloud.zkm.de/index.php/s/gfWEjyEr9X4gyY6

Hello

Thanks for that addon, i want to move some python code i had to c++ in oF but i need to get a new machine.
Seems like now is very hard to find a 2080 so i must go for 3080 but this last one seems to work only with cuda 11 and tf > 2.4

Have you tested this addon with that config?

Thank you!!

Hello

I moved/ported/coded/play or whatever is called correctly this repo https://github.com/PeterL1n/RobustVideoMatting to use it with you addon.

Is supercool, is interesting to add to examples? It has multiple inputs/outputs and use initial recurrent states.

i tried to use this model for music generation: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/audio/music_generation.ipynb#scrollTo=1mil8ZyJNe1w

its very rough yet, and not sure if everything works as expected.
maybe you can find something wrong...
especially my replacement for tf.random.categorical(pitch_logits, num_samples=1) could be wrong.
the model is very small, so it is included in the example...
https://github.com/Jonathhhan/ofxTensorFlow2/tree/music_generation_example/example_music_generation