arbitrary-image-stylization-v1-256 about ofxtensorflow2 HOT 13 CLOSED

Hey @bytosaur,
I think I found a solution and made an example: https://github.com/Jonathhhan/ofxTensorFlow2/tree/universal-sentence-encoder/example_arbitrary-image-stylization
The advantage is (in my opinion) that you just need to load an image and not a model for the style.

from ofxtensorflow2.

hey @Jonathhhan,

thank you for your contribution! I am currently trying out your example on an RTX 3080.
I have two ideas how to improve the performance:

• use multiply by 1/255.0 instead of div by 255

   input2 = cppflow::mul(input2, cppflow::tensor({ 1.0f/255.f}));

• allocate the image beforehand instead of for every image

   imgOut.allocate(videoPlayer.getWidth(), videoPlayer.getHeight(), OF_IMAGE_COLOR);

Please feel free to make a PR for this example!

BTW could you please name the example example_style_transfer_arbitrary? This way they are next to each other :) and i would like to keep the basic example on top

from ofxtensorflow2.

oh and please keep in mind to use the standard OF_ROOT. I.e. leave the variable commented in config.make ;)

from ofxtensorflow2.

hey @Jonathhhan,

i have already pushed a branch with some smaller changes. I ll first adjust and merge your other examples before modifying the download scripts :)

from ofxtensorflow2.

Hey @Jonathhhan,

I am currently on vacation and can only look into it at the mid of next week.
It looks alright how you handle the input. Maybe you are running out of Gpu memory? Try setting memory growth with

if(!ofxTF2::setGPUMaxMemory(ofxTF2::GPU_PERCENT_90, true)) {
  ofLogError() << "failed to set GPU Memory options!";
}

from ofxtensorflow2.

Yup arbitrary style transfer is superior in that sense. How much fps do you get and what hardware do you use?
The same model has been used in Alexander Schubert's crawl3rs ;)

from ofxtensorflow2.

It runs at 60FPS and 1280 x 694 (video resolution) with an RTX3090, but it needs a lot of power (around 340w). Not sure how to reduce that. With 1 FPS it goes down to 120 w.

from ofxtensorflow2.

hey @bytosaur,
thanks for the improvement ideas. mul instead of div does not seem to make a noticable difference and i need to allocate the output image after running the model, because sometimes the output shape differs slightly from the input (in my case the width difference is 2px).
Here is a branch with just the example. I changed the name and made some small changes:
https://github.com/Jonathhhan/ofxTensorFlow2/tree/example_style_transfer_arbitrary/example_style_transfer_arbitrary
I also tried to adapt the example_style_transfer example but I think I had problems to setup the model in the model class, so it didnt work (but in theory it should ;)).
And I copied config.make from the other example, cant remember that I changed something (also not sure how it works, exactly).
I do not know how to keep the empty folders in github with the help of gitkeep files, they always disappear (there should be a data/model folder)...
And the readme is not written yet :)

from ofxtensorflow2.

Hey,
Yes the mul doesnt make much of a difference in compute time but I still prefer it actually. Division just give me the chills :D
The difference in pixels sounds really really odd... It worked pretty well for me and improved the performance quite a bit.
No worries about using inheritance. I think it s good to have some examples that are more straight forward.
I'd be happy if you add a little README (please also put in your and the models reference :)).
Then we can merge the example and make our small and more stylistic changes ;P
Never had any issue with leaving and empty .gitkeep file in the folder... are you sure you added it to the commit?

from ofxtensorflow2.

Alright :) It seemed that the video had a strange resolution or was corrupted, with another video it works without allocating from the shape. I also replaced div with mul. And I read the video texture directly into float pixels now, so no further conversion is needed. I will have a look into the other things...

from ofxtensorflow2.

hey @bytosaur,
thanks a lot for improving and polishing the examples (and for the addon itself, of course). I am still a beginner and already learned something from it.
I also found some small things that I could have done better (but nothing with a big increase in performance).
I can do some pull requests from your branches.
This way for example, the ofFloatPixels object is not needed anymore:

		input = ofxTF2::pixelsToTensor(videoPlayer.getPixelsRef());
		input = cppflow::expand_dims(input, 0);
		input = cppflow::resize_bicubic(input, cppflow::tensor({ 360, 480 }), true);
		input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
		input = cppflow::div(input, cppflow::tensor({ 255.f }));

from ofxtensorflow2.

hey @Jonathhhan,
we highly appreciate your examples! I didnt do much on the code side of things except for packing the inference part into a separate function. Your examples have been pushed to this branch and will be merged soon :)

from ofxtensorflow2.

included in #29

from ofxtensorflow2.