Android: RGB bitmap from YUV image about tflite-support HOT 13 OPEN

Got it. Supporting YUV conversion utils is in our roadmap. I'll give updates here once we have any progresses.

So far, we only touched image processing on CPU. We have some undergoing projects related to GPU computation, but they are still in an early stage.

from tflite-support.

@lu-wang-g, no worries, thanks for the update!

Perhaps I can ask for a consult. Do you have a suggestion regarding the best approach to pass images from camera to a model with a possible persistence to a JPEG / WebP file? Maybe some team at Google had a similar workflow. Our model consumes RGB bitmaps, we’ve used RenderScript for the YUV → RGB conversion so far. Unfortunately, RenderScript is deprecated (as a tool) — we’ve tried libyuv as a replacement but it’s a bit less performant. The trick is — we have to persist resulting images in the original resolution (4K) so we cannot request small-ish images from the camera. I guess one approach might be to switch a YUV-consuming model but it will require a retraining and it still might be less performant IRL.

from tflite-support.

@lu-wang-g Hi Lu, Any thoughts on this?

from tflite-support.

Converting to RGB and image rotation are supported in Task library through ImageProcessingOptions, such as ImageClassifier#classify(TensorImage, ImageProcessingOptions). You don't need to convert it to Bitmap. We're working on publish examples and tutorials to introduce consuming YUV images directly.

Surfacing YUV image conversion to RGB in TensorImage is not prioritized at this moment. We have the C++ implementation, but haven't wrapped it in Java yet. In C++, image processing is done through FrameBuffer. See the code here for an example of how to.

from tflite-support.

@lu-wang-g, thanks for details!

I understand that the Bitmap becomes kind of redundant but we still need it (for our use case) to save the image as a JPEG file. Maybe it can be done on the native level as well though. We don’t save all images however since not everything is useful. Not sure how popular this scenario is to do the investment but I imagine that saving model-enriched images after their processing is used by someone besides us 😉

from tflite-support.

@arturdryomov I agree with that saving images could be very useful, at least for debugging purposes I can imagine.

If adding implementation to getBitmap on MediaImageContainer is all your need, and not very concern about performance, we can add an implementation there (as far as I know, our C++ YUV library is really fast, Java implementation might not be that efficient), but we can not promise a date due to workload.

If you're interested, any contribution is really welcomed!

from tflite-support.

@xunkai55, thanks, this might be useful indeed. In the meanwhile I guess we might want to explore using FrameBuffer API instead 🤔

from tflite-support.

@lu-wang-g, hey, is there a reason for this issue to be closed? Seems like a valid use case (not of a top priority of course but still).

from tflite-support.

Ha, I thought the issue was walked around by using FrameBuffer. Let me reopen it to track.

from tflite-support.

@lu-wang-g, unfortunately we weren’t able to use FrameBuffer for this since we didn’t want to pull this repo to get access to header files. We did something with libyuv which kinda works but it would be useful to have this built-in to avoid dealing with NDK / JNI.

BTW, as a side-note — have you tried using Vulkan for the YUV → RGB conversion? Would be interesting to know if there are benefits.

from tflite-support.

@lu-wang-g, hey, are there news regarding this issue?

from tflite-support.

We had to shift our roadmap on image utils soon after I left my previous comments last year. We won't be able to deliver this feature in the near feature. Sorry about it.

from tflite-support.

Libyuv should be very performant as far as I know. We use it to perform image processing in Task Library as well. Alternatively, you can request RGBA output from CameraX directly. See this blogpost: https://medium.com/androiddevelopers/convert-yuv-to-rgb-for-camerax-imageanalysis-6c627f3a0292.

from tflite-support.