Sync animation to video about unityplugin-avprovideo HOT 6 CLOSED

Hi,

Currently we don't really have a system for this. There might be a few ways you can try though.

On some platforms (like Windows) you can retrieve the timestamp of the texture. This is more accurate than querying the mediaPlayer for it's time, as the mediaPlayer time is the time is decoding but not the frame you're about to display.

long time = mediaPlayer.TextureProducer.GetTextureTimeStamp();

This 64-bit value is returned in HNS format (hundreds of nanoseconds), so you might want to convert it to seconds or frames:

double seconds = time / 10000000.0;

float frameRate = mediaPlayer.Info.GetVideoFrameRate();
int frame = Helper.ConvertTimeSecondsToFrame((float)seconds, frameRate);

You then have an accurate way to update your animation to the correct frame.

Also a good idea when you need to sync a video to something is for the video to be encoded in a way that is very fast to seek randomly. For this you need a video with lots of key-frames. Ideally each frame should be a key-frame. See the PDF documentation where we have a whole section on seeking and key-frames.

from unityplugin-avprovideo.

The technique works perfectly on Windows with GetTextureTimeStamp(). Unfortunately, on Android the results remain too inconsistent to be useable.

Part of the problem is that on Android the timestamp does not start at 0 for frame 0. Instead it seems to be time of day or some such. I'm not choosing the exactly correct moment to store the timestamp. It's not entirely clear how to go about it; I tried storing the first reported number above zero, but that was off by 35-37 frames. Then I tried storing it based on when GetCurrentTimeMs() is zero, which ends up being off by 0-2 frames. I also tried using MediaPlayerEvent.EventType.FirstFrameReady to set the initial timestep, which again was off by 35-37 frames. And that range changes as the video plays so I go out of sync. Generally it does not seem to be more reliable timing than just using GetCurrentTimeMs().

On Android you can change the video API, but results were the same with MediaPlayer and ExoPlayer, except that with MediaPlayer the 35-37 frame gap became 4-5 frames.

from unityplugin-avprovideo.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from unityplugin-avprovideo.

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

from unityplugin-avprovideo.

I found a workaround by storing frame information in the video and reading it out of a shader, but it's inconvenient and leaves artifacts in the video. So I'm hoping the new Android version will work better for this use case.

from unityplugin-avprovideo.

In version 2.x we have improved the time precision and frames have an accurate timestamp. There is always latency though when asking for a video frame, so to keep them in sync you could either buffer decoded frames and then select the one with the matching timestamp, or you could wait/block while the decoder is processing the requested frame and poll for it in a loop with some sane timeout value.

Closing this as there's nothing we can do to improve this

from unityplugin-avprovideo.