paulhoux / cinder Goto Github PK

When running the VideoTest application full screen with v-sync enabled, rendering video frames takes (much) longer. I'll copy-paste an email conversation with Rich Eakin and Andrew Bell about it:
#1 (Rich)

I'll definitely try to see if VSync and / or fullscreen are contributing to the issue on Monday. I'm planning to make a standalone test to reproduce outside of our app, which I didn't have time to do last week. We might also need to think about removing two of the cards to narrow down that the quadro sync isn't causing any issues.

When does 'unlocking' the texture take place? One thing to note is that sometimes I am grabbing the video's texture and placing it in an ImageView, as the videos live in an animated carousel like the one in this video.

I was assuming the quadros have hardware accelerated video just because they cause more than geforce's, Andrew do you know? Is there anyway to verify via the nvidia control panel or some other stats display?
#2 (Paul)

Here’s some info on how texture creation and sharing is implemented:

1. When a video file is loaded, a topology (or graph) is created by Media Foundation, consisting of nodes necessary to decode and synchronize the audio and video streams. One of the nodes, the StreamSink, receives pointers to individual video frames, sometimes serveral at once, each of which has a timestamp. The StreamSink then schedules the frames for presentation.
2. Depending on the version of DirectX used, either the PresenterDX9::ProcessFrame() or the PresenterDX11::ProcessFrame() method is called. A pointer to the video frame is obtained (see: GetBufferByIndex and ConvertToContiguousBuffer). Important to know is that the video frame is not yet decoded.
3. After a few indirections, the BlitToShared() method is called. Here, I create a new texture and blit the video frame to it. I use a dynamically resizing pool of textures here. Remember that this is not an extra texture copy! Actually it is the first time the frame is decoded to a texture, either using a call to StretchRect (on DX9, which converts YUV to RGB in the process) or using the hardware video processor in DX11.
4. The decoded video texture is then made available as the “most recent texture” (m_pNextFrame) of which there can only be one at any moment. If another video frame is rendered, the current most recent texture is returned to the pool and replaced by the new one.
5. The scheduler calls PresentFrame() when the latest frame must be presented (according to its time stamp). Here we update the m_pCurrentFrame pointer to point to the latest frame.
6. The OpenGL player calls GetFrame() regularly. It checks to see if m_pCurrentFrame is valid and retrieves it if it is.
7. Upon retrieving the frame, the interop extension is enabled (if not done already) and the texture is shared with OpenGL. Sharing means assigning an OpenGL id to existing GPU memory and then locking it so DirectX can’t access it. While shared, the texture is never returned to the DirectX pool. The texture also prevents the DirectX side of things from being destroyed (using reference counting) until the texture itself is destroyed. Note that I only keep texture memory, the video and its topology are of course allowed to destruct as soon as the video is closed.
8. Upon texture destruction, it is unlocked and unregistered. Reference count of the DirectX stuff is decremented and if it reaches zero will release all DirectX memory associated with it.
   #3 (Andrew)

I can say with pretty high confidence that it's unique to fullscreen+vsync. This is less mysterious than it might first appear; in Win 10 you cannot disable Aero compositing normally. This is why enabling/disabling vsync in windowed mode doesn't have any impact on perf, nor do we ever see tearing. However I think the NVIDIA drivers switch to a different mode when your app is fullscreen, and GL's SwapBuffers() is no longer ignored. I believe this difference is what causes stuttering.

This note in the WGL_DX_interop extension itself is an interesting clue:
2) Can DirectX and OpenGL render concurrently to the same DirectX
resource?

RESOLUTION: Concurrent rendering by OpenGL and DirectX to the same
resource is not supported.

DISCUSSION: The Lock/Unlock calls serve as synchronization points
between OpenGL and DirectX. They ensure that any rendering
operations that affect the resource on one driver are complete
before the other driver takes ownership of it.

The bold portion I think sheds some light on the problem, though not necessarily a solution.

• One approach that might improve this:
  • Create a cache of say, 3 or 4 gl::Textures that we recycle, rather than creating a fresh Texture every frame (this is probably worthwhile anyway)
  • When we recycle a gl::Texture, the first thing we do is unlock/unregister its previous shared resource, and ReturnFrame()s on its D3D texture. We don't do this unlocking on the destructor anymore, but delay it til the next time we require a free Texture.
  • The theory here would be that we're creating some kind of bubble around a specific resource, probably due to frame lag or similar.
• Another approach I'm not totally sure how to tackle would be to call PresentFrame() more frequently on our SwapChain than we are now. Again, if there's some sort of synch bubble here, the theory would be regular time slices to DX are important. I'm guessing that DX only gets woken up at say, 24Hz, by the WMF decoding thread. When Aero (DX) is handling compositing, this is not a problem because DX is always receiving time slices, but when NV driver switches into its special fullscreen GL mode, DX can only service the synch point trigged by wglDXUnlockObjectsNV() infrequently. Are we allowed to call IDXGISwapChain1::Present() from the primary thread? That would be one approach; perhaps letting MovieGl::getTexture() request PresenterDX11 to issue Present() against its swapchain?
  #4 (Paul)

Regarding your solutions:

• Textures are already reused. There is a dynamically resizing pool of textures that are used to render new video frames to. All of this is managed on the DirectX side of things.
• You can’t unlock/unregister a texture that late in the process, because of the strict separation between the OpenGL code and DirectX. Unlocking a texture just before rendering a new frame to it would require us to mix OpenGL code with DirectX, which is a maintenance nightmare and reduces the code’s flexibility, making it impossible to use the code for DirectX-only targets. Also remember that we’re using a pool of textures and a texture that has been unlocked and returned to the pool is not reused instantly.
  It’s also impossible to never unlock the texture, because this prevents DirectX from rendering to it.
  Precisely this part of the code is why it took so long for me to find a workable solution. I have tried at least half a dozen approaches, most of which simply did not work and one that created a mess of tangled code. The current solution is by far the best one.
  I believe the call to unlock is relatively slow because of the internal driver synchronization, as you’ve mentioned. Knowing more about this synchronization will help us finding a better algorithm. See next bullet point.
• I am not sure how precisely the call to Present() is handled, where it comes from and how frequently it is called (I’ll have to run the application again to see how it works). I do know that we’re using an invisible window to create the message pump and I assume that Windows is calling Present() for us. In a recent commit, I disabled vertical sync for the swap chain, in hopes of not having to wait twice for the blank signal, but I am not sure this improved things, only that it did not degrade performance. Learning more about the Present() stuff is probably a good thing and might lead to a better algorithm. I do know that getting rid of it entirely will also remove all scheduling and an hour long movie will play in mere seconds 
  #5 (Andrew)

Ad 1: Sorry, was referring to gl::Textures, which are not being reused:
https://github.com/paulhoux/Cinder/blob/video/src/cinder/wmf/MediaFoundationGlImpl.cpp#L272

Ad 2: Imagine a 4 frame cache; we receive frame A B C D, and then we need to reuse a gl::Texture. At this point we release/unlock the old GL resources (and the D3D texture back to our internal pool) associated with the now defunct A, and then we assign to it the new frame. The concept being, if the GPU is ~3 frames behind, then we could appear to be done with a frame and attempt to reuse it when in actuality the driver must hold on to it, resulting in spinning until that frame has rendered. Instead, we delay unlocking the GL resource until we actually need it, decreasing the odds that the GPU is actually in need of that resource.

Ad 3: OK - would you be able to experiment with forcing the message pump to make more frequent calls to Present (I bet they only occur at the framerate of the video right now) and I can take a look at redesigning the cache?
#6 (Rich)

I actually had tried borderless + maximized window + vsync and while it performed better than fullscreen + vsync, there are still noticeable hitches and the framerate drops off after 3 or 4 videos, unlike when I just disable vsync. So - do we gain anything by switching to win10? Reason why I ask, is that my PC at home is handling 7-9 videos at the same time much better, and there I can switch on vsync although I'm not really sure what it does if anything. Biggest worry I have is that we're going to disable vsync, then show up at the install and see nasty frame tearing once we hook up to the 9 displays (in mosaic, again).

Interesting you're seeing the com error messages on win10 too, I'm not. Although I agree its probably harmless, it's also mysterious as to what combination of hardware / software is causing it, and it just so happens that for me they are coming up on a more powerful machine that is performing much worse. Btw Paul you can't break on the exceptions, they appear to be beyondthe COM boundary.
#7 (Andrew)

OK - sorry, has been tricky for me to get in hours against this since I can't repro it on my work PC.

I implemented a more sophisticated caching scheme that recycled textures, and delayed unlocking a resource as long as possible. Unfortunately it did not improve performance discernibly. However I also created a 60 FPS movie and found something interesting. The frame processing time is exactly what I'd expect it to be if my other synch point theory were true. Frame processing time is about 40% of what it is with a 24 FPS clip.

So to reiterate, I believe that when wglDXUnlockObjectsNV() is called, the primary thread spin-waits until the D3D context gets some time to respond to the request. We already know that this one call is what causes video to stutter. This is further supported by the statement I quoted from the GL extension, which cites this call as a "synchronization point" between DX and GL. My theory is that D3D is constantly getting time slices when the app is not fullscreen because Aero is handling all the compositing, and it is in turn D3D. Also if you just reason about it, it makes sense - there's no way for GL to "get into" the D3D driver to mark a resource as available, so some sort of synch mechanism needs to be received by D3D to service this call. This is why the time to service a single frame exactly corresponds with the framerate of the media - the video thread gets woken up at 24Hz, and notices a pending synch point initiated by the primary thread (via wglDXUnlockObjectsNV()) which it services (marking the resource as available for D3D's use), thus freeing the primary thread to continue. In particular, my theory is that this line in the "media" thread is the one that allows wglDXUnlockObjectsNV() in the primary thread to continue.

Paul, I attempted to force PresenterDx11::m_pD3DDevice to Flush() both before and after wglDXUnlockObjectsNV() from the primary thread, which had no effect. However this is not too surprising - I can easily imagine a scenario where the D3D time slice must be provided by a different thread - presumably the "media" thread associated with the Presenter. What I could use your help on would be determining a way to issue events on the "media" thread, which will likely call something like m_pD3DDevice->Flush() in order to give D3D a time slice to service any possible synch points with GL / primary thread. What do you think the best way to do this would be? What I'm imagining would be a new type of StreamOperation on StreamSink which we enqueue at regular intervals.

I have a lot of optimism that this could fix our issue, based on the evidence the 60 FPS video provided, and also the fact that it just makes sense.

E-mail from Rich Eakin:

Hey guys, just writing this note to you here, although I haven't been able to even repro in debug mode but I'm seeing what I think is a deadlock when a video loads or unloads, but usually after many usages... it's pretty random.

When I pause the app it is usually hung on this line, which obviously doesn't make sense because it is a simple macro checking whether hr == 0, so it's probably the line right above. I also saw the deadlock happen one time when setting the volume, how

Anyway, just letting you guys know at this point, I'll continue investigating on my end..

This is happening because the MovieGl calls close() when the app's window closes, which destroys the MovieGl's internal state, yet the MovieGl's destructor hasn't been called yet. I hit this as I was trying to pause the video from my VideoView when shutdown began. However at this point, calling almost any of the MovieGl's methods will cause assertions to fire in debug and try to access a null mObj in release.

Maybe all those assertions aren't necessary, and instead the methods just don't do anything and return default values when mObj or mObj->mPlayerPtr don't exist (shutdown)?

Hi All!

The vc2015_winrt project didn't compile.
Errors:

• Error C2065 'GL_BGRA': undeclared identifier cinder d:\Projects\Temp\Cinder\src\cinder\gl\TextureFormatParsers.cpp 283
• Error C2065 'GL_BGR': undeclared identifier cinder d:\Projects\Temp\Cinder\src\cinder\gl\TextureFormatParsers.cpp 289

p.s.
VS 2013 - ok

Because MovieGl registers for app main window's getSignalClose() here. I'm wondering if this will be problematic in multi-window situations?

Also, not sure why the MovieGl is tied to a specific window, or would it be OK to close all MovieGl's on app's cleanup signal?

I'm not sure what could be the cause of this, but it looks like the last 6-10 horizontal rows (bottom) of the video's texture is stretched:

I created this images by placing the following line in the VideoTest's draw() function, right after it draws the texture:

writeImage( "video.png", texture->createSource() );

Here is the video that I used to create the image with the otter.

If you do the following:

mVideo = MovieGl::create( fullPath );
auto size = mVideo->getSize();

size is [0,0], which is unexpected if you wanted to use it for laying things out. Probably because the size isn't set until the firs texture is grabbed, but I think it would be ideal if the video properties were queried to determine the size before draw time.