example/video/main.go hangs after 3 frames on macOS only about go-scrap HOT 7 OPEN

Yeah, the code here is quite small, so feel free to debug as needed to see where it's bloicking. Maybe something on the Rust side which is where the would block is returned. In many cases you might be able to ignore would block errors, I am unsure (I don't use mac).

I did go ahead and merge that PR to make mac builds work

from go-scrap.

Okay, I've made some progress on understanding this.

The quartz (macOS) implementation of screen capture is significantly different from the others. It uses a CGDisplayStream to grab frames. A parameter passed to this during initialization gives it a queue length (number of frames to queue). This is configurable from 3 to 8 frames, but defaults to 3 (and in the rust scrap implementation, it is actually set to 3). I confirmed that changing this variable to 8 (forking the repo, and modifying the go-scrap/scrap-sys/Cargo.toml file to have scrap = { git = "url/of/forked/modified/repo" }), results in the video demo hanging after 8 frames instead of 3.

When we start the CGDisplayStream, a CGDisplayStreamFrameAvailableHandler starts running constantly. When a frame becomes available, the frame gets transferred to a surface and then gets passed off, locks the frame, and runs IOSurfaceIncrementUseCount(). After it returns the frame, it drops the frame, unlocks the frame, and runs IOSurfaceDecrementUseCount().

This actually works fine in the background, when we haven't started trying to call capturer_frame from go-scrap. So the queue isn't filling up just by the capturer running. However, when we call frame HERE, which is what go-scrap actually hits, we end up running IOSurfaceIncrementUseCount(), BUT IOSurfaceDecrementUseCount() never gets called. So it looks like we end up running up the buffer by one frame every time we try to grab a frame from go-scrap.

So, perhaps in capturer_frame we are throwing away the frame from memory and it never gets dropped? Changing this to std::mem::drop(frame) breaks things, but I feel like there should be a way to handle it here.

from go-scrap.

Good catch. There is most definitely a drop implementation in the underlying frame data at https://github.com/quadrupleslap/scrap/blob/de3cd4cd8610f1cac7f80e29f59dbdb93ffefc6d/src/quartz/frame.rs#L37. What should probably happen is I should copy over the u8 slice (e.g. copy_from_slice) in Rust and return that and then let the RAII of the frame call drop when it leaves scope like it normally would instead of mem::forget'ing it. This technically reduces performance, but is safer. There's probably a way to just mem forget the slice itself or something instead of copying but still let frame's drop get called. I don't really work on this library much anymore, but yeah that'd be the first step is to see if you can get drop to be called while returning the data as cheaply as possible. I'd gladly merge a PR to this effect.

from go-scrap.

I got it to work! No guarantees on performance though. Ended up copying the frame over as you suggested. I agree it should be possible to just forget the slice, just haven't dug into it far enough to find out.

Here's what I ended up doing:
efunn@e59765e

This will definitely slow down all implementations, so maybe not ready for a pull request yet, but if you have any suggestions to clean it up, please let me know!

from go-scrap.

Awesome. One thing I was thinking is giving back the frame pointer along with the byte slice, storing it on the Go side, and dropping it before grabbing again each frame call. Then add a Close call to the capturer to make sure that last frame is collected. Or really, just keep a capturer on the rust side updating the frame over and over and let it have a call to get the bye array. But meh, lots of solutions.

from go-scrap.

So actually, in the Rust implementation of scrap on MacOS, the capturer is updating the frame over and over constantly. When you do your frame call, it think it just tries to lock the frame and then grab it from that constantly updating capturer. So it might already be doing what you envision.

Or are you suggesting you add another layer to avoid interacting with the Quartz CGDisplayStream directly from golang? Like, always grabbing the frame into memory somewhere in your lib.rs rust code regardless of whether you actually need it, and then simplifying your frame call to just swap the latest frame over to the golang side? This would avoid having the golang side directly grabbing from the Rust scrap library (and the underlying CGDisplayStream objects). My concern with that is how it will affect performance across platforms - since it's only really broken on MacOS currently, so this complexity would only be a benefit for one platform.

I think your first suggestion of just more intelligently grabbing the frame pointer/byte slice (rather than making a copy in memory) would be simpler... I feel like it should just be a couple lines of code to change. If we can ensure it isn't doing any memory copying then performance shouldn't be affected on other platforms. I would still do some timing tests to double-check, though.

from go-scrap.

Or are you suggesting you add another layer to avoid interacting with the Quartz CGDisplayStream directly from golang?

I suppose. Basically you just have to make sure the frame is properly dropped (which calls the proper macOS code) at the right time from Rust which happens when there are no more references to it. This is why when you copied that slice, it was able to be dropped which called the right code. There are many ways to do this. One way is to return the opaque frame mem::forget'ed reference/pointer back to Go, then mem::drop'ing it before next frame or on close.

from go-scrap.