Using QUIC for real time media about mvfst HOT 8 CLOSED

Hey there!

mvfst offers partial reliability and scatter/gather API which may address some of the real time video needs you're talking about.

Partial reliability allows you to "reject" or "skip" bytes to certain offset on a given QUIC stream. For example, you're sending 100 bytes of video, but at some point during transmission application sees that it's taking too long to send the bytes and decides to "cancel" first 50 bytes but still send the rest - it can do using partial reliability API. Partial reliability API is represented by sendDataExpired() and sendDataRejected() calls in quic/api/QuicTransportBase.h.

There is also scatter/gather (or peek&consume) API represented by peek() and consume() calls in the same file. Peek allows one to get a notification when bytes arriver to mvfst's read buffer, not necessarily in order. What that gives to the application is ability to be notified whenever new bytes arrive on the wire. Application can then keep its own view/state of the bytes in the read buffer (e.g. how many gaps there are, how long it's been waiting for bytes to arrive etc). And the app can use consume() call to consume unwanted bytes and move the (consecutive) read pointer froward if need be.

Disclaimer is that both API are still raw as we're literally working on the features as we speak. Let me know if you have more questions or suggestions, I'd be happy to chat.

from mvfst.

...there are also delivery callbacks that might be useful as far as timing feedback goes. Look for DeliveryCallback in quic/api/QuicTransportBase.h. It basically allows application to get notified when certain bytes (offset) were delivered (or not delivered) on a stream.

from mvfst.

Interesting! This is really helpful, thank you.

So thinking about this more, in order to use FEC, I need a way to send discrete blobs of data in each packet. That is, I'd like to define my own framing. The reason is that is that if I have a 10kB encoded frame, I could split it up into 10 1kB chunks. Then, I could apply 30% FEC to this and get 13 1kB chunks. The receiver needs any 10 of these 13 chunks to generate the initial 10 chunks (and thus the 10kB encoded frame).

I don't see any obvious way to do this in QUIC or mvfst. I see, for example:
stream.writeBuffer.append(std::move(data));
A piece of code that I found through
EchoClient.h:sendMessage -> writeChain -> writeDataToQuicStream

Which is as I hypothesized: QUIC only has a byte stream abstraction, not a messaging abstraction like UDP/RTP.

Apparently TCP has some sort of support, as described in https://tools.ietf.org/html/rfc4571 .. but I think it just runs the whole byte stream through and at the end breaks up the byte stream into packets.

Perhaps one way to get around this is to write data blobs to the stream that are exactly the size of the MTU. But I'm guessing there's probably a difference between the packetSize setting in these params, and the amount of data that should be sent, due to headers and maybe other things like encryption (not sure if that makes it bigger or not). And, er, maybe the header size is variable in certain instances.. so maybe this strategy wouldn't work.

But it doesn't mean I can't change things, right? :). It seems mayyybe I could just edit QuickPacketBuilder.cpp? Maybe I'm being too optimistic

from mvfst.

You are correct, QUIC only offers byte stream API as far as read/write interface goes. Of course, you could roll your own framed protocol on top of QUIC streams.

FEC, looks like there was a proposal: https://datatracker.ietf.org/meeting/104/materials/slides-104-nwcrg-05-michel-fec-quic-results-00 for QUIC FEC, but I’m not sure where things stand with that proposal.

from mvfst.

Do you really mean "on top" of QUIC? My understanding is I would have to modify QUIC itself.

from mvfst.

You can certainly fork and modify the code to make it egress bytes in the way you want it. I was talking about building protocol on top of QUIC without QUIC modification.

You could also use multiple streams to deliver video segments, e. g. one video segment per stream. Since each QUIC stream is independent, that will give you somewhat independent (and potentially out of order) delivery of data across multiple short lived streams.

from mvfst.

Got it, thank you for the guidance, I appreciate it! I'll send an update if I do go down this path

from mvfst.

Looks like questions got answered thanks to @kvtsoy . I'm closing the issue. Feel free to open any further issues if you feel something isn't answered yet, or if there is API changes you'd like to see us make to better support your use case.

from mvfst.