Long term reference support for AVC/HEVC about webcodecs HOT 6 OPEN

It's also not clear to me what the exact behavior is when a large number of IDs are outstanding. There is only so much room to store references, eventually the encoder has to drop them. How much latency can there be before these schemes fall apart? How does this vary between encoder implementations?

[BA] The viability of LTR in conferencing scenarios relies on assumptions about how participants manage long-term references. They're called long-term references because the participants need to keep them long-term or else an encoder can't be assured that it can rely on an LTR to create a delta frame that all the conference participants will be able to decode. And of course, regardless of how long LTRs are kept, you can have a recent joiner who won't have received the LTR. So it is quite possible for an encoder to create a delta frame based on an LTR and then receive a PLI in response. Ooops!

from webcodecs.

Related: #285, aboba/hevc-webrtc#17, w3c/webrtc-nv-use-cases#118

from webcodecs.

We (Chrome implementers) discussed this recently. It's possible, but there are some design considerations that need to be solved:

• Not all encoders support it (eg. MediaCodec), so it needs to be feature-detectable.
• Different encoders use different signaling mechanisms, so we probably need to find a way to unify them all.
• Rather than creating serializable IDs, we could use opaque objects and leave mapping/serialization up to apps.
• Putting IDs on encoded chunks is ambiguous since there can be more than one frame in an encoded chunk. In practice this feature probably won't be used together with frame reordering, and if superframes do get involved it's probably fine to interpret a single ID as representing all frames in the chunk.

It's also not clear to me what the exact behavior is when a large number of IDs are outstanding. There is only so much room to store references, eventually the encoder has to drop them. How much latency can there be before these schemes fall apart? How does this vary between encoder implementations?

from webcodecs.

We (Chrome implementers) discussed this recently. It's possible, but there are some design considerations that need to be solved:

• Not all encoders support it (eg. MediaCodec), so it needs to be feature-detectable.

Are you suggesting to add something to be checked by MediaCapabilities API?

• Different encoders use different signaling mechanisms, so we probably need to find a way to unify them all.
• Rather than creating serializable IDs, we could use opaque objects and leave mapping/serialization up to apps.
• Putting IDs on encoded chunks is ambiguous since there can be more than one frame in an encoded chunk. In practice this feature probably won't be used together with frame reordering, and if superframes do get involved it's probably fine to interpret a single ID as representing all frames in the chunk.

It's also not clear to me what the exact behavior is when a large number of IDs are outstanding. There is only so much room to store references, eventually the encoder has to drop them. How much latency can there be before these schemes fall apart? How does this vary between encoder implementations?

I may create prototype CL in Chromium to examine the behavior on Windows at least.

from webcodecs.

Are you suggesting to add something to be checked by MediaCapabilities API?

More likely a flag in VideoEncoderConfig so that VideoEncoder.isConfigSupported() can be used to detect it.

from webcodecs.

sandersdan@ max number of outstanding long term ref picture indices(as well as max value of long_term_ref_frame_idx + 1 ) must not exceed maximum possible reference pictures (for AVC/HEVC, it is 16);
If application explicitly specifies maximum reference frame numbers, that will be further reduced to application specified limitations.

from webcodecs.