Decoding errors about webcodecs HOT 13 CLOSED

Closing in favor of #49. Fundamentally the same but the constraints are different now that we are not using Streams.

from webcodecs.

The streams will be errored, and the error object should include some information about the failure.

It may also make sense to add an event, which would work better for warnings.

from webcodecs.

At least in webrtc, when a decoder error happens on video, a PLI/FIR request is sent to the encoder, so I would say that the event would be needed in any case.

Also, would the stream error be clearer if a later frame is decoded correctly?

from webcodecs.

A stream error is fatal.

An event isn't technically required, as the stream entering an error state can be detected (and managed using pipe options). It may be more ergonomic though.

from webcodecs.

Then, what I am requesting is not a fatal stream error, but a transient one so the app can take actions to recover it.

A quite common case is if I frame is lost and only P frames are received. The app should get an error somehow so he can request an picture update to the other side encoder.

from webcodecs.

An app can create a new decoder to recover.

from webcodecs.

That's not how decoder apis work, would be the first decoder api that requires me to create a new decoder on a frame decoding error.

from webcodecs.

I am unaware of a decode API for which a decode error is not fatal. There are of course many media player APIs that hide such details.

Can you provide an example of an API upon which WebCodecs could perform better by having recoverable errors?

from webcodecs.

Any video decoder in webrtc, check for example vp9 impl:
https://cs.chromium.org/chromium/src/third_party/webrtc/modules/video_coding/codecs/vp9/vp9_impl.cc?sq=package:chromium&g=0&l=1676

How about to extend the VideoDecoderOuput with a state which could be OK|NOFRAME|DISCARDED|ERROR and make optional/nullable the frame attribute? That way we would have a 1-to-1 mapping between inputs and outputs.

An ERROR state would not make the stream errored, which we could use for fatal errors (out of memory or whatever non-recoverable error)

from webcodecs.

VP9DecoderImpl is built on the libvpx decode API. In the libvpx decode API, most decode errors are fatal, but there are two error codes that are fatal only to the current GOP (VPX_CODEC_UNSUP_FEATURE and VPX_CODEC_CORRUPT_FRAME). There does not seem to be any (time or resource) advantage in recovering in this case vs. replacing the decoder.

In general I prefer to have one error path, to make it loud, and to use it frequently. This makes it more likely that apps implement the cases correctly, and do not operate forever in a silent error state.

In general, there is no 1:1 mapping from inputs to outputs for a decoder, but I can imagine special cases where it may be useful to mark outputs as missing or partial (DC-only decode, reference-only decode, enhanced error recovery modes, etc). These cases are optimizations, so ideally they would be optional. They seem similar to changing encode parameters (such as target bitrate), perhaps there is a converged API that can cover both cases?

from webcodecs.

There does not seem to be any (time or resource) advantage in recovering in this case vs. replacing the decoder.

It provides a cleaner API, as recreating the decoder is not needed and the stream can be decodable in following frames without having to do anything. Moreover, as the streams are async, I could have pushed several frames to the writable queue that could solve the problem (for example an I frame) and would be lost if the stream is errored and the app has to recreate the decoder.

I am ok to error the stream on fatal errors (btw, which stream would be errored, the writable or the readable one?), but on transient frame decoding errors, I don't see absolutely no benefit. If you feel more confortable, we could call them frame decoding warnings instead to differentiate for the ones that would require a decoder restart.

These cases are optimizations, so ideally they would be optional. They seem similar to changing encode parameters (such as target bitrate), perhaps there is a converged API that can cover both cases?

Not sure if I understood your proposal, could you elaborate what do you mean?

from webcodecs.

I'm not convinced that either is cleaner, but fatal errors are simpler and give the app more control.

I could have pushed several frames to the writable queue that could solve the problem (for example an I frame).

This makes sense for live streaming cases. A wrapper could keep references to undecoded inputs and feed them again in this case.

Chrome does this internally to enable fallback from hardware to software decoders, but it keeps all the way back to the start of the current GOP. I think I am saying that the definition of "transient frame decoding errors" is app-specific.

which stream would be errored, the writable or the readable one?

Both.

Not sure if I understood your proposal, could you elaborate what do you mean?

There are optional decoder features that an app may want to control separately from the current configuration, such as post-processing quality to tune performance, or reference-only decode for stream catch-up. Similar for encode with target bitrate, target resolution, or keyframe request.

Not all decoders support these features, nor should every app need to understand them. An opt-in API that unifies decoding and encoding tunable parameters could be a big improvement.

from webcodecs.

As you said in #46 (comment) there is no 1:1 mapping between input chunks and decoded video frames, so I don't see a viable way for the app to know which chunk has caused the "fatal" error or even when to discard a chunk because they have been already processed.

The usage of streams as decoder input and maintaining a api "purity" starts to look like a solution looking for a problem and not the correct way of abstracting what app developer needs.

from webcodecs.