Importing / exporting out of band side data about webcodecs HOT 9 CLOSED

If we had a grab bag of codec-specific parameters in (a list of key value pairs), H264 could have some for SPS and PPS, right?

But would that be a static or dynamic or frame encoder parameter?

from webcodecs.

I believe extra data specified in contains like mkv only need to be specified when initializing the decoder, so they would be static parameters.

from webcodecs.

I found a same this issue as what I questioned.

Should WebApp parse byte stream to pick up SPS/PPS?
For instance, AVC RBSP (Raw Byte Sequence Payload) just be along with startcode, no payload size. It seems to be bother...
It may be easy to initialize a encoder by using SPS/PPS at transcoding if SPS/PPS and frame encoded data are already separated.

Android MediaCodec has already provided such way, I think. (Maybe, if my understanding is correct.)

Although above things are depending on codec spec

from webcodecs.

The in-progress implementation in Chrome has EncodedVideoConfig::description, which is where an avcC/vpcC would (must) go.

There is no equivalent field on EncodedVideoChunk. Internally, Chrome does have DecoderBuffer::side_data, and it is used to hold an alpha channel for VP8. I don't think this needs to be exposed in the WebCodecs API, apps should be responsible for handling side data themselves, and WebCodecs shouldn't be trying to interpret such data.

As currently implemented in Chrome, H.264 must be in AVC format; Annex B is not supported. The implementation also requires that each submitted buffer contains exactly one primary coded picture, so even if Annex B were added, apps would need to identify frame boundaries themselves. I am hesitant to relax that requirement, it makes many other things much simpler and easier to specify.

from webcodecs.

As currently implemented in Chrome, H.264 must be in AVC format; Annex B is not supported.

I'm sorry that I forgot that spec. It makes sense.

However for use-case of partial transcoding (for video-editing use-case), it needs to encode compatibly with SPS/PPS which come from original, I think.
Is it difficult to do ?

from webcodecs.

As currently implemented in Chrome, H.264 must be in AVC format; Annex B is not supported.

I might have misunderstanding. Please let me confirm.
Does it means that encoder of WebCodecs will generate each Access Unit (includes AUDelimiter/SPS/PPS/PrimarySlices) in each Buffer without startcode?

If so, doesn't it look so hard to find boundaries of NAL Unit? (^_^;)

from webcodecs.

it needs to encode compatibly with SPS/PPS which come from original

FYI we (Chrome WebCodecs implementors) do not intend to support this. WebCodecs in Chrome will expose the existing RTC encoders, with the same limitations they currently have. SPS and PPS are not part of an encoder config, rather they are an output of the encoding process.

I suspect that to support what you describe in general, you would need to use the same encoder for the original encode and the re-encode. WebCodecs doesn't guarantee that happens, even if you use it end-to-end on the same device. It could use different encoder implementations internally.

Does it means that encoder of WebCodecs will generate each Access Unit (includes AUDelimiter/SPS/PPS/PrimarySlices) in each Buffer without startcode?

AVC format does not have start codes, and it doesn't require the generation of AUD/SPS/PPS (AUD is implied by frame boundaries, SPS/PPS are in the avcC).

But basically yes. Chrome's current MSE path actually does exactly that (converts to Annex B format) as part of bitstream validation.

from webcodecs.

Thank you for the information. It make me clear.
As you said, it seems to be hard to do what I considered with different encoder. It looks like that I should handle a unit of IDR period.

I've checked VideoEncoder between latest idl and google doc. It looks work in progress. So I'll be wait to release them.
(It's just be RTC use-case though, I concern that I wonder if a path from VideoEncoder to VideoDecoder may be complicated to initialized.)

Anyway thanks!

from webcodecs.

I think all issues here are now addressed, but please re-open if more to discuss.

The current state is:

• avcc information may be provided via the VideoDecoderConfig.description
• if you fail to provide the avcc, we will assume you intend to use annex-b format with SPS/PPS periodically throughout the bitstream. Work to clarify this in the spec tracked by #143
• We also support producing annex-b from the VideoEncoder by specifying the desired format from in the encoder config (see #117)
• VideoEncoder configured to use avc format will emit the avcc information via the output callback (see #113)

from webcodecs.