Comments (14)
Nit: decoding_sequence
should be called decodingSequence
to follow our naming policies (check TAG design principles)
from webcodecs.
Sorry to comment on an old ticket but I'm seeing behavior in Safari where VideoDecoder (with h264) seems to be returning frames in decoder order rather than presentation order, creating strange playback artifacts. (Same thing plays fine in Chrome.)
I can't tell if this is something I need to handle in JavaScript or is it a Safari implementation bug?
from webcodecs.
Frames are to always be output in presentation order, and it's the job of the implementation to ensure this is the case.
https://w3c.github.io/webcodecs/#dom-videodecoder-decode has the spec text for this, specifically this line: https://w3c.github.io/webcodecs/#ref-for-dom-videodecoder-codec-implementation-slot%E2%91%A8
https://bugs.webkit.org/ is the bug tracker for WebKit. @youennf might also know if there's something opened already.
As always, test-cases are appreciated by implementers, if you have one handy you can share with them.
from webcodecs.
@marcello3d, please file a bug report. I believe VTB is providing in decode order and WebKit is missing some logic to do the necessary reordering.
from webcodecs.
This actually is very similar to #7, perhaps they should be merged.
Two key things to consider:
- Not all platform decoders can operate in this mode. A WebCodecs implementation may need to rewrite slice headers to remove reordering to get the described behavior, and this requires parsing of the bitstream.
- This only really applies to H.264/H.265. VPx and AV1 use a different system of non-output frames that doesn't fit this shape well. Because of that it is more consistent to output in presentation order.
On the plus side, outputting in decode order resolves some difficult questions about backpressure.
from webcodecs.
Are there other use cases where a web app would want to receive frames in decode order? I wonder if this flexibility is needed, and if it's worth the additional complexity (from the web app's point of view).
from webcodecs.
"UA does not need to maintain a queue of decoded VideoFrames that will be given to the Web App sometimes later after other frames."
Of all decoding APIs, only Apple VideoToolbox decoder output frames in decoding order.
FFmpeg, Microsoft's WMF, Android MediaCodec all return frames in presentation order, and this is what you want.
The UA typically doesn't maintain a re-ordering queue, the low-level decoder itself does.
I don't believe that there's a use case that could ever be useful to the user in returning in decode order.
It makes the life of the user more complicated, requires them to write more code and it's not something the user-agent could even provide even if it wanted to.
As mentioned, only H264/H265 require such re-ordering queue.
from webcodecs.
I believe the most important point is to decide if there ever is a use-case where having the frames in decode order is useful. For now there seem to be a clear preference for outputting in presentation order, except in the initial message in this issue, and the second part of #55 (comment), but it's not an argument strong enough to decide considering the Web API design principles (authors come before implementers/speccers). @sandersdan, can you clarify what you mean by "resolving difficult questions about backpressure"?
Some more comments:
* UA does not need to maintain a queue of decoded `VideoFrames` that will be given to the Web App sometimes later after other frames.
If all apps need to write more or less the same code to have something working, it's a good sign that we can make the API better.
* VideoFrames are given to Web App as soon as possible which gives the Web App more time to do some processing to frames before displaying them.
This is real-time, the frame processing need to be shorter than a particular processing quantum (possibly with artificial added delay) for things to work well. Some frames might have more time, but other won't.
* Useful if the Web App is in full control of the synchronization and displaying the frames
I can't think of a use-case where using the frames in decode order makes sense. The web app is in control of synchronization and display of the frames anyways.
from webcodecs.
I think we're more or less in consensus here that inputs should be in decode order while outputs should be in presentation order. I'll leave this open for a bit more to hear objections or use cases for alternative modes.
@sandersdan assuming we maintain course, do you think we need clarification in the spec on this point?
from webcodecs.
We probably do, the decode algorithm is unclear about the relationship, and seems to imply a direct relationship between inputs and outputs. For example, I read it to say that if one chunk results in multiple output frames, then they must be adjacent; that's probably true but not something we should specify.
The actual output order is codec-specific, but could probably be worded in a generic way ("output in the presentation order specified by the codec"?). It's important not to imply involvement of the timestamps though.
from webcodecs.
triage note: marking 'editorial' as conclusion is to update spec wording to clarify that output is presentation order. if we later desire a mode to output in decode order, this can be achieved by extending the config.
from webcodecs.
@marcello3d, I filed https://bugs.webkit.org/show_bug.cgi?id=263901.
If you already have a test case, could you upload it add a link to it there?
from webcodecs.
yea, so I just tried adding logging to the sample on this repo and it reproduces:
here is my changeset: https://github.com/w3c/webcodecs/pull/734/files
I'm on Safari Version 17.0 (19616.1.27.211.1), macOS 14.0 (23A344)
It also reproduces on Safari Technology Preview Release 181 (Safari 17.4, WebKit 19618.1.3.1)
this also happens with h265 on Safari Technology Preview Release 181 (vp8, vp9, av1 have no errors):
![image](https://private-user-images.githubusercontent.com/21952/279158186-ec4758c6-b2d1-4bf5-a7df-63a168631b0b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTU4Mjg4MjgsIm5iZiI6MTcxNTgyODUyOCwicGF0aCI6Ii8yMTk1Mi8yNzkxNTgxODYtZWM0NzU4YzYtYjJkMS00YmY1LWE3ZGYtNjNhMTY4NjMxYjBiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTE2VDAzMDIwOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdmMTQ4ZWQ1YjVlMmMyMGM3ODU1NmIzNWQwNTA1ZGNkNTgyM2Q3MjQ3MTRhNTJjYjJjYTIzYzk5YmZjZTc5NWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.qwTdgn42kZxyxBLr63UjA1wTU6X1vEGuw8olaVvqdW0)
from webcodecs.
@youennf okay added some comments on bugzilla, let me know if there's anything else you need!
from webcodecs.
Related Issues (20)
- Support for AV1 switch frames HOT 5
- Fatal encoding/decoding error on Windows 10/11 HOT 2
- Why is isTypeSupported() promise-based? HOT 8
- Receiving Uncompressed Webcam Data without Browser Compression HOT 5
- After how many decode should the codec process the frames? HOT 10
- Clarify `reset()` behaviour when multiple things are being output HOT 3
- key-frame request handling when scalability mode is not L1T1 for encoder HOT 6
- Clarify the `Clone configuration` algorithm HOT 4
- Figure out what should happen to the unused bits in 10-bits and 12-bits pixel formats
- Rephrase non-normative uses of RFC2119 keywords HOT 1
- Web Audio API compatibility HOT 1
- Assign VideoFrame resource to [[resource reference]] in BufferSource constructor HOT 4
- VideoFrameBufferInit metadata field missing
- sourceWidthBytes from sampleWidth, not sampleHeight
- Vekil
- Sporadic build failures HOT 4
- Define scope for w3c candidate recommendation HOT 3
- Candidate Recommendation tracking issue
- VideoFrame copyTo() behavior with non-RGBA/RGBX/BGRA/BGRX formats HOT 1
- VideoPixelFormat enum values do not follow casing rule guidelines HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.