Comments (7)
Time as an integer fraction of seconds (ie. an integer multiple of a timebase) is indeed the most common format for interchanging media timestamps. There isn't a Rational type already though, so the ergonomics are not great in JS.
from webcodecs.
I'm not sure what API we should have for timestamps, because the options can differ between codecs and codec implementations. Typically media containers provide timestamps, and sometimes durations also, but these can be in the bitstream as well. Reordering is always signaled in the bitstream AFAIK.
Some codec implementations allow us to perfectly correspond frames with encoded chunks, while others give special meaning to timestamps. (Eg. some Android MediaCodec video decoder implementations will reorder frames to match the timestamps provided with the input chunks.) We can usually work around this by disabling post-processing filters and generating artificial timestamps in the browser implementation.
There exist codecs without a reliable encoded chunk to frame mapping. (Eg. VPx without superframes has chunks that map to no frame.) We can probably avoid this by requiring that each encoded chunk corresponds to exactly one frame as a matter of conformance, although I still expect WebCodecs implementations to sometimes drop frames for unspecified reasons.
If we require a 1:1 mapping, then we could allow apps to specify an opaque object that flows through the encode/decode process, which could be a timestamp or some other structure. We could also determine codec-by-codec whether implementations should add additional annotations for in-band metadata that they find.
Edit: The above applies to audio and video, but not images. We don't expect apps to parse their own timestamps from animations.
from webcodecs.
You probably also want to use a high res timestamp for actually timestamps, no?
from webcodecs.
That's an open question. We (Chrome implementors) are using an integer type so that we can make guarantees about exact preservation through our whole media stack, but some aspects of that are Chrome-specific and we'll need to make sure all browsers can meet the requirements equally.
DOMHighResTimeStamp is a double type in milliseconds. It could potentially meet our needs as well, would be easier to use correctly from JS, but may be less nice for WASM users.
from webcodecs.
Re: duration use cases, one good one is ImageDecoder for animated images. The ImageDecoder will emit VideoFrames, and duration is needed to know how long the last frame is presented.
from webcodecs.
In terms of compatibility, how about adopting timebase
.
timebase
figures numbers of clock ticks in 1sec as integer. So it achieve completely to preserve and keep compatibility if timebase==1
.
And it may be needed for muxing into some format.
from webcodecs.
Closing this issue as the timestamp type question is now tracked separately.
Conclusions on duration, lets keep it, but also keep it as optional.
Duration isn't required by video decoders and encoders, but it is potentially a nice-to-have passthrough attribute for authors. For these uses its fine to keep it and leave it as optional.
Duration is however very useful for VideoFrames output by ImageDecoder in cases where the image is animated and you need to know how long to present the final frame before looping. For this use, the ImageDecoder will set it accordingly.
from webcodecs.
Related Issues (20)
- Define scope for w3c candidate recommendation HOT 3
- Candidate Recommendation tracking issue
- VideoFrame copyTo() behavior with non-RGBA/RGBX/BGRA/BGRX formats HOT 1
- VideoPixelFormat enum values do not follow casing rule guidelines HOT 2
- EncodedAudioChunkInit should probably also support AllowSharedBufferSource HOT 1
- Issue
- numberOfChannels/sampleRate check in AudioDecoderConfig/AudioEncoderConfig HOT 1
- Under what conditions should the ImageDecoder [[completed promise]] and ImageTrackList [[ready promise]] be rejected? HOT 3
- Spec differentiates between ImageDecoder initialization with unsupported non-image MIME type and an unsupported image MIME type HOT 2
- how to config videoEncoder that output frame is annexB not avcC? HOT 1
- `ImageDecoder` is `[SecureContext]` but related interfaces aren't HOT 2
- `ImageBufferSource` should be `[AllowShared]` HOT 1
- Announcement: Background Segmentation metadata entry to WebCodecs VideoFrame Metadata Registry HOT 2
- Why is the audio encoder output timestamp different from the input timestamp
- Video encoder bitrate control in case frame rate is not provided
- Video frames on demand HOT 1
- Error code: SIGSEGV HOT 3
- VideoFrame.timestamp and EncodedVideoChunk.timestamp HOT 6
- video encoder memory leaks HOT 1
- Include some frame info with `VideoDecoder` Errors? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.