Comments (4)
In particular, when it comes to connecting WebCodec objects to MediaStream
, there need to be provision for dealing with multiple clock domains. This is especially important for audio, but also quite critical for video.
from webcodecs.
@padenot can you give toy example for how the problem manifests?
from webcodecs.
Here is a typical setup:
- Record a microphone/webcam (those are
MediaStreamTrack
s) using Web Codecs. This gives you a series of encoded packets with timestamps in the clock domain of the input device. In particular, there is going to be a very specific speed at which the frames are made available, that might not be the speed of the system clock. For example it's almost guaranteed that 1s second of audio captures at 44100Hz (i.e. 44100 frames) was captured in a bit less or more than 1s on the system clock: this is a first clock drift. - Decode it using WebCodecs, and play it out using another
MediaStreamTrack
. Depending on the output device (screen/speakers), it will probably be clocked differently than the input device. This means that, say, 30 frames captured at 30fps on the webcam is not going to be played out in exactly 1 second on a screen that is 60fps: the clock domain of the webcam and the screen are different, and also different compared to the system clock, a webcam frame will eventually be either dropped or displayed for 3 screen frames, because of this second drift.
For video, simply repeating the frame is the approach usually taken (not always). For audio, it is absolutely essential to do adaptive resampling, or else problems will appear: either latency buildups if the drift goes in one direction, or gaps (translating into very problematic audio glitches) if the drift goes in the other direction.
from webcodecs.
Most of the concern about playout via MediaStreamTrack is now out of scope for WebCodecs, and might instead go to https://github.com/w3c/mediacapture-transform/issues. Issue w3c/mediacapture-transform#35 might be a good a starting point.
For sync generally, I think resolution here is the same as discussion in #39 (comment). In short, apps can overcome skew concerns by letting audio drive the clock (as a function of its output samples and sample rate), and then paint video frames to match wherever audio is at.
Closing as I don't think there's anything else in scope for WC here, but please reopen if I've overlooked anything.
from webcodecs.
Related Issues (20)
- Web Audio API compatibility HOT 1
- Assign VideoFrame resource to [[resource reference]] in BufferSource constructor HOT 4
- VideoFrameBufferInit metadata field missing
- sourceWidthBytes from sampleWidth, not sampleHeight
- Vekil
- Sporadic build failures HOT 4
- Define scope for w3c candidate recommendation HOT 3
- Candidate Recommendation tracking issue
- VideoFrame copyTo() behavior with non-RGBA/RGBX/BGRA/BGRX formats HOT 1
- VideoPixelFormat enum values do not follow casing rule guidelines HOT 2
- EncodedAudioChunkInit should probably also support AllowSharedBufferSource HOT 1
- Issue
- numberOfChannels/sampleRate check in AudioDecoderConfig/AudioEncoderConfig HOT 1
- Under what conditions should the ImageDecoder [[completed promise]] and ImageTrackList [[ready promise]] be rejected? HOT 3
- Spec differentiates between ImageDecoder initialization with unsupported non-image MIME type and an unsupported image MIME type HOT 2
- how to config videoEncoder that output frame is annexB not avcC? HOT 1
- `ImageDecoder` is `[SecureContext]` but related interfaces aren't HOT 2
- `ImageBufferSource` should be `[AllowShared]` HOT 1
- Announcement: Background Segmentation metadata entry to WebCodecs VideoFrame Metadata Registry HOT 2
- Why is the audio encoder output timestamp different from the input timestamp
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.