Comments (13)
Note that worker integration is discussed in the explainer under "Execution environment." Might be worth expanding that section to be a bit more concrete though.
from webcodecs.
This is currently planned. The spec doesn't reflect it yet, but it's on the short list of things to add in Chromium. There's currently discussion on the WebAudio spec, which needs to be modified to expose AudioBuffer (tracking bug for Chromium).
from webcodecs.
The first question is whether WebCodecs should work in a worker at all. The current implementation in Chromium does not, as the existing media stack in Chromium assumes there is a frame. I hope to support dedicated workers in the future.
There is a separate question about background and shared workers, which should be a separate issue.
Options for getting data in and out:
TransferableStreams
: in theory, all streams can be transferred. Support for atransferList
is not in the spec yet though, and it's possible that WebCodecs could be specified in an incompatible way. Such a design precludes using an single object to represent a decode frame, which is annoying. I do not know if this feature is under active development.MediaStreamTrack
: there has not been much work onMediaStream
integration yet, but being able to transfer aMediaStream
would directly solve the problem. This is probably out of scope for the first version of WebCodecs.- Transferable encoded and decoded chunks: would enable
postMessage()
, but this seems like a worse API than Streams. Also requires having an explicit type, so we won't be able to useIDLDictionary
. - Manual transfer: It should be possible to deconstruct the input/output chunks, transfer them via
postMessage()
, and reconstitute them on the other side. This should work with Chromium's VideoDecoder implementation as soon as I implement ImageBitmap support.
from webcodecs.
My main use case is to be able to do video manipulation on gUM
video track on web worker
Currently I can captureStream
on an offscreencanvas
, but I cannot easily grab the video contents . I would need to do a video.requestAnimationFrame
get the createImageBitmap
and transfer it to the web corker. If i could transfer the mediatrack (or just the reader/writter) to the web worker it would be fine for me.
from webcodecs.
@murillo128 In WEBRTC WG, we have discussed whether access to raw video is best provided by adding a method on the track, or on the sender/receiver (the approach currently taken by
Insertable Streams). In a number of use cases such as "funny hats" or "machine learning", the raw video processing would probably be done on the sender side, in which case adding a track method might work. The major question is how to make this efficient enough. Insertable Streams combined with web workers and transferrable streams seems performant enough to enable scenarios involving encoded frames such as e2e encryption and Virtual Reality Gaming. However, raw video frames are much larger and so a track method providing a stream of raw video frames might not provide enough performance even with web workers and transferrable streams.
from webcodecs.
I'd like to add my voice to the requests for web worker support.
We currently have a decode and render pipeline that happens entirely in a worker (using WebAssembly and an offscreen canvas). It'd be a real shame, and rather perverse, to have to shuffle samples to the main thread to get them decoded, then back to the worker to render them.
from webcodecs.
This issue hasn't been updated in a while, but Chromium does plan on supporting WebCodecs in workers. In fact, VideoDecoder
should work on the latest Canary (crbug.com/1138676), and VideoEncoder
will be following relatively soon (crbug.com/1094169).
Transferring MediaStreamTrack
directly is still currently out of scope, as there are spec implications beyond the simple technical feasability. However, the combination of a stream based track readers and native stream transfer optimizations will allow frames to be delivered onto a worker thread directly, without hopping onto the main thread.
The TrackProcessor
and TrackGenerator
will ship on the same timeline as WebCodecs. Last I heard, the stream transfer optimizations were planned for Q1 2020.
from webcodecs.
Note that the support above is only for dedicated workers and that there are still open questions about shared workers and service workers.
from webcodecs.
Having access to VideoDecoder from a dedicated worker is exactly what I need, so I'm happy!
from webcodecs.
What about AudioDecoder? I notice that it outputs AudioBuffers, but these aren't exposed to Worker, nor are they transferable.
from webcodecs.
The AudioDecoder support should have been added at the same time as VideoDecoder.
It landed in 89.0.4332.2:
https://chromium.googlesource.com/chromium/src/+/acbd2eae0763947f78244814c2bdd5628778813d
It was merged back to 88.0.4324.29, so it should be available in Beta:
https://chromium.googlesource.com/chromium/src.git/+/a9b34930a44749da8e6ba5da4cf1df9daa1c3a94
I guess you are saying that underneath our worker-exposed AudioFrame lies the not-worker-exposed AudioBuffer, and therefore AudioWorker is not useable in workers? That is a bug, I've opened crbug.com/1160580.
from webcodecs.
Is it possible to make AudioFrame
transferable?
AudioDecoder
and AudioFrame
are exposed to DedicatedWorker
, but Web Audio APIs(not only AudioBuffer
, but also other APIs) are exposed to Window
only. At this moment, we need to getChannelData
in worker and create a new AudioBuffer
in the main thread. Then decoded audio frames can be processed by Web Audio.
from webcodecs.
I think we can close this, but let me give a overview of where we landed.
We now support all decoder/encoder interfaces in DedicatedWorker. I think WebAudio WG is near consensus to expose AudioBuffer to workers as well.
VideoFrame and AudioFrame will be transferable. Spec work tracked in #129.
To transfer frames from a MediaStream, the idea is to setup https://w3c.github.io/mediacapture-insertable-streams/#track-processor in the main window, but transfer it's readable
stream to the worker for input into the encoder. Under the hood UA's may will optimize that transfer such that frames arriving from the camera do not create main-thread contention in the original Window context.
Support for Service and Shared workers is not planned at this point.
Feel free to re-open or file new sub issues if more to discuss.
from webcodecs.
Related Issues (20)
- Support for AV1 switch frames HOT 5
- Fatal encoding/decoding error on Windows 10/11 HOT 2
- Why is isTypeSupported() promise-based? HOT 8
- Receiving Uncompressed Webcam Data without Browser Compression HOT 5
- After how many decode should the codec process the frames? HOT 10
- Clarify `reset()` behaviour when multiple things are being output HOT 3
- key-frame request handling when scalability mode is not L1T1 for encoder HOT 6
- Clarify the `Clone configuration` algorithm HOT 4
- Figure out what should happen to the unused bits in 10-bits and 12-bits pixel formats
- Rephrase non-normative uses of RFC2119 keywords HOT 1
- Web Audio API compatibility HOT 1
- Assign VideoFrame resource to [[resource reference]] in BufferSource constructor HOT 4
- VideoFrameBufferInit metadata field missing
- sourceWidthBytes from sampleWidth, not sampleHeight
- Vekil
- Sporadic build failures HOT 4
- Define scope for w3c candidate recommendation HOT 3
- Candidate Recommendation tracking issue
- VideoFrame copyTo() behavior with non-RGBA/RGBX/BGRA/BGRX formats HOT 1
- VideoPixelFormat enum values do not follow casing rule guidelines HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.