Comments (17)
Having access to a decoded plane would be useful for a use case like super-sampling via WebGL.
I'm thinking of a complete flow where it's possible to put a shader based preprocessor to super-res the image, route this back to the WebCodecs exposed web encoder and generate a segment of media ready for MSE to take over for regular playback.
from webcodecs.
The proposal from Jie @ Intel is now public
https://docs.google.com/document/d/1ULhbTWlS_vz3AUKAZXNa0TSlwtFB7ItuiFhlqseLUTU/edit
Chromium bug where prototyping work is tracked is here
https://bugs.chromium.org/p/chromium/issues/detail?id=1142754
FYI, there is also a proposal(s) for accessing planes for GPU backed VideoFrames using WebGPU
https://docs.google.com/document/d/1MmLTO7gmOBp9fIccrlbqYcGOEydxYKbQwOYBY6gxZ5M/edit#
from webcodecs.
There is a proposal in progress for a WebGL extension that would enable this, hopefully it will be available publicly soon.
from webcodecs.
Any news on this? There no information anywhere on what happens when one does:
const internalFormat = gl.RGBA;
const srcFormat = /* something ?! use RGBA because a conversion is implied? How is the color space known? */;
const srcType = gl.UNSIGNED_BYTE;
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.texImage2D(gl.TEXTURE_2D, 0, internalFormat, srcFormat, srcType, videoFrame);
where texture
is a texture already initialized and videoFrame
a VideoFrame
object.
from webcodecs.
I think @dalecurtis html PR intends to address this
whatwg/html#6589
Adding VideoFrame as a CanvasImageSource allows drawImage(), createImageBitmap(), and texImage() to interoperate with VideoFrames.
But perhaps additional updates are needed? Where is the behavior defined for earlier types of CanvasImageSource?
For the webGL extensions described earlier in this issue, that work is paused for a bit to focus on the webGPU strategy instead. Updates in the crbug describe a bit more
https://bugs.chromium.org/p/chromium/issues/detail?id=1142754#c21
https://bugs.chromium.org/p/chromium/issues/detail?id=1142754#c45
from webcodecs.
Hmm, these are good questions, and they fall right into an area of GL that I've never fully understood. From a logical standpoint, srcFormat
and srcType
aren't meaningful for a WebGL CanvasImageSource
; the source buffer doesn't need to be described because the CanvasImageSource
knows, and in practice it's not usually a block of memory like GL texImage2D()
expects.
My mental model is that this acts like a texture-to-texture copy; that is, sample at each coordinate of videoFrame
and write that sample to the texture in whatever format the destination texture is in.
In Chrome's implementation, the CanvasImageSource
for VideoFrame
handles color spaces the same way as the <video>
rendering path, which I believe looks like sRGB in WebGL. I'm not actually sure if there is a way to get linear RGB by the texImage2D()
path, but that sure would be useful. In the future, when the WebGL implementation supports HDR/WCG, there will be other intermediate color spaces available to the implementation.
(aside: I believe Chris means to say "WebGPU strategy" above.)
from webcodecs.
(aside: I believe Chris means to say "WebGPU strategy" above.)
Correct! I've just made that edit.
from webcodecs.
@kenrussell @domenic to opine.
Background: for now WebCodecs implicitly assumes sRGB. We plan to support more (#47). But, looking just at current support, should the spec (or the HTML PR) say more about how color is handled by canvas.drawImage() and textImage*() APIs?
from webcodecs.
I think @ccameron-chromium is the expert you need. In whatwg/html#6562 he added color space support to HTML canvas, so we need to at least be compatible with what's there. It looks like the spec currently says
When drawing content to a 2D context, all inputs must be converted to the context's color space before drawing.
and
There do not exist any inputs to a 2D context for which the color space is undefined. The color space for CSS colors is defined in CSS Color. The color space for images that specify no color profile information is assumed to be 'srgb', as specified in the Color Spaces of Untagged Colors section of CSS Color.
which might cover you?
from webcodecs.
My inclination about adding Y/U/V plane access is that if it is to be added, one should request something to effect of "I want the Rec709 Y plane", specifying both the color space and the plane. This is very well-defined, and doesn't depend on any implementation details.
If the video is Rec709, then one can end up in the optimal situation of sampling the Y plane that came out of the decoder. Or one may be sampling a copy that was made (but one that was made by just copying the one plane, which is more efficient). Or perhaps this is on a platform with some sort of workaround where the video came out as something totally different (say, 444, or RGBA), because of some bizarre bug, and one ends up getting a color-converted copy.
This requires that a complementary API be present which can tell the application "this frame is Rec709". That enables the application to select the Y plane format that is going to be the most efficient. (And if the application is sloppy and always says "I want Rec709 Y", and gets a video that is something totally different, then they suffer in terms of performance but not correctness).
from webcodecs.
Triage note: marking 'extension' as the proposed WebGL API adds new interfaces without breaking WebCodecs.
from webcodecs.
@Djuffin @padenot Can we close this issue?
from webcodecs.
I've proposed that we talk about this at TPAC, in a joint meeting with webgl/webgpu folks. There has been demand from the industry some time ago.
from webcodecs.
@padenot would you email me at kbr at chromium dot org? If we're to advance this feature then I think it would be most productive to do it in the WebGL and WebGPU working group meetings rather than waiting for TPAC. WebGL is hosted at Khronos rather than the W3C and WG members are not currently planning to attend TPAC.
Note that WebGPU's functionality of importing external textures provides a zero-copy way to bring YUV VideoFrames into WebGPU shaders. I wonder whether that path addresses this use case.
from webcodecs.
I'm also interested in this from the encoding perspective - writing a VideoFrame currently requires converting to a Uint8Array from gl.readPixels
rather than passing a Texture or Framebuffer.
Up until this point I've used a MediaStreamTrackProcessor
on the canvas which is generally performant, but I'd like to be able to write the individual passes from a Framebuffer to my VideoFrame, and currently reading from the framebuffer is very slow.
Happy to email separately about this @kenrussell.
from webcodecs.
@akre54 you can construct a VideoFrame
from either an HTMLCanvasElement
or OffscreenCanvas
, using WebGL to render to either of those surfaces. The data will remain on the GPU, though a copy is necessary.
It's not possible to render directly to a VideoFrame
from WebGL, nor to copy individual WebGLTextures
into a VideoFrame
. The browser's media stack and WebGL implementation may be hosted on different graphics APIs (and, in Chrome, currently are, on multiple platforms), so the surfaces that interoperate between the two APIs have to be allocated specially. This is the case for all of the input types to VideoFrame
per the MDN documentation, such as HTMLCanvasElement
and OffscreenCanvas
.
from webcodecs.
Ah that's a bummer but thanks for the info.
Up until this point we've been using the canvas for capturing the stream but the goal is to take the separate shader passes and write them out individually with an alpha channel, similar to Blender's render passes, instead of one unified composited image. I'll do some experimentation to see if drawing the individual layers and capturing the ReadableStream value is faster than writing to Framebuffers - I suspect it might be.
from webcodecs.
Related Issues (20)
- numberOfChannels/sampleRate check in AudioDecoderConfig/AudioEncoderConfig HOT 1
- Under what conditions should the ImageDecoder [[completed promise]] and ImageTrackList [[ready promise]] be rejected? HOT 3
- Spec differentiates between ImageDecoder initialization with unsupported non-image MIME type and an unsupported image MIME type HOT 2
- how to config videoEncoder that output frame is annexB not avcC? HOT 1
- `ImageDecoder` is `[SecureContext]` but related interfaces aren't HOT 2
- `ImageBufferSource` should be `[AllowShared]` HOT 1
- Announcement: Background Segmentation metadata entry to WebCodecs VideoFrame Metadata Registry HOT 2
- Why is the audio encoder output timestamp different from the input timestamp
- Video encoder bitrate control in case frame rate is not provided
- Video frames on demand HOT 1
- Error code: SIGSEGV HOT 3
- VideoFrame.timestamp and EncodedVideoChunk.timestamp HOT 6
- video encoder memory leaks HOT 1
- Include some frame info with `VideoDecoder` Errors? HOT 2
- VideoFrameMetadata is defined in WebCodecs and videoFrameCallback HOT 2
- Wrong chroma values in videoFrame.copyTo() on Intel MacOS HOT 1
- Attempting to configure an encoder with a bitrate of 0 should fail HOT 1
- display size is not set in Convert to RGB frame algorithm
- copyTo from planar to interleave should copy / convert all channels HOT 2
- Do Linear PCM AudioDecoders not work? HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.