Recording a <canvas> and audio stream about webm-muxer HOT 12 CLOSED

Perfectly fine, take your time 😊 Just sent a coffee shipment 📦

from webm-muxer.

Awesome!! Thanks for the coffee shipment :)

from webm-muxer.

Thank you so much! I have a Ko-fi! https://ko-fi.com/vanilagy

I'll look into your issue later, I'm quite busy right now.

from webm-muxer.

Thank you so much for your generosity!

Alright so, I understand your use case - sounds like you want to build some sort of video cropper / editor. I think a good solution for the audio is using AudioContext.prototype.decodeAudioData on the raw video bytes to extract the audio (that works!) I wrote up some code to test it locally:

let response = await fetch('test.mp4'); // Get your video from wherever
let videoArrayBuffer = await response.arrayBuffer();
let audioBuffer = await audioContext.decodeAudioData(videoArrayBuffer);

// This was so I could hear the sound working
let node = audioContext.createBufferSource();
node.buffer = audioBuffer;
node.start();
node.connect(audioContext.destination);

// Define your "trimming extents"
let startTime = 3;
let endTime = 5.5;
let duration = endTime - startTime;

// You should ensure that startTime * sampleRate and endTime * sampleRate are integers, like:
// startTime = Math.round(startTime * sampleRate) / sampleRate

// Now we want to create an array packed with all of the data
let data = new Float32Array(duration * audioBuffer.sampleRate * audioBuffer.numberOfChannels);

for (let i = 0; i < audioBuffer.numberOfChannels; i++) {
    data.set(
        audioBuffer.getChannelData(i).subarray(startTime * audioBuffer.sampleRate, endTime * audioBuffer.sampleRate),
        i * duration * audioBuffer.sampleRate
    );
}

let audioData = new AudioData({
    format: 'f32-planar',
    sampleRate: audioBuffer.sampleRate,
    numberOfChannels: audioBuffer.numberOfChannels,
    numberOfFrames: data.length,
    timestamp: 0,
    data
});

// Pass audioData into audioEncoder.encode

See if this gets you somewhere! Gotta see if you get a "clicking" sound when the video ends or not (that usually happens when you randomly cut off an audio buffer). Although there's a good chance the compression codec will smooth that out. If it doesn't, you can try fading in the audio in the first 0.01 seconds or so (and fading out again) by multiplying the numbers in the Float32Array.

from webm-muxer.

Thanks! The audio is playing, so the first part is working. My test mp4 has 6 channels, and a sampleRate of 48000.

I copy pasted your code, and for testing set the start and end to 0 and the total video duration.

TypeError: Failed to construct 'AudioData': data is too small: needs 870580224 bytes, received 145096704.
    at http://localhost:3000/pintura-video/pinturavideo.js:9567:27
    at async process (http://localhost:3000/pintura/module/pintura.js:7278:21)
    at async http://localhost:3000/pintura/module/pintura.js:7489:33

As 870580224 is 6 times 145096704 I suspect somehow the data.set is not adding enough data?

from webm-muxer.

Aah no, I think I set numberOfFrames wrong. I think it's frames per channel. So it should be duration * sampleRate.

from webm-muxer.

Alright, yeah that works! Alright, back to it, will let you know if it works :)

from webm-muxer.

I'm very much out of my comfort zone with this 😅

const audioData = new AudioData({
    format: 'f32-planar',
    sampleRate: sampleRate,
    numberOfChannels: numberOfChannels,
    numberOfFrames: totalDuration * sampleRate,
    timestamp: 0,
    data,
});

audioEncoder = new AudioEncoder({
    output: (chunk, meta) => muxer.addAudioChunk(chunk, meta),
    error: (err) => {
        throw err;
    },
});

audioEncoder.configure({
    codec: 'opus',
    numberOfChannels: 2,
    sampleRate: sampleRate
});

audioEncoder.encode(audioData);

It throws

Input audio buffer is incompatible with codec parameters

I had to set number of channels to 2 on the encoder, it didn't accept 6.

from webm-muxer.

This is the muxer:

const muxer = new Muxer({
    target: new ArrayBufferTarget(),
    video: {
        codec: 'V_VP9',
        width,
        height,
        frameRate,
    },
    audio: {
                    codec: 'A_OPUS',
                    sampleRate: sampleRate,
                    numberOfChannels: 2
    },
    firstTimestampBehavior: 'offset',
});

from webm-muxer.

Well, your audio data has 6 channels tho! So you'll probably have to reduce it to stereo, picking two channels that you wanna keep. Is keeping the first 2 channels sufficient? (Should be front left and front right)

from webm-muxer.

@Vanilagy Ooh I was expecting it to know how to limit that? Like merge all the channels.

Trying with 2 channels now :)

Truly appreciate your help.

from webm-muxer.

It works 🎉 This is awesome, thank you so much.

from webm-muxer.