mbebenita / broadway Goto Github PK

At times the video moves either too fast or too slow,
In order to provide a smooth user experience displaying frames should be done in a constant interval (1000/FPS)

Hey Michael,

Awesome work, but noticed after forking that a couple of the js (avc-codec.js, maybe others) files were minified, Any chance we can get the unminified source into the repository? Will make it easier for others to make work on this awesome proj.

Steve

is there any way to tune performance?
i wonder if the filter settings or some other configuration could impact performance.
i am aware of the fact that this is no mplayer but i want to take every speed option i can get.

if anyone could be so kind to have a look at my configuration and tell me if i could do something better i would be happy ;)

https://github.com/soliton4/nodeMirror/blob/master/src/avc/Wgt.js

I am trying to get broadway to work with a HTTP live video stream coming from VLC. My goal is to get a webcam feed displayed in the browser without any plugins.

Now I am aware that XHR does not support progressive download and as such does not support live streams. Is there a way around this? I am willing to contribute some development time to get live streaming to work with Broadway as I think it could be very valuable for things like TV streams or videoconferencing solutions. However I need some pointers:
1.) Is live streaming possible at all with Broadway?
2.) Is there an alternative to XHR that lets me do live streaming? I am thinking of using WebSocket and creating a custom streaming server that sends a video stream using it.
3.) If I do use WebSocket, which areas of the Broadway code do I need to modify in order for it to get the video data?

Thanks!
Ben

Hello guys

Have you ever tried to have a video encoder in JavaScript instead? I'm trying to do one with emscripten + avconv but am struggling.

Cheers
Michael

hi @mbebenita
broadway refuses to render the output of this:

ffmpeg -y -f x11grab -r 30 -s 1920x1080 -i :0+0,0  -b:a 2M -bt 4M -vcodec libx264 -pass 1 -coder 0 -bf 0 -flags -loop -wpredp 0 -an -f h264 -s 1920x1080 screentest2.h264

if i do the same thing with avconv it works

also strange: regular file transcoding works fine with ffmpeg + broadway

do you have any idea / hint why?

so i have a brand new intel 4770s
4 cores - 8 logical core units.
but the performance is very poor because only one core is used for rendering.
i get only 10 fps @ 1080p

can i easily utilize the other cores? 4 threads would be optimum for me.
if i can get to 25 fps i am happy

i am really puzzled how to render the frames without webgl. is there any fallback code?

Hi, I have an issue in Chrome and Firefox when decoding frames with webgl set to "auto" or true.

My code:

script src="broadway/Player.js"/script
script src="broadway/Decoder.js"/script
script src="broadway/YUVWebGLCanvas.js"/script

script
var player=new Player({
useWorker: true,
workerFile: "broadway/Decoder.js",
webgl: "auto"
});
document.body.appendChild(player.canvas);

while(1) {
---Feed h264 frames in_h264 buffer---
player.decode(hit_h264);
}
script

Chrome says exception is in Player.js line 119:
self.webGLCanvas = new YUVWebGLCanvas(self.canvas, self._size);

Firefox says exception "TypeError: YUVWebGLCanvas is not a constructor" on Player.js:119:27

When I use "webgl:false" option video plays ok.
Also when "webgl:"auto" or true, canvas resize to video size, but when "webgl:false" it's just stays as it is by default (does not adapt size to video frame size).

Please advise if someone experienced the same?

Hi All,

I am trying to compile FFMPEG scaling library of android but i am not able to compile.I want to scale the video.Currently it is taking the video height and width but i want to use it with my size.

Can anybody help me in that?

Thanks,
Pranav Shah

hi
the live demo does not seem to work.
i dont see anything happening and i have 0 network traffic.
is there a working demo anywhere i can use?

i basically want to display the output from x264 binary. is this possible?
is this project still maintained?

In mac mini safari broadway ,performance is not so good.I tried demo fps come as ~3-5 fps.On same mac chrome is working as expected(>30 fps for demo).In safari after decoding certain number of frames screen corruption start to happen and it is consistent.

With -O2 and setting assertions in Decoder/make.py, by adding the line:

 '-s', 'ASSERTIONS=1',

in emcc_args, the compiled code won't work (tested on FF 34.0).

When the console is enabled, I get this log:

Successfully compiled asm.js code (total compilation time 112ms; not stored in cache) avc-codec.js
"increasing TOTAL_MEMORY to 67108864 to be compliant with the asm.js spec" avc-codec.js:4
"run() called, but dependencies remain, so not running" avc-codec.js:4
"Assertion failed: you need to wait for the runtime to be ready (e.g. wait for main() to be called)" avc-codec.js:4
uncaught exception: abort() at Ja@file:///home/stefano/src/Broadway/Player/avc-codec.js:26:127
A@file:///home/stefano/src/Broadway/Player/avc-codec.js:165:58
u@file:///home/stefano/src/Broadway/Player/avc-codec.js:12:211
$._get_h264bsdClip@file:///home/stefano/src/Broadway/Player/avc-codec.js:151:302
@file:///home/stefano/src/Broadway/Player/avc.js:11:16

The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol. storyDemo.html
TypeError: Avc is not a constructor

The same code will work if compiled with -O1.

By setting the workers flag to "true" storyDemo.html is not working at all (not generating any content).
Tested with Firefox 34.0 and 30.0 on Linux, with the current master e6d1c4dd9275db323c793f3ca5c2843cb63159fd

This is the log I get from the Firefox console:

Successfully compiled asm.js code (total compilation time 109ms; not stored in cache) avc-codec.js
"increasing TOTAL_MEMORY to 67108864 to be compliant with the asm.js spec" avc-codec.js:4
The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol. storyDemo.html
: Successfully compiled asm.js code (total compilation time 212ms; stored in cache) avc-codec.js
ReferenceError: Module is not defined avc.js:5
"MP4Player::readAll()" mp4.js:825
"Parsed stream in 6 ms" mp4.js:457
"MP4Player::readAll(), length: 9064588" mp4.js:831
"creatingTextures: size: (640, 360)"

I think the main problem is related to line:

ReferenceError: Module is not defined avc.js:5

Tested on various platforms, always getting the same result.

My plan is to create H.264 encoded frames in an android and send them via websockets to a web browser and decode them using broadway. Note : It's not an mp4 file but H.264 frames.
Can anyone guide me on decoding and rendering them after I have received them ?
A code snippet or something would do.

Thanks.

Hi,

I have h264 main profile encoded mp4 file.When i use this file decode is not able to decode the video on the browser.If i use the Mp4 file with H264 base line profile encoded video.Decode is able to decode the video.

Is there anything special to provide ?

Help please....

Thanks,

Pranav Shah

Hello,

1. How do I use Decoder to override onPictureDecoded?
2. Do I have to include Decoder.js only? Or all other js files are required as well?
3. What input data Decoder expect? Is preloaded binary mp4 file ok?

Can't get it to work and that is why I decided to write these questions here..
Thanks for your efforts in Broadway.js

Every time I create more than one Avc object and then try to pass video data to it, the Javascript console spits out an error that says:

Uncaught RangeError: Maximum call stack size exceeded
    OptimizedFilterVerLumaEdge
    OptimizedFilterVerLumaEdge

I want to be able to decode multiple video streams at once within the same webpage. Is there a better way to do this?

hi broadway.js i am proud to announce i solved the worker thread problem ;)

pls refer to -> http://updates.html5rocks.com/2011/12/Transferable-Objects-Lightning-Fast

instead of cloning the uint8array you create a local copy in the worker thread and then transfer ownership like shown in the link.
the performance boost is unbelievable.

i also created a multithreading implementation for my application. but it requires that 4 seperate h264 streams exist.
using this together with the new worker threads i am over 40fps at 720p. ;))))))))

if you wanna try it out use the latest bleeding edge nodemirror commit.

the worker.js implementation in broadway.js will not work out of the box.
if you solve the window dependency i will provide a pull request.
otherwise you can look at my changes and feel free to copy them.

thank you
me

There have been quite some changes to emscripten lately, a good summary can found here: http://mozakai.blogspot.de/2012/11/emscripten-compiler-upgrades.html

I could not get the current master of emscripten to build a working Broadway.js without disabling every optimization: closure-compiler breaks the modules patching, and even -O1 only results in a grey canvas (no error messages in the console).

Has anybody gotten this to work already? I'd love to put those the new optimizations to work.

undefined:60
throw msg;
^
Assertion failed: asm.js heap must be power of 2
Traceback (most recent call last):
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/emscripten.py", line 860, in
_main(environ=os.environ)
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/emscripten.py", line 848, in _main
temp_files.run_and_clean(lambda: main(
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/tools/tempfiles.py", line 38, in run_and_clean
return func()
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/emscripten.py", line 856, in
DEBUG_CACHE=DEBUG_CACHE,
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/emscripten.py", line 742, in main
jcache=jcache, temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE)
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/emscripten.py", line 184, in emscript
assert '//FORWARDED_DATA:' in out, 'Did not receive forwarded data in pre output - process failed?'
AssertionError: Did not receive forwarded data in pre output - process failed?
Traceback (most recent call last):
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/emcc", line 1527, in
final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/tools/shared.py", line 1159, in emscripten
assert os.path.exists(filename + '.o.js') and len(open(filename + '.o.js', 'r').read()) > 0, 'Emscripten failed to generate .js: ' + str(compiler_output)
AssertionError: Emscripten failed to generate .js:
Traceback (most recent call last):
File "make.py", line 77, in
emscripten.Building.emcc('avc.bc', emcc_args, os.path.join(JS_DIR, 'avc.js'))
File "/Users/a-jie/Downloads/emsdk_portable/emscripten/1.5.6/tools/shared.py", line 1137, in emcc
assert os.path.exists(output_filename), 'emcc could not create output file: ' + output_filename
AssertionError: emcc could not create output file: js/avc.js

I copied the repository but I am having issues when I try to load in an external video, I get a CORS error from Broadway:

XMLHttpRequest cannot load http://www.w3schools.com/html/mov_bbb.mp4. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8080' is therefore not allowed access.

However, I am able to play that same URL (http://www.w3schools.com/html/mov_bbb.mp4) when I just use a video tag. I am using storyDemo.html and here is my code:

// this will throw a CORS error
<div class="broadway" src="http://www.w3schools.com/html/mov_bbb.mp4" width="640" height="360" style="float: left; position: relative;" workers="false" render="true" webgl="auto"></div>

// this will play as expected
<video src="http://www.w3schools.com/html/mov_bbb.mp4"></video>

Is there a reason why an external video would get a CORS error from Broadway but not when I just try to play the video as is?

Can I use this library to compress video before upload? It doesn't appear so after looking at the source, but I thought I would ask.

Thanks.

Hi,
Is this correct that webGL need here only to convert YUV image to rgb, right?
So is it possible to convert it to rgb without webGL, by this it will be possible to draw image in 2D canvas, right?

Thanks.

    this.webGLCanvas.YTexture.fill(buffer.subarray(0, lumaSize));
    this.webGLCanvas.UTexture.fill(buffer.subarray(lumaSize, lumaSize + chromaSize));
    this.webGLCanvas.VTexture.fill(buffer.subarray(lumaSize + chromaSize, lumaSize + 2 * chromaSize));
    this.webGLCanvas.drawScene();

When running $ python make.py i get the following errors...

emcc avc.bc -> js/avc.js
Traceback (most recent call last):
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/emscripten.py", line 1656, in <module>
    _main(environ=os.environ)
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/emscripten.py", line 1644, in _main
    temp_files.run_and_clean(lambda: main(
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/tools/tempfiles.py", line 39, in run_and_clean
    return func()
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/emscripten.py", line 1652, in <lambda>
    DEBUG_CACHE=DEBUG_CACHE,
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/emscripten.py", line 1536, in main
    shared.Building.ensure_struct_info(struct_info)
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/tools/shared.py", line 1767, in ensure_struct_info
    gen_struct_info.main(['-qo', info_path, path_from_root('src/struct_info.json')])
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/tools/gen_struct_info.py", line 505, in main
    struct_info = inspect_code(header_files, cpp_opts, structs, defines)
  File "/mnt/nethome/users/joe.blogs/Downloads/emsdk_portable/emscripten/master/tools/gen_struct_info.py", line 380, in inspect_code
    info = subprocess.check_output([shared.LLVM_INTERPRETER, bin_file[1]]).splitlines()
AttributeError: 'module' object has no attribute 'check_output'
copying js/avc.js -> ../Player/avc-codec.js

Any ideas?

Building appears to fail now due to missing a file called "yuv2rgb.h"

I thought I'd put this here to let people know I'm working (in my branch) on changing mp4.js to support progressive downloading of a h.264/ac file from a remote server, as currently the whole file needs to be loaded. Live streaming will not be in scope however. Let me know if you have any questions! Once completed, I'll throw up a pull request.

Latest FF, latest Ubuntu. 3fps in demo (main thread), same for other demos. Really slow down my PC.

http://bkw.github.io/Broadway/storyDemo.html

i believe i am not the only one stupid enough to not click on the screen and wonder why it doesnt work

a "click me" message or something similar would be helpful. also it should not be much effort to implement ;)

thx
great project

is there anyway to get a png stream of the video being decoded using broadway, a silly way is to call "toDataURL" on the canvas element every X ms. is it possible?

I'm using Aurora.js (and previously JSMad) to decode audio to display as a visual waveform. I'd really like to do the same thing with audio tracks out of a video file, and I figure I can do that by using mp4.js to parse a video file and then just extract the audio data, but I can't quite figure out how to do that. So, how would one go about accessing any audio data in an mp4 file?

Mobile browsers do not support the autoplay feature of HTML5 video.

<video src="myVideo.mp4" autoplay>

This will autoplay using non-mobile browsers but not on mobile browsers.

Is it possible to decode the MP4 and then autoplay it using Broadway?

It would be super cool if we could use openH264 for decoding, since it supports a lot more feature than Android's limited software decoder.

Is it possible to add a click event to start the video. There's broadway.play() but it doesn't seem to work. Is there any good documentation explaining the decoder and how to use it. I'm kinda new to this would really appreciate some help.

How? From where?

I wanna track the time of the frame being displayed in the canvas? Also is ti possible to limit the number of frames rendered per second?

what is this file used for?

https://github.com/mbebenita/Broadway/blob/master/Player/.project

The demo pages give a 404

Hi mbebenita ,
I known that the HTML5 video tag is very excellent for static video files. but I don't known whether the HTML5 video tag can be used to stream live video now?
Waiting for your help!
Thanks!!!

I'm compiling the code with the latest version of emscripten (build of the day) and the storyDemo.html demo is displaying gray frames. The onPictureDecoded callback is called regularly.

If I compile the decoder code with -O1 or -O0 (by editing the Decoder/make.py script) the code is working correctly. When compiled with -O2 the average framerate increases with respect to -O1, but this may be due to the fact that the decoding code is not called.

Tested with Firefox 30.0 on Linux mint and with Firefox 34.0 on Windows 7.

I get this error

[03:18:11.181] Use of attributes' specified attribute is deprecated. It always returns true. @ http://mbebenita.github.com/Broadway/broadway.html
[03:18:11.197] Use of XMLHttpRequest's responseType attribute is no longer supported in the synchronous mode in window context. @ http://mbebenita.github.com/Broadway/broadway.html
[03:18:11.207] uncaught exception: [Exception... "A parameter or an operation is not supported by the underlying object"  code: "15" nsresult: "0x8053000f (NS_ERROR_DOM_INVALID_ACCESS_ERR)"  location: "http://mbebenita.github.com/Broadway/broadway.html Line: 56"]

Hi,

I am trying to see if its possible to add event handlers on the canvas such that will allow for canvas manipulations. For instance I have implemented zooming and panning, using the mousemove event listeners etc, functions using the canvas context and SVG matrix separately outside of Broadway. I am now trying to integrate this with Broadway, but i seem to be running into problems.

What is the best way to achieve this? What part of the code should i look at to effectively integrate this functionality?

Thanks!

Hi,

I am trying to use Broadway to implement multiple canvases such that the video frames from each canvas is synchronized. I'm not sure what part of the code i need to alter in order to achieve this.

Also to test the synchronization of frames i made an mp4 video from images consisting of numbers 1-100, using ffmpeg. When i test the demos using this mp4 file, there is nothing displayed. Is there something that im missing?

Help will be appreciated!

Thanks!

Hi,

I am trying to get the Broadway demo to work on my computer and I seem to have run into some problems. Apparently the avc.js and avs-codec.js files have not been properly handled when I run the make.py script. I was unable to run the demo, so I copied the source files from the demo's browser provided here, to pin point where the issue came from, being the 2 avc files.

I simply ran the make.py script after downloading and configuring Emscripten. Is there anything that im missing at this step that I should take note of? Also the make.py script does not seem to create the Broadway.js and Decoder.js files. I compiled the corresponding C files manually to get the javascript. Is this right?

Thanks!
Saima

Hello,

I would like to know if this library supports raw H264 Annex B ? if not, is it planned?

Thanks.

hi @ALL,

first congratulations to the team for this awesome piece of code! I was able to build a working prototype within hours.

My h264 data is send from a mobile phone via a websocket server to the browser. Right now I want to upscale the canvas on the html page, I tried to set the width and height of the element, which works fine until new data is drawn.

Is there a trick how to upscale the canvas?

many thanks in advance,
axel

What about support commonjs and package.json for using browserify?

i updated my amd modules with your recently submitted newly compiled code.

its not working without changes. the _broadwayOnHeadersDecoded and _broadwayOnPictureDecoded are now supposed to be properties of the window object.
i find that disturbing.

multiple videos are now impossible.

i dont know if i can measure a perormance increase.
i can play videos at 15fps@720p
my previous measure was 10fps@1080p so maybe a little increase, maybe not.

the codec code seems to make a step into the direction of leaving the global namespace alone, however the reqired window properties nullify that effect and make the usability even worse, since the result functions are now mandatory global.

best wishes
sol

With IE11, the demo on http://bkw.github.io/Broadway/storyDemo.html will only show a green picture:

This can be attributed to IE's incomplete WebGL support. The error console will show many instances of errors such as

WEBGL11048: INVALID_ENUM: texImage2D: Invalid Format
File: canvas.js, Row: 114, Column: 5

WEBGL11094: INVALID_OPERATION: texImage2D: The enumeration LUMINANCE is currently not supported
File: canvas.js, Row: 114, Column: 5

Although IE11 may not be a primary target for Broadway, it would still be a nice-to-have. Apparently the decoder itself is functional and performing well.