advice for creating a python-based wrapper to projectM? about projectm HOT 24 CLOSED

from projectm.

Hi sorry for my late reply back. The reason for python was to eventually integrate in to the audio visualizer mentioned above, which is based in python and has a QT gui plus command line interface on top. But that may be a long term goal, so the immediate need was to get an understanding of how to create a wrapper for projectM to begin with. For starters, is there functionality in place such that if I have a sound file (typically in mp3 format) could I invoke a visualization with projectM from the command line? Just trying to get a feel for how to interact with it.

from projectm.

Please let me know if there's anything I can assist you with

from projectm.

Is there guidance you can offer on how to get a footing in understanding this codebase so I may begin to understand what needs to be wrapped? Is all core functionality in src/libprojectM/? Is there some other code I can read as a reference for using the library?

The Python project, Audio Visualizer, currently sends sections of raw PCM signed 16-bit audio data to plugins, and each plugin draws the corresponding video frame for that point in the audio, drawn on top of transparency to be composited into the final frame which may contain other layers.

Therefore, is there a way to receive a video frame (1/30th of a second) for a corresponding piece of audio data in this format? Can ProjectM create visualizations on top of transparency when applicable to the preset in use?

from projectm.

I am not sure what "on top of transparency" means. It can be output as an openGL texture.
For a very simple example of usage, see https://github.com/projectM-visualizer/projectm/blob/master/src/projectM-sdl/projectM_SDL_main.cpp
I don't think there is currently a way to take any video data as input, only PCM data. You could try making a "native preset" which can do anything and take video input that way. I don't know if you can combine that with another preset though.

from projectm.

In my comment above, ProjectM would be the plugin receiving PCM data. The 'video frame' is what ProjectM creates and returns to Audio Visualizer, which I suppose is the OpenGL texture you mention. Thanks for that link, it is informative.

By "on top of transparency" I mean that our program normally expects its plugins to give a Pillow RGBA image or a Qt QImage, so any background should have an alpha value of 0. It would be ideal for compositing with layers below the plugin, but not a mandatory feature.

from projectm.

Chiming in to say I would be very interested in a Python wrapper to ProjectM.

I am also interested in the question of how Python can send PCM data to ProjectM and receive video data as output.

I am not at all familiar with the codebase, but based on the example it appears that ProjectM is fed a sequence of PCM16 data - however much you want. This data is stored in projectM-> _pcm - a PCM() object. It will keep a history of up to 2048 samples it seems.

Then, when you want to request a frame, projectM's renderFrame() is called. First, this calls BeatDetect->detectFromSamples. BeatDetect appears to do a lot more than just detect beats, and stores all of the data that is passed through to presets, such as vol, treb, mid, bass, vol_history etc - which IIRC are what the preset scripts use to build visualizations. Once BeatDetect has characterized the PCM history attributes, m_activePreset->Render() is called using that data. The preset will then construct a GL pipeline based on this, stored in m_activePreset->pipeline.

The active preset GL pipeline is then finally passed into renderer->RenderFrame(), which draws out the pipeline.

Does this sound at all close to what is actually happening? Again, I'm just skimming through so I'm probably missing some critical details.

If so, given a sequence of PCM data, to construct a single frame it sounds like you need to:

0. Set up some kind of GL context
1. Set up projectM() using a bunch of settings, you could probably just take his defaults. Do selectRandom(true) or something to set an initial preset and projectM_resetGL(width, height) to set up GL.
2. Use projectM->pcm()-> addPCM16() to add in your PCM history
3. Call projectM->renderFrame(), which will take the PCM history, run beat detect on it, pass that info into the preset, which will then yield a pipeline, which is then passed into the Renderer(), which goes through and uses GL to draw all the drawables stored in the pipeline.

At this point GL will have rendered your frame. Now the real trick is in taking that finished GL context and displaying it to a window, or drawing it to an image, or dumping the frame to an array somehow. Don't know much about openGL, so not sure how this would work.

At a Python level, you might either want to turn this process into a script that Python can call if you don't want to interface with C++ directly, or you could perhaps use something like SWIG to make calls directly into projectM. I don't have enough experience with C++<->Python interop to provide much help here - this is just brainstorming.

In his example, he seems to use SDL_RenderPresent() which just renders everything drawn in openGL up to that point. SDL also gives him a window, so that is where the results will be displayed upon that call.

Anyways, I am just trying to better understand the codebase and how this wrapper might work. Does this sound about right?

Let me know if you get started on some Python code for this, I'd love to take a look! I am very interested in a wrapper.

from projectm.

Thanks @dnola that's a great writeup. I can verify the single frame construction part but it sounds like you've done more digging than I have into the internal flow.
I believe there are a couple ways to draw the frame. When you have a GL context I think it's usually handled by the application where you want to display it. For higher-level toolkits like SDL, you create a context in a window. I assume you can also create an off-screen context if desired. The other way that I'm aware of for drawing is you can have projectM render into a GL texture and bind that to whatever you want. I have not personally tried to implement this yet.

In python with sdl2 you can create a context with something like this:

        SDL_GL_SetAttribute(SDL_GL_MULTISAMPLEBUFFERS, 1);
        SDL_GL_SetAttribute(SDL_GL_MULTISAMPLESAMPLES,16);
        self.glcontext = SDL_GL_CreateContext(window.window)
        glEnable(GL_BLEND)
        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)

from https://github.com/revmischa/tri

I haven't done C/C++ bindings for python yet but I think this is a wonderful idea and I'd be happy to help where I can (not a ton of free time at the moment but I'll do what I can.) What's the first step, you think? We can add it to this project. Send a PR or I can give you contributor access.

from projectm.

I think SWIG would probably be the best approach, but I have next to no experience in it unfortunately. I think the idea is that you would write a SWIG interface to create a Python module, and that interface file would include projectM.hpp.

I think the big question is how you would create an openGL context in Python land, then call projectM->renderFrame() in C++ land, and have it be aware of the Python context so it can render to it. I think someone with SWIG experience would probably need to answer that one.

Unfortunately I'm short on time myself, but if someone knows how a shared GL context could be created between Python and C++ - i.e. create context in Python, draw in C++ - with something like SWIG I would be happy to look into it!

from projectm.

SWIG is definitely appropriate if you want to bind to more than one language, like Python, Swift, Java etc. It may add extra complexity and overheard if we only care about making Python bindings.
I could be mistaken but I'm pretty sure you don't have to pass any GL context to projectM. You simply create a context and when you call renderFrame() it will draw into the current active GL context.
This is how it is currently done but this may fall down when moving to GLES which probably has more reentrant context management (big assumption here).

Do you want to write up your findings (what you described above) in the readme or wiki or a document at some point? That'd be helpful for other folks that want to contribute.

from projectm.

I'm definitely not enough of an expert to recommend SWIG vs some other C++ interop method, so you could absolutely be right. I've just mostly heard of people using SWIG when trying to wrap large, existing C++ codebases - the SWIG interface files seem to do a lot of work/magic for you.

Yes, I also agree that it looks like you don't have to pass a context to projectM - my question would be that if you create a GL context in Python, and then call renderFrame() using some interop method, will renderFrame() be aware of the active GL context? I don't know enough about openGL/interop to be clear on this one - sorry I can't be more helpful!

And yes, I would be happy to write a wiki page or something. I'm guessing you are referring to the 'PCM data -> frame' workflow? Or my somewhat oversimplified explanation of what is happening inside ProjectM? Both?

Thanks!

from projectm.

GL is still GL, if you're calling renderFrame() from python it should still be aware of the active GL context.

from projectm.

Am I late to the party? When I first heard about this I thought it would be a cool project that I could help out with, but was too busy moving across the country at that time. Things have finally settled down. I had some extra time this last week. I took a stab at prototyping out python bindings using cython. I've created a new repo to hold the project here for now. I was able to recreate the projectm-sdl example with it, but I still think the PCM interface needs some work. I would be interested in getting people's thoughts on whether this is useful or not and future directions to take with it.

Thanks!

Bryan

from projectm.

Oh great! I hope to take a look soon

from projectm.

Let me know how I can help out, this looks awesome

from projectm.

There are a couple of headers that don't get installed that are needed to use projectm.hpp.

Can you add these to src/libprojectM/Makefile.am?

ie:

include_HEADERS = projectM.hpp Common.hpp dlldefs.h event.h fatal.h PCM.hpp

Thanks

Bryan

from projectm.

Sure, open a PR!

from projectm.

Still relatively new to githhub. Do you need to grant me permissions for me to do that?

from projectm.

Nope! You fork the repo and then you have write access to your fork. Then you can push to a branch on your fork repo and then open a PR to merge your branch to this repo. Or do it on master if you're into that sorta thing.

from projectm.

Seems like a cool project!
Do you think it could be possible to get access to presets variables, in python?
I mean, to modify a variable externally?

from projectm.

I don't see why you couldn't modify the preset variables while a preset is running. I think they're accessible from the Preset param list?
I'd check this out: https://github.com/projectM-visualizer/projectm/blob/master/src/libprojectM/MilkdropPresetFactory/PerFrameEqn.hpp#L41

from projectm.

It could be useful to trigger external events/changes, even when not synced to the music

from projectm.

We would have to add some plumbing to the projectM class to expose PerFrameEqn and related classes correct? From a python bindings standpoint I would have to take a closer look. My worries would be exposing to much of ProjectM's internals to python. I would want to have some well understood use cases and a tight public facing C++ API for these elements first.

from projectm.

Yes, you'd most likely have to add some setters/accessors to such things. If you have any more questions feel free to join us on IRC or open a new issue.

from projectm.