Giter VIP home page Giter VIP logo

isf4ae's Introduction


ISF4AE - Interactive Shader Format for After Effects




After Effects plugin to allows you to use GLSL code written in the Interactive Shader Format as an effect. In short, it's an effect that lets you build effects by yourself -- without struggling with the complicated C++ SDK.

Supported Environments

The plugin has been confirmed to work in the following environments. However, it should generally work in CC 2022 or later on any macOS and Windows:

  • CC 2023 on macOS Monterey (Apple Silicon, Intel)
  • СС 2023 on Windows x86_64

Additional ISF Specification

Since ISF was originally designed for real-time purposes, such as VJing, it has some limitations that do not align with After Effects' characteristics. This section describes how the plugin interprets ISFs and interoperates with After Effects. The following explanation assumes that you have already understood the specification of ISF.

Limitations

Currently, the plugin does not support several types of inputs, persistent buffers, or custom vertex shaders. It also has a maximum of 16 inputs (excluding "inputImage" and reserved uniforms "i4a_" as mentioned later).

Supported Input Types

Note that "event", "audio", and "audioFFT" are not yet supported currently.

  • "bool": displayed as a checkbox.
  • "long": displayed as a dropdown list by default. But if no VALUES key exists, an integer slider between MIN and MAX is used instead.
  • "float": displayed as a slider by default. You can specify a type of UI by setting either constant below as a property "UNIT".
    • "default": just a scalar; the value will be passed as it is.
    • "length": represents a length in px. It is also displayed as a slider, but the value will be mapped from 0...<layer's width> to 0...1 when passed to a shader.
    • "percent": displayed with "%" suffix. The value will be bounded as a uniform divided by 100. For example, 50% in the Effect Controls panel will be passed as 0.5 to the shader.
    • "direction": displayed as an angle input, and the range of value will be mapped so that radians in OpenGL coordinate (Y-up right-handed) match the direction that the rotary knob UI is pointing. The internal conversion from AE to GLSL is equivalent to the formula radians(90 - value). It would be suitable for representing an absolute angle that should be consistent visually between the knob UI and the output.
    • "angle": similar to the above, displayed as an angle input but uses a conversion formula radians(-value) instead. It is suitable for representing relative angle changes -- hue offset of HSL adjustment, for instance.
  • "point2D": represents a position in px. It will be mapped to OpenGL's normalized coordinate; (0, 0) at the bottom-left corner, (1, 1) at the top-right corner of the layer.
  • "color": displayed as a color picker.
  • "image": displayed as a layer reference input.

Other Addotional Properties for Inputs

"MIN" and "MAX" only affect the range of slider UI in the Effect Controls panel and users can still set the values outside of the range. To forcibly constrain the value within the range, you can use the plugin's custom properties "CLAMP_MIN" and "CLAMP_MAX" in a boolean value.

ISF Built-in Uniforms

Here is how the plugin determines the value of ISF built-in uniforms

Uniform Description
TIME When you load a shader referring to this uniform, the hidden parameters appear in the Effect Controls panel. You can use either layer time or custom value.
TIMEDELTA Always set to a value that equals to an expression thisComp.frameDuration
FRAMEINDEX Equivalent to TIME / TIMEDELTA, or TIME * fps.

ISF4AE-specific Uniforms

Inputs with names beginning with i4a_ are reserved by the plugin. When you define the inputs shown below, the plugin automatically binds values that can be useful for accessing the status of the Preview panel from a shader.

Name ISF Type Description
"i4a_Downsample" point2D Downsampling factor in Preview panel. For instance, the value is set to (0.5, 0.5) in half resolution.
"i4a_CustomUI" bool Set to true when the effect requests an image for Custom Comp UI, which is only visible when you click and focus the effect title in the Timeline / Effect Controls panels. The pass returned by a shader will be overlayed onto the result layer.
NOTE: you cannot refer to any image inputs on this pass.
"i4a_UIForegroundColor"
"i4a_UIShadowColor"
bool For referring to After Effects' current color scheme to draw Custom Comp UI. Only available when i4a_CustomUI is true.
"i4a_UIShadowOffset" point2D Same as above. Unlike user-defined inputs, the range of value is not normalized and will be passed in absolute px.
"i4a_UIStrokeWidth"
"i4a_UIVertexSize"
float Same as above. Passed in px.

How to Build

  1. Download After Effects Plug-in SDK 2022 Mac - Oct 2021.

  2. Clone this repository, including its submodule, by following the steps below:

# At the root folder of SDK
cd Examples/Effect
git clone https://github.com/bskl-xyz/ISF4AE.git
cd ISF4AE
git submodule update --init

On Mac

  1. Open the Xcode project and build it. The binary will automatically be copied under /Library/Application Support/Adobe/Common/Plug-ins/7.0/MediaCore/.

On Windows

  1. Set environment variable AE_PLUGIN_BUILD_DIR to C:\Program Files\Adobe\Common\Plug-ins\7.0\MediaCore.

  2. Copy the glew32.dll dynamic library from VVISF-GL\external\GLEW\win_x64 to C:\Windows\System32\ folder.

  3. Apply VVISF-GL patch in the root of this repo:

git apply VVISF-GL-Submodule.patch

License

This plugin has been published under an MIT License. See the included LICENSE file.

Similar Projects

  • PixelsWorld: Scripting environment supporting Lua, GLSL, and shadertoy.com.
  • Pixel Blender Accelator: A plugin to run Pixel Blender, a shader format supported in Adobe products previously, in modern versions of AE.
  • AE-GLSL: Working-in-progress repository by cryo9, which hinted me how to interoperate OpenGL and After Effects SDK.
  • AE_tl_math: an plugin project which allows users to write math expression for each pixels, or GLSL shaders. It also bundles a nice pane to edit and update code in realtime.

Acknowledgments

  • mizt: One of the visual artists having an ability to buidl a plugin for After Effects. We've traded lots of tips on AESDK.
  • 0b5vr: He also gave me many advices for handling OpenGL.
  • Patricio Gonzalez Vivo: One of my reason for being hooked on shaders is his cool project The Book of Shaders. This plugin once aimed to support the format of its editor.

isf4ae's People

Contributors

baku89 avatar gmatters avatar timurco avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

isf4ae's Issues

Feature Request: Windows support

Not sure if this is planned or not, but would be really nice to see this get Windows support! Looking at the source with my limited knowledge of cpp, I think the file open/save dialogs and such are the only parts that require macOS, but maybe there's more to it I'm not aware of.

Cant install, patch problems from git?

Hey, sorry im newb but i get this error when installing:

error: git diff header lacks filename information when removing 1 leading pathname component (line 98)

any hints on how to fix this?

Request for Enhancement: Effect compilation features

It could be helpful if the effect had a sort of “export“ button in ECP and could compile the currently loaded ISF into a binary.
There are several merits for realizing the feature:

  • Much easier to distribute effects for teams, as users don't need to load the specific ISF manually after applying the ISF4AE effect.
  • No need to worry about some references via expressions being broken since the parameter lists are declared statically.

Suggested by @mizt

AE 2018 Crash

First, let me thank You for creating and sharing GLSLCanvas!

I've tried it with two versions of After Effects.
AE CC 2017.2 (14.2.1.34) - loads and renders some of shaders from Book of Shaders that I've tried. So far no crashes, performance is ok (just a little choppiness while adjusting parameters in plug-in UI).

AE CC 2018 (15.0.0, build 180) - after attempting to load shader via "Load Shader.." AE crashes, same if I just add GLSLCanvas to empty solid and move playhead. Few times it even froze my whole system (not a kernel panic, just complete freeze and lots of artefacts). I also tried to change AE renderer from Metal to Software Only, same thing.
Crash report:
AE_2018_GLSLCanvas_CRASH.txt

I've compiled Your plugin on MacOS 10.12.6 & XCode 9.2 (9C40b) and used latest AE SDK.

Set an expression 'thisLayer.time' as a default value for Time property

As discarding the 'Use Layer Time' checkbox,
it'd be useful if the remaining 'Time' slider automatically refers the layer time that the effect is applied.

Since PF_AD_PARAM only supports non-expression values as a default,
I tried to use AEGP suites to modify a parameter like the below code at the timing PF_Cmd_SEQUENCE_SETUP has called:

// Set the default expression to parameters
AEGP_EffectRefH effectH = NULL;
AEGP_StreamRefH streamH = NULL;

ERR(suites.PFInterfaceSuite1()->AEGP_GetNewEffectForEffect(G_my_plugin_id, in_data->effect_ref, &effectH));
ERR(suites.StreamSuite5()->AEGP_GetNewEffectStreamByIndex(G_my_plugin_id, effectH, FILTER_TIME_ID, &streamH));
ERR(suites.StreamSuite2()->AEGP_SetExpressionState(G_my_plugin_id, streamH, true));
ERR(suites.StreamSuite2()->AEGP_SetExpression(G_my_plugin_id, streamH, "thisLayer.time"));
    
if (effectH) ERR(suites.EffectSuite3()->AEGP_DisposeEffect(effectH));
if (streamH) ERR(suites.StreamSuite2()->AEGP_DisposeStream(streamH));    

But it raises an error, which says ”internal verification failure, sorry! {child not found in parent} (20::0)”.
Does any guru know how to handle this?

Support referencing ISF inputs via expressions

Since the plug-in dynamically changes the names of parameters to match with ISF inputs, it raises an error when you try to retrieve the value by expression.

An expression like this:

const d = thisLayer.effect("Polar Displacement")("Distance");

triggers an error with the message: ”Error: property or method named 'Distance' in Class 'Group' is missing or does not exist.”.

Crash when MFR enabled and glBindBuffer() Issues on Windows Version

Hello everyone! I decided to create a separate Issue, which was previously mentioned in the Enhancement regarding Windows in this issue. Now there is a Windows version, but it seems to work with certain limitations, apparently related to issues with multithreading in OpenGL, as mentioned here.

When we simply move the sliders or the playhead in the composition, everything is fine. Also, if we turn off the MFR mode in the settings, the plugin's output renders perfectly.

image

However, with MFR mode enabled, a crash occurs because the cpuBackingPtr is found to be Null. I've reflected this more explicitly in this assert inside SmartRender at this line:

if (!outputImageCPU->cpuBackingPtr) {
      err = PF_Err_OUT_OF_MEMORY;
      // A more explicit assert that appears with the new versions 
      // of the Nvidia Studio driver described in this post:
      // [link](https://github.com/baku89/ISF4AE/issues/19#issuecomment-1724631129)
      assert("FATAL! CPU backing pointer is NULL! Cannot copy CPU buffer to EffectWorld!" && false);
    }

But there are some peculiarities, many of which I mentioned in the comments to that issue. Although I've now found that the driver version doesn't actually affect the issue. During rendering or background frame calculation, cpuBackingPtr will always be NULL.

Also, the scene behaves very strangely if, for some reason, you don't do this line before compiling it:

// Without this USELESS variable I'm getting a glitch, where the scene
// doesn't work without any errors
// FIXME: Dig into it to figure it out the reason
ISF4AESceneRef useless = VVISF::CreateISF4AESceneRefUsing(globalData->context->newContextSharingMe());

In doing so, you actually need to assign this variable; otherwise, it won't fix the problem. I made these useless variables in a separate commit.

Without it, the screen will sometimes be black, and the gl2aeScene won't work either. There's still more to figure out with this issue, and it may not be as critical if it isn't related to the main problem causing the crash.


Possible Cause

Firstly, for what it's worth, the context on Windows is initialized slightly differently, and I did this based on intuition and from examples; perhaps I did something wrong in GlobalSetup:

#ifdef _WIN32
  GLContext::bootstrapGLEnvironmentIfNecessary();
  // VVGL on Windows Doesn't have pixel format functions
  globalData->context = VVGL::CreateNewGLContextRef(nullptr, nullptr);
#else
  globalData->context = VVGL::CreateNewGLContextRef(NULL, CreateCompatibilityGLPixelFormat());
#endif
  VVGL::CreateGlobalBufferPool(globalData->context->newContextSharingMe());

globalData->downloader = VVGL::CreateGLTexToCPUCopierRefUsing(globalData->context->newContextSharingMe());
globalData->uploader = VVGL::CreateGLCPUToTexCopierRefUsing(globalData->context->newContextSharingMe());

Here the global pool is initialized, which is subsequently used in the downloader and uploader, and I think the problem lies therein. Because later on, when we copy data from CPU to GPU through the uploader in the SmartRender function, here:

ERR(uploadCPUBufferInSmartRender(globalData, in_data->effect_ref, extra, checkoutIndex, layerSize, image));

Further in this function here:

auto imageAE = globalData->uploader->uploadCPUToTex(imageAECPU);

And then further down the calls, we come to this function: GLCPUToTexCopier::_beginProcessing inside of which we fail to bind the buffer:

glBindBuffer(inPBOBuffer->desc.target, inPBOBuffer->name);
if (glGetError() != 0) {
  assert("Cannot Bind to Buffer" && false);
}

All I found about this is a similar issue. But I'm not particularly knowledgeable about working with glew, and maybe @baku89 has an idea why, for instance, our buffer gets cleared through VVGL::GetGlobalBufferPool()->housekeeping(); in the renderISFToCPUBuffer function and not, say, through purge()? Or why does the cleanup happen here and not at the end of SmartRender()?
Spoiler: I tried changing and moving them, it doesn't make any difference.

Crash when long TYPE does not have LABELS or VALUES field defined.

The ISF spec says long "has" the extra LABELS and VALUES fields, but does not explicitly state whether they are required or optional.

ISF4AE in both renderISFToCPUBuffer and UpdateParamsUI assume that LABELS and VALUES exist for every long, and AE will crash if those fields are not defined for a long.

ISF Editor has logic to use values as labels if there is no LABELS field (or LABELS size mismatches), and values between MIN and MAX if there is no VALUES field defined. See ISFEditor/ISFEditor_app/misc_ui/ISFUIItem.cpp

The ISF Editor behaviour allows long to be used a generic integer input. At least, ISF4AE should fail to load rather than crash.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.