Giter VIP home page Giter VIP logo

shaderkit's Introduction

Size Version Downloads

shaderkit

Tools and IntelliSense for GLSL and WGSL.

Table of Contents

Installation

To install, use your preferred package manager:

npm install shaderkit
yarn add shaderkit
pnpm add shaderkit

Or, use a CDN:

<script type="module">
  import * as shaderkit from 'https://unpkg.com/shaderkit'
</script>

Tokenize

Tokenizes a string of GLSL or WGSL code, returning an array of Token objects, where each Token object represents a single syntax feature in the input code.

interface Token {
  type: 'whitespace' | 'comment' | 'symbol' | 'bool' | 'float' | 'int' | 'identifier' | 'keyword'
  value: string
}
GLSL Example
import { tokenize } from 'shaderkit'

const code = 'void main() { gl_Position = vec4(0, 0, 0, 1); }'
const tokens = tokenize(code)

console.log(tokens)

The output of the above code will be:

[
  { "type": "keyword", "value": "void" },
  { "type": "whitespace", "value": " " },
  { "type": "identifier", "value": "main" },
  { "type": "symbol", "value": "(" },
  { "type": "symbol", "value": ")" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "{" },
  { "type": "whitespace", "value": " " },
  { "type": "keyword", "value": "gl_Position" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "=" },
  { "type": "whitespace", "value": " " },
  { "type": "keyword", "value": "vec4" },
  { "type": "symbol", "value": "(" },
  { "type": "int", "value": "0" },
  { "type": "symbol", "value": "," },
  { "type": "whitespace", "value": " " },
  { "type": "int", "value": "0" },
  { "type": "symbol", "value": "," },
  { "type": "whitespace", "value": " " },
  { "type": "int", "value": "0" },
  { "type": "symbol", "value": "," },
  { "type": "whitespace", "value": " " },
  { "type": "int", "value": "1" },
  { "type": "symbol", "value": ")" },
  { "type": "symbol", "value": ";" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "}" }
]
WGSL Example
import { tokenize } from 'shaderkit'

const code = '@vertex fn main() -> @builtin(position) vec4<f32> { return vec4(0, 0, 0, 1); }'
const tokens = tokenize(code)

console.log(tokens)

The output of the above code will be:

[
  { "type": "symbol", "value": "@" },
  { "type": "keyword", "value": "vertex" },
  { "type": "whitespace", "value": " " },
  { "type": "keyword", "value": "fn" },
  { "type": "whitespace", "value": " " },
  { "type": "identifier", "value": "main" },
  { "type": "symbol", "value": "(" },
  { "type": "symbol", "value": ")" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "->" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "@" },
  { "type": "keyword", "value": "builtin" },
  { "type": "symbol", "value": "(" },
  { "type": "keyword", "value": "position" },
  { "type": "symbol", "value": ")" },
  { "type": "whitespace", "value": " " },
  { "type": "keyword", "value": "vec4" },
  { "type": "symbol", "value": "<" },
  { "type": "keyword", "value": "f32" },
  { "type": "symbol", "value": ">" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "{" },
  { "type": "whitespace", "value": " " },
  { "type": "keyword", "value": "return" },
  { "type": "whitespace", "value": " " },
  { "type": "keyword", "value": "vec4" },
  { "type": "symbol", "value": "(" },
  { "type": "int", "value": "0" },
  { "type": "symbol", "value": "," },
  { "type": "whitespace", "value": " " },
  { "type": "int", "value": "0" },
  { "type": "symbol", "value": "," },
  { "type": "whitespace", "value": " " },
  { "type": "int", "value": "0" },
  { "type": "symbol", "value": "," },
  { "type": "whitespace", "value": " " },
  { "type": "int", "value": "1" },
  { "type": "symbol", "value": ")" },
  { "type": "symbol", "value": ";" },
  { "type": "whitespace", "value": " " },
  { "type": "symbol", "value": "}" }
]

The following are the supported token types and their descriptions:

Type Description
whitespace A sequence of one or more whitespace characters.
comment A single-line or multi-line comment.
symbol A symbol, such as an operator or punctuation mark.
bool A boolean value, either true or false.
float A floating-point number, represented by a sequence of digits and symbols.
int An integer number, represented by a sequence of digits.
identifier A user-defined identifier, such as a variable name or function name.
keyword A keyword reserved by the language, such as if, else, for, etc.

Minify

Minifies a string of GLSL or WGSL code, returning a minified version of the input code.

const minified: string = minify(code: string, {
  /** Whether to rename variables. Will call a MangleMatcher if specified. Default is `false`. */
  mangle: boolean | ((token: Token, index: number, tokens: Token[]) => boolean)
  /** A map to read and write renamed variables to when mangling. */
  mangleMap: Map<string, string>
  /** Whether to rename external variables such as uniforms or varyings. Default is `false`. */
  mangleExternals: boolean
})

To shared mangled interfaces when using mangleExternal, declare and re-use a mangleMap between shaders:

const options = { mangle: true, mangleExternals: true, mangleMap: new Map() }

// #version 300 es\nin vec2 a;out vec2 b;void main(){b=a;}
minify(`#version 300 es\nin vec2 sstt;out vec2 c;void main(){c=sstt;}`, options)

// #version 300 es\nin vec2 b;out vec4 a[gl_MaxDrawBuffers];void main(){a[0]=b.sstt;}
minify(`#version 300 es\nin vec2 c;out vec4 data[gl_MaxDrawBuffers];void main(){data[0]=c.sstt;}`, options)

Parse

Parses a string of GLSL (WGSL is WIP) code into an AST.

const ast: AST[] = parse(code: string)

Generate

Generates a string of GLSL (WGSL is WIP) code from an AST.

const code: string = generate(ast: AST[], {
  target: 'GLSL' // | 'WGSL'
})

AST

An Abstract Syntax Tree loosely based on ESTree for GLSL and WGSL grammars.

Literal

A shader literal representing a bool, float, int, or uint type.

class Literal {
  value: string
}

Identifier

A variable identifier.

class Identifier {
  value: string
}

Type

Represents a type specifier and its parameters (WGSL specific).

class Type {
  name: string
  parameters: (Type | Literal | Identifier)[] | null
}

VariableDeclaration

A variable declaration with an optional binding layout, type qualifiers, kind (WGSL only), and declarators (e.g. a comma-separated list).

class VariableDeclaration {
  layout: Record<string, string | boolean> | null
  qualifiers: string[]
  kind: 'var' | 'let' | 'const' | null
  type: Type | Identifier
  declarations: VariableDeclarator[]
}

VariableDeclarator

A single named declarator as part of a VariableDeclaration.

class VariableDeclarator {
  name: string
  value: AST | null
}

StructDeclaration

A struct declaration. Can be used as a type or constructor.

class StructDeclaration {
  name: string
  members: VariableDeclaration[]
}

FunctionDeclaration

A function declaration with an optional type qualifier and arguments.

class FunctionDeclaration {
  name: string
  type: Type | Identifier
  qualifiers: string[]
  args: VariableDeclaration[]
  body: BlockStatement | null
}

UnaryExpression

A unary expression with a left or right handed operator.

class UnaryExpression {
  operator: string
  left: AST | null
  right: AST | null
}

BinaryExpression

A binary expression with a left and right operand.

class BinaryExpression {
  operator: string
  left: AST
  right: AST
}

TernaryExpression

A ternary or conditional expression.

class TernaryExpression {
  test: AST
  consequent: AST
  alternate: AST
}

CallExpression

A call expression.

class CallExpression {
  callee: AST
  args: AST[]
}

MemberExpression

A member expression.

class MemberExpression {
  object: AST
  property: AST
}

ArrayExpression

An array expression. members can be empty if uninitialized.

class ArrayExpression {
  type: Type
  members: AST[]
}

BlockStatement

A block statement.

class BlockStatement {
  body: AST[]
}

IfStatement

An if statement.

class IfStatement {
  test: AST
  consequent: AST
  alternate: AST | null
}

ForStatement

A for statement.

class ForStatement {
  init: AST | null
  test: AST | null
  update: AST | null
  body: AST
}

WhileStatement

A while statement.

class WhileStatement {
  test: AST
  body: AST
}

DoWhileStatement

A do-while statement.

class DoWhileStatement {
  test: AST
  body: AST
}

SwitchStatement

A switch statement.

class SwitchStatement {
  discriminant: AST
  cases: SwitchCase[]
}

SwitchCase

A switch-case statement. test is null for a default case.

class SwitchCase {
  test: AST | null
  consequent: AST[]
}

ReturnStatement

A return statement with an optional argument.

class ReturnStatement {
  argument: Literal | Identifier | UnaryExpression | null
}

PreprocessorStatement

A GLSL preprocessor statement with an optional value.

class PreprocessorStatement {
  name: string
  value: AST[] | null
}

PrecisionStatement

A GLSL precision statement.

class PrecisionStatement {
  precision: 'lowp' | 'mediump' | 'highp'
  type: Type
}

ContinueStatement

A continue statement.

class ContinueStatement {}

BreakStatement

A break statement.

class BreakStatement {}

DiscardStatement

A discard statement.

class DiscardStatement {}

shaderkit's People

Contributors

codyjasonbennett avatar renaudrohlinger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

shaderkit's Issues

minifier mangles external when token appears after curly brace

the following code breaks during mangling

out vec3 vLight;
void main() {
  if(true) {
  }
  vLight = vec3(1.0);
}

=> out vec3 vLight;void main(){if(true){}a=vec3(1.0);}

Adding a semicolon after the braces gets mangled properly:

out vec3 vLight;
void main() {
  if(true) {
  };
  vLight = vec3(1.0);
}

=> out vec3 vLight;void main(){if(true){};vLight=vec3(1.0);}

WGSL variable is ignored by the mangler

The minifier is unable to identify some variables in WSGL.

The issue can already be observed by looking at the current snapshots where output is not mangled.

Reproduction

it('mangle WGSL variables', () => {
  const mangleMap = new Map()
  const shader = /* wgsl */ `@vertex fn main() -> void { var test: f32; }`
  expect(minify(shader, { mangle: true, mangleMap })).toMatchSnapshot()
  expect(mangleMap).toContain('test')
  //                ^^^^^^^^^ AssertionError: expected Map{} to include 'test'
})

mangler charset includes "`" character

When mangling, the "`" character will be used to generate mangled names, which breaks GLSL code.

Example manglemap snippet:

'PI' => 'x',
'facing' => 'y',
'lambertian' => 'a`',
'reflectDir' => 'aa',
'specular' => 'ab',

Float tokenization requires whitespace after each float

Thank you for making and open sourcing this tool!

I ran into an issue with source code without whitespace between numbers, operators and identifiers:

tokenize('2.2+bar')
// [ { type: 'float', value: '2.2+bar' } ]

presumably an issue with

const isNumber = (c: number) => isAlpha(c) || isDigit(c) || c === PLUS || c === MINUS || c === DOT

and/or

while (isNumber(code.charCodeAt(index))) value += code[index++]

mangle function result is ignored if manglemap has seen token of same value before

Hey,

I enjoy using the mangle function to determine whether a given token should be mangled.
In some situations, where I conditionally return false in the mangle function, the token still gets mangled if the mangleMap contains a value of that token from a previous occurrence. I am not sure if that is intentional, but it seems unexpected to me.

Related lines:

!renamed &&

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.