Giter VIP home page Giter VIP logo

wetzel's Introduction

wetzel

Generate Markdown or AsciiDoctor documentation from JSON Schema

Purpose and Limitations

This tool was developed to generate reference documentation for the glTF schema. As such, it doesn't support the entire JSON Schema spec, only what is needed by the glTF schema. Currently it accepts JSON Schema drafts 3, 4, 7, and 2020-12.

Example

This JSON Schema:

{
    "$schema" : "https://json-schema.org/draft/2020-12/schema",
    "title" : "example",
    "type" : "object",
    "description" : "Example description.",
    "properties" : {
        "byteOffset" : {
            "type" : "integer",
            "description" : "The offset relative to the start of the buffer in bytes.",
            "minimum" : 0,
            "default" : 0
        },
        "type" : {
            "type" : "string",
            "description" : "Specifies if the elements are scalars, vectors, or matrices.",
            "enum" : ["SCALAR", "VEC2", "VEC3", "VEC4", "MAT2", "MAT3", "MAT4"]
        }
    },
    "required" : ["type"],
    "additionalProperties" : false
}

can be used to generate this Markdown documentation:


example

Example description.

example Properties

Type Description Required
byteOffset integer The offset relative to the start of the buffer in bytes. No, default: 0
type string Specifies if the elements are scalars, vectors, or matrices. ✓ Yes

Additional properties are not allowed.

example.byteOffset

The offset relative to the start of the buffer in bytes.

  • Type: integer
  • Required: No, default: 0
  • Minimum: >= 0

example.type

Specifies if the elements are scalars, vectors, or matrices.

  • Type: string
  • Required: ✓ Yes
  • Allowed values:
    • "SCALAR"
    • "VEC2"
    • "VEC3"
    • "VEC4"
    • "MAT2"
    • "MAT3"
    • "MAT4"

Getting Started

Install Node.js if you don't already have it, clone this repo, and then:

cd wetzel
npm install

Run node bin/wetzel.js and pass it the path to a file with a JSON Schema, and the generated Markdown is output to the console.

It is useful to pipe the Markdown output to the clipboard and then paste into a temporary GitHub issue for testing.

On Mac:

wetzel ../glTF/specification/2.0/schema/accessor.schema.json -l 2 | pbcopy

On Windows:

wetzel.js ../glTF/specification/2.0/schema/accessor.schema.json -l 2 | clip

Run the tests:

npm run test

There's also a version published on npm.

Command-Line Options

  • The -l option specifies the starting header level.
  • The -c option lets you specify a custom symbol to place in front of required properties.
  • The -k option replaces the word must with a specified keyword, such as **MUST**.
  • The -p option lets you specify the relative path that should be used when referencing the schema, relative to where you store the documentation.
  • The -s option lets you specify the path string that should be used when loading the schema reference paths.
  • The -e option writes an additional output file that embeds the full text of JSON schemas (AsciiDoctor mode only).
  • The -m option controls the output style mode. The default is Markdown, use -m=a for AsciiDoctor mode.
  • The -n option will skip writing a Table of Contents.
  • The -w option will suppress any warnings about potential documentation problems that wetzel normally prints by default.
  • The -d option lets you specify the root filename that will be used for writing intermediate wetzel artifacts that are useful when doing wetzel development.
  • The -a option will attempt to aggressively auto-link referenced type names in descriptions between each other. If it's too aggressive, you can add =cqo so that it only attempts to auto-link type names that are within "code-quotes only" (cqo) (e.g.: typeName)
  • The -i option lets you specify an array of schema filenames that might be referenced by others, but shouldn't get their own documentation section.

Common Usage

This tool is used to generate the glTF Properties Reference section and the JSON Schema Reference Appendix of the glTF specification, using the glTF JSON Schema files as its input data.

The process is initiated from a GitHub Action in the glTF repository (CI.yml). This action runs glTF's Makefile. The Makefile calls wetzel with a command similar to the following:

~/bin/wetzel.js \
    -n -a=cqo -m=a \
    -p "schema" \
    -e "JsonSchemaReference.adoc" \
    -i '["gltfchildofrootproperty.schema.json", "gltfid.schema.json", "gltfproperty.schema.json"]' \
    -c "icon:check[]" \
    -k "**MUST**" \
    schema/glTF.schema.json > PropertiesReference.adoc

This will read schema/glTF.schema.json and all referenced sub-schemas, and produce two different output files:

These files are then included into glTF's Specification.adoc using AsciiDoc include commands:

[[properties-reference]]
= Properties Reference

// Generated with wetzel
include::PropertiesReference.adoc[]

and later:

[appendix]
[[appendix-a-json-schema-reference]]
= JSON Schema Reference (Informative)

// Generated with wetzel
include::JsonSchemaReference.adoc[]

Finally, the Makefile uses asciidoctor to convert Specification.adoc and its included, generated documentation, into HTML and PDF forms of the final glTF specification document, which are then posted to the glTF Registry.

Contributions

Pull requests are appreciated! Please use the same Contributor License Agreement (CLA) used for Cesium.


Developed by the Cesium team and external contributors.

wetzel's People

Contributors

analyticalgraphics avatar beardedgnome avatar emackey avatar ggetz avatar howardwolosky avatar javagl avatar lilleyse avatar mattmcmullan avatar mohd-akram avatar pjcozzi avatar sanjeetsuhag avatar shehzan10 avatar theory avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wetzel's Issues

Handle 3D Tiles schema

In order to support the 3d-tiles schema these two changes are needed:

  • Handle schema references that create circular dependencies. For example in tile.schema.json.
  • Support references to internal definitions, like in batchTable.schema.json.
    { "$ref" : "#/definitions/binaryBodyReference" }

Relative `$ref`s are always resolved against root schema path (?)

I'm not sure if it's the intended behaviour, but it feels like it shouldn't be.

./main.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "main",
    "$ref": "./subdir/child1.json"
}

./subdir/child1.json:

{
    "title": "child1",
    "$ref": "../child2.json"
}

./child2.json:

{
    "title": "child2",
    "type": "string"
}

wetzel main.json fails with Error: Unable to find $ref ../child2.json, but works if I change the $ref in child1 to ./child2.json, as if child1 is located in the same directory as child2.

Items missing from Schema with array type

I have a JSON schema named things.schema.json that looks like this:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "things.schema.json",
    "title": "Things",
    "description": "A list of things.",
    "type": "array",
    "uniqueItems": true,
    "minItems": 1,
    "items": { "$ref": "thing.schema.json" }
}

It references thing.schema.json, which looks like this:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "thing.schema.json",
    "title": "Thing",
    "description": "A thing.",
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "age": { "type": "number" }
    },
    "required": ["name", "age"]
}

Then I run wetzel like so:

wetzel.js --autoLink=cqo things.schema.json > things.md

The resulting file looks like this:

# Objects
* [`Thing`](#reference-thing)
* [`Things`](#reference-things) (root object)


---------------------------------------
<a name="reference-thing"></a>
## Thing

A thing.

**`Thing` Properties**

|   |Type|Description|Required|
|---|---|---|---|
|**name**|`string`|| &#10003; Yes|
|**age**|`number`|| &#10003; Yes|

Additional properties are allowed.

### thing.name

* **Type**: `string`
* **Required**:  &#10003; Yes

### thing.age

* **Type**: `number`
* **Required**:  &#10003; Yes




---------------------------------------
<a name="reference-things"></a>
## Things

A list of things.

Note that the Things reference at the bottom is missing any information about the items. The same is true even if I set "items": { "type": "string" }; the resulting Markdown is simply:

# Objects
* [`Things`](#reference-things) (root object)


---------------------------------------
<a name="reference-things"></a>
## Things

A list of things.

If I add minItems uniqueItems, or any of the rest, none of it appears, either. Shouldn't there be more details on what constitutes an an array schema?

jsonpointer Security Vulnerability

Dependabot on my employer's repository reports:

CVE-2021-23807

moderate severity
Vulnerable versions: < 5.0.0
Patched version: 5.0.0
This affects the package jsonpointer before 5.0.0. A type confusion vulnerability can lead to a bypass of a previous Prototype Pollution fix when the pointer components are arrays.

And:

Remediation

Upgrade jsonpointer to version 5.0.0 or later. For example:

"dependencies": {
  "jsonpointer": ">=5.0.0"
}
```

or…

```
"devDependencies": {
  "jsonpointer": ">=5.0.0"
}
```

Add AsciiDoctor output style

Would be nice to have an option to output AsciiDoctor instead of Markdown.

Looks like most of the Markdown flavoring is handled in style.js.

Could another file asciiDoctorStyle.js be added, to produce AsciiDoctor flavoring instead of Markdown, on command? If the styles aren't too different, this might be low-ish hanging fruit.

AsciiDoctor has a Markdown conversion cheat-sheet.

Definitions cannot refer to other definitions

Here is an example schema that refers to a type exampleReferenceDefinition that is defined in another file:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title" : "Definitions user example",
    "description" : "An example that uses definitions from another file",
    "type" : "object",
    "properties": {
        "exampleProperty": {
            "allOf": [
                {
                    "$ref": "definitions.schema.json#/definitions/exampleReferenceDefinition"
                }
            ]
        }
    }
}

The definitions.schema.json looks as follows:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title" : "Definitions example",
    "description" : "An example with definitions",
    "type" : "object",
    "definitions": {
        "exampleDefinition": {
            "title": "An example definition",
            "type": "string"
        },
        "exampleReferenceDefinition": {
            "title": "An example definition that refers to another one",
            "allOf": [
                {
                    "$ref": "#/definitions/exampleDefinition"
                }
            ]
        }
    }
}

This causes wezel to bail out:

C:\wetzel\lib\replaceRef.js:54
            throw new Error(`Unable to find $ref ${ref}`);
            ^

Error: Unable to find $ref #/definitions/exampleDefinition
    at replaceRef (C:\wetzel\lib\replaceRef.js:54:19)
    at replaceRef (C:\wetzel\lib\replaceRef.js:100:32)
...

Roughly: The definitions are not properly transported through the recursive calls of replaceRef.

My gut feeling is that this is related to #56 . More generally: The replaceRef approach of trying to completely "inline" the references may not be the most sustainable here. It does/will also cause trouble when/if allOf is replaced with $ref.

I've spent a few hours debugging the surroundings of replaceRef. Maybe I'll try to refactor this in an attempt to kill a few bugs with one stone, but it's hard to make promises here...

Roadmap

  • New properties beyond JSON Schema?
    • webglNotes
    • detailedDescription

Handle "oneOf" { "required" ... }

"Required" isn't always a yes/no condition in the glTF schema. Some types like image have a "one or the other is required" kind of condition, that is not reflected in the documentation.

This tool should find a way to output that detail.

Either $ref resolution doesn't work, or $id is ignored.

Wetzel version: Whatever it is in git right now.
OS: Windows 10
Node: 16.13.1

Given the following two schemas placed in a subdirectory named schemas:

schemas\a.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "http://example.com/schema/schema_a",
    "title": "schema a",
    "type": "object",
    "properties": {
        "something": { "type": "string" }
    }
}

schemas\b.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "http://example.com/schema/schema_b",
    "$ref": "http://example.com/schema/schema_a",
    "title": "schema b",
    "description": "schema a with an additional property",
    "type": "object",
    "properties": {
        "something_else": { "type": "string" }
    }
}

When I run:

node bin\wetzel.js -p schemas -i "[\"a.json\"]" schemas\b.json

Wetzel fails with:

Error: Unable to find $ref http://example.com/schema/schema_a
    at replaceRef (C:\...\wetzel\lib\replaceRef.js:54:19)

Why isn't it loading a.json and how do I make it find the references? Is my understanding of the -i option incorrect?

I tried hand-wavily adding -s schemas as well, but the result was the same.

Thanks!

Consider a refactoring for the tests

The approach for the tests in wetzel is explained in a comment in test.js : Based on a set of example schemas, the tests consist of automatically generating the property reference in different configurations, and comparing the resulting files with the "golden" output that is checked in into the repository. The exact inputs and parametrizations are summarized in the index.json.

The example schemas for the tests are apparently based on glTF. But additional features have been added to some of these schema files, and it is not always clear which aspect of the schema files are supposed to cover which functionality. One specific example: The image.schema.json is largely a glTF image, but contains some test for fractions. I think that it could make sense to break these tests down into smaller pieces that have a "semantic meaning". For example, these 'fractions' could be tested with a dedicated fractions.schema.json.

Other aspects that could be covered with dedicated tests could be

  • circular references
  • nested type definitions
  • the handling of additionalProperties
  • details about strings (patterns, lengths, formats)
  • Maybe important: Subtle differences regarding the JSON schema version. For example, the change of the meaning of minimum/exclusiveMinimum that was done after Draft 04 - see Range for details

The advantage would be that these schemas can be documented via the description, clearly explaining which aspect of the schema is tested there, and that it could be easier to apply specific changes or add and test specific new functionality.

There are some questions that will certainly come up either in this process, or in the medium term in general:

  • Which parts of JSON Schema are supposed to be supported in the first place?
  • What should the generated documentation look like, exactly?
  • In how far should details of the generated result be configurable (e.g. via CLI parameters)?

But I think that a few, first steps for creating such a set of example schemas could be done independently.

(Note: all this could be done as a pure addition to the current tests. But we might as well try to "clean up" the current schemas so that they more closely resemble the relevant parts of the current glTF schema, and use this part as a more coarse-grained "integration test". And these changes would solely be on the test schemas, and not affect any part of the actual schema generation code, just to avoid regressions)

Add page break before appendix / section headers?

@emackey Suggest turning the ''' markup (horizontal rule) preceding the generated appendix title and each appendix section title into <<< (page break). The way it's working now, the long schema listings tend to force a page break after the title in the PDF, which looks weird.

No object recursion

If I have a schema such as:

    "level_1": {
      "title": "level_1",
      "description": "level_1 description",
      "type": "object",
      "properties": {
        "level_2": {
          "title": "level_2",
          "description": "level_2 description",
          "type": "object",
          "properties": {
            "level_3": {
              "title": "level_3",
              "description": "level_3 description",
              "type": "object",
              "properties": {
                "level_4": {
                  "title": "level_4",
                  "description": "level_4 description",
                  "type": "string"
                }
              }
            }
          }
        }
      }
    }

Only the top level (level_1) is documented.

Handle circular refs in schema

I use a circular model for my JSON schema. So as an example JSON, something like this:

{
  "guides": [{
    // This can go on forever recursively
     "children": [{
        "children": [{
           "name": "My Item"
        }]
     }]
  }]
}

Here is JSON schema

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "definitions": {
        "DocsConfigNavigation": {
            "anyOf": [
                {
                    "$ref": "#/definitions/DocsConfigNavigationPage"
                },
                {
                    "$ref": "#/definitions/DocsConfigNavigationCategory"
                }
            ]
        },
        "DocsConfigNavigationCategory": {
            "properties": {
                "children": {
                    "items": {
                        "$ref": "#/definitions/DocsConfigNavigation"
                    },
                    "type": "array"
                }
            },
            "required": [
                "children"
            ],
            "type": "object"
        },
        "DocsConfigNavigationPage": {
            "properties": {
                "name": {
                    "type": "string"
                }
            },
            "type": "object"
        }
    },
    "properties": {
        "guides": {
            "items": {
                "$ref": "#/definitions/DocsConfigNavigation"
            },
            "type": "array"
        }
    },
    "type": "object"
}

This results in the following error:

$ wetzel docs.schema.json
/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/defaultValue.js:14
function defaultValue(value, fallback) {
                     ^

RangeError: Maximum call stack size exceeded
    at defaultValue (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/defaultValue.js:14:22)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:28:24)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:96:32)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:96:32)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:90:16)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:96:32)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:96:32)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:96:32)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:90:16)
    at replaceRef (/Users/brunnel6/.npm-packages/lib/node_modules/wetzel/lib/replaceRef.js:96:32)

Handling of allOf types

While trying to do a seemingly simple fix ( KhronosGroup/glTF#1013 ), I noticed that the issue refers to other properties as well.

The general question is: How are allOf type definitions supposed to be handled?

In a recent issue in glTF, people argued that true multiple inheritance should not be allowed in glTF anyhow. Nevertheless, even for single inheritance, the types are listed in an allOf property.

In many cases, this is not so critical - specifically, for glTF, because glTFProperty, glTFChildOfRootProperty and glTFid are usually omitted via the command line or boil down to object/integer anyhow. But for other properties (textureInfo, camera.orthographic, material.pbrMetallicRoughness etc) this is relevant.

I think that it could be easy to fix it for the cases where allOf only contains a single element. I could add a check in https://github.com/AnalyticalGraphicsInc/wetzel/blob/46878d1f97c132ed7a01a3c05dedd6ec119d3c7e/lib/generateMarkdown.js#L472 , roughly like

if (property.has("allOf")) {
    if (property.allOf.length != 0) printSomeWarning(?);
    else return property.allOf[0].type;
}

Otherwise, it would be necessary to pass not a single type (and formattedType) through the downstream pipeline, but always use a types[] array, including some contortions for properly generating the bullet point lists in the markdown for the property detail descriptions ...

Would it be OK to assume that allOf only contains a single type, and console.log a warning otherwise?

(One could also add this warning somewhere in https://github.com/AnalyticalGraphicsInc/wetzel/blob/46878d1f97c132ed7a01a3c05dedd6ec119d3c7e/lib/schema4Resolver.js#L79 , if this is the preferred way...)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.