unifiedjs / unified Goto Github PK

View Code? Open in Web Editor NEW

4.2K 33.0 105.0 749 KB

☔️ interface for parsing, inspecting, transforming, and serializing content through syntax trees

Home Page: https://unifiedjs.com

License: MIT License

JavaScript 87.23% TypeScript 12.77%

syntax-tree ast cst javascript processor plugins vfile unist

unified's People

Contributors

Stargazers

Watchers

Forkers

zkochan damianof alexxnica kryndex jdrew1303 cbforks miguelramosfdz ajoslin woodycharming melioratus stgerman kimballjohnson bluepine carrotzpc johnking christianmurphy tarsbase bsyslient shaunstanislauslau devhttps doniyor2109 bradparks spencerx aomcloudenterprise rokt33r swinx bhanditz matt-d-brown the-cc-dev diessica osdiab jamesmessinger godfreyd tokyoshare marko-knoebl jamiek-acl dissident-app sandhose millette doc22940 katerinapal begueradj honai bmorelax meesrutten lokeshn yunwuxin zhangaz1 zg-lib watervenice7 enricopolanski sshyran therakeshpurohit rodrigoieh armenr mummybot magicianred bertho-zero unpolinomio guiltydolphin daveek jsbrain leocete sosukesuzuki guilhermesantos001 acse-zc220 carltonhenderson beckzayi alvinleung1996 artyompsofteq takazudo jasjamjos quant-daddy lokhmakov jablko ericdmoore fansonly unifiedts eploko ndinhbang littlegreay amy-juan-li spread0x anima-os-dev crguezl 00mjk webdevsharex zenryokukun anchan828 hedaayat vitaly-z kkpan11 zignis grogogoll x-oss-byte cappo hkshell guzily john-rice teleki

unified's Issues

tsc reports incorrect / missing types

Subject of the issue

$ yarn tsc
node_modules/@types/vfile/index.d.ts:11:31 - error TS7016: Could not find a declaration file for module 'vfile-message'. '/{path to my project}/node_modules/vfile-message/index.js' implicitly has an 'any' type.
  Try `npm install @types/vfile-message` if it exists or add a new declaration (.d.ts) file containing `declare module 'vfile-message';`

11 import * as vfileMessage from 'vfile-message';
                                 ~~~~~~~~~~~~~~~

Your environment

$ yarn why vfile-message
yarn why v1.22.0
[1/4] Why do we have the module "vfile-message"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "[email protected]"
info Reasons this module exists
   - "stylelint#postcss-markdown#remark#unified#vfile" depends on it
   - Hoisted from "stylelint#postcss-markdown#remark#unified#vfile#vfile-message"
info Disk size without dependencies: "20KB"
info Disk size with unique dependencies: "36KB"
info Disk size with transitive dependencies: "36KB"
info Number of shared dependencies: 1
=> Found "@types/vfile-message#[email protected]"
info This module exists because "stylelint#postcss-markdown#remark#unified#@types#vfile#@types#vfile-message" depends on it.
info Disk size without dependencies: "28KB"
info Disk size with unique dependencies: "60KB"
info Disk size with transitive dependencies: "60KB"
info Number of shared dependencies: 2
Done in 1.18s.

Steps to reproduce

Be using TypeScript.
Check Types.

Tell us how to reproduce this issue. Please provide a working and simplified example.

repo here: https://github.com/NullVoxPopuli/emberclear/pulls

cd packages/frontend
yarn tsc

Expected behaviour

no type failures from vfile-message

Return type of `stringify` could be not a `string`

The TypeScript definition specifies it as string.
But - if you use eg. rehype-react as the last stage of the pipeline, then the return type is actually a React.VNode.
This causes TS to issue an error.

Not sure what's the best solution here.

Specify the "stringed" type as a generic parameter somewhere?
Make the return type any?

Cannot use the same plugin multiple times in a processor

I have a use case where I want to apply the same plugin twice in a processor which doesn't seem to work

Steps to reproduce

Here's a simple example that uses rehype-format twice. Only the first use seems to succeed:

const unified = require("unified");
const rehypeParse = require("rehype-parse");
const rehypeFormat = require("rehype-format");
const rehypeDocument = require("rehype-document");
const rehypeStringify = require("rehype-stringify");

const processor = unified()
  .use(rehypeParse, { fragment: true })
  .use(rehypeFormat)
  .use(rehypeDocument)
  .use(rehypeFormat)
  .use(rehypeStringify);

const input = "<div><h1>hello world</h1></div>";

const output = processor.processSync(input).toString();
console.log(output);

this produces the following output:

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

<div>
  <h1>hello world</h1>
</div>

</body>
</html>

note that the original content was formatted, but the rest was not - the second pass of formatting did not occur

The same behavior occurs with other plugins

Expected behavior

Allow plugins to be used multiple times

If that's not possible, it would be nice if attempting to use a plugin more than once threw an error

Alternatively, this should at least be documented with possible workarounds (I didn't find documentation on this)

this.use() doesn't respect plugin order when combined with unified().use()

Calling unified with

unified().use(function() {
    this.use(plugin1)
}).use(plugin2)

seems to execute the plugins in the order pluign2 then plugin1. Is this the intended behavior? In my case, both plugins are transformer plugins.

Use one `settings` object per processor

proc.data('settings', {a: true, b: false})

Then, presets with settings (GH-22) can extend that object.

Additionally, options should be removed from parse / stringify / process / pipe.

@types packages should be in devDependencies

Subject of the issue

This package includes @types/unist and @types/vfile as dependencies

Your environment

MacOS 10.14.3
node v10.15.2, yarn 1.13.0
Package [email protected] (from Create React App) includes unified

Steps to reproduce

yarn add unified
yarn why @types/node

Expected behaviour

No @types packages should be installed from adding unified.

Actual behaviour

Adding unified adds @types/unist, @types/vfile, @types/node and other packages into node_modules.

Allow `parser`, `compiler` as non-constructors

Currently, parser and compiler must be constructors whose instances have a parse or compile function, respectively.

It makes sense in some cases, such as retext-stringify, rehype-parse, rehype-stringify, to allow them to return a tree or stringified document immediately.

var proc = unified();

proc.use(parse, {b: true}).use(stringify, {c: true}).data('settings', {a: true});

proces.process('foo', console.log);

function parse(options) {
  var proc = this;
  proc.parser = parser;
  function parser(contents, file) {
    var settings = xtend({e: true}, options, proc && proc.data ? proc.data('settings') : {});
    // {a: true, b: true, e: true}
    return {type: 'bar', value: contents};
  }
}

function stringify(options) {
  var proc = this;
  proc.compiler = compiler;
  function compiler(tree, file) {
    var settings = xtend({f: true}, options, proc && proc.data ? proc.data('settings') : {});
    // {a: true, c: true, f: true}
    return tree.value;
  }
}

Logo not readable on github dark mode

Subject of the issue

The logo of Unified is not well displayed in the Readme of the repository when I use Github's dark-theme

Your environment

Firefox, Github dark theme

Steps to reproduce

Enable Github Dark Theme, and open the repository page.

Expected behavior

The logo should be fully readable

isarray warns on install

The isarray dependency throws a warning on install to just use Array.isArray() directly.

As I don't see an engines field in the package.json saying this needs to be compatible with an ancient version of Node.js that doesn't have that, it can probably just be removed.

3rd overload signature of `FrozenProcessor.run` should accept `ParseTree` instead of `CompileTree`

Initial checklist

I read the support docs
I read the contributing guide
I agree to follow the code of conduct
I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions

unified 10.1.0

Link to runnable example

No response

Steps to reproduce

The 3rd overload signature of FrozenProcessor.run is incorrect.

unified/index.d.ts

Lines 307 to 321 in c3ba2cd

 /** 

  * Run transforms on the given node. 

  * 

  * @param node 

  * Tree to transform. 

  * @param file 

  * File associated with `node`. 

  * `VFile` or anything that can be given to `new VFile()`. 

  * @returns 

  * Promise that resolves to the resulting tree. 

  */ 

 run( 

 node: Specific<Node, CompileTree>, 

 file?: VFileCompatible | undefined 

 ): Promise<Specific<Node, CompileTree>>

Expected behavior

run(
    node: Specific<Node, ParseTree>,
    file?: VFileCompatible | undefined
): Promise<Specific<Node, CompileTree>>

Actual behavior

run(
    node: Specific<Node, CompileTree>,
    file?: VFileCompatible | undefined
): Promise<Specific<Node, CompileTree>>

Runtime

Node v16

Package manager

Other (please specify in steps to reproduce)

OS

Linux

Build and bundle tools

Webpack

Don’t pass `processor` as argument to `attacher`s

Attachers should be invoked with the processor as the context object, instead of a parameter.

Often, attachers don’t even need to interact with the processor, so it makes no sense to pass it first.
Removing it from the arguments makes it easier to use attachers to without unified, such as in the following code:

var toc = require('remark-toc')();
var tree = {type: 'root', children: [/* … */]};
console.log(toc(tree));

Async 'process' callback argument order wrong

According to the documentation, when calling process async, it will call its callback with the arguments (err, result, file). It actually calls it with the arguments (err, file, result) link

The same mistake is in remark's documentation

dtslint silently started failing

Subject of the issue

dtslint currently fails:

> [email protected] test-types unified
> dtslint types

Error: unified/types/index.d.ts:3:1
ERROR: 3:1     strict-export-declare-modifiers  All declarations in this module are exported automatically. Prefer to explicitly write 'export' for clarity. If you have a good reason not to export this declaration, add 'export {}' to the module to shut off automatic exporting. See: https://github.com/Microsoft/dtslint/blob/master/docs/strict-export-declare-modifiers.md
ERROR: 4:1     strict-export-declare-modifiers  All declarations in this module are exported automatically. Prefer to explicitly write 'export' for clarity. If you have a good reason not to export this declaration, add 'export {}' to the module to shut off automatic exporting. See: https://github.com/Microsoft/dtslint/blob/master/docs/strict-export-declare-modifiers.md
ERROR: 5:8     strict-export-declare-modifiers  All declarations in this module are exported automatically. Prefer to explicitly write 'export' for clarity. If you have a good reason not to export this declaration, add 'export {}' to the module to shut off automatic exporting. See: https://github.com/Microsoft/dtslint/blob/master/docs/strict-export-declare-modifiers.md
ERROR: 7:19    strict-export-declare-modifiers  All declarations in this module are exported automatically. Prefer to explicitly write 'export' for clarity. If you have a good reason not to export this declaration, add 'export {}' to the module to shut off automatic exporting. See: https://github.com/Microsoft/dtslint/blob/master/docs/strict-export-declare-modifiers.md
ERROR: 412:18  strict-export-declare-modifiers  All declarations in this module are exported automatically. Prefer to explicitly write 'export' for clarity. If you have a good reason not to export this declaration, add 'export {}' to the module to shut off automatic exporting. See: https://github.com/Microsoft/dtslint/blob/master/docs/strict-export-declare-modifiers.md
ERROR: 413:1   strict-export-declare-modifiers  All declarations in this module are exported automatically. Prefer to explicitly write 'export' for clarity. If you have a good reason not to export this declaration, add 'export {}' to the module to shut off automatic exporting. See: https://github.com/Microsoft/dtslint/blob/master/docs/strict-export-declare-modifiers.md

Your environment

OS: macOS Mojave 10.14.6
Packages: latest
Env: Node 12.2, npm 6.11

Steps to reproduce

git clone, npm install, and npm test.

Expected behaviour

Shouldn’t fail

Actual behaviour

Fails

/cc @ChristianMurphy @Rokt33r

Webpack support

Your basic markdown example isn't working in a browser.

var unified = require('unified');
var markdown = require('remark-parse');
var remark2rehype = require('remark-rehype');
var doc = require('rehype-document');
var format = require('rehype-format');
var html = require('rehype-stringify');

unified()
  .use(markdown)
  .use(remark2rehype)
  .use(doc)
  .use(format)
  .use(html)
  .process('# Hello world!')
  .then(function (file) {
    console.log(String(file));
  }, function (err) {
    console.error(String(err));
  });

I will get an error

ReferenceError: process is not defined
    at new VFile (VM11132 core.js:47)
    at VFile (VM11132 core.js:41)
    at executor (VM11126 index.js:369)
    at Function.process (VM11126 index.js:366)

The problem is this line in vfile

this.cwd = process.cwd();

When commenting this line out everything is fine. Is this the expected behavior? I don't see anything in the docs that unified is only made for node.

types: revist transformer typescript interface

Subject of the feature

Update the Transformer interface to better reflect the types being passed in.

Problem

Transformer expects to receive exactly a unist Node

unified/types/index.d.ts

Lines 296 to 316 in e408747

 /** 

  * Transformers modify the syntax tree or metadata of a file. A transformer is a function which is invoked each time a file is passed through the transform phase. 

  * If an error occurs (either because it’s thrown, returned, rejected, or passed to `next`), the process stops. 

  * 

  * The transformation process in unified is handled by `trough`, see it’s documentation for the exact semantics of transformers. 

  * 

  * @param node Node or tree to be transformed 

  * @param file File associated with node or tree 

  * @param next If the signature of a transformer includes `next` (third argument), the function may finish asynchronous, and must invoke `next()`. 

  * @returns 

  * - `Error` — Can be returned to stop the process 

  * - `Node` — Can be returned and results in further transformations and `stringify`s to be performed on the new tree 

  * - `Promise` — If a promise is returned, the function is asynchronous, and must be resolved (optionally with a `Node`) or rejected (optionally with an `Error`) 

  */ 

 interface Transformer { 

 ( 

 node: Node, 

 file: VFile, 

 next?: (error: Error | null, tree: Node, file: VFile) => {} 

 ): Error | Node | Promise<Node> 

 }

When in most cases parsers will return a RootNode which extends ParentNode which adds an extra children attribute that Node does not expect.

remark ran into a similar issue to this with the visitor interface, see remarkjs/remark#426

Expected behaviour

Ideally avoid requiring as casting the node parameter.

Alternatives

do nothing, casting may be an okay option
use a generic <T extends Node> to guarantee the incoming node is a super type of Node, though it may or may not reflect the correct super type.
Allow Processor to accept a generic of all possible types (E.G. all mdast nodes) and pass that generic type union in (Open question: How would plugins add new types to the union?)
Other ideas?

React Native support

Subject of the issue

If you use unified on React Native app, you get following error while compilation:

error: bundling failed: Error: Unable to resolve module `path` from `/Users/.../node_modules/vfile/core.js`: Module `path` does not exist in the Haste module map

vfile requires path but there is no such module on RN app.

Your environment

I just wanted to parse Markdown with remark.
A workaround is to install path npm module but I report it just in case if there is a better solution.

Throw on preset w/ neither `settings` nor `plugins`

Initial checklist

I read the support docs
I read the contributing guide
I agree to follow the code of conduct
I searched issues and couldn’t find anything (or linked relevant results below)

Problem

mdx-js/mdx#2136

Solution

Throw here when neither plugins nor settings is defined.

Alternatives

Keep as is.

I don’t imagine most folks want empty presets being silently ignored.

For unified-engine, we treat config files as presets as well (though importing plugins-as-string). There might be a tiny use case there for empty config files, as they could “restore-to-defaults” a folder, if a higher folder has a config file that prescribes complex things.
This would break that use case.
But I dunno, I don’t think people actually do this?

Dependencies need to be ESM

Initial checklist

I read the support docs
I read the contributing guide
I agree to follow the code of conduct
I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions

all

Link to runnable example

No response

Steps to reproduce

Provided in sveltejs/kit#1961

Expected behavior

All dependencies should be ESM since this package declares "type": "module". The package should work out-of-the-box with SvelteKit and Vite

This can be fixed by distributing an ESM version of is-buffer and extend or by removing those dependencies from this library since they have minimal usage. I've sent a PR for is-buffer feross/is-buffer#43 though I don't know yet whether it will be accepted

This could be added a sub-task to #121

Actual behavior

Vite encounters CJS dependency of this ESM module and fails with the message module is not defined as described in vitejs/vite-plugin-react#30

Runtime

Node v14, Other (please specify in steps to reproduce)

Package manager

npm v7

OS

Linux

Build and bundle tools

Vite

Prevent double configuration for the same plugin

unified().use([[plugin, {foo: true, bar: true}], [plugin, {foo: false, qux: true}]]);

...should not attach the same plugin twice. Rather, the config should be merged as if the following was given:

unified().use(plugin, {foo: false, bar: true, qux: true});

This would allow, for example, some configuration for remark-lint to live in a preset, whereas other configuration is passed by the user:

unified().use({plugins: [
  require('remark-lint-preset-recommended'),
  [require('remark-lint'), {finalNewline: false}]
]});

Unfortunately it’s not possible to merge configuration from different .use calls:

unified().use(plugin, {foo: true, bar: true}).use(plugin, {foo: false, qux: true});

😒

Refactor `use` mechanics for multiple plug-ins

The current support is wonky in some places. But, the fix will be in a major release.

Support for accepting a processor as argument to use() is wrong;
Support for a list of plug-ins and options is not intuitive. Accepting a list
of plugin--options tuples is more logical.

Stop allowing extraSettings

Yes, it is possible that there are third and other parameters. On the API side, unified and the engine support it, but it’s not used by the plugins developed inside the collective. The second parameter is used, such as by remark-rehype, and the engine passes a second parameter, which is only used by remark-validate-links. I’m not very happy with how second parameter of the engine works, and would like to change it (maybe in “uniflow”?)

@wooorm mentioned in #58 (comment)

I think allowing extraSettings is giving too much unnecessary freedom.

For example, remark-rehype is accepting two optional settings, a function for a destination processor and an object for ToHast settings. So remark-rehype should check the first argument's type if it is a function or an object. And we need to provide extra types like the below....

function remark2rehype(settings?: ToHastSettings)
function remark2rehype(destination?: Processor, settings?: ToHastSettings)
function remark2rehype(destinationOrSettings?: Processor | ToHastSettings, settings?: ToHastSettings)

In my opinion, it might cause a little tiny bit of confusing for plugin providers and adopters. Haha

If we don't allow extra settings, remark-rehype could be coerced to become a bit simpler like the below example.

interface RemarkRehypeSettings extends ToHastSettings {
  destination: Processor
}

function remark2rehype(settings?: RemarkRehypeSettings)

So, I'm down with this idea. But I also admit that this is a very trivial issue so we can just ignore.

Support `presets`

Currently, presets (an object with two keys: plugins and settings) are only available on the engine and related projects. Users have been asking for this to be added to unified’s core and I think that’s a very nice addition to the ecosystem.

plugins is a matrix/list/plugins;
settings are an object that‘s now passed to parse/run/stringify/process.

This does mean that now settings need to be stored on processors. That makes sense: to configure options in the same phase as .use() calls. The current way doesn’t: to pass them to methods, allowing different settings per call.

Problem with unused library functions

My name is Hernan; with a group of colleagues we are conducting a research about unused code present in dependencies of JavaScript projects. We call this functions, UFF (Unused foreign functions). We found that in most projects there exist a great amount of UFF that are being included in the final bundle.

In the case of unified (v 5.1.0) our tools detected approximately 25 unused function in dependencies. Removing those functions, the size of unified bundled could be reduced at least 10% (All tests passed). I’m attaching the reduced version of your project.
unified(optimized).txt

I’ll be very grateful if you can answer me the following questions:
-Did you were aware of the existence of these unused functions in your projects?
-Do you think that this is a problem?
-Do you think that can be useful a tool for deal with this kind of problem?

Thanks in advance.

Cheers,

VFile types do not support property result

The following commit distinguishes the return value of process and processSync as follows.

c3ba172

If the result a string, it is placed on the property contents, if not on result.

Currently the VFile Typescript Definition only defines contents. To achieve compatibility of the above described behavior the Typescript must be extended locally, this is of course not pretty.

I don't know how the communication to the VFile project is and if a change of this kind complies with the VFile specifications.

Thanks!

Support async compilers

Initial checklist

I read the support docs
I read the contributing guide
I agree to follow the code of conduct
I searched issues and couldn’t find anything (or linked relevant results below)

Problem

Prettier 3.0.0-alpha.0 was just released. This makes all public APIs asynchronous.

remark-prettier uses those APIs in a unified compiler. It would be nice if this can keep working when it updates Prettier.

Solution

In addition to a string of buffer, a compiler may also return a promise that resolves to a string or buffer.

Of course just like with transformers, an async compiler means the unified pipeline no longer supports the synchronous methods.

Alternatives

remark-prettier could use deasync, but IMO that’s not a nice solution.

Component/duo support, try catch?

Can you explain the idea behind the try/catch for a module that isn't part of the package in 3eacf6a? I get that the result is that parent apps may supply their own copy of node-extend by depending on it themselves, but I can't tell why they'd want to do that. I also don't know what "component/duo support" means, and it's hard to work out what's going on just from the diff.

My use case is that I have this module in my dependency tree (via mdast) in a React Native app, and the React Native packager doesn't really love modules that cannot be resolved, so I'm trying to figure out the best way to handle this. It looks like the node-extend module is quite old and the github repo doesn't even exist anymore, so I don't particularly want to add it to my app.

Thanks.

Define mutators

“Mutators” (working title) transform the syntax tree from one
processor to another processor.

Typical transformation between languages and syntax trees occur
something like the following

     / -- mdast -- \                  / -- hast ...
    /               \                /
   /                 \              /
parse            remark-html     parse
                       \          /
          remark        \ ------ /         hast

With a mutator, the following is possible:

     / -- mdast -------- mutator --------- hast ...
    /
   /
parse

         remark                            hast

Mutators are a construct which skips the Compiler of the origin
processor and the Parser of the destination processor. Instead,
implementing a tree transformation between the run phase of both
processors. Mutators transform the document to the language
the destination processor is in (in the aforementioned examples,
they transform from markdown to HTML).

Parser & Compiler can't be ES6 classes

The following function doesn't detect ES6 classes as constructor functions:

unified/index.js

Lines 403 to 405 in 612d8dd

 function newable(value) { 

 return typeof value === 'function' && keys(value.prototype) 

 }

It should be more something like:

const isConstructorFunction = (value) => Boolean(value && value.prototype && value.prototype.constructor);

plugin transformer does not support async functions

node 8+

Steps to reproduce

function plugin() {
  return async function transformer(tree) {
    tree.children = await somethingAsync()
    return tree
  }
}

Expected behaviour

Expect to be the same as

function plugin() {
  return function transformer(tree) {
    return somethingAsync()
      .then((result) => {
        tree.children = result
        return tree
      })
  }
}

Actual behaviour

The plugin is skipped.

Root cause

serapath/x-is-function#1

TypeError: done is not a function

This code is throwing this error

TypeError: done is not a function
    /Users/moox/Sync/Development/phenomic/node_modules/unified/index.js:391:9
    next (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:83:14)
    done (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:163:12)
    then (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:174:5)
    wrapped (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:149:9)
    next (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:81:24)
    done (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:163:12)
    /Users/moox/Sync/Development/phenomic/node_modules/unified/index.js:38:9
    /Users/moox/Sync/Development/phenomic/node_modules/unified/index.js:323:21
    next (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:83:14)
    done (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:163:12)
    then (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:174:5)
    wrapped (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:149:9)
    next (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:81:24)
    Object.run (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:48:10)
    Function.run (/Users/moox/Sync/Development/phenomic/node_modules/unified/index.js:319:18)
    /Users/moox/Sync/Development/phenomic/node_modules/unified/index.js:32:7
    wrapped (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:128:19)
    next (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:81:24)
    done (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:163:12)
    then (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:174:5)
    wrapped (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:149:9)
    next (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:81:24)
    Object.run (/Users/moox/Sync/Development/phenomic/node_modules/trough/index.js:48:10)
    Function.process (/Users/moox/Sync/Development/phenomic/node_modules/unified/index.js:380:14)
    description (/Users/moox/Sync/Development/phenomic/src/content-loader/description.js:45:74)
    Test.fn (description.js:24:15)

It seems to be new and this code "was working" a few days ago. Any idea? I can see from git history that you made some changes/fixes related to trough and "done".

Move projects over to `unifiedjs` organisation

Readme on npm link to homepage not working

Subject of the issue

npm README links to https://unified.js.org/ which is broken link

Add list of processors

...and mutators, bridges when they’re there.

Make VFileContents generic to support processors that return objects

Subject of the feature

Right now, compilers that return objects (DOM nodes, React nodes, etc.) can not be type checked correctly. I opened an issue on vfile side to make the VFileContent generic. See vfile/vfile#45.

Once fixed on vfile side, we should extend the Processor as well as the types related to plugins (Plugin, Pluggable, Preset, etc.) to have a VFile or VFileContent type parameter.

Related to #47

Problem

Compilers that do not return text are not type checkable.

Expected behaviour

Compilers should be able to override the VFileContent type somehow.

Alternatives

Casting the output of process[Sync]().contents, although it might not work with some strict TS compiler options/linters?

unify AST types

Initial checklist

I read the support docs
I read the contributing guide
I agree to follow the code of conduct
I searched issues and couldn’t find anything (or linked relevant results below)
i hate checklists

Problem

xast-parse produces a different AST than hast-parse
so i cannot use hast-util-select on xast

part of rehypejs/rehype#112 (trying to use hast tools for xml)

Solution

unify ASTs across all parsers in the unifiedjs ecosystem
so that tools like select "just work" on all ASTs

interface Node {
  name: string,
  type: string,
  props: Record<string, any> | null,
  children: Node[] | null, // node is a branch node
  value: string?, // node is a leaf node
  location: { start: number, end: number },
  internal: Record<string, any> | null, // like props, but for unifiedjs internal data
  // example: path to source file of a root node
  isRoot: boolean, // shortcut to detect "context switch" between different ASTs
}

~~this would also allow to just "plug in" tools like graphQL, or interface with graph databases ...~~

benefit: ASTs are composable, for example, i can embed a python AST in a html code block.
or, i can parse <img src="image.svg">, parse the svg file at src, transform the svg, inline the svg in html

maybe extend the Node type of a parser-generator like tree-sitter or lezer-parser

Alternatives

write N tools for M ASTs ...

which is the opposite of "unified"

"what would pandoc do?"

examples of tools?

Hello, I'm just wondering... I feel like a while ago I saw a tool that would highlight unnecessary words in a piece of text. And a few other examples of interesting prose-level tooling built on top of Unified. I could have sworn I found them via the Unified website, but cannot find them anymore.

Did these exist? Do they still exist?

(This is vague, I'm sorry!)

Browser support

The dependency "is-plain-obj" npm package is not compatible with ...... IE

tl;dr is-plain-object seems like a good alternative.

Unlike other packages, "is-plain-obj" is publishing arrow functions without transpiling. It will cause the applications that use IE as the javascript engine failing to load unifiedjs. For my case, it's on-premise MS Word 2019. In addition, although "is-plain-obj" is a 4 line function, it's claiming to support node8+ only. I believe unifiedjs is aiming to support both browser and backends, so a backend only package might not be a good dependency.

FYI, my use case is writing a text processor as an Add-on for MS Word with typescript and unified. My current workaround is using babel to load this package first in my webpack.config.js. Which is obviously an anti-pattern because I'm compiling the package in node_modules

      rules: [
        {
          test: /node_modules\/.*is\-plain\-obj.*\.js$/,
          use: {
            loader: "babel-loader",
            options: { presets: ["@babel/preset-env"] },
          },
        },

Your environment

OS: Windows 10
Packages: unified, webpack
Env: IE11

Steps to reproduce

Create a minimal project

#!/bin/bash
mkdir -p minimal-project/src
cd minimal-project
echo '<script src="main.js"></script>' > index.html
echo 'import unified from "unified"' > src/index.js
yarn add unified webpack webpack-cli webpack-dev-server
./node_modules/.bin/webpack-dev-server --mode development

Open Intenet Explorer and navigate to http://localhost:8080

Then see syntax error from is-plain-obj

Expected behaviour

unifiedjs should load correctly

Actual behaviour

The 'is-plain-obj' npm package contains ES6 syntax

Define bridges

“Bridges” (working title) transform the syntax tree from one
processor to another processor. Then, they apply the destination
processors plug-ins, and transform the tree back into the origin
processor and continue on running its plug-ins. Bridges can
be implemented as a plug-in for the origin processor accepting
a destination processor.

Typical transformation between languages and syntax trees occur
something like the following (although it’s not really possible yet)

     / -- mdast ---------------- ... ---------------------------------- ...
    /              \                               nlcst -- ...
   /                \                              /
parse                \                          parse
 /                    \                          /
/         remark       \ -- strip-markdown ---- /  retext               remark

With a bridge, the following is possible:

     / -- mdast -- ... -- bridge: enter -- nlcst -- ... -- bridge: exit -- ... -- mdast -- ...
    /
   /
parse
 /
/        remark                            retext                                 remark

Bridges provide support for tree transformations in another language.
Then, they run the destination plug-ins on that tree.

This is especially useful for languages embedded in others, like
natural language. Or, for example, JSDoc comments inside JavaScript.

Note that the exit step can be omitted (currently the case for
mdast > nlcst, which doesn’t have exit support).

write after end

Subject of the issue

Trying to install remark causes an error thrown by npm.

Environment

macOS 10.14 Mojave, remark (latest), node v10.1.0 (npm 6.0.1)

Steps to reproduce

npm i remark

Expected behavior

Do I have to explain this?

Actual behavior

oganDark@base ~/nitrogen> npm i --save remark remark-html
events.js:167░░░░░░⸩ ⠦ extract:remark-stringify: sill extract [email protected]
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:292:5)
    at PassThrough.Writable.end (_stream_writable.js:582:10)
    at ReadEntry.entry.on (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:296:25)
    at ReadEntry.[maybeEmitEnd] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:249:12)
    at ReadEntry.end (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:162:27)
    at Unpack.[consumeBody] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:296:25)
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:99:17)
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:292:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167░░░░░░⸩ ⠏ extract:property-information: sill extract [email protected]
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:292:5)
    at PassThrough.Writable.end (_stream_writable.js:582:10)
    at ReadEntry.entry.on (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:296:25)
    at ReadEntry.[maybeEmitEnd] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:249:12)
    at ReadEntry.end (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:162:27)
    at Unpack.[consumeBody] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:296:25)
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:99:17)
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:292:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167░░░░░░⸩ ⠏ extract:kebab-case: timing audit body Completed in 7ms
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:292:5)
    at PassThrough.Writable.end (_stream_writable.js:582:10)
    at ReadEntry.entry.on (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:296:25)
    at ReadEntry.[maybeEmitEnd] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:249:12)
    at ReadEntry.end (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:162:27)
    at Unpack.[consumeBody] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:296:25)
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:99:17)
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:292:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/Users/LoganDark/.nvm/versions/node/v10.1.0/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
npm ERR! cancel after 1 retries!

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/LoganDark/.npm/_logs/2018-06-07T00_56_49_797Z-debug.log
⏎

p.s. you spelled "behavior" wrong in the template. /s :^)

Broken TypeScript definitions

Subject of the issue

The TypeScript type definitions were refactored in PR #49, but the new definitions are invalid due to the use of export default instead of export =. See the TypeScript documentation to understand the difference between the two.

Your environment

OS: Windows
Packages:
- unified v8.0.0
- typescript: 3.5.2
Env:
- Node v12.2.0
- NPM v6.9.0

Steps to reproduce

You can see this bug in the wild in the rehype-inline-svg plugin.

Clone the repo

git clone https://github.com/JS-DevTools/rehype-inline-svg.git
Install dependencies

npm install
Build the code

npm run build

The build will succeed because the repo is currently using Unified v7.1.0 which has the old TypeScript definitions, which correctly used export = syntax.
Upgrade Unified
npm install unified@8
Build the code again

npm run build

This time the build will fail because of the new type definitions with incorrect export default syntax.

Expected behaviour

Option 1
The TypeScript definitions should correctly reflect the fact that Unified exports a CommonJS module, not an ESM module.

Option 2
Change Unified to export an ESM module instead of, or in addition to a CommonJS module. This way, the new TypeScript definitions will be correct.

Actual behaviour

TypeScript builds fail because the type definitions are incorrect.

Remove streaming interface from core

Now:

var unified = require('unified');
var parse = require('rehype-parse');
var rehype2remark = require('rehype-remark');
var stringify = require('remark-stringify');

var processor = unified()
  .use(parse)
  .use(rehype2remark)
  .use(stringify);

process.stdin
  .pipe(processor)
  .pipe(process.stdout);

Future:

 var unified = require('unified');
+var stream = require('unified-stream');
 var parse = require('rehype-parse');
 var rehype2remark = require('rehype-remark');
 var stringify = require('remark-stringify');
 
 var processor = unified()
   .use(parse)
   .use(rehype2remark)
   .use(stringify);
 
 process.stdin
-  .pipe(processor)
+  .pipe(stream(processor))
   .pipe(process.stdout);

angular-cli support

Subject of the issue

Because Unified requires node globals & libraries, it cannot be used in Angular applications that use angular-cli >= 6.0.0

See this comment from angular-cli:
angular/angular-cli#9827 (comment)

Your environment

angular-cli >= 6.0

Expected behaviour

I can use unified in my app and ng serve does not fail

Actual behaviour

ng serve fails with

ERROR in ./node_modules/replace-ext/index.js
Module not found: Error: Can’t resolve ‘path’ in ‘/my-project/node_modules/replace-ext’

Next major for the ecosystem

Soon (April), it’s time to switch to ESM (without CJS backup) for all packages in the ecosystem.
That’s a lot of projects switching over, and a round of majors that bubbles from the bottom through to the top.

I’ve had some experience with that recently in a push in micromark (with CJS fallback, causing issues). My recent own projects dioscuri and xdm are fully ESM. It’s going to be a lot of fun for the ecosystem 😅

Otherwise, here are some changes slate for this next bubbling:

Big breaking changes:

Small breaking changes:

hastscript’s h and s and a JSX pragma syntax-tree/hastscript#15
unist-util-find-all-before’s results should be reversed probably syntax-tree/unist-util-find-all-before#6

For ES++ features (unifiedjs/rfcs#4), I’d say we can do that after we switch to ESM, because ESM will already be a lot of work, and it gives us a baseline of what ES features / engines to support.

Add `.runSync()`, `.processSync()`

...and switch to returning promises without callback in .run(), .process().

Supersedes: remarkjs/remark-lint#76.

Which versions of Node are supported?

Subject of the feature

Just add supported node range in engines into package.json

Problem

Hard to undestand what is node supported, for example prettier/prettier#6563 we can't update because we don't know supported versions

Expected behaviour

Field present

Alternatives

Maybe we can just add this information to documentation

Thanks

Change `processor.parser`, `processor.compiler` signatures

parser(vfile, settings, processor) > parser(doc, vfile)
compiler(vfile, settings, processor) > compiler(tree, vfile)

That makes it more similar to transformers, and allows for settings to be removed (GH-23).

var parse = require('./parse')
var proc = unified().use(parse, {b: true}).data('settings', {a: true});

proces.parse('foo', console.log);

parse.js:

module.exports = parser;

Parser.prototype.parse = parse;
Parser.prototype.defaults = {c: true};

// Invoked when attaching.
function parser(options) {
  var proc = this;
  var LocalParser = proc.parser = unherit(Parser);
  LocalParser.prototype.defaults = xtend(LocalParser.prototype.defaults, options);
  LocalParser.prototype.processor = proc;
}

// Constructed just before parsing.
function Parser(contents, file) {
  // allow use without unified.
  var settings = this.processor ? this.processor.data('settings') : {};
  this.doc = contents;
  this.file = file;
  this.settings = xtend(this.defaults, settings);
}

// Constructed when parsing, with `this` as an instance of Parser.
function parse() {
  this.settings; //=> {a: true, b: true, c: true}
  return this.start(); // ...or something
}

Support for using processors

Take the following example:

var prose = unified()
  .use(retextParse)
  .use(equality)
  .use(quotes)

var html = unified()
  .use(minify)
  .use(rehypeStringify)

var markdown = unified()
  .use(remarkParse)
  .use(remark2retext, prose)
  .use(lint, {rules: {/*...*/}})
  .use(remark2rehype)
  .use(html)

Currently, the last line doesn’t work, we have to replace it with:

  .use(minify)
  .use(rehypeStringify)

...which is okay for two lines but when using many plugins it becomes
cumbersome, and hampers reusability.

If passing a processor to use would be supported we walk into a new
problem though: the given processor, if it has a parser or compiler,
would overwrite the origin parser/compiler with them. That’s OK for
the compiler (I think?) but not for the parser.

Accepting TypeScript definitions

Unified seems really cool! Would you be willing to accept TypeScript definition files?

Rethink how `*-stringify` should work

Personally, I love the idea remark-prettier.

Because even with https://github.com/remarkjs/remark-preset-prettier, remark-stringify's default or few custom formatting can still change the input.

For example, there is no option to preserve two spaces:

Line.  <!-- two spaces here -->
Next line

It will always to formated as:

Line.\
Next line

This is very unexpected for prettier users.

Originally posted by @JounQin in #196 (comment)

Update vfile to 4.2.1

Subject of the feature

I'd like you to release a new version of unified with vfile 4.2.1

Problem

vfile 4.2.1 includes this PR. With this change, users are able to use unified (and remark) in browser without path polyfill.

This change is a big improvement but the latest unified (9.2.0) still includes vfile 4.2.0.

Also, many repositories in unified ecosystem ignore package-lock.json and yarn.lock by .gitignore. Without them, the version of dependencies determined by the timing of npm install, so we cannot reproduce the behavior of a specific version. Maybe you should include them in the repository.

Expected behavior

The new version of unified and remark with latest dependencies are released.

	/**
	* Run transforms on the given node.
	*
	* @param node
	* Tree to transform.
	* @param file
	* File associated with `node`.
	* `VFile` or anything that can be given to `new VFile()`.
	* @returns
	* Promise that resolves to the resulting tree.
	*/
	run(
	node: Specific<Node, CompileTree>,
	file?: VFileCompatible \| undefined
	): Promise<Specific<Node, CompileTree>>

	/**
	* Transformers modify the syntax tree or metadata of a file. A transformer is a function which is invoked each time a file is passed through the transform phase.
	* If an error occurs (either because it’s thrown, returned, rejected, or passed to `next`), the process stops.
	*
	* The transformation process in unified is handled by `trough`, see it’s documentation for the exact semantics of transformers.
	*
	* @param node Node or tree to be transformed
	* @param file File associated with node or tree
	* @param next If the signature of a transformer includes `next` (third argument), the function may finish asynchronous, and must invoke `next()`.
	* @returns
	* - `Error` — Can be returned to stop the process
	* - `Node` — Can be returned and results in further transformations and `stringify`s to be performed on the new tree
	* - `Promise` — If a promise is returned, the function is asynchronous, and must be resolved (optionally with a `Node`) or rejected (optionally with an `Error`)
	*/
	interface Transformer {
	(
	node: Node,
	file: VFile,
	next?: (error: Error \| null, tree: Node, file: VFile) => {}
	): Error \| Node \| Promise<Node>
	}

	function newable(value) {
	return typeof value === 'function' && keys(value.prototype)
	}