Load tree from package-lock.json and package.json without reading tree (Ie, just get the presumed tree, without reaffirming from disk)

NB: this should be unblocked now that pacote v10 is ready to go.

Get the list of nodes to remove in an earlier walk so we don't have to walk the diff a second time.

• Add an accept field to the Edge class.
• Update depValid so that a dep is valid if it satisfies the spec or the accept specification. (Note: should depValid take an Edge object rather than a spec? Should it just be a part of the Edge class? It smells like feature envy, but we do often want to know if a dep would be valid, without committing to attaching it into the tree so an Edge points to it, but again, maybe this would be better handled with a method like Edge.wouldBeSatisfiedBy(node) or something?)
• Update Node[_loadDepType] method to include the acceptDependencies range when creating an Edge.

That should be it. Everything else that's done with dep analysis happens by calling depValid with the edge values and target node.

What / Why

When the tree is flattened, deep dependencies are reported as direct dependencies, i.e. the node.parent will be the node.root rather than the dependent.

This may, of course, be as designed, in which case the documentation needs clarification. In that case, I can raise a separate feature request for detecting node.direct dependencies (i.e. ones in package.json, rather than just the top level of the node_modules) - or rename this issue.

When

• n/a

Where

• n/a

How

Current Behavior

• node.parent === node.root when node is an indirect dependency.

Steps to Reproduce

• npm install debug
• arborist.buildIdealTree()
• the Node for ms will have node.parent.isTop === true, even though its parent is debug.

Expected Behavior

• The Node for ms should have node.parent be the Node for debug.

Who

• n/a

References

• n/a

Depends on #13

Provide a way to tell the buildIdealTree process to also build out the dependency trees of link targets.

Necessary for npm workspace update etc.

Right now, node.resolve() does this:

  resolve (name) {
    const mine = this.children.get(name)
    return mine ? mine
      : this.isTop ? null
      : this.parent.resolve(name)
  }

However, we might have a folder structure like this:

root
+-- x (depends on y)
+-- packages
|   +-- y (depends on z and a)
|   |   +-- node_modules
|   |       +-- a
|   +-- z
+-- node_modules
    +-- x -> ../x
    +-- y -> ../packages/y
    +-- z -> ../packages/z

When loading the tree in x, the top of tree will be root/x. No node_modules, so it'll look like it's missing its deps.

Similarly, in packages/y, it'll find the a dep, but not the z dep.

We need a way to identify other points in the fs tree where a dep might be found: ie, any ancestor of the top node which contains a node_modules folder.

Some options:

• Represent the node_modules folder as a special type of node in the tree. Then the "parent" relationship can be a direct folder relationship, and we can say that the root/x node's parent is the root node. Then resolve looks like this:

resolve (name) {
  const mine = this.node_modules && this.node_modules.get(name)
  if (mine) return mine
  if (this.parent.isNodeModules && this.parent.parent)
    return this.parent.parent.resolve(name)
  else if (this.parent)
    return this.parent.resolve(name)
  else
    return null
}

Downside of this is that we'd have to effectively always walk the full tree loading every folder in the path to really do it correctly.

• Define a set of "potential parents" in the loadActual step if a node is top of tree. A potential parent is an ancestor folder containing a node_modules folder. Then resolve() looks like this:

resolve (name) {
  const mine = this.children.get(name)
  if (mine)
    return mine
  else if (this.parent)
    return this.parent.resolve(name)
  else {
    let ppFound
    this.potentialParents.some(pp => ppFound = pp.resolve(name))
    return ppFound
  }
}

The potential parents list would have to be an array, not a Set, because order is important.

The important thing about getting this right in Node.resolve() is that that's how the Edge class keeps itself automatically up to date. Challenge is to get this set up in loadActual so that we're not doing fs stuff in the Node class directly, since that should be completely agnostic about whether it refers to an actual node on disk or a hypothetical node in an ideal tree.

Failed optional deps should remain in the idealTree, but be removed from the actual tree.

Clone the tree (maybe by implementing some kind of Node.clone() method?) removing any nodes whose paths are in this[_trashList].

Need to be able to install global packages with Arborist.

Create top level arborist.prune() method/mixin.

Rough sketch algorithm:

• Load virtual tree, or failing that, actual tree
• For node in tree.inventory, compare node dep flags to requested prune args, according to this chart
• If shouldPrune, then set node.parent = null
• this.reify()

Depends on pacote v10

Add options to omit dev, optional, and/or peer deps in Arborist.reify() (but still include them in the ideal tree and shrinkwrap).

• binLinks - boolean, default true, whether to link bins
• rebuildBundle - boolean, default false, rebuild bundled deps #80
  • This actually snuck on b28a43f, and it's probably good to have true by default rather than false? I think the only reason it's not true by default is that it was kind of an afterthought back in the npm v1 days, and that behavior was just carried forward.
  • Update It actually does default to true in npm v6, not sure where I got the impression it didn't. So that's good. Just have to make it configurable.
• packageLock - boolean, default true, respect/update the package-lock/shrinkwrap file
  • #93
• packageLockOnly - boolean, default false, don't reify anything, just update the lockfile
  • depends on #73, if this config is set reify() calls buildIdealTree({complete:true}) and then saves package{,-lock}.json files
  • #78
• globalStyle - boolean, install as if it's a top-level global package (ie, don't resolve any deps above the top level, except peer deps)
• dryRun - boolean, don't actually touch anything on the filesystem
  • depends on #73. If this config is set, calls buildIdealTree({complete:true}), calculates the diff, and returns diff and ideal tree, without saving anything.
  • #94

1. loadActual
2. loadVirtual
3. buildIdealTree
4. reify

The better to test and organize it with. So that the actual Arborist class looks like:

class Arborist extends actualLoader(virtualLoader(idealTreeBuilder(reifier(EE)))) {
  constructor (options) {
    // set this.whatevers
  }
}
// that's it!

Right now, Arborist.buildIdealTree can add/remove nodes, and will prune by default. However, it really is an "install" and not an "update", in that it only visits the root node, and then any node that has an invalid edge going out from it.

npm update is different:

If a list of names is provided

• Start with the actual tree if there's no shrinkwrap (not an empty tree with just the root node)
• Walk the whole tree no matter what, with the exception of bundled deps. (Ie, don't just start at the root and only add problems to the queue.)
• While we may visit every node, only re-evaluate edges if the name is on the list, or if it has a problem.

If a numeric depth is provided

• Do not continue the "walk the entire tree" process beyond nodes with a depth property > the depth specified.
• Still always evaluate nodes if they're a problem, to any depth (they're likely a problem because of something replaced further up in the tree, or because they are missing.)

If no names are provided, and no depth provided

• Ignore the shrinkwrap. Start from the root node, rebuild the whole thing. (This is essentially just buildIdealTree with shrinkwrap: false, so npm update becomes the same as npm install --no-shrinkwrap or rm -rf node_modules package-lock.json; npm i.)

But, only print deprecation for a given pkg._id one time, or else it could get really awful.

Dep graph:

a -> (b@1, [email protected])
b -> (PEER [email protected])

where versions 1.2.3 and 1.2.4 exist of c.

Should be smart enough to note that 1.2.3 satisfies the peer dep and the parent, but that should be verified and captured in a test.

Arborist needs to run build scripts so that it can fail (and roll back) the install if they fail (or ignore, but clean up if it's an optional dep).

Write a copy of the lockfile to node_modules/.package-lock.json whenever we reify the tree.

If the mtime on this lockfile is >= the mtime on all folders in node_modules, assume it's a valid representation of the packages on disk, and have loadActual return its data by default instead of reading the tree. An option to loadActual can tell it to prefer this cached data if present, since npm ls, npm fund, etc. are typically fine with cached data.

When building the ideal tree, we should not use this fallback lockfile as an authoritative reference, however. It's a record of what is, but npm-shrinkwrap.json, package-lock.json, and yarn.lock capture what should be.

Using each of loadActual, loadVirtual, and buildIdealTree (since I use all three in what I'm building), how can i omit/ignore dev deps?

For loadActual, I can run npm prune --production first, and I get the tree I want; for loadVirtual, i'd presumably have to remove all lockfiles and run npm install --package-lock-only --package-lock --production; for buildIdealTree, it's not clear there's any way to work around it.

unsupported platforms should be failures (and cleaned up appropriately) if the node is optional

otherwise, we try anyway.

this will require the dist.shasum, which npm
no longer stashes in package.json. Will have to pull that out of
cacache or something, or fetch it if necessary?

This may require an update to pacote (and potentially ssri?) to calculate the sha1 integrity hash that yarn expects for resolved tarballs.

Another thought: what if the resolved value from pacote for registry deps just always had the #${shasum} tacked onto the tgz url, like yarn does? Would certainly make it easier to generate, and wouldn't be too hard. Fetch would have to know to strip it off, but I would expect it to anyway.

Set up a folder like described in npm/cli#750

Run the scripts/reify.js script on it, with --save

1. Does not create link to ./lib in node_modules.
2. symlink is absolute url, should be relative.
3. package-lock.json looks like this:

{
  "name": "monorepo",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "monorepo",
      "dependencies": {
        "app": "file:./app"
      }
    },
    "app": {},
    "node_modules/app": {
      "resolved": "app",
      "link": true
    }
  },
  "dependencies": {
    "app": {
      "version": "file:app"
    }
  }
}

Run scripts/reify.js --save on it again, and now package-lock.json looks like this:

{
  "name": "monorepo",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "monorepo",
      "dependencies": {
        "app": "file:./app"
      }
    },
    "app": {
      "resolved": "git+ssh://[email protected]/node_modules/app.git"
    },
    "node_modules/app": {
      "resolved": "app",
      "link": true
    }
  },
  "dependencies": {
    "app": {
      "version": "file:app"
    }
  }
}

Super weird, looks like it's interpreting the node_modules/app reference as a github shorthand? That's wrong.

What / Why

npm verb stack TypeError: Cannot read property 'name' of null
npm verb stack     at npa (/Users/mperrotte/npminc/cli/node_modules/npm-package-arg/npa.js:28:20)
npm verb stack     at FetcherBase.get (/Users/mperrotte/npminc/cli/node_modules/pacote/lib/fetcher.js:448:16)
npm verb stack     at Object.extract (/Users/mperrotte/npminc/cli/node_modules/pacote/lib/index.js:4:34)
npm verb stack     at Arborist.[extractOrLink] (/Users/mperrotte/npminc/cli/node_modules/@npmcli/arborist/lib/arborist/reify.js:344:16)
npm verb stack     at p.(anonymous function).then (/Users/mperrotte/npminc/cli/node_modules/@npmcli/arborist/lib/arborist/reify.js:330:39)

Get this nebulous error when trying to run arborist.buildIdealTree and arborist.reify in a path that doesn't contain a package.json

When

• Running .buildIdealTree()
• Running .reify()

Where

• opts = { path: 'some/path/with/no/package/json' }

How

Current Behavior

• Nebulous error

Expected Behavior

• Should throw with intelligent error message which can be caught and used

It would be nice if reify() (and, for that matter, other functions as well) would track how long things take.

Call process.emit('time', name) when a thing starts, and process.emit('timeEnd', name) when they end.

move a node to a different spot in the tree. Update parent/child links,
and re-resolve dependencies of any nodes that depend on it.

Required for #9

npm dedupe needs to find and de-duplicate dependencies. This should be available as a top-level Arborist function/mixin.

Rough algo sketch:

• Set this[_preferDedupe] to true
• Load virtual tree (or, failing that, load actual tree)
• For each name in tree.inventory.query('name')
  • set nodes to tree.inventory.query('name', name)
  • if nodes.size is 1, continue to next name
  • for each node in nodes
    • for each edge in node.edgesIn
      • add edge.from to this[_depsQueue]
• set this.idealTree to tree
• return this.reify()

This will require that we make the _depsQueue and _preferDedupe symbols shared (ie, Symbol.for('_depsQueue') instead of Symbol('_depsQueue').

cc @claudiahdz

Right now, if you do this:

npm i -D tap@12
npm i rimraf@2

then it'll add rimraf as a dep, and tap as a devDep.

And then later, if you do:

npm i tap@latest rimraf@latest

it'll be smart enough to update the tap in your devDeps, and the rimraf in your deps.

However, the API for adding deps in arborist requires that the caller decide where to place it in the package.json.

That means that the CLI can't just willy-nilly say "add this thing, idk, put it wherever it goes". It has to read the package.json, see if it's there, if not, decide where to add it, etc.

And then, arborist has to read the package.json again, and re-do some of that same complexity.

We've already diverged from mapping the options directly onto the package.json data structure (since we don't have the names for some deps at request time), so I feel like we may as well model the API more closely to what the CLI presents to the user.

Proposed API

// the first install, adding tap to devDeps
arb.reify({
  add: ['tap@12'],
  saveType: 'dev', // overrides wherever it might already be
  saveBundled: false,
  saveOptional: false,
})

// next install
arb.reify({
  add: ['rimraf@2'],
  saveType: null, // defaults to prod if missing
  saveBundled: false,
  saveOptional: false,
})

// third install, update both
arb.reify({
  add: ['tap@latest', 'rimraf@latest'],
  saveType: null, // just update where they are
  saveBundled: false,
  saveOptional: false,
})

Implementation

• saveBundled will place the names of all added deps into bundleDependencies
• saveType can have the following values:
  • null Just use the current for that dep name (deps/optional/dev/peer), or dependencies if not already present. If present in peer, do not modify the peerDepsMeta entry.
  • dev Save all added deps to devDependencies, removing them from optional/deps/peer/peerMeta if present.
  • prod Save all added deps to dependencies, removing them from dev/optional/peer/peerMeta if present.
  • optional Save all added deps to optionalDependencies, removing from dev/peer/peerMeta/deps if present.
  • peer Save all added deps to peerDependencies, removing them from dev/optional/deps if present, and removing a peerDependenciesMeta[name].optional flag if present.
  • peerOptional Save all added deps to peerDependencies and add a peerDependenciesMeta[name]={optional:true}

inBundle should be set based on a package's bundledDependencies, not
the _inBundle field in the written package.json file. Actually, we
should not really be looking at that package.json file for very much, if
possible, as it makes interoperability with Yarn harder. Rather than
writing to nm/foo/package.json, leave that file untouched, and stash
stuff in nm/.cache/npm instead maybe?

Keep inventories of the nodes in the tree.

• Index of all nodes by location. Would have to be updated when re-parenting. Maybe reparenting should be done by the Arborist, not a Node method? See #13.
• Set of all extraneous/dev/optional/devOptional nodes, the better to prune. See #11.

In the same way that we install something, and then might need to roll it back, bin links need to be preserved and cleaned up on removal.

Example:

$ mkdir x

$ echo '{}' > x/package.json

$ node scripts/reify.js x --add=rimraf --quiet
{
  path: 'x',
  cache: '/Users/isaacs/.npm/_cacache',
  add: { dependencies: { rimraf: '' } },
  quiet: true
}
resolved 12 deps in 0.0865676154s

$ node scripts/reify.js x --rm=rimraf --quiet
{
  path: 'x',
  cache: '/Users/isaacs/.npm/_cacache',
  rm: [ 'rimraf' ],
  quiet: true
}
resolved 0 deps in 0.004495018s

$ tree x
x
├── node_modules
│  └── .bin
│     └── rimraf -> ../rimraf/bin.js
├── package-lock.json
└── package.json

After installing and then removing rimraf, the rimraf bin is still present. That's litter.

If another module were to try to use that bin name, it'd fail, even though the rimraf module was removed!

Should be as simple as adding the bin link targets to the retiredNodes object. (Which should probably be called retiredPaths, since it's a collection of path names, not node objects?)

Required for #7

Provide an option to buildIdealTree to exclude (and prune) optional dependencies.

Note: we cannot ignore optionalDependencies conflicts, because the contract as implemented is that we will provide the version requested or nothing. Problems during reification (fetching, unpacking, building, etc) can be ignored if the dep is optional.

also, should probably log.warn the error.

Default export should be Arborist class, not loadActual (carry-over from read-package-tree)

• Node object should maybe have a hasShrinkwrap field? (Pull from pkg or by reading the actual folder when loaded from disk)
• Track hasShrinkwrap in lockfile
• Dont' bother fetching deps if it has a shrinkwrap, becuase we wont' need that
• Update idealTree during reification (essnetially, do loadVirtual on that node and then slap that tree into the idealTree as the node's children).

What / Why

Woah! This is awesome ❤️ excited to see where this goes!

Then provide these to runScript and binLinks in reify()

It's unnecessary and unused.

• make a list of what kinds of events we care about
• Emit events when that stuff happens
• should enable timing and logging in npm/cli

the dev/optional flag stuff, where it's neither when it's both. Set
a flag saying node.devOrOptional, unset that on deps and peerDeps.
Then, for optional deps, walk any that have devOrOptional, and set
optional=true. For dev deps, same thing, but dev=true. Result is that
something in both trees has both flags, and pruning dev means we remove
any dev that isn't optional; pruning optional means remove any optional
that isn't dev, pruning both is pruning any in either.

What

When loading, if a linked tree top is in a node_modules folder, and has
no children of its own, then load the parent of the node_modules folder
as a node, which will pick up all of the linked deps recursively.

pnpm arranges the tree like this:

root
+-- node_modules
    +-- .pnpm
    |   +-- lock.yaml
    |   +-- registry.npmjs.org
    |       +-- pkg
    |       |   +-- 1.2.3
    |       |       +-- node_modules
    |       |           +-- pkg
    |       |           +-- dep -> .pnpm/registry.npmjs.org/dep/2.3.4/node_modules/dep
    |       +-- dep
    |           +-- 2.3.4
    |           |   +-- node_modules
    |           |       +-- dep
    |           +-- 2.4.5
    |               +-- node_modules
    |                   +-- dep
    +-- pkg -> .pnpm/registry.npmjs.org/pkg/1.2.3/node_modules/pkg

So the root can do require(pkg) but not require(dep).

If we load node_modules/.pnpm/regisitry.npmjs.org/pkg/1.2.3 as if it's
a top level tree node of its own, then it obviously won't have a
package.json, so it'll be kind of boring, but then it'll pick up the pkg
target as a child, along with its deps, and the pkg target won't have any
missing edges.

In fact, we may even be able to work on this tree and capture it in a
package-lock.json!

How

In lib/load-actual.js, if a link target is in a node_modules folder, then load the parent of that node_modules folder as a Node. This will automatically get assigned as the link target node's parent, as well as loading all of its children. (In the case of a pnpm tree, this will be links to other similarly structured package nodes.)

Why

Interoperability with other package managers is an explicit design goal of Arborist and npm v7. Running npm ls in a pnpm-installed tree project should produce useful and correct output. npm commands in such a project should behave as a user would expect.

Limitations of pnpm Interop

We likely will not go so far as to preserve the pnpm tree style in projects that use it, for new dependencies added. For example, in the above tree, if a user runs npm install foo, they'll end up with something like:

root
+-- node_modules
    +-- .pnpm
    |   +-- lock.yaml
    |   +-- registry.npmjs.org
    |       +-- pkg
    |       |   +-- 1.2.3
    |       |       +-- node_modules
    |       |           +-- pkg
    |       |           +-- dep -> .pnpm/registry.npmjs.org/dep/2.3.4/node_modules/dep
    |       +-- dep
    |           +-- 2.3.4
    |           |   +-- node_modules
    |           |       +-- dep
    |           +-- 2.4.5
    |               +-- node_modules
    |                   +-- dep
    +-- pkg -> .pnpm/registry.npmjs.org/pkg/1.2.3/node_modules/pkg
    +-- foo
        +-- node_modules
            +-- foo-dep

If a user runs npm update --deep, and an updated version of dep is available that satisfies pkg's requirements, we might end up with:

root
+-- node_modules
    +-- .pnpm
    |   +-- lock.yaml
    |   +-- registry.npmjs.org
    |       +-- pkg
    |       |   +-- 1.2.3
    |       |       +-- node_modules
    |       |           +-- pkg
    |       |           |   +-- node_modules
    |       |           |       +-- dep
    |       |           +-- dep -> .pnpm/registry.npmjs.org/dep/2.3.4/node_modules/dep
    |       +-- dep
    |           +-- 2.3.4
    |           |   +-- node_modules
    |           |       +-- dep
    |           +-- 2.4.5
    |               +-- node_modules
    |                   +-- dep
    +-- pkg -> .pnpm/registry.npmjs.org/pkg/1.2.3/node_modules/pkg

While this is messy, it's not incorrect, and running npm prune would delete nodes that have no edges pointing into them.

quick audit on reify

Arborist should construct the audit payload and submit the quick audit for all reifications (unless audit: false is in the options) and have a function to perform a full audit.

full audit without fix

Load the actual tree, and submit to the full audit endpoint. Return the audit report object.

full audit with fix

Submit the full audit, then perform the necessary remediations as specified.

• Copy actualTree to this.idealTree
• Make remediation changes to this.idealTree
• Call this.reify() (triggers another quick audit, which should pass)

Get rid of node.logical. It serves no purpose, and is weirdly
confusing when dealing with link targets that may appear multiple times
in a tree, since it only captures the first place a node was
encountered. Better to just use the "physical" real path for nodes, and
always have node.path refer to that.

Refactor the tests into per-file 100% coverage-generating unit tests.