ipld-resolver has been renamed to ipld, hence all packages which depend on the old ipld won't receive any updates and it will be confusing as it looks like there's a new version of ipld available, but in reality it's a completely different thing.

Packages using the old "ipld" should either be deprecated or adapted so they don't use "ipld" anymore. According to npm those packages are:

Those are all related to IPRS. Is that still a thing? What should we do with the repositories below?

• https://www.npmjs.com/package/ipfs-record
• https://www.npmjs.com/package/abstract-record-store
• https://www.npmjs.com/package/ipfs-kad-record-store
• https://www.npmjs.com/package/interface-record-store
• https://www.npmjs.com/package/libp2p-kad-record-store
• https://www.npmjs.com/package/ipfs-drs

What about:

• https://www.npmjs.com/package/ipfs-ipld
  It now links to the js-ipld repo, but surely isn't the same thing. I think that one should also be deprecated.

Already deprecated:

• https://www.npmjs.com/package/merkle-trie

It would be nice to be able to get the full merkle proof of a path using resolver.resolve (would be exposed in ipld resolver i suppose). ipld-graph-builder could use the proofs to build its state object.

@lgierth @RichardLitt @whyrusleeping @jbenet

Instead of having to ping you every time. It might be easier and totally worth it to just make me an admin of the org

To make this work we need

• ipfs/js-ipfs-block-service#5
• ipfs/js-ipfs-repo#36
• ipld/js-ipld-dag-cbor#11

to be merged and published.

joinPath here is returning the path separated by platform-specific path separators.

The code afterwards in the resolvers is expecting forward slashes, so it doesn't work on Windows platforms.

I suggest replacing all backslashes by slashes after joinPath on Windows platforms.

In ipld/js-ipld-ethereum#25 @olizilla ran into some issues with the node representation that is returned from ipld.get(cid) for Ethereum objects. After I looked into it, it turns out the same issue exists for the Bitcoin and Zcash resolvers.

Specifically, when calling ipld.get(cid), some resolvers (dag-pb, dag-cbor, git) return an object where the keys represent resolvable paths. Other resolvers (bitcoin, ethereum, zcash) return an object where the keys don’t represent resolvable paths (it’s just the parsed representation from whatever underlying library is being used). For example:

// dag-cbor
const cborCid = new CID('zdpuAnhvECCigEM92DCZVwFguAHTSg1yJcPLyPdsJsVTYRuXB')
ipld.get(cborCid, (error, results) => {
  console.log(results)
})

// outputs:
{
  child: { '/': Buffer },
  title: 'A simple example of an IPLD node'
}

// and each key in that object represents a resolvable path:
ipld.get(cborCid, 'title', {localResolve: true}, (error, results) => {
  console.log(results)
})
// outputs:
'A simple example of an IPLD node'

// or
ipld.get(cborCid, 'child', {localResolve: true}, (error, results) => {
  console.log(results)
})
// outputs:
{ '/': Buffer }

// But for ethereum:
const etherumBlock = new CID('z43AaGEvwdfzjrCZ3Sq7DKxdDHrwoaPQDtqF4jfdkNEVTiqGVFW')
ipld.get(etherumBlock, (error, results) => {
  console.log(results)
})

// outputs:
{
  raw: [/* array of buffers */],
  _fields: [
    'parentHash',
    'uncleHash',
    'coinbase',
    'stateRoot',
    'transactionsTrie',
    'receiptTrie',
    'bloom',
    'difficulty',
    'number',
    'gasLimit',
    'gasUsed',
    'timestamp',
    'extraData',
    'mixHash',
    'nonce'
  ],
  toJSON: function() {},
  serialize: function() {},
  parentHash: Buffer,
  uncleHash: Buffer,
  coinbase: Buffer,
  stateRoot: Buffer,
  transactionsTrie: Buffer,
  receiptTrie: Buffer,
  bloom: Buffer,
  difficulty: Buffer,
  number: Buffer,
  gasLimit: Buffer,
  gasUsed: Buffer,
  timestamp: Buffer,
  extraData: Buffer,
  mixHash: Buffer,
  nonce: Buffer
}

// But many of these keys are not resolvable. This fails:
ipld.get(etherumBlock, '_fields', {localResolve: true}, (error, results) => {
  console.log(results)
})

// And this is *missing* valid, resolvable paths. This succeeds, even though `parent` is not in the above object:
ipld.get(etherumBlock, 'parent', {localResolve: true}, (error, results) => {
  console.log(results)
})

After talking with @vmx on Friday, we decided it was important that the object you get back is a reasonably accurate representation of the IPLD conception of the node — that is, the keys in the on the object should also represent paths you can resolve.

We need to make this requirement clear in the README or some other documentation.

Note: this is somewhat related to ipld/ipld#44, but after discussing with @vmx and @mikeal on Friday, I think we can treat it separately.

One other open question is that it might be important to have a way to get the underlying representation (either the raw bytes or the parsed representation from which we created the IPLD-focused representation [like what the Ethereum resolver currently returns]). What would be the best way to do that? We had three ideas:

1. Create a separate API, like ipld.getRawNode(cid, callback).

2. Return the raw bytes (and maybe the parsed representation) as special symbol keys so they don’t get serialized, but are still available. e.g:

   ipld.get(etherumBlock, (error, results) => {
     console.log(results)
   })
   // outputs:
   {
     [ipld.rawBytes]: Buffer,
     [ipld.parsedData]: {},  // The object we currently get from ipld.get()
     bloom: Buffer,
     coinbase: Buffer,
     difficulty: 11966502474733,  // A number or BigInt, not just bytes
     parent: CID,  // or {'/': 'hash'} -- see https://github.com/ipld/ipld/issues/44
     // etc.
   }

3. Return the underlying object (like today) if just getting the CID, but return an IPLD-focused representation if getting the path /. e.g:

   // Returns the same thing as today
   ipld.get(etherumBlock, (error, results) => { console.log(results) })
   
   // Returns a nicer, IPLD-focused representation where the keys are resolvable paths:
   ipld.get(etherumBlock, '/', (error, results) => { console.log(results) })

   This is probably the least desirable because it’s not super clear and very easy to get wrong.

ipld-resolver was renamed to ipld. Hence deprecate the ipld-resolver package. This is a follow-up issue of #116.

Currently deprecation fails, it seems to be an npm issue: https://github.com/npm/registry/issues/301

$ npm deprecate ipld-resolver@'<= 0.14.1' 'WARNING: This project has been renamed to "ipld". Install using "ipld" instead.'
npm ERR! code E422
npm ERR! Unprocessable Entity : ipld-resolver

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/vmx/.npm/_logs/2018-03-26T10_27_22_108Z-debug.log

Removed TODO from resolver.put as it prevents a git commit

put (node, options, callback) {
    if (typeof options === 'function') {
      return setImmediate(() => callback(new Error('no options were passed')))
    }
    callback = callback || noop

    if (options.cid && CID.isCID(options.cid)) {
      return this._put(options.cid, node, callback)
    }

    options.hashAlg = options.hashAlg || 'sha2-256'
    const r = this.resolvers[options.format]
    if (!r) {
      return callback(new Error('No resolver found for codec "' + options.format + '"'))
    }
    // TODO add support for different hash funcs in the utils of
    // each format (just really needed for CBOR for now, really
    // r.util.cid(node1, hashAlg, (err, cid) => {
    r.util.cid(node, (err, cid) => {
      if (err) {
        return callback(err)
      }

      this._put(cid, node, callback)
    })
  }

• Version:0.34.4
• Platform:OSX
• Subsystem:*

Type:Enhancement

Severity:Other

Description:Provide an API to encode data without publishing a block

This is related to ipld/ipld#64, more specifically I wish to encode content with arbitrary IPLD format supported, but then encrypt that buffer before publishing it. However API available right now doesn't provide a way to accomplish that because put first encodes data to buffer and then publishes it. It is also impossible to retrieve buffer for that CID as it automatically decoded with a corresponding format.

If public API was further extended with encode(data, option):Buffer that would allow one to do multiple encoding passes before block is published. Doing reverse operation would also be required to provide get that has to unwrap multiple layers.

We've discussed a patch API, but we haven't settled on how it gets implemented and/or if it gets implemented at this level of IPLD.

tl'dr the patch API would enable the set of values withing an IPLD path, bubbling up the changes and returning the new root CID, very much like object patch API, but for any IPLD Format, which means that interface-ipld-format would have to have a local scope patch function too, and that knows how to traverse objects.

Considerations:

• These can be very expensive operations, the longer the path the more nodes it touch (more network round trips, more disk accesses), we need to account, just like MFS, options for : flush/no flush
• It might be or not important to have at this level, an API to commit/revert (very much like what @wanderer built for Ethereum with https://github.com/ethereumjs/merkle-patricia-tree)
• Note that this API is inevitable to exist, it is just a question if we can make something that serves all the potential use cases, or if it gets built application by application (just like MFS has its own)

We need an endpoint for https://github.com/ipfs/js-ipfs-unixfs-engine that can return multiple IPLD nodes from an Array of CIDs.

Leverage ipfs/js-ipfs-block-service#81

This is blocking ipfs/js-ipfs#856.

@vmx do you already have code for this with the GraphSync endeavour?

Due to fixes in the Bitcoin (ipld/js-ipld-bitcoin@ef05359) and Zcash (ipld/js-ipld-zcash@fdfe4e4) format to work only on the header, some tests broke.

It's probably a matter of updating the test code of js-ipld.

Create a document similar to https://github.com/libp2p/js-libp2p/blob/40739e96398fe5715068f5ecb12a5b973c172e2b/RELEASE.md that describes the release process.

This issue is related to #136.

Right now when deserializing data we only pass the block's data into the util.deserialize function of the codec.

If the deserialized version of the data needs to be aware of it's own CID, it must then derive it from the passed data. For example ipld-dag-pb does this.

This has a few side effects:

1. It's inefficient (this takes about 11% of CPU time when using ipfs-npm)
2. It's unnecessary when we have the CID further up the call stack
3. It's inaccurate as the CID version/hash-alg is lost so ipld-dag-pb defaults to sha2-256 which might not be what was used when the DAGNode was originally encoded thus resulting in a different CID

We could pass the CID into util.deserialize instead which would negate all of the above points.

Thoughts @ipld/javascript-team ?

HashAlg option of the put method is not yet supported, as stated here: https://github.com/ipld/js-ipld-resolver/blob/master/src/index.js#L192

However, in the documentation appears as required, as well as in other documentations: https://github.com/ipfs/interface-ipfs-core/tree/master/API/dag#dagput

if this option is provided to the method, a message informing that it is not yet supported and the hash algorithm used (sha2-256) would improve the development experience.

Calling ipld.get() should never result in the direct, deserialized value from a resolver’s util.deserialize() function. It should instead result in the equivalent of resolving the path / if no path is provided.

Right now, we explicitly return the result of resolver.util.deserialize(rawBlockData) if no path is specified:

(where ipld._get(cid) is just a wrapper for getResolverForCid(cid).util.deserialize(rawData))

Instead, we should always call resolver.resolver.resolve(rawBlockData, path) as part of ipld.get(), and the path we pass to the resolver should be / if no path was specified as an argument to ipld.get().

This is because get() should always return an object where the keys reflect paths that could be resolved — i.e. the IPLD focused concept of the DAG node, not some other underlying representation, like a BitcoinjsBlock from the Bitcoin library, for example. See the discussion in #141 for more about this.

I have two examples of treeStream where the output is [] - the first is definitely a bug, the second is probably a bug 😝

1. treeStream('z8mWaJHXieAVxxLagBpdaNWFEBKVWmMiE', { recursive: true }) -> []

   I would expect to see at least the paths returned by the same call without the recursive option i.e.

   ['message',
     'tree',
     'author/original',
     'author/name',
     'author/email',
     'author/date',
     'committer/original',
     'committer/name',
     'committer/email',
     'committer/date',
     'parents/0',
     'parents/1']

   I guess I'd expect a stack overflow or many many many more paths haha! Shouldn't there be a default limit on the recursion? This could be used as a DoS attack.

2. treeStream('z8mWaJHXieAVxxLagBpdaNWFEBKVWmMiE', 'parents/0') -> []

   Probably a bug right? parents/0 is a link to another 'git-raw' commit so I was expecting a similar output to the example above (message, tree etc.). However, this could be expected behaviour - is it? I'm hoping not!

   N.B. this is not a git-raw resolver issue as I've tried the same thing with dag-cbor nodes and I get the same result.

I'm using the following code to test this:

// npm i ipfs-api ipld cids pull-stream
const IpfsApi = require('ipfs-api')
const Ipld = require('ipld')
const CID = require('cids')
const pull = require('pull-stream')

const ipfs = new IpfsApi()
const ipld = new Ipld(ipfs.block)

const cid = new CID('z8mWaJHXieAVxxLagBpdaNWFEBKVWmMiE')
const path = 'parents/0'
const opts = { /* recursive: true */ }

pull(ipld.treeStream(cid, path, opts), pull.collect(console.log))

I am now writing some very demo code to show IPLD.

I run into the following: js-ipld does not resolve a path query, so for example, given an object (that I have in memory), I can't use merkle paths to point to any attribute.

1. It would be great to have a function that resolves a path given a service (resolve(service, path))
2. It would be great if we make an IPLD service that is dum - basically just a memory store, there is no need to run the entire IPFS

Ref #3

I have a use case were I want to have some sort of authentication when resolving through each node. This requires the local resolver to have access to the authentication data.

It seems that the options object could transport any arbitrary data and could fit my purpose as well as other ones. Right now the options object is not passed down to resolver.resolve(binaryBlob, path, callback

I'm opening this issue as it was suggested by @vmx here

Sometimes getting a timeout from Travis CI

  1) Browser,IPLD Resolver with eth-* (resolving across various Eth objects)
       "before all" hook:
     Timeout of 5000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

const node = {
  'test': 1
}
resolve.put(node, { format: 'dag-cbor', hashAlg: 'sha2-256'}, () => {
   console.log('done!')
})

node.test = 2

This will produce an invalid node, it will have the contents of {test: 2} with the cid of {test: 1}
Most of the time changing the node before resolve.put is done would be a mistake but it can easily prevented at the level of this module. The problem is that serialize the data twice (here and here). If it would only be serialized once then hashed this would be prevented. It would help to have this first though.

It seems weird to me that you can do this sort of thing, where there's no guarentee that the passed CID has any relation to the node we're trying to store:

const cid = new CID('QmTmxQfEHbQzntsXPTU4ae2ZgBGwseBmS12AkZnKCkuf2G')

resolver.put(Buffer.from('Hello world'), {
  cid
}, cb)

How about deprecating/removing the cid argument in favour of requiring version, hashAlg and format and letting the resolvers decide on the CID used?

Maybe also accept codec in place of format too in order to be more consistent with other terminology around ipld?

This test fails (add to the suite in test/ipld-dag-pb.js):

it('should return a v1 CID for a DAGNode created with a v1 CID', (done) => {
  dagPB.DAGNode.create(Buffer.from([0, 1, 2, 3]), (err, node) => {
    expect(err).to.not.exist()

    const cid = new CID(node.multihash).toV1()

    resolver.put(node, {
      cid,
      format: cid.codec
    }, (error, nodeCid) => {
      expect(error).to.not.exist()
      expect(nodeCid instanceof CID).to.be.true()
      expect(nodeCid).to.deep.equal(cid)

      resolver.get(nodeCid, (error, result) => {
        expect(error).to.not.exist()
        expect(result.value).to.deep.equal(node)

        done()
      })
    })
  })
})

If you remove the toV1() conversion it passes.

? Following the discussion on the first IPLD deep dive -- ipfs/team-mgmt#484 --

@wanderer has volunteered to our Captain of Performance Testing 🙌🏽 🌟 ⚡️ Thank you! ❤️

@wanderer has already been developing benchmarks for IPLD and doing a lot of work with IPLD (e.g https://github.com/ipld/js-ipld-graph-builder) and identified some serious performance issues such as: ipld/interface-ipld-format#7.

The ideal goal is reach a state where IPLD has a set of powerful benchmarks for multiple use cases and the right primitives that let the user tune for its use case, as I believe that finding a one size fits all solution is going to be hard (thinking of the caching question). That being said, we do have a lot of work to do just in raising the bar and that will improve perf for all use cases.

Another great endeavour related to this one is the development of WASM resolvers and hashing functions so that we can go at max speed we could go in a Browser/Node.js env. Crypto operations are some of the primary perf bottlenecks of the whole js-ipfs.

Removes dependence on ipfs-repo and ipfs-block-service.

As pointed out by @wanderer here ipld/interface-ipld-format#7, we need to update the spec. I'm using this issue to track of the work that needs to happen.

Edit: Moved tracking info to #96

Currently, the js implementation is missing git and Bitcoin support to match the go-ipfs implementation

http://ipld.io/implementations/

We will get to this at some point, however, if there is someone in the community that would like to take on that task right now, it would be really appreciated! Let me know if you need help getting started :)

This repo got renamed from js-ipld-resolver to just js-ipld. What should the description say?

The GitHub description is: "JavaScript implementation of the IPLDService"
The package.json description is: "IPLD Resolver Implementation in JavaScript"
The Readme description is: "The JavaScript implementation of the IPLD, InterPlanetary Linked-Data"

I guess all should be the same.

This repo currently is published as ipld-resolver. @diasdavid mentions in #110 (comment) that this package should be deprecated an republished as just ipld.

Though the ipld npm package currently points to https://github.com/diasdavid/js-ipld which currently redirects to https://github.com/ipld/js-ipld-dag-cbor.

And then there's also the ipfs-ipld npm package which points to https://github.com/ipfs/js-ipfs-ipld which redirects to https://github.com/ipld/js-ipld.

There are some examples in js-ipfs using the DAG API]. Those should be run as part of the release process.

Seems that #169 (comment) was not a hiccup. I get it to happen often on my machine.

  1) Browser,IPLD Resolver with eth-* (resolving across various Eth objects),resolver.get
       block-to-account resolve:
     Error: Uncaught AssertionError: expected [Error: Path not found ("nonce").] to not exist (webpack://ipld/node_modules/chai/lib/chai/assertion.js:141 <- node_modules/aegir/src/config/karma-entry.js:23107)

» node -v
v8.12.0

Hey,

I'm currently trying to create a new IPLD format resolver. It works fine, however a strange thing happen ... Sometime, when the object i'm trying to push is "too" long i got the following when a I call the put method:

Error: ENAMETOOLONG: name too long, open '/Users/osarrouy/Documents/devs/dapps/@wespr/pando/test/mocks/.pando/ipfs/blocks/J6/AFKRXWIBPMREA5DZOBSSEORCONXGC4DTNBXXIIRMEJQXK5DIN52HI4RCHIRDEZBUO5UGM2TLNBRGM43ENNVGM2DTNJSGW2BCFQRHI5BCHIRDEZBUO5UGM2TLNBRGM43ENNVGMZDTMZZWIZTEONTWMZTEM5SGMZ3EON5GM2DTNJSGW2BCFQRG2ZLTONQWOZJCHIRFIZLTOQQHG3TBOBZWQ33UEIWCE4DBOJSW45DTEI5FWXJMEJ2HEZLFEI5HWIRPEARDUITUN52G6IT5FQRHI2LNMVTWIZTIM5UHG5DBNVYCEORCPJXXU332N55G66TPPJXXU332N55G66TPPJXXU332N55G66T2EJ6Q.data.301414793'

That's really weird because the length of the block file seems linked to the 'length' of the object - and seems to exceed the limit of the filesystem.

It happens on macOS with a resolver whose serialization method is just a Buffer.from(JSON.stringify(dagNode))

Any idea ?

This is a catch all issue for the API review. Issues related to an API review should be linked here.

The new API might have breaking changes. It will also use async/await, which is kind of part of the Awesome Endeavour: Async Iterators.

Already opened issue should get the label api-review attached.

There's also a sibling issue for IPLD Formats.

It would be nice when writing resolver to have an abstract interface to follow like the way its done in libp2p.

working version:
const node = {
hello: {
world: 11,
some: 12
}
}
const mh = ipld.multihash(ipld.marshal(node))
ipldService.add(node, (err) => { resolve(ipldService, ${mh}/hello/world, (err, res) => { console.log(res) })})

The BlockService is an intermediate abstraction/interface that's not really needed. It makes sure that blocks are retrieved from remote Peers if an Exchange (i.e. Bitswap) is defined. Bitswap already checks by itself if the data is locally available, hence we don't need the BlockService, but we can use Bitswap directly.

JS-IPLD doesn't even need a full JS Repo implementation, but only the block storage part of it. This means that IPLD could be initialised with just the block store, which doesn't even need to implement the full datastore interface, but only the part the block API is using (currently put(), get(), getMany() and delete()).
So passing in a something that implements the block API subset of the datastore interface is enough.

Steps to get rid of the BlockService

1. Implement the datastore interface for Bitswap which implements the subset to make it usable as block API datastore.
2. Change the IPLD API to use the a block API datastore instead of a full Repo implementation
3. Make IPLD use the Bitswap based datastore

@diasdavid Can you please confirm that I'm not missing something here?

Inspiration - https://github.com/ipfs/js-ipfs#packages

This module was named ipld-resolver because it was designed to the mother of all local resolvers (dag-pb, dag-cbor, eth-*, etc) and act as a facade for a future IPLD cli (for people wanting to play with IPLD in the terminal), an IPLD playground (a project @nicola and @victorbjelkholm were working on) and finally, by js-ipfs witht he DAG API.

It turned out that the DAG API became pretty much the API that this module implements directly, the IPLD cli is a bit dormant and we definitely have to wake up the IPLD Playground.

What I'm thinking, to reduce fragmentation and make it more obvious that "hey world, this is your entry point to IPLD if you want to play with IPLD" is to rename this module to js-ipld and add the CLI feature here, possibly following this feature ipfs/notes#194

This would also encourage us to make this module work very much standalone (spawning its own in memory or local repo) so that users can play with IPLD at will without having to boot a full node.

Thoughts?

Currently the tests don't pass due to ipfs/js-ipfs-repo@27553a3. There's now a version mismatch between ipfs-repo and the example repository for the tests.

The solution is to recreate the example repository.

A workaround is to pin ipfs-repo to 0.17.0.

Is it possible to

• have a function (.putAll ?) when one could put many objects instead of just one?
• or have put to accept an array instead of a single object?

Is this possible, @vmx? For the js-ipfs 0.28 release?

The isLink() function was removed from the IPLD Format spec (ipld/interface-ipld-format#23), in favour of putting this functionality into js-ipld instead.

The implementation from js-ipld-dag-pb can be used.

It first serialization the data to create the CID then once again to store the data.

would be nice to do a ipldResolver.get and get back a pullStream that passes back all nodes touched

would also be nice (for analytics) to be able to spy on all nodes being resolved by any request, something like a ipldResolver.createSpyStream()

cc @diasdavid

We've discussed some potential changes for ipld-resolver. I wanted to make sure we're tracking them all.

• get paths only / lazy value lookups ( some objects like etherum-tx's have implicit data (e.g.: fromAddress) that is computed via expensive crypto - we should only return values that were explicitly asked for)
• get external / local paths only
• toJSON + fromJSON - roundtrip (de-)serialize data via json format
• put + patch api #66
• text summary - output text that explains the content of a given ipfs-block
• globbing ( this should be handled outside of the resolvers )

@ipld/javascript-team see thread at Awesome Endeavour: Async Iterators

Currently the IPLD constructor is taking a BlockService object. Change the API to take it an options object (similar to what libp2p is doing).

This makes it possible to pass in the IPLD Formats that should be used. It will include ipld-dag-cbor and ipld-dag-pb by default and make others optional. This way the Browser bundle size will shrink a lot.

@diasdavid I was trying to create a branch windows-interop and I was denied access.