Giter VIP home page Giter VIP logo

merkle-patricia-tree's Introduction

SYNOPSIS

NPM Status Actions Status Coverage Status Discord

This is an implementation of the modified merkle patricia tree as specified in the Ethereum Yellow Paper:

The modified Merkle Patricia tree (trie) provides a persistent data structure to map between arbitrary-length binary data (byte arrays). It is defined in terms of a mutable data structure to map between 256-bit binary fragments and arbitrary-length binary data. The core of the trie, and its sole requirement in terms of the protocol specification is to provide a single 32-byte value that identifies a given set of key-value pairs.

The only backing store supported is LevelDB through the levelup module.

INSTALL

npm install merkle-patricia-tree

USAGE

There are 3 variants of the tree implemented in this library, namely: BaseTrie, CheckpointTrie and SecureTrie. CheckpointTrie adds checkpointing functionality to the BaseTrie with the methods checkpoint, commit and revert. SecureTrie extends CheckpointTrie and is the most suitable variant for Ethereum applications. It stores values under the keccak256 hash of their keys.

Initialization and Basic Usage

import level from 'level'
import { BaseTrie as Trie } from 'merkle-patricia-tree'

const db = level('./testdb')
const trie = new Trie(db)

async function test() {
  await trie.put(Buffer.from('test'), Buffer.from('one'))
  const value = await trie.get(Buffer.from('test'))
  console.log(value.toString()) // 'one'
}

test()

Merkle Proofs

const trie = new Trie()

async function test() {
  await trie.put(Buffer.from('test'), Buffer.from('one'))
  const proof = await Trie.createProof(trie, Buffer.from('test'))
  const value = await Trie.verifyProof(trie.root, Buffer.from('test'), proof)
  console.log(value.toString()) // 'one'
}

test()

Read stream on Geth DB

import level from 'level'
import { SecureTrie as Trie } from 'merkle-patricia-tree'

const db = level('YOUR_PATH_TO_THE_GETH_CHAIN_DB')
// Set stateRoot to block #222
const stateRoot = '0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544'
// Initialize trie
const trie = new Trie(db, stateRoot)

trie
  .createReadStream()
  .on('data', console.log)
  .on('end', () => {
    console.log('End.')
  })

Read Account State including Storage from Geth DB

import level from 'level'
import rlp from 'rlp'
import { BN, bufferToHex } from 'ethereumjs-util'
import Account from 'ethereumjs-account'
import { SecureTrie as Trie } from 'merkle-patricia-tree'

const stateRoot = 'STATE_ROOT_OF_A_BLOCK'

const db = level('YOUR_PATH_TO_THE_GETH_CHAINDATA_FOLDER')
const trie = new Trie(db, stateRoot)

const address = 'AN_ETHEREUM_ACCOUNT_ADDRESS'

async function test() {
  const data = await trie.get(address)
  const acc = new Account(data)

  console.log('-------State-------')
  console.log(`nonce: ${new BN(acc.nonce)}`)
  console.log(`balance in wei: ${new BN(acc.balance)}`)
  console.log(`storageRoot: ${bufferToHex(acc.stateRoot)}`)
  console.log(`codeHash: ${bufferToHex(acc.codeHash)}`)

  let storageTrie = trie.copy()
  storageTrie.root = acc.stateRoot

  console.log('------Storage------')
  const stream = storageTrie.createReadStream()
  stream
    .on('data', (data) => {
      console.log(`key: ${bufferToHex(data.key)}`)
      console.log(`Value: ${bufferToHex(rlp.decode(data.value))}`)
    })
    .on('end', () => {
      console.log('Finished reading storage.')
    })
}

test()

Additional examples with detailed explanations are available here.

API

Documentation

TESTING

npm test

BENCHMARKS

There are two simple benchmarks in the benchmarks folder:

  • random.ts runs random PUT operations on the tree.
  • checkpointing.ts runs checkpoints and commits between PUT operations.

A third benchmark using mainnet data to simulate real load is also under consideration.

Benchmarks can be run with:

npm run benchmarks

To run a profiler on the random.ts benchmark and generate a flamegraph with 0x you can use:

npm run profiling

0x processes the stacks and generates a profile folder (<pid>.0x) containing flamegraph.html.

REFERENCES

EthereumJS

See our organizational documentation for an introduction to EthereumJS as well as information on current standards and best practices.

If you want to join for work or do improvements on the libraries have a look at our contribution guidelines.

LICENSE

MPL-2.0

merkle-patricia-tree's People

Contributors

alanshaw avatar alextsg avatar axic avatar cvan avatar evertonfraga avatar fanatid avatar federicobond avatar gabrocheleau avatar holgerd77 avatar jbaylina avatar jochem-brouwer avatar jwasinger avatar kumavis avatar medvedev1088 avatar no2chem avatar ryanio avatar s1na avatar vpulim avatar wanderer avatar whymarrh avatar xcrux avatar zmitton avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

merkle-patricia-tree's Issues

Extract db-related logic from baseTrie

In addition to MPT logic, BaseTrie also has methods for the underlying key-value db. This logic can be extracted to a separate class, to make BaseTrie more readable and reduce coupling. The methods include getRaw, 'putRaw, 'batchNodes, etc.

missing typescript types

On the compiler options please add "declaration": true:
example:

{ "extends": "@ethereumjs/config-tsc", "compilerOptions": { "outDir": "./dist", "declaration": true }, "include": ["src/**/*.ts"],

Does this handle nested nodes and where in the code is it?

The generated tries in Ethereum account for nodes that are less than 32 bytes when RLP-encoded to be included in it's parent in it's entirety as opposed to being reference to by its hash.

A popular library eth-proof seems to generate incorrect tries for some blocks on the mainnet as noted here which uses this library to construct the tries and I suspected it could be due to this edge-case.

I can't find a reference to where this edge-case referenced as clause 193 in the Ethereum Yellow Paper is in the code. Is this case handled or am I just blind?

Support Question, headline replaced by admin

Hello, my company is developing a ether wallet, ether node has been set up, with the Geth. But now there is a problem, that is, how to query the wallet address under the transaction records. Do you need to parse the node's leveldb file, can you give me some ideas? Thank you.

update checkpointing API

The current check-pointing API isn't the best and it is inefficient. The API could be simplified. Instead of checkpoint, commit and revert there should be a single commit or flush function that writes everything to the DB and returns the resulting state root. Instead of checkpoint and commit we should have a functional API such as this. This relegates the functionality of checkpoint and revert to JS's GC which will have better performance then manually tracking the different referenced states.

Trie only accepts one `put` call

If I clone the repository and then import CheckpointTrie then this does not seem to happen. However, if I use the package via npm and then use the default Trie type then the root hash does not change after I put 1 key-value pair:

const Trie = require('merkle-patricia-tree') // v3.0.0

async function test() {
  console.log("Trie with default order")
  let trie1 = new Trie()
  console.log(trie1.root.toString('hex'))
  await trie1.put("a", "d")
  console.log(trie1.root.toString('hex'))
  await trie1.put("b", "b")
  console.log(trie1.root.toString('hex'))
  console.log("Trie with flipped order")
  let trie2 = new Trie()
  console.log(trie2.root.toString('hex'))
  await trie2.put("b", "b")
  console.log(trie2.root.toString('hex'))
  await trie2.put("a", "d")
  console.log(trie2.root.toString('hex'))

}

test()
/*

Output:

Trie with default order
56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
cd243c64836a746bb39de18312bd10e010ef4c8df513fb6700c20a7e120cdf7c
cd243c64836a746bb39de18312bd10e010ef4c8df513fb6700c20a7e120cdf7c
Trie with flipped order
56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
4330eb1bb29013a46e5c00795cea13ca615491cc42fbfe76e4f139b349af330d
4330eb1bb29013a46e5c00795cea13ca615491cc42fbfe76e4f139b349af330d

Binary tree support

Data structures in Ethereum sharding will use a binary instead of a hex tree for shorter merkle proofs, see e.g. the Chunk tree description for Collations in the sharding phase 1 spec.

It should be analyzed if this library can easily be adopted to also support binary tree structures (first guess: just change the maximum number of child nodes).

Or if e.g. another tree implementation is already existing (e.g. https://github.com/dfinity/js-dfinity-radix-tree, also consider license compatibility), that would better serve the needs for sharding.

Add merkle proofs to the secure trie version

Currently merkle proof functionality is only added to the base trie, see index.js for the integration.

This should be done accordingly for the secure trie in secure.js and at least one new test should be added to the test/proof.js test file.

Unable to read stream from the latest Ethereum mainnet

Hi,

I was trying to iterate a readStream to loop through all accounts in the latest Ethereum mainnet. I ran a script like the following:

var Trie = require('merkle-patricia-tree/secure');
var levelup = require('levelup');
var leveldown = require('leveldown');
var RLP = require('rlp');

//Connecting to the leveldb database
var db = levelup(leveldown('/data3/data/geth/chaindata/'));
var root = '0x6e18ee7df9043dad34470e71729a14ba946750e017e523815478ff11d3bbaccc';    // block 5676026

var trie = new Trie(db, root);

//get a read stream object
var stream = trie.createReadStream();

stream.on('data', function (data) {
  console.log(data);
});

stream.on('end', function (val) {
  console.log('done');
});

The script works on small private nets I set up for other experiments, however, when I try to run it against the fast-sync'd mainnet geth, there seems to have no data event at all. The script ended printing "done" immediately.

Trie.prototype._updateNode(); TypeError: Cannot read property 'pop' of undefined

_updateNode is receiving undefined stack

Receiving TypeError: Cannot read property 'pop' of undefined

patricia-tree/baseTrie.js:359
  var lastNode = stack.pop()

                      ^

TypeError: Cannot read property 'pop' of undefined

Suspect WalkController is not passing stack in the callback of processNode();

Trie.prototype._findPath = function (targetKey, cb) {
  var self = this
  var root = self.root
  var stack = []
  targetKey = TrieNode.stringToNibbles(targetKey)

  this._walkTrie(root, processNode, cb)

  function processNode (root, node, keyProgress, walkController) {
    var nodeKey = node.key || []
    var keyRemainder = targetKey.slice(matchingNibbleLength(keyProgress, targetKey))
    var matchingLen = matchingNibbleLength(keyRemainder, nodeKey)

    stack.push(node)

    if (node.type === 'branch') {
      if (keyRemainder.length === 0) {
        walkController.return(null, node, [], stack)
      // we exhausted the key without finding a node
      } else {
        var branchIndex = keyRemainder[0]
        var branchNode = node.getValue(branchIndex)
        if (!branchNode) {
          // there are no more nodes to find and we didn't find the key
          walkController.return(null, null, keyRemainder, stack)
        } else {
          // node found, continuing search
          walkController.only(branchIndex)
        }
      }
    } else if (node.type === 'leaf') {
      if (doKeysMatch(keyRemainder, nodeKey)) {
        // keys match, return node with empty key
        walkController.return(null, node, [], stack)
      } else {
        // reached leaf but keys dont match
        walkController.return(null, null, keyRemainder, stack)
      }
    } else if (node.type === 'extention') {
      if (matchingLen !== nodeKey.length) {
        // keys dont match, fail
        walkController.return(null, null, keyRemainder, stack)
      } else {
        // keys match, continue search
        walkController.next()
      }
    }
  }
}

Proof for non-existence values (0x0)

Currently when I get the merkle-proof from parity for a storage-value with 0x0 it will return me a array of nodes which are only branches because there is no leaf-node for the 0x0-value.
When I use the verifyProof it will return with a Error 'Unexpected end of proof', which is actually correct, but it raises the question, how are we able to prove non-existend or 0x0-values?

For now I rewrote the verifyProof-Function with a additional Argument (expectedValue) in our repo.
This checks in case of a branch before throwing this error:

  // if we expected this to be null and there is no further node since wantedHash is empty, than it is ok not to find leafs, because it proofs, that this value does not exist
  if (expectedValue === null && wantHash.length === 0)
    return null

  // we reached the end of the proof, but not of the path
  throw new Error('Unexpected end of proof')

In the case the last node is a leaf pointing to a different node, which can happen, I accept this as a proof that this node does not exist:

        // if the relativeKey in the leaf does not match our rest key, we throw!
        if (matchingNibbleLength(node.key, key) !== node.key.length) {
          // so we have a wrong leaf here, if we actually expected this node to not exist,
          // the last node in this path may be a different leaf or a branch with a empty hash
          if (key.length === node.key.length && i === proof.length - 1 && expectedValue === null)
            return val

          throw new Error(errorPrefix + 'Key does not match with the proof one (extention|leaf)')
        }

Library fails branch-value-update test

Upstream in the official tests, a new test "branch-value-update" was added:

  "branch-value-update": {
    "in": [
      [ "abc", "123" ],
      [ "abcd", "abcd" ],
      [ "abc", "abc" ]
    ],
    "root": "0x7a320748f780ad9ad5b0837302075ce0eeba6c26e3d8562c67ccc0f1b273298a"
  }

This library fails that test. In particular, it seems that search stops at an extension node when it should check the value in the branch node.

npm run build fails

npm run build fails with the following error:

Error: Cannot find module 'async' from '/path/merkle-patricia-tree'

Although npm run babel succeeds.

better proof/verification support

The Implementation of proof verification creates its own sort of "new" version of traversing the trie. There is a better way to verify proofs: the prooving nodes or branch nodes should be put into the key-value DB (just the ephemeral one) using the nodes given with their keccak as key. Then the the regular get function should be called on this small tree to see if it can pull out the expected value at the end. This should be much dryer and easier to maintain. I plan to PR this myself soon

In the meantime, Im having trouble verifying proofs of the state tree. So far Im stuck trying to make sense of a function that claims to turn strings and buffers into nibbles, but as far as I can tell does nothing of the sort:

https://github.com/ethereumjs/merkle-patricia-tree/blob/master/src/trieNode.js#L207

my results:

> stringToNibbles(2)
[ 0, 0, 0, 0 ]
> stringToNibbles('2')
[ 3, 2 ]
> stringToNibbles('02')
[ 3, 0, 3, 2 ]
> stringToNibbles('0x02')
[ 3, 0, 7, 8, 3, 0, 3, 2 ]
> stringToNibbles(0x02)
[ 0, 0, 0, 0 ]

UPDATE:
Seems to work for converting from buffers. maybe this is the only current use. Either way, name might want to change

empty stack

somehow we got to a place where we're popping an empty stack. not sure how to debug, the code is a little dense.

/Users/kumavis/Development/Node/vapor-rpc/node_modules/node-ethereum/node_modules/ethereumjs-lib/node_modules/merkle-patricia-tree/index.js:285
  if (lastNode.type === 'branch') {
              ^
TypeError: Cannot read property 'type' of undefined
    at internals.Trie._updateNode (/Users/kumavis/Development/Node/vapor-rpc/node_modules/node-ethereum/node_modules/ethereumjs-lib/node_modules/merkle-patricia-tree/index.js:285:15)
    at /Users/kumavis/Development/Node/vapor-rpc/node_modules/node-ethereum/node_modules/ethereumjs-lib/node_modules/merkle-patricia-tree/index.js:106:18
    at processNode (/Users/kumavis/Development/Node/vapor-rpc/node_modules/node-ethereum/node_modules/ethereumjs-lib/node_modules/merkle-patricia-tree/index.js:154:7)
    at /Users/kumavis/Development/Node/vapor-rpc/node_modules/node-ethereum/node_modules/ethereumjs-lib/node_modules/merkle-patricia-tree/index.js:565:9
    at Stream.<anonymous> (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/rpc-stream/index.js:77:10)
    at Stream.stream.write (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/rpc-stream/node_modules/through/index.js:26:11)
    at Stream.ondata (stream.js:51:26)
    at Stream.emit (events.js:107:17)
    at Stream.<anonymous> (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/mux-demux/inject.js:61:14)
    at Stream.emit (events.js:107:17)
    at Stream.stream.emit (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/mux-demux/node_modules/stream-serializer/index.js:60:33)
    at Stream.stream.write (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/mux-demux/node_modules/duplex/index.js:68:12)
    at parse (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/mux-demux/node_modules/stream-serializer/index.js:29:13)
    at Stream.onData [as write] (/Users/kumavis/Development/Node/vapor-rpc/node_modules/multilevel/node_modules/mux-demux/node_modules/stream-serializer/index.js:36:7)
    at Socket.ondata (_stream_readable.js:540:20)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:163:16)
    at Socket.Readable.push (_stream_readable.js:126:10)
    at TCP.onread (net.js:529:20)

Getting empty Buffer

Hi,
I am documenting the Ethereum database structure
Here
and
here

When I run the following code on my Ethereum Private Network I just get an empty buffer.

//requirements
var Trie = require('merkle-patricia-tree');
var levelup = require('levelup');
var leveldown = require('leveldown');
var rlp = require('rlp');
//set the database variable
var db = levelup(leveldown('/home/timothymccallum/gethDataDir/geth/chaindata'));
//set the genesis state root variable
var root = '0xe403429877a3fd13eb3da17503af458dd0741ec3b617f2eaf2338ca945edb0e2';
//set the trie variable
var trie = new Trie(db, root);
//set the variable tim which is my account address
var tim = new Buffer('0xb7714bd73152a27939914c644fb7471966378626', 'hex');
//get results from leveldb
trie.get(tim, function (err, val) {
  var decoded = rlp.decode(val);
  console.log(decoded);
});

Please let me know if you have any suggestions as well as useful links and documentation. I am looking to explore the entire workings of Ethereum database.

Thank you

Trying to access nonce, balance, storageRoot and codeHash via the state trie

Hi,
Firstly, thanks for this code, much appreciated.
I am wanting to use this merkle-patricia-tree library to fetch nonce, balance, storageRoot and codeHash from my Ethereum (private network) state trie.

I am accessing the keys of the state trie here

I then am using the following code to access (and print the values of each of the 3 accounts which I created on my private Ethereum network).

var Trie = require('merkle-patricia-tree/secure');
var levelup = require('levelup');
var leveldown = require('leveldown');
var RLP = require('rlp');
var assert = require('assert');

//Connecting to the leveldb database
var db = levelup(leveldown('/home/timothymccallum/gethDataDir/geth/chaindata'));

//Obtaining the state root (of the latest block) using this command web3.eth.getBlock(web3.eth.defaultBlock).stateRoot
var root = '0x8c77785e3e9171715dd34117b047dffe44575c32ede59bde39fbf5dc074f2976';

//Creating a trie object of the merkle-patricia-tree library
var trie = new Trie(db, root);

//Creating a nodejs stream object so that we can access the data
var stream = trie.createReadStream()

//Turning on the stream (because the node js stream is set to pause by default)
stream.on('data', function (data){
  //Capture the RLP serialised data structure
  var arrayOfStateValues = RLP.decode(data.value);
  //Loop through the data of the 3 accounts in the array to obtain the nonce, balance, storageRoot and codeHash for each one 
  for (i = 0; i < arrayOfStateValues.length; i++){ 
    console.log(RLP.decode(arrayOfStateValues[i], [skipRemainderCheck=false])); 
    console.log("");
  }
  
});

The above code gives me the following results

<Buffer >

{ data: <Buffer a2 bc 2e c5 00 00>, remainder: <Buffer > }

{ data: <Buffer 56>,
  remainder: <Buffer e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21> }

{ data: [ [ <Buffer 46>, <Buffer 01>, <Buffer f7> ] ],
  remainder: <Buffer 23 3c 92 7e 7d b2 dc c7 03 c0 e5 00 b6 53 ca 82 27 3b 7b fa d8 04 5d 85 a4 70> }

<Buffer >

{ data: <Buffer 04>, remainder: <Buffer 93 e0> }

{ data: <Buffer 56>,
  remainder: <Buffer e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21> }

{ data: [ [ <Buffer 46>, <Buffer 01>, <Buffer f7> ] ],
  remainder: <Buffer 23 3c 92 7e 7d b2 dc c7 03 c0 e5 00 b6 53 ca 82 27 3b 7b fa d8 04 5d 85 a4 70> }

{ data: <Buffer 01>, remainder: <Buffer > }

{ data: <Buffer 06>,
  remainder: <Buffer 3b 40 6f f8 03 2d 1a 80> }

{ data: <Buffer 56>,
  remainder: <Buffer e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21> }

{ data: [ [ <Buffer 46>, <Buffer 01>, <Buffer f7> ] ],
  remainder: <Buffer 23 3c 92 7e 7d b2 dc c7 03 c0 e5 00 b6 53 ca 82 27 3b 7b fa d8 04 5d 85 a4 70> }

Can you please advise on how I would access these values using your code.

Many thanks
Tim

Rename Trie.prove to Trie.createProof

I always had problems with "reading" the proof-related functions of MPT (in the sense of: grasp the functionality by seeing the names), see e.g. the examples. For a long time I thought that I am just not getting the whole picture.

Now on a closer look it seems to me that the name of the Trie.prove function is just plain wrong, I'll repaste the example code here:

const prove = await Trie.prove(trie, Buffer.from('test'))
const value = await Trie.verifyProof(trie.root, Buffer.from('test'), prove)
console.log(value.toString())

prove is is a noun, in the example above the const prove name makes no sense at all, but also when looking at the implementation of Trie.prove one realizes that this proves nothing at all but just creates a proof, which can then be used (therefore the double confusion on my side) to prove that an item is contained in the trie by using the Trie.verifyProof (or - here it would fit better - an imaginary Trie.prove) function. All this gets also verified by seeing the intuitively used naming conventions and descriptions in the tests.

So I would suggest to rename to:

const proof = await Trie.createProof(trie, Buffer.from('test'))
const value = await Trie.verifyProof(trie.root, Buffer.from('test'), prove)
console.log(value.toString())

Does this make sense respectively am I still missing something?

Along I would suggest the following simplifications/changes to the proof related API:

Old/current:

static async fromProof(proofNodes: Buffer[], proofTrie?: Trie): Promise<Trie>
static async prove(trie: Trie, key: Buffer): Promise<Buffer[]>
static async verifyProof(rootHash: Buffer, key: Buffer, proofNodes: Buffer[])

Suggestion:

type Proof = Buffer[]

static async fromProof(proof: Proof, trie?: Trie): Promise<Trie>
static async createProof(trie: Trie, key: Buffer): Promise<Proof>
static async verifyProof(rootHash: Buffer, key: Buffer, proof: Proof)

It would also be good if these methods could get some doc comments along for signature description.

I would really love to see this integrated in v4 if all makes sense (at least the first part), think this is really some not-to-underestimate mental blocker on using this.

//cc also @gabrocheleau

Add Ethereum name or address to package.json

I would like to see an Ethereum name or address added to the ethereum field of the package.json file. This will allow consumers of the project to know a cannonical address to send funds to. For example, I'm building an application that will allow developers to automatically donate ETH to all of their npm dependencies that have Ethereum addresses in their package.json files: https://sustainus.io (not quite ready, alpha soon)

Verify Proof fails if there is an extension with an embedded branch

This can occur if you insert lots of small key/values into the tree.

A simple example of this bug is below:

const Tree = require('merkle-patricia-tree');
const tree = new Tree();
tree.put('a', 'a', (err) => {
    tree.put('b', 'b', (err) => {
         Tree.prove(tree, 'a', (err, proof) => {
            Tree.verifyProof(tree.root, 'a', proof, (err, val) => {
                // err = "Error: Unexpected end of proof"
                // expected : val === 'a' 
            }
        } 
    }
}

PR #51 fixes this issue.

OutOfMemory when trying to traverse the state trie of the latest Ethereum mainnet block

I have the JavaScript code below, which is traversing the state trie from geth's leveldb for Ethereum mainnet Block #5200035.

After some time with high cpu usage and having reserved about 6Gb RAM, it crashes with FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory.

It looks like the library is having trouble with the size of the mainnet database (~70Gb after a fast sync). A simple key/value lookup with db.get works fine.

Any idea how to approach this problem?

    var Trie = require('merkle-patricia-tree/secure');
    var levelup = require('levelup');
    var leveldown = require('leveldown');
    
    //Connecting to the leveldb database
    var db = levelup(leveldown('../datadir/geth/chaindata'));
    
    //Adding the "stateRoot" value from the block so that we can inspect the state root at that block height.
    var root = "0x4d675087a16f13fa5f61af74d79e08c82de7cf200e63d5b225b4a7937705a3e2"; // Block #5200035
    
    //Creating a trie object of the merkle-patricia-tree library
    var trie = new Trie(db, root);
    
    //Creating a nodejs stream object so that we can access the data
    var stream = trie.createReadStream()
    
    //Turning on the stream (because the node js stream is set to pause by default)
    stream.on('data', function (data){
      console.log(data)
    });

Delete node error

When you try to remove a single node from a branch with a value, an error occurs.

To reproduce:

var Trie = require('merkle-patricia-tree')

trie = new Trie()
var ops = [
   { type: 'put', key: 'do', value: 'verb' }
 , { type: 'put', key: 'dog', value: 'puppy' }
]

trie.batch(ops,
 () => {
  trie.del('dog')
 }
)

Output:

node test_trie.js
/home/local/tmp/node_modules/merkle-patricia-tree/baseTrie.js:631
var branchNodeKey = branchNode.key;
^

TypeError: Cannot read property 'key' of undefined
at processBranchNode (/home/local/tmp/node_modules/merkle-patricia-tree/baseTrie.js:631:40)
at /home/local/tmp/node_modules/merkle-patricia-tree/baseTrie.js:730:19
at /home/local/tmp/node_modules/merkle-patricia-tree/baseTrie.js:212:11
at /home/local/tmp/node_modules/merkle-patricia-tree/util.js:87:7
at /home/local/tmp/node_modules/async/dist/async.js:473:16
at replenish (/home/local/tmp/node_modules/async/dist/async.js:1006:25)
at iterateeCallback (/home/local/tmp/node_modules/async/dist/async.js:995:17)
at /home/local/tmp/node_modules/async/dist/async.js:969:16
at /home/local/tmp/node_modules/merkle-patricia-tree/util.js:83:7
at /home/local/tmp/node_modules/merkle-patricia-tree/baseTrie.js:187:13

The reason is that when nodes in a branch are counted (baseTrie.js:573), the value of the node is taken as a reference to the node.
Next, there is a try to get the node (baseTrie.js:588) and access it (baseTrie.js:490).

missing error message?

So when using a presumably existing key, there is nothing returned, neither an error message nor a value. For this I was just using the example from the Readme, with some example values given or gathered. So if no error is given, despite err in the callback function, how do I know what is happening?

Data gets lost when using string as a key. (memdown + 2 tries)

The test below fails. If you comment line 50 with key thisbreaks the test passes. Changing the key to a Buffer and the test passes. I'm not confident that the pass is a fix, but is possibly just an intermittent fix, as I've been experiencing other oddities.

const levelup = require('levelup');
const memdown = require('memdown');
const MerklePatriciaTree = require('merkle-patricia-tree');

class Trie {
  constructor(db) {
    this.trie = new MerklePatriciaTree(db);
  }

  async get(key) {
    return new Promise((resolve, reject) => {
      this.trie.get(key, (err, value) => {
        if (err) {
          return reject(err);
        } else {
          return resolve(value);
        }
      });
    });
  }

  async put(key, value) {
    return new Promise((resolve, reject) =>
      this.trie.put(key, value, err => {
        if (err) {
          return reject(err);
        } else {
          return resolve();
        }
      }),
    );
  }
}

describe('trie', () => {
  it('minimum test that breaks', async () => {
    const db = levelup(memdown());

    const trie1 = new Trie(db);
    const trie2 = new Trie(db);

    const key1 = Buffer.from('ffffff', 'hex');
    const val1 = Buffer.from('afafaf', 'hex');
    await trie1.put(key1, val1);

    const key2 = Buffer.from('ababab', 'hex');
    const val2 = Buffer.from('deadbeef', 'hex');
    await trie2.put(key2, val2);

    await db.put('thisbreaks', Buffer.from('bdbdbd', 'hex'));

    const result1 = await trie1.get(key1);
    const result2 = await trie2.get(key2);

    expect(result1).toEqual(val1);
    expect(result2).toEqual(val2);
  });
});

Add secondary index

Since leveldb is more performant when fetching ordered data it may give us a speed boost to add a second index. This index would map all the hash of a given root to a uuid. then each path in the tree (with respect to the root) would be prefixed with the UUID and stored as a key with the value being the sha3 of the node.

emit error on missing node

just blows up

/Users/kumavis/Development/Node/vapor-db/node_modules/merkle-patricia-tree/index.js:220
    if (node.type === 'leaf') {
            ^
TypeError: Cannot read property 'type' of null
    at processNode (/Users/kumavis/Development/Node/vapor-db/node_modules/merkle-patricia-tree/index.js:220:13)
    at /Users/kumavis/Development/Node/vapor-db/node_modules/merkle-patricia-tree/index.js:560:9
    at dispatchError (/Users/kumavis/Development/Node/vapor-db/node_modules/level/node_modules/level-packager/node_modules/levelup/lib/util.js:59:35)
    at /Users/kumavis/Development/Node/vapor-db/node_modules/level/node_modules/level-packager/node_modules/levelup/lib/levelup.js:222:14

npm ERR! Darwin 14.0.0
npm ERR! argv "node" "/usr/local/share/npm/bin/npm" "start"
npm ERR! node v0.12.0
npm ERR! npm  v2.9.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: `node index.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] start script 'node index.js'.
npm ERR! This is most likely a problem with the vapor-db package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node index.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls vapor-db
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/kumavis/Development/Node/vapor-db/npm-debug.log

More consistent and better structured directory/file layout

Currently file naming is pretty inconsistent within the source directory of the library and it's relatively hard to grasp for newcomers what the code within different files do and how task/responsibility areas are structured.

File/directory structure should be updated - with respect (or at least considering) to API changes which would occur when moving certain files (e.g. src/secure.js). The following is a proposal which came out of discussion within PR #74:

  • main directory
    • base.js
    • checkpoint.js
    • index.js
    • secure.js
    • node.js
    • proof.js
  • util directory
    • async.js
    • hex.js
    • nibbles.js
    • tasks.js (or eventually merge with async.js)
  • db directory
    • index.js
    • scratch.js
  • readStream directory
    • index.js
    • scratch.js

Replace istanbul with nyc

This library is still using istanbul to generate coverage, even though most of the other ethereumjs libraries have migrated to nyc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.