Description

Currently, each trie key is generated with a call to the nativelib. However, moving to hash instead of compressed leads to a big optimisation opportunity: using hashMany is far more efficient.
This PR aims at batching all trie keys generation for a block into a single call to the native lib.

This depends on PR 158

Currently, root hash is computed using the commit function, which returns the mapToScalars of the commitment.
However, it should be the serialized compressed representation of that commitment instead.

It is therefore expected that getRootHash() returns the value of commit_root instead of commit.
One should make the proper modifications to the code and to the test cases.

Description

Currently, the batch processor gathers all dirty nodes and process them for crypto-commitments and hashing.
However, persisting in the DB is done separately, meaning we need to climb the tree up again.
It would be more efficient to commit to the DB while processing batches.

Currently, commitments' scalars are stored inside the children nodes in memory and in database.
However, for generating proofs, we need to get all scalars either way.
We propose to store them along the commitment in the database at the parent node.

To minimize refactoring for now, the idea would be for a StoredNode's NodeFactory to retrieve the parent node info + children hashes from DB, create the parent node, and distribute the children hashes to the children.
This way, a proof generation will not need to load the children node only to retrieve hashes.

Description

There are some discrepancies in the commitments when looking into logs from Geth and from Besu on the genesis block creation.
See attachments for the logs:
besu-genesis.txt
geth-genesis.dot.txt

Examples of discrepancies:

1. 0x00 is ok, but 0x0000 is not.
2. 0xc4 is not good.

Your goal is to investigate this issue, find where and why there is such discrepancies.
If all that is needed is a simple fix, you can fix it. Otherwise we can decompose the task further.

Description

Actually, triekeyadaptor uses commitAsCompressed to generate trie keys from addresses and slots.
To conform to the new specs for trie keys generation, we need to change the hash function from commitAsCompressed to hash.

This depends on merging PR 158 from besu-native

Description

Currently, trie keys generation will gather all transactions data and generate a single large batch of trie keys at once.
However, there are already mechanisms implemented in Besu to preload those trie keys in a background thread during block processing.
We should investigate using that mechanism.

Description

Currently, hashing is done node by node.
In geth, they use hashMany native function to make use of Montgoméry's optimisation to batch hashing level by level.
We should implement the same strategy in a new HashVisitor.
This new visitor might require a new Interface as well, returning List of Updated Nodes instead of the updated node.

Description

Currently, values are stored in the database one key-value pair at a time. However, for performance reasons, it seems to be more adequate to store all 256 leaf-values under a single stem together.

This PR should introduce a new CommitVisitor and StoredNodeFactory implementing this logic.
Keeping the previous visistors and factory will allow us to benchmark both solutions.

For the sparse update of the commitment, we'll need some workaround our java trie implemtation. Specifically:
https://github.com/hyperledger/besu-verkle-trie/blob/main/src/main/java/org/hyperledger/besu/ethereum/trie/verkle/visitor/HashVisitor.java#L79

This method currently just does this:

    hashes[2] = hasher.groupToField(hasher.commit(leftValues));
    hashes[3] = hasher.groupToField(hasher.commit(rightValues));
    final Bytes32 hash = hasher.groupToField(hasher.commit(hashes));

For LeftValues it should be:

hashes[2] = hasher.updateCommitmentSparse(oldC1, oldValuesArray, newValuesArray, indexes)

For right values:

hashes[3] = hasher.updateCommitmentSparse(oldC2, oldValuesArray, newValuesArray, indexes)

And for hash:

final Bytes32 hash = hasher.groupToField(hasher.updateCommitmentSparse(OldCommitment, oldValuesArray, newValuesArray, indexes));

Following the signature:
https://github.com/hyperledger/besu-native/blob/main/ipa-multipoint/src/main/java/org/hyperledger/besu/nativelib/ipamultipoint/LibIpaMultipoint.java#L75

Potential issue is that maybe oldValues are not available in the context of hashVisitor, but i need to double-check that.

Description

Computation of commitments should be done only if not already computed and up-to-date.
Currently, HashVisitor returns the cached value if the node is not dirty (has not been modified since last write in database) and if the commitment is non-empty.
Note that it cannot detect if the commitment should be modified, or if commitment is up-to-date but other attributes are not.
Moreover, currently calling two getRootHash() in a row recomputes the hash, which it should not.

We therefore want to separate the logic of being up-to-date in the database and for the commitment.
Therefore, whenever the commitment is out-of-sync, it should be set to empty.

Expected

1. Write a test that shows that commitments are not recomputed between two consecutive calls to getRootHash(). You could for example call getRootHash a first time, then hack the commitment value to a wrong value, make a second call and check that the second call returns the wrong value.
2. Modify PutVisitor to set commitments as empty whenever a leaf is modified.
3. Modify RemoveVisitor to set commitments as empty whenever a leaf is removed.

Description

Currently, chunkifyCode in TrieKeyAdaptor has been implemented and tested on two contract's bytecode.
However, we are uncertain of the validity of the current implementation in the case of PUSH32.
The ultimate edge-case would be to have a PUSH32 opcode at a position a multiple of 31. This would mean that the next code chunk is made of only push data, and that data even spills one byte into the next-next chunk.

Expected
A testcase for the described edge-case. It should be added to the JSON test file.
If test fails, correct the current implementation so that it passes.

Description

Currently, nodes have a flag dirty that should be set whenever the node has been modified since last write in the database.
It is also used to check if commitments should be recomputed or not.
However, the CommitVisitor does not seem to set written nodes as clean.

Expected

1. Write a test that validates that the root node is clean after calling commit().
2. If test fails, ensure that CommitVisitor sets nodes clean after storing node in database.