umbracle / minimal Goto Github PK

View Code? Open in Web Editor NEW

74.0 74.0 13.0 24.9 MB

Modular Ethereum client

License: GNU Lesser General Public License v3.0

Go 99.65% Makefile 0.10% Shell 0.24%

blockchain client ethereum

minimal's People

Contributors

Stargazers

Watchers

Forkers

matrixy gitter-badger 0xpolygon zeina1i blockchain-research-2020

minimal's Issues

Precompiled contracts

Select the precompiled contracts from the genesis file. A factory method for each contract with configuration flags like in the consensus factory.

panic: Some of the data has been pending for too long

When running minimal (commit 1b541ab) sync for a while, getting this stack trace:

SYNC RECEIPTS (90a4c4c617ae6431929dc83d3b720c7c0bbe754f38821042ec528d983be39e95a0f49790692f72fc34acfcd7b5d2662691f631e8eb9e33fd888675605b5cee28): 47
deliver receipts 392: 50
deliver receipts 393: 54
SYNC BODIES (90a4c4c617ae6431929dc83d3b720c7c0bbe754f38821042ec528d983be39e95a0f49790692f72fc34acfcd7b5d2662691f631e8eb9e33fd888675605b5cee28): 52
SYNC HEADERS (867c7bfd6df41ad8537eced23c8c48a4b474151156be0bbd9920f391ae0bc38ef6b2319b4f1afd96a8ab1e3bc5a8e9f340cf94a6f6ef8b9726436c013f7d39bb): 165301 190
deliver bodies 392: 53
block 101461: Completed Pending Completed
block 101651: Completed Completed Pending
block 101841: Completed Completed Completed
block 102031: Completed Completed Completed
block 102221: Completed Completed Completed
block 102411: Completed Completed Completed
block 102601: Completed Completed Completed
block 102791: Completed Completed Completed
block 102981: Completed Completed Completed
block 103171: Completed Completed Completed
block 103361: Completed Completed Completed
block 103551: Completed Completed Completed
block 103741: Completed Completed Completed
block 103931: Completed Completed Completed
block 104121: Completed Completed Completed
block 104311: Completed Completed Completed
block 104501: Completed Completed Completed
block 104691: Completed Completed Completed
block 104881: Completed Completed Completed
block 105071: Completed Completed Completed
block 105261: Completed Completed Completed
block 105451: Completed Completed Completed
block 105641: Completed Completed Completed
block 105831: Completed Completed Completed
block 106021: Completed Completed Completed
block 106211: Completed Completed Completed
block 106401: Completed Completed Completed
block 106591: Completed Completed Completed
block 106781: Completed Completed Completed
block 106971: Completed Completed Completed
block 107161: Completed Completed Completed
block 107351: Completed Completed Completed
block 107541: Completed Completed Completed
block 107731: Completed Completed Completed
block 107921: Completed Completed Completed
block 108111: Completed Completed Completed
block 108301: Completed Completed Completed
block 108491: Completed Completed Completed
block 108681: Completed Completed Completed
block 108871: Completed Completed Completed
block 109061: Completed Completed Completed
block 109251: Completed Completed Completed
block 109441: Completed Completed Completed
block 109631: Completed Completed Completed
block 109821: Completed Completed Completed
block 110011: Completed Completed Completed
block 110201: Completed Completed Completed
block 110391: Completed Completed Completed
block 110581: Completed Completed Completed
block 110771: Completed Completed Completed
block 110961: Completed Completed Completed
block 111151: Completed Completed Completed
block 111341: Completed Completed Completed
block 111531: Completed Completed Completed
block 111721: Completed Completed Completed
block 111911: Completed Completed Completed
block 112101: Completed Completed Completed
block 112291: Completed Completed Completed
block 112481: Completed Completed Completed
block 112671: Completed Completed Completed
block 112861: Completed Completed Completed
block 113051: Completed Completed Completed
block 113241: Completed Completed Completed
block 113431: Completed Completed Completed
block 113621: Completed Completed Completed
block 113811: Completed Completed Completed
block 114001: Completed Completed Completed
block 114191: Completed Completed Completed
block 114381: Completed Completed Completed
block 114571: Completed Completed Completed
block 114761: Completed Completed Completed
block 114951: Completed Completed Completed
block 115141: Completed Completed Completed
block 115331: Completed Completed Completed
block 115521: Completed Completed Completed
block 115711: Completed Completed Completed
block 115901: Completed Completed Completed
block 116091: Completed Completed Completed
block 116281: Completed Completed Completed
block 116471: Completed Completed Completed
block 116661: Completed Completed Completed
block 116851: Completed Completed Completed
block 117041: Completed Completed Completed
block 117231: Completed Completed Completed
block 117421: Completed Completed Completed
block 117611: Completed Completed Completed
block 117801: Completed Completed Completed
block 117991: Completed Completed Completed
block 118181: Completed Completed Completed
block 118371: Completed Completed Completed
block 118561: Completed Completed Completed
block 118751: Completed Completed Completed
block 118941: Completed Completed Completed
block 119131: Completed Completed Completed
block 119321: Completed Completed Completed
block 119511: Completed Completed Completed
block 119701: Completed Completed Completed
block 119891: Completed Completed Completed
block 120081: Completed Completed Completed
block 120271: Completed Completed Completed
block 120461: Completed Completed Completed
block 120651: Completed Completed Completed
block 120841: Completed Completed Completed
block 121031: Completed Completed Completed
block 121221: Completed Completed Completed
block 121411: Completed Completed Completed
block 121601: Completed Completed Completed
block 121791: Completed Completed Completed
block 121981: Completed Completed Completed
block 122171: Completed Completed Completed
block 122361: Completed Completed Completed
block 122551: Completed Completed Completed
block 122741: Completed Completed Completed
block 122931: Completed Completed Completed
block 123121: Completed Completed Completed
block 123311: Completed Completed Completed
block 123501: Completed Completed Completed
block 123691: Completed Completed Completed
block 123881: Completed Completed Completed
block 124071: Completed Completed Completed
block 124261: Completed Completed Completed
block 124451: Completed Completed Completed
block 124641: Completed Completed Completed
block 124831: Completed Completed Completed
block 125021: Completed Completed Completed
block 125211: Completed Completed Completed
block 125401: Completed Completed Completed
block 125591: Completed Completed Completed
block 125781: Completed Completed Completed
block 125971: Completed Completed Completed
block 126161: Completed Completed Completed
block 126351: Completed Completed Completed
block 126541: Completed Completed Completed
block 126731: Completed Completed Completed
block 126921: Completed Completed Completed
block 127111: Completed Completed Completed
block 127301: Completed Completed Completed
block 127491: Completed Completed Completed
block 127681: Completed Completed Completed
block 127871: Completed Completed Completed
block 128061: Completed Completed Completed
block 128251: Completed Completed Completed
block 128441: Completed Completed Completed
block 128631: Completed Completed Completed
block 128821: Completed Completed Completed
block 129011: Completed Completed Completed
block 129201: Completed Completed Completed
block 129391: Completed Completed Completed
block 129581: Completed Completed Completed
block 129771: Completed Completed Completed
block 129961: Completed Completed Completed
block 130151: Completed Completed Completed
block 130341: Completed Completed Completed
block 130531: Completed Completed Completed
block 130721: Completed Completed Completed
block 130911: Completed Completed Completed
block 131101: Completed Completed Completed
block 131291: Completed Completed Completed
block 131481: Completed Completed Completed
block 131671: Completed Completed Completed
block 131861: Completed Completed Completed
block 132051: Completed Completed Completed
block 132241: Completed Completed Completed
block 132431: Completed Completed Completed
block 132621: Completed Completed Completed
block 132811: Completed Completed Completed
block 133001: Completed Completed Completed
block 133191: Completed Completed Completed
block 133381: Completed Completed Completed
block 133571: Completed Completed Completed
block 133761: Completed Completed Completed
block 133951: Completed Completed Completed
block 134141: Completed Completed Completed
block 134331: Completed Completed Completed
block 134521: Completed Completed Completed
block 134711: Completed Completed Completed
block 134901: Completed Completed Completed
block 135091: Completed Completed Completed
block 135281: Completed Completed Completed
block 135471: Completed Completed Completed
block 135661: Completed Completed Completed
block 135851: Completed Completed Completed
block 136041: Completed Completed Completed
block 136231: Completed Completed Completed
block 136421: Completed Completed Completed
block 136611: Completed Completed Completed
block 136801: Completed Completed Completed
block 136991: Completed Completed Completed
block 137181: Completed Completed Completed
block 137371: Completed Completed Completed
block 137561: Completed Completed Completed
block 137751: Completed Completed Completed
block 137941: Completed Completed Completed
block 138131: Completed Completed Completed
block 138321: Completed Completed Completed
block 138511: Completed Completed Completed
block 138701: Completed Completed Completed
block 138891: Completed Completed Completed
block 139081: Completed Completed Completed
block 139271: Completed Completed Completed
block 139461: Completed Completed Completed
block 139651: Completed Completed Completed
block 139841: Completed Completed Completed
block 140031: Completed Completed Completed
block 140221: Completed Completed Completed
block 140411: Completed Completed Completed
block 140601: Completed Completed Completed
block 140791: Completed Completed Completed
block 140981: Completed Completed Completed
block 141171: Completed Completed Completed
block 141361: Completed Completed Completed
block 141551: Completed Completed Completed
block 141741: Completed Completed Completed
block 141931: Completed Completed Completed
block 142121: Completed Completed Completed
block 142311: Completed Completed Completed
block 142501: Completed Completed Completed
block 142691: Completed Completed Completed
block 142881: Completed Completed Completed
block 143071: Completed Completed Completed
block 143261: Completed Completed Completed
block 143451: Completed Completed Completed
block 143641: Completed Completed Completed
block 143831: Completed Completed Completed
block 144021: Completed Completed Completed
block 144211: Completed Completed Completed
block 144401: Completed Completed Completed
block 144591: Completed Completed Completed
block 144781: Completed Completed Completed
block 144971: Completed Completed Completed
block 145161: Completed Completed Completed
block 145351: Completed Completed Completed
block 145541: Completed Completed Completed
block 145731: Completed Completed Completed
block 145921: Completed Completed Completed
block 146111: Completed Completed Completed
block 146301: Completed Completed Completed
block 146491: Completed Completed Completed
block 146681: Completed Completed Completed
block 146871: Completed Completed Completed
block 147061: Completed Completed Completed
block 147251: Completed Completed Completed
block 147441: Completed Completed Completed
block 147631: Completed Completed Completed
block 147821: Completed Completed Completed
block 148011: Completed Completed Completed
block 148201: Completed Completed Completed
block 148391: Completed Completed Completed
block 148581: Completed Completed Completed
block 148771: Completed Completed Completed
block 148961: Completed Completed Completed
block 149151: Completed Completed Completed
block 149341: Completed Completed Completed
block 149531: Completed Completed Completed
block 149721: Completed Completed Completed
block 149911: Completed Completed Completed
block 150101: Completed Completed Completed
block 150291: Completed Completed Completed
block 150481: Completed Completed Completed
block 150671: Completed Completed Completed
block 150861: Completed Completed Completed
block 151051: Completed Completed Completed
block 151241: Completed Completed Completed
block 151431: Completed Completed Completed
block 151621: Completed Completed Completed
block 151811: Completed Completed Completed
block 152001: Completed Completed Completed
block 152191: Completed Completed Completed
block 152381: Completed Completed Completed
block 152571: Completed Completed Completed
block 152761: Completed Completed Completed
block 152951: Completed Completed Completed
block 153141: Completed Completed Completed
block 153331: Completed Completed Completed
block 153521: Completed Completed Completed
block 153711: Completed Completed Completed
block 153901: Completed Completed Completed
block 154091: Completed Completed Completed
block 154281: Completed Completed Completed
block 154471: Completed Completed Completed
block 154661: Completed Completed Completed
block 154851: Completed Completed Completed
block 155041: Completed Completed Completed
block 155231: Completed Completed Completed
block 155421: Completed Completed Completed
block 155611: Completed Completed Completed
block 155801: Completed Completed Completed
block 155991: Completed Completed Completed
block 156181: Completed Completed Completed
block 156371: Completed Completed Completed
block 156561: Completed Completed Pending
block 156751: Completed Completed Completed
block 156941: Completed Completed Completed
block 157131: Completed Completed Completed
block 157321: Completed Completed Completed
block 157511: Completed Completed Completed
block 157701: Completed Completed Completed
block 157891: Completed Completed Completed
block 158081: Completed Completed Completed
block 158271: Completed Completed Completed
block 158461: Completed Completed Completed
block 158651: Completed Completed Completed
block 158841: Completed Completed Completed
block 159031: Completed Completed Completed
block 159221: Completed Completed Completed
block 159411: Completed Completed Completed
block 159601: Completed Completed Completed
block 159791: Completed Completed Completed
block 159981: Completed Completed Completed
block 160171: Completed Completed Completed
block 160361: Completed Completed Completed
block 160551: Completed Completed Completed
block 160741: Completed Completed Completed
block 160931: Completed Completed Completed
block 161121: Completed Completed Completed
block 161311: Completed Completed Completed
block 161501: Completed Completed Completed
block 161691: Completed Completed Completed
block 161881: Completed Completed Completed
block 162071: Completed Completed Completed
block 162261: Completed Completed Completed
block 162451: Completed Completed Completed
block 162641: Completed Completed Completed
block 162831: Completed Completed Completed
block 163021: Completed Completed Completed
block 163211: Completed Completed Completed
block 163401: Completed Completed Completed
block 163591: Completed Completed Completed
block 163781: Completed Completed Completed
block 163971: Completed Completed Completed
block 164161: Completed Completed Pending
block 164351: Completed Completed Completed
block 164541: Completed Completed Completed
block 164731: Completed Pending Completed
block 164921: Completed Pending Pending
block 165111: Pending Completed Completed
block 165301: Pending Completed Completed
block 165491: Waiting Completed Completed
block 7796835: Waiting Completed Completed
panic: Some of the data has been pending for too long!

goroutine 12743 [running]:
github.com/umbracle/minimal/protocol/ethereum.(*Backend).Deliver(0xc0000f0630, 0x49525ec, 0x6, 0xc000000188, 0x4802e60, 0xc00f051f80, 0x0, 0x0)
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:399 +0xb35
github.com/umbracle/minimal/protocol/ethereum.(*task).Run(0xc004dd68c0, 0xc00022e2d0)
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:699 +0x1f5
github.com/umbracle/minimal/protocol/ethereum.(*Backend).runTask(0xc0000f0630, 0xc004dd68c0)
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:167 +0x2f
created by github.com/umbracle/minimal/protocol/ethereum.(*Backend).runSync
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:249 +0x5d
exit status 2

Remove rlpx.Capabilities from server

Server depends on rlpx.Capabilities to match protocols. Add a generic capabilities and info struct that when passed to rlpx is mapped into their own structs.

peers info not working

When running minimal in one terminal, and running commands from another terminal, getting this error:

$ go run main.go peers list
ID
cfbfdb...99
31bf0e...cc
$ go run main.go peers info cfbfdb...99
invalid character 'j' looking for beginning of value
exit status 1

Add an instantSeal field to the sealer. If its on, the sealer will use a 'dummy' consensus protocol and will seal a new block for each transaction. This feature is intended to be used for dev testnets. Not sure if there is a use case where we would use instantseal with ethash or clique.

[Tracking] Clique consensus

Add Clique consensus protocol.

Crash while syncing

Sometimes during synching to mainnet, minimal crashes with this stack trace:

SYNC RECEIPTS (7992fa8471ae906764a6069f98696e767e63aef6b7e8dba6788ae01426ee8e48adb74c2a3db39e51677ab9384ac835e17c49965ac615fdb85e2d39345da4eec6): 72
deliver headers 526: 190
SYNC HEADERS (31bf0e6c8e717affe5970f8c65f53da5225162c5a65f0b0f8f5e227f96f740639878e1660eae37e444f26d8812b3d4e5fec94d9b6dbfba2efffb1891cc8bd7cc): 103361 190
SYNC BODIES (7992fa8471ae906764a6069f98696e767e63aef6b7e8dba6788ae01426ee8e48adb74c2a3db39e51677ab9384ac835e17c49965ac615fdb85e2d39345da4eec6): 79
deliver headers 527: 190
panic: runtime error: index out of range

goroutine 43211 [running]:
github.com/umbracle/minimal/protocol/ethereum.workersHeapImpl.Swap(...)
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:620
container/heap.Remove(0x4bd7060, 0xc000336ec8, 0xffffffffffffffff, 0x80, 0xc0007b68c0)
	/usr/local/Cellar/go/1.12/libexec/src/container/heap/heap.go:72 +0x9c
github.com/umbracle/minimal/protocol/ethereum.(*workersHeap).Remove(0xc000336ec0, 0xc009b810e2, 0x80)
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:571 +0x88
github.com/umbracle/minimal/protocol/ethereum.(*Backend).runTask(0xc0000ea840, 0xc00ac16480)
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:179 +0x5f
created by github.com/umbracle/minimal/protocol/ethereum.(*Backend).runSync
	/Users/alexeyakhunov/workspace/src/github.com/umbracle/minimal/protocol/ethereum/backend.go:252 +0x5d
exit status 2

panic: runtime error: invalid memory address or nil pointer dereference

When I run master from my local comupter (Mac) using go run main.go agent command I got

-- addr --
127.0.0.1
30303
Enode: enode://939adec24cbf8102eaf2fdc09574aac3d1cb4e1cc192b0e3d0445ffecaf4a8fa9cc00bf4bc6af17d3b27ae746933606923ab081b5b42d45a03a8ffc56d01327c@127.0.0.1:30303
Current header (0): 0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x457e11d]

goroutine 1 [running]:
github.com/umbracle/minimal/network.(*Server).Schedule(0xc0008be1c0, 0x508a598, 0x0)
	/Users/user/g/minimal/network/server.go:185 +0x13d
github.com/umbracle/minimal/minimal.NewMinimal(0xc0008a0550, 0xc0008a2aa0, 0x2, 0xc0002ea4f0, 0xe)
	/Users/user/g/minimal/minimal/minimal.go:195 +0x1a18
github.com/umbracle/minimal/command/agent.(*Agent).Start(0xc0000c3620, 0xc0000c3620, 0x0)
	/Users/user/g/minimal/command/agent/agent.go:147 +0x851
github.com/umbracle/minimal/command/agent.(*AgentCommand).Run(0xc0000c58d0, 0xc0000c2010, 0x0, 0x0, 0xc0000c58a0)
	/Users/user/g/minimal/command/agent/command.go:96 +0x1b1
github.com/mitchellh/cli.(*CLI).Run(0xc0001288c0, 0xc0001288c0, 0xc0000fd530, 0xc0001fdf58)
	/Users/user/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:255 +0x1f1
main.Run(0xc0000c2010, 0x1, 0x1, 0xc000092058)
	/Users/user/g/minimal/main.go:24 +0xa3
main.main()
	/Users/user/g/minimal/main.go:12 +0x62
exit status 2
➜  minimal git:(master) ✗

go version go1.12.4 darwin/amd64

Fix Spurious dragon bug with ripemd

Add state and trie

Replace go-ethereum trie with an immutable merkle tree and model the state in the following way:

s := NewState()
txn := s.Txn()
txn.ApplyEthTransaction(...)
txn.ApplyEthTransaction(...)
x := txn.Commit()

After the commit a new immutable trie with a new state root (x) is created. Dereferencing the state root is as easy as dereferencing the variable x and Go gc would remove the values from memory.

[Tracking] Fast sync protocol

Syncer has to sync the ethereum state. Following fast mode algorithms, pick a pivot point and download recursively from that root.
It would be necessary to get an approximate equation of the total number of nodes to download per block. Thus, it would be possible to estimate an approximate download time for the total state.
The scheduler could split the download bandwidth between headers data (header, body and receipts) and state. By modifying this quotas is possible to control the total time to sync the blockchain.

Network max connections and peer management

Configuration options on the server for the max number of connections and the max number of incoming connections allowed. Dialing tasks are idle if there are no more slots to open a connection.

Syncer (or any other service connected to the network) should be able to set a peer as unnecessary at any time (i.e. server.SetUnnecessary(peerID string)). Server would evict that peer from the peerset and dial for another one.

Add state cache

Add state cache to the state transition for the SLOAD operations.

Add api endpoints

Add different RPC endpoints in the Api. Only 'eth_getBlockByHash' and 'eth_getBlockByNumber' are implemented now.

Eth protocol failed queries

Some eth queries (i.e. getHeaders, getReceipts) may return empty if the peer does not have those values. Since we use fields in the response to return the values, it wont be able to return on queries that return those empty sets.

For example, we send four header queries but the peer does not have any of the values, it will return empty set for all the queries. All the queries will block until the timeout despite having receive already the requests.

To fix this we keep a list of outstanding requests per each set (headers, receipts, bodies and data). Every time we receive a response (either empty or not) we reduce the outstanding requests. If the number of outstanding requests is zero but there are still pending requests to be fulfilled we finished them since we can assume that the empty responses received where to fulfill them.

Dial gets stuck

Dial in the server gets sometimes stuck. I think it may be because Rlpx handshake operations wait for messages to arrive.

Invalid pipe on Windows

C:\Users\Bruno\repos\minimal (develop -> origin)
λ go run main.go agent
2019-08-24T01:42:25.439+0200 [INFO]  minimal.server: ID: enode=enode://f9a97ab44ede11c5d7f944148c91a2f7234b97611f93d7ec3399fa21e045827eb786a59f22535ac694796c90f8d73aec84a536ff3b437005019fea7686ed9c71@0.0.0.0:30303
2019-08-24T01:42:25.597+0200 [INFO]  minimal.ethereum: Header: num=0 hash=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
2019-08-24T01:42:25.597+0200 [INFO]  JsonRPC-HTTP: Running: addr=127.0.0.1:8545
panic: Invalid pipe address '/tmp/minimal.ipc'.

goroutine 1 [running]:
github.com/umbracle/minimal/command/agent.(*Agent).Start(0xc000214fe0, 0xc000214fe0, 0x0)
        C:/Users/Bruno/repos/minimal/command/agent/agent.go:189 +0x1278
github.com/umbracle/minimal/command/agent.agentRunE(0x13aa9a0, 0x13d5c18, 0x0, 0x0, 0x0, 0x0)
        C:/Users/Bruno/repos/minimal/command/agent/root.go:110 +0x217
github.com/spf13/cobra.(*Command).execute(0x13aa9a0, 0x13d5c18, 0x0, 0x0, 0x13aa9a0, 0x13d5c18)
        C:/Users/Bruno/go/pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x46c
github.com/spf13/cobra.(*Command).ExecuteC(0x13aa740, 0xa, 0xc00009c500, 0xc00009c500)
        C:/Users/Bruno/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2f3
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/Bruno/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
        C:/Users/Bruno/repos/minimal/main.go:15 +0x34
exit status 2

Add consensus algorithm selection in network json

Network spec json files should have a field to select the consensus algorithm (i.e. ethash, clique...).

Remove go-ethereum dependencies

These are the last go-ethereum dependencies to be removed:

Optimized immutable trie for state transactions

State uses this radix library to implement the transaction object that the VM uses to transition state. I think it could be more efficient with an adhoc Hamt trie.

[Tracking] Graphql api

Fair rate upload

Use a network scheduler different from the sync scheduler in #13 to rate limit the number of eth requests we receive. We could either set a bandwidth rate in the peer or in the ethereum protocol.

Join protocol and syncer

What are actually the protocol backends? The only thing ethereum backend does is to offer and interface to syncer to send eth64 messages and get responses. Besides, it resolves messages related to requests from other peers. So, in general terms is just a big communications (i.e. rpc) library that is used exclusively by syncer. It would make sense to put syncer and all the sync logic in ethereum.

Then, the protocol backend folder becomes the sync algorithm itself. This backend would have the callbacks to start new handlers for the peers (what is now called the protocol) and the backend would keep all the specific handlers for the server.

Benefits

We remove one big entity from the minimal core.
Backend could have as an argument a bandwidth limiter that could be shared between different backend protocol competing to use bandwidth.
It has a way to notify minimal that he can start to seal blocks (this is, is only valid to seal blocks when the state is fully synced). Then, it will be possible to implement a raft backend that would notify minimal to seal blocks once he is the leader.

Note that this is actually the approach followed in geth.

Add logger

Improve syncer

Current

Syncer runs N workers and each one queries a job (headers, bodies or receipts) to a given peer at a time. The peer selection is done by sorting all the active peers to which we are connected and sorting by number of outstanding queries (4 is the max number of outstanding queries).

Syncer keeps a queue (linked list) of query blocks. Each item in the queue is a group of 100 blocks. The query jobs are done per item, this is, the max number of elements in the headers query will be 100 (when the max number of elements we can retrieve from geth is 190). In the case of bodies and receipts, if only six blocks in the item have bodies, the bodies query will ask for only six elements. This approach underuses the network.

The only query limit factor for a given peer is the max number of outstanding queries (4). It does not take into account the congestion (time to receive the response).

Proposal

The sync scheduler creates a peer connection service for each peer. That connection will request bandwidth from the scheduler to make the queries. Then, it will ask the scheduler picker for a job. Peer connection will try to do as many as concurrent requests it can do (if there is available bandwidth). Periodically, it measures the rtt time from the queries to change the number of concurrent requests it can make. The higher the rtt the lower the number of requests.

It will periodically check the rate of each connection. If there are too many active connections competing for the bandwidth it will idle the connections with the lower rate. At the same time, if there is unused bandwidth it will active idle peers if there are any.

Uncles logic

There is no much uncles logic right now on place.

Adhoc deriveSha

TxHash and ReceiptsHash from the block header is generated with the deriveSha function. It creates a trie with the list of values. The value is the item and the key is the rlp encoding of the index of the item.

In the current implementation, it builds a full merkle trie and hashes the values using the generic implementation for the state. However, due to the nature of the values (monotonal increasing values) it should be possible to create an adhoc version to exploid some features. For example: all the leaf values will be the type 'short' with '0x10' as key, groups of 16 consecutive values will be hashed together (except for the zero)...

Rlpx stream transport as a net.Conn interface

We have to difference between the Rlpx transport protocol (ping/pong, connect/disconnect) and the stream inside that transport that the protocols use to communicate. The goal of this issue is to make the protocol stream a net.Conn interface.

We can achieve two goals with this approach:

Use any transport protocol (i.e. plain net/http) that fulfills net.Conn as the communication for the protocols. This makes the client flexible and testing the protocols will be easier since it would not require to mockup a complete Rlpx transport with handshake.
Improve memory management since transport and protocol will take care of their own memory.

Add build script

Add a build script to create binaries for different platforms

Initial state of the genesis

One of the first operations in minimal is the build of the initial genesis allocations to create the genesis block hash (It takes considerable time in mainnet). We can do this operation the first time if the storage is completely empty and there is no genesis file.

Add Goerli chain

Add Goerli chain json file and add a test here to check that the computed state root is correct.

Replace metrics library

Replace go-metrics with metrics.

Go-metrics has a generic interface to dispatch metrics to different sinks (Prometheus, Datadog...), though we only use the prometheus sink.

On the other side, metrics only supports Prometheus, but the library is much lighter and there seem to be already some agents to move metrics from Prometheus to other sinks like Datadog or InfluxDB. Besides, go-metrics does not support WASM compilation.

Remove iradix dependency in the state trie

The state trie takes an iradix from the state transition object every time there is a new state change. We should remove that and only send a map of accounts and a map of state changes. This makes it easier to add #61 and a tracer.

[Tracking] LibP2P

Store Ethash cache in file

Minimal loads the chain in json format every time it starts. In the case of the foundation chain, the list of predefined accounts is huge. It may take up to half a second to load those accounts. We should avoid loading those accounts and only unmarshal them when necessary (writing the genesis file).

Update

It turns out that the slow initial time was not due to the loading of the chain .json but the generation of the Ethash cache.

EVM array overflow with memory set operations

There is a case for an overflow in the EVM with Opcodes that resize the memory like here. The memory set operation does not check if the resize length of the memory overflows the max size of an array in Go. This would fail in the gas cost, however, the gas cost is computed later so the error is not catched.

It will require a change in the flow of all the memory operations:

Resize the memory to the correct size (or at least compute the cost)
Calculate gas cost
Set the value and resize if necessary

This issue is not covered in the ethereum/tests repo.
There are 3 txns around block 300K that produce this error.

Add config file

Add config file (preferably HCL) to run the agent. The config file defines the parameters for the whole client: syncer, network, traffic, blockchain...

Blockchain forks

Save more information about the forks. Right now, there is only a list stored in RLP format in the blockchain storage. The new architecture would store forks (aka HeaderChain) in boltdb with some contextual information (ancestor, height...).

Helper method to parse enode format

Add a new helper method under /helper/enode to parse enode format.

Ethereum protocol rlp decoder bottleneck

In the Ethereum protocol (i.e. eth63) there is a bottleneck decoding the message with rlp.DecodeBytes. It uses the old rlp decoder with reflection and with reflect.New to create new objects.

We need to solve two issues. First, how do we decode structs with rlpv2. the Rlp parser should copy the bytes from the message as in fastjson, then, we could even process the messages in parallel. Second, how do we reuse the structs in the syncer (we may use the headers in the queue).

Discovery interface

Expose to the server only a discover interface so that we can plug any discovery protocol (i.e discv5 or consul).

type Discovery interface {
   Start()
   GetServices() ([]*Service, error)
   RegisterService(s *Service)
}

The directory structure would be:

/network
  /discovery
    /consul
    /disc
    /discv5
    discovery.go
  server.go