System information
Geth version:
Geth
Version: 1.8.26-stable
Git Commit: 9ab0e40ee61d2e23008a45087ef7056a8bc228cd
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.13.8
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.13
OS & Version:
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
Commit hash :
ommit 9ab0e40ee61d2e23008a45087ef7056a8bc228cd (HEAD -> newton, origin/newton, master)
Author: VieYang <[email protected]>
Date: Tue Jun 16 07:15:20 2020 +0800
README: Update NewChain ReadMe
Description
Setup a read-only node (archive mode) first.
The geth process was accidentally killed (for example, suddenly power off, or kill -9 {pid} to simulate the unclean shutdown) and the chain database was corrupted.
It cannot recover by itself. The only way is to totally remove the chain database and sync from the beginning.
As a comparison, the original ethereum node can be smoothly recovered from the accident.
Steps to reproduce the behaviour
newchain: archive mode (read-only), Geth/v1.8.26-stable-9ab0e40e/linux-amd64/go1.13.8
source code: https://github.com/newtonproject/newchain.git [branch: newton]
datadir: ~/.newchain/test/
init: geth --datadir . init ../mainnet/genesis.json
start: geth --config ~/.newchain/test/config.toml
then kill it: kill -9 {geth pid}
INFO [06-20|12:37:07.329] Maximum peer count ETH=25 LES=0 total=25
INFO [06-20|12:37:07.443] Starting peer-to-peer node instance=Geth/v1.8.26-stable-9ab0e40e/linux-amd64/go1.13.8
INFO [06-20|12:37:07.444] Allocated cache and file handles database=/home/evan/.newchain/test/geth/chaindata cache=768 handles=524288
INFO [06-20|12:37:08.035] Initialised chain configuration config="{ChainID: 1012 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: <nil> ConstantinopleFix: <nil> Engine: clique}"
INFO [06-20|12:37:08.035] Initialising Ethereum protocol versions="[63 62]" network=1012
INFO [06-20|12:37:08.088] Loaded most recent local header number=0 hash=76905a…7d1ed6 td=1 age=1y8mo2w
INFO [06-20|12:37:08.088] Loaded most recent local full block number=0 hash=76905a…7d1ed6 td=1 age=1y8mo2w
INFO [06-20|12:37:08.088] Loaded most recent local fast block number=0 hash=76905a…7d1ed6 td=1 age=1y8mo2w
INFO [06-20|12:37:08.089] Stored checkpoint snapshot to disk number=0 hash=76905a…7d1ed6
INFO [06-20|12:37:08.330] New local node record seq=1 id=6c964dcb2546f50e ip=127.0.0.1 udp=38311 tcp=38311
INFO [06-20|12:37:08.331] Started P2P networking self=enode://e5717bc40d5e8895ab8a693ffc09a323a81e363e1f38b7957b7cb280d82ee945210bf8dbbd7344ca930e7f5d936ae065133d6204f4ed811b6c09ec23008be264@127.0.0.1:38311
INFO [06-20|12:37:08.335] IPC endpoint opened url=/tmp/geth.ipc
INFO [06-20|12:37:08.336] HTTP endpoint opened url=http://127.0.0.1:8801 cors= vhosts=localhost
INFO [06-20|12:37:18.333] Block synchronisation started
INFO [06-20|12:37:24.125] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=51.622ms mgasps=0.000 number=192 hash=a16801…47cd98 age=1y6mo1w cache=0.00B
INFO [06-20|12:37:26.068] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=48.523ms mgasps=0.000 number=384 hash=05ca8e…e68cf6 age=1y6mo1w cache=0.00B
INFO [06-20|12:37:27.076] Imported new chain segment blocks=157 txs=0 mgas=0.000 elapsed=43.255ms mgasps=0.000 number=541 hash=3f6993…bacc8d age=1y6mo1w cache=0.00B
INFO [06-20|12:37:27.860] Imported new chain segment blocks=227 txs=1 mgas=0.028 elapsed=55.408ms mgasps=0.508 number=768 hash=a13c2d…f5629c age=1y6mo1w cache=844.00B
INFO [06-20|12:37:28.256] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=74.266ms mgasps=0.000 number=960 hash=a57230…9939e7 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:28.723] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=69.723ms mgasps=0.000 number=1152 hash=58a18e…3540ba age=1y6mo1w cache=844.00B
INFO [06-20|12:37:29.324] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=50.019ms mgasps=0.000 number=1344 hash=39d212…d320e3 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:30.041] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=52.563ms mgasps=0.000 number=1536 hash=496460…cb6183 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:30.878] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=48.214ms mgasps=0.000 number=1728 hash=2357be…bc4941 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:31.961] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=78.605ms mgasps=0.000 number=1920 hash=b693b6…e18d60 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:32.712] Imported new chain segment blocks=192 txs=0 mgas=0.000 elapsed=72.173ms mgasps=0.000 number=2112 hash=d3714c…f39ba4 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:33.786] Imported new chain segment blocks=83 txs=0 mgas=0.000 elapsed=53.902ms mgasps=0.000 number=2195 hash=89bdd2…f30925 age=1y6mo1w cache=844.00B
INFO [06-20|12:37:35.734] Imported new chain segment blocks=33 txs=3 mgas=5.634 elapsed=11.767ms mgasps=478.759 number=2228 hash=97b6b7…b68463 age=1y6mo1w cache=13.13kB
INFO [06-20|12:37:36.529] Imported new chain segment blocks=264 txs=5 mgas=7.448 elapsed=92.288ms mgasps=80.707 number=2492 hash=51c449…f32b1b age=1y6mo1w cache=15.28kB
INFO [06-20|12:37:37.489] Imported new chain segment blocks=207 txs=5 mgas=0.105 elapsed=79.985ms mgasps=1.313 number=2699 hash=433e2e…80425a age=1y6mo1w cache=15.46kB
INFO [06-20|12:37:37.955] Imported new chain segment blocks=257 txs=2 mgas=0.225 elapsed=85.366ms mgasps=2.634 number=2956 hash=87951a…d930d2 age=1y6mo1w cache=19.69kB
INFO [06-20|12:37:38.865] Imported new chain segment blocks=116 txs=1 mgas=0.080 elapsed=58.415ms mgasps=1.372 number=3072 hash=bd6b7c…34b005 age=1y6mo1w cache=21.80kB
INFO [06-20|12:37:38.868] Imported new chain segment blocks=12 txs=0 mgas=0.000 elapsed=3.166ms mgasps=0.000 number=3084 hash=573c88…a525ee age=1y6mo1w cache=18.93kB
INFO [06-20|12:37:40.085] Imported new chain segment blocks=180 txs=2 mgas=0.185 elapsed=46.078ms mgasps=4.013 number=3264 hash=10264c…2a831d age=1y6mo1w cache=22.64kB
INFO [06-20|12:37:40.103] Imported new chain segment blocks=70 txs=0 mgas=0.000 elapsed=17.088ms mgasps=0.000 number=3334 hash=ebc7a6…e1fd4d age=1y6mo1w cache=20.52kB
INFO [06-20|12:37:41.892] Imported new chain segment blocks=122 txs=2 mgas=0.104 elapsed=56.560ms mgasps=1.841 number=3456 hash=268970…d70f95 age=1y6mo1w cache=25.59kB
INFO [06-20|12:37:42.903] Imported new chain segment blocks=384 txs=4 mgas=0.090 elapsed=111.686ms mgasps=0.806 number=3840 hash=cfad3c…0a3818 age=1y6mo1w cache=20.92kB
INFO [06-20|12:37:43.891] Imported new chain segment blocks=77 txs=0 mgas=0.000 elapsed=45.505ms mgasps=0.000 number=3917 hash=1f8ba2…7ddbf3 age=1y6mo1w cache=20.92kB
Killed
restart to see if it crash/recover: geth --config ~/.newchain/test/config.toml
failed to recover:
INFO [06-20|12:40:45.220] Block synchronisation started
INFO [06-20|12:40:55.611] Importing heavy sidechain segment blocks=2048 start=542 end=2589
WARN [06-20|12:40:55.619] Synchronisation failed, dropping peer peer=423f38c2ae6ee4cf err="retrieved hash chain is invalid"
INFO [06-20|12:40:58.882] Importing heavy sidechain segment blocks=2048 start=2196 end=4243
ERROR[06-20|12:40:58.945] Impossible reorg, please file an issue oldnum=542 oldhash=e47a16…1ccb0d newnum=542 newhash=e47a16…1ccb0d
WARN [06-20|12:40:58.984] Synchronisation failed, dropping peer peer=b33bb3d280e7e401 err="retrieved hash chain is invalid"
INFO [06-20|12:41:15.825] Importing heavy sidechain segment blocks=2048 start=2214 end=4261
ERROR[06-20|12:41:15.830] Impossible reorg, please file an issue oldnum=2196 oldhash=96e063…44cbe7 newnum=2196 newhash=96e063…44cbe7
WARN [06-20|12:41:15.831] Synchronisation failed, dropping peer peer=53227d4fe44350b1 err="retrieved hash chain is invalid"
INFO [06-20|12:41:19.484] Importing heavy sidechain segment blocks=2048 start=2222 end=4269
ERROR[06-20|12:41:19.489] Impossible reorg, please file an issue oldnum=2214 oldhash=1ec9f8…a50418 newnum=2214 newhash=1ec9f8…a50418
WARN [06-20|12:41:19.490] Synchronisation failed, dropping peer peer=ca3e9e0aea417754 err="retrieved hash chain is invalid"
it cannot be recovered any more. the only way is to clean the full database and re-sync from the beginning.
ethereum test
go-ethereum: archive mode, Geth/v1.9.16-unstable-9a188c97-20200619/linux-amd64/go1.13.8
source code: https://github.com/newtonproject/newchain.git [branch: master]
clone the source, set up the go proxy first:
$ go env -w GO111MODULE=on
$ go env -w GOPROXY=https://goproxy.cn,direct
then, make
run: ./build/bin/geth
it will init ~/.ethereum automatically
NFO [06-20|16:41:32.451] Looking for peers peercount=0 tried=5 static=0
INFO [06-20|16:41:42.452] Looking for peers peercount=0 tried=4 static=0
INFO [06-20|16:41:52.453] Looking for peers peercount=0 tried=4 static=0
INFO [06-20|16:42:02.454] Looking for peers peercount=1 tried=44 static=0
INFO [06-20|16:42:03.016] Imported new state entries count=273 elapsed="2.185µs" processed=273 pending=4369 retry=0 duplicate=0 unexpected=0
INFO [06-20|16:42:03.904] Imported new block headers count=192 elapsed=571.002ms number=192 hash="723899…123390" age=4y11mo2w
INFO [06-20|16:42:03.906] Migrated ancient blocks count=1 elapsed="123.472µs"
INFO [06-20|16:42:03.929] Imported new block receipts count=2 elapsed=23.343ms number=2 hash="b495a1…4698c9" age=4y11mo2w size=1.69KiB
INFO [06-20|16:42:08.178] Imported new state entries count=384 elapsed="3.393µs" processed=657 pending=10513 retry=0 duplicate=0 unexpected=0
INFO [06-20|16:42:08.498] Imported new block headers count=192 elapsed=11.811ms number=384 hash="d3d5d5…c79cf3" age=4y11mo2w
INFO [06-20|16:42:08.498] Imported new block receipts count=4 elapsed=14.817ms number=6 hash="1f1aed…6b326e" age=4y11mo2w size=3.30KiB
INFO [06-20|16:42:12.454] Looking for peers peercount=1 tried=27 static=0
INFO [06-20|16:42:14.668] Imported new state entries count=384 elapsed="1.683µs" processed=1041 pending=16657 retry=0 duplicate=0 unexpected=0
INFO [06-20|16:42:17.795] Imported new block headers count=192 elapsed=11.414ms number=576 hash="41a746…6a8b38" age=4y11mo2w
check syncing:
$ ~/git/go-ethereum/build/bin/geth attach ~/.ethereum/geth.ipc
> eth.syncing
{
currentBlock: 842350,
highestBlock: 10302878,
knownStates: 1028559,
pulledStates: 1027298,
startingBlock: 48071
}
kill -9 {geth pid}:
INFO [06-20|21:46:05.158] Looking for peers peercount=1 tried=5 static=0
INFO [06-20|21:46:09.068] Imported new block headers count=192 elapsed=45.391ms number=869669 hash="9b6fee…dc6fc0" age=4y5mo3w
INFO [06-20|21:46:10.045] Imported new block receipts count=32 elapsed=35.892ms number=842326 hash="51a9cb…ddc717" age=4y5mo4w size=33.46KiB
INFO [06-20|21:46:15.159] Looking for peers peercount=1 tried=4 static=0
INFO [06-20|21:46:15.408] Imported new block headers count=192 elapsed=46.451ms number=869861 hash="deb641…fc4527" age=4y5mo3w
WARN [06-20|21:46:15.475] Served eth_coinbase reqid=3 t="22.196µs" err="etherbase must be explicitly specified"
INFO [06-20|21:46:16.007] Imported new block receipts count=24 elapsed=28.009ms number=842350 hash="2b7a7b…622e5a" age=4y5mo4w size=27.10KiB
INFO [06-20|21:46:23.323] Imported new block headers count=192 elapsed=33.512ms number=870053 hash="b3caeb…884172" age=4y5mo3w
INFO [06-20|21:46:24.014] Imported new block receipts count=31 elapsed=27.383ms number=842381 hash="94ba54…900062" age=4y5mo4w size=37.53KiB
INFO [06-20|21:46:25.159] Looking for peers peercount=1 tried=5 static=0
INFO [06-20|21:46:29.331] Imported new state entries count=963 elapsed=13.531ms processed=1028261 pending=1389 retry=0 duplicate=691 unexpected=1724
Killed
restart. it successfully recovered:
$ ./build/bin/geth
INFO [06-20|21:48:05.681] Starting Geth on Ethereum mainnet...
INFO [06-20|21:48:05.681] Bumping default cache on mainnet provided=1024 updated=4096
INFO [06-20|21:48:05.682] Maximum peer count ETH=50 LES=0 total=50
INFO [06-20|21:48:05.682] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [06-20|21:48:05.683] Starting peer-to-peer node instance=Geth/v1.9.16-unstable-9a188c97-20200619/linux-amd64/go1.13.8
INFO [06-20|21:48:05.683] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB
INFO [06-20|21:48:05.683] Allocated cache and file handles database=/home/evan/.ethereum/geth/chaindata cache=2.00GiB handles=524288
INFO [06-20|21:48:16.539] Opened ancient database database=/home/evan/.ethereum/geth/chaindata/ancient
INFO [06-20|21:48:16.566] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, YOLO v1: <nil>, Engine: ethash}"
INFO [06-20|21:48:16.566] Disk storage enabled for ethash caches dir=/home/evan/.ethereum/geth/ethash count=3
INFO [06-20|21:48:16.566] Disk storage enabled for ethash DAGs dir=/home/evan/.ethash count=2
INFO [06-20|21:48:16.566] Initialising Ethereum protocol versions="[65 64 63]" network=1 dbversion=7
INFO [06-20|21:48:16.567] Loaded most recent local header number=870053 hash="b3caeb…884172" td=5735210297559978996 age=4y5mo3w
INFO [06-20|21:48:16.567] Loaded most recent local full block number=0 hash="d4e567…cb8fa3" td=17179869184 age=51y2mo1w
INFO [06-20|21:48:16.567] Loaded most recent local fast block number=842381 hash="94ba54…900062" td=5484495146596747535 age=4y5mo4w
INFO [06-20|21:48:16.567] Loaded local transaction journal transactions=0 dropped=0
INFO [06-20|21:48:16.567] Upgrading chain index type=bloombits percentage=8
INFO [06-20|21:48:16.567] Regenerated local transaction journal transactions=0 accounts=0
INFO [06-20|21:48:16.594] Allocated fast sync bloom size=1.60GiB
INFO [06-20|21:48:16.658] New local node record seq=3 id=daedc92bfc565412 ip=127.0.0.1 udp=30303 tcp=30303
INFO [06-20|21:48:16.660] IPC endpoint opened url=/home/evan/.ethereum/geth.ipc
INFO [06-20|21:48:16.660] Started P2P networking self=enode://d9bc30d6ca2626969a2158180654d8dc071d1f795e0a122f3587686b35698bdbddc171717f31a7da16aaf6648b9414a4719d708d691dd67935630672b7096a95@127.0.0.1:30303
INFO [06-20|21:48:21.121] Initialized fast sync bloom items=1024542 errorrate=0.000 elapsed=4.527s
INFO [06-20|21:48:24.623] Upgrading chain index type=bloombits percentage=26
INFO [06-20|21:48:26.950] Looking for peers peercount=0 tried=4 static=0
INFO [06-20|21:48:32.806] Upgrading chain index type=bloombits percentage=43
INFO [06-20|21:48:37.337] Looking for peers peercount=0 tried=17 static=0
INFO [06-20|21:48:40.857] Upgrading chain index type=bloombits percentage=61
INFO [06-20|21:48:47.445] Looking for peers peercount=1 tried=10 static=0
INFO [06-20|21:48:48.934] Upgrading chain index type=bloombits percentage=78
WARN [06-20|21:48:57.003] Served eth_coinbase reqid=3 t="22.334µs" err="etherbase must be explicitly specified"
INFO [06-20|21:48:57.112] Upgrading chain index type=bloombits percentage=96
INFO [06-20|21:48:59.040] Finished upgrading chain index type=bloombits
INFO [06-20|21:49:01.005] Looking for peers peercount=0 tried=5 static=0
INFO [06-20|21:49:11.538] Looking for peers peercount=0 tried=2 static=0
INFO [06-20|21:49:21.539] Looking for peers peercount=0 tried=4 static=0
INFO [06-20|21:49:31.540] Looking for peers peercount=0 tried=6 static=0
INFO [06-20|21:49:41.542] Looking for peers peercount=0 tried=4 static=0
WARN [06-20|21:49:44.036] Served eth_getWork reqid=12 t="123.111µs" err="no mining work available yet"
INFO [06-20|21:49:51.542] Looking for peers peercount=0 tried=4 static=0
INFO [06-20|21:50:01.543] Looking for peers peercount=0 tried=4 static=0
INFO [06-20|21:50:11.544] Looking for peers peercount=0 tried=2 static=0
INFO [06-20|21:50:21.544] Looking for peers peercount=0 tried=5 static=0
...
INFO [06-20|22:09:47.659] Looking for peers peercount=0 tried=5 static=0
INFO [06-20|22:09:57.660] Looking for peers peercount=1 tried=4 static=0
INFO [06-20|22:09:58.978] Imported new block headers count=0 elapsed=19.358ms number=842573 hash="0dca2c…3cd85d" age=4y5mo4w ignored=192
INFO [06-20|22:09:59.987] Imported new block headers count=0 elapsed=19.261ms number=842765 hash="291620…eb1cae" age=4y5mo4w ignored=192
INFO [06-20|22:09:59.988] Imported new block receipts count=2 elapsed=28.811ms number=842383 hash="61a8d4…e9d407" age=4y5mo4w size=1.94KiB
INFO [06-20|22:10:02.742] Imported new block headers count=0 elapsed=20.157ms number=842957 hash="11bb49…0210a3" age=4y5mo4w ignored=192
INFO [06-20|22:10:02.800] Imported new block receipts count=6 elapsed=21.766ms number=842389 hash="a46a6f…bd23c8" age=4y5mo4w size=8.64KiB
INFO [06-20|22:10:02.811] Imported new block receipts count=1 elapsed=11.208ms number=842390 hash="e121e2…e1db22" age=4y5mo4w size=581.00B
INFO [06-20|22:10:05.026] Imported new state entries count=294 elapsed="3.939µs" processed=1028555 pending=4689 retry=0 duplicate=0 unexpected=0
INFO [06-20|22:10:05.392] Imported new block receipts count=6 elapsed=21.955ms number=842396 hash="667b36…5f88f7" age=4y5mo4w size=8.92KiB
INFO [06-20|22:10:05.404] Imported new block headers count=0 elapsed=25.691ms number=843149 hash="ceddb9…7ff77d" age=4y5mo4w ignored=192
INFO [06-20|22:10:07.640] Imported new state entries count=227 elapsed="4.694µs" processed=1028782 pending=8268 retry=0 duplicate=0 unexpected=0
INFO [06-20|22:10:07.660] Looking for peers peercount=1 tried=1 static=0
INFO [06-20|22:10:08.420] Imported new block headers count=0 elapsed=18.581ms number=843341 hash="2c094c…6bfdb6" age=4y5mo4w ignored=192
INFO [06-20|22:10:09.045] Imported new block receipts count=12 elapsed=24.972ms number=842408 hash="96a474…5329c5" age=4y5mo4w size=18.00KiB
INFO [06-20|22:10:11.541] Imported new block headers count=0 elapsed=20.226ms number=843533 hash="c47e91…11fa98" age=4y5mo4w ignored=192
INFO [06-20|22:10:12.162] Imported new block receipts count=17 elapsed=37.383ms number=842425 hash="f62971…691224" age=4y5mo4w size=41.47KiB
INFO [06-20|22:10:14.483] Imported new block headers count=0 elapsed=21.276ms number=843725 hash="aa535f…842be2" age=4y5mo4w ignored=192
INFO [06-20|22:10:14.769] Imported new block receipts count=34 elapsed=33.283ms number=842459 hash="a148c4…af705d" age=4y5mo4w size=49.22KiB
...
it works well.