kinecosystem / blockchain-docker-quickstart Goto Github PK

Steps to reproduce:

1. Start the docker-quickstart until it fully syncs.

2)Reboot your machine to be sure that the docker restarts safe without any problems

3)Press the restart button(hard shutdown) and boot up again + start the docker container
localhost:11626/info status is :
"Catching up: Awaiting 1/1 prerequisites of: get-history-archive-state"
and that's where it will stuck forever

Changing state LM_BOOTING_STATE -> LM_CATCHING_UP_STATE
[History INFO] Catchup downloading history archive state at checkpoint 6733183
[Work WARNING] Scheduling retry #1/5 in 36 sec, for get-history-archive-state
Some random : Catching up: Awaiting checkpoint (ETA: 21 seconds)

and seriously many many :
Scheduling retry #2/5 in 4 sec, for get-remote-file history/00/66/bd/history-0066bd7f.json
messages.

Mostly but not always there is also a replay error:
Catching up: Retrying in 1 sec: apply-ledger-chain
[History ERROR] Replay failed: replay of [seq=6731602, hash=2d5e90] at LCL [seq=6731602, hash=6bcd84] disagreed on hash [ApplyLedgerChainWork.cpp:275]

and then it will just start again
lastClosedLedger: 6731602
toLedger: 6733326

[Ledger INFO] Lost sync, local LCL is 6731602, network closed ledger 6733327
[Ledger INFO] Close of ledger 6733327 buffered, starting catchup
[Ledger INFO] Changing state LM_BOOTING_STATE -> LM_CATCHING_UP_STATE
[History INFO] Starting catchup with configuration:

It will then catch up again and download the ledger files (29 files to be specific) followed by

[History INFO] Catchup verifying ledger chain for checkpointRange [6731602..6733437]
It verifies the checkpoints and then

GCA3Q [History INFO] Verifying catchup candidate 6733437 with LedgerManager
[History INFO] History chain [6731602,6733437] verified
[History INFO] Catchup downloading history archive state for applying buckets at checkpoint 6731647 not needed
[History INFO] Catchup downloading transactions for range [6731647..6733439]

after the download is complete it says:
[History INFO] Catchup downloading history archive state for applying buckets at checkpoint 6731647 not needed

then it tries to apply the transactions but it fails:
Catching up: applying checkpoint 1/29 (3%)
2020-02-09T00:14:43.735 GCA3Q [History ERROR] Replay failed: replay of [seq=6731602, hash=2d5e90] at LCL [seq=6731602, hash=6bcd84] disagreed on hash [ApplyLedgerChainWork.cpp:275]

And then the game repeats itself.

The only solution is to wipe out everything and start from scratch but this isn't a permanent solution for sure.

It will stay like this forever.That's the most frequent one and the second error is the known "bridge the gap" error which is solvable but still painful.

Note: I can't reproduce this problem by using Stellars current version. Googling the problem revealed that the top recommendations are:
a) wiping out the DB + the buckets
or
b) upgrading to a newer Stellar version because the newer version has 99 problems but the mentioned error above ain't one