Giter VIP home page Giter VIP logo

Comments (25)

2019jack avatar 2019jack commented on June 25, 2024 1

With global state root not found? It should help avoid issues like that, yes; @2019jack was your node restarted in the meantime, after having used the snapshot?

Yes, when my node uses a snapshot, I restart it, and the node is still stuck.

from snarkos.

joske avatar joske commented on June 25, 2024

Hi @2019jack, can you try with this fix? #2903

I'm not 100% sure it's the same issue, but it fixes a sync issue we were seeing.

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Hi @2019jack, can you try with this fix? #2903

I'm not 100% sure it's the same issue, but it fixes a sync issue we were seeing.

Hey joske,Thank you very much for your response. I tried your method, modified /snarkOS/node/sync/src/block_sync.rs, and recompiled and ran it, but it seems that there is still a block stuck problem.
20231211-160948

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Missing solution

I think the main problem is how to deal with Syncing certificate after Missing solution in snarkos node bft module, maybe I have something wrong, please point it out

from snarkos.

joske avatar joske commented on June 25, 2024

Can you run with --verbosity 5 and provide the logs?

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Can you run with --verbosity 5 and provide the logs?

no problem, doing now

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Can you run with --verbosity 5 and provide the logs?

aleo_last_20_min.log
Hey @joske , here log

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Can you run with --verbosity 5 and provide the logs?

no problem, doing now

https://github.com/AleoHQ/snarkOS/pull/2903/files#diff-c2e724c7020cd2ef76ea842b503a4cb85f626f19d85d60e9c685fee70e3725ddR579

log display here: Dec 11 16:34:33 10-7-167-1 snarkos[688557]: 2023-12-11T08:34:33.936137Z TRACE snarkos_node_sync::block_sync: Removing all block requests to peer 185.119.116.244:5000

from snarkos.

joske avatar joske commented on June 25, 2024

Thx, will look into it!

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Thx, will look into it!

Thanks

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Can you run with --verbosity 5 and provide the logs?

aleo_last_20_min.log Hey @joske , here log

Hey @joske , until it gets stuck at block height 894240

from snarkos.

joske avatar joske commented on June 25, 2024

Yes, I'm looking at it, not clear to me (yet) why the block 894241 is rejected.

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Yes, I'm looking at it, not clear to me (yet) why the block 894241 is rejected.

Thank you very much for your time. If you need any help, feel free to let me know.

from snarkos.

joske avatar joske commented on June 25, 2024

When you reported the issue, the node got stuck with a similar error on a different block, so it was able to get past that in a next attempt. If you retry, does it get past 894241?

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

When you reported the issue, the node got stuck with a similar error on a different block, so it was able to get past that in a next attempt. If you retry, does it get past 894241?

When my node block was stuck at 882720, I used the snapshot height at 894008. When the node reached height 894240, it got stuck again and could not proceed.

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

When you reported the issue, the node got stuck with a similar error on a different block, so it was able to get past that in a next attempt. If you retry, does it get past 894241?

Hey @joske ,from my observation, trying next time won't fix the issue, the node will stay stuck at block height 894240

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Update

Hey @joske , I have used snapshot again, snapshot block height 909742, when the node proceeds to block height 912600, again it keeps stuck at block height 912600
aleo_last_10_min.log

Intercepted some very important logs. Please see the complete logs above.

Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.213157Z TRACE snarkos_node_sync::block_sync: Updating is_block_synced: greatest_peer_height = 912698, canon_height = 912600
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.213804Z TRACE snarkos_node_sync::block_sync: Prepared 50 block requests
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.213861Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912601' to '15.235.53.79:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.215820Z DEBUG snarkos_node_bft::primary: Skipping batch proposal (node is syncing)
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.225138Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912602' to '35.230.64.227:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.236485Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912603' to '167.235.163.202:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.252049Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'PrimaryPing' from '185.119.116.244:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.254002Z TRACE snarkos_node_bft::primary: Skipping a primary ping from '185.119.116.244:5000' (node is syncing)
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.254150Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912604' to '35.230.64.227:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.265429Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912605' to '204.93.241.110:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.276753Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912606' to '51.158.55.189:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.288053Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912607' to '34.134.137.195:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.299238Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912608' to '173.208.52.137:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.310407Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912609' to '15.235.53.79:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.321597Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912610' to '34.134.137.195:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.332781Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912611' to '146.59.118.20:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.343713Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912612' to '212.126.35.133:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.358580Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'PrimaryPing' from '135.181.228.130:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.360451Z TRACE snarkos_node_bft::primary: Skipping a primary ping from '135.181.228.130:5000' (node is syncing)
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.360612Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912613' to '44.224.120.105:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.371998Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912614' to '146.59.118.20:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.373341Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'WorkerPing' from '51.158.55.189:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.383239Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912615' to '160.202.128.199:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.394537Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912616' to '173.208.52.137:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.405732Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912617' to '34.134.137.195:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.417013Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912618' to '173.208.52.137:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.428252Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912619' to '35.230.64.227:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.439475Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912620' to '15.188.23.173:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.446480Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'PrimaryPing' from '162.55.4.41:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.448361Z TRACE snarkos_node_bft::primary: Skipping a primary ping from '162.55.4.41:5000' (node is syncing)
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.451566Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912621' to '15.235.53.79:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.462863Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912622' to '34.134.137.195:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.474192Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912623' to '15.188.23.173:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.485551Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912624' to '34.138.224.141:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.493155Z TRACE snarkos_node_router::inbound: Received 'UnconfirmedTransaction' from '146.59.118.20:4133'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.497451Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912625' to '212.126.35.133:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.508723Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912626' to '160.202.128.199:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.509643Z TRACE snarkos_node_consensus: Received unconfirmed transaction 'at1m55ue09c0x2xd..' in the queue
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.509691Z TRACE snarkos_node_consensus: Adding unconfirmed transaction 'at1m55ue09c0x2xd..' to the memory pool...
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.509741Z TRACE snarkos_node_bft::primary: Primary - Received an unconfirmed transaction 'at1m55ue09c0x2xd..'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.520367Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912627' to '212.126.35.133:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.531629Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912628' to '167.235.163.202:5000'
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.542450Z  WARN snarkos_node_consensus: Failed to add unconfirmed transaction 'at1m55ue09c0x2xd..' to the memory pool - Invalid unconfirmed transaction 'at1m55ue09c0x2xd..': Fee verification failed: global state root not found
Dec 12 11:41:28 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:28.543512Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'BlockRequest 912629' to '34.138.224.141:5000'

What I think is how to deal with the Missing solution problem

Dec 12 11:41:30 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:30.784275Z  WARN snarkos_node_consensus: Failed to add unconfirmed transaction 'at19zsjgkxjgs76k..' to the memory pool - Invalid unconfirmed transaction 'at19zsjgkxjgs76k..': Fee verification failed: global state root not found
Dec 12 11:41:31 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:31.178510Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1gyec8fq903g8njtjqrvu0x9tvy8s8wefckh3y3tvlyhnmh267a5wgtwuum8ze3dqh5pxsjc2qdlsz2c7umu in block 912601
Dec 12 11:41:31 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:31.178571Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5448241441979125..' for round 1869928 with 5 transmissions
Dec 12 11:41:31 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:31.191225Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5448241441979125..' from block 912601 - Failed to provide a transmission for round 1869928 (gc = 1869878)
Dec 12 11:41:31 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:31.191735Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1gyec8fq903g8njtjqrvu0x9tvy8s8wefckh3y3tvlyhnmh267a5wgtwuum8ze3dqh5pxsjc2qdlsz2c7umu in block 912601
Dec 12 11:41:31 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:31.191764Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7913086030938995..' for round 1869928 with 3 transmissions
Dec 12 11:41:31 10-7-167-1 snarkos[1519530]: 2023-12-12T03:41:31.204435Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7913086030938995..' from block 912601 - Failed to provide a transmission for round 1869928 (gc = 1869878)

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Hey @howardwu @raychu86, take a look? thanks

from snarkos.

joske avatar joske commented on June 25, 2024

Hi @2019jack we're trying to get this fixed ASAP.

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Hi @2019jack we're trying to get this fixed ASAP.

thank you very much for your hard work

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Hi @2019jack we're trying to get this fixed ASAP.

Hey, have fixed? I can trying it

from snarkos.

joske avatar joske commented on June 25, 2024

Not yet, sorry. @ljedrz will your PR help here?

from snarkos.

2019jack avatar 2019jack commented on June 25, 2024

Not yet, sorry. @ljedrz will your PR help here?

ok, thanks

from snarkos.

ljedrz avatar ljedrz commented on June 25, 2024

@ljedrz will your PR help here?

With global state root not found? It should help avoid issues like that, yes; @2019jack was your node restarted in the meantime, after having used the snapshot?

from snarkos.

ljedrz avatar ljedrz commented on June 25, 2024

Closing the issue as stale; there is nothing special about that block height, so it must have been storage corruption, which is tracked in another issue (or fixed altogether if it's related to CDN sync).

from snarkos.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.