I followed the instructions for running a binary light client with default config on t

Again, thanks for the report <a class="user-mention notranslate" data-hovercard-type="

Light Client Crash about avail-light HOT 5 CLOSED

keyneom commented on June 1, 2024

Light Client Crash

from avail-light.

Comments (5)

keyneom commented on June 1, 2024

Light client crashed again but looks to be a different error this time:

2023-12-13T23:01:31.509184Z  INFO avail_light::api::v2: Message published to clients topic=HeaderVerified published=0 failed=0
thread 'tokio-runtime-worker' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-relay-0.17.0/src/priv_client/handler.rs:387:26:
must have active request for stream
stack backtrace:
   0: rust_begin_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
   2: core::panicking::panic_display
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:168:5
   3: core::panicking::panic_str
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:152:5
   4: core::option::expect_failed
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/option.rs:1988:5
   5: <libp2p_relay::priv_client::handler::Handler as libp2p_swarm::handler::ConnectionHandler>::poll
   6: libp2p_swarm::connection::Connection<THandler>::poll
   7: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   8: tokio::runtime::task::raw::poll
   9: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  10: tokio::runtime::task::raw::poll
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aborted

Let me know if you'd rather me open a separate issue for this.

from avail-light.

keyneom commented on June 1, 2024

Haha sorry if this is spamming. I keep running it and increasing the backtrace verbosity and I keep gettting crashes with different errors:

2023-12-14T00:51:24.343830Z  INFO avail_light::network::rpc::event_loop: Sending finalized block 155309
2023-12-14T00:51:24.343906Z  INFO avail_light::api::v2: Message published to clients topic=HeaderVerified published=0 failed=0
thread 'tokio-runtime-worker' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-relay-0.17.0/src/priv_client/handler.rs:387:26:
must have active request for stream
stack backtrace:
   0:     0x5621587bfd53 - std::backtrace_rs::backtrace::libunwind::trace::h67a838aed1f4d6ec
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5621587bfd53 - std::backtrace_rs::backtrace::trace_unsynchronized::h1d1786bb1962baf8
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5621587bfd53 - std::sys_common::backtrace::_print_fmt::h5a0b1f807a002d23
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x5621587bfd53 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf84ab6ad0b91784c
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x56215824561c - core::fmt::rt::Argument::fmt::h28f463bd1fdabed5
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/fmt/rt.rs:138:9
   5:     0x56215824561c - core::fmt::write::ha37c23b175e921b3
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/fmt/mod.rs:1114:21
   6:     0x56215878d79d - std::io::Write::write_fmt::haa1b000741bcbbe1
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/io/mod.rs:1763:15
   7:     0x5621587c14be - std::sys_common::backtrace::_print::h1ff1030b04dfb157
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5621587c14be - std::sys_common::backtrace::print::hb982056c6f29541c
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5621587c1084 - std::panicking::default_hook::{{closure}}::h11f92f82c62fbd68
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:272:22
  10:     0x5621587c207c - std::panicking::default_hook::hb8810fe276772c66
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:292:9
  11:     0x5621587c207c - std::panicking::rust_panic_with_hook::hd2f0efd2fec86cb0
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:731:13
  12:     0x5621587c1b9c - std::panicking::begin_panic_handler::{{closure}}::h3651b7fc4f61d784
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:609:13
  13:     0x5621587c1af6 - std::sys_common::backtrace::__rust_end_short_backtrace::hbc468e4b98c7ae04
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:170:18
  14:     0x5621587c1ae1 - rust_begin_unwind
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
  15:     0x562157d369b4 - core::panicking::panic_fmt::h979245e2fdb2fabd
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
  16:     0x562157d36f92 - core::panicking::panic_display::h9b355c58fd35af38
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:168:5
  17:     0x562157d36f92 - core::panicking::panic_str::h187a5146d72e7d2f
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:152:5
  18:     0x562157d36f92 - core::option::expect_failed::h7cdfa49208a82a89
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/option.rs:1988:5
  19:     0x562158456706 - <libp2p_relay::priv_client::handler::Handler as libp2p_swarm::handler::ConnectionHandler>::poll::h1866f851f4cd59a7
  20:     0x562157e7f5fa - libp2p_swarm::connection::Connection<THandler>::poll::h15676eaf51cf006f
  21:     0x562157f7e2d2 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hc6a162c87583877a
  22:     0x56215845bfbf - tokio::runtime::task::raw::poll::h01d0650cbebccb15
  23:     0x5621587ff0d3 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hf863ea11c4b1a4d6
  24:     0x562158801ff0 - tokio::runtime::task::raw::poll::h6a4470c18a93bbb5
  25:     0x5621587f6a53 - std::sys_common::backtrace::__rust_begin_short_backtrace::h795a60b793af8c6f
  26:     0x5621587f6842 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hdf10a96b5265e57d
  27:     0x5621587c3925 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hfa37c25e0ad051b0
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/boxed.rs:2007:9
  28:     0x5621587c3925 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h9486bed8ab2e65ad
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/boxed.rs:2007:9
  29:     0x5621587c3925 - std::sys::unix::thread::Thread::new::thread_start::hd28b46dbf5673d17
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys/unix/thread.rs:108:17
  30:     0x7f89cea86b43 - start_thread
                               at ./nptl/pthread_create.c:442:8
  31:     0x7f89ceb18a00 - clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  32:                0x0 - <unknown>
Aborted

from avail-light.

luka-ethernal commented on June 1, 2024

Hi,

Thanks for reporting the issue. These seem to be two separate issues, both are already known to us and have to do with network connection issues. Currently, on any sort of network issue, the Light Client will just panic and crash and currently we are relying on external services to restart the client. However, this is actively being worked on on various fronts. We are improving logs, implementing retry, recovery and fallback mechanisms and expect this to improve both issues.

Which version of Light client are you running? In the latest pre-release, various things have changed and logging has been overhauled, so if you don't mind retesting, that would be of great help!

Regards,
Luka

from avail-light.

keyneom commented on June 1, 2024

I see, no worries, just didn't realize that was the expected state! Hoping to test it out soon. Feel free to close this if you'd like.

from avail-light.

sh3ll3x3c commented on June 1, 2024

Again, thanks for the report @keyneom!

We're currently review a PR that introduces proper RPC retry mechanism, so this issue shouldn't be happening from going forward. If you see anything else anything else that's suspicious, don't hesitate to contact us!

from avail-light.

Light Client Crash about avail-light HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent