ERROR Version: 0.1.0-3c3a48f865c

Please provide as much information as possible to help the community/ team resolve your issue quicker


Hello. I start a new node and farmer (CLI last release 13jun). At some point in time I went to the server I saw the following message in the logs of the NODE:

Version: 0.1.0-3c3a48f865c
   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/std/src/panicking.rs:588:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/core/src/panicking.rs:142:14
   6: core::result::unwrap_failed
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/core/src/result.rs:1785:5
   7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
   8: sp_io::storage::get_version_1
   9: sp_io::storage::ExtStorageGetVersion1::call
  10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: wasmtime_runtime::traphandlers::catch_traps::call_closure
  17: wasmtime_setjmp
  18: sc_executor_wasmtime::runtime::perform_call
  19: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call
  20: sc_executor_common::wasm_runtime::WasmInstance::call_export
  21: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  22: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  23: <subspace_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_execute_block_runtime_api_impl
  24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  25: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  26: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  27: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  29: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
  30: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  31: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  32: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  33: tokio::runtime::task::raw::poll
  34: std::sys_common::backtrace::__rust_begin_short_backtrace
  35: core::ops::function::FnOnce::call_once{{vtable.shim}}
  36: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/alloc/src/boxed.rs:1872:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/42e1761c704f074b7b8c7ff8a7433acfd65d2ee9/library/std/src/sys/unix/thread.rs:108:17
  37: start_thread
  38: clone
Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x31f989396f55753f9db4cc780cea3bcd98a865097cddf91a6ed61954d0dd5e52"', /home/runner/.cargo/git/checkouts/substrate-7bc20b373ca3e834/5f0aa1f/primitives/state-machine/src/ext.rs:189
This is a bug. Please report it at:
        https://discord.gg/vhKF9w3x
subspace-node.service: Main process exited, code=exited, status=1/FAILURE
subspace-node.service: Failed with result 'exit-code'.

Server: 6xIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 16 GB RAM, 400 GB SSD, Ubuntu 20.04.

The discord link in message is broken, so I report here. Please take this problem to work and let me know what I need to do about my node!

@nazar-pc Weird issue with tokio-runtime-worker on node, not sure if that Discord link is supposed to be ours or some other maintainers link, but probably worth getting fixed if its on our end.

This typically indicates a hardware error. If error repeats on every restart try to check disk’s SMART to make sure there are no errors on it.

Just had the same error thrown though the stack trace indicates on a different call. SMART is coming back clean FWIW. This is on a Windows 10 machine with 4 Intel cores and 16GiB RAM It’s running prerelease gemini-1b-2022-jul-27 in Docker Desktop and plotting 2 farms in this configuration:

      "--farm", "hdd=/var/subspace-hdd1,ssd=/var/subspace-ssd1,size=200G",
      "--farm", "hdd=/var/subspace-hdd2,ssd=/var/subspace-ssd2,size=300G",

Error:

subspace-node    | ====================
subspace-node    | 
subspace-node    | Version: 0.1.0-3c98a67882cd7454a6882f56a726bb337cce6242
subspace-node    | 
subspace-node    |    0: sp_panic_handler::set::{{closure}}
subspace-node    |    1: std::panicking::rust_panic_with_hook
subspace-node    |    2: std::panicking::begin_panic_handler::{{closure}}
subspace-node    |    3: std::sys_common::backtrace::__rust_end_short_backtrace
subspace-node    |    4: rust_begin_unwind
subspace-node    |    5: core::panicking::panic_fmt
subspace-node    |    6: core::result::unwrap_failed
subspace-node    |    7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
subspace-node    |    8: sp_io::storage::get_version_1
subspace-node    |    9: sp_io::storage::ExtStorageGetVersion1::call
subspace-node    |   10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
subspace-node    |   11: <unknown>
subspace-node    |   12: <unknown>
subspace-node    |   13: wasmtime_runtime::traphandlers::catch_traps::call_closure
subspace-node    |   14: wasmtime_setjmp
subspace-node    |   15: sc_executor_wasmtime::runtime::perform_call
subspace-node    |   16: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call
subspace-node    |   17: sc_executor_common::wasm_runtime::WasmInstance::call_export
subspace-node    |   18: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
subspace-node    |   19: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
subspace-node    |   20: sp_consensus_subspace::SubspaceApi::records_root
subspace-node    |   21: sc_consensus_subspace::SubspaceBlockImport<Block,Client,I,CAW,CIDP>::block_import_verification::{{closure}}
subspace-node    |   22: <sc_consensus_subspace::SubspaceBlockImport<Block,Client,Inner,CAW,CIDP> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
subspace-node    |   23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
subspace-node    |   24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
subspace-node    |   25: sc_consensus::import_queue::basic_queue::block_import_process::{{closure}}
subspace-node    |   26: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
subspace-node    |   27: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
subspace-node    |   28: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
subspace-node    |   29: tokio::runtime::task::raw::poll
subspace-node    |   30: std::sys_common::backtrace::__rust_begin_short_backtrace
subspace-node    |   31: core::ops::function::FnOnce::call_once{{vtable.shim}}
subspace-node    |   32: std::sys::unix::thread::Thread::new::thread_start
subspace-node    |   33: start_thread
subspace-node    |   34: clone
subspace-node    | 
subspace-node    | 
subspace-node    | Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x73129487c5f059100efea050a3d65f3b444795a7c38c8f0d7d317250f4342493"', /root/.cargo/git/checkouts/substrate-7bc20b373ca3e834/1399afd/primitives/state-machine/src/ext.rs:189
subspace-node    | 
subspace-node    | This is a bug. Please report it at:
subspace-node    | 
subspace-node    |      https://forum.subspace.network
subspace-node    | 

Appreciate this could well be a failing drive that can’t be detected yet but thought I’d document it just in case a pattern starts to emerge.

1 Like

This looks like database corruption, did your node ran out of space at any point?

Plenty of space on the drives. I’ll monitor the situation and report back if I see it again.

2 Likes