Consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList

after update to jun-13 i have this problem:

Jun 13 23:45:37 subspace subspace-node[52477]: 2022-06-13 23:45:37 [PrimaryChain] :zzz: Idle (40 peers), best: #188014 (0x0014…1da1), finalized #0 (0x9ee8…ccf0), :arrow_down: 4.2MiB/s :arrow_up: 5.5MiB/s
Jun 13 23:45:38 subspace subspace-node[52477]: 2022-06-13 23:45:38 [PrimaryChain] :broken_heart: Error importing block 0x6621d6daee09b63e72ee83424b5d648a8c4aad88e6542ae78da72906573bc6c2: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList
Jun 13 23:45:40 subspace subspace-node[52477]: 2022-06-13 23:45:40 [PrimaryChain] :broken_heart: Error importing block 0x6621d6daee09b63e72ee83424b5d648a8c4aad88e6542ae78da72906573bc6c2: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList
Jun 13 23:45:42 subspace subspace-node[52477]: 2022-06-13 23:45:42 [PrimaryChain] :zzz: Idle (39 peers), best: #188014 (0x0014…1da1), finalized #0 (0x9ee8…ccf0), :arrow_down: 2.9MiB/s :arrow_up: 2.9MiB/s
Jun 13 23:45:43 subspace subspace-node[52477]: 2022-06-13 23:45:43 [PrimaryChain] :broken_heart: Error importing block 0x6621d6daee09b63e72ee83424b5d648a8c4aad88e6542ae78da72906573bc6c2: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList
Jun 13 23:45:45 subspace subspace-node[52477]: 2022-06-13 23:45:45 [PrimaryChain] :broken_heart: Error importing block 0x6621d6daee09b63e72ee83424b5d648a8c4aad88e6542ae78da72906573bc6c2: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList
Jun 13 23:45:47 subspace subspace-node[52477]: 2022-06-13 23:45:47 [PrimaryChain] :zzz: Idle (41 peers), best: #188014 (0x0014…1da1), finalized #0 (0x9ee8…ccf0), :arrow_down: 2.8MiB/s :arrow_up: 1.3MiB/s
Jun 13 23:45:51 subspace subspace-node[52477]: 2022-06-13 23:45:51 [PrimaryChain] :broken_heart: Error importing block 0x6621d6daee09b63e72ee83424b5d648a8c4aad88e6542ae78da72906573bc6c2: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList
Jun 13 23:45:52 subspace subspace-node[52477]: 2022-06-13 23:45:52 [PrimaryChain] :zzz: Idle (36 peers), best: #188014 (0x0014…1da1), finalized #0 (0x9ee8…ccf0), :arrow_down: 3.6MiB/s :arrow_up: 1.1MiB/s
Jun 13 23:45:52 subspace subspace-node[52477]: 2022-06-13 23:45:52 [PrimaryChain] :broken_heart: Error importing block 0x6621d6daee09b63e72ee83424b5d648a8c4aad88e6542ae78da72906573bc6c2: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList
Jun 13 23:45:57 subspace subspace-node[52477]: 2022-06-13 23:45:57 [PrimaryChain] :zzz: Idle (38 peers), best: #188014 (0x0014…1da1), finalized #0 (0x9ee8…ccf0), :arrow_down: 1.2MiB/s :arrow_up: 1.1MiB/s
Jun 13 23:46:02 subspace subspace-node[52477]: 2022-06-13 23:46:02 [PrimaryChain] :zzz: Idle (38 peers), best: #188014 (0x0014…1da1), finalized #0 (0x9ee8…ccf0), :arrow_down: 278.7kiB/s :arrow_up: 1.3MiB/s
Jun 13 23:46:04 subspace subspace-node[52477]: 2022-06-13 23:46:04 [PrimaryChain] :broken_heart: Verification failed for block 0x2de12b35e99965c8ca1f21bd4c736001675e983917272c53b5e5f265cc02d6ef received from peer: 12D3KooWKMcYhMgpK3EvToCzEEUpujtvbFddP3YfShb4xiu12oLJ, “Unexpected block author during early history of Gemini 1b, please ignore this message”

I have the same issue:

2022-06-13 22:33:11 [PrimaryChain] ?? Idle (45 peers), best: #190393 (0x7dd8??39c9), finalized #0 (0x9ee8??ccf0), ?? 8.4MiB/s ?? 999.4kiB/s    
2022-06-13 22:33:13 [PrimaryChain] ?? Error importing block 0x3f5f7458d3b4b628a40ed2490b8717fd872d1854f3935ed1f67e27a400eabec4: consensus error: Import failed: Checking inherents failed: IncorrectRootBlocksList

This message repeats continuously.

@nazar-pc mind taking a look

This is an expected error in case your node received blocks from the network that it can’t verify because they belong to alternative history of the blockchain. When blocks are incorrect node will rightfully complain and reject/ignore them.

The node repeats this message for hours on end. It’s stuck. Can I revert the chain back a couple thousand blocks? If so, how? This node was running fine on the correct chain up until earlier today so I’m surprised this would happen.

If your best block very close to one in explorer then everything works fine and you can just ignore those messages for now.

However I just checked block 188014 and it seems like your node is not on the canonical chain and would require a reset, that explains why when nodes on the network send your node some blocks, node can’t import them.

If what you say is true then this is a huge problem.

Nodes should be able to revert backwards until they find a block they can import correctly and then resume sync from there. Otherwise what you’re saying is that if a node goes off on a fork for whatever reason it’s impossible for it to re-sync back to the correct chain unless it’s completely wiped and restarted from scratch.

There is a “revert” command available on the node. Does this work?

Default instructions are not storing state for blocks deeper than 1024 from the tip of the chain you’re on. In which case it is not possible to switch to a different fork deeper than that. Also we have a process of archiving of the history of the blockchain for blocks at depth of 100 blocks, that essentially also means you can’t revert below that point.

As such we don’t really expect nodes to have deeper forks than 100 blocks, but due to series of circumstances we still have a significant number of nodes on non-canonical chain right now. Good news is that over 50% of nodes are synced on the correct chain and the number of growing, so things are improving quickly.

What should I do? Synchronization has not been restored in the last 12 hours. I updated the version while on block #188014 and I’m still on it. The farmer issued (successfully signed hash of the reward) 6 times during this time, but the reward was not credited after this update.

Are you on the correct chain? Type block 188014 in explorer and see if it shows the same block hash as you see in logs of the node.

I had the same issue. I stopped farmer and node. Then I deleted the local “chains” directory and rsynced the “chains” directly from a different node, that I knew was still working local. Then I started node and farmer, and things seems to work.

Got this error on 3 of my nodes.
They are was sync’ed and working good, but then - …

Ubuntu 20.04 LTS,
SubSpace sw version from 06 october.

I’ve already try to revert with

sudo systemctl stop subspaced && subspace-node revert 100 --chain gemini-2a --base-path ~/.local/share/subspace-node --state-pruning archive

No result. It comes again to same point and sync stuck.

Any suggestions how to fix it without new sync?

1 Like