Consensus error: Checking inherents failed: IncorrectSegmentHeadersList

Support
,
1

Issue Report

Environment

  • Operating System: Ubuntu 22.04
  • Pulsar/Advanced CLI/Docker:: Advanced CLI

Problem

The node (feb-19) started throwing the error and can’t progress past the block.

2024-03-02T06:00:48.188264Z  INFO Consensus: substrate: ⚙️  Syncing  0.0 bps, target=#450113 (9 peers), best: #439117 (0x644e…bd67), finalized #378227 (0xaf45…f891), ⬇ 4.5MiB/s ⬆ 2.6kiB/s
2024-03-02T06:00:49.549177Z  WARN Consensus: sync: 💔 Error importing block 0xd1dcd1035d8bf1f48eb156098997ee4343ef0db87944bb0ae77383975712a7ef: consensus error: Checking inherents failed: IncorrectSegmentHeadersList { expected: [V0 { segment_index: SegmentIndex(202), segment_commitment: SegmentCommitment([139, 23, 48, 0, 164, 228, 224, 122, 19, 158, 189, 39, 17, 151, 164, 78, 255, 56, 32, 36, 79, 97, 123, 133, 234, 162, 245, 26, 127, 43, 160, 165, 244, 15, 245, 111, 125, 193, 210, 106, 17, 195, 67, 242, 202, 177, 140, 32]), prev_segment_header_hash: [46, 147, 49, 72, 186, 99, 19, 190, 7, 113, 60, 137, 210, 78, 213, 116, 253, 17, 32, 44, 67, 52, 167, 93, 102, 168, 151, 179, 220, 4, 232, 73], last_archived_block: LastArchivedBlock { number: 439017, archived_progress: Partial(3048) } }], actual: [V0 { segment_index: SegmentIndex(202), segment_commitment: SegmentCommitment([145, 115, 83, 167, 121, 79, 231, 246, 147, 233, 95, 52, 60, 231, 24, 18, 78, 152, 106, 153, 9, 170, 62, 117, 121, 145, 64, 111, 104, 203, 226, 69, 26, 112, 33, 190, 40, 100, 150, 137, 115, 210, 236, 78, 36, 235, 147, 173]), prev_segment_header_hash: [46, 147, 49, 72, 186, 99, 19, 190, 7, 113, 60, 137, 210, 78, 213, 116, 253, 17, 32, 44, 67, 52, 167, 93, 102, 168, 151, 179, 220, 4, 232, 73], last_archived_block: LastArchivedBlock { number: 439017, archived_progress: Partial(3048) } }] }
2024-03-02T06:00:51.029229Z  WARN Consensus: sync: 💔 Error importing block 0xd1dcd1035d8bf1f48eb156098997ee4343ef0db87944bb0ae77383975712a7ef: consensus error: Checking inherents failed: IncorrectSegmentHeadersList { expected: [V0 { segment_index: SegmentIndex(202), segment_commitment: SegmentCommitment([139, 23, 48, 0, 164, 228, 224, 122, 19, 158, 189, 39, 17, 151, 164, 78, 255, 56, 32, 36, 79, 97, 123, 133, 234, 162, 245, 26, 127, 43, 160, 165, 244, 15, 245, 111, 125, 193, 210, 106, 17, 195, 67, 242, 202, 177, 140, 32]), prev_segment_header_hash: [46, 147, 49, 72, 186, 99, 19, 190, 7, 113, 60, 137, 210, 78, 213, 116, 253, 17, 32, 44, 67, 52, 167, 93, 102, 168, 151, 179, 220, 4, 232, 73], last_archived_block: LastArchivedBlock { number: 439017, archived_progress: Partial(3048) } }], actual: [V0 { segment_index: SegmentIndex(202), segment_commitment: SegmentCommitment([145, 115, 83, 167, 121, 79, 231, 246, 147, 233, 95, 52, 60, 231, 24, 18, 78, 152, 106, 153, 9, 170, 62, 117, 121, 145, 64, 111, 104, 203, 226, 69, 26, 112, 33, 190, 40, 100, 150, 137, 115, 210, 236, 78, 36, 235, 147, 173]), prev_segment_header_hash: [46, 147, 49, 72, 186, 99, 19, 190, 7, 113, 60, 137, 210, 78, 213, 116, 253, 17, 32, 44, 67, 52, 167, 93, 102, 168, 151, 179, 220, 4, 232, 73], last_archived_block: LastArchivedBlock { number: 439017, archived_progress: Partial(3048) } }] }
2024-03-02T06:00:52.303741Z  WARN Consensus: sync: 💔 Error importing block 0xd1dcd1035d8bf1f48eb156098997ee4343ef0db87944bb0ae77383975712a7ef: consensus error: Checking inherents failed: IncorrectSegmentHeadersList { expected: [V0 { segment_index: SegmentIndex(202), segment_commitment: SegmentCommitment([139, 23, 48, 0, 164, 228, 224, 122, 19, 158, 189, 39, 17, 151, 164, 78, 255, 56, 32, 36, 79, 97, 123, 133, 234, 162, 245, 26, 127, 43, 160, 165, 244, 15, 245, 111, 125, 193, 210, 106, 17, 195, 67, 242, 202, 177, 140, 32]), prev_segment_header_hash: [46, 147, 49, 72, 186, 99, 19, 190, 7, 113, 60, 137, 210, 78, 213, 116, 253, 17, 32, 44, 67, 52, 167, 93, 102, 168, 151, 179, 220, 4, 232, 73], last_archived_block: LastArchivedBlock { number: 439017, archived_progress: Partial(3048) } }], actual: [V0 { segment_index: SegmentIndex(202), segment_commitment: SegmentCommitment([145, 115, 83, 167, 121, 79, 231, 246, 147, 233, 95, 52, 60, 231, 24, 18, 78, 152, 106, 153, 9, 170, 62, 117, 121, 145, 64, 111, 104, 203, 226, 69, 26, 112, 33, 190, 40, 100, 150, 137, 115, 210, 236, 78, 36, 235, 147, 173]), prev_segment_header_hash: [46, 147, 49, 72, 186, 99, 19, 190, 7, 113, 60, 137, 210, 78, 213, 116, 253, 17, 32, 44, 67, 52, 167, 93, 102, 168, 151, 179, 220, 4, 232, 73], last_archived_block: LastArchivedBlock { number: 439017, archived_progress: Partial(3048) } }] }
2024-03-02T06:00:53.125720Z  INFO Consensus: subspace_service::sync_from_dsn: Received notification to sync from DSN reason=NoImportedBlocks

I tried the previous release but it didn’t solve the issue.

Probably it will be solved if I wipe the node but don’t really wanna do that and waste time on syncing. Is there a faster way to resolve the problem?

2

I don’t think you can recover from this without wiping. It has also been a while since we saw errors of this kind.
Would you be able to archive db directory of the node and upload it somewhere so I can debug how you could end up in that situation? db shouldn’t have any sensitive information. Torrent file would work, which might be the easiest.

3

Fortunately I found a snapshot from a few weeks ago and resync took only half a day.

Here is db folder archived.

4

Downloaded, thanks a lot!
I’ll check when I have time to see if this is a bug in the code or some one-off glitch.