Database errors from Subspace CLI

Issue Report

Environment

  • Operating System: Linux distribution: Ubuntu 20.04.5 LTS
  • CPU Architecture: x86_64 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU cores: 30
  • RAM: 117919MB
  • Storage: 3 TB
  • Plot Size: 800 GB
  • Subspace Deployment Method: Nodes Guru automated deploy script (Subspace CLI under systemd service with --verbose)

Problem

Error messages appearing in the logs related to database issues and then failing to get header for hash.

Note the plot size of 800G.

Note that the user was following a third party guide.

Error messages encountered:

  • sc_service::client::client: Block import error: Database
  • slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash
  • sync: 💔 Error importing block 0xcc3f4910b76799b870e372f8293c05ce066eb41b001d448dca078ee1bcda1f0e: consensus error: Import failed: Database
  • `DB saving error
  • ERROR subspace_networking::behavior::provider_storage::providers: DB saving error. key=[248, 185, 206, 5, 32, 204, 156, 36, 161, 42, 60, 149, 134, 148, 89, 250, 20, 176, 49, 134, 2, 7, 172, 173, 202, 153, 191, 87, 116, 222, 11, 69, 111, 51, 182, 9, 72] err=Background(Io(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))

Steps to reproduce

  1. Start farming using Nodes Guru guide.
  2. (Possibly) start an 800G plot.

Expected result

  • Syncing completes and plotting starts.

What happens instead

  • Errors in the logs as per below.
2023-03-18T13:49:01.249829Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494075 (79 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 1.2MiB/s ⬆ 6.6kiB/s
2023-03-18T13:49:02.001219Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:02.903390Z  WARN sc_service::client::client: Block import error: Database
2023-03-18T13:49:03.001210Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:03.050278Z  WARN sync: 💔 Error importing block 0xcc3f4910b76799b870e372f8293c05ce066eb41b001d448dca078ee1bcda1f0e: consensus error: Import failed: Database
2023-03-18T13:49:04.000919Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:05.000813Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:06.001138Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:06.250781Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494075 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 4.8MiB/s ⬆ 8.6kiB/s
2023-03-18T13:49:07.000669Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:08.001360Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:09.000702Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:10.001026Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:11.000547Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:11.251426Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494075 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.9MiB/s ⬆ 7.1kiB/s
2023-03-18T13:49:12.001253Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:13.001083Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:14.000802Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:15.001274Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:16.000653Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:16.252277Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494076 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.5MiB/s ⬆ 7.8kiB/s
2023-03-18T13:49:17.001103Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:18.000639Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:19.001033Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:20.000407Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:21.001072Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:21.252763Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494077 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.9MiB/s ⬆ 7.3kiB/s
2023-03-18T13:49:22.000513Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:23.000997Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:24.000431Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:25.001249Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:26.000803Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:26.253796Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494080 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 8.0MiB/s ⬆ 7.9kiB/s
2023-03-18T13:49:27.001705Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:28.001259Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:29.000720Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:30.001104Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:31.000757Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:31.255037Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494082 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.7MiB/s ⬆ 7.8kiB/s
2023-03-18T13:49:32.001467Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:33.000865Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:34.001797Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:35.001467Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:36.000971Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:36.255748Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494083 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.6MiB/s ⬆ 7.7kiB/s
2023-03-18T13:49:37.000447Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:38.000840Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:39.001239Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:40.000899Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:41.001240Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:41.256625Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494084 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.9MiB/s ⬆ 8.0kiB/s
2023-03-18T13:49:42.000624Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:43.000989Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:44.000333Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:45.000689Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:46.001283Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:46.257318Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494084 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.8MiB/s ⬆ 7.4kiB/s
2023-03-18T13:49:47.001226Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:48.000598Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:49.001287Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:50.000813Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:51.001459Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:51.257880Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.8MiB/s ⬆ 7.8kiB/s
2023-03-18T13:49:52.000861Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:53.001707Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:54.001975Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:55.000391Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:56.000953Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:56.258606Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.8MiB/s ⬆ 7.9kiB/s
2023-03-18T13:49:57.000482Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:58.001174Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:59.000680Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:00.001310Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:01.000895Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:01.259157Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.5MiB/s ⬆ 7.9kiB/s
2023-03-18T13:50:02.001367Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:03.000677Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:04.001281Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:05.000878Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:06.001151Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:06.260559Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.6MiB/s ⬆ 7.2kiB/s
2023-03-18T13:50:07.000666Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:08.001699Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:09.001961Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:10.000705Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:11.001572Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:11.273920Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.6MiB/s ⬆ 7.7kiB/s
2023-03-18T13:50:12.001142Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:13.000529Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:14.001032Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:15.001197Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:16.000644Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:16.275278Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.5MiB/s ⬆ 7.6kiB/s
2023-03-18T13:50:17.001069Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:18.000368Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:19.001014Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:20.000775Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:21.001302Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:21.276067Z  INFO substrate: ⚙️  Syncing  0.0 bps, target=#494087 (78 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.9MiB/s ⬆ 20.6kiB/s
2023-03-18T13:50:22.000777Z  WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e

Another user of the same script got some different database errors:

Related post: Stopping synchronization and node errors.

For the “slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash” error

I found a similar error & solution from previous networks, seems to be an issue based off user permissions

1 Like

Interestingly enough they both occur at 62xxx block height… we may want to escalate this one up to the developers for further investigation

1 Like