gemini-3g-2023-dec-15 - Database error: Too many sibling blocks

Support
, ,
1

Issue Report

Environment

  • Operating System: Ubuntu 20.04.6 LTS
  • Pulsar/Advanced CLI/Docker: Advanced CLI

Problem

2023-12-16T23:05:28.654291Z [Domain] [apply_extrinsic] after: 0x3f518c7888065c00705dae030d89f28c1a85915f5c4e7231fb43930fbd901d47    
2023-12-16T23:05:28.655243Z [Domain] [apply_extrinsic] after: 0x0cb67abd6b49fb03d468ae40a96c8df2ce42c7c37e09f88f11d8399c15b20d0c    
2023-12-16T23:05:28.656174Z [Domain] [apply_extrinsic] after: 0xe366c8f7cf63711a9d564ec0b86fd88b1628dc51569d0aaf3191814865033348    
2023-12-16T23:05:30.250819Z [Consensus] 💤 Idle (40 peers), best: #684379 (0xbf99…1c96), finalized #638037 (0x0226…6210), ⬇ 163.2kiB/s ⬆ 1.4MiB/s    
2023-12-16T23:05:34.155899Z [Consensus] ✨ Imported #684380 (0x5571…d069)    
2023-12-16T23:05:35.251087Z [Consensus] 💤 Idle (40 peers), best: #684380 (0x5571…d069), finalized #638037 (0x0226…6210), ⬇ 62.3kiB/s ⬆ 147.5kiB/s    
2023-12-16T23:05:35.259549Z [Consensus] ✨ Imported #684380 (0x73bc…11fa)    
2023-12-16T23:05:36.342982Z [Domain] Block import error: State Database error: Too many sibling blocks at #378217 inserted    
2023-12-16T23:05:36.344827Z [Domain] 💤 Idle (31 peers), best: #378433 (0x4874…8a17), finalized #0 (0xf886…aeb8), ⬇ 0.9kiB/s ⬆ 0.8kiB/s    
2023-12-16T23:05:36.344228Z [Domain] Failed to process consensus block error=Consensus(ClientImport("State Database error: Too many sibling blocks at #378217 inserted"))
2023-12-16T23:05:36.345669Z [Domain] Essential task `domain-operator-worker` failed. Shutting down service.    
2023-12-16T23:05:36.356158Z [Domain] Domain starter exited with an error Other("Essential task failed.")    
2023-12-16T23:05:36.356175Z [Domain] Essential task `domain` failed. Shutting down service.    
Error: SubstrateService(Other("Essential task failed."))
3 Likes
2

Additional information: other operators have also seen this issue on the dec-15 build. Restarting seems to fix it.