Node can't catch up to 100 units, doesn't fully sync

Issue Report

Environment

Operating system: linux
2023-03-28T04:32:30.853626Z INFO sc_sysinfo: :computer: CPU architecture: x86_64
2023-03-28T04:32:30.853682Z INFO sc_sysinfo: :computer: Target environment: gnu
2023-03-28T04:32:30.853697Z INFO sc_sysinfo: :computer: CPU: DO-Regular
2023-03-28T04:32:30.853711Z INFO sc_sysinfo: :computer: CPU cores: 4
2023-03-28T04:32:30.853724Z INFO sc_sysinfo: :computer: Memory: 7948MB
2023-03-28T04:32:30.853738Z INFO sc_sysinfo: :computer: Kernel: 5.19.0-23-generic
2023-03-28T04:32:30.853751Z INFO sc_sysinfo: :computer: Linux distribution: Ubuntu 22.10
2023-03-28T04:32:30.853764Z INFO sc_sysinfo: :computer: Virtual machine: yes
2023-03-28T04:32:30.891643Z INFO sc_service::builder: :package: Highest known block at #637318

Problem

Steps to reproduce

type or paste code here

Expected result

What happens instead

2023-03-28T15:11:47.302886Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::reward_signing: Successfully signed reward hash 0xb680f20f738b8d69eceda39dcf5009840e62daff23f6df0388e1a65c2ce21234
2023-03-28T15:11:48.184855Z  INFO substrate: 💤 Idle (100 peers), best: #643840 (0x3796…b00e), finalized #643740 (0x5773…af90), ⬇ 32.9kiB/s ⬆ 1.9MiB/s
2023-03-28T15:11:50.998178Z  INFO substrate: ✨ Imported #643841 (0x070d…8b0e)
2023-03-28T15:11:51.652878Z  INFO substrate: ✨ Imported #643842 (0x4ac8…54c0)
2023-03-28T15:11:52.290634Z  INFO substrate: ✨ Imported #643841 (0x8802…b211)
2023-03-28T15:11:53.185560Z  INFO substrate: 💤 Idle (85 peers), best: #643842 (0x4ac8…54c0), finalized #643742 (0xcf81…7f25), ⬇ 46.2kiB/s ⬆ 2.5MiB/s
2023-03-28T15:11:54.036704Z  INFO substrate: ✨ Imported #643843 (0x213a…fe5e)
2023-03-28T15:11:58.186520Z  INFO substrate: 💤 Idle (100 peers), best: #643843 (0x213a…fe5e), finalized #643743 (0xd91f…fe62), ⬇ 50.3kiB/s ⬆ 2.6MiB/s
2023-03-28T15:12:03.188401Z  INFO substrate: 💤 Idle (100 peers), best: #643843 (0x213a…fe5e), finalized #643743 (0xd91f…fe62), ⬇ 155.9kiB/s ⬆ 2.9MiB/s
2023-03-28T15:12:03.541381Z  INFO substrate: ✨ Imported #643844 (0x763b…d7c0)
2023-03-28T15:12:05.583674Z  INFO sc_informant: ♻️  Reorg on #643844,0x763b…d7c0 to #643844,0xef00…f7a1, common ancestor #643843,0x213a…fe5e
2023-03-28T15:12:05.583726Z  INFO substrate: ✨ Imported #643844 (0xef00…f7a1)
2023-03-28T15:12:06.935679Z  INFO substrate: ✨ Imported #643845 (0x9881…2273)
2023-03-28T15:12:07.416518Z  INFO substrate: ✨ Imported #643846 (0x0cd8…3151)
2023-03-28T15:12:08.189159Z  INFO substrate: 💤 Idle (90 peers), best: #643846 (0x0cd8…3151), finalized #643746 (0x85ec…05fc), ⬇ 79.1kiB/s ⬆ 2.0MiB/s
2023-03-28T15:12:13.189825Z  INFO substrate: 💤 Idle (100 peers), best: #643846 (0x0cd8…3151), finalized #643746 (0x85ec…05fc), ⬇ 43.1kiB/s ⬆ 391.7kiB/s
2023-03-28T15:12:17.465064Z  INFO substrate: ✨ Imported #643847 (0x5674…f92c)
2023-03-28T15:12:18.190286Z  INFO substrate: 💤 Idle (93 peers), best: #643847 (0x5674…f92c), finalized #643747 (0x30d5…daf1), ⬇ 79.9kiB/s ⬆ 186.9kiB/s
2023-03-28T15:12:19.326271Z  INFO sc_informant: ♻️  Reorg on #643847,0x5674…f92c to #643847,0x264f…dbc5, common ancestor #643846,0x0cd8…3151
2023-03-28T15:12:19.326337Z  INFO substrate: ✨ Imported #643847 (0x264f…dbc5)
2023-03-28T15:12:23.192598Z  INFO substrate: 💤 Idle (100 peers), best: #643847 (0x264f…dbc5), finalized #643747 (0x30d5…daf1), ⬇ 59.9kiB/s ⬆ 708.4kiB/s
2023-03-28T15:12:28.193240Z  INFO substrate: 💤 Idle (100 peers), best: #643847 (0x264f…dbc5), finalized #643747 (0x30d5…daf1), ⬇ 133.6kiB/s ⬆ 1.2MiB/s
2023-03-28T15:12:33.195073Z  INFO substrate: 💤 Idle (100 peers), best: #643847 (0x264f…dbc5), finalized #643747 (0x30d5…daf1), ⬇ 120.6kiB/s ⬆ 2.6MiB/s
2023-03-28T15:12:33.339518Z  INFO subspace: 🚜 Claimed block at slot 1680016352
2023-03-28T15:12:33.340592Z  INFO sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0x264f20dbad97f4096db3366ce4478b54f2f1ccb01d68e284fce750ad3838dbc5
2023-03-28T15:12:34.110873Z  INFO substrate: ✨ Imported #643848 (0x79e0…d05e)
2023-03-28T15:12:35.151074Z  INFO sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 643848 (1804 ms) [hash: 0xd874f1901141c3213dc81bc696751ba364ac98fbb1999b140c3f2f025c0048b6; parent_hash: 0x264f…dbc5; extrinsics (20): [0xf0c3…acb0, 0x25b6…8cd3, 0x5fde…cbbc, 0x52af…a1e3, 0xa7eb…8476, 0xf389…97de, 0xabb5…b65f, 0xc007…2d97, 0x253b…6c9e, 0x8074…3277, 0x668d…b966, 0xf6cc…3f4f, 0x219a…5352, 0x7f2a…6119, 0x484e…348d, 0x47e3…661b, 0x7725…4652, 0x1665…db01, 0x865b…e32c, 0x2bec…d9d4]]
2023-03-28T15:12:35.151677Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::reward_signing: Successfully signed reward hash 0xd874f1901141c3213dc81bc696751ba364ac98fbb1999b140c3f2f025c0048b6
2023-03-28T15:12:35.152027Z  INFO subspace: 🔖 Pre-sealed block for proposal at 643848. Hash now 0x1657d13ae8ecdbc7af4e09229904c1912f143ced1b61320891738164e10adc04, previously 0xd874f1901141c3213dc81bc696751ba364ac98fbb1999b140c3f2f025c0048b6.
2023-03-28T15:12:35.233243Z  INFO substrate: ✨ Imported #643848 (0x1657…dc04)
2023-03-28T15:12:37.294692Z  INFO substrate: ✨ Imported #643849 (0xee04…ede5)
2023-03-28T15:12:38.196906Z  INFO substrate: 💤 Idle (90 peers), best: #643849 (0xee04…ede5), finalized #643749 (0xe8db…0984), ⬇ 60.6kiB/s ⬆ 2.0MiB/s
2023-03-28T15:12:43.199022Z  INFO substrate: 💤 Idle (100 peers), best: #643849 (0xee04…ede5), finalized #643749 (0xe8db…0984), ⬇ 52.5kiB/s ⬆ 1.9MiB/s
2023-03-28T15:12:43.766698Z  INFO substrate: ✨ Imported #643850 (0xac77…f7b4)
2023-03-28T15:12:45.501073Z  INFO sc_informant: ♻️  Reorg on #643850,0xac77…f7b4 to #643850,0xb1f5…256d, common ancestor #643849,0xee04…ede5
2023-03-28T15:12:45.507957Z  INFO substrate: ✨ Imported #643850 (0xb1f5…256d)
2023-03-28T15:12:46.962647Z  INFO substrate: ✨ Imported #643851 (0xfddb…ad80)
2023-03-28T15:12:48.199657Z  INFO substrate: 💤 Idle (96 peers), best: #643851 (0xfddb…ad80), finalized #643751 (0x5f48…0a5c), ⬇ 35.1kiB/s ⬆ 2.4MiB/s

Hello Insperative

We just published a new release covering this issue:

Ty, but can you tell me how to clear the cache and is it possible to do this so that the node starts faster?

I updated, launched and also does not catch up with 100

Farmer started successfully!
2023-03-28T17:37:55.890194Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::single_disk_plot: Subscribing to slot info notifications
2023-03-28T17:37:55.893098Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-03-28T17:37:56.487500Z  INFO substrate: ✨ Imported #645344 (0x2903…c356)
2023-03-28T17:37:58.679560Z  INFO substrate: ✨ Imported #645344 (0x0dea…6d66)
2023-03-28T17:37:59.514831Z  INFO substrate: 💤 Idle (43 peers), best: #645344 (0x2903…c356), finalized #645244 (0x195f…c625), ⬇ 23.3kiB/s ⬆ 512.1kiB/s
2023-03-28T17:38:02.196905Z  INFO substrate: ✨ Imported #645345 (0x354f…12bf)
2023-03-28T17:38:04.186979Z  INFO sc_informant: ♻️  Reorg on #645345,0x354f…12bf to #645345,0xf909…16f5, common ancestor #645344,0x2903…c356
2023-03-28T17:38:04.187039Z  INFO substrate: ✨ Imported #645345 (0xf909…16f5)
2023-03-28T17:38:04.515315Z  INFO substrate: 💤 Idle (28 peers), best: #645345 (0xf909…16f5), finalized #645245 (0x7ee3…01fc), ⬇ 25.7kiB/s ⬆ 709.0kiB/s
2023-03-28T17:38:05.429565Z  INFO sc_informant: ♻️  Reorg on #645345,0xf909…16f5 to #645346,0x2d5f…f954, common ancestor #645344,0x2903…c356
2023-03-28T17:38:05.429614Z  INFO substrate: ✨ Imported #645346 (0x2d5f…f954)
2023-03-28T17:38:07.548643Z  INFO substrate: ✨ Imported #645345 (0x3f54…3124)
2023-03-28T17:38:09.203331Z  INFO substrate: ✨ Imported #645346 (0xbd17…8a37)
2023-03-28T17:38:09.517318Z  INFO substrate: 💤 Idle (58 peers), best: #645346 (0x2d5f…f954), finalized #645246 (0x727b…f470), ⬇ 65.2kiB/s ⬆ 665.1kiB/s
2023-03-28T17:38:14.517875Z  INFO substrate: 💤 Idle (100 peers), best: #645346 (0x2d5f…f954), finalized #645246 (0x727b…f470), ⬇ 32.6kiB/s ⬆ 1.1MiB/s
2023-03-28T17:38:14.968856Z  INFO substrate: ✨ Imported #645347 (0x02e8…634c)
2023-03-28T17:38:16.939173Z  INFO sc_informant: ♻️  Reorg on #645347,0x02e8…634c to #645347,0x696b…cedf, common ancestor #645346,0x2d5f…f954
2023-03-28T17:38:16.939219Z  INFO substrate: ✨ Imported #645347 (0x696b…cedf)
2023-03-28T17:38:18.896521Z  INFO substrate: ✨ Imported #645347 (0xd64c…c1a0)
2023-03-28T17:38:19.518261Z  INFO substrate: 💤 Idle (69 peers), best: #645347 (0x696b…cedf), finalized #645247 (0xd064…d5c1), ⬇ 59.8kiB/s ⬆ 1.1MiB/s
2023-03-28T17:38:21.804486Z  INFO substrate: ✨ Imported #645348 (0x7304…8ab9)
2023-03-28T17:38:23.765609Z  INFO substrate: ✨ Imported #645348 (0x7c87…26cb)
2023-03-28T17:38:24.525427Z  INFO substrate: 💤 Idle (57 peers), best: #645348 (0x7304…8ab9), finalized #645248 (0x1f7d…b518), ⬇ 48.1kiB/s ⬆ 766.4kiB/s
2023-03-28T17:38:26.265612Z  INFO substrate: ✨ Imported #645349 (0x378e…299a)
2023-03-28T17:38:26.879510Z  INFO substrate: ✨ Imported #645350 (0x935d…6c37)
2023-03-28T17:38:27.282679Z  INFO substrate: ✨ Imported #645351 (0x36a8…2e0e)
2023-03-28T17:38:29.526208Z  INFO substrate: 💤 Idle (47 peers), best: #645351 (0x36a8…2e0e), finalized #645251 (0x0404…bd41), ⬇ 30.8kiB/s ⬆ 692.0kiB/s
2023-03-28T17:38:32.430497Z  INFO substrate: ✨ Imported #645352 (0xe8ea…1123)
2023-03-28T17:38:32.878117Z  INFO substrate: ✨ Imported #645352 (0xb8f5…d7b0)
2023-03-28T17:38:34.528047Z  INFO substrate: 💤 Idle (33 peers), best: #645352 (0xe8ea…1123), finalized #645252 (0x5daf…ca68), ⬇ 28.6kiB/s ⬆ 1.0MiB/s
2023-03-28T17:38:37.623071Z  INFO substrate: ✨ Imported #645353 (0x0c36…e269)
2023-03-28T17:38:39.529076Z  INFO substrate: 💤 Idle (63 peers), best: #645353 (0x0c36…e269), finalized #645253 (0x4330…207b), ⬇ 28.8kiB/s ⬆ 1.6MiB/s
2023-03-28T17:38:44.529457Z  INFO substrate: 💤 Idle (100 peers), best: #645353 (0x0c36…e269), finalized #645253 (0x4330…207b), ⬇ 42.9kiB/s ⬆ 1.9MiB/s
2023-03-28T17:38:45.000201Z  INFO substrate: ✨ Imported #645354 (0x3969…ca5d)
2023-03-28T17:38:49.529883Z  INFO substrate: 💤 Idle (100 peers), best: #645354 (0x3969…ca5d), finalized #645254 (0x5c76…e80a), ⬇ 83.9kiB/s ⬆ 1.9MiB/s
2023-03-28T17:38:53.677917Z  INFO substrate: ✨ Imported #645355 (0x1895…2b0c)
2023-03-28T17:38:54.544252Z  INFO substrate: 💤 Idle (51 peers), best: #645355 (0x1895…2b0c), finalized #645255 (0x343d…33cf), ⬇ 182.1kiB/s ⬆ 1.9MiB/s
2023-03-28T17:38:57.114316Z  INFO substrate: ✨ Imported #645356 (0xb081…1f11)
2023-03-28T17:38:59.545772Z  INFO substrate: 💤 Idle (98 peers), best: #645356 (0xb081…1f11), finalized #645256 (0x7524…50fc), ⬇ 28.3kiB/s ⬆ 1.8MiB/s
2023-03-28T17:38:59.792111Z  INFO substrate: ✨ Imported #645356 (0xdb22…5939)
2023-03-28T17:39:01.385704Z  INFO substrate: ✨ Imported #645357 (0x5f8b…e631)
2023-03-28T17:39:01.930147Z  INFO substrate: ✨ Imported #645358 (0x7c28…5a1e)
2023-03-28T17:39:02.499287Z  INFO substrate: ✨ Imported #645358 (0xe380…9447)
2023-03-28T17:39:04.546520Z  INFO substrate: 💤 Idle (39 peers), best: #645358 (0x7c28…5a1e), finalized #645258 (0x24d4…cf54), ⬇ 22.9kiB/s ⬆ 3.1MiB/s
2023-03-28T17:39:09.547020Z  INFO substrate: 💤 Idle (94 peers), best: #645358 (0x7c28…5a1e), finalized #645258 (0x24d4…cf54), ⬇ 29.3kiB/s ⬆ 3.5MiB/s
2023-03-28T17:39:10.333200Z  INFO substrate: ✨ Imported #645359 (0x53fc…aa5f)
2023-03-28T17:39:11.214722Z  INFO substrate: ✨ Imported #645360 (0xef12…a817)

Hello Insperative. Thank for your feedback!
Will escalate to devs. Just to make sure, you running only one instance, correct?

Do you have one node per wallet? If I understand correctly, then yes. But I don’t see my node in telemetry, either I somehow entered the name wrong at the beginning, or something is wrong. How can I view or change the name of a node?

I found how to see the name - the name of the “Insperation”

1 Like

There will always be a 100 block difference between best and finalized as that is how many blocks are required for finality. This is perfectly normal and your node is fine.

1 Like

…And it appears you on the latest block on telemetry :slight_smile:

1 Like

Yes, in telemetry it shows the last block, but in the logs it seems to be behind. But since it’s normal, then for now there’s nothing to worry about, thanks

3-03-29T05:42:20.954292Z  INFO substrate: 💤 Idle (100 peers), best: #653019 (0xeecd…b211), finalized #652919 (0xb0a4…fa49), ⬇ 94.7kiB/s ⬆ 8.7MiB/s
2023-03-29T05:42:21.739266Z  INFO sc_informant: ♻️  Reorg on #653019,0xeecd…b211 to #653020,0xac0e…770b, common ancestor #653018,0x55fe…4333
2023-03-29T05:42:21.739319Z  INFO substrate: ✨ Imported #653020 (0xac0e…770b)
2023-03-29T05:42:22.703783Z  INFO substrate: ✨ Imported #653021 (0xc9f2…c1aa)
2023-03-29T05:42:25.955099Z  INFO substrate: 💤 Idle (100 peers), best: #653021 (0xc9f2…c1aa), finalized #652921 (0x9478…135f), ⬇ 44.2kiB/s ⬆ 3.9MiB/s
2023-03-29T05:42:30.955599Z  INFO substrate: 💤 Idle (100 peers), best: #653021 (0xc9f2…c1aa), finalized #652921 (0x9478…135f), ⬇ 105.4kiB/s ⬆ 1.6MiB/s
2023-03-29T05:42:32.908971Z  INFO substrate: ✨ Imported #653022 (0xf8c8…46b6)
2023-03-29T05:42:34.244993Z  INFO substrate: ✨ Imported #653022 (0x9da9…bba9)
2023-03-29T05:42:35.956028Z  INFO substrate: 💤 Idle (100 peers), best: #653022 (0xf8c8…46b6), finalized #652922 (0x3dcc…0184), ⬇ 99.3kiB/s ⬆ 14.8MiB/s
2023-03-29T05:42:40.956580Z  INFO substrate: 💤 Idle (100 peers), best: #653022 (0xf8c8…46b6), finalized #652922 (0x3dcc…0184), ⬇ 85.6kiB/s ⬆ 7.1MiB/s
2023-03-29T05:42:42.425516Z  INFO substrate: ✨ Imported #653023 (0x429e…edb6)
2023-03-29T05:42:44.025858Z  INFO sc_informant: ♻️  Reorg on #653023,0x429e…edb6 to #653023,0xdef2…3d88, common ancestor #653022,0xf8c8…46b6
2023-03-29T05:42:44.032238Z  INFO substrate: ✨ Imported #653023 (0xdef2…3d88)
2023-03-29T05:42:45.634651Z  INFO substrate: ✨ Imported #653024 (0x199d…53c4)
2023-03-29T05:42:45.957450Z  INFO substrate: 💤 Idle (94 peers), best: #653024 (0x199d…53c4), finalized #652924 (0x0374…6809), ⬇ 82.1kiB/s ⬆ 1.4MiB/s
2023-03-29T05:42:47.585500Z  INFO subspace: 🗳️ Claimed vote at slot 1680068566
2023-03-29T05:42:47.586003Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::reward_signing: Successfully signed reward hash 0xcbed700564b8078fb3b6f98d6d4b8b3eedbe3cc2a352ae85bc362b408a62c7f3
2023-03-29T05:42:47.833966Z  INFO sc_informant: ♻️  Reorg on #653024,0x199d…53c4 to #653024,0xfc99…0c33, common ancestor #653023,0xdef2…3d88
2023-03-29T05:42:47.834019Z  INFO substrate: ✨ Imported #653024 (0xfc99…0c33)
2023-03-29T05:42:47.933131Z  INFO sc_informant: ♻️  Reorg on #653024,0xfc99…0c33 to #653025,0x56ae…439a, common ancestor #653023,0xdef2…3d88
2023-03-29T05:42:47.937200Z  INFO substrate: ✨ Imported #653025 (0x56ae…439a)
2023-03-29T05:42:50.110403Z  INFO substrate: ✨ Imported #653026 (0xbd93…12a6)
2023-03-29T05:42:50.959266Z  INFO substrate: 💤 Idle (93 peers), best: #653026 (0xbd93…12a6), finalized #652926 (0x2b46…74d0), ⬇ 35.6kiB/s ⬆ 1.3MiB/s
2023-03-29T05:42:55.959782Z  INFO substrate: 💤 Idle (100 peers), best: #653026 (0xbd93…12a6), finalized #652926 (0x2b46…74d0), ⬇ 51.5kiB/s ⬆ 1.8MiB/s
2023-03-29T05:43:00.960148Z  INFO substrate: 💤 Idle (100 peers), best: #653026 (0xbd93…12a6), finalized #652926 (0x2b46…74d0), ⬇ 135.0kiB/s ⬆ 2.9MiB/s
2023-03-29T05:43:05.960603Z  INFO substrate: 💤 Idle (100 peers), best: #653026 (0xbd93…12a6), finalized #652926 (0x2b46…74d0), ⬇ 168.1kiB/s ⬆ 1.8MiB/s
2023-03-29T05:43:07.399599Z  INFO substrate: ✨ Imported #653027 (0xcecd…6d82)
2023-03-29T05:43:09.404709Z  INFO sc_informant: ♻️  Reorg on #653027,0xcecd…6d82 to #653027,0x02b0…af1c, common ancestor #653026,0xbd93…12a6
2023-03-29T05:43:09.404772Z  INFO substrate: ✨ Imported #653027 (0x02b0…af1c)
2023-03-29T05:43:10.963752Z  INFO substrate: 💤 Idle (100 peers), best: #653027 (0x02b0…af1c), finalized #652927 (0xc947…ae95), ⬇ 119.0kiB/s ⬆ 1.7MiB/s
2023-03-29T05:43:14.576736Z  INFO substrate: ✨ Imported #653028 (0x1557…5182)
2023-03-29T05:43:15.965348Z  INFO substrate: 💤 Idle (100 peers), best: #653028 (0x1557…5182), finalized #652928 (0x23b8…8959), ⬇ 142.9kiB/s ⬆ 1.6MiB/s
2023-03-29T05:43:18.409894Z  INFO substrate: ✨ Imported #653029 (0x3619…bb2d)

But if in a couple of days not a single token is farmed, is it worth sounding the alarm again and looking for the reason?

You should be earning tokens. Where are you checking your balance?

https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Feu-0.gemini-2a.subspace.network%2Fws#/accounts

my wallet: st8ojZJMwBqLpyQTswGc7WqmHuhXF1XiqbJPP9mJeZDHr8xzs

I think I figured it out and you can see the accrual of tokens

2 Likes