Node sync stuck

Support Incentivized Testnet
, ,
1 
2023-09-07 17:56:13 [Consensus] ⚙️  Syncing  0.0 bps, target=#320152 (91 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 71.8kiB/s ⬆ 2.6kiB/s
2023-09-07 17:56:18 [Consensus] ⚙️  Syncing  0.0 bps, target=#320152 (90 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 100.2kiB/s ⬆ 1.4kiB/s
2023-09-07 17:56:23 [Consensus] ⚙️  Syncing  0.0 bps, target=#320153 (94 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 120.1kiB/s ⬆ 2.7kiB/s
2023-09-07 17:56:28 [Consensus] ⚙️  Syncing  0.0 bps, target=#320154 (96 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 102.3kiB/s ⬆ 1.3kiB/s
2023-09-07 17:56:33 [Consensus] ⚙️  Syncing  0.0 bps, target=#320154 (101 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 77.1kiB/s ⬆ 9.6kiB/s
2023-09-07 17:56:38 [Consensus] ⚙️  Syncing  0.0 bps, target=#320155 (100 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 142.3kiB/s ⬆ 13.5kiB/s
2023-09-07 17:56:43 [Consensus] ⚙️  Syncing  0.0 bps, target=#320156 (110 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 109.2kiB/s ⬆ 14.6kiB/s
2023-09-07 17:56:48 [Consensus] ⚙️  Syncing  0.0 bps, target=#320156 (111 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 94.7kiB/s ⬆ 10.4kiB/s
2023-09-07 17:56:53 [Consensus] ⚙️  Syncing  0.0 bps, target=#320158 (119 peers), best: #320054 (0x69ec…6ce5), finalized #246951 (0x0248…042c), ⬇ 104.7kiB/s ⬆ 3.8kiB/s
2

Relavent conversation with Pascal from Discord here: Discord

Adding some more context to this issue as I’ve observed it on multiple farmers. What seems to be occurring is that users are getting an error such as

2023-09-02T12:17:34.049744Z  INFO subspace_networking::node_runner: Public address status changed. old=Unknown new=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]")
2023-09-02T12:19:04.131726Z  INFO subspace_networking::node_runner: Public address status changed. old=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]") new=Private
2023-09-02T12:22:04.049771Z  INFO subspace_networking::node_runner: Public address status changed. old=Private new=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]")
2023-09-02T12:29:34.052060Z  INFO subspace_networking::node_runner: Public address status changed. old=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]") new=Private
2023-09-02T12:31:04.068498Z  INFO subspace_networking::node_runner: Public address status changed. old=Private new=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]")
2023-09-02T12:32:34.062546Z  INFO subspace_networking::node_runner: Public address status changed. old=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]") new=Private
2023-09-02T12:34:04.054590Z  INFO subspace_networking::node_runner: Public address status changed. old=Private new=Public("/ip4/[REDACTED]/tcp/30533/p2p/[REDACTED]")

This seems to be potentially related a few different issues

  1. Lack of sync entirely - Some nodes aren’t syncing at all as seen by Pascal
  2. Extremely Slow sync, were only a few peers are connecting but little progress is made
  3. I personally started getting this error aswell and shortly after started facing “Block too far in future” errors which are still persistent.
{"v":0,"name":"pulsar","msg":"Public address status changed.","level":30,"hostname":"LAPTOP-5ULCJ5GR","pid":9212,"time":"2023-09-07T16:01:17.9232668Z","target":"subspace_networking::node_runner","line":1172,"file":"C:\\Users\\Administrator\\.cargo\\git\\checkouts\\subspace-5c1447fb849a5554\\117e99d\\crates\\subspace-networking\\src\\node_runner.rs","old":"Private","new":"Public(\"/ip4/71.209.182.119/tcp/30433/p2p/12D3KooWAGfGjAAJzUjUTuJUu8DvXjNxDA8UwcQzcvrAgXs2j3Ve\")"}
{"v":0,"name":"pulsar","msg":"✨ Imported #\u001b[1;37m323887\u001b[0m (0x01ed…c4cc)","level":30,"hostname":"LAPTOP-5ULCJ5GR","pid":9212,"time":"2023-09-07T16:01:18.5940509Z","target":"log","line":null,"file":null,"log.file":"C:\\Users\\Administrator\\.cargo\\git\\checkouts\\substrate-7bc20b373ca3e834\\55c157c\\client\\informant\\src\\lib.rs","log.line":146,"log.module_path":"sc_informant","log.target":"substrate"}
{"v":0,"name":"pulsar","msg":"✨ Imported #\u001b[1;37m323888\u001b[0m (0x83ca…2f2e)","level":30,"hostname":"LAPTOP-5ULCJ5GR","pid":9212,"time":"2023-09-07T16:01:21.2393725Z","target":"log","line":null,"file":null,"log.file":"C:\\Users\\Administrator\\.cargo\\git\\checkouts\\substrate-7bc20b373ca3e834\\55c157c\\client\\informant\\src\\lib.rs","log.module_path":"sc_informant","log.line":146,"log.target":"substrate"}
{"v":0,"name":"pulsar","msg":"💤 \u001b[1;37mIdle\u001b[0m (\u001b[1;37m17\u001b[0m peers), best: #\u001b[1;37m323888\u001b[0m (0x83ca…2f2e), finalized #\u001b[1;37m246951\u001b[0m (0x0248…042c), \u001b[32m⬇ 5.2kiB/s\u001b[0m \u001b[31m⬆ 5.1kiB/s\u001b[0m","level":30,"hostname":"LAPTOP-5ULCJ5GR","pid":9212,"time":"2023-09-07T16:01:22.2727294Z","target":"log","line":null,"file":null,"log.target":"substrate","log.module_path":"sc_informant::display","log.line":149,"log.file":"C:\\Users\\Administrator\\.cargo\\git\\checkouts\\substrate-7bc20b373ca3e834\\55c157c\\client\\informant\\src\\display.rs"}
{"v":0,"name":"pulsar","msg":"💔 Verification failed for block 0x83266d6093601a3bafba7bcfca0740d50503972aa3ef1248866f2b3e6b8e2cbb received from (12D3KooWNGd9pwbDNPRrf2RxGj2m7zpwPFxVEnNKCKx6bXvPqCoZ): \"Header 0x83266d6093601a3bafba7bcfca0740d50503972aa3ef1248866f2b3e6b8e2cbb rejected: too far in the future

Links to relevant discord conversations with this issue:

Curious your thoughts @nazar-pc, I understand the private = public is just informational but it seems there may be some other networking issues going on which is just displaying itself with this informational log. Was also thinking that i may just be due to the load on the network with inc. launching yesterday. any insights you may have are appreciated :pray:

3

Looks like you have customized connection limits, why? It shouldn’t be necessary and might actually harm things. Try to use instructions exactly as provided in official documentation.

Not related at all, those are just informational messages that should really be debug messages, @shamil, users try to associate them with various things unnecessarily.

What about CPU usage, I/O? Without any details hard to provide a useful answer.

This only means that either your machine doesn’t have system time synced properly or the machine that produced a block (in which case these messages should be rare), just like error indicates.

1 Like