Hello, I see a farmer stuck in a cycle failing to recover missing piece. The farmer never leaves this cycle. This farmer uses Sept-13-2 right now, but error also occurred with Sept-11. I saw some users reporting the same issue/error message on discord also.
Blockquote
2023-09-14T16:47:42.836080Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Node is synced, resuming plotting
2023-09-14T16:47:42.836139Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Node is synced, resuming plotting
2023-09-14T16:48:51.106731Z ERROR yamux::connection: 7d36dc01: maximum number of streams reached
2023-09-14T17:19:58.324072Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=6086
2023-09-14T17:20:22.417245Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=1920
2023-09-14T20:14:11.132484Z ERROR single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=6086 received_pieces=35 required_pieces_number=128
2023-09-14T20:14:11.132597Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=6086
2023-09-14T20:16:07.707771Z ERROR single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=1920 received_pieces=48 required_pieces_number=128
2023-09-14T20:16:07.708105Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=1695
I think the farmer had finished the plotting previously. This is one of my earliest farmers. Must have gone through most re-plotting cycles of all my farmers. Not sure if that matters.
This is probably the reason. Have you modified something, also did you change default CLI parameters related to networking? It can result in this (exceeding number of streams).
Self compiled but unchanged code (I have only changed the code on one single machine and reverted that after no success). Node and farmers run with defaults, only changed (from defaults) configuration parameters have to do with the ports.
We have seen this rarely, we’ll investigate why this happens. I do not have a good answer for you right now. Overall this shouldn’t be fatal, did you see plotting continue eventually or was it stuck permanently?
It was stuck in there for a while, several hours, so i restarted, meanwhile also reinstalled sep-11, the nodes with sep-13-2 all lost sync and didn’t get back on track. with sep-11 they all resynced.
but this particular reported issue, i didn’t see it again so far.
Oh, this is a long-known problem. Farmer does not come out of this state for days (I just did not check longer). This has been discussed many times in discord, the solution helps with reducing the number of allowed connections for the farmer and, possibly, for the node too.
This should no longer be necessary, recent releases addressed this specific issue, specifically sep-05 release if I’m not mistaken. As mentioned in other threads, things are changing and we do make improvements that should remove the need of workarounds.
Well, then would be great to report it here right away and help developers with debugging so we can improve things going forward.
Everything except --dsn-in-connections on the node is irrelevant for DSN, so I have no idea why would you customize it at all. And for farmer I’m not sure how it would help either to be completely honest.
Are you sure it is those parameters that help and not a coincidence? Can be checked by creating backup of the node data and starting once without those customized arguments and another time with them.
There is another dedicated (remote) farmer connected to this node (in addition to the farmer working on the same machine where the node itself), it has finished plotting today. It has the same startup parameters. Is it possible that the node does not support more than two farmers (which are already connected to it)?
Looks like they are having issues with those arguments added, not without, am I reading it correctly?
It would be ideal if people didn’t tweak those parameters unless requested by developers, it wasn’t once or twice that it was the reason things didn’t work as expected.
No, node can work with many more farmers implementation-wise.
With those parameters added plotters at least start plotting. The community does not resort to their help out of boredom…
Of course, perhaps it’s not about these para, but about the limit on the maximum number of plots that farmer agrees to plot, but this is not mentioned anywhere. What remains for the community? Look for solutions to their problems on their own. And if these measures help, so be it.
Don’t you think this is an empty argument? After all, it seems that no one has really said yet why the plotter enters this conditionally eternal cycle of recovering missing pieces, what causes this behavior and how to deal with it…
I understand that users are not doing it for no reason, but at the same time thigns might be fixed on the latest release and in some cases evidence is anecdotal.
It would be great to have such things shared on the forum, we can then request more detailed logs with and without those parameters to understand why it is helping (if it does) to make it work well by default. In the end the goal is to make it “just work”.
Interesting question actually, what was the first release you have used? I’m asking because before aug-21 there was a bug in the code that may result in node being banned on DSN. If you wiped the plot after upgrade or started with aug-21 or later release then this is not applicable.
Have a farmer doesn’t even start to plot, setup was newly created. sep-13-2, trying 21 now
Blockquote
2023-09-21T00:55:50.998877Z INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)…
2023-09-21T00:55:50.998883Z INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2023-09-21T00:55:50.998936Z INFO subspace_farmer::piece_cache: Initializing piece cache
2023-09-21T00:55:51.000043Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000199Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.000216Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000220Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.000233Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000237Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.000378Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000484Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.001288Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.004252Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.004865Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/30406/p2p/…
2023-09-21T00:55:51.004974Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/10.1.100.30/tcp/30406/p2p/…
2023-09-21T00:55:51.009945Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.013137Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.075974Z INFO subspace_networking::node_runner: Added observed address as external: /ip4/8.8.8.8/tcp/30406
2023-09-21T00:56:03.952011Z INFO subspace_networking::node_runner: Public address status changed. old=Unknown new=Private
2023-09-21T00:58:35.271670Z INFO subspace_farmer::piece_cache: Synchronizing cache
2023-09-21T01:06:34.280205Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=4615
2023-09-21T01:06:34.297562Z ERROR yamux::connection: 9de79bbf: maximum number of streams reached
2023-09-21T01:06:34.329215Z ERROR yamux::connection: 4eed3dfb: maximum number of streams reached
2023-09-21T01:06:34.330569Z ERROR yamux::connection: 07407985: maximum number of streams reached
2023-09-21T01:06:34.341485Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=6329
2023-09-21T01:06:34.373631Z ERROR yamux::connection: 2857d6fb: maximum number of streams reached
2023-09-21T01:06:34.374652Z ERROR yamux::connection: 42b44e86: maximum number of streams reached
2023-09-21T01:06:34.378987Z ERROR yamux::connection: c04644bf: maximum number of streams reached
2023-09-21T01:06:34.380793Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=4526
2023-09-21T01:06:34.381492Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=7436
2023-09-21T01:06:34.390924Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=2497
2023-09-21T01:06:34.391230Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=4726
2023-09-21T01:06:34.392037Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=2285
2023-09-21T01:06:34.392894Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=5042
2023-09-21T01:06:34.395350Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=8152
2023-09-21T01:06:34.434777Z ERROR yamux::connection: 1065f142: maximum number of streams reached
2023-09-21T01:06:34.447526Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=5339
2023-09-21T01:06:34.450208Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=7529
2023-09-21T01:06:34.460856Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=3691
2023-09-21T01:06:34.467599Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=7869
2023-09-21T01:06:34.515328Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=3257
2023-09-21T01:06:34.516873Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=3360
2023-09-21T01:06:34.523081Z ERROR yamux::connection: bf45c269: maximum number of streams reached
