Segment_reconstruction: Recovering missing piece on free disk

Support
,
1

Issue Report

Environment

  • Ubuntu 20.04
  • Advanced CLI ( gemini-3f-2023-sep-11-2 (Skylake edition) )
  • bare metal server (Xeon 2686v4 18 cores, 64gb DDR4, 1tb SSD NVME)

Node is synced and all disk wiped (No data with plots) but farmer after start trying to segment_reconstruction on NEW plot (first run) more then 1day…
Wipe plots and purge chain on node not helped. Maybe i need to clean any cache?

Now i’m trying single plot 370G. ( lesser plot size not helped 185G)

Farmer logs

сен 14 09:40:30 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:30.685086Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://127.0.0.1:9944
сен 14 09:40:30 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:30.691714Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D protocol_version=/subspace/92e91e657747c41eeabed5129ff51689d2e935b9f6abfbd5dfcb2e1d0d035095
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: Single disk farm 0:
сен 14 09:40:33 andrew-node subspace-farmer[3765607]:   ID: 01HA96Q08DK95SHS6V7958J2AX
сен 14 09:40:33 andrew-node subspace-farmer[3765607]:   Genesis hash: 0x92e91e657747c41eeabed5129ff51689d2e935b9f6abfbd5dfcb2e1d0d035095
сен 14 09:40:33 andrew-node subspace-farmer[3765607]:   Public key: 0x943662006cfc265b12cf8547b87d15f562de0fb441c4022642a13062608af846
сен 14 09:40:33 andrew-node subspace-farmer[3765607]:   Allocated space: 344.6 GiB (370.0 GB)
сен 14 09:40:33 andrew-node subspace-farmer[3765607]:   Directory: /media/andrew/nvme/plots/1
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.779975Z  INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.779983Z  INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.780150Z  INFO subspace_farmer::piece_cache: Initializing piece cache
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.782453Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.782668Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.783110Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.784142Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.784177Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.1.136/tcp/30533/p2p/12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.784189Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.122.1/tcp/30533/p2p/12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.784200Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/172.17.0.1/tcp/30533/p2p/12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D
сен 14 09:40:33 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:33.784210Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/172.18.0.1/tcp/30533/p2p/12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D
сен 14 09:40:34 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:34.263960Z  INFO subspace_networking::node_runner: Added observed address as external: /ip4/MY_IP/tcp/30533
сен 14 09:40:39 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:39.511983Z  INFO subspace_farmer::piece_cache: Synchronizing cache
сен 14 09:40:45 andrew-node subspace-farmer[3765607]: 2023-09-14T06:40:45.781458Z  INFO subspace_networking::node_runner: Public address status changed. old=Unknown new=Private
сен 14 09:41:26 andrew-node subspace-farmer[3765607]: 2023-09-14T06:41:26.578424Z  INFO subspace_networking::node_runner: Added observed address as external: /ip4/192.168.33.1/tcp/30533
сен 14 09:42:02 andrew-node subspace-farmer[3765607]: 2023-09-14T06:42:02.570495Z  INFO subspace_networking::node_runner: Added observed address as external: /ip4/10.10.1.1/tcp/30533
сен 14 09:43:45 andrew-node subspace-farmer[3765607]: 2023-09-14T06:43:45.882188Z  INFO subspace_networking::node_runner: Public address status changed. old=Private new=Public("/ip4/MY_IP/tcp/30533/p2p/12D3KooWBMRsSUa7SuGiB1YTZzpcqcxPU6Atbx7aDwUvCMzpij1D")
сен 14 10:21:28 andrew-node subspace-farmer[3765607]: 2023-09-14T07:21:28.681167Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=3099
сен 14 10:33:37 andrew-node subspace-farmer[3765607]: 2023-09-14T07:33:37.166600Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece succeeded. missing_piece_index=3099
сен 14 10:33:37 andrew-node subspace-farmer[3765607]: 2023-09-14T07:33:37.177531Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=3820
сен 14 10:33:56 andrew-node subspace-farmer[3765607]: 2023-09-14T07:33:56.820696Z  INFO subspace_networking::node_runner: Added observed address as external: /ip4/10.10.3.1/tcp/30533
сен 14 10:35:53 andrew-node subspace-farmer[3765607]: 2023-09-14T07:35:53.001390Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece succeeded. missing_piece_index=3820
сен 14 10:35:53 andrew-node subspace-farmer[3765607]: 2023-09-14T07:35:53.008758Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=2849

Farmer config (cat /etc/systemd/system/subspace-farmer.service)

[Unit]
Description=Subspace Farmer
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/subspace-farmer farm --reward-address *** path=/media/andrew/nvme/plots/1,size=370G
Restart=on-failure
LimitNOFILE=1024000

[Install]
WantedBy=multi-user.target

Node config (cat /etc/systemd/system/subspace-node.service)

[Unit]
Description=Subspace NODE
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/subspace-node --chain gemini-3f   --execution wasm   --blocks-pruning 256   --state-pruning archive   --no-private-ipv4   --validator   --name "zug"
Restart=on-failure
LimitNOFILE=1024000

[Install]
WantedBy=multi-user.target
2

Recovery may happen sometimes, as long as it succeeds you should be fine. It just means your node wasn’t able to find piece right away and had to recover, which takes much longer. But there is nothing wrong with your node and restarting will only make it start over, so better let it be.