Subspace farmer Mar 18 is blocking after plot.bin is ready

Support
1

Issue Report

Subspace farmer cannot continue after ploting.

Environment

  • Operating System:
    Ubuntu 22.04 ARM64
  • Pulsar/Advanced CLI/Docker:
    Docker container run the subspace farmer process

Problem

Farmer does not continue, and stay there for more than 10 hours while the plot.bin file is 1.6TB

./subspace-farmer-ubuntu-aarch64-gemini-3h-2024-mar-18 farm --node-rpc-url ws://192.168.0.11:9944 --listen-on /ip4/0.0.0.0/tcp/30533 --listen-on /ip6/::/tcp/30533 --listen-on /ip4/0.0.0.0/udp/30533/quic-v1 --listen-on /ip6/::/udp/30533/quic-v1 --in-connections 300 --out-connections 100 --pending-in-connections 100 --pending-out-connections 100 --allow-private-ips --reward-address stxxxx path=/data/farm/,size=1.6TiB
2024-03-19T06:57:17.572803Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://192.168.0.11:9944
2024-03-19T06:57:17.644019Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWCsQ5uPrVF68Dq461zBEh8QrYbdbDSgavCaMqcg871NR4 protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-03-19T06:57:17.647546Z  INFO libp2p_swarm: local_peer_id=12D3KooWCsQ5uPrVF68Dq461zBEh8QrYbdbDSgavCaMqcg871NR4
2

Interesting and unexpected.

Please help me with a few things:

  • what hardware is it?
  • do you see any CPU usage or disk activity?
  • what file system are you using and what is the total size of the file system?
3

d019adfe-a405-4031-afd3-742a03428767
d019adfe-a405-4031-afd3-742a034287671272×1140 223 KB

nmon screenshot

4

File system is using sshfs mounted a shared folder in LAN.

5

Ubuntu 22.04
8 core ARM64 CPU
12GB DDR

6

Do you have two accounts or something? I’m confused because I asked this to someone else.
Either way SSHFS will clearly not going to work, so…

7

Yes, I have two accounts. Sorry for the confusion.

8

So SSHFS caused this blocking, right, but why

9

I think SSHFS is just horribly slow, so when farmer tries to read thousands of different tiny things at once, it slows down to a crawl, so it looks like it is not doing anything, but it is likely just struggling.