Farming cluster documents

We are testing subspace new feature - farming cluster.

Can Dev team share its design docs or videos helping to better understand how it work inside? such as the detail about how controller, cache, plotter, farmer work together with NATs.

(I have trouble to access discord)

1 Like

Have you checked here? Farming cluster - #44 by nazar-pc there is a lot of info from others testing it.

1 Like

Yes, I have checked it out but it is not well organized for a new feature.

1 Like

That feature has not shipped yet, you need to understand that it is early preview and you’re expected to do some reading and experimentation before full proper documentation arrives.

1 Like

Farming cluster farmer Log shows “Timed out without ping from plotter” frequently.
The plotting never completes (regular farmer takes 200 seconds to plot 1 sector).

./subspace-farmer cluster --nats-server nats://192.168.0.2:4222 farmer --reward-address st9XXXX path=/home/fact/subspace/farm,size=300GiB

2024-05-13T22:40:38.948482Z  INFO async_nats: event: connected
2024-05-13T22:40:38.948539Z  INFO async_nats: event: connected
2024-05-13T22:40:38.948888Z  INFO async_nats: event: connected
2024-05-13T22:40:38.948913Z  INFO async_nats: event: connected
2024-05-13T22:40:38.949089Z  INFO async_nats: event: connected
2024-05-13T22:40:38.949137Z  INFO async_nats: event: connected
2024-05-13T22:40:38.949154Z  INFO async_nats: event: connected
2024-05-13T22:40:38.949664Z  INFO async_nats: event: connected
2024-05-13T22:40:40.047791Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-05-13T22:40:40.050085Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-05-13T22:40:40.051916Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-05-13T22:40:42.773150Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-05-13T22:40:42.836397Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:
2024-05-13T22:40:42.836422Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   ID: 01HSSQNHR8ZDTM9G0MWXE0DYD6
2024-05-13T22:40:42.836428Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-05-13T22:40:42.836437Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Public key: 0xccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b
2024-05-13T22:40:42.836450Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Allocated space: 300.0 GiB (322.1 GB)
2024-05-13T22:40:42.836455Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Directory: /home/fact/subspace/farm
2024-05-13T22:40:42.920249Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-05-13T22:40:42.920283Z  INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-05-13T22:40:42.923447Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-05-13T22:40:42.926821Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.01% complete) sector_index=301
2024-05-13T22:40:42.928840Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.34% complete) sector_index=302
2024-05-13T22:40:42.930768Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.67% complete) sector_index=303
2024-05-13T22:40:52.928272Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:40:52.931972Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:40:53.930086Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-13T22:40:53.933836Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:03.935659Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:05.256981Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:15.258222Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:16.260708Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:26.261824Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:27.264539Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:37.265542Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:38.267672Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:42.957202Z  WARN subspace_farmer::cluster::nats_client: Acknowledgement wait timed out request_type=subspace_farmer::cluster::farmer::ClusterFarmerPlottedSectorsRequest response_type=core::result::Result<subspace_farmer_components::plotting::PlottedSector, alloc::string::String>
2024-05-13T22:41:43.042476Z  WARN subspace_farmer::cluster::nats_client: Acknowledgement wait timed out request_type=subspace_farmer::cluster::farmer::ClusterFarmerPlottedSectorsRequest response_type=core::result::Result<subspace_farmer_components::plotting::PlottedSector, alloc::string::String>
2024-05-13T22:41:48.269453Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:49.272713Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:59.274503Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:00.276414Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:42:10.277172Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:11.279396Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:42:21.280389Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:22.285920Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:42:32.287669Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:33.291103Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303

Try latest build: Farming cluster - #53 by nazar-pc

It should work much better.

What hardware are you using for each farming cluster component?

ARM64 8 cores,12GB DDR,Ubuntu 18.04

Hi Nazar, could you please send me some video or document links about the subspace-farmer (regular farmer not cluster)? I am lost in subspace-farmer RUST code.

Is it the same as latest build: git pull the latest code in farming-cluster branch, and build it with cargo build --profile production --bin subspace-farmer?

What ARM64 cores exactly, do you have a specific model or name of the SBC/server you’re using? Are you running all cluster components on the same machine?

https://docs.subspace.network/ is what you want to read. No need to read Rust code. If you want code docs then Index of crates is probably where you should look, but some of the documentation might be outdated.

I build and deployed the latest farming-cluster branch code. and run on my another x64 server, also report timeout warnings.

NATs runs on other server, while controller, cache, plotter, farmer run on this x64 server.
x64 server cpu info:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            12th Gen Intel(R) Core(TM) i5-12400F
    CPU family:          6
    Model:               151
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            5
    CPU max MHz:         4400.0000
    CPU min MHz:         800.0000
    BogoMIPS:            4992.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cm
                         ov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pb
                         e syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmo
                         n pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfm
                         perf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx
                          est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic
                          movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand 
                         lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp 
                         ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsb
                         ase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx 
                         smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv
                         1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts
                          hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi
                          umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri 
                         movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d ar
                         ch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   288 KiB (6 instances)
  L1i:                   192 KiB (6 instances)
  L2:                    7.5 MiB (6 instances)
  L3:                    18 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-11
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer san
                         itization
  Spectre v2:            Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RS
                         B filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected
  Tsx async abort:       Not affected

Latest code log:

2024-05-14T13:00:20.777302Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777354Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777376Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777386Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777391Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777394Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777405Z  INFO async_nats: event: connected
2024-05-14T13:00:20.777549Z  INFO async_nats: event: connected
2024-05-14T13:00:21.827205Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-05-14T13:00:21.829385Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-05-14T13:00:21.830731Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-05-14T13:00:24.476067Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-05-14T13:00:24.530805Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:
2024-05-14T13:00:24.530833Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   ID: 01HSSQNHR8ZDTM9G0MWXE0DYD6
2024-05-14T13:00:24.530839Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-05-14T13:00:24.530846Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Public key: 0xccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b
2024-05-14T13:00:24.530857Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Allocated space: 300.0 GiB (322.1 GB)
2024-05-14T13:00:24.530862Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Directory: /home/fact/subspace/farm
2024-05-14T13:00:24.613823Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-05-14T13:00:24.613856Z  INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-05-14T13:00:24.615787Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-05-14T13:00:24.617945Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.01% complete) sector_index=301
2024-05-14T13:00:24.620497Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.34% complete) sector_index=302
2024-05-14T13:00:24.622425Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.67% complete) sector_index=303
2024-05-14T13:00:34.619383Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:34.621701Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:34.624077Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:35.622328Z  INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=302}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:00:35.622986Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:00:35.626273Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-14T13:00:45.623723Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:45.624023Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:45.627370Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:46.626285Z  INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:00:46.626439Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:00:46.629878Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-14T13:00:56.627887Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:56.627919Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:56.631286Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:57.630725Z  INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:00:57.630905Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:00:57.633128Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-14T13:01:07.632059Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:01:07.632087Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:01:07.634511Z  WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:01:08.635073Z  INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=302}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=303}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:01:08.635352Z  INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=302}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=303}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:01:08.636213Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303

It is really difficult when you don’t answer questions I’m asking :disappointed:

Please run all components separately for now and if the issue persists, run them with RUST_LOG=info,subspace_farmer=trace so I have more information to work with. See how vexr reports bugs in farming cluster thread. Please include full commands you’re using to run things and if you build it from source then tell me how exactly are you doing it and exact commit from which you’ve built it.

Also please format messages like I edited above with spoilers and code blocks, it is much easier to read it that way comparing to a wall of text that everyone has to scroll through.

12400F should be fine and it shoud be possible to make it work. However the plotter doesn’t seem to be in logs above despite your claim, which likely indicates plotter is only running on some other machine, likely that underpowered ARM64 computer that you persistently refuse to provide any details on.

With newest commit in farming-cluster branch, this farmer timeout issue has gone.

2 Likes