repost:
Unifi Dream Machine
Thanks, a few more thoughts:
Since there are appliances between pfSense and other machines, try to test speed between them directly:
Install iperf3 on pfSense and machines and try to test bandwidth between them. Typically iperf -s on one end to start server and iperf3 -c server_ip on the other end to connect to it.
20% CPU usage is fine assuming it is uniformly distributed between available CPU cores. Use something like htop to see per-core CPU usage on pfSense. 20% is close to 25%, which would mean CPU is a bottleneck if the load happens to fall on one CPU core only for whatever reason.
repost
February 4, 2024, 3:39pm
22
Working on iperf, but here is htop. I do see it get to about 30% per core at some times -but it is perfectly distributed across all cores.
iperf3:
hakedev@alpha:~$ iperf3 -c 192.168.10.102
Connecting to host 192.168.10.102, port 5201
[ 5] local 192.168.10.101 port 56126 connected to 192.168.10.102 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 115 MBytes 964 Mbits/sec 26 279 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 936 Mbits/sec 38 301 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 50 264 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 56 236 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 935 Mbits/sec 38 287 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec 52 239 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 38 301 KBytes
[ 5] 7.00-8.00 sec 113 MBytes 945 Mbits/sec 51 263 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 50 247 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 936 Mbits/sec 49 232 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 448 sender
[ 5] 0.00-10.04 sec 1.09 GBytes 936 Mbits/sec receiver
1 Like
Hm, so everything on your end looks perfectly fine. Can you try VPN and see if ISP is throttling specific requests? We can also try iperf over Internet to check external connectivity.
You can try iperf3 from pfSense box or one of the machines to this box in Europe:
iperf3 -c 65.109.39.229
-R could be added for benchmark in reverse direction.
repost
February 4, 2024, 4:03pm
24
Yikes - I am in US-west but this… doesn’t look great.
hakedev@bravo:~$ iperf3 -c 65.109.39.229
Connecting to host 65.109.39.229, port 5201
[ 5] local 192.168.10.102 port 41104 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 90.5 KBytes 741 Kbits/sec 8 8.48 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 0 8.48 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.48 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 7 8.48 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.48 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.48 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0 8.48 KBytes
[ 5] 7.00-8.00 sec 63.6 KBytes 521 Kbits/sec 3 2.83 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 2.83 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 154 KBytes 126 Kbits/sec 18 sender
[ 5] 0.00-10.17 sec 18.4 KBytes 14.8 Kbits/sec receiver
Nope, it does not. Try from pfSense box to reduce the number of things in between.
repost
February 4, 2024, 4:08pm
26
In reverse took forever to finish but same;
hakedev@bravo:~$ iperf3 -c 65.109.39.229 -R
Connecting to host 65.109.39.229, port 5201
Reverse mode, remote host 65.109.39.229 is sending
[ 5] local 192.168.10.102 port 54858 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 9.90 KBytes 81.1 Kbits/sec
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 1.41 KBytes 11.6 Kbits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 9.90 KBytes 81.1 Kbits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 12.7 KBytes 104 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.35 sec 164 KBytes 130 Kbits/sec 26 sender
[ 5] 0.00-10.00 sec 33.9 KBytes 27.8 Kbits/sec receiver
Here is from pfsense
[2.7.2-RELEASE][admin@pfSense.hakedev.lan]/root: iperf3 -c 65.109.39.229
Connecting to host 65.109.39.229, port 5201
[ 5] local 97.124.167.XXport 53075 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 64.2 KBytes 525 Kbits/sec 2 2.83 KBytes
[ 5] 1.00-2.01 sec 2.83 KBytes 22.9 Kbits/sec 1 1.41 KBytes
[ 5] 2.01-3.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 3 1.41 KBytes
[ 5] 5.00-6.01 sec 5.66 KBytes 45.7 Kbits/sec 2 5.66 KBytes
[ 5] 6.01-7.00 sec 0.00 Bytes 0.00 bits/sec 5 128 KBytes
[ 5] 7.00-8.00 sec 230 KBytes 1.88 Mbits/sec 24 77.9 KBytes
[ 5] 8.00-9.00 sec 654 KBytes 5.36 Mbits/sec 0 111 KBytes
[ 5] 9.00-10.00 sec 729 KBytes 5.97 Mbits/sec 0 123 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.65 MBytes 1.38 Mbits/sec 39 sender
[ 5] 0.00-10.17 sec 1.49 MBytes 1.23 Mbits/sec receiver
I’m suspecting ISP is applying some restrictions, probably based on number of connections or something like that and Subspace is by design P2P network, which means connections to machines all around the world.
Can you try to talk to support of ISP and see what they say?
repost
February 4, 2024, 4:11pm
28
I will try a VPN here soon - and will also call the ISP. I am running both Subspace + Spacemesh. But just to make sure there was nothing caused by Spacemesh I’ve turned it all of for the testing. Will post results from VPN testing soon.
1 Like
It’d be also helpful if ISP can say if it is number of connections or something else, it’d help to think how we can optimize Subspace around those restrictions.
1 Like
repost
February 4, 2024, 4:25pm
30
Connecting with just my laptop over a VPN the issue is totally resolved. I don’t have a VPN client on the pfsense yet - but seems that may resolve the issue for now. Will give it a test.
Chatting with ISP now but not sure how much support they will give on the weekend.
repost
February 4, 2024, 4:29pm
31
No change on my network other than using NordVPN on my laptop:
Connecting to host 65.109.39.229, port 5201
[ 4] local 10.5.0.2 port 51504 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec
[ 4] 1.00-2.04 sec 1.50 MBytes 12.2 Mbits/sec
[ 4] 2.04-3.01 sec 1.25 MBytes 10.8 Mbits/sec
[ 4] 3.01-4.00 sec 1.62 MBytes 13.6 Mbits/sec
[ 4] 4.00-5.02 sec 1.62 MBytes 13.4 Mbits/sec
[ 4] 5.02-6.01 sec 1.50 MBytes 12.7 Mbits/sec
[ 4] 6.01-7.01 sec 1.50 MBytes 12.6 Mbits/sec
[ 4] 7.01-8.00 sec 1.50 MBytes 12.7 Mbits/sec
[ 4] 8.00-9.00 sec 1.50 MBytes 12.6 Mbits/sec
[ 4] 9.00-10.01 sec 1.75 MBytes 14.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 14.1 MBytes 11.8 Mbits/sec sender
[ 4] 0.00-10.01 sec 14.1 MBytes 11.8 Mbits/sec receiver
iperf Done.
repost
February 4, 2024, 4:41pm
32
I just got off the phone with my ISP - almost 100% chance I was talking with someone outside the US, and they were not much help. Basically they ‘established some connection and found an error’ which they have now allegedly resolved. Sounds like they just made that up but who knows. Will see if anything improves. They wouldn’t tell me what this error was, only that I should see an improvement.
However I am enjoying a speed up in my Space Acres syncing while on this VPN - whatever is happening it REALLY slows down my ability to sync.
repost
February 4, 2024, 4:44pm
33
Turned off my VPN just to test again with iperf3:
Connecting to host 65.109.39.229, port 5201
[ 4] local 192.168.10.34 port 65267 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 256 KBytes 2.09 Mbits/sec
[ 4] 1.00-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.01-4.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.01-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 5.00-6.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 6.01-7.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 7.01-8.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.01-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.00-10.01 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 256 KBytes 209 Kbits/sec sender
[ 4] 0.00-10.01 sec 91.2 KBytes 74.7 Kbits/sec receiver
That is very sad, man! It just goes completely offline after a bit.
repost
February 4, 2024, 4:46pm
35
This is with just 1 node/farm running + Space Acres, No VPN
Connecting to host 65.109.39.229, port 5201
[ 4] local 192.168.10.34 port 53868 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 256 KBytes 2.08 Mbits/sec
[ 4] 1.01-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.01 sec 128 KBytes 1.05 Mbits/sec
[ 4] 3.01-4.01 sec 128 KBytes 1.05 Mbits/sec
[ 4] 4.01-5.01 sec 256 KBytes 2.10 Mbits/sec
[ 4] 5.01-6.01 sec 256 KBytes 2.10 Mbits/sec
[ 4] 6.01-7.01 sec 384 KBytes 3.14 Mbits/sec
[ 4] 7.01-8.00 sec 768 KBytes 6.33 Mbits/sec
[ 4] 8.00-9.00 sec 1.00 MBytes 8.38 Mbits/sec
[ 4] 9.00-10.00 sec 1.12 MBytes 9.46 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.25 MBytes 3.57 Mbits/sec sender
[ 4] 0.00-10.00 sec 4.25 MBytes 3.57 Mbits/sec receiver
iperf Done.
repost
February 4, 2024, 4:48pm
36
And lastly - With ONLY Space Acres running (no VPN):
Connecting to host 65.109.39.229, port 5201
[ 4] local 192.168.10.34 port 65038 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 256 KBytes 2.07 Mbits/sec
[ 4] 1.01-2.01 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 2.01-3.01 sec 1.38 MBytes 11.6 Mbits/sec
[ 4] 3.01-4.00 sec 1.38 MBytes 11.6 Mbits/sec
[ 4] 4.00-5.01 sec 1.50 MBytes 12.4 Mbits/sec
[ 4] 5.01-6.01 sec 1.50 MBytes 12.7 Mbits/sec
[ 4] 6.01-7.01 sec 1.25 MBytes 10.4 Mbits/sec
[ 4] 7.01-8.02 sec 1.50 MBytes 12.5 Mbits/sec
[ 4] 8.02-9.05 sec 1.38 MBytes 11.1 Mbits/sec
[ 4] 9.05-10.02 sec 1.38 MBytes 12.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.02 sec 12.8 MBytes 10.7 Mbits/sec sender
[ 4] 0.00-10.02 sec 12.7 MBytes 10.7 Mbits/sec receiver
iperf Done.
erdnapa
February 4, 2024, 6:52pm
37
Are those last results from the pfSense box as well? If the unify stuff is still in between that might contribute too.
But it looks like you have ISP throttling.
I run ISP ↔ pfSense ↔ Microtik ↔ Node/Farmers
(Sorry Nazar, took the liberty for science! Only once each way.)
iperf3 -c 65.109.39.229
Connecting to host 65.109.39.229, port 5201
[ 5] local 192.168.40.23 port 42148 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.47 MBytes 12.3 Mbits/sec 0 308 KBytes
[ 5] 1.00-2.00 sec 1.31 MBytes 11.0 Mbits/sec 52 232 KBytes
[ 5] 2.00-3.00 sec 730 KBytes 5.98 Mbits/sec 17 127 KBytes
[ 5] 3.00-4.00 sec 669 KBytes 5.48 Mbits/sec 0 139 KBytes
[ 5] 4.00-5.00 sec 669 KBytes 5.48 Mbits/sec 0 143 KBytes
[ 5] 5.00-6.00 sec 669 KBytes 5.48 Mbits/sec 0 143 KBytes
[ 5] 6.00-7.00 sec 669 KBytes 5.48 Mbits/sec 0 143 KBytes
[ 5] 7.00-8.00 sec 730 KBytes 5.98 Mbits/sec 13 110 KBytes
[ 5] 8.00-9.00 sec 669 KBytes 5.48 Mbits/sec 0 130 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 11 99.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.47 MBytes 6.26 Mbits/sec 93 sender
[ 5] 0.00-10.27 sec 6.31 MBytes 5.15 Mbits/sec receiver
iperf Done.
Reverse:
iperf3 -c 65.109.39.229 -R
Connecting to host 65.109.39.229, port 5201
Reverse mode, remote host 65.109.39.229 is sending
[ 5] local 192.168.40.23 port 59044 connected to 65.109.39.229 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.55 MBytes 13.0 Mbits/sec
[ 5] 1.00-2.00 sec 19.5 MBytes 163 Mbits/sec
[ 5] 2.00-3.00 sec 19.9 MBytes 167 Mbits/sec
[ 5] 3.00-4.00 sec 19.9 MBytes 167 Mbits/sec
[ 5] 4.00-5.00 sec 19.6 MBytes 164 Mbits/sec
[ 5] 5.00-6.00 sec 20.2 MBytes 169 Mbits/sec
[ 5] 6.00-7.00 sec 20.4 MBytes 171 Mbits/sec
[ 5] 7.00-8.00 sec 20.9 MBytes 175 Mbits/sec
[ 5] 8.00-9.00 sec 20.7 MBytes 174 Mbits/sec
[ 5] 9.00-10.00 sec 19.9 MBytes 167 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.15 sec 194 MBytes 161 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 183 MBytes 153 Mbits/sec receiver
repost
February 4, 2024, 7:46pm
38
Those latest were from my windows laptop on the network. So all the way through everything. Which is actually even worse because it goes Laptop → Eero Wifi → Dream Machine → pfsense → ISP Router (bridge mode) → Internet
What speed does ISP promise to you BTW?
repost
February 4, 2024, 8:13pm
40
1gbps up/down - and I usually have no problem getting that
