NAS upgrade

NAS

  1. Login
  2. sudo synogear install
  3. iperf3 -s -p 5201

Workstation

  1. Login
  2. iperf3 -c 192.168.129.3 -p 5201

Plan

NAS

  • LAN: 192.168.129.3/24
  • P2P: 192.168.130.9/30

Titan

  • LAN: 192.168.129.217/24
  • P2P: 192.168.130.10/30

Hardware

USB-C 10Gb NIC

enp4s0f4u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 34:c8:d6:b3:08:ac brd ff:ff:ff:ff:ff:ff
  altname enx34c8d6b308ac

Baseline

Titan: 1Gb port / 1Gb link NAS: 1Gb port / 1Gb link

$ iperf3 -c 192.168.129.3 -p 5201
Connecting to host 192.168.129.3, port 5201
[  4] local 192.168.129.217 port 35646 connected to 192.168.129.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec   114 MBytes   955 Mbits/sec    0    355 KBytes       
[  4]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    355 KBytes       
[  4]   2.00-3.00   sec   112 MBytes   940 Mbits/sec    0    386 KBytes       
[  4]   3.00-4.00   sec   113 MBytes   946 Mbits/sec    0    386 KBytes       
[  4]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0    404 KBytes       
[  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    404 KBytes       
[  4]   6.00-7.00   sec   112 MBytes   936 Mbits/sec    0    424 KBytes       
[  4]   7.00-8.00   sec   113 MBytes   947 Mbits/sec    0    424 KBytes       
[  4]   8.00-9.00   sec   112 MBytes   937 Mbits/sec    0    424 KBytes       
[  4]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    0    424 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0            sender
[  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

Use USB 10Gb on Titan

Titan: 10Gb port / 1Gb link NAS: 1Gb port / 1Gb link

$ iperf3 -c 192.168.129.3 -p 5201
Connecting to host 192.168.129.3, port 5201
[  4] local 192.168.129.231 port 37496 connected to 192.168.129.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec   114 MBytes   959 Mbits/sec    0    419 KBytes       
[  4]   1.00-2.00   sec   112 MBytes   937 Mbits/sec    0    419 KBytes       
[  4]   2.00-3.00   sec   112 MBytes   944 Mbits/sec    0    440 KBytes       
[  4]   3.00-4.00   sec   112 MBytes   943 Mbits/sec    0    440 KBytes       
[  4]   4.00-5.00   sec   112 MBytes   943 Mbits/sec    0    440 KBytes       
[  4]   5.00-6.00   sec   112 MBytes   937 Mbits/sec    0    440 KBytes       
[  4]   6.00-7.00   sec   113 MBytes   945 Mbits/sec    0    440 KBytes       
[  4]   7.00-8.00   sec   112 MBytes   938 Mbits/sec    0    440 KBytes       
[  4]   8.00-9.00   sec   112 MBytes   944 Mbits/sec    0    440 KBytes       
[  4]   9.00-10.00  sec   112 MBytes   937 Mbits/sec    0    440 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0            sender
[  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.

Install 10Gb on NAS

Titan: 10Gb port / 1Gb link NAS: 10Gb port / 1Gb link

$ iperf3 -c 192.168.130.9 -p 5201
Connecting to host 192.168.130.9, port 5201
[  4] local 192.168.130.10 port 51120 connected to 192.168.130.9 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec   564 MBytes  4.73 Gbits/sec    0   1.16 MBytes       
[  4]   1.00-2.00   sec   561 MBytes  4.70 Gbits/sec    0   1.16 MBytes       
[  4]   2.00-3.00   sec   561 MBytes  4.71 Gbits/sec    0   1.23 MBytes       
[  4]   3.00-4.00   sec   562 MBytes  4.71 Gbits/sec    0   1.23 MBytes       
[  4]   4.00-5.00   sec   561 MBytes  4.71 Gbits/sec    0   1.23 MBytes       
[  4]   5.00-6.00   sec   561 MBytes  4.71 Gbits/sec    0   1.23 MBytes       
[  4]   6.00-7.00   sec   561 MBytes  4.71 Gbits/sec    0   1.29 MBytes       
[  4]   7.00-8.00   sec   560 MBytes  4.70 Gbits/sec    0   1.29 MBytes       
[  4]   8.00-9.00   sec   562 MBytes  4.71 Gbits/sec    0   1.29 MBytes       
[  4]   9.00-10.00  sec   561 MBytes  4.70 Gbits/sec    0   1.29 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec    0            sender
[  4]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec                  receiver

iperf Done.

File performance testing

Basic rsync from local NVME to $HOME/tmp/

  1. Over 1Gb NICs
$ rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ && sync
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  161.83MB/s    0:00:38 (xfr#1, to-chk=0/1)
  1. Over 10Gb NICs
$ rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ && sync
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  225.15MB/s    0:00:27 (xfr#1, to-chk=0:00:27 (xfr#1, to-chk=0/1)

Performance testing

Trying multiple iperf3 connections

Two connections

$ iperf3 -P 2 -p 5201 -c nas
Connecting to host nas, port 5201
[  4] local 192.168.130.10 port 36980 connected to 192.168.130.9 port 5201
[  6] local 192.168.130.10 port 36994 connected to 192.168.130.9 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec   284 MBytes  2.38 Gbits/sec    0    559 KBytes       
[  6]   0.00-1.00   sec   282 MBytes  2.37 Gbits/sec    0    536 KBytes       
[SUM]   0.00-1.00   sec   566 MBytes  4.75 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec   280 MBytes  2.35 Gbits/sec    0    559 KBytes       
[  6]   1.00-2.00   sec   282 MBytes  2.36 Gbits/sec    0    564 KBytes       
[SUM]   1.00-2.00   sec   562 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    0    559 KBytes       
[  6]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    0    564 KBytes       
[SUM]   2.00-3.00   sec   560 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   281 MBytes  2.35 Gbits/sec    0    559 KBytes       
[  6]   3.00-4.00   sec   281 MBytes  2.35 Gbits/sec    0    594 KBytes       
[SUM]   3.00-4.00   sec   562 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    0    559 KBytes       
[  6]   4.00-5.00   sec   281 MBytes  2.36 Gbits/sec    0    594 KBytes       
[SUM]   4.00-5.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0    559 KBytes       
[  6]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0    622 KBytes       
[SUM]   5.00-6.00   sec   560 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec    0    559 KBytes       
[  6]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0    622 KBytes       
[SUM]   6.00-7.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0    629 KBytes       
[  6]   7.00-8.00   sec   281 MBytes  2.36 Gbits/sec    0    622 KBytes       
[SUM]   7.00-8.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0    629 KBytes       
[  6]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0    622 KBytes       
[SUM]   8.00-9.00   sec   562 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0    629 KBytes       
[  6]   9.00-10.00  sec   282 MBytes  2.36 Gbits/sec    0    622 KBytes       
[SUM]   9.00-10.00  sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec    0            sender
[  4]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec    0            sender
[  6]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  5.49 GBytes  4.71 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec                  receiver

iperf Done.

Result: Same ~4.7 Gbits/sec performance.

Maybe it needs more cores/threads to see a difference?

Four connections

$ iperf3 -P 4 -p 5201 -c nas
Connecting to host nas, port 5201
[  4] local 192.168.130.10 port 50562 connected to 192.168.130.9 port 5201
[  6] local 192.168.130.10 port 50578 connected to 192.168.130.9 port 5201
[  8] local 192.168.130.10 port 50590 connected to 192.168.130.9 port 5201
[ 10] local 192.168.130.10 port 50596 connected to 192.168.130.9 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec   141 MBytes  1.18 Gbits/sec    0    324 KBytes       
[  6]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec    0    322 KBytes       
[  8]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec    0    321 KBytes       
[ 10]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec    0    307 KBytes       
[SUM]   0.00-1.00   sec   567 MBytes  4.75 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec   140 MBytes  1.18 Gbits/sec    0    324 KBytes       
[  6]   1.00-2.00   sec   140 MBytes  1.18 Gbits/sec    0    322 KBytes       
[  8]   1.00-2.00   sec   140 MBytes  1.18 Gbits/sec    0    321 KBytes       
[ 10]   1.00-2.00   sec   140 MBytes  1.17 Gbits/sec    0    322 KBytes       
[SUM]   1.00-2.00   sec   560 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec   140 MBytes  1.18 Gbits/sec    0    324 KBytes       
[  6]   2.00-3.00   sec   140 MBytes  1.18 Gbits/sec    0    338 KBytes       
[  8]   2.00-3.00   sec   141 MBytes  1.18 Gbits/sec    0    321 KBytes       
[ 10]   2.00-3.00   sec   141 MBytes  1.18 Gbits/sec    0    322 KBytes       
[SUM]   2.00-3.00   sec   562 MBytes  4.72 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   141 MBytes  1.18 Gbits/sec    0    341 KBytes       
[  6]   3.00-4.00   sec   140 MBytes  1.18 Gbits/sec    0    356 KBytes       
[  8]   3.00-4.00   sec   140 MBytes  1.18 Gbits/sec    0    355 KBytes       
[ 10]   3.00-4.00   sec   140 MBytes  1.17 Gbits/sec    0    338 KBytes       
[SUM]   3.00-4.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec   140 MBytes  1.17 Gbits/sec    0    341 KBytes       
[  6]   4.00-5.00   sec   141 MBytes  1.18 Gbits/sec    0    356 KBytes       
[  8]   4.00-5.00   sec   141 MBytes  1.18 Gbits/sec    0    355 KBytes       
[ 10]   4.00-5.00   sec   140 MBytes  1.18 Gbits/sec    0    338 KBytes       
[SUM]   4.00-5.00   sec   562 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec   140 MBytes  1.18 Gbits/sec    0    341 KBytes       
[  6]   5.00-6.00   sec   140 MBytes  1.18 Gbits/sec    0    373 KBytes       
[  8]   5.00-6.00   sec   140 MBytes  1.18 Gbits/sec    0    373 KBytes       
[ 10]   5.00-6.00   sec   141 MBytes  1.18 Gbits/sec    0    338 KBytes       
[SUM]   5.00-6.00   sec   562 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec   140 MBytes  1.17 Gbits/sec    0    341 KBytes       
[  6]   6.00-7.00   sec   140 MBytes  1.18 Gbits/sec    0    373 KBytes       
[  8]   6.00-7.00   sec   140 MBytes  1.18 Gbits/sec    0    373 KBytes       
[ 10]   6.00-7.00   sec   140 MBytes  1.18 Gbits/sec    0    338 KBytes       
[SUM]   6.00-7.00   sec   560 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec   140 MBytes  1.18 Gbits/sec    0    341 KBytes       
[  6]   7.00-8.00   sec   140 MBytes  1.18 Gbits/sec    0    373 KBytes       
[  8]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    373 KBytes       
[ 10]   7.00-8.00   sec   140 MBytes  1.18 Gbits/sec    0    338 KBytes       
[SUM]   7.00-8.00   sec   561 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   140 MBytes  1.18 Gbits/sec    0    341 KBytes       
[  6]   8.00-9.00   sec   141 MBytes  1.18 Gbits/sec    0    373 KBytes       
[  8]   8.00-9.00   sec   141 MBytes  1.18 Gbits/sec    0    373 KBytes       
[ 10]   8.00-9.00   sec   141 MBytes  1.18 Gbits/sec    0    338 KBytes       
[SUM]   8.00-9.00   sec   562 MBytes  4.72 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   141 MBytes  1.18 Gbits/sec    0    434 KBytes       
[  6]   9.00-10.00  sec   140 MBytes  1.18 Gbits/sec    0    373 KBytes       
[  8]   9.00-10.00  sec   140 MBytes  1.17 Gbits/sec    0    373 KBytes       
[ 10]   9.00-10.00  sec   140 MBytes  1.18 Gbits/sec    0    338 KBytes       
[SUM]   9.00-10.00  sec   561 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec    0            sender
[  4]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec    0            sender
[  6]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec    0            sender
[  8]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec    0            sender
[ 10]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  5.49 GBytes  4.71 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec                  receiver

iperf Done.

Result: Same ~4.7 Gbits/sec performance.

Nope. But we'll give more threads a test...

(Insert obligitory "MOAR!!!" image here...)

Ten connections

$ iperf3 -P 10 -p 5201 -c nas
Connecting to host nas, port 5201
[  4] local 192.168.130.10 port 48174 connected to 192.168.130.9 port 5201
[  6] local 192.168.130.10 port 48188 connected to 192.168.130.9 port 5201
[  8] local 192.168.130.10 port 48204 connected to 192.168.130.9 port 5201
[ 10] local 192.168.130.10 port 48208 connected to 192.168.130.9 port 5201
[ 12] local 192.168.130.10 port 48212 connected to 192.168.130.9 port 5201
[ 14] local 192.168.130.10 port 48224 connected to 192.168.130.9 port 5201
[ 16] local 192.168.130.10 port 48228 connected to 192.168.130.9 port 5201
[ 18] local 192.168.130.10 port 48230 connected to 192.168.130.9 port 5201
[ 20] local 192.168.130.10 port 48242 connected to 192.168.130.9 port 5201
[ 22] local 192.168.130.10 port 48250 connected to 192.168.130.9 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec  56.8 MBytes   476 Mbits/sec    0    161 KBytes       
[  6]   0.00-1.00   sec  56.9 MBytes   477 Mbits/sec    0    161 KBytes       
[  8]   0.00-1.00   sec  56.8 MBytes   476 Mbits/sec    0    144 KBytes       
[ 10]   0.00-1.00   sec  56.9 MBytes   477 Mbits/sec    0    161 KBytes       
[ 12]   0.00-1.00   sec  56.8 MBytes   476 Mbits/sec    0    161 KBytes       
[ 14]   0.00-1.00   sec  56.8 MBytes   476 Mbits/sec    0    161 KBytes       
[ 16]   0.00-1.00   sec  56.9 MBytes   477 Mbits/sec    0    161 KBytes       
[ 18]   0.00-1.00   sec  56.6 MBytes   475 Mbits/sec    0    156 KBytes       
[ 20]   0.00-1.00   sec  56.8 MBytes   476 Mbits/sec    0    163 KBytes       
[ 22]   0.00-1.00   sec  56.9 MBytes   477 Mbits/sec    0    161 KBytes       
[SUM]   0.00-1.00   sec   568 MBytes  4.76 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   1.00-2.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   1.00-2.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   1.00-2.00   sec   560 MBytes  4.70 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   2.00-3.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   2.00-3.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   2.00-3.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   2.00-3.00   sec   562 MBytes  4.72 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   3.00-4.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   3.00-4.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   3.00-4.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   4.00-5.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   4.00-5.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   4.00-5.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   4.00-5.00   sec   562 MBytes  4.72 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   5.00-6.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   5.00-6.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   5.00-6.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   5.00-6.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   6.00-7.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   6.00-7.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   6.00-7.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   6.00-7.00   sec   562 MBytes  4.72 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   7.00-8.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   7.00-8.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   7.00-8.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   7.00-8.00   sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   8.00-9.00   sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   8.00-9.00   sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   8.00-9.00   sec   562 MBytes  4.72 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec  56.2 MBytes   472 Mbits/sec    0    161 KBytes       
[  6]   9.00-10.00  sec  55.9 MBytes   469 Mbits/sec    0    161 KBytes       
[  8]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    144 KBytes       
[ 10]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 12]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 14]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 16]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[ 18]   9.00-10.00  sec  55.8 MBytes   468 Mbits/sec    0    163 KBytes       
[ 20]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    163 KBytes       
[ 22]   9.00-10.00  sec  56.0 MBytes   470 Mbits/sec    0    161 KBytes       
[SUM]   9.00-10.00  sec   561 MBytes  4.71 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  5.49 GBytes  4.71 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec                  receiver

iperf Done.

Result: Same ~4.7 Gbits/sec performance.

Trying with MTU of 9000

Both ports set to MTU=9000:

Local workstation:

$ ip address show dev enp4s0f4u1
3: enp4s0f4u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether 34:c8:d6:b3:08:ac brd ff:ff:ff:ff:ff:ff
    altname enx34c8d6b308ac
    inet 192.168.130.10/30 brd 192.168.130.11 scope global noprefixroute enp4s0f4u1
       valid_lft forever preferred_lft forever

And on the NAS:

# # ip address show dev eth2
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc mq state DOWN group default qlen 1000
    link/ether 90:09:d0:aa:83:d0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.130.9/30 brd 192.168.130.11 scope global eth2
       valid_lft forever preferred_lft forever
$ iperf3 -p 5201 -c nas
Connecting to host nas, port 5201
[  4] local 192.168.130.10 port 41914 connected to 192.168.130.9 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  4]   0.00-1.00   sec   507 MBytes  4.25 Gbits/sec    0    935 KBytes       
[  4]   1.00-2.00   sec   508 MBytes  4.26 Gbits/sec    0    987 KBytes       
[  4]   2.00-3.00   sec   505 MBytes  4.24 Gbits/sec    0    987 KBytes       
[  4]   3.00-4.00   sec   501 MBytes  4.20 Gbits/sec    0   1.02 MBytes       
[  4]   4.00-5.00   sec   500 MBytes  4.20 Gbits/sec    0   1.02 MBytes       
[  4]   5.00-6.00   sec   506 MBytes  4.25 Gbits/sec    0   1.07 MBytes       
[  4]   6.00-7.00   sec   502 MBytes  4.21 Gbits/sec    0   1.07 MBytes       
[  4]   7.00-8.00   sec   502 MBytes  4.21 Gbits/sec    0   1.07 MBytes       
[  4]   8.00-9.00   sec   498 MBytes  4.18 Gbits/sec    0   1.64 MBytes       
[  4]   9.00-10.00  sec   496 MBytes  4.15 Gbits/sec    0   1.64 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-10.00  sec  4.91 GBytes  4.22 Gbits/sec    0            sender
[  4]   0.00-10.00  sec  4.91 GBytes  4.21 Gbits/sec                  receiver

Result: Dang, a slight decrease, but possibly due to the larger overall packet size?

BUT!!! The "rsync" seems to be quite a bit faster, but also all over the place when it comes to the overall MB/s:

Write performance - local to NAS:

$ for C in $(seq 1 5) ; do rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && date && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ ; done
Sat Oct 31 02:55:27 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  808.13MB/s    0:00:07 (xfr#1, to-chk=0/1)
Sat Oct 31 02:56:02 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  440.86MB/s    0:00:14 (xfr#1, to-chk=0/1)
Sat Oct 31 02:56:57 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  680.04MB/s    0:00:09 (xfr#1, to-chk=0/1)
Sat Oct 31 02:57:46 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  733.48MB/s    0:00:08 (xfr#1, to-chk=0/1)
Sat Oct 31 02:58:37 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  650.65MB/s    0:00:09 (xfr#1, to-chk=0/1)
[dan@titan ~]$

The overall write speed comes out to 662MB/s.

Doing read performance from both locations:

Testing the local read:

$ time (for C in $(seq 1 5) ; do date ; time sha1sum /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso ; done ; echo Total time:)
Sat Oct 31 03:05:02 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso

real    0m3.271s
user    0m2.841s
sys 0m0.418s
Sat Oct 31 03:05:05 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso

real    0m3.266s
user    0m2.853s
sys 0m0.401s
Sat Oct 31 03:05:09 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso

real    0m3.269s
user    0m2.818s
sys 0m0.438s
Sat Oct 31 03:05:12 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso

real    0m3.269s
user    0m2.819s
sys 0m0.436s
Sat Oct 31 03:05:15 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso

real    0m3.282s
user    0m2.803s
sys 0m0.465s

Total time:
real    0m16.364s
user    0m14.135s
sys 0m2.164s

And now a read over NFS:

$ time (for C in $(seq 1 5) ; do date ; time sha1sum /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso ; done)
Sat Oct 31 03:03:33 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso

real    0m11.146s
user    0m3.501s
sys 0m0.894s
Sat Oct 31 03:03:44 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso

real    0m3.279s
user    0m2.858s
sys 0m0.406s
Sat Oct 31 03:03:48 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso

real    0m3.279s
user    0m2.816s
sys 0m0.449s
Sat Oct 31 03:03:51 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso

real    0m3.280s
user    0m2.807s
sys 0m0.459s
Sat Oct 31 03:03:54 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b  /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso

real    0m3.275s
user    0m2.841s
sys 0m0.420s

Total time:
real    0m24.267s
user    0m14.826s
sys 0m2.636s

Reading five times gives 24.267s over NFS, vs 16.364s local.

Heat problems?

I noticed when I went to re-run another test the speeds were decreasing:

$ time (for C in $(seq 1 5) ; do date ; rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ ; done ; echo Total time:)
Sat Oct 31 03:10:55 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  449.69MB/s    0:00:13 (xfr#1, to-chk=0/1)
Sat Oct 31 03:11:47 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  289.55MB/s    0:00:21 (xfr#1, to-chk=0/1)
Sat Oct 31 03:12:25 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  307.83MB/s    0:00:20 (xfr#1, to-chk=0/1)
Sat Oct 31 03:13:19 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  251.13MB/s    0:00:24 (xfr#1, to-chk=0/1)
Sat Oct 31 03:14:06 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  267.18MB/s    0:00:23 (xfr#1, to-chk=0/1)
Total time:

real    3m57.811s
user    0m9.038s
sys 0m25.878s

Admittedly, over the past ~10-15 minutes I've been doing a lot of big transfers and the metal case of the USB NIC is a bit warm, but not ultra hot.

I'll let this cool down and try later.

[Many hours later...]

After letting this sit all day while I was at work, I re-ran the speed tests:

$ time (for C in $(seq 1 5) ; do date ; rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ ; done ; echo Total time:)
Mon Nov  2 06:05:29 PM CST 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  598.01MB/s    0:00:10 (xfr#1, to-chk=0/1)
Mon Nov  2 06:06:15 PM CST 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  503.93MB/s    0:00:12 (xfr#1, to-chk=0/1)
Mon Nov  2 06:07:07 PM CST 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  730.20MB/s    0:00:08 (xfr#1, to-chk=0/1)
Mon Nov  2 06:07:53 PM CST 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  666.13MB/s    0:00:09 (xfr#1, to-chk=0/1)
Mon Nov  2 06:08:42 PM CST 2026
ubuntu-26.04-desktop-amd64.iso
  6,518,974,464 100%  542.73MB/s    0:00:11 (xfr#1, to-chk=0/1)
Total time:

real    4m14.499s
user    0m9.099s
sys 0m31.878s

Intersting - the total time (real 4m14.499s) was more but the MB/s were faster in each case...

USB settings

I came across this writeup (https://kohlschuetter.github.io/blog/posts/2026/04/28/rtl10/) and had some interesting things to check.

  1. Find the USB devices "Bus" and "Device" IDs:
$ sudo lsusb
[...]
Bus 004 Device 002: ID 0bda:815a Realtek Semiconductor Corp. USB 10/100/1G/2.5G/5G/10G LAN
  1. View the details of that specific Bus:Device identifier:
$ sudo sudo lsusb -v -s 004:002
Bus 004 Device 002: ID 0bda:815a Realtek Semiconductor Corp. USB 10/100/1G/2.5G/5G/10G LAN
Negotiated speed: SuperSpeed+ (10Gbps)
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.20
  bDeviceClass            0 [unknown]
  bDeviceSubClass         0 [unknown]
  bDeviceProtocol         0 
  bMaxPacketSize0         9
  idVendor           0x0bda Realtek Semiconductor Corp.
  idProduct          0x815a USB 10/100/1G/2.5G/5G/10G LAN
  bcdDevice           30.00
  iManufacturer           1 Realtek
  iProduct                2 USB 10/100/1G/2.5G/5G/10G LAN
  iSerial                 7 000334C8D6B308AC
  bNumConfigurations      3
[...snip...]

Ah-ha moment...

See the Negotiated speed: line? It reports SuperSpeed+.

In the wacky hodge-podge of USB naming standards, that is a USB 3.2 Gen 2×1 (10Gbps).

To support the full 10Gbs bandwidth of the network adapter over USB, the port must provide more to provide some "room" (overhead) for the protocols. The next step up is 20Gbps, or the USB 3.2 Gen 2×2 standards name.

In the end.

Checking my system BIOS and manua confirm it - that is the best I'll get out of this system. The USB chips in my system only support the USB-C 3.2 Gen 2x1 (10Gbps).

So 5Gbps for me it is. 😅

```