Cache not downloading at full potential even with multiple IPs

[ghost]

Issue Description:
Following the instructions listed on this PR I have successfully managed to get my steam cache to run on 30 total IPs, however that does not seem to be working. My maximum network bandwidth is 8MB/s. With the 30 IPs I never see more than 4MB/s on uncached games. I know 30 is a bit excessive, I want to make sure not enough wasn't the issue first. When downloading on the client, in Resource Manager I do see all 30 connections downloading at least a little, they are working, just not at full speed.

Docker Run Command:

docker run --restart unless-stopped --name lancache-dns --detach -p 192.168.0.13:53:53/udp -e USE_GENERIC_CACHE=true -e UPSTREAM_DNS=8.8.8.8 -e LANCACHE_IP="192.168.0.13 192.168.0.102 192.168.0.103 192.168.0.104 192.168.0.105 192.168.0.106 192.168.0.107 192.168.0.108 192.168.0.109 192.168.0.110 192.168.0.111 192.168.0.112 192.168.0.113 192.168.0.114 192.168.0.115 192.168.0.116 192.168.0.117 192.168.0.118 192.168.0.119 192.168.0.120 192.168.0.121 192.168.0.122 192.168.0.123 192.168.0.124 192.168.0.125 192.168.0.126 192.168.0.127 192.168.0.128 192.168.0.129 192.168.0.130 192.168.0.131" lancachenet/lancache-dns:latest
docker run --restart unless-stopped --name lancache --detach -v /mnt/hdd:/data/cache -v /mnt/hdd:/data/logs -p 80:80 lancachenet/monolithic:latest
docker run --restart unless-stopped --name sniproxy --detach -p 443:443 lancachenet/sniproxy:latest

Cache system specs:
CPU: Ryzen 5 3400G
Motherboard: ASRock B450M/AC
RAM: 4x16GB G.SKILL Ripjaws
Cache Drive: 12TB Western Digital Red Pro
NIC: Onboard gigabit ethernet

Any help is appreciated.

[astrolox]

Honestly more IP addresses will never be your problem. A single IP will be able to achieve much faster throughput that you'll ever be able to utilise.

To solve this you will need to try and find the bottleneck in your configuration.

Network
DISK I/O
RAM
CPU

are the most likely causes - in that order.

Find a way to monitor each of these on both your lancache server and on the computer doing the download. Then install and uninstall the same game a few times. Whichever of these is the flattest and/or highest is probably the cause.

[astrolox]

After further investigation I have found that we have previous evidence of a limit of around 40 Mbps per IP.

[ghost]

I've ran some tests on the potential culprits you suggested, here are the results:

Cache system:
Network: 53-65Mbit/s down 19-20Mbit/s up (for mid-us to east-us respectively)
DISK I/O: , drive was limited to gigabit speeds (reached 250MB+/s when empty)
RAM: There is 64GBs of RAM most of which is unused, a bit difficult to test RAM speeds on Ubuntu but I'd imagine it's not RAM unless it's not using it properly, Linux likes to use it as a cache.
CPU: The CPU does not wait much for the drive and there is little usage when downloading, as you would expect, downloads don't take much.

No matter the game, it downloads painfully slow. I moved the cache so it would re-cache any downloads. I tried my smaller games ranging in size from 240MB to 2.6GB, they were downloading any where from 3.5KB/s to 1.8MB/s, jumping around frequently. During the download the cache hard drive I/O saw uses from < 2-3%, it's not doing much because it's not downloading much.

After the games were cached, the smaller ones downloaded instantly, peaking at 47.5MB/s. The bigger games dropped as low as 1MB/s, drive might as well have been idle at that point. Then spiking back up to 40MB/s occasionally.

Because the download is all over the place, I suspect someone is going to say "Oh well, sounds your drive is bad". No. Simply is not the case, to prove it here are the same games on the boot SSD (Samsung 970 EVO 500GB) Here is a screenshot of the first game starting to download, this is going to take time I don't have but I'll update as they finish.

They all just finished, 4.4GBs of games took 208 minutes (9:30 PM to 12:58 AM) peaking at 2.8MB/s no where near the maximum steam can achieve without the cache.

Here are the downloads from the cached games, off the SSD.

Dino D-Day: 
  Start: 1:06 AM
  Finish: 1:11 AM
  Peak: 32.6MB/s
  Size: 2.6GB
  
Terraria:
  Start: 1:13 AM
  Finish: 1:16 AM
  Peak: 12.6MB/s
  Size: 176MB
  
SUPERHOT:
  Start: 1:16 AM
  Finish: 1:18 AM
  Peak: 20.6MB/s
  Size: 936.2MB  
  
Starbound:
  Start: 1:19 AM
  Finish: 1:22 AM
  Peak: 16.3MB/s
  Size: 755.7MB

Still pretty slow, though I'm pretty sure steam has decided to serve compressed games no matter what the dev uploads.

[markkundinger]

hey just chiming in with a data point on this old topic that this isn't an obvious global performance problem.

I will say that, in general, caching on a HDD can stink for performance. The I/O is highly random which spinners have a hard time with. Store the cache on a SSD for more faster.

I can get about 47MB/s peak (around 400+mbps) on an uncached game.
once it's cached it's way faster, over 291MBs (2.2gbps+) when I tested on a 10g network.

One other thing is that installing the game on steam can sometimes be limited by the client's CPU speed (steam unzipping the files?). At least when it's cached, my Ryzen 1600AF is maxed out at 100% for brief portions