I am developing a whiteboard app in rust using fastwebsockets library. I want my app to handle ~10000 active users: 500 of them are supposed to constantly send data to websocket, which is broadcasted to other users in the room.

I’ve written a load test with 500 rooms where 500 clients send 120 bytes to websocket every 3 seconds. At the beginning users send and receive messages normally, but after a few minutes sending takes forever and continues only if I disable the test. In the current implementation I try to reconnect the client if sending requires more than 5 seconds, but clients just can’t connect anymore.

I tried using this sysctl config, but dropped cause it did’t help:

fs.file-max = 1000000
fs.nr_open = 1000000
net.core.somaxconn=1000000
net.core.netdev_max_backlog=1000000
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.nf_conntrack_max=1048576 
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.ip_forward=1
net.ipv4.conf.ens3.forwarding=1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_tw_reuse=1

My specs: ryzen 5500 + 16GB RAM

lspci | grep -E -i --color 'network|ethernet'
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15)

ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       63378
-n: file descriptors                100000
-l: locked-in-memory size (kbytes)  8192
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 63378
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15: rt cpu time (microseconds)   unlimited

I tried using various sysctl configs, increasing size of the message coming from server to 4kb, running test with 1000 users(it helped a little bit, but eventually app dropped clients and couldn’t connect them again).