After setting up the RDMA interfaces for the first time and trying to test the connection from two different machines, the test just hangs and timeouts from the server side.
Prior to this, I have ran tests locally (both client and server are localhost) and the expected output is generated. The same problem also occurs to other connectivity tests (rping
and ping_pong
)
Server side output:
jun@jun-Latitude-5430:~/test/async-rdma$ ib_send_bw
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : rx0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
RX depth : 512
CQ Moderation : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0077 PSN 0x4af244
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:83
remote address: LID 0000 QPN 0x0026 PSN 0x819b5b
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:02:15
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
Did not get Message for 120 Seconds, exiting..
Total Received=0, Total Iters Required=1000
And here is the output from the client side.
root@jng:/home/jng/async-rdma# ib_send_bw 192.168.100.83
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : rx0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
TX depth : 128
CQ Moderation : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0026 PSN 0x819b5b
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:02:15
remote address: LID 0000 QPN 0x0077 PSN 0x4af244
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:83
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
I also used a packet sniffer and discovered that I get no reply from the client side (192.169.100.83) and the server is constantly sending out RC packets. This also occurs when I switch the server and client machines.
The two machines are Debian 5.0 and Ubuntu 22.04 respectively. The Debian machine is a virtual machine in virtualbox.