I’m trying to write a server and a client in C++ using io_uring. The “protocol” is simple, send an int32_t, and get an array of that int, but if I increase the package size from 128 to 4096 bytes, I’m starting to get some packets which have one half from one request and another from another request. Here’s a logs snippet:

[0] Iterations received: 120000 [139629 it/s]
[0] Incorrect response at index 125629, value 32161280
[0] Received out-of-range response at index 32140800
First 30 and last 30 bytes: 00 6E EA 01 00 6E EA 01 00 6E EA 01 00 6E EA 01 00 6E EA 01 00 6E EA 01  ... 00 BE EA 01 00 BE EA 01 00 BE EA 01 00 BE EA 01 00 BE EA 01 00 BE EA 01 

And then a bit later the other half shows up:

First 30 and last 30 bytes: 00 D3 EA 01 00 D3 EA 01 00 D3 EA 01 00 D3 EA 01 00 D3 EA 01 00 D3 EA 01  ... 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 
[0] Incorrect response at index 125652, value 125652
First 30 and last 30 bytes: 6E EA 01 00 6E EA 01 00 6E EA 01 00 6E EA 01 00 6E EA 01 00 6E EA 01 00  ... D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 D4 EA 01 00 
[0] Iterations received: 130000 [138256 it/s]

I get that TCP is supposed to be a stream of data, but how can I know which packet is which in these partial/mixed recvs?

The code is quite ordinary, here’s are some relevant parts. How the server sends the reply:

  struct io_uring_sqe* sqe = io_uring_get_sqe(&ring);
  auto* req = new RequestData{WRITE_EVENT};

  io_uring_prep_send(sqe, client_socket, data, PAGE_SIZE * sizeof(int32_t), 0);
  io_uring_sqe_set_data(sqe, req);

How the client reads the reply:

  auto* recv_buffer = new int32_t[PAGE_SIZE];
  struct io_uring_sqe* sqe_recv = io_uring_get_sqe(&ring);
  io_uring_prep_recv(sqe_recv, sock, recv_buffer,
                     PAGE_SIZE * sizeof(int32_t), MSG_WAITALL);
  auto* request_data_recv = new RequestData{RECV_EVENT, recv_buffer};
  io_uring_sqe_set_data(sqe_recv, request_data_recv);

You can see MSG_WAITALL here, it made the transfer more reliable.

Otherwise, with reply size of 4096 bytes, and 1024*1024 messages, I get 1048269 correct responses, and 307 incorrect responses over loopback.

You can see the full code here – https://github.com/sssemil/fast_net. The relevant files are (depends only on liburing, spdlog is not used in these):

• https://github.com/sssemil/fast_net/blob/main/src/simple_iou_client.cpp
• https://github.com/sssemil/fast_net/blob/main/src/simple_iou_server.cpp
• https://github.com/sssemil/fast_net/blob/main/src/simple_consts.hpp