I’m writing a simple network packet mangler in C++ which only changes the TTL value (for now). The main loop is receiving requests from the netfilter queue using recv(fd, buf, sizeof(buf), 0) and then processes those requests using nfq_handle_packet(h, buf, rv). The buffer size is 500000000 bytes, which is about 500 megabytes or 4000 megabits. The internet speed is nowhere close to 4 gigabits (and even if it would be 4 gigabits, I’m not transferring that much data), yet the recv function keeps on receiving the ENOBUFS (“No buffer space available” when formatted using strerror) when I, for example, run a speed test on speed.cloudflare.com. I don’t understand how I’m running out of buffer space here. The code that receives the requests is:

// created before in the code: char buf[500000000] = {0};
fd = nfq_fd(h);
while (true) {
    rv = recv(fd, buf, sizeof(buf), 0);
    if (rv < 0) {
        std::cout << "Error in recv: " << strerror(errno) << std::endl;
    } else nfq_handle_packet(h, buf, rv);
}

The code that processes the packets is also very simple, I don’t think the reason behind the “No buffer space left” error is slow processing of packets. As I said before, it just changes the TTL (time-to-live) value.

static int cb(struct nfq_q_handle *qh, struct nfgenmsg *nfmsg, struct nfq_data *nfa, void *_data) {
    struct nfqnl_msg_packet_hdr *ph;
    ph = nfq_get_msg_packet_hdr(nfa);
    if (!ph) {
        data->packetsRejected += 1;
        return -1;
    }
    int id = ntohl(ph->packet_id);
    unsigned char* packetData;
    int length = nfq_get_payload(nfa, &packetData);
    if (length == -1) {
        int result = nfq_set_verdict(qh, id, NF_DROP, 0, NULL);
        data->packetsRejected += 1;
        return result;
    }
    uint8_t ipVersion = (packetData[0] & 0b11110000) >> 4;
    if (ipVersion == 4) {
        uint8_t headerLength = packetData[0] & 0b1111;
        if (headerLength < 5 || headerLength > 15) {
            int result = nfq_set_verdict(qh, id, NF_ACCEPT, 0, NULL);
            data->packetsRejected += 1;
            data->bytesRejected += length;
            return result;
        }
        headerLength *= 4;
        packetData[8] = 64;
        fixIPv4Checksum(packetData, headerLength);
    }
    int result = nfq_set_verdict(qh, id, NF_ACCEPT, length, packetData);
    data->packetsAccepted += 1;
    data->bytesAccepted += length;
    return result;
}