I have an FPGA digitizer connected via FTDI FT4232 USB-UART converter to my Windows 10 computer. The communication speed is 5 Mega Bauds, 8N1. I am trying to read 1048576 bytes of data which is sent as a response to a simple text command. Data is protected by a CRC32 checksum (transferred using another command).

Everything works quite well with RealTerm terminal emulator, and now I need to do the same thing with my C#.NET application. Having read some real experience with the SerialPort class, I currently have the code below. The approach is simple: There is a dedicated reader thread which reads data from the serial port and pushes it to a producer-consumer queue, from which data is read by another thread.

_PicoblazeReadBuffer = new ConcurrentQueue<byte[]>();
_Port = new SerialPort("COM12", 5000000, Parity.None, 8, StopBits.One);
_PicoblazeReaderThreadCanceller = new CancellationTokenSource();
_PicoblazeReaderThread = new Thread(ReaderThread) {
    Priority = ThreadPriority.Highest,
    IsBackground = true,
};
_PicoblazeReaderThread.Start(_PicoblazeReaderThreadCanceller.Token);

...

private void ReaderThread(object cancellationTokenAsObject)
{
    byte[] buffer = new byte[131072];
    int bufferOffset = 0;
    CancellationToken token = (CancellationToken)cancellationTokenAsObject; 

    Stopwatch stopwatch = new Stopwatch();
    while (_Port.IsOpen && !token.IsCancellationRequested) {
        try {
            int readLength = Math.Min(4096, buffer.Length - bufferOffset);
            int actualLength = _Port.BaseStream.Read(buffer, bufferOffset, readLength);
            if (actualLength > 0) {
                bufferOffset += actualLength;
                if (bufferOffset >= 65535 || stopwatch.ElapsedMilliseconds > 100) {
                    byte[] chunk = new byte[bufferOffset];
                    Buffer.BlockCopy(buffer, 0, chunk, 0, bufferOffset);
                    _PicoblazeReadBuffer.Enqueue(chunk);
                    _PicoblazeReadSemaphore.Release();
                    bufferOffset = 0;
                    stopwatch.Restart();
                }
            }
        } catch (Exception ex) {
            ...
        }
    }
}

The problem: Some data get lost during the transfer, and therefore my consumer thread times out. When I tried to save the intermediate result, it seems the data always get trimmed at the end, but the beginning is valid.

• When I increase the Latency Timer from default 16 ms to at least 60 ms, the reading completes successfully. I cannot understand, why this happens, because if I understand it correctly, the latency timer should affect performance only when I send short data packets.
• Although I want to read 4096 bytes (default driver buffer size), the function completes earlier and returns only 250 bytes on average. Therefore, I added a simple data packing condition, which reduces the overhead related to the queue locking. This issue might be connected to the event characters, which forces the chip to flush out the buffers, but the problem persists even if I send all-zero pattern from the FPGA.

Do you have any experience with high baud-rate data transfers in C#?