We are developing a Linux host PCIe driver for an external PCIe card(device) to perform DMA operations. DMA controller is present on the PCIe board. We need to pass the DMA address to the board via MBOX (some registers mechanism) and then DMA controller will perform DMA.

Linux HOST machine <====> PCIe interface <===> External PCIe card

Below steps are involved to perform DMA operation:

1. Pass a user space buffer to driver via IOCTL. This buffer contains the actual data(for DMA write) and location(for DMA read).
2. Convert this user buffer into SG list.
3. Map the SG list for DMA
4. Pass the DMA address to the board using MBOX push mechanism.

Below standard Linux APIs are used in implementation:

Buffer is passed from user space and copied in kernel using copy_from_user.

descriptor = dma_alloc_coherent(&pdev->dev, sizeof(struct dma_descriptor), &dma_handle, GFP_KERNEL);
start = (unsigned long)buffer;
len = do_dma->length;
offset = start & ~PAGE_MASK;
num_pages = (offset + len + PAGE_SIZE - 1) / PAGE_SIZE;
pages = kvmalloc_array(num_pages, sizeof(struct page *), GFP_KERNEL);
pin_user_pages(start, num_pages, FOLL_WRITE | FOLL_LONGTERM | FOLL_FORCE, pages);
sgt = kmalloc(sizeof(*sgt), GFP_KERNEL);
sg_alloc_table_from_pages(sgt, pages, num_pages, offset, length, GFP_KERNEL);
dma_map_sgtable(&rtc_dev->pdev->dev, sgt, DMA_BIDIRECTIONAL, 0);

SG list elements are then copied to struct dma_descriptor

With above implementation, DMA read flow is working fine. We could see the message on Linux host side which is sent by the board.
But our DMA write flow is not working. After the data is passed to the board, the board is not receiving DMA done interrupt from the DMA controller.
After reading DMA controller’s control and status register on board, we came to know that DMA controller is busy in DMA operation but never completes it.

We are suspecting the issue is with the permissions of the pages created by pin_user_pages. But we have tried all flag like FOLL_WRITE | FOLL_FORCE | FOLL_LONGTERM. Unable to find the root cause of this failure. Board software is correct, because same board is working fine when interfaced with Windows setup.