I’m encountering a segmentation fault while training a text recognition model using the PP-OCRv3 configuration in PaddlePaddle. During my first attempt, I ran into an out-of-memory error on the GPU, so I reduced the batch_size_per_card from 128 to 64. However, after this adjustment, the training process initiates but fails almost immediately with a segmentation fault.

Here’s the exact error message:

--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
No stack trace in paddle, may be caused by external reasons.

----------------------
Error Message Summary:
----------------------
FatalError: `Segmentation fault` is detected by the operating system.
  [TimeInfo: *** Aborted at 1726071412 (unix time) try "date -d @1726071412" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x0) received by PID 1421259 (TID 0x731b08f4c000) from PID 0 ***]

LAUNCH INFO 2024-09-11 18:16:56,339 Exit code -11

The segmentation fault occurs almost immediately after the first iteration begins, despite the pretrained model loading successfully and the data loaders being set up correctly.

I’m using PaddlePaddle version 2.6.1 with GPU support, CUDA version 11.8, a GPU with Compute Capability of 8.6, and cuDNN version 9.4, model en_PP-OCRv3_rec, os Ubuntu 22.04.

I’m looking for potential causes and solutions for this issue, especially regarding whether the batch size adjustment might be contributing to this error. Any insights would be greatly appreciated.

Here’s a summary of what I’ve tried:

1. Initial Configuration Adjustment:

   • Reduced the batch_size_per_card from 128 to 64 to address the out-of-memory error.

2. Additional Parameter Modifications:

   • Adjusted various other parameters in the configuration file to further reduce memory usage.

3. Using a Different Dataset:

   • Switched to a different dataset to determine if the issue was related to the original dataset.

Despite these attempts, the segmentation fault error persisted. The problem seems to go beyond memory allocation and dataset specifics.

Has anyone encountered a similar issue with PaddlePaddle? What potential causes could be contributing to this error, and are there any recommended solutions or debugging steps I can take to resolve the segmentation fault? Any advice on how to better diagnose or fix this issue would be greatly appreciated.