I tried to iterate through a directory in Linux system. The program is run on a chip, so OS memory is limited in 25MB. The directory only contains images, each with size about 3MB. What program does is read each image in the file and do some process. But program failed when reading about the 4000th image, and I am not able to find out the reason.

first try with scandir()

I used scandir() function, the code looks like this.

void MyFunc(const std::string &fileDir)
{
    struct dirent **entry_list;
    int32_t count = scandir(fileDir.c_str(), &entry_list, nullptr, alphasort);
    if (count == -1) {
        printf("open dir:%s failed.n", fileDir.c_str());
        return;
    }

    for (int32_t i = 0; i < count; i++) {
        struct dirent *ptr = entry_list[i];
        if (strcmp(ptr->d_name, ".") == 0 || strcmp(ptr->d_name, "..") == 0) {
            continue;
        }

        // read image
        // do process
        // release image
    }
}

This code worked fine until I met a large directory with 30,000+ images in it. So I thought scandir() will go through the entire directory and cause the program collapse.

second try with readdir()

With previous experience, I tried readdir() this time.

void MyFunc(const std::string &fileDir)
{
    DIR *dir = opendir(fileDir.c_str());
    if (dir == nullptr) {
        printf("open dir:%s failed.n", fileDir.c_str());
        return;
    }
    ScopeGuard dirGuard([&] { (void)closedir(dir); }); //RAII

    struct dirent *entry;
    while ((entry = readdir(dir)) != nullptr) {
        if (strcmp(ptr->d_name, ".") == 0 || strcmp(ptr->d_name, "..") == 0) {
            continue;
        }

        // read image, the same scandir verison
        // do process, the same scandir verison
        // release image, the same scandir verison
    }
}

Surprisingly, this time the program will report OOM error when reading approximately the 4000th image, while scandir() version fails when directory has about 10,000+ images in it.

So what may cause the collapse of readdir version? Is entry acquired one by one?

I did closedir() in the code. The code is remedied.