I’m trying to write a download manager with the gallery_dl library in python, but I’m having a issue with multiprocessing(for managing multiple download tasks in a web-ui inerface).

When I try to create a new job in the __init__ method of the DownloadTask class, I get a self = reduction.pickle.load(from_parent) ... EOFError: Ran out of input error.

I’m guessing this is because multiprocessing is using pickle to serialize the stuff in the child processes, and pickle can’t handle the gallery_dl instance, but I have no idea what to do about this.

Is there a way to fix or get around this?

Here’s a simplfied version of my code:

# DownloadManager.py
from typing import Dict
from DownloadTask import DownloadTask
import uuid
import multiprocessing

class DownloadManager:
    def __init__(self):
        self.tasks: Dict[str, DownloadTask] = {}
        self.processes: Dict[str, multiprocessing.Process] = {}

    def add_task(self, url: str) -> str:
        task_id = str(uuid.uuid4())
        new_task = DownloadTask(task_id, url)
        self.tasks[task_id] = new_task

        task_process = multiprocessing.Process(target=new_task.start)
        self.processes[task_id] = task_process
        task_process.start()

        return task_id

if __name__ == '__main__':
    downloader = DownloadManager()
    downloader.add_task("https://lexica.art/?q=cats")

# DownloadTask.py
from gallery_dl import job

class DownloadTask:
    def __init__(self, task_id: str, url: str):
        self.task_id = task_id
        self.url = url
        self.job = job.DownloadJob(url)

    def start(self):
        self._download()

    def _download(self):
        self.job.run()

I tried to create the job directly in the DownloadManager class on the DownloadTask instance, but the same error still occurs.