I run background jobs for different users in our service. Subtasks and runtime is different based on the size and setting of the user account.

Subtasks within the job are something like: downloading data, processing data, caching data, refreshing parts of data, downloading other data if some condition happen, write data, etc. It can run for 30 seconds or 10 hours. Same subtask for different users varies by time too depending on settings.

I want to be able to communicate the progress of the job to the user and the estimated remaning time.

My ideas is a Python module which sets ‘checkpoints’ in the code using a number of items the subtask is estimated to process. After the first run of the job all checkpoints are known and this way the estimation can be made more accurate. How I see it in practice:

```python

start_checkpoint()

# ... background job logic....

new_checkpoint('download data 1', 5)   # Expected 5 REST API calls
data = download('list_of_numbers.json?n=5000&pagesize=1000')

new_checkpoint('process data', len(data))   # process each data item
processed = [i*i for i in data if i % 2 == 0]

new_checkpoint('save data', len(processed))   # store all even squares
save(processed)

# ... even more background job logic....

finished_all_checkpoint()

```

The idea is that if you run the script for the second time, you have an idea which checkpoints are coming for this user and the duration of each checkpoint with an average runtime per item. Then you could use that data for extrapolating the duration for the new number of items. By summing up the checkpoint durations you can have an idea of total runtime and an estimated remaining time of the whole job.

I looked around but I could not find anything for this purpose. Only function timing / profiling wrappers.

An idea what module is there for this purpose? Maybe used in a different way?

PS. I need something like this for PHP and Python.