I’m currently running a parameter search which is going to take a few days, I’m using hyperopt and recording the results using mlflow.
The server is going to be restarted automatically (oversight on my part) and for various reasons can’t be delayed. This would also be useful to know if it crashes.
I didn’t dump the hyperopt trials (lesson learned for next time)
Is it possible to use the MLFlow records stored locally to rebuild the trials object and restart the space search from where it is going to be left off.
I’ve logged all the parameters and the results into MLFlow. I think I can do it manually – read the results and run dummy “experiments” which just return the values then use that trial object to do the rest using the real thing. Is there some formal/quick way of doing this I haven’t been able to find (I’ve looked but I’m not sure I’m finding the right search terms).