How to programmatically set alerts for failure in a W&B run?
I’m trying to set up alerts in my W&B (Weights & Biases) project to notify me if a run fails. I’ve been testing several functions I thought would work based on my research, but none seem to be implemented in the W&B API. Here’s the code snippet where I attempt to set up these notifications:
import wandb
mode = 'dryrun'
run_name = 'my_run'
num_batches = 50
path = '/data'
name = 'experiment1'
today = '2023-08-01'
probabilities = [0.1, 0.9]
batch_size = 32
data_mixture_name = 'mix1'
debug = mode == 'dryrun'
run = wandb.init(mode=mode, project="beyond-scale", name=run_name, save_code=True)
wandb.config.update({
"num_batches": num_batches,
"path": path,
"name": name,
"today": today,
'probabilities': probabilities,
'batch_size': batch_size,
'debug': debug,
'data_mixture_name': data_mixture_name
})
# Attempts to set notifications
run.notify_on_failure()
run.notify_on_crash()
run.notify_on_exit()
run.notify_on_heartbeat()
run.notify_on_abort()
Each of these attempts results in an AttributeError
, stating that the ‘Run’ object has no such attribute. For example:
AttributeError: 'Run' object has no attribute 'notify_on_failure'
Is there a correct method to set up failure or other alerts in W&B? If so, how should I modify my approach?
ref: https://community.wandb.ai/t/how-do-i-set-the-wandb-alert-programatically-for-my-current-run/4891