Recovery#
allow_spot#
Spot machines are cheaper and can be taken back by the provider at any time.
Set false to use on-demand machines only.
checkpoint_every#
How often your job saves a checkpoint, in hours. RunCompute uses it for two things:
- Choosing a machine. Expected lost work is
(1 − reliability) × checkpoint_every, priced into each offer's estimate. - Resuming. After a preemption the job restarts from the last checkpoint on the next machine in the ranking, excluding the one that was just lost.
Your code has to save and load the checkpoints; see training and fine-tuning.
What happens on a preemption#
| Step | Event |
|---|---|
| Provider reclaims the machine | preempt, with how much work since the last checkpoint was lost |
Status becomes recovering |
— |
| Next machine chosen | match |
| Job starts from the checkpoint | resume |
Spend from every attempt counts toward the budget.