# Recovery

## `allow_spot`

Spot machines are cheaper and can be taken back by the provider at any time.
Set `false` to use on-demand machines only.

## `checkpoint_every`

How often your job saves a checkpoint, in hours. RunCompute uses it for two
things:

- **Choosing a machine.** Expected lost work is `(1 − reliability) ×
  checkpoint_every`, priced into each offer's estimate.
- **Resuming.** After a preemption the job restarts from the last checkpoint on
  the next machine in the ranking, excluding the one that was just lost.

Your code has to save and load the checkpoints; see
[training and fine-tuning](/docs/guides/training/).

## What happens on a preemption

| Step | Event |
| --- | --- |
| Provider reclaims the machine | `preempt`, with how much work since the last checkpoint was lost |
| Status becomes `recovering` | — |
| Next machine chosen | `match` |
| Job starts from the checkpoint | `resume` |

Spend from every attempt counts toward the budget.