# Training and fine-tuning

Long jobs are where spot machines save the most money, and where losing a
machine hurts the most. Two things make a long job safe to run on RunCompute:
a checkpoint your code can resume from, and a budget.

## Save and resume checkpoints

RunCompute does not see inside your process. When a machine is lost, the job is
started again on another machine with the same command. Your code decides what
to resume from.

Write checkpoints under `/outputs/checkpoints/` and load the newest one at
start:

```python
import glob, os, torch

CKPT_DIR = "/outputs/checkpoints"
os.makedirs(CKPT_DIR, exist_ok=True)

start_step = 0
existing = sorted(glob.glob(f"{CKPT_DIR}/step-*.pt"))
if existing:
    state = torch.load(existing[-1])
    model.load_state_dict(state["model"])
    optimizer.load_state_dict(state["optimizer"])
    start_step = state["step"] + 1

for step in range(start_step, total_steps):
    train_step()
    if step % 500 == 0:
        torch.save({"model": model.state_dict(), "optimizer": optimizer.state_dict(), "step": step},
                   f"{CKPT_DIR}/step-{step:07d}.pt")
```

Hugging Face `Trainer` and PyTorch Lightning have their own options for saving
and resuming; point their output directory at `/outputs`.

## Match the checkpoint interval

`checkpoint_every` tells the matcher how often you save, in hours. It is used to
estimate how much work a preemption would throw away:

- Save often (`0.25`) and cheap spot machines usually win.
- Save rarely (`2`) and the matcher leans toward reliable on-demand machines.

## Pick a budget

A rough budget is `est_cost × 1.3` from a quote. If the job reaches the budget
it stops with status `stopped`, keeps its checkpoints, and can be submitted
again with a higher budget.

```python
offers = client.quote(min_vram=40, work_units=12, checkpoint_every=0.5)
budget = round(offers[0]["est_cost"] * 1.3, 2)
```