Picking a Model
Model choice moves cost and latency more than prompt tweaks do. Match the model to the job, then measure.
| Job | Reach for | Watch out for |
|---|---|---|
| High-volume chat / classification | A small or "flash"-class model | Quality cliffs on long inputs |
| Reasoning and maths | A reasoning-tuned model | Higher latency and token cost |
| Code generation and refactors | A code-strong model | Context window limits on big repos |
| Document and PDF analysis | A long-context model | Cost scales with input tokens |
| Screenshots and diagrams | A vision-capable model | Image payload limits |
| Tool / function calling | A model with tools enabled | Strict schema validation |
Keep a fallback
Model ids are attached to specific model versions. On a 404 model_not_found, drop to a secondary id from your own shortlist rather than failing the user request — then alert yourself so the fallback does not become permanent.
Measure before optimising
- Read Usage & Analytics for requests, tokens and spend per model.
- Compare latency in the playground before committing to a swap in production.
- Trim the prompt first: input tokens are usually the larger cost line for chat workloads.