Treat configuration as the test unit
Different models can serve different application tasks, but each configuration needs its own evidence. Record the exact model ID, protocol, client version, important parameters, test date, and intended purpose. A model directory helps identify candidates; it does not prove every capability has passed your tests.
Separate release requirements from optional improvements. If the application requires structured output or tool calls, define explicit acceptance criteria for those features. Without evidence, mark them unverified rather than inferring support from a name or successful plain-text response.
Attach evidence to each capability
Use fixed, non-sensitive samples and expected outcomes. Humans can review answer quality, while parsing, field types, and boundary conditions benefit from deterministic validation. Record pass, fail, or untested and link to redacted application test records.
| Dimension | Example acceptance criterion |
|---|---|
| Task quality | Representative samples meet team-defined requirements |
| Structured output | Parses successfully and matches application field rules |
| Tool calls | Arguments, result handling, and termination are correct |
| Streaming | Assembly, termination, and cancellation work |
| Operating limits | Timeouts, rate limits, context, and spending are observable |
Retest when something changes
Run a bounded pilot at expected input lengths and modest concurrency. Observe usage, latency, and error categories. Do not extrapolate one success to the largest context or long-term reliability, and confirm limits before attempting load tests.
Rerun affected samples after changing a model, upgrading an SDK, or modifying prompts. Expand through a controlled set of tasks and keep a tested route back to the previous configuration. Verify that fallback inputs, outputs, and tool protocols still satisfy application expectations.
Sources & further reading
An owned publication maintained by NexoToken. Product references reflect the publisher’s commercial interest and are not independent certification.