When “It Works in the Lab” Isn’t Enough: QA for AI Features That Orchestrate Real Systems
A common QA failure mode for AI-enabled products is assuming that strong offline evaluation implies safe production behavior. Benchmark-style checks (even well-designed ones) tend to confirm that a model can answer questions or generate code under controlled conditions. Production systems, however, ask something different: can a probabilistic component reliably operate inside a distributed application—calling tools, consuming retrieved content, handling timeouts, and responding to messy user input—without creating operational incidents or silent correctness issues?
This article focuses on a concrete QA problem: an AI assistant that combines retrieval-augmented generation (RAG) and tool execution (APIs, workflows, database writes). The recommendations apply broadly to similar copilots and agentic flows.
Define the Real Quality Risk: “Valid Response” vs “Correct System Behavior”
Traditional test automation often treats “no crash + HTTP 200” as success. AI features can fail while still returning syntactically valid output:
- A response can be fluent but wrong, outdated, or non-compliant.
- A tool call can be nearly correct but violate a schema in one field.
- A workflow can retry after a timeout in ways that duplicate side effects.
QA should start by rewriting the definition of “pass” for AI paths:
- Safety/compliance pass: the system avoids prohibited content and follows policy.
- Grounding pass: responses are supported by the provided context (when required).
- Integration pass: structured outputs conform to contracts; side effects are controlled.
- Resilience pass: failures in dependencies lead to deterministic, recoverable outcomes.
- Observability pass: the system emits enough telemetry to detect silent degradation.
These become explicit acceptance criteria, not implicit hopes.
Build a Two-Layer Assertion Strategy (Deterministic + Probabilistic)
AI outputs mix deterministic and non-deterministic elements. A single test style is either too strict (breaks on harmless wording changes) or too loose (lets serious defects slide). Split your checks:
Layer 1: Strict, machine-verifiable invariants
Use hard assertions for anything the system must never violate:
- JSON/tool payload schema (types, required fields, allowed enums)
- Escaping/quoting rules
- Maximum lengths
- Idempotency keys for write operations
- Retry budgets and backoff behavior
- “Must refuse” patterns when context is insufficient (if your product design requires it)
These checks should behave like classic API contract tests: deterministic, reproducible, and blocking in CI.
Layer 2: Semantic/behavioral evaluation
For the parts that are inherently language-driven, use evaluations that tolerate wording variation but still measure intent:
- Did the answer use retrieved context appropriately (where required)?
- Did it remain consistent with policy?
- Did it ask a clarifying question when the input was ambiguous?
- Did it avoid fabricating specifics when context was contradictory?
This layer can be implemented with rubrics, targeted heuristics, and—when appropriate—secondary model grading. The key QA decision is what “good” means operationally, not whether the prose matches a golden string.
Replace Generic Benchmarks with a Versioned “Golden Set” of Edge Cases
Public benchmarks can be useful signals about general capability, but they rarely reflect your product’s actual failure modes. A practical next step is to create a version-controlled golden dataset built from your own domain scenarios and integration edges.
What to include:
- Typical user intents (happy paths)
- Ambiguous requests that require clarification
- Policy-sensitive requests (refunds, access, permissions, HR/legal-style topics—whatever applies)
- Long, messy inputs (pasted logs, multi-part questions, typos)
- Multi-turn conversations with state carried forward
- Retrieval conflicts (two sources disagree; one is outdated)
- Tool invocation cases with optional parameters, nulls, and boundary values
How to manage it:
- Store prompts, retrieved context snapshots (or references + stable fixtures), expected tool calls, and rubric expectations in git.
- Tag each case by risk category (compliance, tool schema, retrieval grounding, etc.).
- When you fix a defect, add the triggering case to the golden set so it becomes regression coverage.
Hypothetical example (labeled):
A support assistant retrieves two internal policy snippets: one says “returns within 14 days,” another says “returns within 30 days for premium accounts,” but the customer’s account type is unknown. A golden test asserts the assistant must ask for account type or present both possibilities with conditions—rather than stating a single definitive policy.
Test the Whole Pipeline, Not Just the Model
Many production failures arise from interfaces around the model. QA should treat the AI feature as a pipeline:
- Input handling (sanitization, truncation, conversation state)
- Retrieval (chunking, ranking, deduplication, freshness)
- Prompt assembly (system instructions, context injection)
- Generation
- Tool planning and execution
- Post-processing (validation, redaction, formatting)
- Logging/telemetry
Create integration tests that run through this full path with controlled fixtures. If your staging environment differs from production (different indexes, different policies, different tool endpoints), your tests may certify the wrong system.
Add Chaos and Adversarial Testing Before Launch
AI features should be tested under failure, not only under ideal conditions. Introduce deliberate “bad weather” to validate resilience and guardrails.
Focus areas:
- Tool/API failures: simulate 500/502/504 responses, slow responses, malformed responses.
- Timeouts and retries: confirm bounded retries, exponential backoff, and no runaway loops.
- Side-effect control: ensure write operations are idempotent or gated; validate deduplication.
- Retrieval chaos: inject empty context, contradictory context, irrelevant context, and duplicated chunks.
- Prompt-injection style inputs: user messages and retrieved documents containing instructions that attempt to override system rules.
Hypothetical example (labeled):
A workflow creates a ticket via API. Under a simulated timeout, the orchestrator retries. QA asserts the second attempt includes the same idempotency key and does not create a duplicate ticket. If the API returns “unknown status,” the system should escalate rather than repeating indefinitely.
Establish Release Gates That Reflect Operational Risk
Treat AI changes like production changes, not experiments that “should be fine.” Add evaluation gates to your delivery pipeline, for example:
- Contract tests for tool schemas must pass 100%.
- Golden dataset must meet minimum rubric thresholds for grounding and policy adherence.
- No new critical regressions in high-risk categories (payments, permissions, compliance).
- Known failure modes must have mitigations (fallback responses, escalation paths).
A key decision is to separate “ship-blocking” criteria from “monitor-only” criteria. Some semantic improvements can be monitored post-release; schema and side-effect safety usually cannot.
Plan for Drift: Post-Launch Monitoring That Captures “Silent Failures”
Even if the model weights do not change, production inputs and upstream data do. Retrieval indexes evolve, documentation is updated, and user behavior shifts. QA should extend into production with monitoring designed for AI-specific degradation:
- Capture execution traces: user input, retrieved chunk identifiers (or content hashes), tool arguments, tool results, final output, and validation outcomes.
- Watch for distribution shifts in inputs (length, language, topics) and in tool usage (retry rates, error codes).
- Track grounding indicators (e.g., how often the system answers without sufficient context when context is required by design).
- Alert on spikes in “fallback invoked,” “validation failed,” or “tool schema rejected.”
This is less about perfect metrics and more about building a feedback loop where suspicious traces become new regression tests.
Use Shadow Mode for Risky Changes
When changing prompts, retrieval configuration, index content, or model versions, consider a shadow deployment approach:
- Mirror real requests to the candidate system
- Disable write operations (or route them to a sandbox)
- Compare tool-call structures, validation failures, and rubric outcomes against the current baseline
Shadow testing helps you evaluate behavior on authentic traffic patterns without exposing users to regressions.
Next Steps Checklist
If you need a concrete starting plan, sequence it:
- Map the pipeline (retrieval → prompt → generation → tools → post-processing) and list failure modes.
- Define two-layer assertions: strict invariants + semantic rubrics.
- Create a first golden set of 50–200 cases dominated by edge conditions and high-risk scenarios.
- Add chaos tests for tool failures, timeouts, and retrieval noise.
- Implement release gates that block on schema/side-effect safety and key compliance rules.
- Instrument traces in production and route flagged cases back into regression.
- Adopt shadow testing for prompt/index/model changes.
Reliable AI features emerge when QA treats the model as a probabilistic component embedded in real software, and tests the system behavior end-to-end—before and after launch.
