Overview
Anbosoft combines software quality assurance with AI agent testing and evaluation. We start with the task your agent needs to complete, the tools and data it can access, and the boundaries it must respect. For support agents, this can include answering knowledge-base questions, retrieving account information, and creating support tickets.
We define the testing scope and acceptance criteria with your product and engineering teams. For retrieval-augmented generation (RAG) applications, we evaluate retrieved context and answer grounding alongside the surrounding application. For tool-using agents, we also inspect actions, permissions, resulting state, and escalation behavior.
AI Quality Challenges We Address
Answers that sound right but lack evidence
An agent may provide a confident answer when documentation is incomplete, outdated, or contradictory. Testing needs to check whether the response is supported by the retrieved material and whether uncertainty is handled appropriately.
Correct words, incorrect actions
A response can claim that a ticket was created even when a tool call failed. Evaluation should inspect the action trace, arguments, resulting state, and user-facing response together.
Untrusted instructions and access boundaries
User input and retrieved documents may contain instructions that conflict with application rules. Cross-account requests and prompt-injection attempts require explicit checks of permissions and data boundaries.
Behavior that changes between releases
Prompt edits, new models, updated documents, and API changes can alter behavior. A repeatable comparison needs versioned inputs, consistent criteria, and analysis of failures by category.
Our Approach
Define behavior and release requirements
Map the support journeys, permitted actions, and human escalation rules. Specify what a successful task looks like and classify failures by business impact. Agree which failures block release rather than treating every check as equally important.
Build representative evaluation datasets
Create normal, ambiguous, adversarial, and failure-recovery scenarios using synthetic or appropriately de-identified data. Record expected behavior and data provenance. Reserve held-out cases for evaluation so improvements are not judged only on examples used during tuning.
Evaluate retrieval, responses, and tool calls
Measure retrieval relevance and answer grounding separately. Use deterministic checks for structured outputs, permissions, and tool arguments. Apply clear rubrics to response quality and task completion; calibrate model-based judges against human-reviewed examples.
Exercise failures and human handoff
Simulate missing knowledge, unavailable tools, timeouts, conflicting instructions, and repeated requests. Check that the agent reports what actually happened, avoids duplicate actions, and escalates when it cannot safely complete a task.
Analyze failures and retest fixes
Group failures by cause and severity, preserve useful traces, and reproduce the failing scenario. After a fix, rerun both the affected cases and the wider regression suite. Record the model, prompt, dataset, and tool versions used for comparison.
Automate checks and review releases
Run appropriate evaluations in CI/CD and review the evidence against agreed thresholds. Assess task completion, groundedness, tool correctness, escalation behavior, latency, and cost where relevant. Feed new production failure patterns back into the test suite.
What We Test
A policy is missing from the knowledge base
Input: a customer asks about an undocumented exception. Expected behavior: identify the missing evidence, avoid inventing a policy, and offer the agreed escalation path. Evidence: retrieved context, answer, and handoff record.
A ticket API times out
Input: a ticket-creation request reaches a slow or unavailable service. Expected behavior: follow the configured retry or recovery policy without falsely claiming success. Evidence: tool calls, service response, actual ticket state, and the message shown to the user.
A retrieved document contains a hostile instruction
Input: a knowledge-base passage asks the agent to reveal restricted information or ignore its rules. Expected behavior: treat the passage as untrusted content and preserve access boundaries. Evidence: response, action trace, and attempted tool calls.
A user repeats an action after a delay
Input: the same ticket request is sent again before a response arrives. Expected behavior: follow the product’s duplicate-handling policy and report the resulting state accurately. Evidence: request identifiers, ticket records, and user-visible confirmation.
What You Receive
A reusable evaluation dataset
A versioned set of representative scenarios with expected behavior, coverage notes, and known limitations. We document how to maintain these cases as your product, integrations, and risk profile change.
A prioritized failure report
Reproducible findings with severity, supporting evidence, and retest status. Our reports distinguish observed behavior from possible causes and make unresolved risks visible to your team.
Release evaluation criteria
Evaluation reports and agreed acceptance criteria that help your team compare versions. Automated checks support release decisions, with human review for ambiguous or higher-impact cases.
How We Assess Release Readiness
We assess a candidate release against the criteria agreed for your product. The review brings together task completion, response quality, tool correctness, recovery behavior, and unresolved failures. We examine critical workflows individually so a strong aggregate score does not hide a serious failure.
Our evaluation reports document measurement definitions, test-set size, model and configuration versions, evaluation conditions, and the comparison baseline. Where useful, we repeat runs to examine variation. We explain the limits of the tested scope and identify cases that need human review or continued monitoring after release.
Build an AI Testing Process Around Your Product
Anbosoft helps teams define quality requirements and test AI-powered applications, agents, and their integrations. Explore our AI Agent Testing & Evaluation service, learn about our cybersecurity testing and regression testing services, or contact us to discuss your workflow.
Methodology Reference
The NIST Generative AI Profile provides a public reference for considering generative AI risks across the lifecycle. It informs our focus on evaluation context, documented evidence, and ongoing review.