Who Tests the AI? A Practical Guide to AI Testing Before Production

AI testing is the work of proving that an AI-powered application behaves accurately, safely, and affordably enough to put in front of real users. Most teams find out they need it the same way: the demo impressed everyone, the pilot went live, and within weeks someone saw an answer the system should never have given.

The hard question is rarely which tool to use. It is who owns the decision that the AI is ready. In conventional software, QA signs off against written requirements. In an AI application, “correct” is often a judgment about accuracy, tone, and risk, and that judgment is spread across product, engineering, data science, legal, and the business. When nobody owns it, testing shrinks to a few people trying prompts the week before launch.

The cost shows up in the numbers. Gartner reported in January 2026 that at least 50% of generative AI projects were abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs, and unclear business value (Gartner). Weak testing sits behind at least two of those four.

The useful question, then, is less about tools and more about proof: what has to hold true before real users arrive, and who should test what before anyone signs off.

What is AI Testing?

AI testing is the process of evaluating an AI-powered system, such as a chatbot, RAG assistant, AI agent or machine learning model, to confirm that it produces accurate, safe, fair and reliable outputs before and after it reaches real users. Unlike conventional testing, it measures how often a system’s answers are acceptable across many realistic inputs, rather than checking for one fixed correct result.

The term is also used in a second sense, and the difference decides who you hire and what you buy. Testing of AI validates the AI system itself: whether a chatbot answers correctly, whether a model treats groups fairly, whether an agent can be manipulated. Testing with AI uses AI to test conventional software, for example by generating test cases or repairing broken automation scripts.

Both matter, but they answer different questions. Using AI to speed up QA is the territory of AI in quality assurance. The rest of this piece stays with testing the AI itself.

Why Does Traditional QA Break on AI Applications?

Traditional QA assumes the same input always produces the same output, and that the correct output can be written down in advance. AI applications break both assumptions, so pass/fail scripts alone cannot tell you whether an AI system is ready.

The last two rows catch teams out most often. A hosted model can be updated by its provider without any change on your side, and a prompt tweak that fixes one answer can quietly break ten others. It is a common failure: nearly one-third of respondents in McKinsey’s 2025 State of AI survey reported negative consequences from AI inaccuracy. We saw the cost of late discovery on HireSales AI, an AI recruitment platform: with QA tracked in spreadsheets, problems surfaced only in user acceptance testing, where backend crashes blocked releases. Moving regression testing into the delivery pipeline meant every change was checked before it shipped, not after.

Who Tests the AI?

No single team can test an AI application alone. QA engineers own the test process and the evidence, ML engineers own model and retrieval quality, domain experts judge whether answers are actually right, security runs adversarial tests, and one named business owner signs the release against agreed thresholds.

The domain expert row is the one most often left out. An automated judge can check whether an answer matches its source document. It cannot tell you whether a refund policy or a lending rule was explained correctly to a customer. Budget expert review time from the first sprint, not the last week.

Skills and governance are the other gaps. Half of organizations cite limited in-house AI expertise as a barrier (Capgemini), and 63% of breached organizations had no AI governance policy in place (IBM). So many teams keep ownership internal and bring in a specialist partner for evaluation and red teaming.

What Should You Test Before an AI Application Goes Live?

Before launch, test seven areas: task quality and hallucination, retrieval grounding, agent actions, security, fairness, performance and cost, and human handoff. Which ones matter most depends on what the application does and who it affects.

Task quality and hallucination

Measure whether the application completes the real task correctly, not whether it sounds fluent. Build a set of real user requests with reference answers or scoring rubrics, then track the share of responses that are correct, partly correct, or contain unsupported claims. Hallucination testing targets that last group: statements the system presents as fact without any basis in the information it was given. Picking a stronger model does not remove the need; on one benchmark in the Stanford AI Index 2026, hallucination rates across leading models ranged from 22% to 94%.

RAG retrieval and grounding

For retrieval-augmented generation (RAG) systems, test retrieval and generation separately. First check that the right documents come back for each question, then check that the answer stays faithful to them. A wrong answer often starts as a retrieval miss, so scoring only the final answer hides where the fix belongs. Also test permissions: a user should never receive content from a document they are not allowed to see. That access layer is a large part of what separates a governed enterprise assistant from a plain chatbot, a trade-off we cover in RAG vs LLMs.

Agent actions and permissions

AI agents that call tools, APIs, or business systems need testing at the level of each action. Check that the agent chooses the right tool, passes valid parameters, stops when it should, and cannot act beyond its role. Run consequential actions (refunds, record updates, outbound emails) in a sandbox with human approval enforced. Our guide to agentic AI for the enterprise covers where agents fit.

Security and adversarial testing

Red-team the whole application, not just the model. The OWASP Top 10 for LLM Applications (2025) is a practical starting checklist: prompt injection, sensitive information disclosure, system prompt leakage, vector and embedding weaknesses, excessive agency, and unbounded consumption all apply to a typical business chatbot (OWASP). For agents, OWASP’s Top 10 for Agentic Applications, published in December 2025, extends the list to risks specific to autonomous systems (OWASP). Include indirect injection: instructions hidden in documents, emails, or web pages the AI reads.

Performance, latency, and cost

Test under realistic load, including long documents and multi-turn conversations. Track response time at the 95th percentile and cost per completed task, not just cost per call. A prompt change that improves accuracy can also multiply token usage, so cost belongs in every regression run.

AI logic is often the slowest part of the stack. On HireSales AI, an AI recruitment platform, our QA team found a 60% error rate under peak load and API responses above 5 seconds, traced to synchronous AI logic, missing database indexes, and uncached endpoints. After inference was decoupled, batching and caching were added, and the platform was load-tested at 500+ concurrent users; the error rate fell below 0.5% and response time dropped to 1.1 seconds.

Fallbacks and human handoff

Test what happens when the AI should not answer: out-of-scope questions, low-confidence retrieval, frustrated customers, requests that need a person. A well-built AI application declines clearly and hands over to a human with the conversation context attached.

How Do You Build a Pre-production AI Testing Workflow?

A workable pre-production workflow has seven steps, and the first happens before any test runs: agreeing in writing what “good enough” means.

1. Define acceptance criteria per use case. Set thresholds with the business owner for task success, maximum unsafe-response rate, grounding, latency, and cost per task. Base them on the risk of the use case, not on a vendor benchmark.
2. Build an evaluation dataset from real workflows. Use real, anonymized requests, including edge cases and the questions you hope users never ask. Version it like code so every release is tested against the same baseline.
3. Choose how each test is graded. Use code-based checks for format and facts, expert review for judgment calls, and an LLM-as-a-judge for scale. Calibrate any LLM judge against human graders on a sample before trusting its scores.
4. Red-team the application. Run adversarial tests for injection, data leakage, and misuse, including attacks hidden in documents and web content the AI reads.
5. Hold a release-gate review. Compare results with the thresholds. The business owner signs off with the evidence pack in front of them, and any accepted risk is written down.
6. Release in stages. Move from shadow mode to a limited pilot to wider rollout, logging inputs, outputs, retrieved sources, and tool calls at each stage.
7. Evaluate continuously. Re-run the regression suite on every prompt, data, retrieval, or model change, including provider model updates, and sample live conversations for expert review. MLOps and model monitoring make this routine rather than heroic.

The weight of each test depends on the application type:

Where do AI testing tools fit?

Tools make AI testing repeatable, but they do not decide what “good” looks like. That decision comes from your acceptance criteria. Most teams combine three kinds of tooling:

• Evaluation frameworks that score outputs against datasets and rubrics, including open-source options such as DeepEval, Ragas, and promptfoo.
• Observability and tracing platforms that record prompts, retrieved context, tool calls, and cost in production.
• Red-teaming and security scanners that generate adversarial inputs at scale.

AI-powered test automation tools are a separate category. They speed up testing of the conventional parts of an AI product, such as its interface, APIs, and integrations. Choose any tool after you have settled data-handling rules: a platform that sends production conversations to a third-party service can create a compliance problem of its own.

Which Regulations and Standards Shape AI Testing?

Testing evidence is becoming a compliance requirement, not just good engineering. The EU AI Act expects providers of high-risk AI systems to demonstrate appropriate accuracy, robustness and cybersecurity, backed by technical documentation. Under the Digital Omnibus amendments published on 24 July 2026, high-risk obligations for Annex III systems (such as employment and credit use cases) now apply from 2 December 2027, and for Annex I systems embedded in regulated products from 2 August 2028. Transparency obligations under Article 50 generally apply from 2 August 2026 (Future of Privacy Forum).

Sector regulators in financial services and healthcare, and national AI guidance in markets such as the UK, UAE and Saudi Arabia, can add requirements of their own. Check them before you set thresholds, not after. For EU-specific detail, see our article on what the EU AI Act means for enterprise software.

Should You Test AI In-house or Work With an AI Testing Partner?

Keep ownership in-house and bring in a partner for capacity and specialist skills. Acceptance thresholds, domain review, and the release decision should stay with your team. Evaluation pipelines, red teaming, and regression automation can be built and run by a team that has done it before.

What Should an AI Software Testing Consultation and Quote Cover?

A useful consultation scopes the risk before it prices the work. Expect questions about:

• How many use cases and user journeys the AI handles
• Whether any use case is high-risk or regulated
• Where test data may be processed, and how sensitive it is
• Which systems the AI integrates with, and which actions an agent can take
• Languages and channels, such as chat, email, and voice
• Whether you need a one-time pre-launch assessment or ongoing regression

A good quotation separates one-time setup (evaluation datasets, pipelines, red teaming) from recurring cost (regression runs, expert review hours, model and judge API usage). If a quote cannot tell you what evidence you will receive at the release gate, ask before you sign.

How Hidden Brains Approaches AI Testing

Hidden Brains has built and tested software for clients since 2003, under ISO/IEC 27001:2022 information security and ISO 9001:2015 quality management certifications. Our AI application testing services sit alongside functional, security, and regression QA, so AI behavior is tested together with the integrations around it. Engagements usually start before any test runs, by agreeing thresholds, owners, and data rules with the business. Where governance and ownership are the open question, our AI strategy consulting team helps set the operating model first.

Frequently Asked Questions