Why AI Agents Break After the Demo
Published . Updated .
AI means artificial intelligence. An AI agent uses a model and tools to complete tasks. A successful demonstration does not prove production reliability. Failures can come from the model or the surrounding system. Test both before deciding which part to change.
Production introduces varied questions, incomplete inputs, changing sources, access limits, and real consequences. A demonstration can omit these conditions. Build tests from the task that users actually need to complete.
What does the wider evidence show?
In July 2024, Gartner forecast abandonment of at least 30% of generative AI projects after proof of concept by end-2025. A proof of concept tests whether an idea can work. Gartner named poor data quality, weak risk controls, rising costs, and unclear business value. This was a forecast, not a measured failure rate. It does not rule out model limitations. Gartner forecast
S&P Global reports that organisations scrap an average of 46% of AI proofs of concept before production. That reported average does not establish the failure rate for every agent or industry. S&P Global research
How does production differ from a demonstration?
| Area | A limited demonstration | What production testing needs |
|---|---|---|
| Questions | Selected examples | Representative user questions and variations |
| Inputs | Clean files | Missing fields, scans, typos, and conflicting records |
| Data | Selected sources | Current sources, duplicates, and access restrictions |
| Actions | Simulated changes | Real consequences and approval rules |
| Visibility | The builder watches | Records that show where failures occur |
| Ownership | The builder handles problems | A named owner and operating measures |
These are risks to check, not claims that every demonstration has these weaknesses. A well-designed demonstration can include production-like conditions.
Why can multiple steps reduce reliability?
Consider independent steps that each succeed with a probability of 95%. Assume every step must succeed and no retries or recovery occur. Under those assumptions, task success is 0.95 raised to the number of steps.
That gives about 77% success for five steps and 60% for ten steps. This is an arithmetic illustration, not an agent benchmark. Real failures can depend on each other. Checks, retries, recovery, and human review also change task outcomes.
Illustration only: equal 95% step success, independent steps, and no retries or recovery.
Measure the whole task on representative cases. A high score for one step does not prove the full task meets its acceptance criteria.
Which parts should the team check?
| Risk | What to inspect | Possible response |
|---|---|---|
| Limited tests | Cases do not represent real usage | Add representative and difficult cases |
| Wrong source retrieval | Answers use stale or unrelated records | Check sources, search results, and citations |
| Input errors | Scans or incomplete fields change the result | Validate inputs and test exceptions |
| Excessive permissions | The agent can take unnecessary actions | Limit access and apply approval rules |
| Missing activity records | The team cannot locate the failed step | Record relevant inputs, tool actions, and outputs |
| Unclear ownership | Nobody reviews failures or value | Assign an owner and operating measures |
| Model limits | Errors persist with correct inputs and context | Compare models or narrow the task |
Source retrieval means finding the information that the model needs. Activity records must protect sensitive data and follow retention rules. More logging is not useful when it creates unnecessary privacy risk.
How Chiri approaches this. Chiri's small teams of senior operators and engineers work with the customer's team. They plan testing, permissions, activity records, and ownership during delivery. The scope and customer requirements determine the controls. See Chiri's services
How do you separate model errors from system errors?
Repeat representative tests with the same inputs and checked source context. Compare model outputs with the complete agent's outputs. Inspect source retrieval, tool actions, prompts, and calculation logic. Keep the test conditions and versions recorded.
One successful answer does not prove the model is reliable. One failed answer also does not establish the full cause. Use repeated cases and error analysis before changing models or integrations.
When should a company use simpler automation?
A fixed, predictable task can suit a script or conventional automation. Compare those options before adding model-driven decisions. Choose an agent when its flexibility provides value that justifies the extra testing and controls.
Define an owner and acceptance criteria before launch. Apply controls that match risks to money, customer data, and regulated records. Chiri's governance article explains that control layer. Where AI Rollouts Go to Die covers organisational risks.
Check the failing step and its evidence before assigning a cause. Talk to Chiri or see Chiri Brain.
Questions and answers
Will a newer model fix an unreliable agent?
It can help when model limitations cause the errors. It will not necessarily correct wrong sources, missing permissions, or broken tools. Test the model and the complete system separately.
How many test cases does an agent need?
The number depends on task variety, risk, and required confidence. Use representative cases and difficult inputs. Repeat tests after changes to prompts, data, tools, or models.
Why can multi-step reliability decline?
Every required step can introduce failure. Independent 95% steps give about 77% success across five steps without recovery. Real systems need their own task-level measurements.
Who should own production reliability?
Assign a named business owner and clear technical support responsibilities. Agree measures, exception handling, review dates, and change controls before launch.