Section 3 – Orchestrating
An agent that works once is unreliable. Like any software, an agentic system needs to be tested, observed and improved over time. The difference is that an agentic system introduces variability: it depends on the context, data, rules and integrations andskills it mobilizes.
Testing agentic therefore requires a clear engineering discipline.
An agentic system in production must work hundreds, thousands of times, under imperfect conditions. The question is not “Does it work? but “Under what conditions does it stop working, and how does the system react?
An agent, skill or workflow that is not rigorously tested quickly becomes an operational risk. Conversely, a system that is continuously evaluated, layer by layer, becomes a reliable component that the organization can grow with confidence.
In this article
What you really need to test
A common misconception is that it’s enough to test the agent. In practice, the quality of an agentic system depends on several distinct layers, all of which need to be validated.
First,skills. A skill must produce a coherent result when used in different contexts. It must be testable independently of the rest of the system.
Secondly, the agents themselves. They must demonstrate their ability to select the right skills, make the right decisions and achieve the objectives entrusted to them.
Then there are agentic workflows. Several agents and several skills can work together to execute a complete process. This is often where the most costly errors occur.
Finally, the entire agentic system must be validated in its real environment: data, integrations, permissions, business rules and human supervision.
Testing only the agent is tantamount to testing only the application interface, without checking the services that feed it.
Defining what it means to operate
Even before writing tests, it’s important to define what “good” means. An agentic system is not evaluated solely on the quality of a response, but on its ability to produce a useful, reliable and consistent result in a real context.
This involves clarifying :
- the expected result ;
- conditions of acceptance ;
- performance thresholds ;
- when the system must request validation or shut down;
- criteria specific to skills, agents and workflows.
Without it, it is impossible to judge whether an agentic system is ready for production. An agent can produce convincing answers and still be unusable if it doesn’t respect business rules, acts at the wrong time or creates inconsistencies elsewhere in the process.
Testing an agentique doesn’t just mean checking that the answer is correct. It also means checking that it acts correctly in the system in which it is inserted.
Test skills separately
Organizations are quickly discovering that it’s much easier to maintain an agentic system when skills are tested independently of agents.
A skill is generally evaluated according to :
- the accuracy of the result produced
- compliance with business rules
- behavioral stability
- speed of execution
- cost of execution
- the ability to return a clear error when it cannot complete the task
Let’s take a simple example. A sales agent can use :
- CRM research competence
- qualification skills
- writing skills
- proposal generation skills
If proposal quality deteriorates, the team needs to be able to quickly identify which skill is responsible. Testing each skill individually helps to isolate problems, speed up corrective action, reduce regressions and reuse the same skills in multiple agents.
On a large scale, skills become reusable software components that deserve their own quality strategy.
Testing beyond the ideal scenario
Demonstrations often focus on an ideal scenario. In production, data is imperfect, edge cases frequent and situations unexpected.
Testing an agentic system consists of :
- check its behavior in a variety of scenarios
- observe its reaction to errors and incomplete data
- validate your ability to respect rules and permissions
- measure the consistency of your decisions over time
- ensure that the right skills are called up at the right time
- validate that errors in one skill do not compromise the entire workflow.
Traditional software engineering practices remain relevant: unit testing, scenario testing, regression testing. The difference is that an agentic system introduces a degree of variability. So we need to test not only what it does when everything’s going well, but also what it does when the context changes.
A reliable agentic system is not one that always succeeds. It’s the one that behaves predictably when it can’t succeed.
Measuring performance, not just response
In an agentic system, performance is measured not only by the quality of a textual response, but also by its impact on the process. It is measured by the impact on the process.
An agent may produce a correct response, but slow down a flow, trigger the wrong action or create a bottleneck.
It is therefore necessary to define appropriate indicators:
- processing time
- error rate
- frequency of human validations
- decision consistency
- cost per share or transaction
- success rate by skill
- skills reuse rate
- frequency of escalations towards a human
- breakpoints in workflows
These indicators make it possible to evaluate the agentic system as an operational component, not as a simple text generation tool. They bring agent evaluation closer to software engineering and operational quality standards.
Observe and measure in production
Testing doesn’t stop with deployment. An agentic system needs to be observed continuously. You need to be able to understand what it does, why it does it and with what results.
This requires observability mechanisms: activity logs, performance indicators, error tracking and human validation.
In an agentic system, observability must also make it possible to understand which skills have been called up, in what order, with what data and with what results.
This information is used to identify drifts, adjust rules and gradually improve the system.
The evaluation of agents in production becomes a lever for optimization. Real-life usage data can be used to adjust instructions, rules, integrations and reusable skills. It also enables us to identify cases where the system needs to become more autonomous, and those where it needs to remain supervised.
A mature agentic system is based on a continuous loop: observe, measure, adjust.
Improve without starting from scratch
Agents evolve. Models change, data transform, business rules evolve. Regular testing makes it possible to adjust these elements without rebuilding the whole system.
This is even truer when skills are designed as reusable components. A well-tested skill can be improved, replaced or reused in several agents (and even in different platforms) without having to rebuild the whole system.
This logic of continuous improvement transforms agentic into a sustainable software component rather than a one-off project.
An agent is not a fixed object. It’s a living system that needs to be maintained, evaluated and improved, just like any other mission-critical software. But in a mature agentic architecture, it’s not just the agents that evolve. Skills, workflows, rules and integrations must also be maintained over time. Evaluation is not a final step. It’s a permanent capability of the organization.
TL;DR
- – An agent, skill or workflow that “works once” is worthless in production.
- – An agentic system is software that operates in real processes and needs to be tested as such: clear definition of correct operation, skills testing, agent testing, workflow testing, performance measurement and continuous observability.
- – Skills must be validated separately, because they become reusable components in several agents.
- – Without rigorous assessment, agentique becomes an operational risk.
- – With the discipline of engineering and measurement, it becomes a reliable lever that can be improved over time.
