Why Cisco Ditched RAG for Fine-Tuning in Production AI Agents
Ola Mabadeje from Cisco's Outshift group reveals how they built a 5-agent system for network change management, why fine-tuning beat RAG for knowledge graph queries, and how digital twins prevent production failures.
"We initially started by attempting to use RAG to do some querying of the knowledge graph, but that was not working out well. So we decided that for immediate results, we're going to fine-tune it."
Cisco's decision to replace RAG with fine-tuning for knowledge graph queries
Ola Mabadeje, Cisco (Product Manager, Outshift)Specialized Agents
Token Reduction
Knowledge Graph Layers
Standards (MCP, A2A)
Why Fine-Tuning Beat RAG
A pragmatic decision that challenges the default approach
When Cisco's team needed to query their knowledge graph using natural language, they started with the default approach: RAG. But the results were disappointing—slow queries and excessive token consumption. Their solution? Fine-tuning a model specifically for ArangoDB Query Language (AQL) generation.
RAG Approach
- • Queries navigated all graph layers in reasoning loops
- • Excessive token consumption
- • Slow response times
- • Context-heavy retrieval
Fine-Tuning Approach
- • Drastic reduction in tokens consumed
- • Faster query generation
- • Direct, efficient graph interaction
- • Learned schema patterns deeply
"After fine-tuning, we saw a drastic reduction in number of tokens consumed as well as the amount of time it took to actually come back with the results."
Results after fine-tuning the query agent
"Every time the AQL queries were going through all of the layers of the knowledge graph and in a reasoning loop was consuming lots of tokens and taking a lot of time for it to result to return results."
Why RAG was failing - excessive token consumption in reasoning loops
"OpenConfig schema is a schema that is designed around networking primarily and there's a lot of documentation about it on the internet. So LLMs understand this very well."
Why OpenConfig schema was key for fine-tuning success
The Fine-Tuning Secret Weapon: OpenConfig Schema
The key to successful fine-tuning was choosing the right schema. OpenConfig is an industry-standard networking schema with extensive documentation on the internet. This meant LLMs already had pre-trained familiarity with it. Fine-tuning just sharpened that existing knowledge, enabling the model to generate efficient AQL queries without retrieving context for every query.
Cisco's 5-Agent Architecture
Specialized agents collaborating for network change management
Instead of building a monolithic "do-everything" agent, Cisco's team broke the problem into specialized components. Each agent has a specific responsibility and uses ReAct (Reasoning + Acting) loops for decision-making.
Assistant Agent
Coordinates the entire workflow, manages other agents, and handles human interaction
Query Agent
Fine-tuned agent that directly queries ArangoDB knowledge graph using AQL
Impact Assessment Agent
Analyzes potential downstream effects and dependencies of proposed changes
Test Planning Agent
Generates comprehensive, targeted test plans based on change impact and network context
Execution Agent
Creates snapshots, runs tests in digital twin, generates pass/fail reports with recommendations
"We have agents that are tasked at doing specific things. So an agent that is tasked as doing impact assessment, doing testing, doing reasoning around potential failures that could happen in the network."
Specialized agents with distinct responsibilities
"The assistant agent that's kind of the planner that kind of orchestrates things across all of these agents."
Orchestrator agent coordinating the workflow
"We are looking at a system... that allows agents from across the world to talk to each other without having to do heavy lifting of reconstructing your agents every time you want to integrate them."
Open standards vision for agent interoperability
Knowledge Graphs as Digital Twins
Building safe testing environments before production
Cisco's knowledge graph represents a complete digital twin of the production network. By layering the architecture and combining it with testing tools like Batfish and Routenet, they can validate changes in isolation before risking production.
Layered Architecture
Raw Configuration Layer
Device configurations for drift detection and change tracking
Control Plane Layer
Network routing and protocol information
Data Plane Layer
Traffic flow and telemetry data
Topology Layer
Network relationships and connections
Why ArangoDB?
- Multi-model flexibility: Graph, document, and key-value in one database
- Vector indexing: Native support for semantic search (GraphRAG)
- Instant node access: Performance regardless of node location
- Ecosystem stability: Operational flexibility for enterprise use
OpenConfig Schema Advantage
OpenConfig is an industry-standard networking schema with extensive documentation. LLMs already understand it well from pre-training, making it the perfect choice for knowledge graph representation and fine-tuning.
"What we're trying to do here is to build a twin of the actual production network. And that twin includes a knowledge graph plus a set of tools to execute testing."
Digital twin concept for network change testing
"It's structured in a way that when the agents are making their calls to this system, they understand what the request is from the system and they're able to actually go to the right layer to pick up the information that they need to execute on it."
Layered knowledge graph architecture
"We wanted a knowledge graph that can have multi-model flexibility. That means you can talk key-value pairs, you understand JSON files, it understands relationships across different entities in a network."
Requirements for knowledge graph technology selection
"If an engineer is querying a knowledge graph, we want to have instant access to the node information about the node no matter where the location of that node is."
Performance requirements for production queries
Complete Automation Workflow
From ServiceNow ticket to tested network change
Cisco's system orchestrates across enterprise tools—from ServiceNow tickets to GitHub PRs—creating a seamless workflow where AI handles analysis and testing while humans stay in the loop for approval.
ServiceNow Ticket Created
Network engineer submits change request through ITSM system
Intent Synthesis & Impact Assessment
Assistant agent summarizes ticket, Impact agent analyzes downstream effects
Test Plan Generation
Test Planning agent creates targeted test list based on change intent
GitHub PR with Config Changes
Configuration stored in GitHub repo, PR linked to ticket for review
Digital Twin Testing
Executor agent creates snapshot, runs tests sequentially in isolated environment
Results & Recommendations
Comprehensive report with pass/fail status and fix recommendations
"Test plan is one of the biggest problems that our customers are facing. They run a lot of test but they miss out on the right test to run."
The test planning problem in network operations
"The first thing is that this agent called the executor agent goes looks at the test cases and then it goes into the knowledge graph and it's going to go ahead and actually do a snapshot of the most recent visual or most recent information about the network."
Executor agent creating network snapshots for testing
"So all of this is happening in what we call a digital twin. So a digital twin again is a combination of the knowledge graph, a set of tools that you can use to run the test."
All testing happens in digital twin environment
Open Standards for Agent Interoperability
Cisco's bet on Alliance for Agency and open protocols
Cisco is investing in open standards through the Alliance for Agency (agency.org collective). The vision: agents from different vendors should be able to communicate without heavy integration work. They're integrating with MCP (Model Context Protocol) and A2A (Agent-to-Agent) protocols.
Alliance for Agency
Open source collective including Cisco, LangChain, Galileo, and others. Building protocols for agent identity, skills definition, discovery, and composition.
github.com/interagent →Protocol Integrations
Cisco's multi-agent system integrates with MCP (Model Context Protocol) and A2A (Agent-to-Agent) protocols for cross-platform communication.
"The goal again is not to create something that is bespoke. We want to make it open to everyone to be able to create agents and be able to make these agents work in production environments."
Cisco's commitment to open standards
Key Takeaways
Practical insights for AI engineers
1. Fine-Tuning > RAG for Specialized Tasks
When to Fine-Tune
- •RAG consumes excessive tokens for repetitive queries
- •Fine-tuning with schema examples reduces cost and latency
- •Best for structured domains with clear query patterns
- •OpenConfig schema: LLMs already know it from pre-training
2. Multi-Agent Systems Scale Better
Agent Architecture
- •Specialized agents with clear responsibilities outperform monolithic agents
- •Use orchestrator agent to coordinate workflow
- •ReAct loops for reasoning + acting
- •Agent-to-agent communication across systems (ServiceNow ↔ Cisco)
3. Digital Twins Enable Safe Testing
Knowledge Graphs
- •Knowledge graph + testing tools = digital twin
- •Layered architecture for efficient querying
- •Test changes in isolated environment before production
- •ArangoDB: multi-model flexibility + vector indexing
4. Open Standards for Interoperability
Future-Proofing
- •Alliance for Agency: industry-wide agent communication standards
- •MCP and A2A protocol integrations
- •Build for interoperability, not walled gardens
- •github.com/interagent/
Source Video
Multi Agent AI and Network Knowledge Graphs for Change
Ola Mabadeje • Cisco (Outshift Group)
Research Note: All quotes in this report are from the full transcript analysis of Ola Mabadeje's talk at the AI Engineer Conference. This analysis covers Cisco's production multi-agent system architecture, fine-tuning vs RAG decision-making, knowledge graph digital twins, and open standards for agent interoperability.
Key Concepts: Multi-agent AI, knowledge graphs, fine-tuning, ArangoDB, OpenConfig, digital twins, network automation, ServiceNow integration, GraphRAG, agent interoperability, Alliance for Agency, MCP, A2A protocols, ReAct loops