Lost in Translation: Why AI Agents Cannot Scale Without Semantic Foundations
As enterprises struggle to scale AI agents beyond the pilot stage, BCG Platinion is helping organizations build the semantic foundations that turn fragmented business data into machine-readable meaning and unlock new, enterprise-wide AI potential.
Key Takeaways
- Semantic debt is why enterprise AI fails to scale: Inconsistent definitions and business rules buried in legacy systems or undocumented expertise prevent AI agents from moving beyond the pilot stage.
- The semantic layer is the missing piece in AI architecture: A machine-readable, governed layer giving agents consistent definitions, relationships, and actions is what enables reliable enterprise AI.
- Build semantic foundations early to compound AI returns: Each maturity step unlocks new AI capabilities, and every new use case benefits from the structured knowledge already encoded.
Enterprises are deploying AI at pace — but despite having the right models and technology, many attempts to scale stall. Intelligence is not the constraint either, it is meaning.
Terms like ‘customer’ or ‘revenue’ can exist in multiple places simultaneously, like dashboards and process definitions, and while each may be locally valid, none is authoritative — humans navigate this through experience, new AI initiatives stumble.
Most organizations are still not equipping their AI systems with consistent knowledge or a machine-readable business language. The crucial semantics exist, but they are often buried in legacy systems or a domain expert’s intuition.
Often these critical semantics have never been externalized or formalized, and until this is done, every new agent will need to tackle and overcome the same fragmentation from scratch.
Why this matters:
- For CEOs and COOs: This article explains why initiatives stall after the pilot, and why structural investment separates those that will compound advantage from those stuck in a cycle of rebuilding.
- For CIOs and CTOs: The missing architectural layer is the bottleneck, and the article precisely maps the gap and sets out an actionable maturity journey.
- For Chief Data Officers: The semantic layer is the bridge between the data platform you have built, and what AI agents actually need to reason reliably.
- For Chief Risk or Compliance Officers: Tracing agentic outputs to governed data and explicit logic is a critical regulatory and operational requirement.
1. The Scale Constraint: Hidden Semantic Debt
We are seeing a consistent pattern across sectors. AI use cases are piloted, the results impress, the enterprise moves to scale them, then progress stalls. The cost is not just friction; it is value destroyed. Most AI delivery effort goes to reconstructing meaning rather than building new capability.
When customer records do not match across systems, costly anomalies arise. Summarization tools that work for one team can produce different outputs for another, procurement agents may approve suppliers, but miss financial distress signals, and customer service assistants might resolve a ticket, but fail to update billing systems.
These issues surface when models lack a coherent picture of themselves. This can also cause a critical gap in terms of understanding how products link to relevant regulations. At the root of this is semantic debt hidden beneath the surface (see this depicted below in Exhibit 1).
.jpg)
Organizations that have addressed it report measurable gains, in some cases tripling AI accuracy on the same data – or compressing months of manual work to minutes.
2. The Semantic Layer: An Overlooked Enabler
To understand why AI scaling breaks, it helps to look at how most enterprise technology stacks are actually built. Most large enterprises have built out three core layers: core transaction systems, data infrastructure, and the integration and security layer — each of which has matured considerably. Above them sits the (Gen) AI layer, where organizations are now directing significant investment.
What is still being neglected, is the component which provides AI consumers a consistent, machine-readable representation of what the data actually means: the semantic layer. This is visualized in Exhibit 2 below.
.jpg)
Think of the semantic layer as a shared dictionary combined with a map of how business concepts connect — one that machines can read and act on, not just humans. Without a semantic layer at the core of an enterprise stack to translate meaning for AI agents, semantic fragmentation and output variability is inevitable — this ultimately destroys trust.
3. Making Meaning Machine-Readable
The semantic layer externalizes business meaning from the systems it is embedded in, making it consistent and available to every system, application, and agent that needs it. To define meaning consistently, information must be organized:
- Standardized definitions and taxonomy: Core entities, like customers and products, must be defined once and reused everywhere, with shared logic ensuring consistent metrics and reporting. This reduces drift and duplication across teams but doesn’t capture relationships between entities (which more advanced AI use cases require).
- Ontologies: Ontologies go further by explicitly defining not just entities, but also how they relate to each other. They model a domain’s structure and enable systems to understand connections like customers, products, obligations, risks, and regulations (rather than relying on implicit joins or assumptions).
- Knowledge graphs: Knowledge graphs operationalize these relationships, allowing systems to traverse connected information in real time. This supports multi-step reasoning like tracing regulations to products and evidence, while complementing traditional relational databases (rather than replacing them).
Crucially, the semantic layer must also serve as the governed action plane: the interface through which agents connect to enterprise systems to execute, not merely retrieve. An agent that can reason over a customer's risk profile but cannot trigger a workflow, update a record, or initiate a process is an advisor — not an operator.
4. The Limitations of Retrieval
Retrieval-Augmented Generation (RAG) — feeding the AI agent relevant documents to draw its answers from, is the dominant approach used today when it comes to grounding AI agents. For simple questions and document retrieval, it works — but not when a chain of logic must be followed, consistent outputs produced, or when reasoning needs to be explained.
Four ways RAG falls short:
- Documents do not encode business logic: A policy document describes an obligation, but not the entity it applies to, the conditions that trigger it, or the process that must execute in response.
- Retrieval does not guarantee consistency: Two agents handling related tasks may receive different chunks, leading to contradictory outputs from the same model.
- Multi-hop reasoning is not naturally supported: Linking a customer to a product, and a risk category to a regulatory threshold, requires structured traversal, not document similarity.
- Outputs are difficult to audit: The reasoning chain passes through unstructured document fragments with no traceable path back to governed data. In regulated industries, that is not a sufficient basis for consequential decisions.
When an agent is grounded in an explicit ontology, these failure modes are structurally addressed. Tasks decompose into structured steps using domain relationships and inferences are constrained to what has been explicitly encoded. The agent cannot hallucinate relationships that do not exist.
This semantic grounding means outputs trace back to specific entities, relationships, and source data, because each inference step maps to an explicit, governed relationship. Crucially, it makes outputs deterministic and repeatable, which is fundamentally different from RAG approaches, where outputs are probabilistic. It also enables agents to act (not just answer), because the semantic layer connects to the governed action interfaces of the enterprise.
The goal here is not to displace RAG. Unstructured content still benefits from retrieval. But the semantic layer is what enables the agent to know which type of source each question demands, and to apply the right approach to each.
5. The Evidence in Practice
The argument is not theoretical. The same pattern and measurable improvement appear across industries.
In insurance, fraud detection depends on identifying connections across claims, policies, parties, and repair histories that sit in separate systems. A knowledge graph makes those connections explicit, enabling agents to surface networks that no model on isolated records can see.
A peer-reviewed benchmark on an enterprise insurance database found that adding a knowledge graph raised GPT-4's accuracy from 16% to 54% on the same data, a threefold improvement without changing the model.
When it comes to regulatory intelligence, determining which obligations apply to which products across which jurisdictions have traditionally required months of analyst effort (and up to 300 manual screening questions per assessment).
When obligations, conditions, and applicability criteria are modeled as a typed ontology, that same screening compresses to a ten minute conversation. Applicability is inferred dynamically from the structure, not rebuilt from scratch for every new product or market.
The underlying dynamic is the same in both cases, and it is illustrated in Exhibit 3 below. An upfront investment in ontology is required, but integration effort then flattens as definitions are reused, while the traditional approach compounds upward with every new use case.
6. From Retrieval to Reasoning
The evolution from AI assistants to AI agents changes the requirements fundamentally. An assistant answers questions. An agent plans, decides, coordinates, and acts across systems — where errors have real consequences. To do this reliably, it needs to know what entities exist, how they relate, what actions are permitted, and what the downstream consequences are.
Agents must also know which type of source each question demands. When evaluating a loan applicant's creditworthiness, they need to be able to query a credit score database with exact arithmetic, not ask language models to reason those numbers from market commentary. The semantic layer encodes this distinction, without it, agents default to language models for everything, producing ‘confident’ answers grounded in the wrong type of evidence.
Building a semantic layer is not a greenfield undertaking, nor does it require a major transformation. In most enterprises, partial semantic structure already exists in taxonomies, data dictionaries, BI metric definitions, and MDM systems.
The maturity model below maps where you are today and what the path forward looks like:
Five moves that matter:
- Start where you already are: Identify the current maturity level of your top AI use cases. Most enterprises have the foundations for metric consistency or a canonical model already, build on these.
- Design a two-tiered ontology: A shared core defines the entities and relationships that are common across use cases. Use case specific tiers add domain detail, a procurement use case adds supplier structures and compliance requirements, and a quality management use case adds production runs, material lots, and failure modes.
- Move up one level at a time: Each level in the maturity model unlocks a new class of AI capability. Metric consistency enables reliable analytics agents. A canonical model enables cross-domain AI use cases. Semantic reasoning is what separates agents that answer from agents that act — reliably, traceably, at scale. Define which level your most critical use cases demand, and close that gap first.
- Design for compounding returns: The value of the semantic layer increases non-linearly as you progress. Each new entity enriches the graph, each new relationship opens new reasoning paths, and each new use case benefits from everything encoded before it. Organizations that invest early compound that advantage.
- The organizational reality: Ontology design is a political exercise as much as a technical one. Defining shared concepts means reconciling assumptions that different teams have held for years — what "customer" means in sales, versus finance, versus legal. Organizations that hand this to IT alone will produce models that business teams ignore.
Governance is ultimately what separates a semantic layer that scales from one that stalls. The semantic layer is not an IT asset, only the COO and heads of functions have the authority and domain knowledge to define what entities mean and how relationships must be modeled. When framed as a business leadership commitment, it becomes the foundation on which every AI initiative compounds.
Three Key Questions
The next phase of enterprise AI will not be defined by improvements in models. It will be defined by how well organizations represent their own business to those models.
Future industry leaders will have successfully externalized their meaning, turning implicit knowledge into a governed, machine-readable structure that every agent can rely on. They will compound that advantage with every new dataset, use case, and capability.
Before you make your next AI investment, ask these three questions:
- Where in your enterprise do your core entity definitions differ, and which team has authority over them?
- During your last three AI initiatives, how much time did you spend reconstructing data relationships and business logic that already existed somewhere in the organization?
- Which of your planned agent use cases require following a chain of logic across multiple systems, and does your current architecture support it reliably, with consistent and auditable outputs?
If any of these three questions are difficult to answer, the semantic layer should not merely be treated as a future investment – its absence is already limiting value creation.
BCG Platinion works with enterprises across all industries to design agents with semantic foundations for AI at scale, from maturity assessments and ontology design to knowledge graph implementation and agent integration.
If your organization is asking why AI initiatives are not compounding, or how to move from retrieval to reasoning, get in touch with our experts to continue the conversation.
