The model gets most of the attention. Where does it run? Who hosts it? Is it open weight? Does the data stay in-region? These are important questions, but they can create a false sense of security. Underneath the model is a much larger collection of services and infrastructure that can also process, move, or store sensitive data.
I think of this as the hidden AI stack.
The Model Is Only One Layer
Embedding models, vector databases, document repositories, parsing tools, memory systems, logging, observability, agent infrastructure, caches and backups are all part of an enterprise AI system. Yet they tend to get far less attention than the model itself. Any one of them can potentially move data outside the boundary an organization thought it had created.
This means you can make the right decision about the model and still end up with an AI system that violates your sovereignty or data residency requirements.
Embedding models are a good example. A company might deploy its primary model entirely within its own environment, but use an external embedding API to process internal documents. From the organization's perspective, the AI model is local. But the documents may have already left the environment before that model ever sees them.
Vector databases introduce another consideration. They can contain representations of internal research, customer information, contracts, policies, source code and private communications. Organizations still need to understand where that database is hosted, where its replicas and backups live, who can access it and what operational metadata may be collected.
Then there are the document repositories themselves. Enterprise AI systems are increasingly connected to SharePoint, Google Drive, internal databases, file systems and object storage. Each has its own residency rules, permissions, backup practices and regional configurations. Those controls need to carry through to the AI system as well.
Permissions Have to Travel With the Data
This is particularly important for permissions. Just because an AI system can retrieve a document does not mean the person asking the question should be able to see it. If permissions from the original repository are not carried through correctly, AI can unintentionally become a new way of exposing information internally.
The retrieval process adds even more infrastructure. Documents may be parsed, cleaned, divided into chunks, enriched, embedded, cached and indexed before the model ever receives them. Each step could involve another service or provider. An organization can spend months evaluating its model provider while confidential documents are quietly being processed by an external OCR service or document parser that received comparatively little scrutiny.
Memory and Observability Create New Data Stores
Memory creates another layer. AI systems are increasingly able to remember conversations, users, decisions and organizational context over long periods of time. That can make them dramatically more useful, but it also creates another data store that needs to be governed. Where does that memory live? How long is it retained? Who can access it? Can it be deleted? Does it remain inside the same residency boundary as the rest of the system?
Logging and observability may be even easier to overlook. Prompts, model outputs, retrieved documents, user feedback and error traces are often copied into monitoring and evaluation platforms so teams can understand how their systems are performing. Those logs can contain some of the most sensitive information in the application. It is entirely possible to build a private AI deployment and then unknowingly send production traces outside the environment.
Agents widen the stack again. Once an AI system can search internal systems, execute code, call APIs, send messages or update records, there are now credentials, tool permissions, execution environments and audit trails to consider. The sovereignty boundary is no longer simply about where information is stored. It includes where information is processed and where actions are executed.
None of this means every enterprise AI system needs to be isolated or that using external services is inherently a problem. Different workloads should have different requirements. A marketing assistant does not need the same controls as a system working with patient information, financial records, defence workloads, legal documents or proprietary research.
The important thing is knowing what you are choosing.
Today, I think too much of the enterprise AI architecture conversation stops at the model. Companies spend enormous amounts of time deciding which model they can use and where it can run, while some of the less visible layers underneath it receive considerably less attention.
That needs to change.
Before an organization can really say that an AI system meets its sovereignty or residency requirements, it needs to understand the entire path its data can take. Not just which model receives the final prompt, but which services touch the documents, where embeddings are created, where information is indexed, where memory is stored, where logs are sent and where backups ultimately live.
The question is no longer simply, “Where does our model run?”
It is, “Where can our data travel across the entire AI stack?”
Because the model might be the most visible part of an enterprise AI system, but it is far from the only part that matters.
Rob Imbeault
SHARE