Yes. Every part of a RAG chatbot, including the document store, search index, embedding model and language model, can run inside infrastructure you control, up to a fully private network with no outbound internet access. The trade-off is some model capability and more operating work the further you move from a hosted service. Check every component, including embeddings, logging and document parsing, since each can send data to a third party even when the final model does not. Most businesses land on a model hosted in their own cloud account with no-retention terms, reserving full self-hosting for data that requires it.
Can a RAG chatbot run without sending data to a third party?
Yes. Every part of a RAG chatbot - the document store, the search index, the embedding model and the language model that writes the answer - can run inside infrastructure you control, including a private network with no outbound internet access. What you give up is some model capability and the convenience of a managed service, so the real decision is how far along that scale your data requires you to go.
"Third party" also covers more than the model provider. Logs, analytics, monitoring, document parsing services and the vector database host can all see your data. A private setup has to account for each of them.
Where does data actually leave your environment in a RAG system?
| Component | What it sees | How to keep it in-house |
|---|---|---|
| Language model | The question and the retrieved passages | Self-hosted open-weight model, or a model in your own cloud account |
| Embedding model | Every document chunk and every query | Run an open embedding model locally |
| Vector database | Chunks, embeddings and metadata | Self-host, or use a database in your own account or region |
| Document parsing and OCR | The full text of every file | Run parsing libraries or OCR on your own servers |
| Logging and tracing | Prompts, answers and retrieved text | Log to your own store; redact before logging |
| Analytics and error tracking | Often whole requests | Self-host or strip content from events |
The embedding step is the one people forget. If you embed documents with a hosted API, your whole corpus has gone to that provider even if the final answer is generated locally.
What are the options between fully public and fully private?
| Option | Who can see the data | Model quality | Effort to run |
|---|---|---|---|
| Hosted model API on standard terms | Your provider, under its stated retention and training policy | Highest | Lowest |
| Hosted model API with a no-retention agreement | Your provider, transiently | Highest | Low |
| Model in your own cloud account | Only you and the cloud provider's infrastructure | High | Moderate |
| Self-hosted open-weight model in a private network | Only you | Good, and improving; some tasks lag | Highest |
| On-premise or air-gapped | Only people on site | As above, limited by your hardware | Highest, with the most constraints |
Most businesses land on the middle options: a model hosted in their own cloud account, with contractual terms on retention and no training on their data. Fully self-hosted is worth the extra work when contracts, regulation or the sensitivity of the material require it.
What do you give up by keeping everything in-house?
Some model capability
The strongest models are usually offered as hosted services. Open-weight models have closed much of the gap for answering from supplied documents, which is what RAG asks of a model, but they can be weaker at long, multi-step reasoning. Test both on your own questions rather than assuming either way.
Operating work
Someone has to run the model servers, apply updates, watch capacity and plan for hardware. That is a real running cost, separate from the build. The comparison in what AI agents cost to run applies here too.
Speed of improvement
A hosted provider upgrades its model without you doing anything. A self-hosted one only changes when you change it, and an upgrade needs re-testing, as described in why an AI agent can get worse after a model upgrade.
How do you build a RAG chatbot that keeps data private?
- 01
Classify the data
Decide which documents are public, internal, confidential or regulated. The class sets how far from your control any of it may go.
- 02
Draw the data-flow map
List every component from the table above, where it runs and who operates it. Anything you cannot place is a leak you have not found yet.
- 03
Choose the deployment tier per class
Public material can use a hosted model; confidential material may need your own cloud account or a self-hosted model.
- 04
Enforce permissions at retrieval
Tag each chunk with who may read it and filter before anything reaches the model, so the chatbot cannot quote a document the asker could not open.
- 05
Control logs and retention
Decide what is logged, redact personal data, set a retention period and restrict who can read the logs.
- 06
Test quality on your own questions
Score answers and citations on real queries with the model you can actually use, and compare options before committing to one.
For the build itself, see how to implement retrieval-augmented generation, and for personal data specifically, keeping sensitive data out of an AI agent's training set.
What else should a private RAG setup include?
- Network controls: restrict outbound traffic from the servers so a misconfiguration cannot quietly send data out.
- Encryption: at rest and in transit, with keys you control.
- Source citations: every answer shows the documents it used, as in how a RAG chatbot cites its sources.
- Injection defences: private does not mean safe from malicious text inside documents - see prompt injection security.
- Regular review: check access, logs and answer quality on a schedule.
Can a RAG chatbot run without sending data to a third party?
Yes. Run the model, the embeddings, the vector store, the document parsing and the logging in your own cloud account, private network or on-premise, and check every component for where it sends data. Pick the lightest tier your data allows: a hosted model with no-retention terms, a model in your own account, or a fully self-hosted one, and accept the trade-off in capability and operating effort that comes with each step. Enforce permissions at retrieval and test quality on your own questions.
If this is a requirement for your project, our RAG development team can design it, and the security page describes how we handle client data.
