RAG: How to Connect AI to Company Data
By early 2025, Sber publicly announced the industrial deployment of RAG systems based on corporate documents. By mid-year, VTB and Yandex made similar announcements, signaling the transition of the technology to the industrial implementation stage. According to industry research, the use of RAG in corporate projects doubled in 2024 compared to 2023, and growth continued in 2025. An early 2026 benchmark from Onyx showed that a properly configured RAG system answers work-related questions better than ChatGPT, Claude, and Notion AI in 64–76% of cases. The numbers are impressive, but they are backed by a specific architectural idea that should be understood before implementation.
In our practice at West Star Ltd, we encountered a typical request: "We want AI to answer questions about our regulations and knowledge base, but the data should not leave the company's perimeter." This is a request specifically for RAG. Let's explain how it works, what it offers, and where it doesn't work.
WHAT IS RAG AND WHY IT'S NOT JUST "UPLOAD DOCUMENTS TO CHATGPT"
RAG stands for Retrieval-Augmented Generation. The idea is simple: when a user asks a question, the system first searches your documents for the most relevant fragments, then provides them to the language model as context. The model answers based only on what the search system found—not on its training, not on the internet.
This is fundamentally different from uploading documents to a cloud chat. In a cloud chat, the file is entirely sent to the provider (OpenAI, Anthropic, Google). With RAG, the data stays with you, and only a small fragment needed for answering a specific question goes outside.
Technically, the chain looks like this. Documents are transformed into vector representations—numerical imprints of the meaning of each fragment. These vectors are stored in a special database. When a question is asked, it is also turned into a vector, the system finds the closest fragments in meaning from the database, and passes them to the language model. The model generates an answer based on these fragments and indicates the source—the specific document and section.
WHAT THE ARCHITECTURE LOOKS LIKE IN PRACTICE
In a typical corporate RAG system, there are three levels.
The first is loading and indexing. Documents (PDF, Word, Excel, knowledge bases, websites, reports) enter the pipeline, are divided into fragments of 200–500 words, each fragment is turned into a vector using a special model, and recorded in the vector database along with metadata: file name, date, department, access level.
The second is search. The user's request is processed by the same model, the vector database returns the top-5 or top-10 most similar fragments. Additionally, classical keyword search (BM25) can be applied and the results mixed—this is called hybrid search. It is more accurate for queries with specific terms, numbers, or names.
The third is generation and output. The language model receives the question plus the found fragments and forms an answer. A good system always indicates the source: "According to regulation IOT-14, section 3.2." This allows the user to verify the information independently.
In terms of infrastructure, all this can work on a single server with 32–64 GB RAM and a graphics card from 16 GB (or without it—slower, but possible). Qdrant—a vector database in Rust—processes 10 million vectors on a VPS for $30–50 per month in 2026. If a company's knowledge base is a few thousand documents, no expensive hardware is needed.
HOW DATA STAYS WITHIN THE COMPANY
The key question when implementing RAG is where the language model resides. If a cloud provider's API is used, document fragments go to external servers with each request. This may not comply with security policies, regulatory requirements, or client contract terms.
The alternative is local open language models. In 2026, their quality on corporate text tasks closely approached commercial solutions. Such models are deployed on a corporate server and do not transmit data anywhere. The request is processed entirely within the company's infrastructure.
Additionally, access control is configured at the vector database level. Contextual access control allows specifying: HR department documents are seen only by HR, security regulations—only by employees with clearance. The search system filters out fragments to which the requester has no rights when forming an answer. The user will not receive closed information, even if they ask a direct question.
Microsoft GraphRAG, introduced in 2024, offered another approach: instead of a flat vector index, a knowledge graph is built where entities—people, products, processes—are connected by explicit relationships. This significantly improves answers to questions requiring aggregation from multiple sources: "Who is responsible for process X" or "How have rules Y changed since last year." In practice, GraphRAG is useful where documents have many cross-references; for simple regulation searches, standard vector RAG is more effective.
SCENARIOS WHERE RAG WORKS
It's important to understand: RAG is not a universal answer to all document tasks. It works well where information needs to be found and explained from a large corpus of texts. Below are four scenarios where the technology has a real effect.
Internal knowledge base and regulations. Employees ask "how to arrange business trips" or "what is the response time to a claim" and receive an accurate answer with a document link. Instead of searching through folders and reading a forty-page regulation—one question.
Technical documentation. Engineers find equipment parameters, maintenance instructions, fault logs. The system works as a smart search. When instructions are updated, the new version automatically replaces the old one in the index—an engineer doesn't risk working with an outdated scheme.
Client support. Operators ask a question in the interface, and the system instantly finds the required contract clause, tariff plan, or return procedure. Processing time is reduced, answer quality becomes more stable.
Onboarding new employees. Instead of burdening an experienced colleague with repetitive questions, a newcomer asks them directly to the system. Answers appear instantly with a source link.
Contract and legal document analysis. A lawyer asks a question about a package of contracts—the system finds all mentions of a specific condition or term without manually rereading each file.
LIMITATIONS AND WEAKNESSES
Quality depends on document quality. RAG does not correct contradictory or outdated regulations—it reproduces them. If there are three versions of one document in the knowledge base, the system may answer based on an outdated one. Preparing documents for indexing is a separate task often underestimated during planning.
Hallucinations do not disappear completely. The language model may invent details not contained in the found fragments—especially if there is no relevant document in the base at all. It's important to train users to check the source the system indicates next to the answer.
Non-standard formats are poorly parsed. Scanned PDFs without a text layer, tables with complex structures, diagrams, and drawings require separate tools. OCR and converters add errors that then appear in answers.
Maintaining relevance requires a process. RAG is not a one-time implementation. When regulations are updated, old vectors need to be removed and new ones added. Without automation, the knowledge base gradually becomes outdated, and trust in the system falls.
Slow start with a large document base. Initial indexing of thousands of documents can take hours. Incremental updates require thoughtful implementation from the start.
Small companies overestimate complexity and do not start. RAG from open components is a realistic project for several weeks. But it is perceived as an "AI project" with an unclear budget and postponed.
HOW TO ASSESS COMPANY READINESS FOR RAG
Three questions to help make a decision. First: are there at least 200–300 documents that employees actually search for—regulations, instructions, contracts? Second: are there recurring questions that take more than 10–15 minutes to find an answer? Third: is there a requirement for data not to leave the company's perimeter?
If all three answers are "yes"—RAG will provide measurable results even in a pilot. If the knowledge base is poorly structured or hasn't been updated in the last couple of years—document work is needed first.
For a pilot, one department, one category of documents, and an open stack are enough: LangChain or LlamaIndex as an orchestrator, Qdrant or Chroma as a vector database, Ollama for a local language model. The first prototype—in one to two weeks with a dedicated developer. The main thing from the business side is a clear list of documents and one department ready to provide feedback.
CONCLUSION FOR AUDIENCES
Specialist. RAG is a mature technology with an open stack. Qdrant, LlamaIndex, Ollama are deployed locally in a few hours. The main difficulty is not in the code, but in the quality of the document processing pipeline and access control setup. Spend time on the fragment splitting strategy—too short or long pieces directly affect search accuracy. Start with one type of document and iterate.
Manager. The main metric is employee time spent searching for information. Measure the current average, implement a pilot in one department, measure again. The project justifies itself with a 30% or more reduction in search time. It's important to ensure a document update process—without it, the system degrades.
Owner. RAG provides a competitive advantage where company knowledge is accumulated but poorly accessible. It speeds up the adaptation of new employees, reduces dependence on expertise holders who may leave, and creates a foundation for the next step: AI agents that not only search but act. With the right architecture, data remains entirely within the company.
FREQUENTLY ASKED QUESTIONS
— Is it necessary to purchase expensive equipment?
For a typical corporate knowledge base of several thousand documents, a server with 32–64 GB RAM is sufficient. A GPU is desirable for speed but not mandatory at the start. The cost of a cloud VPS ranges from $30 to $150 per month depending on volume.
— Can RAG be used with existing accounting systems?
Yes, but an integration layer is required. Data is exported, cleaned, and indexed. Changes are synchronized on schedule or in real-time. A standard task, but requires separate planning.
— Which is better—cloud RAG or self-hosted?
Cloud is faster to launch. Self-hosted provides full control over data and lower operational costs as volume grows. For companies with data localization requirements—self-hosted is preferable.
— How to ensure the system does not give outdated answers?
Set up mandatory source indication in each answer—the employee sees a link to the document and update date. Additionally, monitor changes: if a file is updated, the system marks related fragments as requiring re-indexing.