RAG — Retrieval-Augmented Generation — is behind many of the “AI chatbot trained on our documents” projects libraries are exploring.
The good news? You don’t need a huge technical stack to start.
You need a defined document set, good retrieval, source citations, and a chatbot that knows when it doesn’t have the answer.
Here’s the basic roadmap.
*1. Start small*
Don’t begin with everything your library owns.
Choose one collection: FAQs, policies, research guides, catalogue records or an institutional repository.
Scope matters more than most people realise.
*2. No-code option: n8n*
If you don’t want to code, n8n can handle much of the workflow — document processing, chunking, embeddings, vector database connections and the chatbot.
AI4LIB has a useful walkthrough by Bhavesh Patel:
Building a RAG Chatbot with n8n https://youtu.be/EFGtmFyrKFk
It covers the process from chunking and embeddings to a working chatbot.
*3. Clean and chunk documents*
• LangChain — python.langchain.com Open source, with document loaders and text splitters.
• LlamaIndex — llamaindex.ai Open source and focused strongly on working with your own data.
*4. Create embeddings*
• sentence-transformers — www.sbert.net Free, open source and can run locally.
• OpenAI embeddings — platform.openai.com/docs/guides/embeddings Paid, but relatively inexpensive for many small projects.
*5. Choose a vector database*
• Chroma — www.trychroma.com — free and open source.
• Qdrant — qdrant.tech — open source and free to self-host.
• Weaviate — weaviate.io — open source with a free managed option.
*6. Connect a language model*
Claude is one option for working with retrieved context.
Ollama is worth considering if you want to run open-source models locally, especially where privacy and API costs matter.
*7. Cite every answer*
For a library chatbot, this is non-negotiable.
Users should be able to see where an answer came from and return to the original document.
*8. Test what happens when the answer isn’t there*
Ask questions with missing, ambiguous or outdated information.
The chatbot should be able to say:
“I don’t have enough information to answer that.”
A confident guess is worse than admitting it doesn’t know.
*9. Plan for updates*
Policies change. Guides are revised. Old documents become obsolete.
Decide how new versions enter the system and how older versions are removed.
For a first build with zero budget and no coding, I’d consider:
*n8n + Qdrant*
For a coded, self-hosted setup:
*Chroma + sentence-transformers + Ollama*
You don’t need the perfect architecture on day one.
Start with one collection. Make it cite sources. Test its limits.
That’s where the real learning begins.
*Has your library attempted a RAG build yet? Where did it get stuck?*