AAmogh·Panhale
All projects
LLMRAG2026

Website Crawler (RAG)

Crawls URLs into a vector database queryable through a local Ollama LLM.

Code

Problem

Ad-hoc web content is hard to ask questions against; you need it chunked, embedded, and retrievable by a local model.

What I built

  • Built a crawler that ingests a list of URLs and builds a vector database from the content.
  • Wired retrieval into a locally-hosted Ollama model for question answering over the crawled corpus.

Impact

  • TODO(amogh): repo has no README — tell me the embedding model, vector store, and use case and I'll finish this.

Stack

PythonOllamaVector DB