Abstract
Conventional web scraping and information retrieval systems are predominantly designed for data extraction; however, they lack the ability to provide context-aware and user-oriented responses. These systems typically retrieve unprocessed data with inadequate semantic interpretation. It leads to inefficiencies in obtaining meaningful insights. To address these limitations, we propose a Retrieval-Augmented Generation content extraction model (RAGBOT). It is an AI-based web scraping and question-answering system. It leverages Retrieval-Augmented Generation (RAG) techniques alongside lightweight Large Language Models (LLMs). RAGBOT integrates semantic search with real-time scraping to enable efficient retrieval and comprehension of online articles. Our method demonstrates superior performance in terms of accuracy (89.4%) and response time (3.8 s) compared to conventional retrieval and summarization systems. The proposed method was evaluated on standard hardware to assess its performance. The results demonstrate its practicality and scalability, confirming it as a robust solution for intelligent web-based information retrieval.