Published · IEEE ICCCNT 2025
Shakthivel Arumugam, Arsh Pawar, Siddhesh More, Shivraj Murali, Rizwana Shaikh, Deepali JagtapSIES Graduate School of Technology, Navi Mumbai
IEEE ICCCNT 2025, IIT Indore, 2025
Engineering students face challenges in accessing reliable information due to the proliferation of unverified online resources and the potential for inaccuracies in Large Language Model (LLM) outputs. This paper presents a novel question answering system that leverages a vector datastore of researched reference books and an LLM for generating responses. A key contribution is a confidence score metric that evaluates the relevance of the retrieved context to the user's query, combining keyword matching, short-term filtering and key term frequency analysis so that the LLM is given pertinent information. The paper analyses the limitations of existing answer evaluation methods in assessing context relevance and highlights the novelty of the proposed metric in its direct, interpretable, keyword-focused approach.