Abstract
The LIC (Life Insurance Corporation) has many different types of policies with increasing complexity, and additionally, with the increase in demand for personalised digital services, there is an expectation of intelligent recommendation systems that can interpret user intent and provide relevant product suggestions that extend beyond the use of keyword matching. We developed a semantic retrieval-based LIC Policy Recommendation System using transformer embedding models to find the best matching LIC Policy for each user’s needs. We compiled all LIC Policies into a single canonical text corpus and created a domain-specific dataset by combining this corpus with user query data. We evaluated many forms of embedding approaches (TF-IDF, transformer, etc.) and focused primarily on Sentence-BERT (MPNet) and an LIC domain term fine-tuned version. LIC Policies were indexed using FAISS and HNSW to allow for scale via efficient similarity searches across thousands of policies. The experimental results showed that the fine-tuned system consistently outperformed the base system in all major information retrieval metrics (Precision@1, Precision@2, Recall@5 and Mean Reciprocal Rank [MRR]). More relevant policies found in higher-ranking retrieves indicate a strong semantic understanding and alignment with user intent for this system. This research lays the groundwork for developing a production-level LIC recommendation system, demonstrating the benefits of utilizing both domain-adaptive embeddings and hybrid retrieval methods. Continuing enhancements could include providing richer fine-tuning data, BM25 Lexical Signals reranking, and offering real-time recommendations based upon user profiles.