Heart disease prediction is important in healthcare because it enables timely identification and intervention of actual condition of the patient. However, the task of accurately predicting disease remains a challenging task. In this paper, we have proposed a framework for heart disease prediction using explainable artificial intelligence (XAI) based Machine Learning (ML) models such as Decision Tree (DT), Random Forest (RF), k-nearest neighbors (KNN), AdaBoost, Logistic Regression (LR), Naive Bayes (NB), and Neural Network (NN). The efficiency of those models were evaluated using MCC, accuracy, precision, recall, and AUC. Finally, it is observed that, DT emerges as the most effective model offering interpretability. This study underscores the importance of transparent models in healthcare and advocates in order to incorporate XAI to enhance interpretability and medical decision-making.