The generation of artificial intelligence (AI) based chest X-ray medical reports assists radiologists by saving time and improving decision-making through explainable AI techniques. Chest X-rays are extensively used to detect a variety of conditions, like pulmonary diseases and granulomatous disorders. However, existing methods for medical report generation often struggle with consistency due to their reliance on convolutional features. The proposed approach integrates gradient weighted class activation mapping (Grad-CAM) with a multilayer perceptron (MLP) and long short-term memory (LSTM) networks to produce coherent medical reports. GradCAM generates weighted localized features from activation maps of the last convolutional layer, which are ranked in descending order. These ordered features are processed by the MLP to capture feature dependencies and are subsequently analyzed by the LSTM with impressions of medical reports. The proposed model was tested on the Indiana University X-ray (IUX) dataset, which has chest X-ray images and corresponding medical reports. The impressions in medical reports are considered ground truth when training the LSTM model. EfficientNet-B0, DenseNet169 are the convolutional neural network (CNN) models that have been trained with X-rays of IUX. Integrating Grad-CAM with CNN+LSTM have significant improvement in BLEU and ROUGE scores.