Breast cancer persists as a major worldwide health issue, and increasing patient survival rates require an early and precise diagnosis. To divide breast ultrasound images into benign, malignant, and normal categories, this research compares two deep learning architectures: The Vision Transformer (ViT) and the Convolutional Neural Network (CNN) based ResNet-101. Addressing the common challenge of limited data, we employed a data augmentation strategy to expand a benchmark dataset of 780 images to over 10,000 images, creating a robust training set. Both models were trained on this augmented dataset, achieving test accuracies of 98.64% for the Transformer model and 97.57% for the Resnet-101 model. The result indicates that the ViT model performed better in terms of accuracy than the ResNet-101. Furthermore, the existing Deep learning models are black box models. To increase model openness and foster clinician confidence, Gradient-weighted Class Activation Mapping (Grad-CAM), an Explainable AI (XAI) technique, is utilized to generate visual heatmaps, emphasizing the specific regions in the ultrasound images that were most influential in the models’ diagnostic decisions. The proposed model harnesses GPU-based parallel infrastructure. The results indicate that the ViT model outperforms the other existing models.