Mammography is an inexpensive and noninvasive imaging tool that is commonly used in detection of breast lesions. However, manual analysis of a mammogramic image can be both time intensive and prone to unwanted error. In recent times, there has been a lot of interest in using computer-aided techniques to classify medical images. The current study explores the efficacy of an Earth Mover’s Distance (EMD)-based mammographic image classification technique to identify the benign and the malignant lumps in the images. We further present a novel leader recognition (LR) technique which aids in the classification process to identify the most benign and malignant images from their respective cohort in the training set. The effect of image diversity in training sets on classification efficacy is also studied by considering training sets of different sizes. The proposed classification technique is found to identify malignant images with up to 80 % sensitivity and also provides a maximum F1 score of 72.73 %.