Alzheimer’s Disease (AD) has emerged as one of the serious causes of death worldwide effecting the elderly population. As the development of this disease is related to neuronal loss in brain sub regions, the focus of this paper is on segmenting the brain into sub regions like GM (Grey Matter), WM (White Matter) and CSF (Cerebrospinal Fluid) followed by AD classification. This paper presents a complete framework for AD classification using feature maps of segmented tissue of brain. Two novel approaches have been proposed for classification task, where in first approach Self-Augmenting CNN and in the second approach Modified ResNet model has been used. Segmentation model achieves the highest similarity score of 99.53% for DS (Dice Similarity) and 99.09% for IOU (Intersection of Union) on GM segment. For Multi-class classification, accuracy of 97.92% is achieved with Self-Augmenting CNN, while Modified-ResNet model achieves 98.75% accuracy on segmented feature maps.