Severity classification is the most pivotal task in Alzheimer’s disease diagnosis. Detection of brain structural changes from brain MR images is crucial for Alzheimer’s classification. In this paper, we have proposed a transfer learning and residual separable convolution network for the classification of Alzheimer’s. The proposed network includes three separable convolution layers with two average pooling layers. An upsampling has been performed to regain its spatial resolution for the residual connection. The main intuition of separable convolution is to optimize parameters with depth-wise convolution. Similarly, the residual connection has been used to reduce the vanishing gradient problem. Finally, a three-layer fully connected dense network has been used for the four-class Alzheimer’s classification. Kaggle dataset has been utilized for the experiments to report results. We have achieved an accuracy of 97.32% on the dataset with five-fold cross-validation. Our model has reported an improvement of 1% in jaccard similarity and outperforms the competing models in all vital metrics.