Brain tumor grade detection is one of the perpetual tasks in brain image classification. Deep learning models are the most successful for multi-class classification which are trained for non-medical image classification. Thus, there is a need for re-training and feature enhancement for better performance in medical image classification. In this paper, we have proposed a residual multi-head attention network to uplift the retraining process with polished feature extraction. The proposed model consists of three parts including a pre-trained EfficientNetB4, a residual multi-head attention network, and a dense network. The residual multi-head attention network utilizes the attention block with three convolution layers for better tumor detection. The residual connection used in the network avoids the vanishing gradient problem. We have extracted a two-class (low-grade/high-grade) dataset from REMBRANDT repository. The proposed model has attained an accuracy of 96.39% and outperforms its competing models in vital metrics.