Electroencephalogram (EEG) is widely used in emotion recognition which is achieved by recording the electrical activity of the brain. It is a complex signal because of its high temporal resolution and thus requires sophisticated methods and expertise to be interpreted with a reasonable degree of accuracy. Over the years, significant strides have been made in the field of supervised and unsupervised feature learning from data, using deep architectures. The purpose of this study is to build on some of these improvements for better classification of emotions from EEG signals. This is a rather challenging task, and more so if the data we are reliant on is noted for being unsteady, as it changes from person to person. There is a need for an intricate deep learning algorithm that can achieve high levels of abstraction and can still dish out robust/accurate results. In this paper, we have used the Xception (Chollet in 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1800–1807, 2017 [1]) model from Keras API, further reinforced by fine tuning, to classify emotions into three categories namely NEGATIVE, POSITIVE and NEUTRAL. An open-source EEG dataset from Kaggle (Bird et al. in The international conference on digital image and signal processing (DISP’19). Springer, Berlin, 2019 [2]) was used in this study for the purpose of classification. Our experimental results achieved a precision score of 98.34%, a recall value of 98.33%, and an F1-score of 98.336%. This result outperforms many other popular models based upon support vector machine, k-nearest neighbor, self-organizing maps, etc., whose accuracy usually ranges from anywhere between 53 and 92%.