With the change from laboratory controlled to challenging facial expression recognition (FER) in the wild and the recent success of deep learning techniques in different fields, deep neural networks have been increasingly leveraged for automated FER to learn discriminatory representations. Here, in this survey, we include a brief overview of deep FER literatures and provide insights into some essential issues. Firstly, we represent the existing datasets that are widely used for the purpose and then we define a deep FER system’s standard pipeline with the associated context information and suggestions for applicable executions for each level. We then present already existing novel deep neural networks (DNN) and related training approaches for the state-of-the-art deep FER techniques that are optimized on the basis of both static and dynamic image sequences. A competitive comparison of the experimental works is also presented along with an analysis of relevant problems and implementation scenarios. Lastly, an overview of the obstacles and appropriate opportunities in this area is presented.