The field of Facial Emotion Recognition (FER) has advanced considerably in the last few years. Much research relies on lab-controlled datasets, characterized by limitations in size, quantity, and quality. These datasets feature high-resolution static images captured in ideal conditions but lack fidelity in representing real-world scenarios. Hence, FER systems must be trained on primary data that includes real-world scenarios like facial expressions captured from various angles and in different lighting conditions, images with occlusion etc., broadly termed as unconstrained environment. To leverage the gap, this study emphasizes utilizing an AffectNet dataset that has samples close to real-world scenarios. In addition, we propose a novel ensemble framework to increase the accuracy of emotion recognition by harnessing the complementary strengths of three distinct deep-learning models: DenseNet169, EfficientNetB7 and InceptionV3. The key innovation lies in our novel ranking-based fusion technique, which introduces a unique perspective on model confidence and its relationship with prediction quality. The rank-based fusion approach optimally harnesses each base model’s unique characteristics and strengths. Our experiments confirm the ensemble framework’s effectiveness, outperforming individual models in facial emotion recognition.