Abnormal event detection in videos has predominantly been explored as a one-class or binary classification problem, limiting the granularity and interpretability of detected anomalies. This paper introduces the Squeeze-Attentive Encoder Network (SAE-Net), a novel encoder architecture that integrates semantic-driven analysis with a Squeeze-and-Excitation (SE) attention mechanism. By emphasizing semantic features, the model effectively discerns and categorizes various types of anomalous events within video sequences, enabling precise multi-class classification. SAE-Net is thoroughly evaluated on the CUHK Avenue and ShanghaiTech datasets, well-known large benchmarks for video anomaly detection. The inclusion of the SE attention mechanism into deep convolutional neural network enhances the network’s ability to focus on semantically significant features within complex video frames, leading to more accurate and reliable anomaly classification across multiple categories. Experimental results highlight the model’s superior performance, demonstrating its robustness and applicability in real-world scenarios, particularly in surveillance and security systems.