For agricultural production increase, early detection of plant diseases and nutrient deficiencies is essential. When it comes to identifying leaf patterns caused by nutritional deficiencies, deep learning techniques are highly effective. In this paper, we propose a novel hybrid method for classifying nutrient deficiencies, combining feature extraction approaches such as MobileNetV3, EfficientNet B0, Vision Transformer, and Swin Transformer with an SVM classifier. Among these proposed HybridSoilNet, the Swin Transformer with SVM achieved 99.2% accuracy in recognizing five nutrient deficiencies: iron, magnesium, nitrogen, potassium, and zinc. For interpretability, Grad-CAM++ was used to produce heatmaps, highlighting nutrient deficiency features better than other state-of-the-art (SOTA) models. This work proposes an efficient smart agriculture system improving production oversight and early disease detection.