The Internet of Things (IoT) is one of the most widely used technologies in the world and security is a major threat to IoT networks. In this paper, we propose and implement a machine learning (ML) approach to design an intrusion detection and classification system (IDS) to detect cyber threats. The proposed work uses a novel feature selection approach, which indicates the most significant attributes from the high-dimensionality data. We use the Whale Optimization Algorithm (WOA) and the Genetic Algorithm (GA) for feature selection and the Simulated Annealing (SA) algorithm to optimize the model’s hyperparameters. We have applied the models in the IoT Intrusion Dataset 2020 (IoTID20) dataset to assess the effectiveness and sustainability of our proposed strategy. The results are concluded after performing the optimization algorithm on both the Extreme Gradient Boosting (XGBoost) and light gradient-boosting machine (LightGBM) models, we have achieved the highest accuracy of 99.2% with simulated annealing optimization on the WOA-selected features. In addition, we provide a complete development environment, validation environment, configurations, and extensive simulation results to better understand the proposed solution methodology.