Human matrix metalloproteinase 3 (MMP3), also known as Stromelysin-1, is involved in various cellular processes, including skin aging, making it an attractive drug target against skin aging. This study aims to apply different ML algorithms to develop a prediction model for the MMP3 inhibitor dataset (ChEMBL283) from the ChEMBL database. ML experiments were performed using the Python programming language. Seven machine learning algorithms, namely neural network, decision tree, Xgboost, CatBoost, random forest, LightGBM, and extra trees, were applied to classify molecules as active or inactive (coded 1 or 0) using AutoML. ML models underwent an evaluation process that included ROC plots, a confusion matrix, and a set of statistical measures. These evaluations demonstrated the exceptional predictive capability of the Extra Trees algorithm, achieving a remarkable accuracy rate of 85.8%. The most effective ML model identified 79 active MMP3 inhibitory phytochemicals in Nelumbo nucifera. Molecular docking confirmed the strong binding of seven phytochemicals to MMP3, suggesting their potential as inhibitors. Following Lipinski’s rule, three compounds—liensinin, isoliensinin, and isovitex—showed promise in molecular dynamics studies (100 ns) and MM-PBSA analysis (last 30 ns). They exhibited the lowest binding free energies, namely − 112.684 kJ/mol, − 194.871 kJ/mol, and − 101.551 kJ/mol, respectively, compared to the HQQ-MMP3 complex (− 95.410 kJ/mol), suggesting their potential as candidates for MMP3 inhibition. The study highlights the effectiveness of ML and the relative accuracy of MD simulations in screening phytochemicals for dermatological research and provides innovative opportunities for designing MMP3 inhibitors in the future.