In colorectal cancer analysis, polyps segmentation is one of the crucial task where encoder–decoder style architecture plays a significant role as a base model. However, it suffers from the issue of loosing contextual and spatial information, which ultimately results poor performance. To address these issues, we introduce a residual multi-scale semantic polyp segmentation approach named MSPolypNet for efficient polyps segmentation. MSPolypNet improves contextual understanding and preserves spatial information by integrating innovative modules namely, residual multi-path atrous spatial pyramid pooling block (RMAB) and cross-spatial attention (CSA) enriched with dilated convolutions to capture intricate details across varying scales while maintaining computational efficiency. The proposed model was rigorously trained and tested on six independent datasets. Additionally, to assess cross-dataset performance, two separate datasets not used during training were exclusively utilized for testing. MSPolypNet achieved Dice Score and mIoU scores of 90.86% and 88.75% on the Kvasir-Seg and 94.92% and 90.56% on the CVC-ClinicDB, demonstrating MSPolypNet robustness and efficiency. Experimental results show a substantial improvement in segmentation accuracy, highlighting the potential of the proposed model to become a new benchmark for polyp segmentation. Its fewer parameters, compared to other models, provide an advantage for using our MSPolypNet in reliable -time clinical evaluations.