Polyp segmentation is essential for the early detection and treatment of colorectal cancer using colonoscopy images. The computer-aided diagnosis (CAD) systems assist colonoscopists in improving the diagnosis process and reducing missed detection rates. One of the major challenges in polyp segmentation is the variability in the size and shape of the polyps that affect the segmentation performance. Another significant difficulty is distinguishing between the polyps and the surrounding tissues. This research proposes an encoder–decoder based architecture that incorporates a Global Cross-Dimensional Attention (GCDA) module to capture complex patterns, facilitating cross-dimensional interactions with a global perspective. This enhances segmentation accuracy by providing broader contextual information, effectively addressing the issue of variability in polyp size. Additionally, the feature extraction is further enhanced using a Scale-Aware Feature Extraction (SAFE) to address the issue of polyp shape and size variability more effectively. The proposed method combines Local-Global Features Fusion (LGFF), which effectively differentiates polyps from the surrounding mucosa. The proposed architecture was assessed on two benchmark datasets and achieved a mean Intersection over Union (mIoU) of 0.952 and 0.893 on the CVC-ClinicDB and the Kvasir-SEG dataset, respectively. The proposed model outperformed nine state-of-the-art (SOTA) models in terms of performance. The proposed model demonstrated strong performance in the generalizability tests involving cross-dataset evaluation along with two additional datasets: Hyper Kvasir and ETIS Larib dataset.