Early detection of colorectal polyps is crucial in preventing colorectal cancer. While convolutional neural networks (CNN) have achieved overwhelming success in classifying medical images, they face challenges in accurately categorizing different types of polyps from colonoscopy images due to their varying sizes and shapes. To address these issues, we introduce a wavelet pooling-based multiscale contextual fusion network called WMCF-Net for efficient polyp classification. The WMCF-Net comprises a sequence of multiscale contextual fusion (MCF) blocks and different pooling layers in its three branches. The MCF block aids in capturing contextual features at various scales. The features from each branch are finally fused to learn more detailed information from colonoscopy images comprehensively. The network also leverages wavelet pooling for feature map size reduction while preserving more detailed spatial information. The WMCF-Net model is evaluated on four benchmark datasets: Kvasir-V2 and CP-CHILD dataset for poly vs. no polyp classification, and Piccolo and Endoscopic dataset for adenoma vs. hyperplastic classification. The proposed model obtains a classification accuracy of 98.00% and 99.75% on the Kvasir-V2 and CP-CHILD datasets, respectively. While an accuracy of 99.80%, 100.00%, and 93.14% is achieved on set-1 and set-2 of the Endoscopic dataset and Piccolo datasets, respectively. The ablation studies and comparisons with state-of-the-art methods further demonstrate its efficacy. The WMCF-Net is end-to-end learnable, lightweight, and can hence be efficiently utilized in real-time clinical applications.