Early detection of colorectal cancer (CRC) can improve the survival rate with a better treatment plan. It is essential to classify the pathological findings of the gastrointestinal (GI) tract to determine GI diseases, such as ulcers or cancerous polyps. The diagnosis of a pathological finding is performed using an automated computer-aided diagnosis (CAD) system. It is always a challenging issue between computational complexity and performance accuracy when developing a CNN model. The proposed work focuses on the development of an improved MobileNet with reduced computational complexity and better accuracy, aiming at deployment in resource-constrained devices such as FPGAs. The proposed MobileNet is an improved version of MobileNet-V2 with efficient use of depthwise separable convolution, along with a multi-kernel feature fusion mechanism to improve performance. The reduction in the layers is compensated with cross-bottleneck feature fusion to improve performance with lower computation. The performance of the proposed model was evaluated with the Kvasir dataset for the pathological finding, and it achieved a classification accuracy of 95.6% with a significant reduction in computational parameters. The hardware efficiency was validated by deploying the model on an FPGA, giving a throughput of 457 frames per second with a better accuracythroughput trade-off. The performance of the proposed model was also validated and compared with other SoTA CNN models, and it outperformed other models.