The early detection of colorectal cancer (CRC) is critical and yet persistently challenging objective for clinical practice. This paper introduces a robust transformer based framework that integrates Deep learning (DL) models (Convolutional Neural Networks: ADaRDEV2I-22 and ResNet-152), Vision Transformers (ViTs), and XGBoost to enhance CRC classification from medical images. The DL model performs high-quality initial feature extraction, while ViTs effectively model long-range spatial dependencies often missed by traditional architectures. XGBoost is employed for final structured data classification. This helps in boosting diagnostic precision and robustness. In order to address the imbalance nature of the dataset, an improved generalization, synthetic data generation has been employed using Generative Adversarial Networks (GANs) and standard augmentation techniques. Under 5-fold stratified cross-validation (the primary evaluation protocol), the framework achieves an accuracy of 95.8 ± 0.7%, precision of 94.8 ± 0.9%, recall of 95.1 ± 0.8%, and F1 score of 94.9 ± 0.7%, surpassing standalone DL models and ViT models by 4.1% – 5.2 percentage points over the best baseline (90.6%). The data augmentation techniques led to a 7.3% gain in recall for underrepresented classes. Furthermore, model interpretability was increased through SHAP, and LIME. Despite its complexity, the framework maintains computational efficiency, with only a 12% increase in training time compared to single-model systems. The proposed methodology offers an interpretable, scalable and high-performance solution for DL based transformer based colorectal cancer detection, supporting its integration into real-world clinical decision support systems.