Adversarial patches are assaults on deep learning networks that try to misclassify image data by altering pixels in a confined region. Patch assaults are a threat to several current Artificial Intelligence (AI) systems because they have been physically and digitally realized. Although significant research has been conducted to develop defenses against such attacks, the evolution of Explainable Artificial Intelligence (XAI) has opened new avenues for identifying adversarial behavior through perturbation-based attribution analysis. Additionally, most defenses cannot distinguish between malignant and benign inputs, which forces the constant use of computationally expensive defense methods. In light of considering the aforementioned shortcomings, a perturbation-based attribution framework is proposed that detects adversarial patches using different methods such as Occlusion, Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive explanations (Kernel SHAP) and provide a comparative analysis of these three methods alongside an evaluation of contemporary gradient-based attribution techniques. Experimental results using the Kernel SHAP method demonstrated a high patch detection accuracy of 90.86% which indicates its potential effectiveness compared to other current algorithms.