Handwritten script identification plays an important role in Optical character recognition (OCR), specially when multilingual and regional language settings like Odia are present. A comparative analysis of K-Nearest Neighbours (KNN) and Support Vector Machine (SVM) classifiers for distinguishing between Odia and Non-Odia handwritten characters can be found here. Two feature representations are being analysed: raw greyscale image pixels and Histogram of Orientated Gradients (HOG) descriptors for better accuracy. Our Experimental results demonstrate that HOG features significantly enhance classification performance. The SVM classifier attains up to maximum accuracy of 91.99% utilising HOG, whereas KNN usually achieves 88.44%. Using raw image data yields lower accuracy (i.e. 55.33 for SVM and 44.20 % for KNN), on the other hand. Importance of skilful feature extraction for improving model performance is emphasised in this study. SVM exceeds KNN because of its resilience in high-dimensional environments. From this we can conclude that by using affective preprocessing on dataset, the performance of model accuracy can be significantly high and it also suggests available options, such as dataset augmentation and deep learning.