Face recognition has been a dominant biometric modality in surveillance and access control systems. The performance of biometric characteristics are usually impaired by the changes in illumination, occlusion, and expression. Conventional Local Binary Pattern (LBP) descriptors are computationally efficient but are vulnerable to noise and light. To overcome these problems, this paper provides a comparative analysis of Local Ternary Pattern (LTP) variants such as Rule-based LTP (RLTP), Directional LTP (DLTP) and Modified LTP (MLTP) to unconstrained face datasets. All the variants are tested based on their capability to encode strong local texture features and maintain discriminatory information in unfavorable circumstances. Extensive testing is done on standard data sets such as ORL, YALE and Extended Yale B. The experiments are performed using classifiers, such as Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Decision Tree (DT). The findings indicate that all the LTP variants behave better than the traditional LBP during changes in illumination, but the combination of directional and ternary encoding provide better resistance to occlusions and changes in expressions. The paper identifies the swapping between accuracy and complexity of computation. This paper also gives the essential information of the lightweight face recognition systems in real time smart surveillance systems.