Among other deep learning (DL) architectures, the convolutional neural network (CNN) has wide applications in speech recognition, face detection, natural language processing, and computer vision. Multiply and Accumulate (MAC) unit is a core part of CNN and requires large computations and memory resources. They result in more power dissipation for low-power embedded devices. Hence, the hardware implementation of CNN to produce high throughput is one of the challenges nowadays. Therefore, sparsity is introduced in weights by a non-linear method with a minor compromise in accuracy. Experimental results also show the enhancement of 52% sparsity with a 4% loss in accuracy. In addition, an indexing module is proposed to perform Single Instruction Multiple Data (SIMD) operations in the fully connected layer to perform only effective operations without multiplication. This module is used along with sparsity to offer better results as compared to SOTA methods. Cadence RTL compiler results show that the proposed indexing module saves 1.3 nJ of energy as compared to the existing methods.