Infrastructure plays an important role in the development of any nation. Roads are very important for the movement of humans and goods. Good road infrastructure reduces travel time and cost. Poor road infrastructure can cause an increase in travel time, more consumption of fuel and lead to accidents due to difficulty in driving. All these factors will have a cascading effect on the economy as the prices of essential commodities are directly related to the easy of transportation. Hence, it is necessary to detect the distresses on roads like potholes and do regular repairs. Initially, manual inspection was done by the concerned employers, who go check the roads and take note of the areas where there is a requirement of repair. This method involves a lot of time and labor. With the development in technology, the focus on implementing a smart method of inspection increased. To build an accurate and quick detection system, an Earth Mover’s Distance (EMD) based model is proposed in this paper. This algorithm finds the presence of road distress by comparing the image of the current testing road with that of the reference image. The experiment also considers the chance of occurrence of Gaussian noise and Salt and Pepper noise while capturing the images, to fit the real-world situations.