Between the year 1993 and 2019, a considerable new and different metaheuristic optimization techniques have been presented in the literature for automated cryptanalysis of classical substitution cipher. This paper compares the performance of these new and different metaheuristic techniques. Three main comparison measures are considered to assess the performance of presented metaheuristics: efficiency, effectiveness, and success rate. To the best of author knowledge, first time this kind of review has been carried out. It is noteworthy that among the presented metaheuristics, the performance of genetic algorithm technique is best with respect to effectiveness and success rate.