The memory function formalism is an important tool to evaluate the frequency dependent electronic conductivity. It is previously used within some approximations in the case of electrons interacting with various other degrees of freedom in metals with great success. However, one needs to go beyond those approximations as the interaction strengths become stronger. In this work, we propose a systematic expansion of the memory function involving its various moments. We calculate the higher order contribution to the generalized Drude scattering rate in case of electron-impurity interactions. Further we compare our results with the results from previously studied lowest order calculations. We find larger contributions from the higher moments in the low frequency regime and also in the case of larger interaction strength.