Surveillance cameras create enormous amounts of data. Processing huge volumes of data requires an intelligent system to identify objects in varied conditions. The requirement of such a system is a primary concern and can be implemented in different locations which may include housing societies, parks, airports, railway stations, etc. Video is being recorded for a specific duration. The motion of an object plays a vital role in video surveillance videos. Moving objects are the primary candidates to analyze. Surveillance videos have an intrinsic property that most of the time camera would be capturing all static backgrounds without any alterations. This video usually does not have any acoustic features. Any important event is marked by the change or alteration in the environment which could mark the beginning of a normal or an abnormal event. The motion of an object or person is of the most important and examinable features which can assist in a better decision-making process. In this paper, a Motion Aware Video Surveillance system is proposed which takes spatial and spectral features into consideration to detect and track the objects of interest in videos.