Security is a major concern in vehicular networks for reliable communication between the source and the destination in smart cities. Data, these days, is in the form of safety or non-safety messages in formats like text, audio, images, video, etc. These information exchanges between the two parties need to be updated with a trust value (TV) by analyzing the communication data. In this paper, a machine learning-based misbehavior detection system (ML-MDS) is proposed for cognitive software-defined multimedia vehicular networks (CSDMV) in smart cities. In the proposed system, before communication, the vehicle must be aware of the TV of other vehicles. If the TV for a vehicle is higher than a threshold (th), then the communication happens and the whole transaction information is sent to the local software-defined network controller (LSDNC) for classification of behavior using the ML algorithm. After this, the TV is updated as per the last transaction status at LSDNC and the updated TV of the vehicle is sent to the main SDN controller for information gathering. In this system, the best ML algorithm for the ML-MDS model is selected by considering decision tree, support vector machine (SVM), neural network (NN), and logistic regression (LR) algorithms. The classification accuracy performance is evaluated using UNSW_NB-15 standard dataset for detecting the normal and malicious vehicles. NN shows better classification accuracy than other algorithms. The proposed ML-MDS is implemented and evaluated using OMNeT++ network simulator and the Simulation of Urban Mobility (SUMO) road traffic simulator by considering various parameters such as detection accuracy, detection time, and energy consumption. From the results, it is observed that the detection accuracy of proposed ML-MDS system is 98.4% as compared to Grover et al. scheme which was 80.2%. Also, for scalability issue the dataset size is increased and performance is evaluated in Orange 3.26.0 machine analytics tool and NN is found to be the best algorithm which shows high accuracy in detecting the attackers.