Vehicular Fog Computing (VFC) has emerged as a captivating research paradigm to increase computing resources for Internet of Vehicles (IoV) applications. However, the allocation of the best possible computing resources is challenging due to the enormous growth of services in vehicular networks, such as augmented reality, 3-D gaming, autonomous driving, etc., and their severe delay limitations, processing demands, and mobility of vehicles. Mainly, achieving strict task latency is a challenging problem while vehicles traverse through the roadside unit (RSU) regions. In this work, the vehicle’s task processing latency and energy consumption are considered by deciding which tasks should be scheduled to either the local RSU or the neighbour RSUs along the vehicle’s path. We study the mobility-aware task scheduling problem to jointly optimize the service latency of the tasks and energy consumption of the vehicles. Initially, we employ the Markov renewal process (MRP) to model the vehicle’s mobility. Subsequently, we establish that our proposed objective function is a monotone submodular function and provide a greedy heuristic for the maximization of the submodular function with an approximation of (1 − e1 ). Further, we propose a hybrid optimization algorithm that combines particle swarm optimization (PSO) and simulated annealing (SA) algorithms to solve mobility-aware task scheduling problems for large-scale networks. We compare our proposed algorithm with existing mobility-aware scheduling approaches and simulation results show that the proposed algorithm performs efficiently for task scheduling problems.