Real-time roadway-environment perception is one of the primary applications of IoT based autonomous driving to improve road safety. Roadway-environment insights include on-road detection of any type of moving vehicles, non-vehicle (persons, animals, etc.), curves and lanes. There have been various studies that provided Artificial Intelligence (AI)-based detection approaches, however, most of the methods are atomistic which are not well suited for such real-time autonomous driving owing to high detection latency and low accuracy. Therefore, in this paper, we propose a holistic AI-based roadway-environment learning system for simultaneous real-time detection of various on-road objects with high accuracy (more than 90%) at reduced computation complexity.