Abstract

This paper explores the potential of machine learning (ML) systems which use data from in-vehicle sensors as well as external IoT data sources to enhance autonomous driving for efficiency and safety in urban environments. We propose a system which combines sensor data from autonomous vehicles and IoT data collected from pedestrians' mobile devices. Our approach includes two methods for vulnerable road user (VRU) detection and pedestrian movement intention prediction, and a model to combine the two outputs for potentially improving the autonomous decision-making. The first method creates a world model (WM) and accurately localizes VRUs using in-vehicle cameras and external mobile device data. The second method is a deep learning model to predict pedestrian's next movement steps using real-time trajectory and training with historical mobile device data. To test the system, we conduct three pilot tests at a university campus with a custom-built autonomous car and mobile devices carried by pedestrians. The results from our controlled experiments show that VRU detection can more accurately distinguish locations of pedestrians using IoT data. Furthermore, up to five future steps of pedestrians can be predicted within 2 m.