Using an ISP for Real-time Data Augmentation

Image signal processors (ISPs) are tasked with processing raw pixels delivered by image sensors in order to optimize the quality of images. In computer vision applications, much attention is focused on tuning the ISP, both to obtain good training data and to optimize image quality in deployed systems when faced with widely varying, dynamic imaging conditions (e.g., changes in lighting). We describe a data-driven solution to these challenges developed in a collaboration between Pony.AI and On Semiconductor. Our approach manipulates the ISP configuration during training data collection in order to perform real-time data augmentation. By training deep neural networks (DNNs) with the resulting augmented data, we are able to create DNNs that are robust to variations in imaging conditions and ISP settings.