Optimized Image Processing for Automotive Image Sensors with Novel Color Filter Arrays

Traditionally, image sensors have been optimized to produce images that look natural to humans. As more and more images are consumed by algorithms rather than humans, the optimization goals change. For images consumed by algorithms, what matters is capturing as much information as possible in the image — including in low-light conditions. One way to capture more information is higher sensor resolution. But with higher resolution comes lower sensitivity. To increase resolution and also maintain high sensitivity, color information can be sacrificed. However, in automotive applications, color information is critical. In response, suppliers are offering image sensors that capture color information using novel color filter arrays (CFAs). Instead of the traditional RGGB array, some sensors use patterns like red-clear-clear-green (RCCG). These approaches can yield good results for images consumed by algorithms. But what about applications where images are consumed by both algorithms and humans? Can one reconstruct a natural-looking image from an image sensor using a non-standard color filter array? In this talk, we explore the pros and cons of novel CFAs and introduce Nextchip’s vision processor, which efficiently supports reconstruction of natural looking images from image sensors with novel CFAs, including RGB-IR sensors commonly used for in-cabin monitoring.