Color and NIR Joint Acquisition with the Bayer CFA

Abstract

In this project, we address the problem of color and NIR joint acquisition using only a single silicon sensor and a modified version of the Bayer color filter array (CFA).  As the Bayer CFA is already mounted in most color cameras, implementing such a design for the joint acquisition of color and NIR images requires only minor changes to the hardware of commercial color cameras.

The main difference between this design and the conventional color camera is the post processing applied to the captured values to reconstruct full resolution images. By using a CFA similar to Bayer, the sensor records a mixture of NIR and one color channel in each pixel. In this case, separating NIR and color channels in different pixels is equivalent to solving an underdetermined system of linear equations. To solve this problem, we propose a novel algorithm that uses the tools developed in the field of compressive sensing. Our method results in high-quality RGB and NIR images (the average PSNR of more than 30 dB for the reconstructed images in our dataset) and shows a promising path towards RGB and NIR cameras.

Results

From left to right: crops of original images, the images reconstructed using the customized CFA [1], and the results of using the modified Bayer CFA and the compressive sensing based algorithm. For each example, top row illustrates color and the second row shows the NIR counterparts.

     
     
     
     
     

Note that implementing the customized CFA is more complicated than the modified Bayer CFA proposed in this project. For more details on the customized CFA, see the corresponding project page.

  • Z. Sadeghipoor Kermani, Y. Lu and S. Süsstrunk, A Novel Compressive Sensing Approach to Simultaneously Acquire Color and Near-Infrared Images on a Single Sensor, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2013. 
    Detailed Record    Full Text

  • Z. Sadeghipoor Kermani, Y. Lu, and S. Süsstrunk, Compressive Acquisition of Color and Near-Infrared Images, IEEE International Conference on Computational Photography (ICCP), 2014.

    Detailed Record    Full Text

Reference

[1] Y. M. Lu, C. Fredembach, M. Vetterli, and S. Süsstrunk, Designing Color Filter Arrays for the Joint Capture of Visible and Near-infrared Images, IEEE International Conference on Image Processing (ICIP), 2009.