Abstract
Near-infrared images combined with the color representations of the scene have been recently used in several computational photography and computer vision tasks (to see some examples, see the corresponding project page). All these applications require an aligned pair of color and NIR images.
Silicon, the light-sensitive material in sensors of most digital cameras, is inherently sensitive to both visible and NIR bands of the electromagnetic spectrum. Thus, a single silicon sensor is potentially enough to capture both color and NIR images of a scene. In this project, we study an acquisition system that simultaneously captures the color and NIR pair using only a single sensor. We propose using an array of color filters (CFA) to spectrally sample the scene. Each filter of the proposed CFA transmits a mixture of all color channels as well as NIR. We pose the CFA design as a novel spatial domain optimization problem. To reconstruct the full-resolution images, we use an efficient linear demosaicing operator. For more details about this algorithm, please see [1].
In the next step of the project, we study the joint characteristics of natural RGB and NIR images. Specifically, we analyze the spatial-spectral correlation of these images in different frequency bands. We show that NIR and color images are almost not correlated in low-frequencies (representing the intensities) and highly correlated in midband to high-frequencies (strong edges). The details of this study are explained in [2]. We use these results as an additional regularizer in the design process of CFA and the demosaicing operator.
Results
From left to right: Original images, the results of the algorithm explained in [1], and the results of [2]. For each example, top row illustrates color and the second row shows the NIR images.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
You can download more images here. If you use these images in research and publications, please put a reference to [2]. Thank you!
Related Publications
, , and IEEE International Conference on Image Processing (ICIP), 2009.
[2] Z. Sadeghipoor Kermani, Y. Lu, and S. Süsstrunk, Correlation-based Joint Acquisition and Demosaicing of Visible and Near-Infrared Images, IEEE International Conference on Image Processing (ICIP), 2011.