1. Introduction
Thermal images captured via uncooled long-wave infrared cameras typically suffer from column/stripe non-uniformity [1], which caused by the different characteristics of column-parallel accumulators and analog-to-digital converters (ADCs) in infrared focal plane array (FPA) [2]-[4]. This type of non-uniformity, or so-called column fixed pattern noise (FPN), appears as column-oriented stripes (see in Fig. 1). Obvious stripes will significantly degrade the quality and radiometric accuracy of captured infrared images, leading to performance drop of subsequent infrared imaging applications (e.g., objection detection, thermal diagnosis, target tracking). stripe-free infrared images are highly demanded or required among these tasks. Therefore, stripe non-uniformity correction (NUC), which aims to remove the stripe noise, has attracted significant attention among both the academic and industrial communities over the past decades.
 |
Fig. 1 Infrared images with obvious column stripe noise. |
However, it is difficult to achieve perfect stripe NUC by using the traditional calibration-based [5], [6] or scene-based [7]-[10] methods. These methods are designed for compensating slowly drifted spatial non-uniformity of infrared cameras, which are not suitable for high-frequency stripe noise. Recently, filtering methods, which transfer the stripe NUC into an image processing problem, have achieved promising results. Nevertheless, it is challenging to distinguish the texture from stripe noise in some cases. For example, a vertical edge caused by specific object in the scene is similar to the stripe noise, making it easy to be smoothed or blurred.
To address the above problem, we propose a single image based processing algorithm called edge-preserving stripe noise removal (EPSNR) method to accurately filter out the stripe noise without blurring vertical edges (see in Fig. 2). The key idea of EPSNR is to employ side-window filtering technique [11]-[13] into the first step for information decomposition. Specifically, the filtering window’s side is aligned with the pixel being processed (treat each pixel as a potential edge). To fit our task and accelerate the speed, there are only left or right side windows placed on one side of the pixel, and the one whose output is closer to the current pixel value is chosen. Then we leverage the 1D column guided filter from [14] to accurately extract stripe noise.
 |
Fig. 2 Overall diagram of our proposed EPSNR method. |
The workflow of our proposed EPSNR method is illustrated in Fig. 2, and the main contributions of our work can be summarized as follows:
-
Through analyzing the procedure of conventional filtering, we theoretically reveal that placing the center of the filtering window on the pixel being processed (i.e., traditional filtering practice) will inevitably blur vertical edges.
-
Side window strategy is firstly introduced for infrared image stripe noise removal. By aligning the window’s side with the pixel being processed, the vertical edges can be well preserved.
2. Related Work
The traditional non-uniformity correction (NUC) methods can be divided into two main categories: calibration-based [5], [6] and scene-based [7]-[10]. The calibration-based methods need shutter or blackbody as the uniform temperature reference to correct the non-uniformity. This practice will periodically freeze the capturing for a few seconds, therefore is not suitable for real-time infrared applications. Many scene-based methods [7], [8], [10], [15] are proposed to perform NUC, and they typically require processing multiple frames. A noticeable limitation is the running speed, and if the image sequences lack enough scene motions, “ghosting” artifacts will appear in the current frame.
In this letter, we focus on stripe non-uniformity, which can be seen as the column FPN. Both calibration-based and scene-based methods cannot achieve satisfactory results. In [2], [16], the stripe NUC is translated into a gradient-constrained optimization problem. They try to compute the optimal image where the energy of horizontal gradients is as small as possible. Cao and Li [17] set up a thermal calibration experiment to derive the behavioral model of the stripe noise, and then embedded the least-squares optimization to find the optimal model parameters. Once parameters are obtained, stripe noise can be removed from the input. Tendero et al. [3], [4] proposed the famous Midway Histogram Equalization (MHE) method, which adjusts pixels’ intensities within a column. The basic hypothesis behind MHE is the statistical similarity between two adjacent columns. MHE can effectively remove stripe noise without blurring fine details. However, it tends to generate false artifacts.
Filtering methods belong to another research line. M\(\ddot{\mathrm{u}}\)nch et al. [18] designed a destriping method based on wavelet decomposition and Fourier transform (WD-FT). The column-oriented stripe noise is decomposed into the vertical component, and filtered out in the Fourier domain. Cao et al. [14] built a local linear model to reveal the relationship between stripe noise and the thermal radiation. Then, a 1D row guided filter is applied to perform information decomposition (both stripe noise and textures are divided into the high-frequency part) while a 1D column guided filter is applied to extract stripe noise. Cao et al. [1] improved the algorithm by firstly introducing wavelet decomposition and then conducting 1D column guided filtering on different scale levels (WD-GF).
More recently, deep learning based methods have been applied in stripe NUC [19]-[21], and achieve state-of-the-art performance. The biggest problem of these methods is the lack of interpretability. We do not discuss them in this work.
3. Methodology
Strip noise, which can be regarded as the column FPN, is a typical undesired non-uniformity in long-wave infrared cameras. Many single frame based methods are proposed to remove the column FPN [14], [17], [18]. Among them, one main category is the filtering method, which employs the image processing algorithm as a tool to filter out the noise. In this letter, we develop a filtering method, called edge-preserving stripe noise removal (EPSNR) method, to tackle the column FPN.
3.1 Edge-Preserving Decomposition
By far 1D guided filtering (1D-GF) method proposed by Cao et al. [14] is the most relevant to our research work. As for 1D-GF method, the input infrared image is typically decomposed into a smoothed part and a high-frequency part via a row guided filter. As shown in Fig. 2, we also adopt this profile to conduct information decomposition in the first step. The key difference between 1D-GF and our EPSNR are filters used for decomposition. We argue that the row guided filter used in 1D-GF may cause edge blurring, since it involves averaging operations on pixels on both sides of the edge.
We take step edge as an example to analyze the essential reason why 1D row guided filter cannot maintain vertical edge. In Fig. 3, \(x_{1:3}\) and \(x_{4:6}\) are pixels on the two sides of the step edge, and \(l\) and \(h\) are their values. According to [13], [22], the output of 1D row guided filter at \(x_i\) (i.e., \(q_i\)) can be calculated by averaging the possible values (all the windows that cover \(x_i\) will contribute to \(q_i\)). For simplification, we only consider the situation where the input image \(p\) is regarded as the guidance image \(I\), and the size of local window \(w_k\) is \(1\times 3\). The output values at \(x_3\) and \(x_4\) can be derived as:
| \[\begin{eqnarray*} &&\!\!\!\!\! q_{3} = l + \frac{\epsilon(h-l)}{3(\sigma^2+\epsilon)}, \tag{1} \\ &&\!\!\!\!\! q_{4} = h - \frac{\epsilon(h-l)}{3(\sigma^2+\epsilon)}, \tag{2} \end{eqnarray*}\] |
where \(\sigma^2 = \frac{2(h-l)^2}{9}\) is the variance of \(p\) in \(w_k\), and \(\epsilon\) is a regularization parameter which is non-negative. It is not difficult to note that the output at \(x_3\) is greater than \(l\) and the output at \(x_4\) is smaller than \(h\). Therefore, the step edge will be blurred after the 1D row guided filter.
 |
Fig. 3 A 1D signal with a step edge. Pixels \(x_{1:3}\) and \(x_{4:6}\) are on the two sides of the edge. \(l\) and \(h\) denote the grayscale values of two sides. |
One possible solution is to utilize pixels of the same side of the edge. Here, we align the right end of the local window with \(x_3\) to calculate the output at \(x_3\), instead of aligning the center of window with \(x_3\). Similarly, we align the left end of the local window with \(x_4\). To the best of our knowledge, this is the first work that the side window strategy is applied for infrared image stripe noise removal.
To accelerate the processing speed, we adopt only two side windows (i.e., left and right). Given a 1D local window whose size is \(1\times (2r+1)\), left side window contains pixels from start to \(r+1\) and right side window contains pixels from \(r+2\) to end. The one whose output is closer to the current input value is selected as the final output. Based on this novel method of placing sides of local windows on the target pixels, the output values at \(x_3\) and \(x_4\) are equals to their input values (the simplest box filter is employed in our implementation). Note that the step edge is perfectly preserved. The details of the 1D side-window filter is summarized in Algorithm 1.
 |
After the edge-preserving decomposition by 1D side-window filter, the input image \(p\) is decomposed into a smoothed part (i.e., \(q\)) and a high-frequency part \(hf\).
3.2 1D Column Guided Filter
Following [1], [14], we employ the \(hf\) and \(q\) as the filtering and guidance images to extract the stripe noise \(s\) from \(hf\).
| \[\begin{equation*} s = \mathcal{F}_{CGF}(\text{filtering}=hf,\text{guidance}=q), \tag{3} \end{equation*}\] |
where \(\mathcal{F}_{CGF}(\cdot)\) denotes the 1D column guided filtering operation with local window size \(hei \times 1\).
The reason we use \(q\) as the guidance image is that our thermal calibration experiment observes a local linear relationship exists between \(s\) and \(hf\), and such relationship satisfies the key assumption of guided filtering [1], [14]. The extracted \(s\) is further subtracted from the input \(p\) to obtain the final result \(o\).
4. Experimental Results
4.1 Implementation Details
The hyper-parameter \(r\), which defines the window size of our 1D side-window filter, is set to \(4\). For the step of 1D column guided filter, we set \(\epsilon=0.2^2\) and \(hei=\frac{H}{4}\) (\(H\) means the height of input image) to keep consistent with [14]. We make use of a 20-image infrared dataset from [4] for testing the performance (download link: http://demo.ipol.im/demo/glmt_mire/). All the experiments are conducted in Matlab R2022b on a laptop equipped with Inter Core i7-12700H CPU (2.3 GHz) and 64 GB memory.
4.2 Discussion
We discuss the effectiveness of our 1D side-window filter against 1D row guided filter. Both of them are utilized in step 1 for information decomposition. The intermediate and final results are shown in Fig. 4, and the input infrared image is consistent with that from Fig. 2.
 |
Fig. 4 The edge-preserving results of 1D-GF [14] and our proposed EPSNR. We plot the pixel intensity values of a randomly selected row (highlighted by the horizontal blue line). |
We need to point out that the 1D row guided filter used in 1D-GF will blur the vertical edge in the first step, and the smoothed part contains obvious edge blurring effect. The blurred edge will further affect the stripe noise extraction operation conducted by the column guided filter in step 2. We also plot the pixel intensity values of a selected row crossing the vertical edge to show the priority of 1D side-window filter. The last column of Fig. 4 clearly demonstrates the edge-preserving effect of our proposed EPSNR.
4.3 Comparison
We thoroughly compare the proposed EPSNR method with MHE [4], WD-FT [18], 1D-GF [14], and WD-GF [1] in Fig. 5. There are some undesired artifacts in the processing results of WD-FT and MHE (highlighted with the red ellipses). In addition, textures around the vertical edge are blurred in the processing results of WD-GF and 1D-GF (highlighted with red arrows). In comparison, our EPSNR achieves the best performance, effectively removing the stripe noise and preserving the vertical edges without artifacts. Note that, the removal effect of stripe noise is not the only criterion, and the preservation of original information (e.g., vertical edge) is also crucial. It is necessary to balance the relationship between the two items.
 |
Fig. 5 Comparative results of WD-FT, WD-GF, MHE, 1D-GF, and our proposed EPSNR. We select an infrared image with obvious vertical edge from the 20-image dataset [4] as the input. Please zoom in on screen to see more details. |
To quantitatively evaluate the performance, we use a reference-free index \(D^{S_F}_{S_T}\) proposed in [23]. This indicator can take into account both the removal of stripe noise and the preservation of vertical edges simultaneously. Higher \(D^{S_F}_{S_T}\) value indicates the proposed method has a better ability to suppress stripe noise while preserving original information. The experimental results are illustrated in Table 1.
Table 1 Quantitative results of WD-FT, WD-GF, MHE, 1D-GF, and our proposed EPSNR. \(D^{S_F}_{S_T}\) is tested on infrared images from [4]. The running time is tested on infrared images with \(384\times 288\) resolution. |
 |
Another advantage is the running speed. As shown in Table 1, our EPSNR is the most efficient among the state-of-the-art destriping methods, indicating its practicability.
5. Conclusion
In this letter, we propose EPSNR to remove the stripe noise, meanwhile preserving the vertical details. For the first time, side-window filter is employed to perform edge-preserving decomposition, and then the stripe noise is accurately separated from the high-frequency part via a 1D column guided filter. Our EPSNR outperforms state-of-the-art destriping methods.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China under Grant No. 52305590 and No. 52105526, China Postdoctoral Science Foundation funded project under Grant No. 2022M712792, Zhejiang Provincial Natural Science Foundation of China under Grant No. LQ24F010004.
References
[1] Y. Cao, Z. He, J. Yang, X. Ye, and Y. Cao, “A multi-scale non-uniformity correction method based on wavelet decomposition and guided filtering for uncooled long wave infrared camera,” Signal Processing: Image Communication, vol.60, pp.13-21, Feb. 2018.
CrossRef
[2] W. Qian, Q. Chen, G. Gu, and Z. Guan, “Correction method for stripe nonuniformity,” Appl. Opt., vol.49, no.10, pp.1764-1773, 2010.
CrossRef
[3] Y. Tendero and J. Gilles, “ADMIRE: A locally adaptive single-image, non-uniformity correction and denoising algorithm: Application to uncooled IR camera,” Infrared Technology and Applications XXXVIII, p.83531O, 2012.
CrossRef
[4] Y. Tendero, S. Landeau, and J. Gilles, “Non-uniformity correction of infrared images by midway equalization,” Image Processing On Line, vol.2, pp.134-146, 2012.
CrossRef
[5] J.M. Mooney, F.D. Shepherd, W.S. Ewing, J.E. Murguia, and J. Silverman, “Responsivity nonuniformity limited performance of infrared staring cameras,” Optical Engineering, vol.28, no.11, pp.1151-1161, 1989.
CrossRef
[6] M. Schulz and L. Caldwell, “Nonuniformity correction and correctability of infrared focal plane arrays,” Infrared Physics and Technology, vol.36, no.4, pp.763-777, 1995.
CrossRef
[7] R.C. Hardie, M.M. Hayat, E. Armstrong, and B. Yasuda, “Scene-based nonuniformity correction with video sequences and registration,” Appl. Opt., vol.39, no.8, pp.1241-1250, 2000.
CrossRef
[8] B.M. Ratliff, M.M. Hayat, and R.C. Hardie, “An algebraic algorithm for nonuniformity correction in focal-plane arrays,” J. Opt. Soc. Am., vol.19, no.9, pp.1737-1747, 2002.
CrossRef
[9] S.N. Torres, J.E. Pezoa, and M.M. Hayat, “Scene-based nonuniformity correction for focal plane arrays by the method of the inverse covariance form,” Appl. Opt., vol.42, no.29, pp.5872-5881, 2003.
CrossRef
[10] S.N. Torres and M.M. Hayat, “Kalman filtering for adaptive nonuniformity correction in infrared focal-plane arrays,” J. Opt. Soc. Am. A, vol.20, no.3, pp.470-480, 2003.
CrossRef
[11] Y. Gong, B. Liu, X. Hou, and G. Qiu, “Sub-window box filter,” VCIP 2018 - IEEE International Conference on Visual Communications and Image Processing, pp.1-4, IEEE, 2018.
CrossRef
[12] H. Yin, Y. Gong, and G. Qiu, “Side Window Filtering,” CVPR, pp.8758-8766, 2019.
CrossRef
[13] H. Yin, Y. Gong, and G. Qiu, “Side window guided filtering,” Signal Process., vol.165, pp.315-330, 2019.
CrossRef
[14] Y. Cao, M.Y. Yang, and C.l. Tisse, “Effective strip noise removal for low-textured infrared images based on 1D guided filtering,” IEEE Trans. Circuits Syst. Video Technol., vol.26, no.12, pp.2176-2188, 2016.
CrossRef
[15] J.G. Harris and Y.M. Chiang, “Nonuniformity correction of infrared image sequences using the constant-statistics constraint,” IEEE Trans. Image Process., vol.8, no.8, pp.1148-1151, 1999.
CrossRef
[16] J. Zhao, Q. Zhou, Y. Chen, T. Liu, H. Feng, Z. Xu, and Q. Li, “Single image stripe nonuniformity correction with gradient-constrained optimization model for infrared focal plane arrays,” Optics Communications, vol.296, pp.47-52, 2013.
CrossRef
[17] Y. Cao and Y. Li, “Strip non-uniformity correction in uncooled long-wave infrared focal plane array based on noise source characterization,” Optics Communications, vol.339, pp.236-242, 2015.
CrossRef
[18] B. Münch, P. Trtik, F. Marone, and M. Stampanoni, “Stripe and ring artifact removal with combined wavelet ― Fourier filtering,” Opt. Express, vol.17, no.10, pp.8567-8591, 2009.
CrossRef
[19] Z. He, Y. Cao, Y. Dong, J. Yang, Y. Cao, and C.l. Tisse, “Single image based non-uniformity correction of uncooled long-wave infrared detectors: A deep learning approach,” Appl. Opt., vol.57, no.18, pp.D155-D164, 2018.
CrossRef
[20] X. Kuang, X. Sui, Q. Chen, and G. Gu, “Single infrared image stripe noise removal using deep convolutional networks,” IEEE Photon. J., vol.9, no.4, p.3900913, 2017.
CrossRef
[21] K. Xu, Y. Zhao, F. Li, and W. Xiang, “Single infrared image stripe removal via deep multi-scale dense connection convolutional neural network,” Infrared Physics and Technology, vol.121, p.104008, 2022.
CrossRef
[22] K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Trans. Pattern Anal. Mach. Intell., vol.35, pp.1397-1409, 2013.
CrossRef
[23] Y. Cao, Z. He, J. Yang, Y. Cao, and M.Y. Yang, “Spatially adaptive column fixed-pattern noise correction in infrared imaging system using 1D horizontal differential statistics,” IEEE Photon. J., vol.9, no.5, p.7803513, 2017.
CrossRef
