Method and device for correcting raw thermal images

Abstract

The invention relates to a device for processing a raw infrared image (IB), the device comprising: - a memory (206) configured to store at least one gain (G) and a family of at least one bias image (BF); and - a processor (210) configured to calculate a corrected image (IC) on the basis of the raw image, of the at least one gain and of a bias image (B) associated with the raw image, wherein the bias image associated with the raw image is generated from the at least one gain (G), from the family of at least one bias image (BF), and from an image quality metric associated with the corrected image (IC).

Description

DESCRIPTION TITLE: Method and device for correcting raw thermal images This patent application claims priority from French patent application FR2412352, filed on November 12, 2024, entitled "Method and device for correcting raw thermal images", which will be considered as forming an integral part of this description. technical field

[0001] This description relates in general to infrared image processing devices, infrared image sensors, and infrared image processing methods. Previous technique

[0002] Infrared imaging devices, such as bolometers, comprise an array of detectors sensitive to infrared radiation forming a pixel array.

[0003] Spatial non-uniformity between pixels in the pixel array, which causes a shift in their values ​​to be corrected in the image, varies over time depending on the temperature of the focal plane array (TFPA), acquisition parameters, and the thermal environment. This problem is generally addressed by using an internal mechanical shutter in the imaging device. This involves periodically capturing an image while the shutter is closed to obtain a reference image of a relatively uniform scene, which can then be used for calibration, for example, by subtraction. However, using a shutter has several drawbacks, such as added weight and cost, and the fragility of this component. Furthermore, for some applications, the use of a shutter is unacceptable due to the time lost while the shutter is closed and calibration is performed. During this calibration time, no image of the scene can be captured.

[0004] Image processing techniques to correct the shift have been proposed as an alternative to using a shutter. Some image processing methods are based on a calibration phase during which the noise of the image sensor is measured. Other methods rely on assumptions about the properties of the acquired image and use, for example, statistical analysis to correct it.

[0005] However, the known methods are expensive, complex, not very robust to changes in the thermal environment of the image sensor and sometimes have a significant computation time.

[0006] Therefore, there is a need in technology for an improved process and device for correcting an infrared image. Summary of the invention

[0007] According to one aspect, a device for processing a raw infrared image is planned, comprising: - a memory configured to store at least one gain and a family of at least one bias image; and - a processor configured to calculate a corrected image from the raw image, at least one gain and a bias image associated with the raw image, wherein the bias image associated with the raw image is generated from at least one gain, the family of at least one bias image, and an image quality metric associated with the corrected image.

[0008] According to one embodiment, the family of at least one bias image is generated on the basis of at least one raw reference image, for example captured during a calibration or calibration phase.

[0009] According to one embodiment, each bias image in the family of at least one bias image is calculated by multiplying pixel by pixel a raw reference image by at least one gain and then subtracting its spatial mean value.

[0010] According to one embodiment, the device further comprises an infrared camera configured to acquire at least two raw reference images and the raw image and in which the processor is configured to calculate at least one gain from the at least two raw reference images.

[0011] According to one embodiment, the processor is further configured to calculate one from the family of at least one bias image from at least one of the at least two raw reference images.

[0012] According to one embodiment, the infrared camera is uncooled.

[0013] According to one embodiment, the processor is further configured to calculate the image quality metric.

[0014] According to one embodiment, the processor is further configured to calculate an estimate of the bias image associated with the raw image by finding a linear combination of at least one bias image that minimizes the image quality metric, according to the following equation:

[0015] B = £” =1 a, BÎ such that BG argminlil VeclUiFjF G • IB - B)

[0016] where n denotes the number of reference bias images in the at least one bias image, and a denotes a coefficient of the linear combination and Bi corresponds to a bias image of the at least one bias image.

[0017] According to another aspect, a process for processing a raw infrared image is planned, the process comprising: - the recording of at least one gain and a family of at least one bias image in a memory; and - the calculation of a corrected image from the raw image, the gain and a bias image associated with the raw image, in which the bias image associated with the raw image is generated from at least one gain, the family of at least one bias image, and an image quality metric associated with the corrected image.

[0018] According to one embodiment, the method further comprises the acquisition, by an infrared camera, of at least two raw reference images and the raw image and the calculation of at least one gain from the at least two raw reference images.

[0019] According to one embodiment, the process further includes calculating one of at least one biased image from at least one of at least two raw reference images.

[0020] According to one embodiment, at least two raw reference images are captured in a thermally dynamic environment of any kind.

[0021] According to one embodiment, the process further includes a calculation of the image quality metric.

[0022] According to one embodiment, an estimate of the bias image associated with the raw image is calculated by a linear combination of at least one bias image which minimizes the image quality metric, according to the following equation:

[0023] B = ” =1 a. B, such that BE argmin BEVect BF F(G • IB — B)

[0024] where n denotes the number of reference bias images in the at least one bias image, a denotes a coefficient of the linear combination and Bi corresponds to a bias image in the at least one bias image.

[0025] According to one embodiment, the calculation of the image quality metric is performed on the basis of a calculation of the image gradient.

[0026] According to one embodiment, the calculation of the image quality metric is performed according to a Euclidean standard satisfying the following equation:

[0027] ∀v ∈ ℝ^Ω, F1(v) =‖ v ‖₂= √∑_{(x,y)∈Ω} v(x,y)²

[0028] where V denotes an image comprising M rows and N columns and Ω denotes the domain {0, 1, ..., M − 1} × {0, 1,..., N − 1}.

[0029] According to one embodiment, the calculation of the image quality metric is performed according to a semi-norm defined by a norm 2 of the gradient satisfying the following equation:

[0030] ∀v ∈ ℝ^Ω, F2(v) =‖ ∇v ‖₂,₂:= √∑_{(x,y)∈Ω} ‖ ∇v(x,y) ‖²₂

[0031] where V denotes an image comprising M rows and N columns, Ω denotes the domain {0, 1, ..., M − 1} × {0, 1,..., N − 1} and ∇: v ↦ (∇ₓv, ∇_y v) represents a linear gradient operator with finite differences.

[0032] According to one embodiment, the calculation of the image quality metric is performed according to a total variation defined by the following equation:

[0033] ∀v ∈ ℝ^Ω, F3(v) =‖ ∇v ‖₁,₂:= ∑_{(x,y)∈Ω} ‖ ∇v(x,y) ‖₂

[0034] where V denotes an image comprising M rows and N columns, Ω denotes the domain {0, 1, ..., M − 1} × {0, 1,..., N − 1} and ∇: v ↦ (∇ₓv, ∇_y v) represents a linear gradient operator with finite differences.

[0035] According to one embodiment, the bias image associated with the raw image is generated from an orthonormal basis, the orthonormal basis being constructed on the basis of the family of at least one bias image.

[0036] According to another aspect, an uncooled infrared camera is planned, including the above device.

[0037] In another aspect, a system for processing a raw infrared image is planned, comprising: - a calibration device configured to calculate at least one gain from at least two raw reference images and to calculate one from a family of at least one bias image from at least one of the at least two raw reference images; and - a device for processing a raw infrared image configured to record at least one gain, to record the family of at least one bias image, to calculate a corrected image from the raw image, the at least one gain and a bias image associated with the raw image, in which the bias image associated with the raw image is generated from the at least one gain, the family of at least one bias image, and an image quality metric associated with the corrected image. Brief description of the drawings

[0038] These features and advantages, along with others, will be detailed in the following description of modes of specific implementations carried out, by way of non-exhaustive list, in relation to the attached figures, including:

[0039] Figure 1 schematically illustrates an imaging device according to an example of the embodiment of this description;

[0040] Figure 2 schematically illustrates, in more detail, an image processing block of the imaging device of Figure 1 according to an example of the implementation of this description;

[0041] Figure 3 is a flowchart illustrating steps in an image processing procedure using the imaging device of Figure 1 according to one embodiment of the present description;

[0042] Figure 4A represents an example of a responsive image according to one embodiment of the present description;

[0043] Figure 4B represents an example of a shift image according to one embodiment of the present description;

[0044] Figure 5 represents, in the form of a flowchart, images involved in the process of Figure 3 according to one embodiment of the present description;

[0045] Figure 6 represents an example of a raw image corrected by different models and for different temperatures of the focal plane of a camera;

[0046] Figure 7 shows another example of a raw image corrected by different models and for different focal plane temperatures of a camera; and

[0047] Figure 8 is a flowchart illustrating the image processing method of Figure 3 according to one embodiment of the present description. Description of the implementation methods

[0048] The same elements have been designated by the same reference numerals in the different figures. In particular, structural and / or functional elements common to the different embodiments may have the same reference numerals and may have identical structural, dimensional and material properties.

[0049] For the sake of clarity, only the steps and elements necessary for understanding the described implementation methods have been shown and are detailed. In particular, infrared image acquisition procedures are assumed to be familiar to those skilled in the art.

[0050] Unless otherwise specified, when referring to two connected elements, this means directly connected without any intermediate elements other than conductors, and when referring to two coupled elements, this means that these two elements can be connected or linked through one or more other elements.

[0051] In the description that follows, when referring to absolute positional qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative positional qualifiers, such as the terms "above", "below", "superior", "inferior", etc., or to orientational qualifiers, such as the terms "horizontal", "vertical", etc., unless otherwise specified, it refers to the orientation of the figures.

[0052] Unless otherwise specified, the expressions "approximately", "roughly", "about", and "in the order of" mean within 10%, preferably within 5%.

[0053] Although some of the embodiments in this description are described in relation to a microbolometer-type pixel array, those skilled in the art will note that the processes described herein also apply to other types of infrared imaging devices, including cooled devices.

[0054] Figure 1 schematically illustrates a 100 infrared imaging device according to an example embodiment.

[0055] The 100 infrared imaging device is for example a camera, a photo camera, a mobile phone, an electronic tablet, etc.

[0056] The infrared imaging device 100 comprises a 102-pixel array sensitive to infrared light. For example, in some embodiments, the pixel array is sensitive to long-wavelength infrared light, such as light with a wavelength between 7 and 13 pm.

[0057] To simplify the illustration, a 10² matrix of only 144 pixels, arranged in 12 rows and 12 columns, is shown in Figure 1. In other embodiments, the 10² pixel matrix can comprise any number of rows and columns of pixels. Typically, the matrix might include, for example, 640 by 480, or 1024 by 768 pixels.

[0058] Each column of pixels in the matrix 102 is associated with a corresponding reference structure 106. Although not functionally an imaging element, this structure will be called here a "reference pixel" by structural analogy with the imaging (or active) pixels 104. In addition, an output block 108 ("OUTPUT") is coupled to each column of the pixel matrix 102 and to each reference pixel 106, and provides a raw image IB.

[0059] A control circuit 110 (“CTRL”) provides control signals to the pixel matrix, to the reference pixels 106, and to the output block 108. The raw image IB is, for example, provided to an image processing block 112 (“IMAGE PROCESSING”), which applies biases and gains to the pixels of the image to produce a corrected image IC.

[0060] The imaging device fd' 100 is configured, for example, to acquire the raw infrared image IB. The focal plane temperature of the imaging device fd' 100 is at room temperature and varies with the ambient temperature. The imaging device fd' 100 is not cooled or maintained at a constant temperature. The imaging device fd' 100 includes, for example, a temperature probe in its focal plane to measure the focal plane temperature. The imaging device fd' 100 does not include a shutter. In this case, the image processing performed by the image processing block 112 corrects the raw infrared image IB sufficiently to avoid the need for a shutter.In other embodiments, the imaging device 100 includes a shutter, and in this case, the image processing carried out by the image processing block 112 makes it possible to reduce at least partially the need to use the shutter.

[0061] Figure 2 schematically illustrates, in more detail, the image processing block 112 of the imaging device 100 of Figure 1 according to an example of an embodiment.

[0062] The functions of the image processing block 112 are, for example, implemented by software, and the image processing block 112 includes a data processing device 202 ("PROCESSING DEVICE") comprising one or several processors 210 (CPUs, from the English "Central Processing Unit") controlled by instructions stored in an instruction memory 204 (INSTR MEM). In other embodiments, the functions of the image processing block 112 can be implemented at least partially by dedicated hardware. In this case, the data processing device 202 includes, for example, an AS IC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), and the instruction memory 204 can be omitted.

[0063] The processing device 202 receives the raw input image IB, and generates the corrected image IC, which is for example provided to a display (not shown) of the imaging device 100. The data processing device 202 is also connected to a data memory 206 (“MEM”) storing at least one image of a gain G and a set of one or more reference bias images Bl, B2,..., Bn forming a family BF of n bias images described in more detail below.

[0064] According to a variant of the embodiment illustrated in Figures 1 and 3, the image processing unit 112 is not connected to the imaging device 100. The raw image IB is, for example, acquired by the imaging device 100 and sent to the image processing unit 112, for example, via a wired or wireless connection. The image processing unit 112 is, for example, a computer that performs image processing to generate the corrected image IC.

[0065] Figure 3 is a flowchart illustrating the steps of an image processing method 300 using the imaging device 100 of Figure 1 according to one embodiment of the present description. This process is implemented, for example, by image processing block 112.

[0066] It is assumed that a raw image IB was captured by the 102 pixel matrix of Figure 1.

[0067] The raw image IB is written, for example, in the form:

[0068] [Math 1] ∀(x,y) ∈ Ω, IB(x,y) = ℛ(x,y) · S(x,y) + 풪(x,y) (1)

[0069] where x and y denote the coordinates of a pixel of an image comprising M columns and N rows, Ω denotes the domain Ω={0,1,…,M-1}×{0,1,…,N-1}, denotes a responsive image, S denotes a scene image, represented in irradiance, and O denotes an intensity offset image (in English, "of fset").

[0070] Figure 4A represents an example of a 400 responsive image according to one embodiment of the present description.

[0071] The responsiveness of a pixel corresponds to the sensitivity of its voltage response to an received heat flux. It depends, for example, on the optics and the size of the microbolometer that makes up that pixel.

[0072] The 400 responsive image is, for example, the responsive image (7?) of the imaging device 100 in Figure 1, for example, an uncooled infrared camera without a shutter. The 400 responsive image is, for example, generated from two images. The first image corresponds, for example, to a first uniform scene, for example, a black body at a first temperature, for example, between 0°C and 20°C. The second image corresponds, for example, to a second uniform scene, for example, a body black at a second temperature different from the first temperature, and for example between 30°C and 50°C.

[0073] The 400 responsive image exhibits slow spatial variations.

[0074] Figure 4B represents an example of a 450 offset image according to one embodiment of the present description.

[0075] The 450 shift image is the O shift image of the imaging device 100 in Figure 1, for example, an uncooled infrared camera without a shutter. The 450 shift image is generated, for example, from two images. The first image corresponds, for example, to a first uniform scene, such as a black body at a first temperature, for example, between 0°C and 20°C. The second image corresponds, for example, to a second uniform scene, such as a black body at a second temperature different from the first, for example, between 30°C and 50°C.

[0076] The offset is an additive voltage resulting from all parasitic heat fluxes, i.e., those not originating from the scene, present during acquisition. Due to the electronics involved, the offset of a pixel (%, y) depends strongly on its columnar position x; in other words, an offset image has a column effect that makes it highly irregular, meaning that pixels in the same column of the 450 offset image have similar values. The 450 offset image is spatially irregular.

[0077] With reference to Figure 3, the scene image S corresponds to a noise-free image and the responsive image and the O-shift image correspond to noise added by the imaging device 100 to the scene image S. 1

[0078] Let μ(v) = 1 / MN Σ(x,y)∈Ω v(x,y) be the average of an image v and by setting G = the corrected image IC, we can write:

[0079] [Math 2] ∀(x,y) ∈ Ω, IC(x,y) = (IB(x,y) − 풪(x,y)) · G(x,y) (2)

[0080] By setting B = G • O — / z(G • (9) as a bias image of the raw image IB, equation (2) can be rewritten in the form:

[0081] [Math 3] ∀(x,y) ∈ Ω, IC(x,y) = G(x,y) · IB(x,y) − B(x,y) = μ(ℛ).S(x,y) + μ(G · 풪) (3)

[0082] The corrected image IC has a fine dependence on the scene image S. The image processing method 300 aims for example to generate the corrected image IC from the raw image IB.

[0083] Furthermore, it is assumed that at least one gain image G and the family of bias images BF are available.

[0084] Each of the at least one G-gain image is, for example, generated based on at least two raw IBC reference images, for example, captured during a calibration phase. The raw IBC reference images are, for example, image pairs consisting of a first image of a first blackbody at a first temperature and a second image of a second blackbody at a second temperature, both images having been acquired in a thermally spatially homogeneous and time-stabilized environment. The image pairs are, for example, acquired for focal plane temperatures of the imaging device that are 100 times different from each other. For example, raw reference IBC images are obtained during an initial phase prior to the acquisition of the raw IB image.

[0085] According to another embodiment, the raw IBC reference images are captured in a thermally and spatially inhomogeneous environment.

[0086] The BF bias image family is generated, for example, using at least one of the raw IBC reference images captured during the calibration phase. Each of the reference bias images Bl, B2, Bn is calculated, for example, by multiplying a raw IBC reference image pixel by pixel by at least one image with gain G, and then subtracting its spatial mean value. The BF bias image family can be enriched at any time by adding new reference bias images. Recording metadata associated with the raw IBC reference images, such as the focal plane temperature, is not necessary for calculating the reference bias images Bl, B2, ..., Bn. The model thus takes into account correction variations due to various measurement parameters, such as the focal plane temperature, or external factors like parasitic heat fluxes.

[0087] According to one embodiment, new raw IBC reference images are captured subsequently, after the calibration phase, to recalculate and update at least one G-gain image and the BF bias image family, for example with raw IBC reference images acquired over a range of focal plane temperature values ​​of imaging device 100 different from the range of focal plane temperature values ​​of imaging device 100, for example including the focal plane temperature of imaging device 100 at the time of image acquisition. The raw IB image is better suited to the current thermal environment and thus more accurate. At least one gain G image and the BF bias image family are updated, for example, to compensate for fluctuations over time, such as those caused by wear and tear on the imaging device.

[0088] At least one image with gain G is assumed to be independent of the temperature of the focal plane during the acquisition of the raw image IB.

[0089] In the example in Figure 3, it is assumed that the at least one gain image G comprises a single gain image G. In other embodiments, the at least one gain image G comprises several gain images, and each of the reference bias images B1, B2, ..., Bn is, for example, calculated from a gain image selected from among the several gain images. Those skilled in the art will be able to adapt the following calculations accordingly. For example, since the biases of the calibration family have been calculated with their respective gains fs, there are at most as many gains as biases, and it is necessary to add a gain selection step for the current raw image IB to be corrected, for example, deterministically based on the current electronic parameters of the sensor.

[0090] During step 320 (“GENERATE BIAS IMAGE”), the estimated bias image B for the raw image IB is generated by the image processing block 112.

[0091] The image of the estimated bias B is an element of the vector space Vect(BF) spanned by the family of bias images BF. The image of the estimated bias B is therefore written according to the following equation:

[0092] [Math 4] n i=l

[0093] Or denotes a coefficient of a decomposition of B in BF, with â̂ᵢ ∈ ℝⁿ and Bi corresponds to a bias image of the family of bias images BF.

[0094] It is further assumed that the image of the estimated bias B is generated in such a way as to obtain the corrected image IC based on an image quality metric F, and, for example, to maximize the image quality in the sense of the metric F. The metric F is associated with the corrected image, and, for example, the metric F is correlated with the spatial regularity of the corrected image (IC). By convention, the image quality metric F is assumed to decrease with image quality; in other words, the quality of an image v will be greater the lower F(v) is. We therefore seek to minimize F(IC), that is, the image of the corrected image IC by the metric F.

[0095] The image of the estimated bias B, for example, satisfies the following equation:

[0096] [Math 5] B̂ ∈ argmin_{B∈Vect(BF)} F(G · IB − B). (5)

[0097] According to a first embodiment, a metric Fl is used as the metric F. The metric Fl is a Euclidean norm that satisfies the equation:

[0098] [Math 6] F1(v) = ‖v‖₂ = √(Σ_{(x,y)∈Ω} v(x,y)²) (6)

[0099] where v is an image of domain Ω and ‖.‖₂ is the L² norm.

[0100] In this embodiment, equation (5) has a unique solution corresponding to the orthogonal projection of the pixel-by-pixel product of the gain image G and the raw image IB, denoted G-IB, onto Vect (BF). This unique solution also satisfies the following equation:

[0101] [Math 7] for all â̂ ∈ argmin_{α∈ℝⁿ} (7) i = l

[0102] where â̂ = (â̂₁, â̂₂, …, â̂ₙ) corresponds to the coefficients of a decomposition of B̂ in BF and α = (α₁, α₂, …, αₙ) corresponds to the coefficients of a linear combination of the reference bias images Bl, B2,..., Bn.

[0103] The calculation of the coefficients satisfying this last equation is for example carried out explicitly from the pixel-by-pixel product of the gain image G and the raw image IB, denoted G- IB, and the reference bias images Bl, B2,..., Bn. This calculation involves the inversion, in the least squares sense, of a Gram matrix of the family of bias images BF.

[0104] When the BF family is linearly independent, it forms a basis of Vect(BF), and the coordinates â of B in the BF basis are uniquely defined. Otherwise, the image of the estimated bias B admits, for example, several decompositions according to the images of the BF bias family.

[0105] According to one embodiment, the Gram matrix is, for example, arbitrarily ill-conditioned or non-invertible if the BF bias family is bound. The BF bias family is, for example, orthonormalized, for example by performing a singular value decomposition (in English, "singular value decomposition"), or using the Gram-Schmidt algorithm, to construct an orthonormal basis BF' = {B1', B2, Bm'}, with m less than or equal to n, of Vect(BF). Since the families BF and BF' both generate the same vector space (i.e., Vect(BF) = Vect(BF')), the estimated bias image B also corresponds to the orthogonal projection of the term-by-term product of the gain image G and the raw image IB, onto Vect(BF'), which can be calculated explicitly and simply using the following equation:

[0106] [Math 8] m B = ^( G - / B, B;) B; (8) i=l

[0107] where <•,•> denotes the canonical scalar product, the terms < G • IB, B- >, for 1< i < m, correspond to the coordinates of B in the orthonormal basis BF ' and whose evaluation poses no numerical difficulty.

[0108] This first Fl metric has the advantage of being simple and inexpensive to implement.

[0109] According to a second embodiment, an F2 metric is used as the F metric. The F2 metric is a semi-norm that satisfies the equation:

[0110] [Math 9] ∀v ∈ ℝ^Ω, F2(v) = ‖∇v‖₂,₂ := √(Σ_{(x,y)∈Ω} ‖∇v(x,y)‖₂²) (9)

[0111] where v is an image of domain Q. and ∇: v ↦ (∇ₓv, ∇ᵧv) represents a linear gradient operator at finite differences and ‖.‖₂² corresponds to the Euclidean norm in ℝ².

[0112] By denoting ∇BF = {∇B₁, ∇B₂, …, ∇Bₙ} and by linearity of ∇, the image of the estimated bias B̂ is a solution of equation (5) if and only if its finite difference gradient ∇B̂ is equal to the orthogonal projection of the signal ∇(G · IB) onto Vect(∇BF), that is, if and only if:

[0113] [Math 10] V B = Or gmin VBEVect it VBF l - II V(G • / B) — VB \\ 2I 2 (10)

[0114] where VB is an element of Vect ( VBF).

[0115] This orthogonal projection is uniquely defined and can be expressed, for example, in terms of decomposition in the VBF generating family of Vect(VBF). Thus, we have:

[0116] [Math 11] ∇B̂ = Σᵢ₌₁ⁿ â̂ᵢ ∇Bᵢ for all â̂ ∈ argmin_{α∈ℝⁿ} ‖∇(G · IB) − Σᵢ₌₁ⁿ αᵢ ∇Bᵢ‖₂,₂ (11) i=li=l 2,2

[0117] By linearity of the operator V and assumption of zero spatial mean on the biases B, the solution B̂ = argmin_{B∈Vect(BF)} F2(G · IB − B) is defined by coefficients â that are solutions of equation (11). The coefficients from (11), substituted into (4), give the solution of (5) when F=F2.

[0118] The calculation of the coefficients â̂ involves, for example, the inversion, in the least squares sense, of the Gram matrix of VBF.

[0119] According to one embodiment, VBF is orthonormalized, for example by performing a singular value decomposition or by using a Gram-Schmidt orthonormalization procedure, to construct an orthonormal basis ∇BF'=(∇B1',∇B2',…,∇Bp'), with p less than or equal to n, of Vect(∇BF). Once the coordinates of VB in the basis VBF' have been calculated, one can find, for example by change of basis, the coefficients â of a decomposition of VB in VBF and therefore of a decomposition of B in BF.

[0120] Using the F2 metric, the search for the B bias image is optimized so that two neighboring pixels of the corrected image have relatively close values ​​and a relatively low gradient.

[0121] One advantage of using the F2 metric is that it allows the image of the estimated bias B to be calculated explicitly using inexpensive calculations and it leads to relatively sharp corrected images.

[0122] According to a third embodiment, an F3 metric is used as the F metric. The F3 metric is a total variation satisfying the equation:

[0123] [Math 12] Vv G IR n , F3(v) =11 Vv ll 12 It 7v(x,y) (12) (x,y)en

[0124] where v is an image of domain fi.

[0125] Proximal algorithms are used for example to perform a search for minimizers of non-differentiable convex functions and to solve equation (5) for the metric F3.

[0126] In one embodiment, BF is orthonormalized, for example by performing a singular value decomposition, or by using the Gram-Schmidt algorithm, to construct the basis BF' = {B1', B2Bm'}. Such an orthonormal basis makes it easy to establish a numerically simple expression for the projection onto the subspace Vect(BF). In this context, the implementation of schemes proximates to solve equation (5) will be within the competence of the person of the trade.

[0127] According to another embodiment, equation (5) is reformulated in terms of finding decomposition coefficients of the image of the estimated bias B in BF, similarly to the embodiment based on the metric F2. The orthonormalization of the family VBF = {VB 1( VB2, VB n ] and the use of preconditioning techniques for proximal algorithms are used and allow for the establishment of more efficient numerical schemes, without being more complex to implement, than by performing an orthonormalization of BF.

[0128] During step 320 of Figure 3, the estimated bias image B of the raw image IB is generated by the image processing block 112, using the gain image G and the bias image family BF f by solving equations (4) and (5) for the image quality metric F.

[0129] Although three examples of Fl, F2, F3 metrics have been detailed, other image quality measurement metrics could be used.

[0130] During a step 330 (“GENERATE MODIFIED IMAGE”) following step 320, the corrected image IC is generated by the image processing block 112, for example by solving equation (2) using the raw image IB, the gain image G and the bias image B generated in step 320.

[0131] Figure 5 represents, in the form of a flowchart, images involved in process 300 of figure 3 according to an embodiment of the present description.

[0132] During the process 300 described in relation to Figure 3, at least two raw reference images 510 (“IBC”) are captured during the calibration phase, by example by a camera. At least two of the raw reference images 510 are used to generate a gain image 520, represented by the gain image G. At least one of the raw reference images 510 is used to generate at least one reference bias image 530, represented by the set of reference bias images Bl, B2,..., Bn forming the bias family BF.

[0133] The raw 540 image (“IB”) is captured, for example by the camera.

[0134] According to one embodiment, the raw image 540 is corrected using an image processing method different from method 300 in Figure 3 to obtain a new reference bias image Bn+1. The new reference bias image Bn+1 is, for example, added to the bias family BF. In another embodiment, the new reference bias image Bn+1 replaces one or more elements of the bias family BF. The raw image 540 is, for example, an image acquired with a shutter or a blackbody image. The offset image O is, for example, calculated from the raw image 540 using an algorithm known to those skilled in the art and not detailed here, and the new reference bias image Bn+1 is calculated using the following equation:

[0135] [Math 13] B = G • (9 - / z(G • (9). (13)

[0136] A bias image 550, represented by the bias image B, is generated, for example according to step 320 of process 300, based on the gain image 520 and at least two reference bias images 530.

[0137] The corrected image 560 (“IC”) is generated, for example according to step 330 of process 300, based on the bias image 550 and the gain image 520.

[0138] According to one embodiment, the BF bias family is transformed, for example during a dimensionality reduction step, for example by an orthonormalization process, for example by performing a singular value decomposition, for example by using the Gram-Schmidt orthonormalization procedure.

[0139] One advantage of being able to acquire new raw IBC reference images in a less than homogeneous and uncontrolled environment is that the image processing device does not need to be returned to the vendor or manufacturer to update the BF bias family over time. Raw IBC reference images, for example, vary in terms of thermal environment.

[0140] Figure 6 represents an example of a raw image corrected by different models and for different focal plane temperatures of a camera.

[0141] An image of a scene is acquired, for example by the imaging device 100 of Figure 1, for seven focal plane temperatures (“TFPA”): -39.0°C, -24.0°C, -1.0°C, 27.0°C, 38.0°C, 66.0°C and 88.0°C.

[0142] The raw images acquired from the scene are then corrected using a polynomial model of the intensity shift as a function of the focal plane temperature (“(a)”), by the method 300 of Figure 3 using the Fl metric (“(b)”), the F2 metric (“(c)”) and the F3 metric (“(d)”).

[0143] For each of these image processing methods, the same IBC reference raw images are acquired. The IBC reference raw images are obtained, for example, in a stabilized oven, in a thermally spatially homogeneous and time-stabilized environment, and are composed of pairs of a first blackbody at 70 °C and a second blackbody at 10 °C acquired in 15 different thermal environments corresponding to 15 focal plane temperature measurements distributed in the calibration range [-23 °C, 62 °C].

[0144] When the temperature of the focal plane of the image to be corrected is outside the calibration range—that is, for -39.0°C, -24.0°C, 66.0°C, and 88.0°C—images corrected using the F2 and F3 metrics offer a correction where contours and scene texture are more clearly defined than images corrected using the polynomial model. Images corrected using the Fl metric allow for fairly good contour definition, but the rendering of texture, such as the sky, is degraded by line and column effects.

[0145] In conclusion, with focal plane temperature interpolation at -1.0°C, 27.0°C, and 38.0°C, the 300 method provides a correction quality at least similar to the polynomial model. With extrapolation at 39.0°C, -24.0°C, 66.0°C, and 88.0°C, the 300 method offers a corrected image of better quality than the polynomial model.

[0146] Figure 7 represents another example of a raw image corrected by different models and for different focal plane temperatures of a camera.

[0147] The image of a scene in Figure 7 is, for example, acquired by the imaging device 100 in Figure 1, for seven focal plane temperatures (“TFPA”): -39.0°C, -24.0°C, -1.0°C, 27.0°C, 38.0°C, 66.0°C and 88.0°C.

[0148] The raw images acquired from the scene are then corrected by the polynomial model (“(a)”) mentioned in relation to Figure 6 and by the 300 process of Figure 3 using the Fl metric (“(b)”), the F2 metric (“(c)”) and the F3 metric (“(d)”).

[0149] For each of these image processing methods, the same raw IBC reference images are acquired. The raw IBC reference images are obtained, for example, in a spatially and thermally inhomogeneous environment. Calibration image pairs are acquired in 12 different thermal environments. Note that, due to spatial thermal inhomogeneity, the measured focal plane temperature, within the range [-21°C, 43°C], is not representative of the entire thermal environment.

[0150] The IC-corrected images corresponding to focal plane temperatures of -1,0°C, 27,0°C, and 38,0°C are therefore within the calibration range. Images corrected using the F2 and F3 metrics are comparable to, or of better quality than, images corrected using the polynomial model, for which row and column effects are visible, particularly at -1,0°C.

[0151] When the temperature of the focal plane of the image to be corrected is outside the calibration range, i.e. for -39.0 °C, -24.0 °C, 66.0 °C and 88.0 °C, images corrected using the F metric as defined by Fl, F2 and F3 offer a correction where it is possible to better distinguish the contours and texture of the scene than for images corrected using the polynomial model.

[0152] In conclusion, for raw reference IBC images acquired in a spatially thermally Tl The inhomogeneous 300 process offers a higher quality corrected image than the polynomial model. The 300 process is therefore more robust to changes in the thermal environment than the polynomial model.

[0153] Figure 8 is a flowchart 800 illustrating the image processing method 300 detailed in relation to Figure 3, according to one embodiment of the present description.

[0154] Calibration or calibration data 810 of process 300 include at least two raw reference images 510 (“IBC”), for example captured during the calibration or calibration phase.

[0155] Each of the at least one image of gain 520 (“G”) is for example generated based on at least two raw reference images 510, for example based on a pair of raw reference images 510, for example by subtracting (“-”) a first image 812 and a second image 814.

[0156] The calibration phase of process 300 is illustrated by a rectangle referenced 820. During the calibration phase, the set of one or more reference bias images 530 ("B1") is calculated, for example, by multiplying ("x") pixel by pixel each raw reference IBC image by at least one image with gain G and then subtracting its spatial mean value. In the example in Figure 8, the gradient ("V") of each reference bias image 530 is calculated ("VB1") and the family VBF = {VB VB2,..., VB n The calculated J is orthonormalized, for example by performing a singular value decomposition (SVD). In other examples, not shown in Figure 8, the BF family of reference bias images 530 is directly orthonormalized without applying a gradient to the reference bias images 530. A pass-through matrix (“P”) is calculated, for example, during the orthonormalization step. The pass-through matrix P is configured, for example, to obtain an orthonormal basis from the BF family. For example, the orthonormal basis BF' = {B1', B2Bm'} (“Bl'”) is obtained by multiplying (“x”) each reference bias image 530 by the pass-through matrix P. The process 300 is configured, for example, so that the set of one or more reference bias images 530 and the pass-through matrix P are calculated during the calibration phase 820.

[0157] At least one image with a gain of 520 and the orthonormal basis BF' ("Bl'") are, for example, stored in a memory, for example in data memory 206. This embodiment has the advantage of using relatively little storage space. Alternatively, at least one image with a gain of 520, the set of one or more reference bias images 530 ("Bl"), the gradient of the orthonormal basis ("VB1'") and the transition matrix P are stored in a memory, for example in data memory 206. This embodiment has the advantage of using relatively less processing power from the processor 210.

[0158] The execution phase of process 300 is illustrated by a rectangle referenced 830. During the execution phase, the raw image 540 ("IB") is multiplied ("x") by a pixel-by-pixel product to at least one image with a gain of 520 to obtain an image "GIB". In some embodiments, the gradient ("V") of the image "GIB" is calculated ("VGIB"). For example, the gradient ("V") of the orthonormal basis ("Bl'") is calculated ("VB1'") during the execution phase. Alternatively, the gradient ("V") of the basis The orthonormal coefficient ("Bl'") is calculated ("VB1'") during the calibration phase and is stored in memory, for example in data memory 206. The coefficients cc = (â 1; â2, &n) ( " of an image decomposition of the estimated bias B of the raw image 540 in BF' are for example calculated on the basis of VGIB and the gradient of the orthonormal basis ("VB1'") and for example on the basis of the image quality metric F, for example using the equation:

[0159] [Math 14] Armin F(VGIB - Y a t VB^. (14) «i Z— i L

[0160] The image of the estimated bias ("B") of the raw image 540 in BF' is, for example, calculated by multiplying ("x") the coefficients at = â2, â n ~) ("â") to the elements of the orthonormal basis ("Bl'") and adding them together ("") according to: The process 300 is configured for example so that the image of the estimated bias B is calculated during the execution phase 830.

[0161] The corrected image 560 (“IC”) is calculated for example, for example in an application phase 840, by subtracting (“-”) the GIB image and the estimated bias image B.

[0162] In one embodiment, the calibration phase 820 is implemented by a calibration device. In another embodiment, the execution phase 830 and / or the application phase 840 are implemented by a processing device. The processing device is, for example, an embedded device.

[0163] One advantage of the 300 method is that even if the image of the estimated bias B sought is strongly While irregular, this is not the case for the IC-corrected image we wish to calculate. A metric favoring the regularity of the IC-corrected image is therefore very unlikely to lead to the estimation of an aberrant image of the estimated bias B, especially since the number of elements n in the family of bias images BF will, in practice, be significantly smaller than the size, MxN, of the IC-corrected image. Disregarding the metadata associated with the elements of the bias image family BF confers a very high degree of robustness to the 300 method, which allows the exploitation of BF elements by considering only the quality of the IC-corrected image.

[0164] According to one embodiment, the imaging device 100 does not include a shutter. The bias image family BF is generated from the raw reference images IBC, acquired without shutters, during an initial calibration phase or subsequently, and allows the calculation of the estimated bias image B corresponding to the raw image IB. The image processing method 300 allows for continuous and latency-free acquisition of raw images, without interruption to acquire new reference images with a shutter.

[0165] In another embodiment, the imaging device 100 includes a shutter. For example, the raw IBC reference images are acquired using the shutter. The number of raw IBC reference images used for the image processing method 300 can be relatively small and does not require periodic acquisition of new raw reference images.

[0166] One advantage of the described embodiments is that a small number of raw reference images is sufficient, thus requiring little memory. Solving the equations presented in method 300 requires few resources. digital. The 300 process has low algorithmic complexity and is, for example, implemented in an embedded imaging device. According to one embodiment, the generation of the gain image G and the family of bias images BF, and for example of the orthonormal basis BF', is carried out during an initial phase, with a landed electronic device and the calculation of the estimated bias image B of the raw image is carried out by the embedded image processing device 100.

[0167] The responsive image The shift image O, which appears in equation (1), is highly dependent on the thermal environment, particularly the temperature of the focal plane during the acquisition of the raw image IB. Method 300 includes the generation of reference bias images Bl, B2, ..., Bn to solve equations (3) and (4), which do not require the estimation of the responsiveness image or the shift image. The reference bias images Bl, B2, ..., Bn, and therefore Method 300, are less dependent on the thermal environment during the acquisition of the raw reference images Bl, B2, ..., Bn and the raw image IB. In particular, the 300 process allows extrapolation and generation of a good quality IC corrected image for a focal plane temperature outside a range used during the acquisition of the reference bias images Bl, B2,..., Bn.

[0168] In one embodiment, the estimated bias image B associated with the raw image IB is generated from an orthonormal basis, the orthonormal basis being constructed on the basis of the family of bias images BF. As an example, the family of bias images BF is orthonormalized to obtain an orthonormal basis BF'. The estimated bias image B then corresponds to the orthogonal projection of the term product The gain image G and the raw image IB are decomposed onto Vect(BF'). Alternatively, the VBF family of gradients of the bias images is orthonormalized to obtain an orthonormal basis VBF'. The coordinates of VB are, for example, calculated in the basis VBF', and the coefficients α of a decomposition of VB into VBF, and therefore of a decomposition of B into BF, can be recovered, for example, by a change of basis. In one embodiment, the orthonormalization of the bias image family BF or of the VBF family is performed during the calibration phase, for example, by a device external to the imaging device 100. Advantageously, the orthonormalization step thus has a negligible impact on the computational cost of the image processing method 300. The calculated orthonormal basis is, for example, stored in the imaging device 100, for example, in the data memory 206.

[0169] The calculation of the cc coefficients, for example, involves the least-squares inversion of the Gram matrix of BF or VBF. Inverting the Gram matrix, for instance, generates significant numerical errors when its conditioning is too high. For example, beyond about thirty images, the conditioning becomes too poor, because it is too imprecise, to be able to use the projection results. Advantageously, the orthonormalization step avoids these numerical errors. Consequently, a relatively large number of images, for example, the entire set of available images, is used. Furthermore, the entire set of pixels in the images is used. The orthonormalization step allows the current raw IB image to be corrected in a stable and relatively easy way, using the useful information available in the entire set of raw images. IBC reference, and not just in a small subset. Advantageously, the orthonormal basis has a dimension less than or equal to the dimension of the BF or VBF family. Dimensionality reduction is performed, for example, during the orthonormalization step. The orthonormalization step also allows the use of the 300 image processing method in contexts where it is relatively difficult to define a priori a notion of "proximity" between the raw IB image to be corrected and the raw IBC reference images, particularly when image degradation is not only explainable by the focal plane temperature, but is also attributable to the variation of other factors, for example, the thermal environment, parasitic fluxes, electronic parameters, etc.

[0170] In one embodiment, the calculation of the image quality metric F is performed based on a calculation of the image value gradient, for example, according to the F2 or F3 metric. Advantageously, such a metric is suitable for a relatively inhomogeneous raw image IB, for example, one containing relatively large variations in values ​​over a relatively short distance. Compared to the Fl metric described previously, such a metric F2 or F3 provides additional robustness against the heterogeneity of the raw image IB, since the Fl metric performs better when the raw image IB is smooth.

[0171] Various embodiments and variants have been described. Those skilled in the art will understand that certain features of these various embodiments and variants could be combined, and other variants will become apparent to them. In particular, the BF bias image family can be represented by a variety or a vector space and several families of bias images BF are for example used to correct the raw image IB, each family of bias images BF being used to correct an area of ​​the raw image IB.

[0172] Finally, the practical implementation of the described methods and variants is within the reach of the person in the trade, based on the functional indications given above.

Claims

DEMANDS 1. Device for processing a raw infrared (IB) image, comprising: - a memory (206) configured to store at least one gain (G) and a family of at least one bias image (BF); and - a processor (210) configured to calculate a corrected image (IC) from the raw image, at least one gain and a bias image (B) associated with the raw image, wherein the bias image associated with the raw image is generated from at least one gain (G), the family of at least one bias image (BF), and an image quality metric (F, Fl, F2, F3) associated with the corrected image (IC).

2. Device f according to claim 1, further comprising an infrared camera configured to acquire at least two raw reference images (IBC) and the raw image (IB) and in which the processor is configured to calculate at least one gain (G) from the at least two raw reference images (IBC), the processor being further configured to calculate one (B1, B2,..., Bn) from the family of at least one bias image (BF) from at least one of the at least two raw reference images (IBC).

3. Device f according to claim 2, wherein the infrared camera is uncooled.

4. Device f according to any one of claims 1 to 3, wherein the processor is further configured to calculate the image quality metric (Fl, F2, F3).

5. Device according to claim 4, wherein the calculation of the image quality metric (F2, F3) is performed on the basis of a gradient calculation of image values.

6. Device according to claim 4, wherein the calculation of the image quality metric (F2) is performed according to a semi-norm defined by a norm 2 of the gradient satisfying the following equation: Vv e HV 2 , F2(v) =|| Vv ll2,2 : = It rv(xy) ||| where v denotes an image comprising M rows and N columns, 2 denotes the domain {0, 1, M — 1} X {0, 1,..., N — 1} e t (V x v, V y v) represents a linear gradient operator with finite differences.

7. Device according to claim 4, wherein the calculation of the image quality metric (F3) is performed according to a total variation defined by the following equation: Vv e HV 2, F3(v) =|| Vv || 1,2 := Il Vv(x,y) ||2 (x,y)EÎ2 where v denotes an image comprising M rows and N columns, 2 denotes the domain {0, 1, M — 1} X {0, 1,..., N — 1} e t (V x v, V y v) represents a linear gradient operator with finite differences.

8. Device according to any one of claims 1 to 7, wherein the processor is further configured to calculate an estimate (B) of the bias image (B) associated with the raw image (IB) by finding a linear combination of at least one bias image (BF) that minimizes the image quality metric (F), according to the following equation: Bi such as BG argmln BeVect it BF ^FÇG • IB - B) É s ' i=l where n denotes the number of reference bias images in l ' at least one bias image (BF), a, denotes a coefficient of the linear combination and Bi corresponds to a bias image in l ' at least one bias image (BF).

9. Device f according to any one of claims 1 to 8, wherein the bias image associated with the raw image is generated from an orthonormal basis, the orthonormal basis being constructed, during a prior calibration phase, on the basis of the family of at least one bias image (BF).

10. A method for processing a raw image (IB) in the infrared, the method comprising: - the recording of at least one gain (G) and a family of at least one bias image (BF) in a memory (206); and - the calculation of a corrected image (IC) from the raw image, the gain and a bias image (B) associated with the raw image, in which the bias image associated with the raw image is generated from at least one gain (G), the family of at least one bias image (BF), and an image quality metric (F, Fl, F2, F3) associated with the corrected image (IC).

11. Method according to claim 10, further comprising the acquisition, by an infrared camera, of at least two raw reference images (IBC) and of the raw image (IB) and the calculation of at least one gain (G) from the at least two raw reference images (IBC).

12. Method according to claim 11, further comprising calculating one (Bl, B2,..., Bn) from at least one bias image (BF) from at least one of at least two raw reference images (IBC).

13. A method according to claim 11 or 12, wherein at least two raw reference images (RBIs) are captured in a thermally dynamic environment of arbitrary dynamics 14. Method according to any one of claims 10 to 13, further comprising a calculation of the image quality metric (F1, F2, F3).

15. Method according to claim 14, wherein the calculation of the image quality metric (F2, F3) is carried out on the basis of a gradient calculation of image values.

16. A method according to claim 14, wherein the calculation of the image quality metric (F2) is performed according to a semi-norm defined by a norm 2 of the gradient satisfying the following equation: Vv G HV 2 , F2(v) =|| Vv || 2,2 := Il ∇v(x,y) ||2 (x,y)e / 2 where v denotes an image comprising M rows and N columns, 2 denotes the domain {0, 1, M — 1} X {0, 1,..., N — 1} e t (V x v, V y v) represents a linear gradient operator with finite differences.

17. A method according to claim 14, wherein the calculation of the image quality metric (F3) is performed according to a total variation defined by the following equation: Vv e HV 2 , F3(v) =|| Vv || 1,2 := Il ∇v(x,y) ||2 (x,y)E / 2 where v denotes an image comprising M rows and N columns, 2 denotes the domain {0, 1, M — 1} X {0, 1,..., N — 1} e t (V x v, V y v) represents a linear gradient operator with finite differences.

18. A method according to any one of claims 10 to 17, wherein an estimate (B) of the bias image (B) associated with the raw image (IB) is calculated by a linear combination of at least one bias image (BF) that minimizes the image quality metric (F), according to the following equation: B̂ = Σ â_i B_i such that B̂ ∈ argminB∈Vect(BF)F(G · IB - B) i=l where n denotes the number of reference bias images in the at least one bias image (BF), a denotes a coefficient of the linear combination and Bi corresponds to a bias image of the at least one bias image (BF).

19. Method according to any one of claims 10 to 18, wherein the bias image associated with the raw image is generated from an orthonormal basis, the orthonormal basis being constructed, during a prior calibration phase, on the basis of the family of at least one bias image (BF).

20. System for processing a raw infrared (IB) image, comprising: - a calibration device configured to calculate at least one gain (G) from at least two raw images reference (IBC) and to calculate one (Bl, B2, ..., Bn) from a family of at least one bias image (BF) from at least one of the at least two raw reference (IBC) images; and an infrared raw image processing device (IB) configured to record at least one gain (G), to record the family of at least one bias image (BF), to calculate a corrected image (IC) from the raw image (IB), the at least one gain and a bias image (B) associated with the raw image, wherein the bias image associated with the raw image is generated from the at least one gain (G), the family of at least one bias image (BF), and an image quality metric (F, Fl, F2, F3) associated with the corrected image (IC).

Images