Correlation-based adaptive image fusion
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-02-10
- Publication Date
- 2026-08-13
Smart Images
Figure US2025015278_13082026_PF_FP_ABST
Abstract
Description
[0001] Attorney Docket No.: 56113-0847WO1
[0002] CORRELATION-BASED ADAPTIVE IMAGE FUSION
[0003] BACKGROUND
[0004] This specification relates to image processing, and, more particularly, to image enhancement. Many modern computing devices, including mobile phones, personal computers, and tablets, include images sensors, e.g., still or video cameras, that can capture image data. Image data captured by an image sensor is often initially processed as part of an image processing pipeline in order to prepare the captured image data for further processing or consumption.
[0005] SUMMARY
[0006] This specification describes techniques for processing images. In particular, the specification describes techniques for reducing artifacts when performing fusion of a pair of images captured by two different cameras, i.e., processing a source image captured by a first camera and a reference image captured by a second camera to generate a fused image.
[0007] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.
[0008] This specification describes techniques for transforming a pair of images from an original domain (e.g., a spatial domain) to a transformed domain (e.g., a frequency domain) and then determining respective weight values for different image patches obtained from the pair of images based on measures of correlation between the representations of the image patches in the transformed domain. Weight values for the different image patches are dynamically adjusted to account for the variance in the measures of correlation in the transformed domain - for example a lower weight value will be assigned to a first image patch from the source image when the measure of correlation is low.
[0009] In this way, influence of visual artifacts (e.g., aliasing artifacts, warping artifacts, or other visible distortions) in the fused image contributed by the source image can be reduced or eliminated. A fused image generated by the described techniques will thus have with fewer artifacts, e.g., it may have fewer distortions and a more accurate depiction of structures than the source image, or even an alternative fused image that might have been generated by a baseline fusion technique that does not use correlation-based weights.Attorney Docket No.: 56113-0847WO1
[0010] The capability to generate fused images with fewer artifacts is particularly important for mobile computing devices - e.g., mobile phones, tablets, laptops, and wearables - for which the sizes of the devices’ image sensors are typically limited. The limitation on the sizes of the image sensors, in turn, impacts the hardware configuration of the image sensors, resulting in an increased likelihood of generating source images that contain visual artifacts (as compared to a full-sized, standalone camera having larger sizes).
[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
[0012] BRIEF DESCRIPTION OF THE DRAWINGS FIG. l is a block diagram of an example system.
[0013] FIG. 2 is an example illustration of operations performed by a correlation-based adaptive fusion block.
[0014] FIG. 3 is an example illustration of applying a discrete cosine transformation (DCT) to an image patch.
[0015] FIG. 4 is a flowchart of an example process for generating a fused image as part of an image processing pipeline.
[0016] FIG. 5 shows a quantitative example of image enhancement that can be achieved by using the system described in this specification.
[0017] Like reference numbers and designations in the various drawings indicate like elements.
[0018] DETAILED DESCRIPTION FIG. 1 is a diagram of an example system 100. The system 100 includes an image signal processor (ISP) 105 and two or more sensors, e.g., a first image sensor 110A and a second image sensor HOB. In some implementations, the system 100 also includes one or more of: a central processor unit (CPU), a graphics processor (GPU), a memory, and so forth.
[0019] The CPU can be implemented using any suitable instruction set architecture, and can be configured to execute any instructions defined in that instruction set architecture. The GPU can include graphics processing circuitry that can be configured to execute graphics software to perform a part or all of the graphics operation, or hardware acceleration of certain graphicsAttorney Docket No.: 56113-0847WO1
[0020] operations. The memory can be any type of transitory or non-transitory computer readable medium capable of storing data accessible by the system 100, such as volatile and non-volatile memory.
[0021] In some implementations, some or all of the hardware components mentioned above are integrated onto a single system-on-a-chip (SOC). The SOC can be an integrated circuit that includes the aforementioned hardware components, and possibly other hardware components of the system, on a single silicon substrate or on multiple interconnected dies, e.g., using silicon interposers, stacked dies, or interconnect bridges.
[0022] The SOC is an example of a device that can be installed on or integrated into any appropriate computing device. Example computing devices include personal computers (e.g., desktop computers, laptop computers, and tablet computers), gaming devices, mobile communication devices (e.g., smart phones), digital assistant devices, augmented reality devices, virtual reality devices, and wearable computing devices (e.g., smart watches), to name just a few.
[0023] The image sensors 110A-B can each be any type of image sensor suitable for capturing raw image data, i.e., any type of image sensor that is responsive to captured light. Example image sensors include active-pixel sensors (e.g., complementary metal-oxide-semiconductor (CMOS) sensors) or charge coupled device (CCD) sensors on a camera, video camera, or a computing device.
[0024] In some implementations, the image sensors 110A-B have identical optical characteristics or properties (although they may be placed at different locations), whereas, in other implementations, the image sensors 110A-B have different optical characteristics or properties. For example, the image sensors 110A-B can be different ones of a telephoto sensor, a wide-angle sensor, an ultra-wide-angle sensor, or an infrared sensor.
[0025] The system 100 uses the image signal processor 105 to process the raw image data captured by the image sensors 110A-B to produce an output image that can be used by another hardware component (e.g., the CPU or GPU) for various immediate or future purposes, e.g., for rendering and display (e.g., on a display device of the computing device), for video coding (e.g., by a video encoder), for further processing (e.g., by a computer vision application, an image editing application, a photo album application, or another application installed on the computing device), for storage (e.g., at the memory of the system 100), among other uses.Attorney Docket No.: 56113-0847WO1
[0026] As a particular example, the image signal processor 105 can be used in a detail transfer use case to transfer additional details that are recoverable from a first image (e.g., a higher-resolution image) to a second image (e.g., a lower-resolution image).
[0027] As another particular example, the image signal processor 105 can be used in a denoising and / or deblurring use case to remove or decrease noise and / or blurring from a first image. Noise and blurring can be caused by, for example, capturing the first image in low light conditions with low-end cameras.
[0028] As another particular example, the image signal processor 105 can be used in a high dynamic range (HDR) fusion use case to merge a first image (e.g., a short-exposure image) and a second image (e.g., a long-exposure image) by using a suitable HDR fusion algorithm, e.g., an exposure fusion algorithm or a local Laplacian filters algorithm.
[0029] The image signal processor 105 has discrete functional blocks that each apply a specific operation to the raw image data to generate the final output image. In some implementations, each functional block is an independent block that includes dedicated hardware that corresponds to a different stage of an image processing pipeline, whereas, in other implementations, two or more of the functional blocks share some hardware.
[0030] For example, the functional blocks of the image signal processor 105 can include one or more of: a demosaicing block (that converts the raw Bayer-patterned image data into a full-color image), a color correction block (that adjusts the colors in the image to ensure they are accurate and match real-world colors), a noise reduction block (that minimizes pixel defects and reduces noise present), an auto exposure block (that adjusts the brightness of the image by changing the exposure settings based on the amount of light in the scene), an auto white balance block (that corrects the white balance of the image, adjusting for different lighting conditions so that the colors appear natural), and so forth.
[0031] In particular, the functional blocks of the image signal processor 105 include a correlation-based adaptive fusion block 120 that corresponds to a correlation-based adaptive fusion stage of the image processing pipeline. At the correlation-based adaptive fusion stage, the image signal processor 105 generates a fused image 122 based on the first raw image data 112A captured by the first image sensor 110A and the second raw image data 112B captured by the second image sensor HOB.Attorney Docket No.: 56113-0847WO1
[0032] As part of the correlation-based adaptive fusion stage, the correlation-based adaptive fusion block 120 computes correlation-based weights for different image patches and then generates the fused image 122 in accordance with the correlation-based weights. The use of the correlation-based weights improves the quality of the fused image 122 by reducing the likelihood that the fused image 122 might contain visual artifacts, e.g., aliasing artifacts, warping artifacts, or other visible distortions. A fused image 122 having an improved quality, in turn, improve the quality and performance of subsequent computer vision algorithms that can be used to process the fused image 122.
[0033] In the example of FIG. 1, the correlation-based adaptive fusion block 120 corresponds to the last stage of the image processing pipeline that is executed at the image signal processor 105. That is, the fused image 122 generated by the correlation-based adaptive fusion block 120 is the final output image produced by the image signal processor 105. However, this is not required in all implementations.
[0034] In other examples, the correlation-based adaptive fusion block 120 can correspond to any intermediate stage of the image processing pipeline, and the image signal processor 105 can perform various other image processing functions on the fused image 122 before providing the final output image.
[0035] For example, instead of or in addition to the image processing functions mentioned above, the image signal processor 105 can perform one or more of: sharpening, tone mapping, color space conversion (e.g., convert the fused image 122 from one color space format to another color space format (e.g., from RGB format to YCbCr format), autofocus, gamma correction, image translation operation, image resizing (e.g., horizontal and vertical scaling), image stabilization transformation, keypoint detection, generation of histogram-of-orientation gradients (HOG), and normalized cross correlation (NCC), and so forth, to further improve the overall quality and clarity of the fused image 122.
[0036] FIG. 2 is an example illustration of operations performed by the correlation-based adaptive fusion block 120.
[0037] The correlation-based adaptive fusion block 120 receives a source image (“image 1”) 210A and a reference image (“image 2”) 210B. For example, the source image can be generated as a result of the preceding stages of the image processing pipeline based on the first raw image data 112A captured by the first image sensor 110A. Likewise, the reference image can beAttorney Docket No.: 56113-0847WO1
[0038] generated as a result of the preceding stages of the image processing pipeline based on the second raw image data 112B captured by the second image sensor HOB.
[0039] In image fusion, one of the images to be fused may be designated as the “reference image,” and another image that is to be part of the fusion operation may be designated as “source image.” In various implementation, the decision of which image in a pair of captured images should serve as the reference image may be based on, e.g., an image quality measure (such as sharpness, resolution, noise level, etc.), a capture timing measure (such as the image captured closest in time to a received capture request), a hardware requirement (such as an image captured by a designated image sensor), or any other requirements.
[0040] The correlation-based adaptive fusion block 120 receives a pair of image patches that include a first image patch 215A from the source image 210A, and a second image patch 215B from the reference image 210B. For example, the correlation-based adaptive fusion block 120 can obtain the pair of image patches by applying a sliding window to each of the source image and the reference image. The window can have a rectangular shape, a circular shape, a triangular shape, or another shape. The stride (e.g., the number of pixels) by which a window shifts across the source image can be the same as the stride by which a window shifts across the reference image.
[0041] With a sliding window be applied to the source image 210A, the first image patch 215A can be a pixel block, which includes a subset of the pixels of the source image 210A. Likewise, the second image patch 215B can be a pixel block, which includes a subset of the pixels of the reference image 21 OB. When the window has a square shape, the pixel block can have a width and a height of an equivalent number of pixels. For example, in FIG. 2, each of the first image patch 215 A and the second image patch 215B includes N pixels along a horizontal dimension and N pixels along a vertical dimension.
[0042] Other techniques can also be used by the correlation-based adaptive fusion block 120 to obtain the pair of image patches. For example, the correlation-based adaptive fusion block 120 can partition the source image (or, analogously, the reference image) into a plurality of partitions, and use each partition as an image patch. Examples of image partitioning techniques that can be used include an object detection-based technique, a saliency map-based technique, and a superpixel technique.Attorney Docket No.: 56113-0847WO1
[0043] By repeatedly performing the following operations for all of the pairs of image patches that can be obtained from the source image and the reference image to generate a fused image patch 260 that corresponds to each pair of image patches, the correlation-based adaptive fusion block 120 can generate a fused image 122.
[0044] In some implementations, the correlation-based adaptive fusion block 120 can perform the following operations for some or all of the pairs of image patches in parallel, in order to reduce the overall time needed to generate the fused image 122. Thus, a fused image patch that corresponds to one pair of image patches can be generated in parallel with a fused image patch that corresponds to another pair of image patches.
[0045] The correlation-based adaptive fusion block 120 applies a first frequency transformation 220Ato the first image patch 215Ato transform the first image patch in an original domain (e.g., a spatial domain) into a first representation 230A of the first image patch in a transformed domain (e.g., a frequency domain), and applies a second frequency transformation 220B to the second image patch 215B to transform the second image patch that is in the same original domain as the first image patch into a second representation 23 OB of the second image patch that is in the same transformed domain as the representation of the first image patch. In other words, by applying the first and second frequency transformations, the correlation-based adaptive fusion block 120 obtains the transformed domain representations (e.g., frequency domain representations) of the first and second image patches.
[0046] In some implementations, the first frequency transformation 220A and the second frequency transformation 220B can each be a discrete cosine transformation (DCT). Unlike a discrete Fourier transform (DFT) which uses both cosine and sine functions, the discrete cosine transformation (DCT) uses only cosine functions. Using only cosine functions means that the discrete cosine transformation (DCT) produces only real numbers, because all waves have the same phase, while a discrete Fourier transform (DFT) can produce complex numbers that contain a phase and amplitude. This simplifies the computation and storage of the DCT coefficients.
[0047] The first representation 230A of the first image patch (or, analogously, the second representation 230B of the second image patch) in the transformed domain includes a plurality of DCT coefficients. The plurality of DCT coefficients include a DC coefficient and a plurality of AC coefficients. The DC coefficient represents the average color of the first image patch (or, analogously, the second image patch). The plurality of AC coefficients represent color changeAttorney Docket No.: 56113-0847WO1
[0048] across the first image patch (or, analogously, the second image patch). AC coefficients having smaller values represent low-frequency color change, or gradual color change across the region. AC coefficients having greater values represent high-frequency color change, or color which changes rapidly from one pixel to another within the pixel block.
[0049] FIG. 3 is an example illustration of applying a discrete cosine transformation (DCT) to an image patch. In the example of FIG. 3, each of the first image patch and the second image patch is a pixel block in the size of 8*8 (i.e., N = 8), which includes 8 pixels along a horizontal dimension and 8 pixels along a vertical dimension. In this example, the first representation of the first image patch (or, analogously, the second representation of the second image patch) in the transformed domain can include 1 DC coefficient and 63 AC coefficients. The DC coefficient represents the average color of the 8^8 pixel block. The 63 AC coefficients represent color change across the 8x8 pixel block.
[0050] Within the first representation of the first image patch (or, analogously, the second representation of the second image patch) in the transformed domain, the DC coefficient and the plurality of AC coefficients can be arranged in a predetermined order. For example, as illustrated in FIG. 3, the DC coefficient is located in the top left corner of the representation, followed by the 63 AC coefficients that are arranged one after another, from left to right, and from top to bottom.
[0051] The correlation-based adaptive fusion block 120 determines a measure of correlation 235 between the first representation 230A of the first image patch in the transformed domain that is generated by applying the first frequency transformation to the first image patch, and the second representation 230B of the second image patch in the transformed domain that is generated by applying the second frequency transformation to the second image patch.
[0052] In some implementations, the measure of correlation 235 between the first representation and the second representation can be represented as a correlation value. That is, to determine the measure of correlation 235, the correlation-based adaptive fusion block 120 computes a correlation value between (i) the plurality of AC coefficients included in the first representation 230A of the first image patch and (ii) the plurality of AC coefficients included in the second representation 23 OB of the second image patch. The correlation value represents the degree to which the change in (i) and the change in (ii) is associated. For example, the correlation valueAttorney Docket No.: 56113-0847WO1
[0053] can represent the strength of a relationship, e.g., a linear or nonlinear relationship, between (i) and (ii).
[0054] In some implementations, a correlation value greater than zero indicates a positive relationship, while a correlation value less than zero signifies a negative relationship. A correlation value close to zero indicates a weak relationship between (i) and (ii). In some implementations, the possible range of values for the correlation value is -1.0 to 1.0 (inclusive on both ends). In some implementations, the correlation value is computed by dividing a covariance between (i) and (ii) by a product of a standard deviation of (i) and a standard deviation of (ii).
[0055] The correlation-based adaptive fusion block 120 computes a first weight value 240A for the first image patch and a second weight value 240B for the second image patch to be used when generating a fused image patch 260 from the first image patch and the second image patch. For the first image patch, the first weight value 240Ais dependent on the measure of correlation, e.g., dependent on the correlation value. Analogously, for the second image patch, the second weight value 240B is also dependent on the measure of correlation, e.g., dependent on the correlation value. Hence, in FIG. 2, the first and second weight values are referred to as “correlation-guided weights.” A specific example of determining the first and second weight values based on the measure of correlation will be described below with reference to FIG. 4.
[0056] The correlation-based adaptive fusion block 120 generates a weighted representation 245 based on the first representation 230A of the first image patch, the second representation 230B of the second image patch, and in accordance with first and second weight values 240A, 240B that have been determined for the first and second image patches. Just as the first representation of the first image patch and the second representation of the second image patch, the weighted representation 245 is in the transformed domain.
[0057] In the weighted representation 245, the first representation of the first image patch is weighted by the first weight value for the first image patch, and the second representation of the second image patch is weighted by the second weight value for the second image patch. In some implementations, generating the weighted representation involves pixelwise multiplying the first representation of the first image patch to the first weight value for the first image patch to determine a first product, pixelwise multiplying the second representation of the second image patch to the second weight value for the second image patch to determine a second product, and then determining a summation of the first product and the second product.Attorney Docket No.: 56113-0847WO1
[0058] Having generated the weighted representation 245, the correlation-based adaptive fusion block 120 applies an inverse frequency transformation 250 to the weighted representation 245 to transform the weighted representation in the transformed domain to a fused image patch 260 that is in the same original domain as the first image patch 215A and the second image patch 215B. In some implementations, the inverse frequency transformation can be an inverse discrete cosine transformation (IDCT).
[0059] FIG. 4 is a flow diagram of an example process 400 for generating a fused image as part of an image processing pipeline. The image processing pipeline includes multiple stages. The multiple stages include a correlation-based adaptive fusion stage.
[0060] For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system that includes an image signal processor, e.g., the system 100 depicted in FIG. 1 which includes an image signal processor 105, appropriately configured in accordance with this specification, can perform the process 400.
[0061] In general, the system can obtain a pair of image patches, and perform an iteration of the process 400 on the pair of image patches to generate a fused image patch that corresponds to the obtained pair of image patches. The fused image patch can be a subset of the fused image that is being generated by the system as part of the image processing pipeline executed at the image signal processor.
[0062] In some implementations, the system can repeatedly perform iterations of the process 400 on each of some, but fewer than all, pairs of image patches obtained by the system to generate a corresponding, fused image patch for the pair of image patches. For example, an iteration of the process 400 can be performed on one or more pair of image patches that each contain pixels from a region of interest in a source image or a reference image. Then, for remaining pairs of image patches (that are not fused by performing iterations of the process 400), the system can use one or more other image fusion techniques to generate a corresponding, fused image patch for each remaining pair of image patches. Examples of the other image fusion techniques include wavelets-based image fusion techniques, curvelet-based image fusion techniques, and DCT-based image fusion techniques (that do not use correlation-based weights). Correspondingly, in these implementations, a fused image can be generated by using the correlation-based adaptive fusion technique and at least one other image fusion technique.Attorney Docket No.: 56113-0847WO1
[0063] In some implementations, the system can repeatedly perform iterations of the process 400 in parallel on different pairs of image patches obtained by the system to generate a corresponding, fused image patch for each obtained pair of image patches. By repeatedly performing iterations of the process 400 in parallel on a plurality of pairs of image patches to generate a fused image patch for each pair of image patches, and then concatenating the generate, fused image patches, the system can generate an entire, fused image more quickly, i.e., with reduced latency (measured from the timepoint at which the source image and the reference image become available), and with improved hardware utilization, relative to performing iterations of the process 400 in a sequential order.
[0064] In other words, in these implementations, the fused image includes a concatenation of the fused image patches that have been generated substantially in parallel, where the generation times for the different fused image patches included in the concatenation at least partly overlap each other.
[0065] The system receives a pair of image patches captured by two image sensors (step 402). The two image sensors include a first image sensor and a second image sensor. In some implementations, the two image sensors have identical optical characteristics whereas, in other implementations, the two image sensors have different optical characteristics. For example, the first image sensor can be a wide-angle sensor, while the second image sensor can be a telephoto sensor. Thus, in this example, the first image sensor has a wider field of view than the second image sensor.
[0066] The pair of image patches include a first image patch from a source image and a second image patch from a reference image. For example, the source image can be generated as a result of the preceding stages of the image processing pipeline (that are arranged preceding to the correlation-based adaptive fusion stage) based on the first raw image data captured by the first image sensor. Likewise, the reference image can be generated as a result of the preceding stages of the image processing pipeline based on the second raw image data captured by the second image sensor.
[0067] In some implementations, the system can obtain the first image patch and the second image patch by apply a sliding window on each of the source image and the reference image. In some implementations, the system can obtain the first image patch and the second image patch by using an image partitioning technique, e.g., an object detection-based technique, a saliencyAttorney Docket No.: 56113-0847WO1
[0068] map-based technique, or a superpixel technique. In some implementations, the first and second image patches can each be a pixel block having a rectangular shape, a circular shape, a triangular shape, or another shape. For example, the first image patch and the second image patch can each be a square pixel block in the size of 4*4, 8*8, 16x16, and so forth.
[0069] The system applies a first frequency transformation to the first image patch to generate a first representation of the first image patch. In addition, the system applies a second frequency transformation to the second image patch to generate a second representation of the second image patch (step 404). Both the first and second image patches are in an original domain, while both the first and second representations are in a transformed domain.
[0070] In some implementations, the first frequency transformation and the second frequency transformation are each a discrete cosine transformation (DCT), the original domain is a spatial domain, and the transformed domain is a frequency domain. In these implementations, the first representation of the first image patch in the transformed domain includes a DC coefficient and a plurality of AC coefficients. Likewise, the second representation of the second image patch includes a DC coefficient and a plurality of AC coefficients.
[0071] The system determines a measure of correlation between the first representation of the first image patch and the second representation of the second image patch (step 406). In some implementations, the measure of correlation between the first and second representations can be represented as a correlation value. That is, to determine the measure of correlation, the system computes a correlation value between (i) the plurality of AC coefficients included in the first representation of the first image patch and (ii) the plurality of AC coefficients included in the second representation of the second image patch. The correlation value represents the degree to which the change in (i) and the change in (ii) is associated.
[0072] The system determines respective weights values for use when generating a fused image patch from the first image patch and the second image patch (step 408). The respective weights values include a first weight value for the first image patch and a second weight value for the second image patch. For the first image patch, the system can determine the first weight value based on the measure of correlation, e.g., the correlation value. Analogously, for the second image patch, the system can determine the second weight value based on the measure of correlation, e.g., the correlation value.Attorney Docket No.: 56113-0847WO1
[0073] In some implementations, for the first image patch (or, analogously, the second image patch), the system applies a first weighting function (or, analogously, a second weighting function) to the correlation value to provide the first weight value (or, analogously, the second weight value). The weighting function can be any function, or combination of functions, that can map a correlation value and possibly other data values to a weight value. A specific example of the weighting function will be described further below.
[0074] Because the measures of correlation will typically vary from one pair of image patches to another, the weight values that are dynamically computed by the system for these pairs of image patches based on the measures of correlation will also vary. Moreover, even for a single pair of image patches, the weight values computed by the system for the first and second image patches included in the pair may also be different. Furthermore, the first weight values determined for different first image patches obtained from the same source image will usually vary. Likewise, the second weight values determined for different second image patches obtained from the same reference image will usually vary.
[0075] For example, when the measure of correlation is low, e.g., when the correlation value is close to zero, the first weight value determined by the system for the first image patch that is obtained from the source image can be lower, relative to the second weight value determined by the system for the second image patch that is obtained from the reference image. Assigning lower weight values to the first image patch reduces the influence of artifacts in the fused image patch contributed by the source image (from which the first image patch is obtained). In contrast, when the measure of correlation is high, e.g., when the correlation value is close to one, the first weight value determined by the system for the first image patch that is obtained from the same source image can be higher.
[0076] The system generates a fused image patch based on the first image patch and the second image patch, and in accordance with the first and second weight values that have been determined for the first and second image patches (step 410). To do this, the system generates a weighted representation, and then applies an inverse frequency transformation to the weighted representation to transform the weighted representation to the fused image patch.
[0077] In some implementations, generating the weighted representation involves pixelwise multiplying the first representation of the first image patch to the first weight value for the first image patch to determine a first product, pixelwise multiplying the second representation of theAttorney Docket No.: 56113-0847WO1
[0078] second image patch to the second weight value for the second image patch to determine a second product, and then determining a summation of the first product and the second product.
[0079] Just as the first representation of the first image patch and the second representation of the second image patch, the weighted representation is in the transformed domain. In some implementations, the inverse frequency transformation is an inverse discrete cosine transformation (IDCT), and the fused image patch is in the same original domain as the first image and second image patches.
[0080] An example algorithm for generating a fused image patch based on correlation-based adaptive fusion is shown below.
[0081]
[0082] In the example algorithm shown above, Isrcrepresents the first image patch that is obtained from the source image. The source image is generated based on first raw image data captured by a first image sensor. lref represents the second image patch that is obtained from the reference image. The reference image is generated based on second raw image data captured by a second image sensor. lsrcand Iref may be the inputs to the algorithm. I represents a fused image patch, which may be the output of the algorithm.
[0083] An iteration of the algorithm can thus be applied to each pair of image patches captured by two image sensors to generate a fused image patch that corresponds to the pair of image patches. As mentioned above, in some implementations, iterations of the algorithm can be performed in parallel on different pairs of image patches.
[0084] corr represents the correlation value between (i) the plurality of AC coefficients included in the first representation of the first image patch Msrcand (ii) the plurality of AC coefficientsAttorney Docket No.: 56113-0847WO1
[0085] included in the second representation of the second image patch Mref, where the first representation Msrcis generated by applying a discrete cosine transformation (DCT) to the first image patch Isrc, and the second representation Mref is generated by applying a discrete cosine transformation (DCT) to the second image patch lref.
[0086] In the example algorithm shown above, the first weighting function maps the first representation Msrcto an initial first weight value )src, and then computes a product of the correlation value corr and the initial first weight value mS3.c, and then computes a product of (i) a constant (e.g., one) minus the correlation value a>srcand (ii) an output of an erode function that takes the initial first weight value )srcas input. Next, by computing a summation of the two products, the first weighting function outputs the first weight value G)srcfor the first image patch ^STC ■
[0087] As an example of a morphological function, the erode function could remove small anomaly artifacts by subtracting image objects with a radius smaller than an erode setting, which is defined in number of pixels (e.g., three pixels), by removing perimeter pixels.
[0088] The second weighting function maps the second representation Mref to an initial second weight Mref, and then computes a product of the correlation value corr and the initial second weight >ref, and then computes a product of (i) a constant (e.g., one) minus the correlation value (t)ref and (ii) an output of a dilate function that takes the initial second weight )ref as input. Next, by computing a summation of the two products, the second weighting function outputs the second weight value Qref for the second image patch lref .
[0089] As another example of a morphological function, the dilate function could connect areas that are separated by spaces smaller than a minimum dilate size setting, which is defined in number of pixels (e.g., three pixels), by adding pixels of the eroded pixel data values to the perimeter of each image object remaining after the erode function is applied.
[0090] weight represents an initial weigh mapping function that is used to map the first image patch Msrcto an initial first weight value a)srcand to map the second image patch Mref to an initial second weight Ci)ref - The initial weight mapping function can be any suitable mapping function that depends on the magnitudes, frequencies, or both in a frequency domain representation. For example, the initial weight mapping function be the Fourier weight mapping function described in Delbracio, Mauricio, and Guillermo Sapiro. Hand-held video deblurringAttorney Docket No.: 56113-0847WO1
[0091] via efficient fourier aggregation. IEEE Transactions on Computational Imaging 1.4 (2015): 270-283.
[0092] In the example algorithm shown above, the fused image patch I is generated by applying an inverse discrete cosine transformation (IDCT) to a weighted representation M, which is generated by computing a product of the first weight value Qsrcand the first representation Msrc, followed by computing a product of the second weight value Qref and the second representation Mref, followed by determining a summation of the two products, and followed by dividing the summation of the two products by a summation of the first weight value G)srcand the second weight value Qref .
[0093] FIG. 5 shows a quantitative example of image enhancement that can be achieved by using the system described in this specification. In FIG. 5, the system receives a source image 510 and a reference image 520. The source image 510 is generated based on raw image data captured by a wide-angle sensor. The reference image 520 is generated based on raw image data captured by a telephoto sensor. The wide-angle sensor and telephoto sensor have different optical characteristics, e.g., they have at least different field of views.
[0094] Suppose that the source image 510 has visual artifacts. For example, as shown in circle 512 in FIG. 5, the source image 510 has distortions, where some of the straight line segments in the horizontal direction are curved or even appears distorted when viewed by a user. In this example, the visual artifacts might be due to the intended characteristics or properties of optics associated with the wide-angle sensor that captured the source image 510.
[0095] In other examples, however, the visual artifacts in a source image might be due to unintended characteristics or properties of optics associated with a first image sensor that captured the source image (e.g., lens or image sensor variations), or due to the preceding stages of the image processing pipeline performed on the raw image data captured by the first image sensor (e.g., artifacts introduced during the image processing pipeline, or warping artifacts due to alignment issues).
[0096] Because of the visual artifacts in the source image 510, a first fused image 530 that would be generated by applying a baseline fusion technique to the source image 510 and the reference image 520 will similarly have visual artifacts. For example, the baseline fusion technique can be a DCT-based fusion technique that does not use correlation-based weights. For example, as shown in circle 532 in FIG. 5, the fused image 530 has at least some of the distortions transferredAttorney Docket No.: 56113-0847WO1
[0097] from the source image 510, causing some of the straight line segments in the fused image 530 to become curved or even appear distorted when viewed by a user.
[0098] In contrast, a second fused image 540 that can be generated by the system 100 based on the source image 510 and the reference image 520 has significantly fewer visual artifacts, e.g., fewer distortions (e.g., the straight line segments in the horizontal direction are no longer curved). The system 100 effectively prevents the visual artifacts, e.g., aliasing artifacts, warping artifacts, or other visible distortions, that appear in the source image 510 from being transferred to the second fused image 540. Put another way, the system 100 reduces the influence of aliasing artifacts in the second fused image 540 contributed by the source image 510.
[0099] By adopting the correlation-based adaptive fusion technique, the system 100 can assign lower weight values to first image patches obtained from the source image 510, relative to weight values assigned to second image patches obtained from the reference image 520 when generating the weighted representations of the image patches in the transformed domain. Thus, because the first image patches are assigned lower weight values in the weighted representations, the weighted representations will also contain fewer visual artifacts from the first image patches. When transformed back from the transformed domain to the original domain, the second fused image 540 generated based on the weighted representations will similarly contain fewer visual artifacts from the first image patches.
[0100] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
[0101] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
[0102] Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encodedAttorney Docket No.: 56113-0847WO1
[0103] on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
[0104] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0105] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
[0106] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.Attorney Docket No.: 56113-0847WO1
[0107] Similarly, in this specification the term “engine” is used broadly to refer to a softwarebased system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
[0108] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
[0109] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
[0110] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and
[0111] CD ROM and DVD-ROM disks.
[0112] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRTAttorney Docket No.: 56113-0847WO1
[0113] (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
[0114] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
[0115] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.
[0116] In addition to the embodiments described above, the following embodiments are also innovative:
[0117] Embodiment 1 is a method comprising:Attorney Docket No.: 56113-0847WO1
[0118] receiving a pair of image patches captured by image sensors having different optical characteristics, wherein the pair of image patches include a first image patch from a source image and a second image patch from a reference image;
[0119] applying a first frequency transformation to the first image patch to generate a first representation of the first image patch and applying a second frequency transformation to the second image patch to generate a second representation of the second image patch;
[0120] determining a measure of correlation between the first representation of the first image patch and the second representation of the second image patch;
[0121] determining, based on the measure of correlation between the first representation of the first image patch and the second representation of the second image patch, respective weight values for generating a fused image patch from the first image patch and the second image patch; and
[0122] generating a fused image patch according to the respective weight values determined for the first image patch and the second image patch.
[0123] Embodiment 2 is the method of embodiment 1, wherein determining the weight values comprises assigning a lower weight value to the first image patch from the source image when the measure of correlation is low.
[0124] Embodiment 3 is the method of embodiment 2, wherein assigning the lower weight value to the first image patch reduces the influence of artifacts in the fused image patch contributed by the source image.
[0125] Embodiment 4 is the method of any one of embodiments 1-3, wherein determining the weight values comprises assigning a higher weight value to the first image patch from the source image when the measure of correlation is high.
[0126] Embodiment 5 is the method of any one of embodiments 1-4, wherein applying the first frequency transformation and the second frequency transformation comprises applying a discrete cosine transform.
[0127] Embodiment 6 is the method of any one of embodiments 1-5, wherein the source image is captured by a first image sensor having a wider field of view than a second image sensor that captured the reference image.
[0128] Embodiment 7 is the method of any one of embodiments 1-6, wherein generating the fused image patch according to the respective weight values determined for the first image patchAttorney Docket No.: 56113-0847WO1
[0129] and the second image patch comprises:
[0130] generating a weighted representation based on the respective weight values determined for the first image patch and the second image patch; and
[0131] applying an inverse frequency transformation to the weighted representation.
[0132] Embodiment 8 is the method of embodiment 7, wherein applying the inverse frequency transformation comprises applying an inverse discrete cosine transformation.
[0133] Embodiment 9 is the method of any one of embodiments 1-8, wherein each of the pair of image patches comprise a block of pixels, wherein the block has a width and a height of an equivalent number of pixels.
[0134] Embodiment 10 is the method of any one of embodiments 1-9, wherein computing the measure of correlation between the first representation of the first image patch and the second representation of the second image patch comprises computing a correlation value between AC coefficients of the first representation of the first image patch and AC coefficients of the second representation of the second image patch.
[0135] Embodiment 11 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 to 10.
[0136] Embodiment 12 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of embodiments 1 to 10.
[0137] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment.
[0138] Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.Attorney Docket No.: 56113-0847WO1
[0139] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0140] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.
[0141] What is claimed is:
Claims
Attorney Docket No.: 56113-0847WO1CLAIMS1. A method comprising:receiving a pair of image patches captured by image sensors having different optical characteristics, wherein the pair of image patches include a first image patch from a source image and a second image patch from a reference image;applying a first frequency transformation to the first image patch to generate a first representation of the first image patch and applying a second frequency transformation to the second image patch to generate a second representation of the second image patch;determining a measure of correlation between the first representation of the first image patch and the second representation of the second image patch;determining, based on the measure of correlation between the first representation of the first image patch and the second representation of the second image patch, respective weight values for generating a fused image patch from the first image patch and the second image patch; andgenerating a fused image patch according to the respective weight values determined for the first image patch and the second image patch.
2. The method of claim 1, wherein determining the weight values comprises assigning a lower weight value to the first image patch from the source image when the measure of correlation is low.
3. The method of claim 2, wherein assigning the lower weight value to the first image patch reduces the influence of artifacts in the fused image patch contributed by the source image.
4. The method of any one of claims 1-3, wherein determining the weight values comprises assigning a higher weight value to the first image patch from the source image when the measure of correlation is high.
5. The method of any one of claims 1-4, wherein applying the first frequency transformation and the second frequency transformation comprises applying a discrete cosine transform.
6. The method of any one of claims 1-5, wherein the source image is captured by a first image sensor having a wider field of view than a second image sensor that captured the reference image.Attorney Docket No.: 56113-0847WO17. The method of any one of claims 1-6, wherein generating the fused image patch according to the respective weight values determined for the first image patch and the second image patch comprises:generating a weighted representation based on the respective weight values determined for the first image patch and the second image patch; andapplying an inverse frequency transformation to the weighted representation.
8. The method of claim 7, wherein applying the inverse frequency transformation comprises applying an inverse discrete cosine transformation.
9. The method of any one of claims 1-8, wherein each of the pair of image patches comprise a block of pixels, wherein the block has a width and a height of an equivalent number of pixels.
10. The method of any one of claims 1-9, wherein computing the measure of correlation between the first representation of the first image patch and the second representation of the second image patch comprises computing a correlation value between AC coefficients of the first representation of the first image patch and AC coefficients of the second representation of the second image patch.
11. A system comprising one or more computers, and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of claims 1-10.
12. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-10.