Image, video etc. visual content enhancement method

Abstract

The invention relates to a method for enhancing visual content such as images, videos, etc. It takes as input versions of the same visual content exposed to different light levels. Using sparse representation via orthogonal matching pursuit, it generates weight maps that assign greater emphasis to pixel clusters containing detailed and information-rich content. By combining the inputs and these weight maps, it outputs a version of the same visual content that appears more visually appealing to the human eye.

Description

[0001] DESCRIPTION

[0002] IMAGE, VIDEO ETC. VISUAL CONTENT ENHANCEMENT METHOD

[0003] Technical Area:

[0004] The invention relates to a method for improving visual contents such as images and / or videos, which offers a solution to enhance the quality of visual content while addressing the high costs of high dynamic range (HDR) imaging devices. This method provides a lower-cost solution capable of competing with the content quality provided by such devices.

[0005] State of the Art:

[0006] Solutions offered to achieve high-quality digital visual content can generally be examined under two main categories. (It should be noted here that the “quality” metric mentioned in the previous sentence is based on proximity to the human visual system.)

[0007] The first category includes hardware solutions. These solutions aim to improve imaging devices hardware-wise to achieve a higher dynamic range. The disadvantage of these solutions is that they are not cost-effective enough to be included in the consumer electronics industry. Therefore, researchers have started to develop more cost-effective software solutions that can be marketed to the public and have developed specific algorithms that make visuals on devices with a low dynamic range appear as though they have a high dynamic range.

[0008] The MEF technique is an algorithm aimed at producing higher-quality visuals by using different versions of the same visual exposed to varying light levels, selecting the most visually appealing parts of each input to the human eye, and combining these selected parts. To select the most visually appealing pixel clusters of the input visuals, mapping methods dependent on different parameters have been proposed. In these mapping methods, various parameters of the visuals (such as contrast, saturation, etc.) are analyzed, and pixel-based evaluations are made based on these parameters. These values are recorded in a matrix of the same dimensions as the visual. The locations of the recorded values correspond to the locations of the pixels with those values. Thus, the information about which pixels in the visual should be given more weight and which should be given less weight compared to others is recorded. Multiple mappings can be performed for a single lighting level.

[0009] In the state of the art, imaging devices with high dynamic range are significantly costly. Numerous solutions have been proposed in the state of the art to address this high cost.

[0010] Among these solutions, Tom Mertens et al. [1] managed to achieve a milestone chronologically with their proposed solution. The standout aspect of the proposed algorithm is the weight maps and the parameters used to derive these weight maps. The idea of creating weight maps based on parameters has also guided other studies in this field. This structure takes different versions of the same visual exposed to various lighting levels as input. Subsequently, weight maps are created to select the parts of the visual that appear aesthetically pleasing to the human eye.

[0011] The weight maps are designed based on three parameters. The first parameter is contrast. The purpose of the contrast parameter is to identify the edges and corners in the visual and assign greater weight values to these pixel clusters. This ensures that when the inputs are combined using the weight maps, the edges and corners in the visual will appear more prominent. Contrast weight maps are created by passing the grayscale versions of the inputs through a Laplacian filter.

[0012] The second parameter is saturation. The purpose of this parameter is to retain the pixels with more vibrant colors in the inputs. Saturation weight maps are created by calculating the standard deviation among the color channels corresponding to each pixel in the inputs. As a result, pixel clusters with more vibrant colors are assigned higher weight values.

[0013] The final parameter is well-exposedness. This parameter creates a weight map by assigning higher weight values to pixels with color channel values closer to 0.5 (on a scale from 0 to 1) for each input. The weights are calculated using a normal distribution curve. As a result of this process, three weight maps are generated for each input. The three weight parameters for each input are subjected to pixel-based multiplication, resulting in a weight map for each input. In the final stage of the weight map extraction process, the weights corresponding to the same pixel are normalized so that their total equals 1, thus creating a proportional relationship among the weights. At the same time, the process specifies how much weight should be taken from each input for every pixel.

[0014] As seen in Figure 1, in the algorithm developed by Tom Mertens et al., the weight maps are transformed into a Gaussian pyramid after being created. The inputs are also transformed into Laplacian pyramids, and each Laplacian pyramid is multiplied layer by layer with the corresponding Gaussian pyramid created from its weight map. Subsequently, the resulting pyramids are combined layer by layer. Starting from the bottom layer of the resulting pyramid, size adjustments are made, and the layers are added together up to the topmost layer. This process results in the desired visual output as the final output.

[0015] Another study in the state of the art is the algorithm developed by Ulucan et al. [2]. As seen in Figure 2, this algorithm uses parameters such as principal component analysis (PCA), adaptive-exposedness sensitive to light levels, and saliency to create weight maps. After the weight maps are extracted, Gaussian and Laplacian pyramids are again used for the merging process. In their published article, Ulucan et al. [2] compared their visual outputs with those produced by other algorithms in this field using the structural similarity index measure (SSIM) method in terms of quality and aesthetic appeal. According to this comparison, their algorithm achieved SSIM values close to the highest values produced by algorithms up to the time of publication. For one visual, they matched the previous highest value, while for two visuals, they achieved a new record value.

[0016] Another study in the state of the art is the representation-based algorithm developed by Wang et al. [3]. As seen in Figure 3, this algorithm creates a structure that takes as input different versions of the same visual exposed to various lighting levels and produces an output visual based on the sparse representations of these inputs. To sparsely represent the inputs, a dictionary matrix is required. These matrices can be pre-generated using specific functions. However, in their study, Wang et al. [3] trained their dictionary using a predefined indoor / outdoor image dataset and the K- SVD algorithm to better represent the images. Once the trained dictionary was used for sparse representations of the inputs, the sparse representation maps followed a fusion method based on the usage frequency of the atoms (columns) in the dictionary matrix. After the fusion process was completed, the output image was produced through a matrix multiplication with the trained dictionary. The fact that their study is purely based on sparse representation is the most prominent feature of their work.

[0017] REFERENCES

[0018] [1] Mertens, T., Kautz, J., & Van Reeth, F. (2009, March). Exposure fusion: A simple and practical alternative to high dynamic range photography. In Computer graphics forum (Vol. 28, No. 1, pp. 161-171). Oxford, UK: Blackwell Publishing Ltd.

[0019] [2] Ulucan, O., Ulucan, D., & Turkan, M. (2023). Ghosting-free multi-exposure image fusion for static and dynamic scenes. Signal Processing, 202, 108774.

[0020] [3] Wang, J., Liu, H., & He, N. (2014). Exposure fusion based on sparse representation using approximate K-SVD. Neurocomputing, 135, 145-154.

[0021] Purpose of the Invention:

[0022] The invention subject to registration aims to provide a cost-effective solution that can rival the content quality offered by high dynamic range (HDR) imaging devices, which are one of the solutions in the field of visual content quality enhancement, such as for images and videos, but are associated with high costs.

[0023] Additionally, the invention aims to produce a visually appealing version of the same visual content for the human eye by taking as input different versions of the same visual content exposed to various lighting levels, generating weight maps through orthogonal matching pursuit (OMP) using sparse representations, and assigning more weight to pixel clusters with higher detail and information content. The method then combines these inputs and their weight maps using a multi -expo sure fusion (MEF) technique.

[0024] Explanation of the Figures:

[0025] Figures in the state of the art are shown below:

[0026] Figure 1. Schematic of the algorithm developed by Mertens et al.

[0027] Figure 2. Schematic of the algorithm developed by Ulucan et al.

[0028] Figure 3. Schematic of the algorithm developed by Wang et al.

[0029] Figures related to the developed method for enhancing visual content such as images and videos are shown below:

[0030] Figure 4. Schematic of calculating multi-layer orthogonal matching pursuit weight maps

[0031] Figure 5. Schematic of calculating color deviation weight maps

[0032] Figure 6. Schematic of calculating detail weight maps

[0033] Figure 7. Schematic of calculating final weight maps

[0034] Figure 8. Schematic of fusing input images and weight maps

[0035] Explanation of the Invention:

[0036] The invention takes as input different versions of the same visual content exposed to various lighting levels. In each of these inputs, different pixel clusters contain varying degrees of detail and information. The invention generates weight maps through sparse representations using orthogonal matching pursuit (OMP), assigning higher weights to pixel clusters with greater detail and information content in each input. These weight maps and the inputs are then combined using a multi-exposure fusion (MEF) method to produce a visually enhanced version of the same image.

[0037] In addition to the three weight map extraction techniques, the invention also provides an optional wavelet-based preprocessing technique that can be used as needed.

[0038] The first mapping technique forms the foundation of the algorithm. The main idea behind this mapping technique is to capture details at varying levels. To achieve this, the images are processed using the orthogonal matching pursuit algorithm in such a way that they are approximated with sparse representation coefficients in varying numbers. By subtracting these from the original image, high-frequency pixel clusters are identified. The use of varying numbers of sparse representation coefficients allows for the capture of detail levels at different scales. While the orthogonal matching pursuit algorithm examines the detailed portions of the images down to the finest level, the other two mapping techniques aim to highlight the visually appealing parts of the images. For this purpose, a standard deviation-based sorting algorithm is used for both color images and their grayscale versions. Optional wavelet-based preprocessing algorithm focuses on correcting burnt and dead pixels in the images by leveraging information from other exposure levels, thus improving the input images.

[0039] The invention introduces three mapping techniques. The first technique, based on sparse representation and orthogonal matching pursuit, aims to capture and refine details layer by layer without noise. The other mapping techniques aim to broadly capture color changes and details using an algorithm based on standard deviationbased sorting. The fusion process used after mapping is the same as the standard fusion processes commonly used in the literature. Finally, the optional wavelet -based preprocessing technique combines the input images with their well-captured pixel clusters (masks) in the wavelet domain and reconstructs the image from the wavelet domain. This enables learning from other images to correct burnt or dead pixels in an input image.

[0040] In addition to the three novel weight map extraction methods, the invention also provides a preprocessing method. The preprocessing method aims to correct undesired pixel clusters in the inputs (RGB) before they are processed by the mapping algorithms. Afterwards, a weight map is extracted for each processed input. The preprocessing method is wavelet-based. Initially, masks of the inputs are generated. These masks store the locations of non-burnt and non-dead pixels as Is and the remaining pixel locations as Os in a matrix of the same size as the input. The mask of the input to be processed first undergoes a dilation operation, and then all the values in the mask are multiplied by 100 (a coefficient that can be changed). Subsequently, all masks are multiplied by the well exposedness map of their respective input (obtained from a grayscale version of the input adjusted to have a mean of 0.5 using a normal distribution). As a result, a mask is obtained for each input. These masks are normalized between 0 and 1 and adjusted so that their sum equals 1. Then, they are split into approximation coefficients with the maximum number of wavelet layers. After obtaining the absolute values of the approximation coefficients, they are normalized layer-wise so that their sum equals 1 across the masks. These approximation coefficients are combined with the approximation and detail coefficients of their respective inputs. Finally, the wavelet trees generated for each input are summed and converted back into an image. This concludes the preprocessing phase.

[0041] The first weight map is based on orthogonal matching pursuit and sparse representation. Before extracting the weight map using orthogonal matching pursuit and sparse representation, the input image is converted to grayscale, and a result matrix of the same size is initialized with zeros. The weight map extraction process consists of four layers. In the first layer, grayscale inputs are subjected to a loop that continues until all possible 7x7 pixel clusters in the image are selected. In each iteration, the selected pixel cluster is processed to its approximation with single sparse representation coefficient using the orthogonal matching pursuit algorithm and a discrete cosine transform dictionary. This approximation value is added to the result matrix at the corresponding location of the pixels in the cluster. After this process is completed for all possible 7x7 pixel clusters, the average of the approximation values added to the result matrix for pixels with multiple overlaps is calculated and stored back in the result matrix. The resulting matrix is subtracted pixel- wise from the original image to reveal the details. To eliminate negative values that may arise from the approximation and to prevent noise, the absolute value of the resulting matrix is taken, and it is passed through a guided filter. This process is performed for all input images in the first layer, and all outputs are then subjected to normalization. The same process is applied in subsequent layers; however, the sparse representation coefficient is modified to match the number of the layer. After obtaining outputs for different light levels for each layer, a total of 4 different detail maps are obtained for each light level. These detail maps are then subjected to pixelwise multiplication to refine them further. This weight map ensures that details are captured at their finest levels. In addition to the above weight map, two additional weight maps are used to generally preserve the details. To obtain the first of these, a method is followed that records the variability in color changes over the RGB input images. Before this process, zero matrices with the same dimensions as the image and as many as the number of light levels are created. Then, for each light level, starting from the top left of the image, 7x7x3 pixel blocks are extracted, including each color channel. The standard deviation of these pixel blocks is calculated. This standard deviation value is multiplied by a 7x7 matrix of ones with the same dimensions as the pixel block from which it was obtained. Then, the resulting matrix is compared in magnitude to the pixel block in the corresponding location of the zero matrix, and the largest values are recorded in the same locations. Finally, the resulting weight maps are normalized between 0 and 1. As a result of this process, more weight is given to the pixels within pixel blocks that have the most variability in color changes. The same process is applied in two dimensions on the grayscale versions of the images to identify edges and corners.

[0042] Thus, three different weight maps are obtained for each light level. These weight maps are multiplied pixel-wise (element-wise), and as a result, weight maps with the same amount as different lighting levels are obtained. These weight maps are normalized so that their pixel-wise sum equals 1. These obtained weight maps and pre-processed images are combined using Laplacian and Gaussian pyramids.

[0043] The algorithms that constitute the patented method for improving visual content, such as images and videos, are as follows:

[0044] 1. Generation of the Auxiliary Elements Algorithm

[0045] 2. Optional Pre-Processing Algorithm

[0046] 3. Weight Map Calculation Algorithm

[0047] 4. Fusion Algorithm for Input Images and Weight Maps ) Generation of the auxiliary elements algorithm: This forms the first part of the invention. It is an algorithm aimed at collecting (creating) the elements to be used throughout the patented method for improving visual content such as images and videos. First, different light-level versions of the same image are loaded into the software environment, and the grayscale versions of these inputs are also saved for use in the weight map calculation phase. Next, the dictionary to be used for sparse representation in the orthogonal matching algorithm is loaded. This dictionary is a discrete cosine transform dictionary obtained using pre-defined functions, where the magnitude of each column vector has been normalized to equal 1. The library is presented as a matrix. The number of rows in the dictionary indicates the length of the vector on which orthogonal matching will be performed. Once this information is obtained, the next step is initiated. ) Optional pre-processing algorithm: This algorithm aims to correct unwanted pixel clusters in the input images. For this purpose, a mask is first created for each input image. These masks are the same size as the images. The masks contain a value of 1 at the locations of non-dead (brightness level higher than 0.14) and non-bumt (brightness level lower than 0.90) pixels, while other locations have a value of 0. The mask of the input to be pre-processed is subjected to a dilation operation, and then all the values within it are multiplied by 100 (a coefficient determined for this purpose, which can be modified). Subsequently, all masks are multiplied by the well-exposedness weight map of the corresponding input. This well-exposedness map is derived using a normal distribution with a mean of 0.5 based on the grayscale versions of the inputs. The resulting masks and inputs are then passed through a wavelet fusion algorithm. This algorithm first normalizes the masks between 0 and 1 and adjusts their pixel- wise sum to equal 1. Then, the masks are decomposed into approximation coefficients with the maximum number of layers. This decomposition is referred to as the wavelet tree (approximation trees). The absolute value of the approximation coefficients derived from the masks is taken, and to prevent abrupt coefficient changes, they are passed through a Gaussian filter both horizontally and vertically. The pixel-wise sum of the wavelet trees derived from the masks is adjusted to equal to 1. The input images are decomposed into approximation and detail coefficients to the maximum number of layers. The approximation and detail coefficients obtained from the input images (for each of the red, green, and blue channels) are combined by element-wise multiplication with the approximation coefficients obtained from the masks (for this purpose, the approximation coefficients are replicated). The resulting wavelet trees are summed pixel-wise, and reconstruction is performed to create an image. After this process is performed separately for each image, the processed inputs are prepared.

[0048] 3) Calculation of the weight maps algorithm: The third part of the invention consists of weight map calculations. The algorithm first takes versions of the same image exposed to different light levels as input. These inputs are in RGB format. An RGB image contains three channels (red, green, and blue) for each pixel. These three (dimensional) channels can take values between 0 and 255. Since any color can be obtained by the weighted sum of these color channels, each pixel of the RGB image can represent any color. The algorithm, which includes a total of three different weight map extraction techniques, uses both these RGB inputs and their grayscale versions to generate the weight maps. Unlike RGB images, grayscale images contain a single (gray) channel for each pixel, and this (two- dimensional) channel has values ranging from 0 to 255. The gray channel can include shades of gray between black and white.

[0049] To calculate the maps, all input images are first adjusted so that their pixel values are between 0 and 1 (by dividing by 255). The first map is created using a orthogonal matching pursuit method based on sparse representation. Grayscale inputs are used to calculate the weight maps. To extract the first weight map, these grayscale images are given as input to a 4-layer orthogonal matching pursuit algorithm. When a two-dimensional grayscale image is given as input to the orthogonal matching pursuit algorithm, the image is first padded by adding new pixels around its edges. These added pixels are symmetric copies of the pixels at the edges and corners of the input image. After the padding process is completed, a zero matrix is created. This matrix, called the approximation matrix, has the same dimensions as the padded image. Then, starting from the top-left corner of the padded image, the algorithm enters a loop where it selects every distinct 49-pixel (7x7) block using a sliding window technique until the entire image is covered. In each step of the loop, the selected pixel block is approximated with single sparse representation coefficient using a discrete cosine transform dictionary (preloaded into the software environment) and the orthogonal matching pursuit algorithm. After this approximation is achieved, the results are summed at the corresponding locations in the approximation matrix. Once this process is completed for the entire image, each original pixel will have undergone approximation 49 times. To calculate the average of these approximations, each pixel in the approximation matrix is divided by 49, thereby normalizing it. Then, the pixels added during padding are removed from the approximation matrix, leaving only the original pixels. The absolute value of the resulting detail matrix is then taken and subtracted pixel- wise from the associated input image, and resulting matrix is refined using self-guided filtering. This process is repeated for each light level. After the detail maps extracted for each light level of an image are normalized between 0 and 1, the first layer of sparse representation detail maps is created. The above process is repeated for a total of 4 layers, with the number of sparse representation coefficients matching the layer number (e.g., 2 coefficients for the second layer). This ensures that details are preserved for each level of granularity. To further refine the layer maps, the 4 different layer maps for a signle lightning level image are multiplied element- wise. In this way, the first mapping method is completed. By multiplying maps of different detail levels element-wise, the mapping method achieves its most refined form.

[0050] In the second and third mapping techniques, the goal is to extract maps that generally identify visually appealing and interesting parts of the images, serving as a template for the first mapping technique. The second and third weight maps support the first weight map. The purpose of the second weight map is to transfer vivid and visually interesting colors to the output image. The inputs for this weight map extraction process are the RGB (scaled between 0 and 1) versions of the same image exposed to different light levels. Before starting the process, a zero matrix with the same dimensions as the first two dimensions of the input images is created. The inputs are then padded in the same way as in the first mapping technique. Starting from the top-left 7x7x3 (RGB) pixel block, a loop continues, moving one pixel at a time from left to right and top to bottom, until all pixels are selected. In each iteration, the standard deviation of the selected pixel block is calculated. This standard deviation value is compared to the values at the same locations in the zero matrix. If the calculated value is higher, it is written to the zero matrix at the same locations iteratively. After the standard deviation-based color variation algorithm is applied to each RGB input, the padded regions are removed, and the resulting maps are normalized between 0 and 1. This ensures that more weight is assigned to pixels in pixel blocks that experience the most color variation. The same process is applied in two dimensions (7x7 regions) for the grayscale versions of these RGB inputs, allowing edges and corners to be identified.

[0051] After obtaining three weight maps for each light level input, these weight maps are multiplied element- wise. As a result, a single weight map is obtained for each input. These weight maps are normalized so that their pixel-wise sum equals 1. This determines the proportion of information (weight value) to be taken from each input.

[0052] These processes are illustrated in Figures 4, 5, 6, and 7. ) Fusion algorithm for input images and weight maps: The final step of the invention involves combining the input images and the weight maps. The normalized weight maps are converted into Gaussian pyramids, and the input RGB images are converted into Laplacian pyramids. The Gaussian pyramid prevents sudden changes in weights, while the Laplacian pyramid prevents abrupt color and pixel value changes caused by low dynamic range. These pyramids help eliminate unwanted seams (transition artifacts) and edge halos (glows) in the output image.

[0053] The Laplacian pyramid extracted from each RGB image is multiplied layer by layer, pixel by pixel, with the Gaussian pyramid derived from the corresponding weight map. Once this process is completed, the resulting three pyramids are added together layer by layer, pixel by pixel. Then, the resulting pyramid is reconstructed from the bottom layer to the top by adjusting dimensions and adding layers pixel-wise, producing the final output image. These processes are illustrated in Figure 8.

[0054] Industrial

[0055] The invention is used for enhancing visual content such as images, videos, and similar visual media.

Claims

CLAIMS1. The invention is a method for improving visual content such as images, videos, etc., which takes as input different versions of the same visual content exposed to varying light levels. Using sparse representation via orthogonal matching pursuit, it extracts weight maps that give more emphasis to pixel clusters containing detailed and abundant information from the inputs. By combining the inputs and these weight maps through a method of merging images exposed to different light levels, it produces an output version of the same visual content that appears more visually appealing to the human eye. Its characteristics include:Saving the different versions of the same visual content with varying lighting levels and their grayscale version to be used during the calculation of weight mapsLoading a discrete cosine transform dictionary into the algorithm for sparse representation used in the orthogonal matching pursuit algorithm. This dictionary is a discrete cosine transform dictionary created using predefined functions, where the magnitude of each column vector is normalized to 1. The library is a matrix, with the number of rows representing the length of the vector on which orthogonal matching pursuit is performed. which contain steps to the generation of the auxiliary elemets to be used.

2. The invention is a method for enhancing visual content such as images and videos by taking versions of the same visual content exposed to different light levels as input, extracting weight maps using sparse representation through orthogonal matching pursuit, and assigning greater weight to pixel clusters in the inputs that contain more details and extensive information. It then combines the inputs and these weight maps to output a version of the same visual content that is more visually appealing to the human eye. Its features include an optional preprocessing algorithm to correct undesired pixel clusters in the input images, consisting of the following steps:Creating a mask for each input image, where the masks are the same size as the images. These masks assign a value of 1 to pixels that are not dead (brightness level above 0.14) and not overexposed (brightness level below 0.90), and 0 to other pixels,Subjecting the mask of the input to be preprocessed to a dilation operation, followed by multiplying all values within the mask by 100 (a coefficient that can be adjusted),Multiplying the resulting masks by the well-exposedness weight map of their corresponding input. The well-exposedness weight map is derived using a normal distribution with a mean of 0.5 applied to the grayscale versions of the inputs.Providing the resulting masks and inputs to the wavelet fusion algorithm, where the masks are first normalized to values between 0 and 1, and their pixel- wise sums are adjusted to equal 1. Then, the masks are decomposed into approximation coefficients at the maximum number of layers,Taking the absolute value of the approximation trees extracted from the masks and passing them through a Gaussian filter both vertically and horizontally to prevent abrupt coefficient changes. The pixel-wise sums of the wavelet trees derived from the masks are then adjusted to equal to 1, Decomposing the inputs into wavelet trees by breaking them into approximation and detail coefficients at the maximum number of layers, Combining the approximation and detail coefficients (for red, green, and blue channels) of the input images by performing element-wise multiplication with the approximation coefficients derived from the masks. (For this purpose, the approximation coefficients are duplicated.),Finally, summing the resulting wavelet trees pixel-wise and reconstructing them to form a new image.These steps are repeated for each visual input separately, ensuring the preparation of processed inputs using the optional preprocessing algorithm.

3. The invention is a method for enhancing visual content such as images and videos by taking versions of the same visual content exposed to different light levels as input, extracting weight maps using sparse representation through orthogonal matching pursuit, and assigning greater weight to pixel clusters in theinputs that contain more details and extensive information. It then combines the inputs and these weight maps to output a version of the same visual content that is more visually appealing to the human eye. Its feature includes an algorithm for calculating weight maps, consisting of the following steps:Taking as input versions of the same visual exposed to different light levels. These inputs are RGB images where each pixel has three channels — red, green, and blue — each ranging between 0 and 255. Any color can be achieved by a weighted combination of these channels, meaning each pixel in an RGB image can represent any color.Adjusting the pixel values of all input images to range between 0 and 1 by dividing them by 255,Generating the first map using orthogonal matching pursuit based on sparse representation, where grayscale inputs are used for calculating weight maps. The grayscale images are fed into a four-layer orthogonal matching pursuit algorithm to generate the first weight mapPadding the grayscale image symmetrically by copying edge and corner pixels to extend the image,Creating a zero matrix, called the approximation matrix, which is the same size as the padded imageIterating through the padded image using a sliding window technique, selecting each 7x7 pixel cluster,Applying discrete cosine transformation and orthogonal matching pursuit to approximate selected pixel cluster with single sparse representation coefficient. These approximations are accumulated at the corresponding locations in the approximation matrix, after completing the iterations for all input images, dividing each approximated pixel by 49 (the total number of iterations per pixel) to normalize the values,Removing the padded pixels from the approximation matrix to revert to the original image dimensions, subtracting the resulting approximation matrix from the associated input image,Taking the absolute value of the resulting detail matrix, followed by refining it using a self-guided filter,Repeating the above steps for all light levels, normalizing the detail maps for each light level together to range between 0 and 1, thereby generating the first layer of sparse representation detail maps,Repeating the process for a total of four layers, with each layer number matching the number of sparse representation coefficients (e.g., the second layer uses two coefficients). This ensures the preservation of details across different levels of granularity,Refining the layer maps further by performing pixel-wise multiplication of the four detail maps for each input, completing the first mapping method, Refining down to the finest level by mutliplying detail maps pixel-wise for each lighting level,Using this mapping technique as a draft to the first one ensure that vivid and appealing colors are carried over to the output image, scaling the inputs to RGB values between 0 and 1, and preparing a zero matrix with the same dimensions as the first two dimensions of the inputs,Padding the inputs similarly to the first mapping technique and iterating over every 7x7x3 (RGB) pixel cluster,Calculating the standard deviation for each cluster, and comparing it with the values with the same pixel locations in the zero matrix. Higher values are written back to the zero matrix iterativelyAfter computing the standard deviation-based color variation algorithm for each RGB input, removing the padded regions and normalizing the resulting maps to range between 0 and 1. This ensures greater weight is assigned to pixel clusters common across inputs that exhibit the most color variation, Performing the same operations in two dimensions for grayscale versions of the RGB inputs to identify edges and comers,After obtaining three weight maps for each light level input, performing pixel-wise multiplication of these maps to generate a single weight map for each input.Normalizing the weight maps such that their pixel-wise sums equal 1, thereby determining the proportional weight of information (weight value) to be extracted from each input4. The invention is a method for enhancing visual content such as images and videos by taking versions of the same visual content exposed to different light levels as input, extracting weight maps using sparse representation through orthogonal matching pursuit, and assigning greater weight to pixel clusters in the inputs that contain more details and extensive information. It then combines the inputs and these weight maps to output a version of the same visual content that is more visually appealing to the human eye. Its features include:Converting the normalized weight maps into Gaussian pyramids and the input RGB images into Laplace pyramids. The Gaussian pyramids prevent abrupt weight changes, while the Laplace pyramids mitigate sudden color and pixel value changes caused by low dynamic range. These pyramids help eliminate undesired seams (transition marks) and edge glows (halos) in the output image.Multiplying the Laplace pyramid of each RGB input pixel-wise with the corresponding Gaussian pyramid of the weight map layer by layer,Summing the resulting pyramids layer by layer,Adjusting the dimensions of the resulting pyramid from the lowest to the highest layer and adding all the layers from bottom to top pixel-wise together to reconstruct the output image5. The invention is a method for enhancing visual content such as images, videos, etc., which takes as input versions of the same visual content exposed to different light levels. Using sparse representation via orthogonal matching pursuit, it generates weight maps that give more emphasis to pixel clusters with detailed and information-rich content. By combining the inputs and these weight maps, it outputs a version of the same visual content that appears more visually appealing to the human eye. Its features include:The auxiliary element creation algorithm mentioned in Claim 1,The optional pre-processing algorithm mentioned in Claim 2,The weight map calculation algorithm mentioned in Claim 3,The algorithm for merging input images and weight maps mentioned in Claim 4.

Images