Image processing method and apparatus, device, and storage medium

Abstract

The present disclosure provides an image processing method and apparatus, a device, and a storage medium. The method comprises: determining covisibility relationships between images on the basis of point cloud coordinate ranges corresponding to the images in an image sequence, and constructing an undirected graph; constructing a loss function of the undirected graph by using the images as nodes and the covisibility relationships between the images as edges, a function value corresponding to the loss function representing a total color difference of the undirected graph after color values of the nodes are adjusted; performing iterative solving on the loss function to determine a gain coefficient corresponding to each node when the function value satisfies a preset constraint condition; and using the gain coefficient to optimize the color of an image corresponding to the node. In this way, color values of images having covisibility relationships are similar, so that the problems of uneven coloring and poor preview effect of point clouds or three-dimensional models are solved, thereby improving the visual coherence and authenticity of color values of point clouds or three-dimensional models.

Description

Image processing methods, apparatus, devices and storage media

[0001] This disclosure claims priority to Chinese Patent Application No. CN202510104534.7, filed on January 22, 2025, entitled "Image Processing Method, Apparatus, Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to three-dimensional reconstruction technology, image processing technology, and in particular an image processing method, apparatus, device, and storage medium. Background Technology

[0003] Simultaneous Localization and Mapping (SLAM) devices typically include a lidar unit and a camera. The lidar unit performs laser scanning of the surrounding environment and generates a corresponding point cloud to obtain three-dimensional spatial information; the camera unit acquires images of the surrounding environment so that the color values ​​of each point in the point cloud can be used to color the point cloud generated by the lidar or the 3D model obtained from the 3D reconstruction.

[0004] During image acquisition for 3D reconstruction, the camera moves with the SLAM device. As the camera moves, changes in the environment alter the white balance and exposure during image acquisition, leading to inconsistent color and brightness across multiple views in the acquired image sequence. After 3D modeling is completed, when coloring the point cloud or 3D model using the image sequence, overlapping content across multiple frames can occur because the same area of ​​the 3D model may be captured. If the color and brightness of these multiple frames are inconsistent, it can result in uneven coloring of the point cloud or 3D model and poor preview quality. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this disclosure provide an image processing method, apparatus, device, and storage medium.

[0006] One aspect of this disclosure provides an image processing method, including:

[0007] Based on the point cloud coordinate range corresponding to each image in the world coordinate system in the image sequence, the co-view relationship between the images in the image sequence is determined, wherein the point cloud coordinate ranges corresponding to the images with co-view relationship have an intersection;

[0008] An undirected graph corresponding to the image sequence is constructed using the images as nodes and the co-visibility relationships between the images as edges.

[0009] Construct a loss function for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to the edges in the undirected graph. The color difference corresponding to the edge is the difference in color values ​​between the two nodes connected by the edge.

[0010] The loss function is iteratively solved to determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is minimized and the function value is less than a preset value.

[0011] The color of the image corresponding to the node is processed using the gain coefficient.

[0012] Optionally, the loss function for constructing the undirected graph includes:

[0013] For each edge connecting two nodes, the common viewing region of each node is determined based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds in the image corresponding to the node.

[0014] The average color value of the pixels within the shared viewing area is determined as the color value of the node;

[0015] The loss function of the undirected graph is constructed based on the color value of each node and the gain coefficient, wherein the gain coefficient is the variable to be solved in the loss function.

[0016] Optionally, before determining the average color value of the pixels within the shared viewing area as the color value of the node, the method includes:

[0017] The matching block for each pixel is determined based on a preset matching block size. The matching block is an image region centered on the pixel and with a size equal to the preset matching block size.

[0018] The average color value of the pixels within the matching block is determined as the color value of the center pixel of the matching block.

[0019] Optionally, determining the average color value of the pixels within the shared viewing area as the color value of the node includes:

[0020] The first pixel in the common viewing area is determined by the interquartile range method. The first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of the common viewing pixel pair. The common viewing pixel pair is a pair of pixels in the corresponding image that share the same point in the intersection of the point clouds.

[0021] Object recognition is performed on the image, and in response to the existence of a target object belonging to a preset object type, the second pixel point corresponding to the target object is determined;

[0022] Remove the first pixel and the second pixel within the shared viewing area to obtain the target pixel within the shared viewing area;

[0023] The average color value of the target pixel is determined as the color value of the node.

[0024] Optionally, the step of constructing the loss function of the undirected graph based on the color value of each node and the gain coefficient includes:

[0025] The loss weight of the edge is determined based on the number of pixels in the shared viewing area corresponding to the edge, and the loss weight is proportional to the number of pixels.

[0026] The loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient.

[0027] Optionally, the function value of the loss function is specifically the sum of the total color difference and the regularization term. The total color difference is specifically the weighted sum of the squares of the color differences corresponding to the edges in the undirected graph. The color difference is specifically the difference in color values ​​between the two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regular distances. The sum of the regular distances is the sum of the regular distances corresponding to all nodes in the undirected graph. The regular distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

[0028] Optionally, the iterative solution of the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraints includes:

[0029] The gradient descent method is used to iteratively solve the loss function based on the preset constraints to determine the gain coefficient of each node. The preset constraints also include at least one of the following: the gain coefficient is minimized and the gain coefficient is not all zero.

[0030] Another aspect of this disclosure provides an image processing apparatus, comprising:

[0031] The first determining module is used to determine the co-view relationship between images in the image sequence based on the point cloud coordinate range corresponding to each image in the world coordinate system, wherein the point cloud coordinate ranges corresponding to images with co-view relationship have an intersection.

[0032] The construction module is used to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-view relationships between the images as edges.

[0033] A generation module is used to construct a loss function for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to the edges in the undirected graph. The color difference corresponding to the edge is the difference in color values ​​between the two nodes connected by the edge.

[0034] The second determining module is used to iteratively solve the loss function and determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the function value being the minimum and the function value being less than a preset value.

[0035] The processing module is used to process the color of the image corresponding to the node using the gain coefficient.

[0036] In another aspect of this disclosure, an electronic device is provided, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the method described in any embodiment of the present disclosure.

[0039] In another aspect, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described in any embodiment of the present disclosure.

[0040] In another aspect of the present disclosure, a computer program is provided, including computer program instructions that, when executed by a processor, implement the methods described in any embodiment of the present disclosure.

[0041] Based on the embodiments of this disclosure, the co-view relationship between images is determined based on the point cloud coordinate range corresponding to the images in the image sequence. An undirected graph corresponding to the image sequence is constructed with images as nodes and the co-view relationship between images as edges. A corresponding loss function is constructed based on the connected structure of the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color value of the nodes. The color values ​​of each node are constrained to tend to be consistent through preset constraints. By iteratively solving the loss function, the gain coefficient corresponding to each node when the function value satisfies the preset constraints is determined. The color value of the image is optimized using the gain coefficient. When environmental changes cause inconsistencies in the color and brightness of overlapping content in multiple images, the color value of the images can be processed to make the color values ​​of images with co-view relationships similar, avoiding large differences in the color values ​​of points in the same area. This solves the problem of uneven coloring and poor preview effect of point clouds or 3D models caused by this, and helps to improve the visual coherence and realism of the color values ​​of point clouds or 3D models.

[0042] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0043] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0044] Figure 1 is a flowchart of an embodiment of the image processing method of this disclosure;

[0045] Figure 2 is a schematic diagram of an undirected graph provided in an embodiment of this disclosure;

[0046] Figure 3 is a flowchart of another embodiment of the image processing method of this disclosure;

[0047] Figure 4 is a flowchart of yet another embodiment of the image processing method of this disclosure;

[0048] Figure 5 is a schematic diagram of the structure of an embodiment of the image processing apparatus of this disclosure;

[0049] Figure 6 is a schematic diagram of another embodiment of the image processing apparatus of this disclosure;

[0050] Figure 7 is a schematic diagram of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0051] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0052] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0053] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0054] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0055] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0056] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0057] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this disclosure or its application or use.

[0058] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0059] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0060] This disclosure can be applied to electronic devices such as SLAM devices, terminal devices, computer systems, and servers, and can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as SLAM devices, terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0061] Electronic devices such as SLAM devices, terminal devices, computer systems, and servers can be described in the general context of computer-executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0062] Figure 1 is a flowchart of an image processing method provided in an exemplary embodiment of this disclosure. The image processing method of this disclosure can be implemented by an electronic device with image processing capabilities. This electronic device may include, but is not limited to, SLAM devices, smartphones, laptops, desktop computers, etc. This disclosure uses a SLAM device as an example for illustration.

[0063] As shown in Figure 1, the image processing method of this embodiment includes the following steps:

[0064] Step 101: Determine the co-view relationship between images in the image sequence based on the point cloud coordinate range corresponding to each image in the world coordinate system.

[0065] Among them, the point cloud coordinate ranges corresponding to images with co-view relationships have an intersection, and this intersection can be called the point cloud intersection.

[0066] In one possible implementation, the image sequence is obtained during the image acquisition process of 3D reconstruction by a camera in a SLAM device capturing images of the target scanning scene (i.e., a scene to be scanned) at a preset frequency. To avoid missing information, the camera can be configured to acquire images at a higher frequency, resulting in overlapping content between adjacent images. For example, the left half of the first image and the right half of the second image may capture the same area of ​​the target scanning scene. Simultaneously, as the SLAM device moves, the LiDAR in the SLAM device performs laser scanning of the target scanning scene and generates corresponding point clouds to obtain 3D spatial information. The coordinate range of the point cloud corresponding to each image is the coordinate range of the point cloud projected onto that image in the world coordinate system. By determining the coordinate range of the point cloud corresponding to each image, it is possible to determine whether there is content overlap between images in the image sequence, i.e., whether there is a shared view relationship.

[0067] The point cloud coordinate range of an image can be calculated using parameters such as the coordinates of each point in the point cloud in the world coordinate system, camera pose, LiDAR-camera extrinsic parameters, and camera intrinsic parameters. Specific calculation methods can be found in relevant technologies and are not detailed here. The camera pose characterizes the camera's position and attitude in the world coordinate system; the LiDAR-camera extrinsic parameters, i.e., the extrinsic parameters of the LiDAR relative to the camera, characterize the relative position and attitude of the LiDAR relative to the camera in the world coordinate system. Based on whether there is an intersection between the point cloud coordinate ranges corresponding to the images, the co-viewing relationship between images in the image sequence is determined. For example, two images whose intersection point cloud size is larger than a preset point cloud size can be identified as a pair of images with a co-viewing relationship.

[0068] In an optional example, step 101 can be executed by the processor calling the corresponding instruction stored in memory, or by the first determining module 501 run by the processor.

[0069] Step 102: Construct an undirected graph corresponding to the image sequence, using each image in the image sequence as a node and the common-view relationship between images as an edge.

[0070] In this undirected graph, each edge connects nodes corresponding to two images that share a common view relationship. Optionally, if there exists an independent node, that is, a node that is not connected to any other node by any edge, then that node can be removed from the undirected graph. The image corresponding to that node does not share a common view with any other image, so there is no need to adjust its color value. Thus, the nodes in the resulting undirected graph are connected to at least one of the other nodes.

[0071] Schematic, Figure 2 shows an undirected graph. As shown in Figure 2, in this undirected graph, the image corresponding to node 1 shares a common view with the images corresponding to nodes 0, 2, 3, and 4; the image corresponding to node 5 shares a common view with the image corresponding to node 4; and the image corresponding to node 6 shares a common view with the image corresponding to node 3.

[0072] In an optional example, step 102 can be executed by the processor calling the corresponding instruction stored in memory, or by the building module 502 run by the processor.

[0073] Step 103: Construct the loss function for the undirected graph.

[0074] The loss function represents the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​can include at least one of chromaticity and luminance values, and can be, for example, chromaticity values, luminance values, or a vector composed of chromaticity and luminance values. The total color difference includes the sum of the color differences corresponding to each edge in the undirected graph, where the color difference for each edge is the difference between the color values ​​of the two nodes connected by that edge.

[0075] In one possible implementation, the gain coefficient of each node can be set as a variable to be solved or optimized, and a loss function for the undirected graph can be constructed. The value of this loss function is used to characterize the total color difference in the undirected graph after adjusting the color value of the corresponding node based on the gain coefficient of each node. Thus, by constraining this function value, the value of the gain coefficient can be continuously adjusted to solve for the ideal gain coefficient corresponding to each node.

[0076] Optionally, color values ​​can be YCrCb values ​​(also known as YUV values), where Y is the luminance value and U and V are chromaticity values ​​used to describe hue and saturation.

[0077] In an optional example, step 103 can be executed by the processor calling the corresponding instruction stored in memory, or by the generation module 503 run by the processor.

[0078] Step 104: Iteratively solve the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraints.

[0079] The gain coefficient is used to characterize the degree of adjustment required for the color values. Preset constraints may include minimizing the function value and ensuring the loss function value is less than at least one of preset values. The specific values ​​of these preset values ​​can be set according to actual needs and adjusted based on different scenarios.

[0080] During the iterative solution of the loss function, the values ​​of the gain coefficients can be continuously changed and the corresponding function values ​​can be calculated. When the function values ​​corresponding to a certain set of gain coefficients meet the preset constraints, the set of gain coefficients is determined as the final solution result.

[0081] In an optional example, step 103 can be executed by the processor calling the corresponding instruction stored in memory, or by the second determining module 504 run by the processor.

[0082] Step 105: Perform image color processing on the image corresponding to each node using the gain coefficient of each node.

[0083] For example, if the gain coefficient obtained from solving node A is 0.8, the target color value of the image corresponding to node A can be calculated by multiplying the original color value of the image corresponding to node A by the gain coefficient 0.8. Based on the target color value, the image data of the image corresponding to node A can be modified to obtain the image after color processing.

[0084] In an optional example, step 103 can be executed by the processor calling the corresponding instruction stored in memory, or by the processing module 505 run by the processor.

[0085] Based on this embodiment, the co-view relationship between images is determined based on the point cloud coordinate range corresponding to the images in the image sequence. An undirected graph corresponding to the image sequence is constructed with images as nodes and the co-view relationship between images as edges. A corresponding loss function is constructed based on the connected structure of the undirected graph. The function value of this loss function is used to characterize the total color difference of the undirected graph after adjusting the color value of the nodes. The color values ​​of each node are constrained to tend to be consistent through preset constraints. By iteratively solving the loss function, the gain coefficient corresponding to each node when the function value meets the preset constraints is determined. The color value of the image is optimized using this gain coefficient. When environmental changes cause inconsistencies in the color and brightness of overlapping content in multiple images, the color value of the images can be processed to make the color values ​​of images with co-view relationships similar, avoiding large differences in the color values ​​of points in the same area. This solves the problem of uneven coloring and poor preview effect of point clouds or 3D models caused by this, and helps to improve the visual coherence and realism of the color values ​​of point clouds or 3D models.

[0086] In one possible implementation, since this embodiment optimizes the color differences of different images in a shared viewing area and determines the degree of color value adjustment, for a pair of nodes with a shared viewing relationship, the average color value of the pixels in the shared viewing area of ​​the corresponding images can be determined as the color value of that image (node). As shown in Figure 3, step 103 above may include the following steps:

[0087] Step 103a: For each pair of nodes connected by each edge, determine the common viewing region of each node based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds onto the image corresponding to the node.

[0088] Optionally, after determining the point cloud coordinate range corresponding to each image, for each pair of images with a co-view relationship (i.e., two nodes connected by each edge), the co-view region of each image in the pair can be determined based on the projection of the point cloud intersection onto the pair of images. For the same image, the co-view region with different images may be located at different positions within the image. For example, in an image sequence, the co-view region of image a with image b is located in the left half of image a, and the co-view region of image a with image c is located in the right half of image a.

[0089] Step 103b: Determine the average color value of the pixels within the shared viewing area as the color value of that node.

[0090] Optionally, for two nodes in a shared viewing relationship, their respective average color values ​​can be determined directly by the average color value of the pixels within the shared viewing area. The average color value is the ratio of the sum of the color values ​​of all pixels within the shared viewing area to the number of pixels.

[0091] Optionally, considering the color value error and robustness of individual pixels, in this embodiment of the disclosure, a local window of a preset size, i.e., a matching block of a preset matching block size, can be used to determine the color value of a pixel, thereby improving the accuracy of the image color value and gain coefficient. The preset size or preset matching block size can be determined according to actual needs and can be adjusted as required; the specific value of the preset size or preset matching block size is not limited in this embodiment of the disclosure. Therefore, in one possible implementation, before step 103b, this embodiment of the disclosure may further include the following steps:

[0092] The matching block for each pixel is determined based on a preset matching block size. In an optional example, this step can be executed by the processor calling the corresponding instructions stored in memory, or by the third determining module 601 run by the processor.

[0093] The average color value of the pixels within the matching block is determined as the color value of the center pixel of the matching block. In an optional example, this step can be executed by the processor calling the corresponding instruction stored in memory, or by the fourth determining module 602 run by the processor.

[0094] Each pixel's matching block is an image region centered on that pixel and with a size equal to the preset matching block size.

[0095] For example, if the preset matching block size is 3*3, where "3" represents 3 pixels, then for any pixel in the common viewing area, the pixel and the 8 pixels around it are determined as the matching block corresponding to the pixel, and the average of the actual color values ​​of the 9 pixels in the matching block is determined as the color value of the pixel.

[0096] Therefore, based on step 103b, the color value of the pixel participating in the image color value calculation is the average of the actual color values ​​of the pixels in the corresponding matching block.

[0097] In one possible implementation, to further improve the accuracy of the node's color values ​​and avoid interference from special pixels, as shown in Figure 4, step 103b may include the following steps:

[0098] Step b1: Use the interquartile range (IQR) method to determine the first pixel point within the common viewing area.

[0099] The first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of a co-viewing pixel pair. A co-viewing pixel pair refers to a pair of pixels in the corresponding image that are the same point in the intersection of point clouds.

[0100] In this embodiment, the quartile is defined as follows: the pixel color difference of all common-view pixel pairs within the common-view area is arranged in ascending order and divided into four equal parts. The pixel color difference at the three dividing points includes the first quartile (i.e., the pixel color difference at the lower quartile or 25th quartile), the second quartile (i.e., the pixel color difference at the median), and the third quartile (i.e., the pixel color difference at the upper quartile or 75th quartile). Pixels with a pixel color difference greater than the upper quartile are defined as the first pixel.

[0101] Next, proceed to step b3.

[0102] Step b2: Perform object recognition on the image. In response to the existence of a target object belonging to a preset object type, determine the second pixel point corresponding to the target object.

[0103] The preset object type can include, but is not limited to, object types with low reliability of color values ​​such as glass, mirrors, screens, and transient objects. The target object is the object in the image that belongs to the preset object type. Optionally, in one possible implementation, the image corresponding to each node can be input into a pre-trained object recognition model, and the object recognition model outputs the various object types contained in the image and the image regions corresponding to each object type. The pixels in the image region corresponding to the preset object type are determined as the second pixel. The object recognition model can be a deep learning model, such as a neural network (NN) model, and this disclosure does not limit this.

[0104] It should be noted that there is no strict order of execution for steps b1 and b2; they can be executed simultaneously or in any order.

[0105] Step b3: Remove the first and second pixels within the shared viewing area to obtain the target pixels within the shared viewing area.

[0106] Step b4: Determine the average color value of the target pixel as the color value of the node.

[0107] Removing the first pixel reduces interference from abnormal pixels in image acquisition on color value calculation; removing the second pixel avoids interference from pixels with low color value reliability. Obtaining the color value corresponding to a node based on the average color values ​​of the retained target pixels improves the reliability of the node's color value, thereby increasing the accuracy of subsequent gain coefficient calculation.

[0108] Step 103c: Construct the loss function of the undirected graph based on the color value and gain coefficient of each node.

[0109] Here, the gain coefficient is the variable to be solved in the loss function.

[0110] By setting the gain coefficient of each node as the variable to be solved or optimized, a loss function for the undirected graph is constructed. The function value of this loss function is used to characterize the total color difference in the undirected graph after adjusting the color value of the corresponding node based on the gain coefficient of each node. Thus, by constraining the function value of the loss function, the value of the gain coefficient can be continuously adjusted to solve for the ideal gain coefficient corresponding to each node.

[0111] The strength of different co-view relationships varies. For node pairs with larger co-view regions, the co-view relationship is stronger; for node pairs with smaller co-view regions, the co-view relationship is weaker, and the edge stability is lower. Optionally, in one possible implementation, the influence of the color values ​​of node pairs with weak co-view relationships on the function values ​​can be reduced, making the obtained gain coefficients more closely match the image processing requirements of the image sequence. Based on this, step 103c may include the following steps:

[0112] The loss weight of each edge is determined based on the number of pixels in the shared viewing area corresponding to that edge; a loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient. The loss weight is directly proportional to the number of pixels.

[0113] Optionally, a range of values ​​for the loss weight can be preset, and the value of the loss weight can be determined from this range based on the number of pixels in the shared viewing area. An illustrative expression for the loss weight is as follows: W <i,j> =min (1.0, M)<i,j> / M mean ) Formula (1)

[0114] Among them, nodes i and j have a common view relationship, denoted as an edge.<i,j> W <i,j> For the edge<i,j> The loss weight, M <i,j> For the edge<i,j> The number of pixels in the corresponding shared viewing area, M mean This represents the average number of pixels in the shared region corresponding to all edges in an undirected graph. As shown in formula (1), when the edge...<i,j> The number of pixels in the corresponding shared viewing area exceeds the average value M mean At that time, on the side<i,j> The value of is 1, otherwise the edge<i,j> The value is the ratio of the number of pixels in the shared viewing area to the average value.

[0115] Optionally, in one possible implementation, the function value in the loss function is specifically the sum of the total color difference and the regularization term. The total color difference is specifically the weighted sum of the squares of the color differences corresponding to each edge. The color difference is specifically the difference in color values ​​between two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regularization distances. The sum of the regularization distances is the sum of the regularization distances corresponding to all nodes in the undirected graph. The regularization distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

[0116] An illustrative example of a loss function expression is as follows:

[0117] Where L is the loss function, W <i,j> For the edge<i,j> loss weights, For the edge<i,j> The color value of node i, w i Let be the gain coefficient of node i. For the edge<i,j> The color value of node j, w j Let be the gain coefficient of node j. Let λ be the regularization term, where λ is the regularization coefficient. This regularization term can constrain the gain coefficient w. k Not all values ​​are 0 to prevent the obtained gain coefficient from being meaningless. N is the number of nodes in the undirected graph - 1, which is also the number of images in the image sequence - 1. The value of N is an integer greater than 0. Nodes i and j ∈ [0, N] and nodes i and j are not the same node.

[0118] Optionally, in one possible implementation, the gradient descent method can be used to iteratively solve the loss function to determine the gain coefficients, based on constraints, to determine the gain coefficients of each node. The preset constraints may further include at least one of the following: minimizing the gain coefficient and ensuring that not all gain coefficients are zero. By constraining the gain coefficients to be minimized, the degree of color value adjustment in the image can be reduced to the greatest extent possible, preserving the original display effect of the image as much as possible while keeping the color values ​​of shared images similar.

[0119] Based on this embodiment, the color value of a pixel is determined by using a matching block, and pixels with large differences in color values ​​and low reliability of color values ​​in the shared viewing area are filtered out. Then, the image color value corresponding to the node is determined based on the mean of the color values ​​of the pixels in the shared viewing area. This can reduce the impact of errors on the color value of the node, improve robustness, and thus improve the accuracy of the gain coefficient. The loss weight of the edge is determined based on the size of the shared viewing area corresponding to the edge. This can reduce the influence of edges with weak shared viewing relationships on the function value and improve the reliability of the gain coefficient.

[0120] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0121] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0122] Figure 5 shows a structural block diagram of an image processing apparatus provided in an exemplary embodiment of this disclosure. The image processing apparatus of this embodiment can be used to implement the image processing method of the above embodiments. As shown in Figure 5, the image processing apparatus includes:

[0123] The first determining module 501 is used to determine the co-view relationship between images in the image sequence based on the point cloud coordinate range corresponding to each image in the world coordinate system, wherein the point cloud coordinate ranges corresponding to images with co-view relationship have an intersection.

[0124] Construction module 502 is used to construct an undirected graph corresponding to the image sequence, with images as nodes and the common-view relationship between images determined by the first determining module 501 as edges;

[0125] The generation module 503 is used to construct a loss function for the undirected graph constructed by the construction module 502. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color value includes at least one of chromaticity value and luminance value. The total color difference includes the sum of the color difference corresponding to each edge in the undirected graph. The color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge.

[0126] The second determining module 504 is used to iteratively solve the loss function generated by the generating module 503 and determine the gain coefficient of each node when the function value meets the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is the minimum and the function value is less than a preset value.

[0127] The processing module 505 is used to process the color of the image corresponding to the node using the gain coefficient determined by the second determining module 504.

[0128] Optionally, in one possible implementation, the generation module 503 is specifically used for:

[0129] For each edge connecting two nodes, the common viewing region of each node is determined based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds onto the image corresponding to the node.

[0130] The average color value of the pixels within the shared viewing area is determined as the color value of the node;

[0131] A loss function for an undirected graph is constructed based on the color value and gain coefficient of each node, where the gain coefficient is the variable to be solved in the loss function.

[0132] Optionally, in one possible implementation, as shown in FIG6, the image processing apparatus may further include:

[0133] The third determining module 601 is used to determine the matching block of each pixel based on the preset matching block size. The matching block is an image region centered on the pixel and with a size equal to the preset matching block size.

[0134] The fourth determining module 602 is used to determine the average color value of the pixels within the matching block as the color value of the center pixel of the matching block.

[0135] Optionally, in one possible implementation, when the generation module 503 determines the average color value of pixels within the shared viewing area as the color value of a node, it is specifically used for:

[0136] The first pixel within the common viewing area is determined using the interquartile range method. This first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of the common viewing pixel pair. The common viewing pixel pair is a pair of pixels in the corresponding image that share the same point in the intersection of the point clouds.

[0137] Object recognition is performed on the image. In response to the existence of a target object belonging to a preset object type, the second pixel point corresponding to the target object is determined.

[0138] Remove the first and second pixels within the shared viewing area to obtain the target pixels within the shared viewing area;

[0139] The average color value of the target pixel is used as the color value of the node.

[0140] Optionally, in one possible implementation, when the generation module 503 constructs the loss function of the undirected graph based on the color value and gain coefficient of each node, it is specifically used for:

[0141] The loss weight of an edge is determined based on the number of pixels in the shared viewing area corresponding to the edge, and this loss weight is proportional to the number of pixels.

[0142] The loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient.

[0143] Optionally, in one possible implementation, the function value in the loss function is specifically the sum of the total color difference and the regularization term. The total color difference is specifically the weighted sum of the squares of the color differences corresponding to each edge in the undirected graph. The color difference is specifically the difference in color values ​​between two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regularization distances. The sum of the regularization distances is the sum of the regularization distances corresponding to all nodes in the undirected graph. The regularization distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

[0144] Optionally, in one possible implementation, the second determining module 504 may also be used for:

[0145] The gradient descent method is used to iteratively solve the loss function based on preset constraints to determine the gain coefficient of each node. The preset constraints also include at least one of the following: the gain coefficient is minimized and the gain coefficient is not all zero.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. For identical, similar, or corresponding content between embodiments, please refer to the corresponding sections. Since the method and apparatus embodiments are basically corresponding, relevant parts can be referred to accordingly. The methods and apparatuses in the embodiments of this disclosure also correspond to each other in specific implementation and beneficial technical effects; related content can be referenced and cited interchangeably, and will not be repeated here.

[0147] In addition, this disclosure also provides an electronic device, including:

[0148] Memory, used to store computer programs;

[0149] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the image processing method described in any of the above embodiments of the present disclosure.

[0150] Figure 7 is a schematic diagram of an application embodiment of the electronic device of this disclosure. The electronic device according to an embodiment of this disclosure will now be described with reference to Figure 7. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which can communicate with the first and second devices to receive acquired input signals from them.

[0151] As shown in Figure 7, the electronic device includes one or more processors and a memory.

[0152] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0153] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the image processing methods of the various embodiments of this disclosure described above and / or other desired functions.

[0154] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0155] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0156] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0157] Of course, for simplicity, Figure 7 only shows some of the components of the electronic device that are relevant to this disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0158] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0159] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0160] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0161] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0162] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above image processing method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0163] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0165] The block diagrams of the devices involved in this disclosure are merely illustrative examples and are not intended to require or imply that connections, arrangements, or configurations must be made in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0166] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0167] It should also be noted that in the apparatus and method of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0168] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0169] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, characterized in that, include: Based on the point cloud coordinate range corresponding to each image in the world coordinate system in the image sequence, the co-view relationship between the images in the image sequence is determined, wherein the point cloud coordinate ranges corresponding to the images with co-view relationship have an intersection; An undirected graph corresponding to the image sequence is constructed using the images as nodes and the co-visibility relationships between the images as edges. Construct a loss function for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to the edges in the undirected graph. The color difference corresponding to the edge is the difference in color values ​​between the two nodes connected by the edge. The loss function is iteratively solved to determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is minimized and the function value is less than a preset value. The color of the image corresponding to the node is processed using the gain coefficient.

2. The method according to claim 1, characterized in that, The loss function for constructing the undirected graph includes: For each edge connecting two nodes, the common viewing region of each node is determined based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds in the image corresponding to the node. The average color value of the pixels within the shared viewing area is determined as the color value of the node; The loss function of the undirected graph is constructed based on the color value of each node and the gain coefficient, wherein the gain coefficient is the variable to be solved in the loss function.

3. The method according to claim 2, characterized in that, Before determining the average color value of the pixels within the shared viewing area as the color value of the node, the method further includes: The matching block for each pixel is determined based on a preset matching block size. The matching block is an image region centered on the pixel and with a size equal to the preset matching block size. The average color value of the pixels within the matching block is determined as the color value of the center pixel of the matching block.

4. The method according to claim 3, characterized in that, Determining the average color value of the pixels within the shared viewing area as the color value of the node includes: The first pixel in the common viewing area is determined by the interquartile range method. The first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of the common viewing pixel pair. The common viewing pixel pair is a pair of pixels in the corresponding image that share the same point in the intersection of the point clouds. Object recognition is performed on the image, and in response to the existence of a target object belonging to a preset object type, the second pixel point corresponding to the target object is determined; Remove the first pixel and the second pixel within the shared viewing area to obtain the target pixel within the shared viewing area; The average color value of the target pixel is determined as the color value of the node.

5. The method according to any one of claims 2 to 4, characterized in that, The loss function for constructing the undirected graph based on the color value of each node and the gain coefficient includes: The loss weight of each edge is determined based on the number of pixels in the shared viewing area corresponding to each edge, and the loss weight is proportional to the number of pixels. The loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient.

6. The method according to claim 5, characterized in that, The loss function is specifically the sum of the total color difference and the regularization term. The total color difference is specifically the weighted sum of the squares of the color differences corresponding to the edges in the undirected graph. The color difference is specifically the difference in color values ​​between the two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regular distances. The sum of the regular distances is the sum of the regular distances corresponding to all nodes in the undirected graph. The regular distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

7. The method according to any one of claims 1 to 4, characterized in that, The iterative solution of the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraints includes: The gradient descent method is used to iteratively solve the loss function based on the preset constraints to determine the gain coefficient of each node. The preset constraints also include at least one of the following: the gain coefficient is minimized and the gain coefficient is not all zero.

8. An image processing apparatus, characterized in that, include: The first determining module is used to determine the co-view relationship between images in the image sequence based on the point cloud coordinate range corresponding to each image in the world coordinate system, wherein the point cloud coordinate ranges corresponding to images with co-view relationship have an intersection. The construction module is used to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-view relationships between the images as edges. A generation module is used to construct a loss function for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to the edges in the undirected graph. The color difference corresponding to the edge is the difference in color values ​​between the two nodes connected by the edge. The second determining module is used to iteratively solve the loss function and determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the function value being the minimum and the function value being less than a preset value. The processing module is used to process the color of the image corresponding to the node using the gain coefficient.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images