Three-dimensional reconstruction method and apparatus, and computing device cluster

Abstract

Provided in the present invention are a three-dimensional reconstruction method and apparatus, and a computing device cluster. In an embodiment, the method comprises: acquiring a three-dimensional model of a scene, wherein the three-dimensional model of the scene is a three-dimensional reconstruction result of the scene, and the scene comprises a target static object; acquiring a plurality of image frames, wherein the plurality of image frames are obtained by means of photographing the scene from surrounding viewpoints, the plurality of image frames include a target image frame, the target image frame carries annotation information, and the annotation information is used for indicating the position of the target static object in the target image frame; on the basis of the plurality of image frames and the annotation information, determining contour information, wherein the contour information is used for indicating contours of the target static object in the plurality of image frames; and on the basis of the contour information, extracting the target static object from the three-dimensional model of the scene, in order to obtain a three-dimensional model of the target static object. Contours of a specified object from different angles under photographing from surrounding viewpoints are extracted from a plurality of image frames obtained by means of photographing from surrounding viewpoints, and on the basis of the contours of the specified object from different angles under photographing from surrounding viewpoints, a three-dimensional model of the specified model is extracted from a three-dimensional model of a scene.

Description

3D reconstruction methods, devices and computing equipment clusters

[0001] This application claims priority to Chinese Patent Application No. 202411998809.8, filed on December 31, 2024, entitled “Three-dimensional Reconstruction Method, Apparatus and Computing Device Cluster”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of computer vision technology, specifically to a three-dimensional reconstruction method, apparatus, and computing device cluster. Background Technology

[0003] 3D Gaussian is a technique for representing and rendering 3D scenes. It describes volumetric elements in a scene using a large number of 3D Gaussian distributions. Each 3D Gaussian has attributes such as position, color, opacity, and shape, which can be adjusted through optimization algorithms to suit different application requirements.

[0004] However, existing 3D Gaussian reconstruction technology reconstructs objects and the environment as a whole, and cannot distinguish the contact surfaces between objects and the environment. It can only extract objects and the environment together, which increases the difficulty of extracting objects from the environment separately. Summary of the Invention

[0005] This invention provides a three-dimensional reconstruction method, apparatus, and computing device cluster. By using multiple frames of images obtained through surround shooting, the contours of a specified object at different angles under surround shooting are extracted. Based on the contours of the specified object at different angles under surround shooting, the three-dimensional model of the specified object is extracted from the scene's three-dimensional model.

[0006] In a first aspect, embodiments of the present invention provide a three-dimensional reconstruction method, the method comprising:

[0007] The process involves acquiring a 3D scene model, which is the result of a 3D reconstruction of the scene, including the target static object. Before or after acquiring the 3D scene model, multiple frames of images are captured. These frames are obtained by taking panoramic photos of the scene and can be used to indicate the target static object from different angles. Each frame includes a target frame image carrying annotation information indicating the position of the target static object within that frame. After determining the labeled positions of the target static object, the target static object to be extracted from the scene can be identified. Then, based on the multiple frames and annotation information, contour information is determined, indicating the contour of the target static object within the multiple frames. After obtaining the contour information, the contour interface of the target static object can be analyzed. Therefore, based on the contour information, the target static object can be extracted from the 3D scene model, resulting in a 3D model of the target static object.

[0008] In this scheme, the outlines of a specified object at different angles under the surround shooting are extracted from multiple frames of images obtained by surround shooting. Based on the outlines of the specified object at different angles under the surround shooting, the 3D model of the specified object is extracted from the 3D model of the scene.

[0009] In one possible implementation, determining contour information based on multiple frames of images and annotation information includes: extracting the target static object from the target frame image based on the annotation information to obtain a first extraction result of the target frame image, thereby segmenting the contour of the target static object from the image; after extracting the target static object, extracting the target static object from each of the other frames of the multiple frames based on the first extraction result of the target frame image to obtain a second extraction result of each of the other frames of the multiple frames, thereby obtaining the contour of the target static object at various angles; subsequently, obtaining contour information based on the first extraction result of the target frame image and the second extraction result of each of the other frames of the multiple frames, for example, using the first extraction result of the target frame image and the second extraction result of each of the other frames of the multiple frames as contour information, or, for example, correcting the extraction result of each frame of the multiple frames based on the first extraction result of the target frame image and the second extraction result of each of the other frames of the multiple frames, and using the corrected extraction result of each frame of the multiple frames as contour information.

[0010] In this scheme, the outline of the target static object is segmented from the image by extracting the target static object from the image.

[0011] In one example of this implementation, before determining the contour information based on multiple frames of images and annotation information, the method further includes: determining the position range of the target static object in the multiple frames of images based on the multiple frames of images and annotation information; and removing content in the multiple frames of images that is outside the position range.

[0012] In this approach, by removing content from the image other than the approximate location of the target static object, the influence of irrelevant content in the image on the extraction is reduced, thereby improving the accuracy of extracting the outline of the target static object.

[0013] In one example of this implementation, the target frame image is used to indicate the front of the target static object.

[0014] In this approach, the front of the target static object is annotated to include as many features of the target static object as possible, thereby improving the accuracy of extracting the outline of the target static object.

[0015] In one possible implementation, based on contour information, the target static object is extracted from the scene's 3D model to obtain a 3D model of the target static object, including:

[0016] First, the coordinate system of the scene's 3D model is determined, with the target shooting point as the origin. The target shooting point is the shooting point of any frame in the multi-frame images. Then, based on the multi-frame images and the target shooting point, the shooting process of the multi-frame images is reconstructed, with the target shooting point as the origin, to determine the image region of each frame in the scene's 3D model. Subsequently, based on the contour information and the image region, the target region where the target static object is located in the scene's 3D model is determined, and the content of the target region is extracted to obtain the 3D model of the target static object.

[0017] In this solution, an image region is obtained by simulating a surround shooting process in the 3D model of the scene. The image region is combined with the outline of the target static object to obtain the region of the target static object in the 3D model of the scene. The content of the region can be extracted to obtain the 3D model of the target static object.

[0018] In one example of this implementation, the contour information is a mask image of each frame in a multi-frame image, and the mask image is used to indicate the area where the target static object is located. Based on the contour information and the image area, the target area of ​​the target static object in the scene 3D model is determined, including: aligning the mask image and the image area of ​​each frame in the multi-frame image to determine the target area where the target static object is located in the image area.

[0019] In this approach, by aligning the mask image corresponding to each frame of multiple images with the image region in the scene's 3D model, the region where the target static object is located in the scene's 3D model can be analyzed. Subsequently, by extracting the content of this region, the 3D model of the target static object can be extracted.

[0020] In one possible implementation, obtaining a 3D model of the scene includes: performing 3D reconstruction with the goal of restoring multiple frames of images to obtain a 3D model of the scene.

[0021] In this solution, a 3D model is constructed by restoring the image, thereby improving the accuracy of the scene's 3D model.

[0022] In one possible implementation, acquiring multiple frames of images includes: acquiring video, which is obtained by shooting around the scene; extracting keyframes from the video to obtain multiple frames of images with high reference value, so that multiple images can more accurately represent the outline of the target static object from different angles.

[0023] Secondly, embodiments of the present invention provide a three-dimensional reconstruction apparatus, which includes several modules. Each module is used to execute various steps in the three-dimensional reconstruction method provided in the first aspect of the present invention. The division of modules is not limited here. For the specific functions performed by each module of the three-dimensional reconstruction apparatus and the beneficial effects achieved, please refer to the functions of each step in the three-dimensional reconstruction method provided in the first aspect of the present invention; further details will not be repeated here.

[0024] Exemplarily, the three-dimensional reconstruction apparatus includes:

[0025] The model acquisition module is used to acquire the 3D model of the scene. The 3D model of the scene is a 3D representation of the scene, which includes the target static object.

[0026] The image acquisition module is used to acquire multiple frames of images. The multiple frames of images are obtained by taking pictures around the scene. The multiple frames of images include target frame images, which carry annotation information to indicate the position of the target static object.

[0027] The contour analysis module is used to determine contour information based on multiple frames of images and annotation information. The contour information is used to indicate the contour of the target static object in the multiple frames of images.

[0028] The extraction module is used to extract target static objects from the scene's 3D model based on contour information, thereby obtaining a 3D model of the target static object.

[0029] In one possible implementation, the contour analysis module is used to extract the target static object in the target frame image based on the annotation information to obtain a first extraction result of the target frame image; based on the first extraction result of the target frame image, extract the target static object in each of the other frames in the multi-frame image to obtain a second extraction result of each of the other frames in the multi-frame image; and based on the first extraction result of the target frame image and the second extraction result of each of the other frames in the multi-frame image, obtain contour information.

[0030] In one example of this implementation, the device also includes:

[0031] The removal module is used to determine the location range of the target static object in the multi-frame images based on multi-frame images and annotation information; and to remove content outside the location range in the multi-frame images.

[0032] In one example of this implementation, the target frame image is used to indicate the front of the target static object.

[0033] In one possible implementation, an extraction module is used to determine the coordinate system of the scene's 3D model. The coordinate system is constructed based on the target shooting point, which is the shooting point of any frame in the multi-frame images. Based on the multi-frame images and the target shooting point, the shooting process of the multi-frame images is reconstructed, and the image region of each frame in the multi-frame images in the scene's 3D model is determined. Based on the contour information and the image region, the target region where the target static object is located in the scene's 3D model is determined, and the content of the target region is extracted to obtain the 3D model of the target static object.

[0034] In one example of this implementation, the contour information is the mask image of each frame in a multi-frame image, and the mask image is used to indicate the area where the target static object is located; the extraction module is used to align the mask image and the image area of ​​each frame in the multi-frame image to determine the target area where the target static object is located in the image area.

[0035] In one possible implementation, the model acquisition module is used to perform 3D reconstruction with the goal of restoring multiple frames of images to obtain a 3D model of the scene.

[0036] In one possible implementation, an image acquisition module is used to acquire video, which is obtained by shooting around the scene; keyframes are extracted from the video to obtain multiple frames of images.

[0037] Thirdly, embodiments of the present invention provide a three-dimensional reconstruction apparatus, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

[0038] Fourthly, embodiments of the present invention provide a three-dimensional reconstruction apparatus that executes computer program instructions to perform the method provided in the first aspect. Exemplarily, the apparatus may be a chip or a processor.

[0039] In one example, the device may include a processor that can be coupled to memory, read instructions from the memory, and execute the methods provided in the first aspect according to those instructions. The memory may be integrated into the chip or processor, or it may be independent of the chip or processor.

[0040] Fifthly, embodiments of the present invention provide a computing device cluster, the computing device cluster including at least one computing device, each computing device including a processor and a memory;

[0041] A processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device to cause a cluster of computing devices to perform the method provided in the first aspect.

[0042] In a sixth aspect, embodiments of the present invention provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect.

[0043] In a seventh aspect, embodiments of the present invention provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. Attached Figure Description

[0044] Figure 1 is a schematic diagram of an artificial intelligence main framework provided in an embodiment of the present invention;

[0045] Figure 2a is a schematic diagram of the architecture of a three-dimensional reconstruction system provided in an embodiment of the present invention;

[0046] Figure 2b is a schematic diagram of the architecture of a three-dimensional reconstruction system deployed in the cloud according to an embodiment of the present invention;

[0047] Figure 2c is a schematic diagram of the interaction between a management platform 221 and a terminal 210 provided in an embodiment of the present invention;

[0048] Figure 3a is a schematic diagram of a three-dimensional reconstruction scene provided by an embodiment of the present invention;

[0049] Figure 3b is a schematic diagram of a surround shooting method provided by an embodiment of the present invention;

[0050] Figure 3c is a schematic diagram of another surround shooting method provided by an embodiment of the present invention;

[0051] Figure 3d is a schematic diagram of the internal modules of the management platform 221 provided in an embodiment of the present invention;

[0052] Figure 4 is a flowchart illustrating a three-dimensional reconstruction method provided in an embodiment of the present invention;

[0053] Figure 5a is a schematic diagram of the annotation scenario provided in an embodiment of the present invention;

[0054] Figure 5b is a schematic diagram of the management platform 221 and terminal 210 labeled in an embodiment of the present invention;

[0055] Figure 6a is a schematic diagram of a mask image provided in an embodiment of the present invention;

[0056] Figure 6b is a schematic diagram of another mask image provided in an embodiment of the present invention;

[0057] Figure 6c is a schematic diagram of a mask scenario provided in an embodiment of the present invention;

[0058] Figure 6d is a schematic diagram of the scheme for determining shooting pose and scene perspective provided in an embodiment of the present invention;

[0059] Figure 7a is a schematic diagram of the structure of a contour tracking network provided in an embodiment of the present invention;

[0060] Figure 7b is a schematic diagram of another contour tracking network provided in an embodiment of the present invention;

[0061] Figure 8a is a schematic diagram of a three-dimensional model extraction method provided in an embodiment of the present invention;

[0062] Figure 8b is a schematic diagram of another three-dimensional model extraction provided by an embodiment of the present invention;

[0063] Figure 9a is a schematic diagram of a method for restoring M-frame images by surround shooting according to an embodiment of the present invention;

[0064] Figure 9b is a schematic diagram of removing a mask region according to an embodiment of the present invention;

[0065] Figure 10a is a schematic diagram of a three-dimensional reconstruction scene provided by an embodiment of the present invention;

[0066] Figure 10b is a flowchart illustrating the 3D reconstruction method in the scenario shown in Figure 10a;

[0067] Figure 11 is a schematic diagram of the structure of the three-dimensional reconstruction device provided in an embodiment of the present invention;

[0068] Figure 12 is a schematic diagram of the structure of the computing device provided in an embodiment of the present invention;

[0069] Figure 13 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present invention;

[0070] Figure 14 is a schematic diagram of computing devices in a computer cluster connected via a network according to an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0072] In the description of the embodiments of the present invention, the words "exemplary," "for example," or "for instance" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary," "for example," or "for instance" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0073] In the description of the embodiments of this invention, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0074] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0075] The following explanations cover some of the terms used in this embodiment. It should be noted that these explanations are for the convenience of those skilled in the art and are not intended to limit the scope of protection claimed by this invention.

[0076] Artificial Intelligence (AI) is a branch of computer science that attempts to understand the nature of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0077] 3D Gaussian: A technique for representing and rendering 3D scenes, it uses a large number of 3D Gaussian distributions to describe volumetric elements in a scene. Each 3D Gaussian has attributes such as position, color, opacity, and shape, which can be adjusted through optimization algorithms to suit different application requirements.

[0078] 3D reconstruction refers to the creation of mathematical models of 3D objects that are suitable for computer representation and processing. It is the basis for processing, manipulating and analyzing the properties of objects in a computer environment, and a key technology for creating virtual reality that expresses the objective world in a computer.

[0079] Cloud: A software platform that uses application virtualization technology, integrating multiple functions such as software search, download, use, management, and backup.

[0080] Object segmentation refers to dividing objects in an image into different regions to represent different categories.

[0081] Surround shooting: The camera moves in a circular motion around the subject, using the subject as the center point. This technique typically employs stabilizers, rotating tracks, drones, etc., to highlight the subject and showcase the relationship between the subject and its environment, or the relationships between people.

[0082] Scene: Represents a specific area in the geographical world. The scene in this invention needs to be determined in conjunction with user needs, such as an intersection, a garage, etc.

[0083] Outline: The lines that form the outer edge of a figure or object.

[0084] 3D Gaussian Splatting (3DGS) is an advanced 3D modeling and visualization technique primarily used in computer graphics, computer vision, and related fields to achieve efficient and accurate representation of 3D scenes. 3DGS uses Gaussian functions to represent points or volumes in 3D space. Each point is described by a Gaussian function that defines the point's spatial location and the degree of diffusion along each axis. Through Gaussian representation, 3DGS can compress 3D data and perform rapid neighborhood searches and updates, making it particularly suitable for dynamic scenes and real-time applications.

[0085] Video: refers to various technologies that capture, record, process, store, transmit, and reproduce a series of still images in the form of electrical signals. When the continuous image changes exceed 24 frames per second, according to the principle of visual persistence, the human eye cannot distinguish individual still images, and what appears to be a smooth and continuous visual effect is called video.

[0086] Object Storage Service (OBS): This is a cloud storage service based on objects.

[0087] The AI ​​Basic Development Platform is a one-stop AI development platform for developers, providing various capabilities throughout the entire AI development process. For example, the capabilities offered by the AI ​​Basic Development Platform can include the following six main parts: data preprocessing, model building and training, model management, model deployment, data optimization, and model optimization and updates. These capabilities can be integrated for users to utilize throughout the entire AI process, or they can be provided as independent functions.

[0088] Segment Anything (SAM) is a pre-trained two-dimensional image segmentation model proposed by Meta, capable of generating a mask for any object in any image or video. The entire model consists of: (1) an image encoder to compute image embeddings, (2) a cue encoder to embed cues, and (3) a lightweight mask decoder to combine the two information sources and predict the segmentation mask.

[0089] Camera intrinsic parameters describe the internal properties of a camera, primarily including the following: Focal length (f): The focal length of the camera lens, which determines the magnification of the image. Principal point (optical center) coordinates: The center point of the image, usually the center of the image. Distortion coefficients: Image distortion parameters caused by lens design or manufacturing defects. Intrinsic parameters are usually determined during camera calibration because they are fixed for a specific camera model and do not change over time. Once the camera intrinsic parameters are determined, they generally remain constant during the use of the camera.

[0090] Camera extrinsic parameters describe the camera's position and orientation in the world coordinate system, mainly including the following aspects: Rotation matrix: Represents the camera's rotation in the world coordinate system. Translation vector: Represents the camera's translation in the world coordinate system. Extrinsic parameters may change with different camera positions. For example, in stereo vision, if there are two cameras, their relative positions and orientations will change each time the cameras move, resulting in changes in the extrinsic parameters. If the camera's position or orientation changes, the extrinsic parameters need to be updated to reflect these changes.

[0091] Camera calibration is an important problem in computer vision. Its purpose is to determine the camera's intrinsic parameters (camera intrinsics) and extrinsic parameters (camera extrinsics) in order to establish a mapping relationship from the camera to the image.

[0092] Feature matching is an image processing technique used to find similar feature points between different images and match them.

[0093] Pose motion refers to the process of changing the position and orientation of an object in space. Pose includes position and orientation; position represents the object's specific coordinates in space, usually represented by {x, y, z} in a three-dimensional Cartesian coordinate system. Orientation represents the object's direction and angle in space, and can be represented by orthogonal rotation matrices, fixed angles, Euler angles, equivalent axis angles, unit quaternions, etc.

[0094] Shooting point: This refers to the location of the camera and its relative relationship to the subject during photography. Specifically, the shooting point includes three elements: the camera lens's orientation, distance, and height relative to the subject. These three elements together determine the shooting point.

[0095] A key frame (or key frame) is a key image frame in a video that reflects the main content of a shot.

[0096] Input / output interface (I / O interface): This is the connection circuit between the processor and external devices for exchanging information. They are connected to the processor via a bus.

[0097] Extraction: In image processing, extraction refers to removing unwanted elements and extracting only the desired portion of the image, thus separating a specific object from the background.

[0098] Mask: In image processing, a mask is used to define the transparency or visibility of specific areas in an image. It is typically a binary or Boolean image of the same size as the original image, where the selected area is marked as 1 (or True), and the remaining areas are marked as 0 (or False). By using a mask, it is possible to edit specific parts of an image without affecting other parts.

[0099] Figure 1 shows a schematic diagram of an artificial intelligence framework, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence needs.

[0100] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0101] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0102] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of artificial intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0103] (1) Infrastructure:

[0104] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as Central Processing Units (CPUs), Neural Processing Units (NPUs), and Graphics Processing Units (GPUs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0105] (2) Data

[0106] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, as well as IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0107] (3) Data processing

[0108] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0109] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0110] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0111] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0112] (4) General ability

[0113] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0114] (5) Smart Products and Industry Applications

[0115] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0116] In the application scenario of this invention embodiment, the data in Figure 1 can be video or multi-frame images, and correspondingly, the data processing can be various image-related functions such as image processing. Additionally, the general capability can be a 3D reconstruction algorithm, which is used to implement the technical solution provided in this invention embodiment.

[0117] First, the three-dimensional reconstruction system to which the method provided in the embodiments of the present invention may be applied will be described. Figure 2a is a schematic diagram of the architecture of a three-dimensional reconstruction system provided in an embodiment of the present invention. As shown in Figure 2a, the system includes a terminal 210 and a three-dimensional reconstruction cluster 220. The terminal 210 and the management platform 221 are connected via a network. The network can be a wired network and / or a wireless network. It is understood that the network can use any known network communication protocol to achieve different communications, and the aforementioned network communication protocol can be various wired and / or wireless communication protocols. For example, the wired network can be a cable network, an optical fiber network, a Digital Data Network (DDN), etc., and the wireless network can be a telecommunications network, an internal network, the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a ZigBee network, a Global System for Mobile Communications (GSM), etc., or any combination thereof. It is understandable that the network can use any known network communication protocol to achieve communication between different client layers and gateways. The aforementioned network communication protocol can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), and other communication protocols.

[0118] The terminal 210 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Exemplary embodiments of the terminal 210 involved in this solution include, but are not limited to, electronic devices running iOS, Android, Windows, Harmony OS, or other operating systems. This embodiment of the invention does not specifically limit the type of electronic device.

[0119] The 3D reconstruction cluster 220 can be configured as an independent physical server, a server cluster consisting of multiple physical servers, or a cloud server or cloud server cluster. The cloud server or cloud server cluster provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), big data, and artificial intelligence platforms.

[0120] In one possible scenario, the 3D reconstruction cluster 220 can be configured as a management platform 221 and a data center 222, as shown in Figure 2b. The management platform 221 and data center 222 can be deployed in the cloud; in this case, the management platform 221 can be a cloud management platform. The terminal 210 and the 3D reconstruction cluster 220 interact through the management platform 221. For example, as shown in Figure 2c, the management platform 221 provides an I / O interface 201. The terminal 210 can input data to the management platform 221 through the I / O interface 201, and the management platform 221 can output results to the terminal 210 through the I / O interface 201. Additionally, the data center 222 can deploy nodes, which can be virtual machine instances, container instances, physical servers, etc. The number of nodes in the data center 222 is typically massive. The management platform 221 can be deployed independently on servers or virtual machines in data center 222, or distributed across multiple servers or virtual machines in data center 222. It can also be partially deployed independently or distributedly on devices in the edge environment (also called edge devices), while another part can be deployed independently or distributedly within data center 222. The edge environment is the environment geographically close to the user's terminal 210, and includes edge devices such as edge servers and edge stations with computing capabilities.

[0121] In this embodiment of the invention, nodes can be used for model training. In one possible scenario, when the model is large, it can be distributed across multiple nodes for parallel training based on the concept of model parallelism. In another possibility, the model can be trained on a single node. Furthermore, for model development, developers can install an AI development framework on terminal 210 and then develop AI models locally, or they can use an AI development framework on an online platform (e.g., an online open-source framework platform, a public cloud AI infrastructure development platform, etc.) to develop AI models. AI development frameworks in the industry are typically open-source. Typical AI development frameworks used for developing deep learning models, also known as deep learning frameworks, include: PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch, and PyTorch, etc.

[0122] Next, in conjunction with the three-dimensional reconstruction system provided above, a three-dimensional reconstruction method provided by an embodiment of the present invention will be described in detail.

[0123] In related technologies, 3D reconstruction of a scene is performed without considering semantics. If multiple objects exist in the scene and these objects are in contact with each other, the 3D reconstruction process treats these multiple objects as an indivisible whole.

[0124] To address the aforementioned issues, as shown in Figure 3a, the 3D reconstruction method provided in this embodiment of the invention may include: acquiring a 3D model of the scene corresponding to the scene; then, acquiring M (positive integers greater than or equal to 2) frames of images obtained by an image capturing device taking panoramic shots of the scene; the user marking the position of the target static object in the scene in at least some of the M frames to obtain annotation information (used to indicate the position of the target static object in at least some of the frames); subsequently, based on the M frames and the annotation information, obtaining the contour information of the target static object; and segmenting the 3D model of the target static object from the 3D model of the scene using the contour information of the target static object.

[0125] In an optional implementation of this embodiment, as shown in Figure 3a, the scene includes object A and object B, which have a contact surface. After object A and object B are reconstructed as a whole in 3D, a 3D model of the scene is obtained. Since object A and object B are a whole, object A cannot be extracted separately. Subsequently, the image capturing device captures M frames of images around the scene, and marks the position of object A in at least some of the M frames. Based on the M frames and the marked position of object A, the contour information of object A is obtained. The 3D model of object A is segmented from the 3D model of the scene using the contour information of object A.

[0126] The image capturing device can be a drone, a camera, or other equipment with image capturing capabilities.

[0127] The surround shooting method can be a single circular shot as shown in Figure 3b, a multi-circular shot as shown in Figure 3c, or a spiral shot (not shown in the figure). It needs to cover all visible surfaces of the scene as much as possible. This embodiment of the invention does not intend to limit the surround shooting method; it can be flexibly designed according to actual conditions. In this embodiment, the M-frame images can be images extracted from a video of surround shooting of the scene, or images obtained from M shooting points (surrounding the scene). The M-frame images need to cover the visible surfaces of the target static object, thereby allowing for relatively complete extraction of the target static object's outline. Furthermore, for surfaces of the target static object that are occluded in the M-frame images, the method provided in this embodiment can segment the occluded surfaces of the target static object in the scene's 3D model, thereby extracting the 3D model of the target static object. Optionally, the M-frame images are used to indicate the outline of the target static object at different angles, covering the visible surfaces of the target static object at different angles. For example, if the bottom of the target static object is occluded, then different angles include at least the front, left, right, back, and top.

[0128] In an optional implementation of this embodiment, terminal 210 uploads a scene 3D model to management platform 221 via I / O interface 201.

[0129] In an optional implementation of this embodiment, as shown in FIG3d, the management platform 221 may include an I / O interface 201, a three-dimensional reconstruction model 202, an annotation tool 203, and an image segmentation model 204.

[0130] The annotation tool 203 is used by the terminal 210 to annotate the location of the target static object in the scene in at least a portion of the M-frame images. Optionally, the terminal 210 accesses the annotation tool 203 through the I / O interface 201 and uses the annotation tool 203 to annotate the location of the target static object in the frame images.

[0131] Among them, the image segmentation model 204 is used to obtain the contour information of the target static object based on the M-frame image and the labeled information.

[0132] The 3D reconstruction model 202 is used to perform 3D reconstruction of the scene to obtain a 3D model of the scene. This embodiment of the invention does not intend to limit the structure of the 3D reconstruction model; it can be designed according to actual applications. Furthermore, currently available open-source models can be used for the 3D reconstruction model.

[0133] Optionally, the terminal 210 uploads data for 3D reconstruction, such as video or image sequences, to the management platform 221 via the I / O interface 201. The management platform 221 inputs the data for 3D reconstruction into the 3D reconstruction model 202, and the 3D reconstruction model 202 outputs the scene 3D reconstruction model.

[0134] Optionally, the 3D reconstruction model 202 can be implemented using 3DGS technology. 3DGS represents the scene as several ellipsoids, each with attributes such as position, shape, opacity, and color, specifically including:

[0135] (1) Location: Mean (x,y,z) of a 3D Gaussian;

[0136] (2) Covariance matrix: describes the shape of a 3D Gaussian (3*3 matrix), which can be equivalently represented by an ellipsoid: scaling S(sx,sy,sz) + rotation R (quaternion);

[0137] (3) Opacity: Used for rendering blending (alpha);

[0138] (4) Spherical harmonic coefficient: describes the appearance color information related to the shooting angle.

[0139] 3DGS interweaves the optimization of 3D Gaussian parameters (i.e., position, covariance, opacity, and spherical harmonics) with the adaptive control of the density of 3D Gaussians to create a radiation field representation. The key to efficiency is a slice-based rasterizer (renderer) that allows for opacity blending of anisotropic volumetric splatters.

[0140] The training of the 3D reconstruction model 202 is described below.

[0141] Optionally, a training dataset needs to be obtained, which includes multiple image sequence samples, each of which can be obtained by taking photos of a selected scene from all angles. The selected scenes are different for different image sequence samples, and the selected scenes need to cover as many fields as possible.

[0142] Image sequence samples can be the result of video data preprocessing, which may include keyframe extraction, data augmentation, and data augmentation. Optionally, data preprocessing can be implemented by the management platform 221. In specific implementation, the terminal 210 sends a data processing task to the management platform 221, indicating the locations of the first dataset and the second dataset. The first dataset is used to store video, and the second dataset is used to store image sequence samples. The first dataset can be pre-collected by the user according to the actual application scenario, or it can use an existing open-source dataset from the industry.

[0143] For example, a user can pre-purchase an object storage service (OBS) on management platform 221 and store the first dataset in OBS.

[0144] For example, management platform 221 can provide an AI basic development platform. Subsequently, terminal 210 accesses the AI ​​basic development platform and configures data processing tasks. Later, data center 222 needs to use the data preprocessing function of the AI ​​basic development platform to perform data processing tasks. When using the data preprocessing function, data center 222 reads video from a first dataset (e.g., video from OBS), and can perform one or more operations on the data as needed, such as data selection, keyframe extraction, data augmentation, and data broadening, to obtain image sequence samples. These image sequence samples are then stored in a second dataset. Subsequently, management platform 221 can directly obtain image sequence samples from the second dataset.

[0145] Next, the 3D reconstruction model 202 is iteratively trained based on multiple image sequence samples until the training completion condition is met. The training completion condition refers to the conditions for stopping model training, including but not limited to reaching the maximum number of training iterations, the model loss information reaching a preset threshold, and / or the model parameters no longer changing. Optionally, during the training process, for each image sequence sample, after inputting the image sequence sample into the 3D reconstruction model 202, a 3D model output by the 3D reconstruction model 202 is obtained. Based on the 3D model output by the 3D reconstruction model 202, the image sequence sample is reconstructed to obtain the reconstructed image sequence. Based on the error between the reconstructed image sequence and the image sequence sample, the error corresponding to the image sequence sample is obtained. Based on the errors corresponding to multiple image sequence samples, the model parameters of the 3D reconstruction model 202 are updated. Then, the updated 3D reconstruction model 202 replaces the previous 3D reconstruction model 202, and the next iteration is performed until the training completion condition is met.

[0146] In practical applications, terminal 210 accesses the AI ​​infrastructure development platform to configure model training tasks. These tasks can specify the location of the second dataset, the location of the 3D reconstruction model 202, and the configuration information of computing resources. The second dataset is used to store multiple image sequence samples. In one example, the user can store the second dataset and the model training task in OBS. The configuration information of computing resources can include the number of nodes, processor type, processor specifications, etc., which can be determined based on actual needs. For example, the processor type can be CPU or GPU; when the processor type is CPU, the processor specifications indicate the number of CPU cores and the size of memory; when the processor type is GPU, the processor specifications indicate the GPU's video memory size, the number of CPU cores, and the size of memory. In one possible scenario, if the number of nodes in the model training task is one, it means that one node is used for model training; if the number of nodes is multiple, multiple nodes are used to train the model, employing a distributed training approach. Terminal 210 can send model training tasks to management platform 221. Management platform 221 then iteratively trains the 3D reconstruction model 202 based on the model training tasks.

[0147] Figure 3a shows only a basic embodiment of the method of the present invention. Other preferred embodiments of the method can be obtained by making certain optimizations and extensions based on it.

[0148] Figure 4 is a flowchart illustrating the three-dimensional reconstruction method provided in an embodiment of the present invention. The three-dimensional reconstruction method provided in this embodiment can be applied to the management platform 221. Optionally, the three-dimensional reconstruction method can run on the management platform 221 in the form of software, such as a service or an application. As shown in Figure 4, the three-dimensional reconstruction method provided in this embodiment includes at least the following steps:

[0149] Step 401: The management platform 221 obtains the scene 3D model. The scene 3D model is the 3D reconstruction result of the scene, which includes the target static object.

[0150] The scene can include target static objects, which can be stationary objects such as toys, tables, chairs, and mobile phones that have not moved. The scene 3D model is a 3D representation of the scene. It should be noted that the construction process of the scene 3D model does not consider semantic information; therefore, the scene 3D model can be understood as an independent element. The scene 3D model can be composed of multiple objects. For example, if the scene consists of multiple objects on the ground, the scene 3D model will consist of the ground and the multiple objects on the ground. For example, as shown in Figure 3a, objects A and B have a contact surface. After 3D reconstruction, the scene 3D model is obtained. Objects A and B are treated as a whole. Since the base surfaces of objects A and B are not modeled, objects A and B are inseparable.

[0151] In an optional implementation of this embodiment, terminal 210 uploads a scene 3D model to management platform 221 via I / O interface 201.

[0152] In an optional implementation of this embodiment, the management platform 221 can acquire video, which is obtained by an image capturing device capturing a scene from all angles; multiple keyframes are extracted from the video for three-dimensional reconstruction to obtain a three-dimensional model of the scene.

[0153] Optionally, the management platform 221's acquisition of video may include: after the image capturing device acquires the video, it can store the video in a database; then, the terminal 210 can access the database to obtain the video, or the video's download address; subsequently, the terminal 210 can access the management platform 221 through the I / O interface 201. In one example, the terminal 210 can download the video locally and upload it to the management platform 221; in another example, the terminal 210 can upload the video's download address to the management platform 221; subsequently, the management platform 221 can download the video based on the video's download address, thereby acquiring the video.

[0154] Furthermore, after the management platform 221 obtains the video, it can extract key frames from the video and input the obtained key frames into the 3D reconstruction model 202. The 3D reconstruction model 202 performs 3D reconstruction of the scene and outputs a 3D model of the scene.

[0155] Step 402: The management platform 221 acquires M-frame images. The M-frame images are obtained by taking pictures of the scene from all angles. The M-frame images include target frame images. The target frame images carry annotation information. The annotation information is used to indicate the position of the target static object in the target frame image. M is a positive integer greater than or equal to 2.

[0156] In an optional implementation of this embodiment, terminal 210 can annotate at least a portion of the frames in the M-frame images to obtain annotation information, and then send the annotation information to management platform 221. In this embodiment, the frame image annotating the location of the target static object is called the target frame image, and the number of target frame images can be one or more. When annotating the frame images, terminal 210 can add some marks to the area where the target static object is located. These marks can be points, lines, graphics, etc. As shown in Figure 5a, for the second frame image in the M-frame images, dots are added to the area where object A is located in the second frame image. It should be noted that the M-frame images can come from video, and the video used to extract the M-frame images and the video used to construct the 3D model of the scene can be the same or different.

[0157] The following explains some possible annotation scenarios.

[0158] In one optional scenario, the management platform 221 has a built-in annotation tool 203. After the terminal 210 acquires the video, the user can use the annotation tool 203 through the I / O interface 201 to actively view the frame images in the video, select at least some frame images, and annotate the location of the target static object in the selected frame images. For example, some marks can be added to the area where the target static object is located. These marks can be points, lines, graphics, etc. Subsequently, after the user completes the annotation, they can upload the annotated video, or upload the download address of the annotated video to the management platform 221, so that the 3D reconstruction platform 310 can obtain the annotated video, extract the keyframes from the annotated video, and obtain M-frame images (showing the visible situation of the target static object from all angles). It should be noted that for the annotated video, the annotated frame images can be directly used as keyframes.

[0159] Considering that there are generally many frames in a video, if the user selects them manually, they may choose frames of low quality for annotation, affecting subsequent use. In another optional scenario, as shown in Figure 5b, the terminal 210 sends the video to the management platform 221. The management platform 221 extracts keyframes from the video to obtain M-frame images (showing the visible state of the target static object from all angles), and sends the M-frame images to the terminal 210. The terminal 210 selects at least a portion of the frame images, annotates the selected frame images, and sends the annotation information (used to indicate the position of the target static object in the selected frame images) to the management platform 221.

[0160] Step 403: The management platform 221 determines the contour information based on the M-frame image and annotation information. The contour information is used to indicate the contour of the target static object in the M-frame image.

[0161] In this embodiment of the invention, the contour information is the extraction result of each frame in the M-frame image. The extraction result is used to indicate the result of extracting the target static object. Optionally, the extraction result can be a mask image, which is used to indicate the area where the target static object is located. For example, the mask image can be the result of occluding the area outside the target static object in the target frame image. Optionally, in one example, as shown in Figure 6a, the mask image can be understood as a mask, where the pixels in the area where the target static object is located are 1 (or True), and the pixels in the area outside the target static object are 0 (or False). Optionally, in another example, as shown in Figure 6b, the mask image can be understood as the image after combining the image and the mask, where the pixels in the area where the target static object is located are 1 (or True), and the pixels in the area outside the target static object are 0 (or False). In this case, after obtaining the mask image by combining the mask and the image, operations can only be performed on the areas in the mask image where the pixels are 1 (or True). It should be noted that the above 6a and Figure 6b are merely examples and do not constitute specific limitations. The extraction results can be designed according to the actual situation.

[0162] In an optional implementation of this embodiment, the target frame image is one or more images. For example, the target frame image can be a frame image of the front view of a target static object. For each target frame image, the management platform 221 extracts the target static object from the target frame image based on its annotation information, obtaining the extraction result of the target frame image (which can be called the first extraction result for ease of description and distinction). The first extraction result indicates the region of the target static object in the target frame image. After obtaining the first extraction result of the target frame image, the platform then analyzes the position of the target static object in each of the other frames in the M-frame image, based on the first extraction results of all target frame images as a reference, thereby extracting the target static object in each of the other frames in the M-frame image, obtaining the extraction result of each of the other frames in the M-frame image (which can be called the second extraction result for ease of description and distinction). The extraction results are used to indicate the region of the target static object in the frame image. Finally, based on the first extraction results of all target frame images and the second extraction results of each other frame image in the M-frame images, contour information is obtained. For example, the first extraction results of all target frame images and the second extraction results of each other frame image in the M-frame images are used as contour information. Alternatively, the first extraction results of all target frame images and the second extraction results of each other frame image in the M-frame images are fused together to consider the contour changes of the target static object from a global perspective, thereby correcting the extraction results of each frame image in the M-frame images. The corrected extraction results of each frame image in the M-frame images are used as mask information.

[0163] Optionally, as shown in Figure 6c, in the process of extracting the target static object from each frame of the M-frame images, in addition to considering the first extraction results of all target frame images, the shooting pose (position and orientation of the shooting point) and the scene perspective (used to indicate the angle of the scene) of each frame of the M-frame images can also be considered, thereby improving the accuracy of extracting the target static object; wherein, the shooting pose of each frame of the M-frame images can be analyzed to analyze the motion trajectory of the scene being shot around, and then to analyze the changes in the region of the target static object in each frame of the M-frame images; the scene perspective of each frame of the M-frame images can be analyzed to analyze the contour changes of the target static object in each frame of the M-frame images.

[0164] Determining the shooting pose and scene view of each frame in the M-frame images can include: camera calibration can be performed based on the M-frame images to obtain the camera extrinsic parameters of each frame in the M-frame images; based on the camera extrinsic parameters of each frame in the M-frame images, the shooting pose and scene view of each frame in the M-frame images can be determined.

[0165] For example, when the camera extrinsic parameters of a frame image are used to indicate the pose motion from the previous frame image to the current frame image, as shown in Figure 6d, the camera calibration method can be as follows: the camera extrinsic parameter 1 of the first frame image in M ​​frames is set to 0. Subsequently, for the i-th frame image (a positive integer greater than or equal to 2 and less than or equal to M) after the second frame in M ​​frames, feature matching is performed between the i-th frame image and the (i-1)-th frame image to analyze the positional changes of the same objects and obtain the camera extrinsic parameter i of the i-th frame image. Correspondingly, determining the shooting pose of each frame in the M-frame images can include: taking the camera extrinsic parameters of a certain frame in the M-frame images (for ease of description, it can be called the starting frame image), such as the first frame image, as the starting point, i.e., the shooting pose is 0. Considering that the camera extrinsic parameters of the starting frame image are the starting point, i.e., the shooting pose is 0, then taking the starting frame image, such as the first frame image, as the starting point, according to a certain direction, such as the time order of the frame images, the next frame image is obtained. The shooting pose of the starting frame image and the camera extrinsic parameters of the next frame image are added together to obtain the shooting pose of the next frame image. Subsequently, according to a certain direction, such as the time order of the frame images, each of the other frames in the M-frame images is traversed, and the shooting pose of each of the other frames in the M-frame images is obtained sequentially. For example, for each frame image traversed sequentially (for ease of description and distinction, it can be called the i-th frame image), the shooting pose of the (i-1)-th frame image and the camera extrinsic parameters of the i-th frame image are added together to obtain the shooting pose of the i-th frame image.

[0166] For example, when the camera extrinsic parameters of a frame image are used to indicate the pose motion of the frame image and the starting frame image, the camera calibration method can be as follows: Assuming the starting frame image is the first frame image, the camera extrinsic parameter 1 of the first frame image in the M-frame images is set to 0. Subsequently, for the i-th (a positive integer greater than or equal to 2 and less than or equal to M) frame image after the second frame in the M-frame images, feature matching is performed between the i-th frame image and the first frame image to analyze the position changes of the same objects in the images and obtain the shooting pose of the i-th frame image.

[0167] For example, determining the scene perspective of each frame in M ​​frames may include: taking a frame image with a certain perspective, such as a frontal view (which can be called the starting frame image for ease of description), as 0 degrees, analyzing the angle between the shooting point of each other frame image and the shooting point of the starting frame image to obtain the scene perspective of the other frames in M ​​frames; for example, for each other frame image in M ​​frames, based on the shooting pose of the frame image and the starting frame image, calculating the angle between the shooting point of the frame image and the starting frame image and the target static object, thereby obtaining the scene perspective of the frame image.

[0168] Correspondingly, the management platform 221, based on the first extraction results of all target frame images, can extract the target static object in each of the other frames in the M-frame images by: when there is only one target frame image, using the first extraction result as a starting point, combining the shooting pose and scene view of each frame in the M-frame images, tracking the target static object in each of the other frames in the M-frame images, thereby obtaining the second extraction result for each of the other frames in the M-frame images; when there are multiple target frame images, for each first extraction result, using the first extraction result as a starting point, combining the shooting pose and scene view of each frame in the M-frame images, tracking the target static object in each of the other frames in the M-frame images, thereby obtaining the second extraction result for each of the other frames in the M-frame images; The shooting pose of each frame in the M-frame image and the scene viewpoint of each frame in the M-frame image are used to track the target static object in each other frame in the M-frame image, thereby obtaining the extraction results of each other frame in the M-frame image (which can be called candidate extraction results for ease of description and distinction); then, for each other frame in the M-frame image, a second extraction result is determined based on the multiple candidate extraction results of that frame image. For example, the better performing candidate result is selected from multiple candidate results as the second extraction result, or multiple candidate results are fused, such as added together, to obtain the second extraction result.

[0169] The tracking of the target static object can include: for the i-th frame image in M ​​frames (used to indicate any other frame image besides the target frame image), subtracting the shooting pose of the i-th frame image from the shooting pose of the first extraction result, or, based on the shooting pose of each frame image from the target frame image corresponding to the first extraction result to the shooting pose of the i-th frame image, the pose motion from the shooting point of the first extraction result to the shooting point of the i-th frame image can be determined; based on the pose motion from the shooting point of the first extraction result to the shooting point of the i-th frame image and the first extraction result, the position of the target static image in the i-th frame image can be estimated; based on the first extraction result, the scene view of the first extraction result, and the scene view of the i-th frame image, the shape of the target static image in the i-th frame image can be estimated; based on the position and shape of the target static image in the i-th frame image, the extraction of the target static object is assisted, thereby obtaining the second extraction result of the i-th frame image.

[0170] It should be noted that the method for determining the second extraction result and the target static object tracking here is only an example and does not constitute a specific limitation. The method for determining the second extraction result can be reasonably designed according to actual needs.

[0171] It should be noted that the method of extracting static objects by considering the shooting pose and scene view of each frame in the M-frame images is only an example. In practical applications, you can choose either the shooting pose or the scene view of each frame in the M-frame images to extract static objects.

[0172] Optionally, before extracting the target static object from the target frame image, in order to facilitate the extraction of the outline of the target static object, the management platform 221 can roughly delete some interference information in the M frame image in advance, retain the approximate area of ​​the target static object, thereby reducing the influence of the area outside the target static object on the extraction of the target static object.

[0173] Optionally, in one example, the management platform 221 can determine the location range of the target static object in the M-frame image based on the annotation information, such as a circular area; and remove the content of the M-frame image outside the location range.

[0174] For example, the area where the target static object is located in the M-frame images does not fluctuate significantly. Therefore, the management platform 221 can determine the position range of the target static object in the M-frame images based on the annotation information, which may include: determining the center point position of the target static object in any target frame image based on the annotation information; determining the panning radius based on the shooting pose of each frame in the M-frame images (used to indicate the position and pose of the shooting point); the panning radius indicates the distance from the center point position to the shooting point position, such as the minimum distance; and determining the position range of the target static object in the M-frame images based on the center point position and the panning radius, which is the same in each frame of the M-frame images. Optionally, the position range is a circular area constructed with the center point position as the origin and the panning radius as the radius, and the panning radius can be appropriately reduced, for example, reduced by 20%.

[0175] The above method for determining the position range of the target static object in the M-frame images is merely an example. When the area of ​​the target static object in the M-frame images fluctuates significantly, the management platform 221 can analyze the pose movement of the center position point in each other frame of the M-frame images, based on the shooting pose of each frame in the M-frame images, taking the center position point of the target frame image as the starting point, thereby obtaining the center position point of each frame in the M-frame images. For example, for each other frame in the M-frame images, based on the shooting pose of that frame image and the shooting pose of the target frame image, the movement of the center position point of the target frame image to that frame image is determined (e.g., the movement distance in the x-direction and the movement distance in the y-direction). The center position point of the target frame image moves according to the movement, thus obtaining the center position point of that frame image. Subsequently, for each frame in the M-frame images, the position range of that frame image is determined based on the center position of that frame image and the panoramic shooting radius.

[0176] In an optional implementation of this embodiment, the management platform 221 determines the contour information based on the M-frame images and annotation information, which may include: inputting the M-frame images and annotation information into the image segmentation model 204, and the image segmentation model 204 outputting the contour information.

[0177] Optionally, the image segmentation model 204 is used to extract the extraction results of each frame in the M-frame images based on each target frame image, annotation information, and each other frame image in the M-frame images, and the extraction results of each frame in the M-frame images are used as contour information.

[0178] Optionally, the image segmentation model 204 is used to extract the target static object in each target frame image based on the annotation information of the target frame image to obtain the first extraction result of the target frame image. Based on all the first extraction results, the target static object in each other frame image in the M frames is extracted to obtain the second extraction result of each other frame image in the M frames. The first extraction result and the second extraction result are fused to obtain the extraction result of each frame image in the M frames as contour information.

[0179] In one optional example, as shown in Figure 3d, the image segmentation model 204 may include an extraction network 2041 and a contour tracking network 2042. It should be noted that the extraction network 2041 and the contour tracking network 2042 are merely examples of the image segmentation model 204 and do not constitute a specific limitation. The image segmentation model 204 can be designed reasonably according to the actual situation.

[0180] The extraction network 2041 can be used to extract the target static object in each target frame image based on the annotation information of the target frame image, and obtain the first extraction result of the target frame image. The extraction network 2041 can be a neural network model, such as the currently open source SAM.

[0181] The contour tracking network 2042 can be used to track the contours of objects, such as static targets. Optionally, in one example, the contour tracking network 2042 extracts the static target from each of the other frames in the M-frame image set based on all the first extraction results, obtaining second extraction results for each of the other frames in the M-frame image set. The first and second extraction results are then fused to obtain the extraction result for each frame in the M-frame image set as contour information, thereby improving the accuracy of the extraction results. Considering that the M-frame images involve temporal information, the contour tracking network 2042 can be a time-recurrent network, such as a Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM) network, or a Gated Recurrent Unit (GRU).

[0182] Optionally, the contour tracking network 2042 may include N (positive integers greater than or equal to 1) hidden layers and an output layer. Each of the N hidden layers includes M hidden nodes at time points M, incrementing from 1 to M. The M hidden nodes are connected sequentially. When N is greater than or equal to 2, the M hidden nodes of the i-th (positive integers greater than or equal to 1 and less than or equal to N) layer are connected one-to-one with the M hidden nodes of the (i-1)-th layer. For example, the hidden nodes can be recurrent neural networks (RNN), long short-term memory networks (LSTM), and gated recurrent units (GRU). For the first hidden layer among the N hidden layers, such as the first layer, the M hidden nodes in that hidden layer... The input to the hidden nodes are all the first extraction results, each frame image in the M frames other than the target frame image, the scene view and / or shooting pose of each frame image in the M frames; the output layer connects to the last hidden layer in the N hidden layers, such as the Nth hidden layer, and is used to obtain the extraction results of each frame image in the M frames based on the output of each hidden node in the connected Nth hidden layer; optionally, in one example, the output layer may include M output nodes, each of the M output nodes is connected to the M hidden nodes in the Nth hidden layer, and the i-th (a positive integer greater than or equal to 1 and less than or equal to M) output node of the M output nodes outputs the extraction result i of the i-th frame image.

[0183] Additionally, when N is greater than or equal to 2, in one optional example, the computation direction of the N hidden layers can be unidirectional. The computation direction can be understood as the output direction of the M hidden nodes, such as hidden node 1 → hidden node 2 → … → hidden node M. That is, the computation direction of each hidden layer in the multiple hidden layers is the same. In another optional example, the computation direction of the multiple hidden layers can be bidirectional, that is, the computation direction of some hidden layers in the multiple hidden layers is opposite to the computation direction of other hidden layers.

[0184] In one example, as shown in Figure 7a, the contour tracking network 2042 operates in a unidirectional direction, comprising hidden layer 0, hidden layer 1, and an output layer. The computation directions of hidden layer 0 and hidden layer 1 are both hidden node 1 → hidden node 2 → ... → hidden node M. Assuming the target frame image has only one image, which is the i-th frame, the input to hidden node 1 in hidden layer 0 is the first frame image, the shooting pose 1, the scene viewpoint 1, ..., and the input to hidden node i is the first extraction result of the i-th frame image, the shooting pose 1, the scene viewpoint 1, ... The output of hidden node i is the camera pose i, the scene view i, and the output of hidden node i-1. The input of hidden node i+1 is the image of frame i+1, the camera pose i+1, the scene view i+1, and the output of hidden node i, ...; Hidden layer 1 and hidden layer 0 are similar and will not be described again; The output layer outputs the extraction result 1 of the first frame image, the extraction result 2 of the second frame image, ..., the extraction result i of the i frame image, the extraction result i+1 of the i+1 frame image, ..., the extraction result M of the M frame image.

[0185] In another example, as shown in Figure 7b, the contour tracking network 2042 has a bidirectional computation direction, including hidden layer 0 and hidden layer 1. The computation direction of hidden layer 0 is hidden node 1 → hidden node 2 → … → hidden node M, and the computation direction of hidden layer 1 is hidden node M → hidden node M-1 → … → hidden node 2 → hidden node 1. Assuming that the target frame image has only one image, which is the i-th frame image, the input of hidden node 1 in hidden layer 0 is the first frame image, the shooting pose 1, the scene view 1, …, the input of hidden node i is the first extraction result of the i-th frame image, the shooting pose i, the scene view i, and the output of hidden node i-1, and the input of hidden node i+1 is the (i+1)-th frame image and the shooting pose. The output of hidden node i+1, scene view i+1, and hidden node i, ...; The input of hidden node M in hidden layer 1 is the image of frame M and the output of hidden node M in hidden layer 0. The input of hidden node M-1 is the image of frame M-1 and the output of hidden node M in hidden layer 1, ...; The input of hidden node i+1 is the image of frame i+1 and the output of hidden node i+2. The input of hidden node i is the extraction result of frame i of the image of frame i and the output of hidden node i+1, ...; The output layer outputs the extraction result 1 of frame 1, the extraction result 2 of frame 2, ..., the extraction result i of frame i, the extraction result i+1 of frame i+1, ..., the extraction result M of frame M.

[0186] The training of the extraction network 2041 and the contour tracking network 2042 is described below. For details, please refer to the description of the 3D reconstruction model above. The differences are as follows:

[0187] First, the training of the extraction network 2041 will be described. For details, please refer to the description of the 3D reconstruction model above; the differences are as follows:

[0188] The training dataset includes image samples and their labels, where the labels represent the extracted results, such as mask images. During the training process, for each image sample, the image sample is input into the extraction network 2041, and the extraction result output by the extraction network 2041 is obtained. Based on the error between the extraction result and the label, the error corresponding to the image sample is obtained. Based on the errors corresponding to multiple image samples, the model parameters of the extraction network 2041 are updated. Then, the updated extraction network 2041 replaces the previous extraction network 2041, and the next iteration is performed until the training completion condition is met.

[0189] The training of the contour tracking network 2042 will be described below. For details, please refer to the description of the 3D reconstruction model above; the differences are as follows:

[0190] The training dataset includes image sequence samples and labels for the image sequences. The labels are the extracted result sequences corresponding to the image sequence samples, such as mask image sequences. Some frames in the image sequence samples are the extracted results, such as mask images. In some other possible cases, each image in the image sequence samples has a shooting pose and / or scene viewpoint.

[0191] In this model, the labels for image sequence samples are determined through data annotation. One feasible implementation is for the AI ​​infrastructure development platform to have a built-in annotation tool for users to perform manual annotation. Different annotation tasks can be performed using different annotation methods. Furthermore, after a certain number of samples have been manually annotated, the AI ​​infrastructure development platform can automatically train a built-in pre-annotated model based on these manually annotated samples. Once a certain level of accuracy is achieved, the pre-annotated model can perform inference on the remaining unannotated training samples (i.e., intelligent annotation) and evaluate the annotation results. Intelligently annotated samples with high accuracy can be directly stored in a second dataset, such as OBS, for subsequent model training. For difficult examples with low accuracy, users can manually verify or correct the results through display, and the manually verified training samples are then stored in the second dataset, such as OBS.

[0192] In the specific training process, for each image sequence sample, after inputting the image sequence sample into the image segmentation model, the extraction result sequence output by the image segmentation model is obtained. Based on the error between the extraction result sequence and the label, the error corresponding to the image sequence sample is obtained. Based on the errors corresponding to multiple image sequence samples, the model parameters of the image segmentation model are updated. Then, the updated image segmentation model replaces the previous image segmentation model, and the next iteration is performed until the training completion condition is met.

[0193] The above embodiments are merely examples. In other possible embodiments, there are M target frame images, which can be M frame images. In specific implementation, the management platform 221 extracts the target static object from each frame of the M frame images based on the annotation information of each frame, and obtains the contour information. Optionally, the management platform 221 can extract the target static object from each frame of the M frame images through the extraction network 2041.

[0194] Step 404: Based on the contour information, the management platform 221 extracts the target static object from the scene 3D model to obtain the 3D model of the target static object.

[0195] In an optional implementation of this embodiment, as shown in Figure 8a, the management platform 221 extracts the target static object from the scene's 3D model based on contour information. Obtaining the 3D model of the target static object may include: the management platform 221 reconstructing the 3D contour of the target static object based on contour information, such as the extraction results of each frame in the M-frame images, or a mask image; and then, based on the 3D contour of the target static object, extracting the 3D model of the target static object from the scene's 3D model. It should be noted that, as shown in Figure 8b, during the process of reconstructing the 3D contour of the target static object, the scene viewpoint of each frame in the M-frame images can be considered. According to the scene viewpoint of each frame in the M-frame images, and the contour of the target static object in the M-frame images indicated by the contour information, the contour of the target frame image under different scene viewpoints is obtained. Based on the differences in different scene viewpoints, a reasonable fitting method is analyzed to fit the contour of the target frame image under different scene viewpoints, thus reconstructing the 3D contour of the target static object.

[0196] Optionally, the management platform 221 may extract the 3D model of the target static object from the scene 3D model based on the 3D contour of the target static object by: matching the 3D contour of the scene 3D model and the 3D contour of the target static object to obtain a contour region in the scene 3D model that matches the 3D contour of the target static object; then, the management platform 221 may modify the size of the 3D contour of the target static object based on the size of the matched contour region to obtain the modified 3D contour. For example, the management platform 221 may determine a correction distance based on the matched contour region and the 3D contour of the target static object, where the correction distance indicates the distance between the same position point of the matched contour region and the 3D contour of the target static object, and expand or shrink the 3D contour of the target static object based on the correction distance; subsequently, the modified 3D contour and the matched contour region are aligned, and the area outside the modified 3D contour in the scene 3D model is deleted.

[0197] Optionally, in one example, to reduce the difficulty of matching the 3D model of the scene with the 3D contour of the target static object, the management platform 221 can provide the 3D model of the scene to the terminal 210. The terminal 210 can mark the position of the target static object, for example, by randomly marking a point at the location of the target static object, or by marking the center point of the target static object in the 3D model of the scene. Subsequently, the management platform 221 matches the marked area of ​​the 3D model of the scene with the 3D contour of the target static object.

[0198] In one optional implementation of this embodiment, the management platform 221 extracts the target static object from the scene 3D model based on contour information to obtain the 3D model of the target static object. This process may include: determining the coordinate system of the scene 3D model, the coordinate system being constructed based on the target shooting point, for example, using the target shooting point (described by shooting pose) as the origin, the target shooting point being the shooting point of any frame in the M-frame images, for example, the target shooting point being the shooting point of the first frame in the M-frame images; then, based on the M-frame images and the target shooting point, the shooting process of the M-frame images can be reconstructed in the scene 3D model to determine the image region of each frame in the M-frame images in the scene 3D model; subsequently, based on the contour information and the image region of each frame in the M-frame images, the target region where the target static object is located in the scene 3D model is determined, and the content of the target region is extracted to obtain the 3D model of the target static object.

[0199] Optionally, as shown in Figure 9a, based on the M-frame images and the target shooting point, reconstructing the shooting process of the M-frame images in the scene 3D model and determining the image region of each frame in the M-frame images in the scene 3D model may include: taking the shooting pose of the target shooting point as the starting point, and determining the pose motion of each frame in the M-frame images relative to the target shooting point based on the shooting pose of each frame in the M-frame images; then, obtaining the shooting parameters of the M-frame images (e.g., camera intrinsic parameters, image size, etc.). The method for determining the camera intrinsic parameters can refer to the method for determining the camera extrinsic parameters shown in Figure 6d. By performing camera calibration on the (i-1)th frame image and the i-th frame image, the camera extrinsic parameter i and the camera intrinsic parameter i can be obtained. It should be noted that this applies when the M-frame images are captured by the same image capturing device. If the camera intrinsics of the M frames are the same, then camera intrinsics 2 to M can be fused to obtain the camera intrinsics. Then, for the frame image corresponding to the target shooting point in the M frames, assuming it is the i-th frame image, the virtual camera takes a picture of the scene 3D model at the target shooting point (the origin of the scene 3D model) according to shooting parameters such as camera intrinsics and image size, to obtain the image region of the i-th frame image in the scene 3D model. For the j-th frame image other than the i-th frame image in the M frames, the virtual camera moves based on the pose motion of the j-th frame image relative to the target shooting point. After the movement is completed, the virtual camera takes a picture of the scene 3D model according to shooting parameters such as camera intrinsics and image size, to obtain the image region of the j-th frame image in the scene 3D model. The method shown in Figure 9a is merely an example and does not constitute a specific limitation. In some other optional implementations of the virtual camera reconstructing the shooting process, the virtual camera takes the target shooting point as the starting point and the i-th frame image corresponding to the target shooting point as the starting point. Following a certain direction, for the next frame image of the i-th frame image: the i+1-th frame image, based on the shooting pose of the i-th frame image and the i+1-th frame image, the pose motion of the shooting points of the two adjacent frames is determined. The virtual camera moves based on the pose motion. After the movement is completed, the virtual camera takes pictures of the scene 3D model according to shooting parameters such as camera intrinsic parameters and image size, and obtains the image area of ​​the i+1-th frame image in the scene 3D model. The virtual camera is moved continuously in a similar manner to take pictures, thereby obtaining the image area of ​​each frame image other than the i-th frame image in the scene 3D model in the M-frame images.

[0200] Optionally, as shown in Figure 9b, after reconstructing the process of capturing M frames of images in the scene's 3D model, the image region of each frame in the M frames can be obtained in the scene's 3D model. If the contour information is the mask image of each frame in the M frames, then, based on the contour information and the image region of each frame in the M frames, determining the target region where the target static object is located in the scene's 3D model and extracting the content of the target region can include: for each frame in the M frames, aligning the mask image of that frame with the image region of that frame in the scene's 3D model to obtain the target region where the target static object, such as object A, is located and the occlusion region in the image region of the scene's 3D model. Subsequently, removing the occlusion region in the scene's 3D model will extract the content of the target region, thus obtaining the 3D model of the target static object, such as object A.

[0201] In this scheme, the outlines of a specified object at different angles under the surround shooting are extracted from multiple frames of images obtained by surround shooting. Based on the outlines of the specified object at different angles under the surround shooting, the 3D model of the specified object is extracted from the 3D model of the scene.

[0202] Figure 3a shows only a basic embodiment of the method of the present invention. Other preferred embodiments of the method can be obtained by making certain optimizations and extensions based on it.

[0203] In one embodiment, this embodiment provides a more detailed description and some optimization of the 3D reconstruction process based on the aforementioned embodiments. As shown in Figure 10a, terminal 210 sends video to management platform 221. Management platform 221 extracts keyframes from the video to obtain M-frame images. Based on the M-frame images, it performs 3D reconstruction to obtain a 3D model of the scene and sends the M-frame images back to terminal 210. Terminal 210 displays the M-frame images, selects frames to annotate the location of the target static object, obtains the annotation information, and sends it to management platform 221. Based on the M-frame images and the annotation information, management platform 210 obtains an M-frame mask image of the target static object. Based on the M-frame mask image, it segments the 3D model of the scene to obtain a 3D model of the target static object. This is merely an overview of the method; for details, please refer to Figure 10b and the description of the steps in Figure 10b. As shown in Figure 10b, the method in this embodiment includes at least the following steps:

[0204] Step 1001: Terminal 210 sends a video to management platform 221. The video is obtained by shooting around the scene, and the scene includes the target static object.

[0205] In one optional implementation of this embodiment, terminal 210 can access management platform 221 to directly upload videos.

[0206] Step 1002: The management platform 221 extracts keyframes from the video to obtain M-frame images.

[0207] In an optional implementation of this embodiment, the management platform 221 can compare adjacent images in the video, remove redundant images to obtain key frame images, and then calculate the clarity of the key frame images to select the frame images with higher clarity.

[0208] Step 1003: The management platform 221 performs 3D reconstruction based on the M-frame images to obtain a 3D model of the scene.

[0209] In one optional implementation of this embodiment, the management platform 221 inputs M-frame images into the 3D reconstruction model 302. The 3D reconstruction model 302 performs 3D reconstruction of the scene with the goal of restoring the M-frame images and outputs a 3D model of the scene. The 3D reconstruction of the scene with the goal of restoring the M-frame images can be performed using 3DGS technology.

[0210] Step 1004: The management platform 221 sends an M-frame image to the terminal 210.

[0211] Step 1005: Terminal 210 obtains annotation information based on user operation information. The annotation information is used to indicate the position of the target static object in the target frame of the M-frame image.

[0212] In an optional implementation of this embodiment, the management platform 221 may have a built-in annotation tool 203. After the terminal 210 acquires M frames of images, the user can use the annotation tool 203 to select at least a portion of the frames in the M frames and annotate the location of the target static object in the selected frames. For example, the user can add some marks on the area where the target static object is located; these marks can be points, lines, graphics, etc. Subsequently, after the user completes the annotation, the annotation information (used to indicate the location marked by the user on the frame images) can be uploaded to the management platform 221.

[0213] Step 1006: Terminal 210 sends annotation information to management platform 221.

[0214] Step 1007: The management platform 221 determines the mask image of each frame in the M-frame images based on the M-frame images and annotation information. The mask image is used to indicate the outline of the target static object.

[0215] For details, please refer to the description of step 403 above. The difference is that the contour information is the mask image of each frame in the M-frame image, which will not be repeated here.

[0216] Step 1008: The management platform 221 extracts the target static object from the scene 3D model based on the mask image of each frame in the M-frame image, and obtains the 3D model of the target static object.

[0217] For details, please refer to Figures 8a to 9b, as well as the description of step 404 above. The difference is that the extraction result is a mask image, which will not be repeated here.

[0218] In this scheme, a 3D model of the scene is constructed and the contours of a specified object at different angles under the surround shooting are extracted from multiple frames of images obtained by surround shooting. Based on the contours of the specified object at different angles under the surround shooting, the 3D model of the specified object is extracted from the 3D model of the scene.

[0219] As shown in Figure 11, the present invention also provides a three-dimensional reconstruction device, which includes:

[0220] The model acquisition module is used to acquire a 3D model of the scene, which is a 3D representation of the scene, and the scene includes target static objects;

[0221] The image acquisition module is used to acquire multiple frames of images, which are obtained by taking panoramic photos of the scene. The multiple frames of images include target frame images, which carry annotation information to indicate the position of the target static object.

[0222] The contour analysis module is used to determine contour information based on the multi-frame images and the annotation information, wherein the contour information is used to indicate the contour of the target static object in the multi-frame images;

[0223] An extraction module is used to extract the target static object from the scene 3D model based on the contour information, and obtain the 3D model of the target static object.

[0224] The model acquisition module, image acquisition module, contour analysis module, and extraction module can all be implemented in software or hardware. For example, the implementation of the model acquisition module will be described below. Similarly, the implementation methods of the image acquisition module, contour analysis module, and extraction module can refer to the implementation method of the model acquisition module.

[0225] As an example of a software functional unit, the model acquisition module may include code running on a compute instance. A compute instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned compute instance may be one or more. For example, the model acquisition module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0226] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0227] As an example of a hardware functional unit, a model acquisition module may include at least one computing device, such as a server. Alternatively, the model acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0228] The model acquisition module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0229] It should be noted that, in other embodiments, the model acquisition module can be used to execute any step in the above-described 3D reconstruction method, such as any step shown in Figure 4 or Figure 10b; the image acquisition module can be used to execute any step in the above-described 3D reconstruction method, such as any step shown in Figure 4 or Figure 10b; the contour analysis module can be used to execute any step in the above-described 3D reconstruction method, such as any step shown in Figure 4 or Figure 10b; and the extraction module can be used to execute any step in the above-described 3D reconstruction method, such as any step shown in Figure 4 or Figure 10b. The steps implemented by the model acquisition module, image acquisition module, contour analysis module, and extraction module can be specified as needed. The model acquisition module, image acquisition module, contour analysis module, and extraction module respectively implement the above-described 3D reconstruction method, such as the method shown in Figure 4 or Figure 10b, to realize all the functions of the 3D reconstruction device.

[0230] The present invention also provides a computing device 1200. As shown in FIG12, the computing device 1200 includes: a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 may be a server or a terminal device. It should be understood that the present invention does not limit the number of processors and memories in the computing device 1200.

[0231] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 12, but this does not imply that there is only one bus or one type of bus. Bus 1202 can include pathways for transmitting information between various components of computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0232] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0233] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0234] The memory 1206 stores executable program code, and the processor 1204 executes this executable program code to implement the functions of the aforementioned model acquisition module, image acquisition module, contour analysis module, and extraction module, thereby realizing the above-mentioned three-dimensional reconstruction method, as shown in Figure 4 or Figure 10b. That is, the memory 1206 stores instructions for executing the above-mentioned three-dimensional reconstruction method, such as the method shown in Figure 4 or Figure 10b.

[0235] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0236] This invention also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0237] As shown in Figure 13, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for performing the three-dimensional reconstruction method described above, such as the instructions shown in Figure 4 or Figure 10b.

[0238] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the three-dimensional reconstruction method shown above, such as partial instructions for the method shown in FIG4 or FIG10b. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the three-dimensional reconstruction method shown above, such as the instructions for the method shown in FIG4 or FIG10b.

[0239] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned 3D reconstruction device. That is, the instructions stored in the memory 1206 of different computing devices 1200 can implement the functions of one or more modules among the model acquisition module, image acquisition module, contour analysis module, and extraction module.

[0240] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 illustrates one possible implementation. As shown in Figure 14, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the model acquisition module. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the image acquisition module, the contour analysis module, and the extraction module.

[0241] The connection method between the computing device clusters shown in Figure 14 can be such that, considering the need to store the scene 3D model for the 3D reconstruction method provided by the present invention, the functions implemented by the image acquisition module, contour analysis module and extraction module are delegated to the computing device 1200B for execution.

[0242] It should be understood that the functions of computing device 1200A shown in Figure 14 can also be performed by multiple computing devices 1200. Similarly, the functions of computing device 1200B can also be performed by multiple computing devices 1200.

[0243] This invention also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it enables the at least one computing device to perform the aforementioned three-dimensional reconstruction method, such as the method shown in FIG4 or FIG10b.

[0244] This invention also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned three-dimensional reconstruction method, such as the method shown in Figure 4 or Figure 10b.

[0245] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0246] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0247] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of the various embodiments of the present disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of the present disclosure to the necessity of employing the specific details described above.

[0248] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0249] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0250] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

[0251] It is understood that the various numerical designations used in the embodiments of the present invention are merely for descriptive convenience and are not intended to limit the scope of the embodiments of the present invention.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Obtain a 3D model of the scene, which is the 3D reconstruction result of the scene, and the scene includes target static objects; Acquire multiple frames of images, which are obtained by taking panoramic photos of the scene. The multiple frames of images include target frames, which carry annotation information to indicate the position of the target static object in the target frames. Based on the multi-frame images and the annotation information, contour information is determined, and the contour information is used to indicate the contour of the target static object in the multi-frame images; Based on the contour information, the target static object is extracted from the scene 3D model to obtain the 3D model of the target static object.

2. The method according to claim 1, characterized in that, The step of determining contour information based on the multi-frame images and the annotation information includes: Based on the annotation information, the target static object in the target frame image is extracted to obtain the first extraction result of the target frame image; Based on the first extraction result of the target frame image, the target static object is extracted from each of the other frames in the multi-frame image, and a second extraction result of each of the other frames in the multi-frame image is obtained. Based on the first extraction result of the target frame image and the second extraction result of each of the other frames in the multi-frame image, contour information is obtained.

3. The method according to claim 2, characterized in that, Before determining the contour information based on the multi-frame images and the annotation information, the method further includes: Based on the multi-frame images and the annotation information, the location range of the target static object in the multi-frame images is determined; Remove the content of the multi-frame image outside the specified location range.

4. The method according to any one of claims 2 or 3, characterized in that, The target frame image is used to indicate the front of the target static object.

5. The method according to any one of claims 1 to 4, characterized in that, The step of extracting the target static object from the scene 3D model based on the contour information to obtain the 3D model of the target static object includes: The coordinate system of the three-dimensional model of the scene is determined. The coordinate system is constructed based on the target shooting point, which is the shooting point of any frame in the multi-frame images. Based on the multi-frame images and the target shooting point, the shooting process of the multi-frame images is reconstructed, and the image area of ​​each frame in the multi-frame images in the scene 3D model is determined; Based on the contour information and the image region, the target region where the target static object is located in the 3D model of the scene is determined, the content of the target region is extracted, and the 3D model of the target static object is obtained.

6. The method according to claim 5, characterized in that, The contour information is a mask image of each frame in the multi-frame images, and the mask image is used to indicate the area where the target static object is located; The step of determining the target region of the target static object in the scene 3D model based on the contour information and the image region includes: Align the mask image of each frame in the multi-frame image with the image region to determine the target region where the target static object is located in the image region.

7. The method according to any one of claims 1 to 6, characterized in that, The acquisition of the scene 3D model includes: The goal is to reconstruct the multi-frame images to obtain a 3D model of the scene.

8. The method according to any one of claims 1 to 7, characterized in that, The acquisition of multiple frames of images includes: Acquire video, which is obtained by shooting the scene from all angles; Keyframes are extracted from the video to obtain multiple frames of images.

9. A three-dimensional reconstruction device, characterized in that, The device includes: The model acquisition module is used to acquire a 3D model of the scene, which is a 3D representation of the scene, and the scene includes target static objects; The image acquisition module is used to acquire multiple frames of images, which are obtained by taking panoramic photos of the scene. The multiple frames of images include target frame images, which carry annotation information to indicate the position of the target static object. The contour analysis module is used to determine contour information based on the multi-frame images and the annotation information, wherein the contour information is used to indicate the contour of the target static object in the multi-frame images; An extraction module is used to extract the target static object from the scene 3D model based on the contour information, and obtain the 3D model of the target static object.

10. The method according to claim 9, characterized in that, The contour analysis module is used to extract the target static object in the target frame image based on the annotation information, and obtain the first extraction result of the target frame image; Based on the first extraction result of the target frame image, the target static object is extracted from each of the other frames in the multi-frame image, and a second extraction result of each of the other frames in the multi-frame image is obtained. Based on the first extraction result of the target frame image and the second extraction result of each of the other frames in the multi-frame image, contour information is obtained.

11. The method according to claim 10, characterized in that, The device further includes: The removal module is used to determine the position range of the target static object in the multi-frame images based on the multi-frame images and the annotation information; and to remove the content in the multi-frame images outside the position range.

12. The method according to claim 10 or 11, characterized in that, The target frame image is used to indicate the front view of the target static object.

13. The method according to any one of claims 9 to 12, characterized in that, The extraction module is used to determine the coordinate system of the scene 3D model. The coordinate system is constructed based on the target shooting point, which is the shooting point of any frame in the multi-frame images. Based on the multi-frame images and the target shooting point, the shooting process of the multi-frame images is reconstructed, and the image area of ​​each frame in the multi-frame images in the scene 3D model is determined; Based on the contour information and the image region, the target region where the target static object is located in the 3D model of the scene is determined, the content of the target region is extracted, and the 3D model of the target static object is obtained.

14. The method according to claim 13, characterized in that, The contour information is a mask image of each frame in the multi-frame images, and the mask image is used to indicate the area where the target static object is located; the extraction module is used to align the mask image of each frame in the multi-frame images with the image area to determine the target area where the target static object is located in the image area.

15. The method according to any one of claims 9 to 14, characterized in that, The model acquisition module is used to perform three-dimensional reconstruction with the goal of restoring the multi-frame images to obtain a three-dimensional model of the scene.

16. The method according to any one of claims 9 to 15, characterized in that, The image acquisition module is used to acquire video, which is obtained by shooting around the scene; keyframes are extracted from the video to obtain multiple frames of images.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in claims 1 to 8.

Images