41 results about "Structure from motion" patented technology

The invention relates to a true orthophoto generation method based on a three-dimensional Gaussian model, a storage medium and equipment, and aims to solve the problems of low image quality, low generation speed, detail loss and the like in the prior art. The method comprises the following steps: firstly, acquiring image data through an unmanned aerial vehicle or aerial photography, and extracting a camera attitude and sparse point cloud by using a motion recovery structure (SfM) technology; secondly, constructing a three-dimensional Gaussian model based on the sparse point cloud, and performing iterative training through top view orthographic projection in combination with a gradient descent algorithm; in the training process, a densification strategy, a point deleting strategy, a blocking strategy and an image pyramid strategy are innovatively introduced, so that the detail expressive force and the overall quality of the image are remarkably improved. Specifically, according to the densification strategy, fine detail reconstruction is achieved by dynamically increasing Gaussian ball density, and according to the point deletion strategy, rendering efficiency is improved and computing resource allocation is optimized by eliminating redundant Gaussian balls. The image pyramid strategy generates a multi-level visual effect through multi-scale training, and the blocking strategy improves the reconstruction precision through local optimization. Finally, on the basis of the trained three-dimensional Gaussian model, real-time generation of a high-quality true orthophoto in a large-scale scene can be realized. The method has remarkable advantages in the aspects of efficiency, precision and practicability, provides important technical support for the fields of geographic information systems, urban planning, disaster monitoring and the like, and has wide application prospects.

The invention discloses a multi-view underwater three-dimensional point cloud splicing method based on sparse and dense point cloud fusion, and relates to the technical field of structure underwater detection. The method comprises the steps that a binocular camera surface image of an underwater structure is received, the binocular camera comprises a left camera and a right camera, dense point clouds are generated based on stereo matching and triangulation, and the corresponding relation between the dense point clouds and image pixels of the left camera is output; and based on the received left camera image, a sparse point cloud is generated by adopting a motion structure recovery algorithm and utilizing multi-view triangulation, and a corresponding relation between the sparse point cloud and the pixels of the left camera image is output. According to the method, when the point cloud of multi-view reconstruction is spliced, the surface features of the underwater structure can be well restored, feature extraction is not needed, the method has high accuracy and robustness, and high-precision point cloud splicing can be achieved in the underwater complex environment, so that the method has good application prospects in actual underwater structure detection and image splicing tasks.

The invention is suitable for the technical field of radar signal processing, and provides a space target SFM three-dimensional reconstruction method based on a multi-frame ISAR image, and the method comprises the steps: carrying out the preprocessing and imaging of large-angle inverse synthetic aperture radar ISAR echo data of a space target, and obtaining a multi-frame ISAR image sequence; extracting scattering points in each frame of ISAR image sequence according to an orthogonal matching pursuit algorithm, and performing feature matching on adjacent frames of ISAR image sequences to obtain a scattering point set; obtaining instantaneous radar sight line information, and constructing a projection matrix from a space target scattering center three-dimensional coordinate to an ISAR two-dimensional pixel plane according to the instantaneous radar sight line information; and obtaining a three-dimensional reconstruction result according to the scattering point set and the projection matrix in combination with a triangulation principle in an incremental motion recovery structure SFM method. According to the method, the robustness and precision of three-dimensional reconstruction can be improved.

The invention discloses a visual positioning method for satellite on-orbit fuel filling docking service, which comprises two stages of off-line training and on-line positioning, and comprises the following steps: firstly, acquiring image and pose information by using a camera, establishing a pre-matching database, carrying out feature matching, calculating the pose of the camera through structural motion, and generating sparse point cloud; carrying out dense reconstruction on the scene by adopting three-dimensional Gaussian sputtering, and generating a three-dimensional Gaussian sputtering model through training; after a camera at the tail end of the refueling satellite mechanical arm captures an original image in real time, a reference image most similar to the current image is recognized based on the pre-matching database, a corresponding initial pose is obtained, and the image is rendered to serve as a starting point of subsequent iterative optimization; and meanwhile, color migration correction is performed on the captured image by taking the rendered image as a reference, and inter-domain differences are eliminated to generate a target image in an optimization stage. According to the invention, a picture captured at an unknown view angle can be well positioned. And the calculation efficiency is improved while the accuracy is ensured as much as possible.

Images captured by a camera moving in an environment are received, and for each of a plurality of points in one of the images, outputs are computed using a neural network. The outputs comprise: a trajectory depicting the point in each of the plurality of images, as well as, for each trajectory, a prediction of visibility of the trajectory in each of the images and a prediction of whether the trajectory depicts a static or moving surface in the environment. The neural network receives the images and points as input and computes the outputs, wherein the outputs comprise for each of the trajectories, confidence data. The outputs are sent to a downstream process selected from any of: visual odometry, structure from motion, human body tracking, video editing, vehicle tracking.

The invention discloses a multi-view three-dimensional reconstruction method for a slender object based on curve guidance. According to the method, geometric prior and a deep learning segmentation network are fused, two-stage foreground extraction, a curve-guided motion recovery structure (SfM) and a surface optimization module capable of differential Poisson reconstruction are provided, and a set of complete processing flow is established for the problem that a slender and weak-texture object is difficult to accurately reconstruct. The method comprises the following steps: acquiring a multi-view image through a consumer-level terminal and performing frame extraction to obtain an input data set; generating a high-quality foreground mask by utilizing depth estimation and semantic segmentation; performing curve-guided SfM initialization based on the foreground skeleton curve to realize joint estimation of the camera pose and the sparse three-dimensional curve; carrying out surface geometric optimization through a micro Poisson equation and differentiable grid rendering; and designing a multi-branch loss function fusing luminosity, mask, regularization and Gaussian rendering supervision to train the model. Finally, the method can output a three-dimensional reconstruction result of the slender structure with high geometric accuracy, strong structural integrity and smooth surface, and has good robustness and application value.

The invention provides a method, apparatus, and storage medium for reconstructing three-dimensional models of buildings based on missing point cloud data. The method includes integrating image-based point cloud generation, neural network techniques, and skeleton line extraction methods, offering a novel approach to handling missing point cloud data. The generation of point cloud data is achieved using principles of Structure from Motion based on video or panoramic image data. The point cloud is sampled and segmented using a region growing algorithm. A neural network based on PointNet is constructed, utilizing cross-entropy loss functions to assess the missing points in the point cloud. For mapping high-confidence point clouds from sampled points, Truth Points is employed to complete the entire process of real-world three-dimensional reconstruction. The integration of images into the three-dimensional scene is achieved with strict geometric relationships.

Systems, computer readable medium and methods for autonomous drone navigation based on vision are disclosed. Example methods include capturing an image using an image capturing device of the autonomous drone, processing the image to identify an object, and navigating the autonomous drone relative to the object for a period of time. After the period of time a second type of navigation is used based on determining structure from motion navigation. Images are captured during the period of time to transition to the second type of navigation. The second type of navigation uses a downward pointing navigation camera and other sensors.

The invention discloses a multi-camera motion recovery structure method and system fusing feature point tracks, and the method comprises the steps: carrying out the feature point detection and matching of an input multi-camera image set, calculating the relative pose between image pairs, and constructing a scene graph with camera nodes and feature tracks as edges; defining a multi-camera model, and converting a camera pose estimation problem into a rigid unit pose and internal camera pose estimation problem; by taking the multi-camera model as a constraint, optimizing a problem through a decoupling rotation averaging method to obtain camera rotation parameters; in combination with a distance and angle-based mixed objective function, performing joint optimization on rigid unit translation and internal camera relative translation to obtain camera translation parameters and feature tracks; and by taking the rotation parameter, the translation parameter and the characteristic track as initial values, adjusting and optimizing a global pose and sparse point cloud coordinates through multi-camera joint binding, and obtaining robust and accurate camera parameters and scene sparse point cloud.

The invention discloses a text-to-three-dimensional surface generation method and system based on a two-dimensional Gaussian surface element, and belongs to the technical field of three-dimensional reconstruction, and the method comprises the steps: representing a three-dimensional object surface as a Gaussian surface element on a local tangent plane; receiving a text description, generating normal diagrams and texture diagrams of a plurality of visual angles based on a pre-training text-to-image diffusion model, and generating a preliminary background mask at the same time; determining the center of sphere and the maximum radius according to the extrinsic parameters of the camera, and randomly generating Gaussian surface elements in the spherical range; removing the surface elements falling in the preliminary background mask area; on the basis of a two-dimensional Gaussian dot drawing renderer, the normal graph and the texture graph are rendered from multiple perspectives, and curvature consistency regular loss and surface convergence constraint loss are calculated; geometric and texture parameters of Gaussian surface elements are optimized through back propagation, iteration is carried out until convergence, and a final three-dimensional model is output. According to the method, the high-fidelity three-dimensional surface can be generated from the text without SfM (Structural Recovery Motion) initialization, and the method has relatively high geometric consistency and generation speed.

Systems and methods are disclosed herein for using structure from motion (SfM) techniques to reconstruct three-dimensional (3D) surface models of tubular patient anatomies, such as the larynx and trachea, from clinical endoscopy videos. The disclosed methods may improve understanding of upper airway disease including vocal fold paralysis, laryngeal cancer, subglottic hemangiomas, subglottic stenosis, tracheal stenosis, tracheal cartilaginous sleeves, complete tracheal rings and tracheomalacia, and allow for quantitative analysis of complex laryngotracheal geometries. Quantitative measures of airway caliber and shape, which are critical for diagnostic purposes, may be obtained using the disclosed methods as a cost-effective and radiation-free alternative to relying on imaging studies. Results have demonstrated excellent resolution of reconstructions, when compared to high-resolution computed tomography (CT) scans (surface errors <0.300 mm).

The present invention discloses a dynamic scene reconstruction method and device based on multi-scale Gaussian spheres, which relates to the field of computer vision and includes: processing a video frame sequence to be reconstructed by using a structure from motion algorithm to generate a sparse point cloud, initializing the sparse point cloud to generate a 3D Gaussian sphere set; processing the 3D Gaussian sphere set by using a dual-domain deformation model and an adaptive timestamp to obtain a deformed 3D Gaussian sphere set; performing multi-scale Gaussian processing on the deformed 3D Gaussian sphere set to generate a multi-scale Gaussian sphere set; performing Gaussian screening based on pixel coverage rate on the multi-scale Gaussian sphere set to obtain an optimized multi-scale Gaussian sphere set; and performing Alpha blending processing based on the optimized multi-scale Gaussian sphere set to reconstruct an anti-aliased dynamic rendering scene image. The present invention solves the problems of large computational overhead and aliasing effect in current dynamic scene reconstruction.

This disclosure provides systems, methods, and devices that utilize machine learning models to determine corresponding spatial positions and motion trajectories for images. In one aspect, a method is provided that includes receiving an image of a scene captured by a camera; determining, with a first machine learning model, a plurality of position values relative to the camera for at least a subset of pixels within the image; and training a second machine learning model based on the determined position values. The method further includes determining basis trajectories based on movement of the positions relative to previous image frames, and determining, for each of a subset of pixels, a movement trajectory relative to the previous frames as a weighted combination of the basis trajectories. These techniques can be employed as part of a structure-from-motion pipeline to estimate multi-view three-dimensional geometry, camera poses, and per-pixel object motion. Additional aspects are provided.

Systems, methods, and computer program code for motion segmentation using a motion from structure approach; and related image processing tasks. Structure-from-motion systems are also described. In some implementations feature representations for pixels of an image frame are used to extrapolate from high reliability regions to lower reliability regions, based on semantic similarity as represented by the feature representations. This can link different visible parts of the same object, e.g. for generating a segmentation map for moving objects.

The invention relates to the field of rock mass structural surface recognition, in particular to an intelligent recognition method for a rock mass three-dimensional structural surface based on an AI visual large model. An artificial intelligence visual large model is combined with a motion recovery structure technology, and three-dimensional structural surface identification is realized by using data of different photogrammetry devices. According to the method, an identification task is converted from point cloud to original image data through the incidence relation (often neglected by previous research) between image pixels and point cloud in a motion recovery structure, and the implementation difficulty is remarkably reduced. A benchmark test shows that the method realizes the average precision of 0.91 in a three-dimensional database, and 87 structural surfaces are identified. The result shows that the method has high precision and high efficiency, and provides a powerful tool for geological data labeling.

The present invention discloses a three-dimensional reconstruction method based on the fusion of fractional gradient optimization and three-dimensional Gaussian sputtering. A sparse point cloud is generated through the structure from motion technology and the input picture data; Gaussian primitives centered on the point cloud are generated in space; rasterization rendering is performed on the scene represented by the Gaussian primitives; a joint optimization function is calculated for the rendered picture and the real picture; a fractional-order optimizer is designed by trimming the definition of fractional calculus; the fractional-order optimizer is used to optimize the parameters of the Gaussian primitives; according to the adaptive trimming strategy, the trimming of the Gaussian ellipsoid is completed. By using the fractional-order optimizer to guide the Gaussian primitives to perform gradient descent, the problem that the three-dimensional Gaussian sputtering falls into local minima in three-dimensional reconstruction is effectively alleviated by using the guidance of the fractional gradient. By performing joint loss optimization on the rendered image and the real image compared with the original function, the edge information of the image can be better focused on, and a reconstructed scene with richer details can be obtained.

The present invention relates to a three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration, belonging to the field of computer vision. Given a sequence of multi-angle images, the method first classifies the image sequence and uses the structure from motion algorithm to obtain the three-dimensional point cloud data corresponding to each type of image sequence. Then, the scale of each type of point cloud data is unified, and the point cloud registration algorithm is used to register and reconstruct each type of point cloud, thereby realizing the three-dimensional reconstruction of the non-cooperative target in space. The present invention classifies the image sequence and performs parallel reconstruction, thereby reducing the time consumption in the reconstruction process and improving the reconstruction efficiency, and can meet the real-time requirements of spacecraft operations. By registering the point cloud data after various reconstructions, the finally reconstructed point cloud is denser, improving the reconstruction accuracy.

Methods and systems for generating and displaying a virtual bottom view associated with a vehicle. Cameras generate images of a parking zone. A processor is programmed to generate image data associated with a first region of the parking zone that includes an object. Movement sensors generate movement data associated with movement of the vehicle during the time in which the image data is generated. A Structure from Motion (SfM) model generates a three-dimensional view associated with the object. Subsequently, when the object is out of view of the cameras, a synthesis model is executed to synthesize the current live view with the three-dimensional view. A virtual bottom view is displayed on a vehicle display during a parking event based on the synthesis, enabling a user to see a three-dimensional virtual view of the object when the object is not in the current field of view during the parking event.

System for a vehicle, including: receiving an image of the vehicle’s environment from an onboard camera; segmenting the image into a region that includes a road surface; determining depth information for the region; generating an output based on the depth information. Also disclosed: the output is a depth image which is used for training. Segmentation and depth determination may be performed using a trained neural network. Depth data (eg. line of sight range, 3D model, 3D point cloud) may be derived based on camera parameters, eg. 3D position, degrees of freedom, field of view, resolution, focal length. Other image regions may include sky, objects, vehicles, pedestrians. Vehicle navigation or control actions, eg. braking, steering, accelerating, may be determined and implemented. A 3D reconstruction of the scene may be may be created from monocular, monoscopic images using structure from motion by calculating camera displacements, and thence pixel disparities, between successive images. The reconstructed 3D scene may be combined with stereoscopic 3D images.

Embodiments of the present disclosure include a computer-implemented method for displaying three-dimensional viewing content based on a user perspective, the method comprising: receiving point cloud data by one or more structure from motion tools; creating a Gaussian splat scene by associating Gaussian splats with the point cloud data; generating two-dimensional raster images from the Gaussian splat scene; deriving a view matrix from a user position in a three-dimensional coordinate space, the view matrix comprising vectors indicating the user position within the Gaussian splat scene; determining a projection matrix comprising vectors for a transform of the three-dimensional coordinate space to two-dimensional screen space coordinates for display on a user interface; and projecting the one or more first two-dimensional raster images in a position in the two-dimensional screen space determined by matrix multiplication of the view matrix and projection matrix.

The invention discloses a dynamic scene reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: generating a corresponding SFM (motion recovery structure) point cloud according to multiple frames of original images, and carrying out the down-sampling of the SFM point cloud, and obtaining each seed point in the SFM point cloud; for any seed point, calculating a feature vector of the seed point based on the feature information of the seed point, and generating a time Gaussian point corresponding to the seed point according to the feature vector; and constructing a Gaussian point cloud based on each time Gaussian point obtained after traversing each seed point, and generating a target view of the dynamic scene corresponding to the multiple frames of original images according to Gaussian point cloud rendering. According to the method, the complex dynamic scene is divided into a plurality of local space arrays, the problem of dynamic modeling complexity caused by full-time-domain unified modeling in a traditional method is solved, accurate reconstruction of the dynamic motion scene is achieved, and a solution with higher robustness and practicability is provided for dynamic scene reconstruction.

A multi-view interactive digital media representation (MVIDMR) of an object can be generated from live images of an object captured from a camera. Selectable tags can be placed at locations on the object in the MVIDMR. When the selectable tags are selected, media content can be output which shows details of the object at location where the selectable tag is placed. A machine learning algorithm can be used to automatically recognize landmarks on the object in the frames of the MVIDMR and a structure from motion calculation can be used to determine 3-D positions associated with the landmarks. A 3-D skeleton associated with the object can be assembled from the 3-D positions and projected into the frames associated with the MVIDMR. The 3-D skeleton can be used to determine the selectable tag locations in the frames of the MVIDMR of the object.

Imaging systems and methods estimate poses of a fluoroscopic imaging device, which may be used to reconstruct 3D volumetric data of a target area, based on a sequence of fluoroscopic images of a medical device or points, e.g., radiopaque markers, on the medical device captured by performing a fluoroscopic sweep. The systems and methods may identify and track the points along a length of the medical device appearing in the captured fluoroscopic images. The 3D coordinates of the points may be obtained, for example, from electromagnetic sensors or by performing a structure from motion method on the captured fluoroscopic images. In other aspects, a 3D shape of the catheter is determined, then the angle at which the 3D catheter projects onto the 2D catheter in each captured fluoroscopic image is found.

The invention discloses a scaffold engineering compliance intelligent evaluation method based on a three-dimensional reconstruction technology. According to the scheme, an unmanned aerial vehicle aerial photography sampling plan is made according to scaffold engineering characteristics, a machine vision model is trained to detect scaffold key components, and the distribution compliance of the scaffold system key components is evaluated according to the types and the number of the key components. A scaffold three-dimensional point cloud model is generated by adopting an incremental SfM and a multi-view stereo matching algorithm MVS based on an aerial image, a point cloud model of each rod piece is obtained through segmentation, and a central characteristic line equation of the rod piece is further extracted through a cylinder fitting algorithm. According to relevant specifications and standards of scaffolds, a scaffold engineering compliance set is summarized; and calculating the spatial relationship of the characteristic line equation, and comparing the scaffold compliance set to evaluate the construction compliance of the scaffold engineering of the project under construction. According to the method, the unmanned aerial vehicle, intelligent image recognition, three-dimensional reconstruction and feature line extraction technologies are comprehensively used, and autonomous and efficient evaluation of the safety compliance of the floor type fastener scaffold is achieved.

This invention discloses a Gaussian splashing method for dynamic scenes based on spatiotemporal motion distillation. It outputs a set of sparse point clouds from images at different viewpoints and times through a motion inference structure. The sparse point clouds are used to initialize Gaussian point attributes. The initialized Gaussian point set undergoes a fixed number of pre-training iterations to obtain a standard spatial set. Learnable motion feature representations are introduced into the Gaussian point attributes to explicitly model the spatiotemporal motion of the Gaussian points. Motion anchor points are extracted by distilling the motion information of the Gaussian points. During the iteration process, an adaptive density control mechanism continuously updates the density distribution of the Gaussian points, outputting the corresponding Gaussian model and deformation field weights for subsequent rendering evaluation. This invention fully utilizes the spatiotemporal characteristics of dynamic objects to efficiently reconstruct and render dynamic scenes, effectively solving the artifact and noise problems that occur during dynamic scene rendering, thereby outputting high-quality rendered images.

The application discloses a plant phenotype-oriented three-dimensional reconstruction method and a collection system. The method first acquires multi-view image data of a plant through a 360-degree ring collection system integrated with an RGB camera and a depth camera; then generates an accurate foreground mask by using a foreground semantic segmentation model; then combines structure from motion (SfM) and depth point cloud, and performs fusion under the guidance of the foreground mask to obtain an initial point cloud; then performs mask-guided 3D Gaussian splashing (3DGS) optimization reconstruction based on the point cloud, and the optimization process improves the identification of small plant organs by using a mask weighted loss function and a semantic-guided density control strategy; and finally derives a color point cloud from the optimized Gaussian model, which can be directly used for extraction of phenotype parameters such as plant height and leaf area. The application solves the problems of large noise and detail loss in the prior art when reconstructing plants in a complex background, and realizes fast three-dimensional reconstruction which can be directly used for phenotype analysis.

System and method are provided for scaling a 3-D representation of a building structure. The method includes obtaining images of the building structure, including non-camera anchors. The method also includes identifying reference poses for images based on the non-camera anchors. The method also includes obtaining world map data including real-world poses for the images. The method also includes selecting candidate poses from the real-world poses based on corresponding reference poses. The method also includes calculating a scaling factor for a 3-D representation of the building structure based on correlating the reference poses with the selected candidate poses. Some implementations use structure from motion techniques or LiDAR, in addition to augmented reality frameworks, for scaling the 3-D representations of the building structure. In some implementations, the world map data includes environmental data, such as illumination data, and the method includes generating or displaying the 3-D representation.

This disclosure provides systems, methods, and devices that utilize machine learning models to determine corresponding spatial positions and motion trajectories for images. In one aspect, a method is provided that includes receiving an image of a scene captured by a camera; determining, with a first machine learning model, a plurality of position values relative to the camera for at least a subset of pixels within the image; and training a second machine learning model based on the determined position values. The method further includes determining basis trajectories based on movement of the positions relative to previous image frames, and determining, for each of a subset of pixels, a movement trajectory relative to the previous frames as a weighted combination of the basis trajectories. These techniques can be employed as part of a structure-from-motion pipeline to estimate multi-view three-dimensional geometry, camera poses, and per-pixel object motion. Additional aspects are provided.

The invention discloses a bridge damage positioning and quantifying method based on unmanned aerial vehicle panoramic unfolding and TCFormer driving, and the method comprises the steps: 1) unmanned aerial vehicle image collection: employing a matrix type flight path planning strategy, and dividing a bridge bottom into a plurality of independent collection regions; 2) panorama construction: performing single-component three-dimensional reconstruction on the acquired image through a motion recovery structure SfM and a multi-view stereoscopic vision MVS technology; 3) performing multi-class damage segmentation: performing semantic segmentation on the panoramic image cutting area based on a lightweight TCFormer model; 4) component-level positioning: establishing a standardized component coordinate system and an index system, and dividing the components into five types; according to the method, a health evaluation system including PMCI single component scoring and PCCI whole-span comprehensive scoring is constructed, full-process automation from image acquisition, damage identification, spatial positioning, geometric quantification to health evaluation is achieved, and the efficiency, precision and standardization level of bridge detection are improved.

Imaging systems and methods estimate poses of a fluoroscopic imaging device, which may be used to reconstruct 3D volumetric data of a target area, based on a sequence of fluoroscopic images of a medical device or points, e.g., radiopaque markers, on the medical device captured by performing a fluoroscopic sweep. The systems and methods may identify and track the points along a length of the medical device appearing in the captured fluoroscopic images. The 3D coordinates of the points may be obtained, for example, from electromagnetic sensors or by performing a structure from motion method on the captured fluoroscopic images. In other aspects, a 3D shape of the catheter is determined, then the angle at which the 3D catheter projects onto the 2D catheter in each captured fluoroscopic image is found.