718 results about "3d space" patented technology

In a first aspect, a computer-implemented method of fusing visual features obtained with multiple sensor modalities comprises extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non-camera sensor modality; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non- parallel to the camera image plane in 3D space; and computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, without applying monocular depth detection to the camera image. In a second aspect, the fused feature vectors are computed using a transformer neural network cross-attention between the camera feature vectors and the second feature vectors.

The invention discloses an image processing method and device and a storage medium. The method comprises the steps of obtaining a to-be-simulated 3D convolution model and training data; decomposing the 3D convolution model into cascading of a 3D space convolution model and a 3D time convolution model to obtain a pseudo 3D cascading convolution model; training a pseudo 3D cascade convolution modelby using the training data, and obtaining parameters of a 3D spatial convolution model and a 3D time convolution model; converting the 3D space convolution model and the 3D time convolution model intoa 2D space convolution model and a 2D time convolution model; setting a feature rearrangement rule for the 2D spatial convolution model and the 2D time convolution model; mapping model parameters ofthe 3D spatial convolution model and the 3D time convolution model into parameters of a 2D spatial convolution model and a 2D time convolution model to obtain a 2D cascaded convolution model; and performing convolution operation on the image by using the 2D spatial convolution model and the 2D time convolution model. By means of the mode, image processing conducted through 3D convolution operationcan be achieved through the 2D convolution model.

Apparatuses, systems, and techniques of using one or more machine learning processes (e.g., neural network(s)) to detect objects from a plurality of image frames. In at least one embodiment, a plurality of image frames are fused into a feature map using one or more neural networks. In at least one embodiment, a plurality of image frames are processed using one or more neural networks to detect objects in a 3D space.

The invention discloses a tunnel fracture recognition method based on three-dimensional live-action reconstruction and ordered re-acquisition of a virtual camera. The method specifically comprises the following steps: acquiring a multi-view rock fracture image; acquiring three-dimensional point cloud data of a tunnel wall surface according to the acquired rock mass fracture image, reconstructing a geometrical shape of the tunnel, mapping acquired texture information to a three-dimensional grid vertex through an image processing algorithm, and generating a three-dimensional tunnel model; setting parameters of the virtual camera, orderly collecting to form a virtual image, and enabling the position of the virtual camera to correspond to the real position of the model; and extracting information of the crack of the virtual image, and projecting the coordinates of the two-dimensional image back to the three-dimensional space through the projection matrix to obtain the space coordinates of the crack. A virtual image is generated by combining a texture mapping technology of a three-dimensional model, so that the limitation of a visual angle, illumination and space in actual image acquisition is overcome, and high-precision identification of tunnel cracks is realized.

The invention discloses a sports motion recognition method based on 3D space-time attention and a slowfast network, and the method comprises the steps: obtaining an original human body motion video, sampling the original human body motion video through different frame rates, generating Slow branch and Fast branch input images, and taking the Slow branch and Fast branch input images as training samples; constructing a dual-branch network based on spatio-temporal feature extraction; designing a loss function, and training the dual-branch network based on spatial-temporal feature extraction by using the training sample; and sampling a to-be-identified human body behavior video through different frame rates to obtain corresponding video frame images, and inputting the video frame images into the trained dual-branch network based on spatial-temporal feature extraction to obtain an identified human body behavior category. According to the sports action recognition method based on the 3D space-time attention and slowfast network, the problems that in the prior art, the locality of the dependency relationship between the features in the space-time dimension is captured, and the model complexity is high are solved.

The invention relates to the field of unmanned aerial vehicle multispectral remote sensing image analysis and inversion model construction based on a machine learning algorithm, in particular to a temperate zone typical grassland surface biomass estimation method. A multi-rotor unmanned aerial vehicle is used for carrying a multi-spectral sensor, multi-band spectral reflection information of a pasture is obtained from a near low altitude, a physical spectral reflection model is constructed to calculate real and accurate reflectivity data of a same-band spectrum, vegetation coverage, soil attributes and pasture canopy information of a sampling area are recorded, and the earth surface biomass is measured by cutting a sample; according to the method, a plurality of vegetation indexes and a 3D space model are combined with a multi-modal input structure, data enhancement and small sample learning strategies are adopted, a biomass inversion model is constructed through an MANBA machine learning framework, the precision of grassland biomass estimation is improved, and effective technical support is provided for grassland ecological monitoring and management.

An encoder projects a scene represented by 3D gaussian splats into 2D representation(s), where the 2D representation(s) store different parameters of the 3D gaussian splats, along with associated metadata describing transformation from 3D space into 2D representations. The encoder forms the 2D representation(s) and the associated metadata into bitstream(s). The encoder outputs the bitstreams. The decoder receives the bitstream(s) encoding 2D representation(s) and associated metadata. The decoder applies decoding to corresponding individual bitstreams to form the 2D representation(s) and the associated metadata. The decoder reprojects, as described by the associated metadata, the 2D representations) into the scene represented by the 3D gaussian splats. The decoder outputs the scene for rendering, or renders the scene, to a viewer.

Three-dimensional Gaussian splatting mechanisms that initialize a set of 3D Gaussian distributions, un-project pixels from two-dimensional (2D) planes to 3D space by applying queries to the 3D Gaussians at expected un-projected ray depth positions, and splat the 3D Gaussian distributions on the 2D planes based on the expected un-projected ray depth positions.

The invention discloses a 3D zero sample anomaly detection method and device based on a 2D multi-mode large model. Based on a 2D multi-modal model CLIP, 3D anomaly detection is realized under the condition of not depending on a target domain sample, and point cloud information from 3D and 2D is integrated through the following modes: firstly, point cloud is rendered from multiple perspectives through the CLIP, and 2D representation of the point cloud is acquired; projecting the 2D representation back to the 3D space, thereby understanding the 3D representation of the point cloud; additional regularization is performed on the 2D representation to further enhance the understanding of the frame on the 3D representation. After the point information and the pixel information of the point cloud are represented, the invention provides a mixed representation learning method, generalization abnormal modes in the point cloud and the pixels are captured into learnable text prompts, and accurate detection of 3D anomalies is realized. According to the method, high-precision zero-sample anomaly detection for 3D data is realized for the first time, and the method has high innovativeness and industrial application value.

The invention discloses a three-dimensional space GPS navigation system. A three-dimensional space GPS navigation system comprises: a map information collection device for acquiring a GPS satellite map of a target area; the unmanned aerial vehicle flying device flies to the high altitude of the target area and is in signal connection with the signal base station or a GPS satellite so as to position the position of the unmanned aerial vehicle in the high altitude; the high-altitude image acquisition module is used for acquiring high-altitude image data of the target area and establishing a three-dimensional scene model based on the high-altitude image data; and the navigation information generation module is used for determining the position in the three-dimensional scene model based on the position of the unmanned aerial vehicle and generating a target route according to the three-dimensional scene model. The navigation scheme provided by the invention can effectively reduce the influence of natural plant growth on navigation when being used in the field, and can accurately find the satellite route through the constructed three-dimensional scene model.

The invention provides a video data processing method and system based on a digital twin scene. The method comprises the following steps: performing spatial-temporal feature extraction on a video frame sequence to determine a first spatial-temporal feature matrix; segmenting a plurality of video elements from the video frame sequence, and generating an element segmentation mask corresponding to each video element; generating a second spatial-temporal characteristic matrix; obtaining a 2D motion track corresponding to the dynamic element based on the second spatial-temporal characteristic matrix; mapping the 2D motion trails of the dynamic elements into a 3D space to create or update a target virtual 3D scene model, and generating a predicted video frame sequence through the target virtual 3D scene model; and adjusting model parameters in the target virtual 3D scene model according to a comparison result of the predicted video frame sequence and the video frame sequence decomposed by the video data. According to the technical scheme provided by the invention, the video data processing efficiency can be effectively improved.

Methods, apparatus, devices, subsystems, and systems for holographically displaying live scenes including one or more three-dimensional (3D) objects are provided. In one aspect, a system includes a holographic capturing system and a holographic display system. The holographic capturing system includes: an optical system configured to generate an optical hologram of a live scene that comprises one or more three-dimensional (3D) objects, and an optical sensor configured to capture sequential optical holograms of the live scene and output sequential hologram data associated with the sequential optical holograms of the live scene, each optical hologram being associated with respective hologram data. The holographic display system is configured to optically reconstruct the live scene in a 3D space based on at least part of the sequential hologram data.

An unmanned system maneuver controller (USMC) includes an inertial navigation system (INS) for state estimation of the USMC in three-dimensional (3D) space, a communications device configured to communicate with an unmanned system, and a processor configured to receive, via the communications device, flight, maneuver, or dive data from the unmanned system, and generate flight, maneuver, or dive control instructions based at least on the flight, maneuver, or dive data and data received from the INS. The flight, maneuver, or dive control instructions are configured to pilot the unmanned system based on movement of the USMC in 3D space. A remote may selectively control an operation of the USMC. The USMC may be mounted to a weapon or observation device, such that movement of the weapon or observation device in 3D space controls a movement of the unmanned system. Additional systems and associated methods are also provided.

The invention discloses a low-illumination scene image three-dimensional reconstruction method and device based on a 3D Gaussian ball, and electronic equipment. The method comprises the following steps: 1) obtaining a low-illumination scene image; 2) initializing information of a 3D Gaussian ball in a 3D space according to camera parameters and a volume rendering technology to obtain an initial position, an initial radius and an initial color of the 3D Gaussian ball, and writing the initialized 3D Gaussian ball information into the enhanced image; 3) constructing an illumination adjustment network based on the convolutional neural network fused with the attention mechanism; 4) inputting the enhanced image obtained in the step 2) into an illumination adjustment network, and predicting an illumination weight value of each 3D Gaussian ball by using the illumination adjustment network; and 5) based on the initial information of the 3D Gaussian ball and the illumination weight value, performing three-dimensional scene reconstruction by using a three-dimensional reconstruction algorithm to obtain a three-dimensional scene reconstruction view Cnor under normal illumination. According to the invention, the illumination can be dynamically adjusted through the illumination adjusting network, and the real restoration of the low-illumination scene is realized.

Techniques are discussed herein for generating three-dimensional (3D) representations of an environment based on two-dimensional (2D) image data, and using the 3D representations to perform 3D object detection and other 3D analyses of the environment. 2D image data may be received, along with depth estimation data associated with the 2D image data. Using the 2D image data and associated depth data, an image-based object detector may generate 3D representations, including point clouds and / or 3D pixel grids, for the 2D image or particular regions of interest. In some examples, a 3D point cloud may be generated by projecting pixels from the 2D image into 3D space followed by a trained 3D convolutional neural network (CNN) performing object detection. Additionally or alternatively, a top-down view of a 3D pixel grid representation may be used to perform object detection using 2D convolutions.

A method and apparatus generate a 2-dimensional (2D) image. A method for generating a 2-dimensional (2D) image includes obtaining a time index and a view. The method further includes obtaining first coding indices and a first codebook for canonical 3D Gaussians, wherein the canonical 3D Gaussians are 3D Gaussians corresponding to a reference time index, and represent a 3D space corresponding to the reference time index. The method also includes obtaining second encoding indices and a second codebook for a parameter offset, wherein the parameter offset indicates a difference between the canonical 3D Gaussians and 3D Gaussians for the time index. The method further includes reconstructing the canonical 3D Gaussians based on the first coding indices and the first codebook. The method also includes reconstructing parameter offsets of the 3D Gaussians for the time index based on the second coding indices and the second codebook. The method further includes adding the reconstructed canonical 3D Gaussians and the reconstructed parameter offset to reconstruct the 3D Gaussians for the time index. The method also includes generating a second image for the view based on the reconstructed 3D Gaussians.

Methods, apparatus, devices, subsystems, and systems for holographically displaying live scenes are provided. In one aspect, a method includes optically generating an optical hologram of a live scene including one or more three-dimensional (3D) objects; capturing sequential optical holograms of the live scene and generating sequential hologram data associated with the sequential optical holograms of the live scene, each optical hologram being associated with respective hologram data; processing the at least part of the sequential hologram data based on a mathematical function to remove low frequency components to generate digital holograms associated with the live scene; and reconstructing the live scene in a 3D space based on the digital holograms.

Methods, apparatus, devices, subsystems, and systems for holographically displaying live scenes including one or more three-dimensional (3D) objects are provided. In one aspect, a system includes a holographic capturing system and a holographic display system. The holographic capturing system includes: an optical system configured to generate an optical hologram of a live scene that comprises one or more three-dimensional (3D) objects, and an optical sensor configured to capture sequential optical holograms of the live scene and output sequential hologram data associated with the sequential optical holograms of the live scene, each optical hologram being associated with respective hologram data. The holographic display system is configured to optically reconstruct the live scene in a 3D space based on at least part of the sequential hologram data.

The embodiment of the invention discloses a naked eye three-dimensional image display method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a plurality of three-dimensional virtual sub-objects and initial display parameters corresponding to the plurality of three-dimensional virtual sub-objects in a target naked-eye three-dimensional scene, wherein the plurality of three-dimensional virtual sub-objects have a logic association relationship; displaying initial naked-eye three-dimensional pictures corresponding to the plurality of three-dimensional virtual sub-objects in the naked-eye three-dimensional display screen according to the initial display parameters; generating space display parameters of the plurality of three-dimensional virtual sub-objects according to the logic association relationship; and dynamically adjusting the three-dimensional space positions of at least part of the three-dimensional virtual sub-objects in the plurality of three-dimensional virtual sub-objects according to the space display parameters so as to update the initial naked-eye three-dimensional picture to obtain a target naked-eye three-dimensional picture. In this way, the spatial display parameters are generated through the logic incidence relation to adjust the three-dimensional space position of the three-dimensional virtual sub-object, and simulation stimulation of the human brain in the three-dimensional space can be achieved when the human eyes watch the three-dimensional virtual sub-object.

Systems and techniques are described for implementing autonomous control of powered earth-moving vehicles, including to automatically calibrate sensors on a powered earth-moving vehicle, such as to determine position and orientation of directional sensors on movable vehicle parts. For example, an on-vehicle sensor to be calibrated may include a LIDAR sensor located on the powered earth-moving vehicle, such as on a movable component part of the vehicle (e.g., a hydraulic arm, a tool attachment, etc.), and a global common frame of reference is determined for different datasets gathered at different times from such a sensor in order to combine or compare the datasets, such as by determining the sensor position in 3D space at a time of dataset gathering (e.g., relative to another reference point on the vehicle with a known location in the global common frame of reference, such as by using one or more determined transforms).

The invention relates to a zero-sample open vocabulary scene understanding method based on semantic instance generation, and the method comprises the steps: obtaining multi-frame image data including depth information, obtaining 2D mask information through segmentation for each frame of image data, carrying out the semantic feature extraction of a 2D mask, projecting the semantic feature to a 3D space, obtaining a 3D point cloud segment, and carrying out the segmentation of the 3D point cloud segment; combining the 3D point cloud fragments into 3D instances through time sequence grouping combination by adopting a semantic-guided under-segmentation filtering and combination threshold linear attenuation strategy to obtain a 3D instance set; for semantic features of each instance in the 3D instance set, performing feature aggregation processing on the premise of considering intra-class similarity and inter-class difference; and based on the 3D instance set after feature aggregation, constructing a hybrid octree-graph structure to realize zero-sample open vocabulary scene understanding. Compared with the prior art, the problems of inaccurate instance segmentation, inaccurate feature aggregation and low spatial representation efficiency are effectively relieved.

Methods, systems, and apparatus, including medium-encoded computer program products, for searching through spatial recordings by constructing queries. One of the methods include constructing a query based on input received at a three-dimensional (3D) space displayed at a visual programming interface, wherein the input includes positioning of objects at 3D positions within the 3D space over time, wherein the input defines at least one spatial orientation between at least two objects from the objects in the 3D space; executing the query to search a database of 3D recordings to find at least one segment from at least one 3D recording that includes the at least two objects and matches the spatial orientation between the at least two objects, as defined in the input, at a specific point in time; and presenting the at least one segment from the at least one 3D recording via the visual programming interface.

Disclosed is a computer-implemented method that includes receiving, by one or more processors, a three-dimensional (3D) representation of a user's mouth where the 3D representation is a mesh representation comprising a plurality of surfaces; mapping, by the one or more processors, the 3D representation of the user's mouth into a two-dimensional (2D) space; encoding, one or more 3D surface characteristics of the plurality of surfaces using one or more channels of a 2D representation; applying, by the one or more processors, a machine learning model to the representation of the user's mouth in 2D space, the machine learning model trained to enhance the representation of the user's mouth in 2D space; and mapping, by the one or more processors, the enhanced representation of the user's mouth in 2D space to an enhanced representation of the user's mouth in 3D space using the one or more surface characteristics.

[Task to be accomplished]To enable a user to easily check whether or not a sensor is set to properly detect a moving body that appears in a measurement area after adjustment of an installation state of the sensor.[Solution]A traffic flow measurement server places a virtual object of a moving body in a 3D space including 3D point cloud data acquired by a lidar, in response to a user's operation on a terminal device, displays a lidar point cloud image 265 including an image 271 of the virtual object of the moving body, and overlays images 277, 278 of the virtual object on a camera image 263 and a lidar intensity image 264 based on a positional relationship between the 3D point cloud data and the virtual object.

The invention relates to the technical field of optical cable fault positioning, and discloses a cable fault spatial position positioning method, which comprises the following steps: acquiring 3D network data of an optical cable, and constructing a topological graph for describing the connection condition of the optical cable; carrying out OTDR detection on the optical cable, and calculating a fault distance between a fault point and a detection point based on an optical time domain reflection signal; exploring a fault path between a fault point and a detection point in the topological graph; if only one fault path exists, spatial positioning of a fault point is carried out; if at least two fault paths exist, extracting attenuation characteristics of the optical time domain reflection signals; screening a fault path based on the attenuation characteristics, and carrying out the spatial positioning of a fault point; according to the invention, the cable fault point can be accurately mapped to the 3D space model, and powerful support is provided for rapidly positioning and repairing the cable fault.

Aspects of the present disclosure relate to a system comprising a computer-readable storage medium storing at least one program and a method for rendering a virtual modification of a real-world environment depicted in image content. A reference surface is detected in a three-dimensional (3D) space captured within a camera feed generated by a camera of a computing device. An image mask is applied to the reference surface within the 3D space captured within the camera feed. A visual effect is applied to the image mask corresponding to the reference surface in the 3D space. Applying the visual effect to the image mask results in the modified surface being rendered in the camera feed being presented on a display of the computing device.

The invention discloses a 3D anomaly detection method and device based on self-distillation. According to the method, rapid and efficient zero sample anomaly detection is carried out on the 3D point cloud by using the self-distillation technology and the strong generalization ability of the 2D multi-modal model CLIP. According to the method, 3D point cloud features are learned and extracted in the following mode, and anomaly detection is carried out. Firstly, a lightweight learnable point cloud feature extraction module is designed; encoding the extracted point cloud features by using an encoder, and aligning the extracted point cloud features with encoding features of the point cloud rendering graph to a 3D space by using a self-distillation technology; and finally, performing zero sample anomaly detection of the 3D point cloud by using the feature. Compared with the prior art, the method can quickly and efficiently carry out zero sample anomaly detection of the 3D point cloud, and has relatively high practicability and innovativeness.

An example device for detecting objects through processing of media data, such as image data and point cloud data, includes a processing system configured to form voxel representations of a real-world three-dimensional (3D) space using images and point clouds captured for the 3D space at consecutive time steps, extract image and / or point cloud features for voxels in voxel representations of the 3D space, determine correspondences between the voxels at consecutive time steps according to similarities between the extracted features, and determine positions of objects in the 3D space using the correspondences between the voxels. For example, the processing system may perform triangulation according to positions of a moving object to positions of the voxels at the time steps. In this manner, the processing system may generate an accurate bird's eye view (BEV) representation of the real-world 3D space.

An audio conference system for automatically forming a single combined physical microphone array aperture across associated and / or disassociated ad-hoc microphone elements in a shared 3D space is provided. The audio conference system includes a plurality of microphone / speaker units, each including at least one microphone and / or at least one speaker and a system processor communicating with the microphone / speaker units. The system processor instructs the microphone / speaker units to transmit unique calibration signals sequentially or simultaneously and to calculate time difference of arrival (TDOA) between the microphone / speaker units. A physical array structure of the microphone / speaker units is obtained based on TDOA between the microphone / speaker units, and a consolidated target coverage zone common to the microphone / speaker units is generated based on the physical array structure.

The invention discloses a data enhancement method and system for monocular 3D target detection. The method comprises the following steps: respectively constructing a scene database and an object database; randomly sampling a scene in the scene database, randomly sampling a position in a reachable area of the scene, randomly sampling an object in the object database, inserting the random sampling position after collision and shielding detection, and fusing the object and the scene to obtain a recombined image and a depth map; based on the depth map, the recombined image is converted to a 3D space, random disturbance is carried out on the camera attitude, and training images at different visual angles are generated; the object, the scene and the camera attitude are randomly recombined to form a continuously generated new image which serves as training data of a neural network, and the performance of a monocular 3D target detection model is improved; existing data resources can be efficiently utilized, and the model performance is remarkably improved; the application of a camera-based scene perception algorithm in the automatic driving industry is facilitated, and the rapid landing and development of related industries are facilitated.