177 results about "Image diffusion" patented technology

The invention provides a method for adapting a text-to-image (T2I) diffusion model for video editing by using spectral decomposition to achieve controlled spectral shifts in the model's weights. This adaptation involves maintaining constant singular vectors while selectively adjusting singular values in response to a text prompt. A spectral shift regularizer constrains adjustments, particularly limiting changes to larger singular values to ensure minimal deviation from the original model's structure. This approach allows efficient, prompt-driven video editing by modifying specific elements according to the prompt while preserving the original video context. By focusing on selective spectral adjustments, the method reduces adaptation time and computational demands, making it suitable for real-time and resource-sensitive applications, such as dynamic video editing for streaming services.

The invention relates to the technical field of image recognition, in particular to a license plate recognition system and method based on an image technology and a medium. The method comprises the following steps: acquiring area sensing data and a camera image set, and performing deformation effect compensation to obtain an environment compensation image set; performing image diffusion reverse enhancement on the environment compensation image set to obtain a license plate area enhanced image set; performing character region high-dimensional topological mapping based on the license plate region enhanced image set to obtain a character segmentation matrix; extracting character morphological characteristics according to the character segmentation matrix, and performing character recognition on the character morphological characteristics to obtain a character recognition result; and carrying out cross-character semantic compensation on the character recognition result to obtain a semantic compensation license plate character vector, and carrying out multi-target cross verification on the semantic compensation license plate character vector to obtain a license plate recognition result. According to the invention, the accuracy and robustness of license plate recognition can be improved.

A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining a text prompt describing an element and an attribute value for a continuous attribute of the element, embedding the text prompt to obtain a text embedding in a text embedding space, embedding the attribute value to obtain an attribute embedding in the text embedding space, and generating a synthetic image based on the text embedding and the attribute embedding, where the synthetic image depicts the continuous attribute of the element based on the attribute value.

In various examples, systems and methods are disclosed relating to aligning images into frames of a first video using at least one first temporal attention layer of a neural network model. The first video has a first spatial resolution. A second video having a second spatial resolution is generated by up-sampling the first video using at least one second temporal attention layer of an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.

The invention discloses a text generation image diffusion model enhancement method based on multi-target preference optimization. The method comprises the following steps: firstly, determining a plurality of reward models, and constructing a sample pair training set comprising positive and negative samples; and then, generating a loss weight of each sample pair in the sample pair training set, performing fine tuning training on the text map diffusion model by using the sample pair training set, and in the fine tuning training process, calculating a loss function value of each sample pair in combination with the loss weight of each sample pair until the training is completed, thereby obtaining an aligned text map diffusion model. According to the method provided by the invention, manual data annotation is not needed, the problems of preference inconsistency and over-optimization in a multi-reward scene are effectively solved, and the image quality, the text alignment capability and the multi-target optimization performance of the text-to-image generation model are remarkably improved. The method is superior to an existing optimization method under single-reward and multi-reward setting, shows higher generation quality and robustness, and can be seamlessly applied to various picture generation models.

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, a multiview latent tensor is generated based on processing a textual input using a diffusion machine learning model, where the multiview latent tensor corresponds to a plurality of orthographic projections corresponding to the textual input. A triplane latent tensor is generated based on the multiview latent tensor using a conversion machine learning model, and a three-dimensional mesh is generated based on processing the triplane latent tensor using a decoder machine learning model.

Embodiments of the present disclosure relate to training-free consistent text-to-image generation. A pre-trained text-to-image diffusion model is leveraged to generate images depicting a consistent subject for diverse prompts describing scenes. Inputs to the model are a text description of at least one subject with prompts (scene text descriptions) describing scenes, where each prompt is associated with a different generated image and the text description is used for all images that depict the subject. Internal activations (intermediate data) computed by the model during generation of the different images are shared for generation of the different images. A subject-driven shared attention block and correspondence-based feature injection are incorporated into the model to promote subject consistency within each image and / or between images. Additionally, layout diversity is encouraged while maintaining subject consistency. The model achieves state-of-the-art performance on subject consistency and text alignment, without requiring any optimization and naturally extends to multi-subject scenarios.

A video generation framework that is controllable, unsupervised and based on events (CUBE) includes an event camera, which captures changes in light intensity at each pixel of a scene asynchronously and generates event camera data. A text-to-image diffusion model that is conditioned on textual descriptions integrates the event camera data to control video synthesis. Further, an edge extraction module translates event data into a format usable by the text-to-image diffusion model, whereby the diffusion model synthesizes detailed and contextually accurate videos based on textual prompts. Further, an improved system (CUBE Plus) includes a content frame identification module which selectively identifies and uses only the most information-rich event segments of the event camera data to drive cross-frame attention, and an event driven attention mechanism that allows the framework to focus on event-dense moments.

A computer system that performs quality control (QC) on images associated with diffusion and structural magnetic resonance imaging (MRI) is described. This computer may include: a computation device that executes program instructions; and memory that stores the program instructions. During operation, the computer system may automatically perform a set of validation operations, where, when one or more of the validation operations fails, the images are rejected. Moreover, the set of validation operations may include: performing QC on brain-tissue segmentation; performing QC on diffusion MRI processing; and performing QC on bundles determined from the images using a tractometry technique.

A method of generating a 3-dimensional representation of a subject is provided. The method includes receiving one or more descriptions characterizing the subject. The method also includes inputting the one or more descriptions characterizing the subject into a first specialized network of a machine learning model to generate one or more images depicting the subject according to the one or more descriptions. The method further includes inputting the generated one or more images to a second specialized network of the machine learning model to generate the 3-dimensional representation of the subject according to the one or more descriptions characterizing the subject.

The invention belongs to the field of image processing, and relates to a method for embedding a robust watermark in a diffusion model generated image, which comprises the following steps of: acquiring cue words, initial Gaussian noise and watermark information, and inputting the cue words, the initial Gaussian noise and the watermark information into a trained diffusion model based on watermark embedding to obtain a watermark-embedded image; the training process of the diffusion model comprises the following steps: acquiring cue words, initial Gaussian noise and watermark information, and inputting the cue words, the initial Gaussian noise and the watermark information into an encoder to obtain potential vectors; inputting the potential vector into a self-attention module to obtain an embedded position vector; embedding the watermark information into the potential vector according to the embedding position vector; inputting the potential vector embedded with the watermark information into a decoder to obtain an image embedded with the watermark; extracting watermark information from the image embedded with the watermark; updating parameters of a diffusion model according to the image embedded with the watermark and the extracted watermark information until a trained diffusion model is obtained; according to the method, the embedding position is selected by combining the potential of the diffusion model and the accuracy of the self-attention mechanism, so that efficient watermark embedding and extraction are realized.

The invention discloses a single view reconstruction and rendering method, which belongs to the field of view reconstruction and rendering, and comprises the following steps of: firstly, generating multi-view feature representation with strong geometric consistency from a single input image by introducing an image diffusion module of a cross attention mechanism; a point cloud reconstruction module with self-attention and cross-attention is utilized, and multi-view information is fused to reconstruct an accurate three-dimensional point cloud; and finally, constructing differentiable three-dimensional Gaussian representation based on the point cloud, rendering the differentiable three-dimensional Gaussian representation, and outputting a new view angle image, a normal map and a depth map. According to the system, a staged training strategy is adopted, and a composite loss function including multi-scale bidirectional consistency smooth loss and feature consistency loss is innovatively used for optimization. According to the method, the problems of low geometric accuracy, multi-view inconsistency, detail missing and the like in single-view reconstruction are effectively solved, and the method can be widely applied to the fields of virtual reality, digital twinning, cultural heritage digitization and the like.

A processor-implemented method includes receiving a text-semantic input at a first stage of a neural network, including a first convolutional block and no attention layers. The method receives, at a second stage, a first output from the first stage. The second stage comprises a first down sampling block including a first attention layer and a second convolutional block. The method receives, at a third stage, a second output from the second stage. The third stage comprises a first up sampling block including a second attention layer and a first set of convolutional blocks. The method receives, at a fourth stage, the first output from the first stage and a third output from the third stage. The fourth stage comprises a second up sampling block including no attention layers and a second set of convolutional blocks. The method generates an image at the fourth stage, based on the text-semantic input.

A computer-implemented method produces latent representations of text prompts created for use with a text-to-image diffusion model. Training images are generated by providing the latent representations to a first artificial neural network implementing a denoising process of the text-to-image diffusion model. A machine-learned modality inversion module is trained. The training includes performing training iterations for training data pairs, each training data pair being comprised of one of the training images and one of the text prompts. Each training iteration for each training data pair includes: providing the one of the training images of the training data pair to a pre-trained classifier configured to generate alternate conditioning information based upon the one of the training images, converting, by the machine-learned modality inversion module, the alternate conditioning information into an alternate latent representation, and updating parameters of the machine-learned modality inversion module based upon differences between the alternate latent representation and one of the latent representations of the one of the text prompts.

Described is a system for improving machine learning models. In some cases, the system improves such models by identifying a performance characteristic for machine learning model blocks in an iterative denoising process of a machine learning model, connecting a prior machine learning model block with a subsequent machine learning model block of the machine learning model blocks within the machine learning model based on the identified performance characteristic, identifying a prompt of a user, the prompt indicative of an intent of the user for generative images, and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

The invention discloses a defect image diffusion generation method and device, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: determining a target defect type and a defect size parameter corresponding to a target defect according to a natural language prompt text input by a user; building a defect physical vector based on the target defect type and the defect size parameter; adjusting the diffusion noise amplitude in the diffusion process based on the defect physical vector, and generating an initial defect image corresponding to the initial workpiece noise image; and performing physical consistency verification on the initial defect image, and outputting a target defect image according to a verification result. The diffusion noise amplitude in the diffusion process is adjusted based on the defect physical vector corresponding to the target defect, the initial defect image is generated, and the target defect image is output according to the verification result of the initial defect image, so that the problem that the defect image generation method in the prior art cannot pay attention to the physical size of the defect is solved. And the accuracy of the generated defect image is not high.

The invention provides a universal 3D intelligent modeling method for multi-view coupling constraint, belongs to the technical field of three-dimensional simulation modeling, and solves the problems that an existing intelligent three-dimensional modeling generation method is poor in generalization ability, a generation result is easy to over-fit, and multi-view geometric inconsistency is easy to occur. Comprising the following steps: optimizing learnable three-dimensional representation through a pre-trained text generation image diffusion model and a fine-adjusted multi-view generation diffusion model, and generating initial three-dimensional representation which is completely consistent with text semantics; according to the transparency prediction information of each point in the preliminary three-dimensional representation and the multi-view rendering graph of the preliminary three-dimensional representation, extracting and converting into a rough three-dimensional grid representation containing geometric structure and texture information; a geometry and texture decoupling optimization method is adopted, multi-view coupling constraints are combined, the geometrical shape and the surface texture of the triangular mesh are finely adjusted and optimized respectively, and a three-dimensional triangular mesh model with high-quality geometrical morphology and realistic texture is generated.

The invention discloses a method for generating an aviation lifelike tree sample and identifying a tree species by using an image diffusion model. The method comprises the following steps: constructing a multi-modal data set; training an image-text comparison model and a Unet denoising network model; outputting predicted noise in the trained Unet denoising network model, and finally generating final image reconstruction which is in semantic alignment with the input text description; the obtained single tree crown images are synthesized; and constructing a YOLOv11 detection model, and training the YOLOv11 detection model by using the synthesized forest image to realize tree species identification. According to the method, language semantics and a diffusion model are combined to generate a vivid tree sample for capturing specific features of species and seasonal phenological changes, and the vivid tree sample is synthesized into a high-fidelity forest image, so that tree species identification in an aerial image is enhanced, and effective crown detection and accurate tree species identification are realized.

The invention belongs to the technical field of physiological information processing, and particularly discloses a medical image reconstruction method, device and equipment based on a diffusion model, and the method comprises the steps: obtaining a first medical image; generating a second medical image from the first medical image according to a preset conversion rule; inputting the second medical image into a measurement diffusion model to obtain a noise measurement value of any time step; and inputting the second medical image into an image diffusion model, and obtaining a third medical image based on the constraint of the noise value of any time step. According to the method, multiple reconstruction tasks such as under-sampling MRI reconstruction, sparse view CT reconstruction and medical image super-resolution can be executed by utilizing arbitrary diffusion generation priori. In order to improve the reconstruction quality of the diffusion model, a measurement diffusion model with a determined trajectory is provided for guiding reconstruction, and a parallel trajectory constraint mode is used for enabling the image diffusion model and the measurement diffusion model to be subjected to parallel evolution, so that the reconstruction quality and the reconstruction precision are greatly improved.

The invention discloses a sea surface temperature image diffusion completion method and system based on dual-stage fusion constraint, and the method comprises the steps: firstly constructing sea area multi-source priori knowledge features to guide the reconstruction of sea surface temperature through integrating three types of heterogeneous data of geography, space-time and power; deep fusion is carried out on the sea area multi-source priori knowledge features through a cross-modal feature fusion module to obtain fused multi-source priori knowledge, and the fused multi-source priori knowledge is fused with structural knowledge to form comprehensive condition features used for guiding the generation process of a conditional diffusion model; and finally, realizing refined reconstruction of the missing region based on a conditional diffusion model, and outputting a completion result. And obtaining a final complete image according to the complementation result, the sea surface temperature image to be complemented and the missing mask. According to the invention, the accuracy of sea surface temperature image completion is improved.

The invention, which relates to the technical field of image identification, discloses an intelligent analysis method for a wound image at an applet end, comprising an image acquisition port and an auxiliary end set according to the applet end; according to the invention, the image acquisition port and the auxiliary end are set, the image diffusion models under different networks are constructed, the output image influence analysis value can quantify the influence degree of network factors on image analysis, and in combination with the preset image influence virtual mechanism, the real-time image data are reasonably supplemented and recovered, so that the real-time image analysis efficiency is improved. The method comprises the following steps: preliminarily extracting demand features from two-dimensional image data, constructing a wound three-dimensional dynamic model, performing dynamic matching according to a human body standard image, analyzing and judging the healing condition in a future stage, predicting a wound recovery volume value and actual wound image data, and obtaining the healing degree between wounds. And false analysis and judgment are carried out on the wound healing process, so that false images possibly appearing in the wound healing process can be identified, and the wound identification quality is improved.

Text-to-image models are used to generate images based on text prompts. Existing text-to-image models create images that are often unclear and exhibit hybrid characteristics of multiple subjects i.e., each subject present in image exhibit characteristic of multiple subjects. Present disclosure provides a method and a system for personalized multi-subject text to image generation. The system first fine-tunes existing text-to-image diffusion model using a plurality of images of target subjects. Then, the system performs image generation based on local text prompts using the fine-tuned text-to-image diffusion model. In particular, the fine-tuned text-to-image diffusion model uses a composite diffusion algorithm for generating subject images. Thereafter, the system model computes a subject aware segmentation loss for generated images which is then used to correct the subject appearance in the generated images. Finally, the system applies a global diffuser to the generated images to create a harmonized image based on a global text prompt.

The invention provides an image diffusion generation method and system based on multi-modal guidance and feedback closed loop, and the method comprises the steps: obtaining multi-modal input information; performing feature extraction processing on the multi-modal input information, and determining structural features and semantic features; generating a guide vector according to the structural features and the semantic features; according to the guide vector, inputting the structural features and the semantic features into a preset diffusion generation network in stages, and determining an initial image; obtaining user feedback input information according to the initial image; inputting the user feedback input information into a preset feedback interpreter, and determining an updated guide vector; and according to the updated guide vector, inputting the structural features and the semantic features into a preset diffusion generation network in stages, and determining a target diffusion image. According to the method and the device, the multi-modal data is guided in a staged manner, a user feedback closed loop is realized, and the consistency, controllability and user interactivity of the generated image are improved.

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, a reference latent tensor generated based on a reference input to a diffusion machine learning model is accessed. A first latent tensor generated during a first iteration of processing data using a denoising backbone of the diffusion machine learning model is accessed, and a first intermediate tensor is generated based on processing the reference latent tensor and the first latent tensor using an auxiliary machine learning model. A second latent tensor is generated, during a second iteration of processing data using the denoising backbone, based on the first latent tensor and at least in part on the first intermediate tensor.

Systems and methods are disclosed that generate dense blob representations such as blob parameters and blob descriptions, and use the dense blob representations to generate images. For example, embodiments of the present disclosure may decompose a scene into visual primitives (e.g., dense blob representations) and based on the blob representations, embodiments of the present disclosure develop a blob-grounded text-to-image diffusion model (BlobGEN) for compositional generation. For example, in some embodiments, a new masked cross-attention module may be introduced to disentangle the fusion between blob representations and visual features. In some embodiments, to leverage the compositionality of large language models (LLMs), a new in-context learning approach may be introduced to generate blob representations from text prompts.

The invention is suitable for the technical field of building model data processing, and provides an intelligent building texture repairing and beautifying method fusing a deep learning model, and the method comprises the steps: firstly, carrying out the recombination of the texture of a three-dimensional building model, and generating a to-be-processed wall surface and roof texture picture; a to-be-repaired area, needing to be repaired, of the building texture is extracted by combining low-rank matrix decomposition and an SAM image segmentation algorithm, and then whether an LAMA image repair model or a PatchMatch image repair algorithm is used for texture processing is determined according to the proportion of the to-be-repaired area in a whole texture image so as to achieve the maximum repair effect. And finally, carrying out texture distortion repair and material beautification by using an image diffusion model, a ControlNet control chart and cue words, and outputting beautified wall surface and roof textures. The problems of texture shielding, garland, noise, window distortion and the like can be solved at the same time, automatic detection and automatic texture repairing and beautifying of the texture shielding garland area are achieved, and the time consumption of texture editing processing in the building monotonized modeling process is remarkably shortened.

The invention relates to the field of power grid defect retrieval, in particular to a diffusion model-based deep hash image retrieval method for power grid defect maintenance, which comprises the following steps of: 1, performing fusion coding by inputting text data and image data, and constructing an initial hash code generation model; 2, using a Pair-wise loss function to optimize the distribution of sample pairs in a hash space, introducing a quantization loss function, generating an efficient binary hash code, and generating a high-quality binary hash code; 3, constructing a Hash code-image latent diffusion model, performing diffusion generation by encoding and decoding the Hash code / image to a continuous latent space, enabling the Hash code to correspond to the image in a generative manner, and directly fitting spatial distribution; and 4, defining a loss function of the Hash code-image diffusion model, generating a high-quality Hash code and an image, and obtaining a power grid defect type in a mode of searching images by images. Auxiliary training is carried out through fusion of text features and image features, so that the semantic features understand the images more deeply.

Embodiments of the present disclosure relate to training-free consistent text-to-image generation. A pre-trained text-to-image diffusion model is leveraged to generate images depicting a consistent subject for diverse prompts describing scenes. Inputs to the model are a text description of at least one subject with prompts (scene text descriptions) describing scenes, where each prompt is associated with a different generated image and the text description is used for all images that depict the subject. Internal activations (intermediate data) computed by the model during generation of the different images are shared for generation of the different images. A subject-driven shared attention block and correspondence-based feature injection are incorporated into the model to promote subject consistency within each image and / or between images. Additionally, layout diversity is encouraged while maintaining subject consistency. The model achieves state-of-the-art performance on subject consistency and text alignment, without requiring any optimization and naturally extends to multi-subject scenarios.

The invention belongs to the technical field of intelligent automobiles, and particularly relates to a driver driving state image data generation method based on a conditional diffusion model. Comprising the following steps: step 1, acquiring original driver driving state data and constructing camera parameters; 2, potential representation extraction and geometric control vector construction of an original image; step 3, potential representation generation of the target image; step 4, image reconstruction and post-processing; according to the method, an image diffusion generation network is taken as a core, a geometric transformation relation between an original camera and a target camera is combined, and generalization generation from a single image to a multi-view image is completed through a potential spatial modeling and condition guidance mechanism; the objective of the invention is to synthesize lifelike images under configuration of other visual angles and camera parameters by using a small amount of original images.

The invention relates to an image generation method and device for a driving scene, electronic equipment and a storage medium, and the method comprises the steps: obtaining a simulation image and an initial noise matrix of the driving scene, and enabling the initial noise matrix to be determined based on a real image of a real driving scene; inputting the simulation image into an image generation control network of an image generation model for coding to obtain a feature image; and inputting the feature image and the initial noise matrix into an image diffusion network of the image generation model for processing to obtain a driving scene image. By adopting the technical scheme, the real style of image generation is introduced through the initial noise matrix, and the initial noise matrix is determined based on the real image of the real driving scene, so that the generated driving scene image can restore the style of the real world, and the quality and visual fidelity of the driving scene image are improved.