227 results about "Image diffusion" patented technology

The invention provides a method for adapting a text-to-image (T2I) diffusion model for video editing by using spectral decomposition to achieve controlled spectral shifts in the model's weights. This adaptation involves maintaining constant singular vectors while selectively adjusting singular values in response to a text prompt. A spectral shift regularizer constrains adjustments, particularly limiting changes to larger singular values to ensure minimal deviation from the original model's structure. This approach allows efficient, prompt-driven video editing by modifying specific elements according to the prompt while preserving the original video context. By focusing on selective spectral adjustments, the method reduces adaptation time and computational demands, making it suitable for real-time and resource-sensitive applications, such as dynamic video editing for streaming services.

Methods and systems are provided for using generative AI to optimize product search queries. In embodiments described herein, product descriptions and product images for a plurality of products are obtained. A multi-modal style classification model classifies each product into a corresponding style of a plurality of styles based on the product's product description and product image. Relationships of each product to other products in the plurality of products are stored in a knowledge graph based on the corresponding style of each product and the corresponding product description of each product. An image is generated by a text-to-image diffusion model with a set of products of the plurality of products based on the relationships of each product of the plurality of products to other products in the plurality of products.

The invention relates to the technical field of image recognition, in particular to a license plate recognition system and method based on an image technology and a medium. The method comprises the following steps: acquiring area sensing data and a camera image set, and performing deformation effect compensation to obtain an environment compensation image set; performing image diffusion reverse enhancement on the environment compensation image set to obtain a license plate area enhanced image set; performing character region high-dimensional topological mapping based on the license plate region enhanced image set to obtain a character segmentation matrix; extracting character morphological characteristics according to the character segmentation matrix, and performing character recognition on the character morphological characteristics to obtain a character recognition result; and carrying out cross-character semantic compensation on the character recognition result to obtain a semantic compensation license plate character vector, and carrying out multi-target cross verification on the semantic compensation license plate character vector to obtain a license plate recognition result. According to the invention, the accuracy and robustness of license plate recognition can be improved.

A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining a text prompt describing an element and an attribute value for a continuous attribute of the element, embedding the text prompt to obtain a text embedding in a text embedding space, embedding the attribute value to obtain an attribute embedding in the text embedding space, and generating a synthetic image based on the text embedding and the attribute embedding, where the synthetic image depicts the continuous attribute of the element based on the attribute value.

A system and method for enhancing e-commerce product listings is disclosed, performed on a server. The method involves importing listing data through Application Programming Interface (API) connections and analyzing this data to calculate a multimodal vector embedding. A quality score is estimated based on the embedding and real-time market data metrics. The method generates content elements, including product images, textual descriptions, and infographics, by applying a controlled generation algorithm through a text-to-image diffusion model. This model integrates loss-guidance and attention injection mechanisms to produce a controlled layout of the product images, producing content that is visually appealing and market-relevant. The resulting content elements are stored in the server's data storage, ready for e-commerce display.

The present disclosure relates to systems, methods, and non-transitory computer-readable media that recolors a digital design according to colors of a digital image and further generates an enhanced recolored digital design. In particular, in some embodiments, the disclosed systems identify a digital image for recoloring a digital design and recolors the digital design utilizing a color affine transformation algorithm to generate a recolored digital design. Further, in some embodiments, the disclosed systems generate the enhanced recolored digital design by transforming one or more colors of the recolored digital design to be within a range of the colors of the digital image utilizing a convex hull projection method. Moreover, in some embodiments, the disclosed systems further enhance the recolored digital design utilizing a contrast enhancement algorithm to modify luminescence values.

In various examples, systems and methods are disclosed relating to aligning images into frames of a first video using at least one first temporal attention layer of a neural network model. The first video has a first spatial resolution. A second video having a second spatial resolution is generated by up-sampling the first video using at least one second temporal attention layer of an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.

The invention discloses a text generation image diffusion model enhancement method based on multi-target preference optimization. The method comprises the following steps: firstly, determining a plurality of reward models, and constructing a sample pair training set comprising positive and negative samples; and then, generating a loss weight of each sample pair in the sample pair training set, performing fine tuning training on the text map diffusion model by using the sample pair training set, and in the fine tuning training process, calculating a loss function value of each sample pair in combination with the loss weight of each sample pair until the training is completed, thereby obtaining an aligned text map diffusion model. According to the method provided by the invention, manual data annotation is not needed, the problems of preference inconsistency and over-optimization in a multi-reward scene are effectively solved, and the image quality, the text alignment capability and the multi-target optimization performance of the text-to-image generation model are remarkably improved. The method is superior to an existing optimization method under single-reward and multi-reward setting, shows higher generation quality and robustness, and can be seamlessly applied to various picture generation models.

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, a multiview latent tensor is generated based on processing a textual input using a diffusion machine learning model, where the multiview latent tensor corresponds to a plurality of orthographic projections corresponding to the textual input. A triplane latent tensor is generated based on the multiview latent tensor using a conversion machine learning model, and a three-dimensional mesh is generated based on processing the triplane latent tensor using a decoder machine learning model.

Embodiments of the present disclosure relate to training-free consistent text-to-image generation. A pre-trained text-to-image diffusion model is leveraged to generate images depicting a consistent subject for diverse prompts describing scenes. Inputs to the model are a text description of at least one subject with prompts (scene text descriptions) describing scenes, where each prompt is associated with a different generated image and the text description is used for all images that depict the subject. Internal activations (intermediate data) computed by the model during generation of the different images are shared for generation of the different images. A subject-driven shared attention block and correspondence-based feature injection are incorporated into the model to promote subject consistency within each image and / or between images. Additionally, layout diversity is encouraged while maintaining subject consistency. The model achieves state-of-the-art performance on subject consistency and text alignment, without requiring any optimization and naturally extends to multi-subject scenarios.

The invention discloses a video anomaly detection method based on a diffusion model and text embedding, and relates to the technical field of video anomaly detection. The invention provides a video anomaly detection method based on a diffusion model and text embedding, and the method comprises the steps: firstly extracting the text information of each video frame through a graphic and text generation model, and obtaining semantic features through a pre-trained CLIP text encoder; a video anomaly detection model is constructed based on an image diffusion model and a text diffusion model, an image diffusion denoising network and a text diffusion denoising network are respectively guided to reconstruct image features and text features by using a cross attention mechanism, model training is completed through overall optimization, and then video anomaly is detected. Therefore, by the adoption of the method, the problem of excessive generalization in a specific scene can be relieved by means of the powerful generation capacity of the diffusion model and the semantic understanding capacity of the graph-to-text model, the interpretability of video anomaly detection is enhanced, and the performance of the video anomaly detection model is improved.

A video generation framework that is controllable, unsupervised and based on events (CUBE) includes an event camera, which captures changes in light intensity at each pixel of a scene asynchronously and generates event camera data. A text-to-image diffusion model that is conditioned on textual descriptions integrates the event camera data to control video synthesis. Further, an edge extraction module translates event data into a format usable by the text-to-image diffusion model, whereby the diffusion model synthesizes detailed and contextually accurate videos based on textual prompts. Further, an improved system (CUBE Plus) includes a content frame identification module which selectively identifies and uses only the most information-rich event segments of the event camera data to drive cross-frame attention, and an event driven attention mechanism that allows the framework to focus on event-dense moments.

A computer system that performs quality control (QC) on images associated with diffusion and structural magnetic resonance imaging (MRI) is described. This computer may include: a computation device that executes program instructions; and memory that stores the program instructions. During operation, the computer system may automatically perform a set of validation operations, where, when one or more of the validation operations fails, the images are rejected. Moreover, the set of validation operations may include: performing QC on brain-tissue segmentation; performing QC on diffusion MRI processing; and performing QC on bundles determined from the images using a tractometry technique.

The invention relates to the technical field of computers, in particular to an image stylized editing method and device for a three-dimensional scene, equipment and a medium. The method comprises the following steps: acquiring a source three-dimensional model, a reference style image, a first image stylization editing network and a second image stylization editing network; obtaining content features of the to-be-edited image and style features of the reference style image; respectively optimizing two image stylization editing networks added with a low-rank adaptation function by using image diffusion loss to construct a third image stylization editing network, inputting the to-be-edited image and the reference style image into the third image stylization editing network for processing, and outputting a single-view image after stylization editing; and iteratively updating the stylized and edited single-view image by using an optimization strategy based on 3D content and style combination to obtain a stylized and edited 3D model. According to the method, the stylization effect of the reference style image is improved, and the fidelity and multi-view consistency of the 3D content are ensured on the premise of consistent styles.

The invention provides a lace texture image diffusion generation method based on an ADM and a dynamic denoising filtering model, the model combines a learnable Gaussian blur layer and a high-pass filtering technology, and the Gaussian blur layer aims at adaptively adjusting the blur degree of the image according to different input images and reducing high-frequency noise in the lace texture image. According to the high-pass filtering technology, key areas such as the contour and the edge of a main pattern in the image are enhanced, so that main texture features of the image are highlighted. The module is added into a down-sampling layer of an ADM network, the phenomenon that lace texture images generated by an original model are too smooth is improved, and the texture layering sense of the images is enhanced.

A method of generating a 3-dimensional representation of a subject is provided. The method includes receiving one or more descriptions characterizing the subject. The method also includes inputting the one or more descriptions characterizing the subject into a first specialized network of a machine learning model to generate one or more images depicting the subject according to the one or more descriptions. The method further includes inputting the generated one or more images to a second specialized network of the machine learning model to generate the 3-dimensional representation of the subject according to the one or more descriptions characterizing the subject.

The invention discloses a nickel-based superalloy microstructure image diffusion generation method, system and equipment, a medium and a product, and relates to the field of superalloy microstructure reconstruction. The method comprises the following steps: gradually diffusing an original training image into a Gaussian noise-like image by adopting a forward diffusion process of a denoising diffusion generation network, and gradually denoising in a reverse diffusion process to obtain a nickel-based high-temperature alloy microstructure-like image; adopting a geometric structure feature loss penalty mechanism to obtain a trained denoising diffusion generation network based on the geometric feature difference value; and adopting a back diffusion process of the trained de-noising diffusion generation network to obtain a virtual microstructure image, and screening to obtain a final microstructure image. According to the method, the problems that data samples are scarce, and acquisition of the alloy internal microstructure is tedious, time-consuming, high in cost and the like are solved, similar microstructure images are visually represented, and intelligent generation of the virtual microstructure is achieved.

The invention discloses an HRCT image generation method and system based on a conditional control diffusion model, and relates to the technical field of image processing, and the method comprises the steps: collecting a lesion image to construct a data set, and employing an image processing algorithm to recognize and extract a to-be-detected region from the data set; determining a seed point by analyzing texture characteristics of the to-be-detected region, and inputting the seed point into the image segmentation model to obtain an image segmentation result; classifying and marking the segmented images by using an image analysis technology to obtain an image classification result; and based on an image classification result, optimizing a preset image diffusion model, and fusing the image diffusion model with the condition regulation and control network to generate a high-resolution image. According to the method, by establishing the data set and adopting the image processing technology, the high-quality image is generated while the labeling cost is reduced, a more reliable data basis is provided for a subsequent image segmentation task, and the detection efficiency of the image is improved.

The invention belongs to the field of image processing, and relates to a method for embedding a robust watermark in a diffusion model generated image, which comprises the following steps of: acquiring cue words, initial Gaussian noise and watermark information, and inputting the cue words, the initial Gaussian noise and the watermark information into a trained diffusion model based on watermark embedding to obtain a watermark-embedded image; the training process of the diffusion model comprises the following steps: acquiring cue words, initial Gaussian noise and watermark information, and inputting the cue words, the initial Gaussian noise and the watermark information into an encoder to obtain potential vectors; inputting the potential vector into a self-attention module to obtain an embedded position vector; embedding the watermark information into the potential vector according to the embedding position vector; inputting the potential vector embedded with the watermark information into a decoder to obtain an image embedded with the watermark; extracting watermark information from the image embedded with the watermark; updating parameters of a diffusion model according to the image embedded with the watermark and the extracted watermark information until a trained diffusion model is obtained; according to the method, the embedding position is selected by combining the potential of the diffusion model and the accuracy of the self-attention mechanism, so that efficient watermark embedding and extraction are realized.

The invention discloses a single view reconstruction and rendering method, which belongs to the field of view reconstruction and rendering, and comprises the following steps of: firstly, generating multi-view feature representation with strong geometric consistency from a single input image by introducing an image diffusion module of a cross attention mechanism; a point cloud reconstruction module with self-attention and cross-attention is utilized, and multi-view information is fused to reconstruct an accurate three-dimensional point cloud; and finally, constructing differentiable three-dimensional Gaussian representation based on the point cloud, rendering the differentiable three-dimensional Gaussian representation, and outputting a new view angle image, a normal map and a depth map. According to the system, a staged training strategy is adopted, and a composite loss function including multi-scale bidirectional consistency smooth loss and feature consistency loss is innovatively used for optimization. According to the method, the problems of low geometric accuracy, multi-view inconsistency, detail missing and the like in single-view reconstruction are effectively solved, and the method can be widely applied to the fields of virtual reality, digital twinning, cultural heritage digitization and the like.

The invention discloses a video virtual fitting framework in a real scene based on a controllable image diffusion model. The video virtual fitting framework comprises a data preprocessing module used for constructing feature expressions of figures and clothes; the single-stage image fitting network is used for generating a virtual fitting image based on the preprocessed data and performing implicit deformation and fusion of clothes by using a cross attention mechanism; the diffusion guiding module is used for enhancing the time sequence consistency of the generated video, a self-supervision model Video MAE and a self-supervision image model DINO-V2 are introduced, the video is masked by the self-supervision model Video MAE, and a complete video is reconstructed through inter-frame information; and the long video generation module is used for generating a long video and keeping the consistency of the video, splitting the complete video into a plurality of short video clips, and reducing the jitter at the clip joints by using a time sequence common noise reduction strategy. The diffusion guiding module is designed, efficient time sequence consistent video virtual fitting is achieved through self-supervised learning, and the method has the advantages of being excellent in shielding processing, stable in long video generation and the like.

A processor-implemented method includes receiving a text-semantic input at a first stage of a neural network, including a first convolutional block and no attention layers. The method receives, at a second stage, a first output from the first stage. The second stage comprises a first down sampling block including a first attention layer and a second convolutional block. The method receives, at a third stage, a second output from the second stage. The third stage comprises a first up sampling block including a second attention layer and a first set of convolutional blocks. The method receives, at a fourth stage, the first output from the first stage and a third output from the third stage. The fourth stage comprises a second up sampling block including no attention layers and a second set of convolutional blocks. The method generates an image at the fourth stage, based on the text-semantic input.

The invention discloses a three-dimensional generation method based on semantic enhancement hybrid reconstruction. The three-dimensional generation method comprises the following steps: 1, performing semantic enhancement on an input original text by utilizing a GPT-4 model; 2, generating a two-dimensional image from the text; inputting the semantically enhanced text into a text generation image diffusion model, so that the generated two-dimensional image is beneficial to three-dimensional reconstruction; 3, multiple views are generated from the two-dimensional image, the images generated in the text image diffusion model are input into a multi-view diffusion model, and images at different view angles are generated for three-dimensional reconstruction; 4, performing three-dimensional hybrid reconstruction by using the image and the text, taking the generated multi-view and the original text as input, and generating three-plane representation through a three-dimensional large reconstruction model; 5, generating a three-dimensional grid from the three-plane representation; and taking the three-plane representation as input, and performing contour surface extraction by using FlexiCubes to obtain a final three-dimensional grid. According to the invention, the problems of slow three-dimensional object generation, low quality and misalignment with semantics are solved.

A computer-implemented method produces latent representations of text prompts created for use with a text-to-image diffusion model. Training images are generated by providing the latent representations to a first artificial neural network implementing a denoising process of the text-to-image diffusion model. A machine-learned modality inversion module is trained. The training includes performing training iterations for training data pairs, each training data pair being comprised of one of the training images and one of the text prompts. Each training iteration for each training data pair includes: providing the one of the training images of the training data pair to a pre-trained classifier configured to generate alternate conditioning information based upon the one of the training images, converting, by the machine-learned modality inversion module, the alternate conditioning information into an alternate latent representation, and updating parameters of the machine-learned modality inversion module based upon differences between the alternate latent representation and one of the latent representations of the one of the text prompts.

Described is a system for improving machine learning models. In some cases, the system improves such models by identifying a performance characteristic for machine learning model blocks in an iterative denoising process of a machine learning model, connecting a prior machine learning model block with a subsequent machine learning model block of the machine learning model blocks within the machine learning model based on the identified performance characteristic, identifying a prompt of a user, the prompt indicative of an intent of the user for generative images, and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

The invention discloses a defect image diffusion generation method and device, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: determining a target defect type and a defect size parameter corresponding to a target defect according to a natural language prompt text input by a user; building a defect physical vector based on the target defect type and the defect size parameter; adjusting the diffusion noise amplitude in the diffusion process based on the defect physical vector, and generating an initial defect image corresponding to the initial workpiece noise image; and performing physical consistency verification on the initial defect image, and outputting a target defect image according to a verification result. The diffusion noise amplitude in the diffusion process is adjusted based on the defect physical vector corresponding to the target defect, the initial defect image is generated, and the target defect image is output according to the verification result of the initial defect image, so that the problem that the defect image generation method in the prior art cannot pay attention to the physical size of the defect is solved. And the accuracy of the generated defect image is not high.

The invention provides a universal 3D intelligent modeling method for multi-view coupling constraint, belongs to the technical field of three-dimensional simulation modeling, and solves the problems that an existing intelligent three-dimensional modeling generation method is poor in generalization ability, a generation result is easy to over-fit, and multi-view geometric inconsistency is easy to occur. Comprising the following steps: optimizing learnable three-dimensional representation through a pre-trained text generation image diffusion model and a fine-adjusted multi-view generation diffusion model, and generating initial three-dimensional representation which is completely consistent with text semantics; according to the transparency prediction information of each point in the preliminary three-dimensional representation and the multi-view rendering graph of the preliminary three-dimensional representation, extracting and converting into a rough three-dimensional grid representation containing geometric structure and texture information; a geometry and texture decoupling optimization method is adopted, multi-view coupling constraints are combined, the geometrical shape and the surface texture of the triangular mesh are finely adjusted and optimized respectively, and a three-dimensional triangular mesh model with high-quality geometrical morphology and realistic texture is generated.

According to the saliency reference and decoupling esophageal tumor CT image diffusion segmentation method provided by the invention, the accuracy of tumor segmentation is improved by combining a saliency pseudo tag and a diffusion denoising network. The method comprises the following steps: firstly, preprocessing an esophageal tumor CT image, generating a significant pseudo tag, and enhancing the attention of a network to a tumor region; and then, constructing a saliency reference segmentation network and a diffusion denoising network, and gradually carrying out noise adding and denoising to restore the image by adopting a conditional diffusion model. A multi-task learning framework for decoupling segmentation and de-noising tasks is designed, and collaborative optimization between tasks is ensured through consistency constraint. Through joint optimization of cross entropy loss, Dice loss, mean square error and consistency loss, the segmentation performance is improved. Experimental results show that the method can effectively improve the segmentation precision of the esophageal tumor, and has high clinical application potential.

The invention discloses a method for generating an aviation lifelike tree sample and identifying a tree species by using an image diffusion model. The method comprises the following steps: constructing a multi-modal data set; training an image-text comparison model and a Unet denoising network model; outputting predicted noise in the trained Unet denoising network model, and finally generating final image reconstruction which is in semantic alignment with the input text description; the obtained single tree crown images are synthesized; and constructing a YOLOv11 detection model, and training the YOLOv11 detection model by using the synthesized forest image to realize tree species identification. According to the method, language semantics and a diffusion model are combined to generate a vivid tree sample for capturing specific features of species and seasonal phenological changes, and the vivid tree sample is synthesized into a high-fidelity forest image, so that tree species identification in an aerial image is enhanced, and effective crown detection and accurate tree species identification are realized.

The invention belongs to the technical field of physiological information processing, and particularly discloses a medical image reconstruction method, device and equipment based on a diffusion model, and the method comprises the steps: obtaining a first medical image; generating a second medical image from the first medical image according to a preset conversion rule; inputting the second medical image into a measurement diffusion model to obtain a noise measurement value of any time step; and inputting the second medical image into an image diffusion model, and obtaining a third medical image based on the constraint of the noise value of any time step. According to the method, multiple reconstruction tasks such as under-sampling MRI reconstruction, sparse view CT reconstruction and medical image super-resolution can be executed by utilizing arbitrary diffusion generation priori. In order to improve the reconstruction quality of the diffusion model, a measurement diffusion model with a determined trajectory is provided for guiding reconstruction, and a parallel trajectory constraint mode is used for enabling the image diffusion model and the measurement diffusion model to be subjected to parallel evolution, so that the reconstruction quality and the reconstruction precision are greatly improved.