Data set generation method and system for sparse scene

Abstract

The present application relates to the technical field of data generation for sparse scenes, and discloses a data set generation method and system for a sparse scene. The method comprises: inputting an actually acquired target sparse scene image into a target detection model to obtain bounding boxes of a target foreground and a background; using the bounding boxes of the target foreground and the background as prompt words for a preset large-scale segmentation model to perform model inference to respectively obtain masks of the target foreground and the background, and, on the basis of the masks of the target foreground and the background, obtaining coordinate information of a target foreground image and a background region; and using an adaptive image fusion technique to integrate the coordinate information of the target foreground image and the background region to generate a new target sparse scene image, and adding the newly generated target sparse scene image to a training data set of the target detection model. The present application only requires a small number of target objects and background images to generate a large number of new high-quality images, thereby solving the problem of scarce scene data, and providing rich data resources for subsequent model training.

Description

A method and system for generating datasets for sparse scenes

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411698683.2, filed on November 25, 2024, entitled “A Dataset Generation Method and System for Sparse Scenes”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of data generation technology for sparse scenes, specifically to a method and system for generating datasets for sparse scenes. Background Technology

[0004] In actual production and daily life, data acquisition becomes difficult in some scenarios due to challenges such as difficulty in sensor deployment, high data acquisition costs, and unstable data quality. Furthermore, traditional data acquisition methods are often costly and constrained by user data privacy concerns, making comprehensive data collection difficult.

[0005] To address this issue, a potential solution is to utilize image fusion techniques to generate new data. Image fusion integrates image data from different sources to produce a single image with richer information and higher resolution. This approach not only improves data quality but also reduces data acquisition costs. Existing image fusion techniques typically employ fixed parameters when processing images from different sources and of varying quality. This can lead to inaccurate or incomplete datasets in certain situations. For example, when the source images exhibit significant differences in quality or features, fixed-parameter fusion methods may fail to adequately account for these differences, resulting in suboptimal fusion results. Summary of the Invention

[0006] In view of this, this application provides a method and system for generating datasets for sparse scenes, in order to solve the problem of obtaining rich and high-quality image data when there is little data in sparse scenes.

[0007] Firstly, this application provides a method for generating datasets for sparse scenarios, the method comprising:

[0008] The target detection model is input into the image of the sparse target scene to obtain the detection bounding boxes of the target foreground and background;

[0009] Using a pre-defined segmentation model, the detection boxes of the target foreground and background are used as prompt words for model inference to obtain the masks of the target foreground and background respectively, and the target foreground image and background region coordinate information are obtained based on the masks of the target foreground and background.

[0010] Adaptive image fusion technology is used to integrate the target foreground image and background region coordinate information to generate a new target sparse scene image, and the newly generated target sparse scene image is added to the training dataset of the target detection model.

[0011] This application provides a method for generating datasets for sparse scenes. Through object detection technology, it can accurately identify and locate objects in images, providing effective cue words for large-scale model segmentation. Combined with large-scale model segmentation technology, it can accurately segment objects and backgrounds. Furthermore, it can adaptively and flexibly adjust the fusion method based on images from different sources and of different qualities, improving the realism of the fused images. Only a small number of object and background images are needed to generate a large number of new high-quality images, solving the problem of scarce scene data and providing rich data resources for subsequent model training.

[0012] In one optional implementation, the target detection model is trained through the following process:

[0013] Annotate the foreground and background of a small number of images realistically captured from sparse target scenes;

[0014] The labeled data is normalized and combined with data augmentation techniques to generate a training set.

[0015] Based on the training set, a preset machine learning model is trained, and the trained model is used as an object detection model. The object detection model outputs a detection box to detect the foreground and background of the target.

[0016] This application embodiment only requires processing a small number of images collected from real-world scenarios with sparse targets. Compared to large-scale data collection, this significantly reduces the manpower, material resources, and time costs required in the data collection phase. By labeling the target foreground and background, a clear learning objective is provided for the model. The model can accurately understand which areas are the target foreground and which are the background based on the labeled information, effectively improving the accuracy, efficiency, and generalization ability of target detection.

[0017] In one optional implementation, the step of using a preset segmentation model to perform model inference using the detection boxes of the target foreground and background as prompt words to obtain masks for the target foreground and background respectively, and obtaining the target foreground image and background region coordinate information based on the masks for the target foreground and background, includes:

[0018] The background single channel is obtained by using a preset segmentation model to perform model inference with the detection boxes of the target foreground and background as prompt words, and the mask of the target foreground single channel image and the mask of the background single channel image are obtained respectively.

[0019] The mask of the single-channel image of the target foreground is expanded to mask data with the same number of channels as the original real-world image. Based on the original real-world image and the mask data with the same number of channels as the original real-world image, the segmented target foreground image is obtained, and the width and height values ​​of the target object in the target foreground image are calculated.

[0020] Obtain the minimum bounding rectangle corresponding to the background single-channel image, and obtain the center coordinates, width, and height of the minimum bounding rectangle.

[0021] This application uses the detection boxes of the target foreground and background as prompts for the pre-defined segmentation model inference, providing the model with clear location information and effectively combining the advantages of target detection and image segmentation. Target detection provides the approximate location of the target (detection box), while image segmentation can further refine this to the pixel level, generating masks to accurately divide the target foreground and background. This combination leads to a deeper understanding of the image and allows for better application to subsequent processing tasks.

[0022] In one optional implementation, when the width or height of the target object is greater than the width or height of the background, the size of the target object is scaled based on a scaling factor, and the width and height of the target object are updated after scaling.

[0023] In this embodiment, when the size (width or height) of the target object is larger than the background size, the target object is scaled based on a scaling factor. This ensures that the target object can be reasonably placed within the background area. Scaled objects make the proportion between the target and the background more harmonious. In a dataset containing multiple target and background combinations, if the size of the target object is too large, it may lead to data inconsistency. By scaling the target object, all data can be made more uniform in size, facilitating unified processing and analysis.

[0024] In one optional implementation, the step of using adaptive image fusion technology to integrate the target foreground image and background region coordinate information to generate a new target sparse scene image, and adding the newly generated target sparse scene image to the training dataset of the target detection model, includes:

[0025] The target foreground image is subjected to data augmentation processing to introduce new features, resulting in a target foreground image set. The fusion position range is determined randomly based on the center coordinates of the minimum bounding rectangle.

[0026] Based on the fusion location range, the target foreground image set is superimposed and fused with the real-acquired background image to obtain a fused image set, and the divergence value of each image in the fused image set in the fusion region is calculated.

[0027] Images with a divergence value less than or equal to the first threshold in the fused image set are used as the first fused image set;

[0028] Images with a divergence value greater than the first threshold and less than the second threshold in the fused image set are processed using a preset fusion method and used as the second fused image set.

[0029] The images with a divergence value greater than the second threshold in the fused image set are input into the fine-tuned diffusion model to obtain the third fused dataset.

[0030] The first fused image set, the second fused image set, and the third fused image set are added to the training dataset of the target detection model as newly generated target sparse scene images.

[0031] The embodiments of this application describe the entire process from image fusion and divergence value calculation to adaptive image fusion technology processing based on different divergence value ranges, and finally adding the processed image set to the training dataset. This process helps to further optimize the image fusion effect and enrich the content of the training dataset.

[0032] In one alternative implementation, the process of fine-tuning the diffusion model includes:

[0033] A small number of images of a sparse target scene are collected and labeled with target foreground and background. Gaussian noise is gradually added to the labeled images until the final image with pure noise is obtained.

[0034] The image with added noise is input into a preset noise prediction network, and the model is trained using a preset loss function to obtain a fine-tuned diffusion model.

[0035] This application embodiment utilizes a small, pre-defined number of images realistically captured from the target sparse scene for annotation, ensuring that the training data closely revolves around the target scene. This ensures that the diffusion model focuses on learning the features of the target foreground and background within that specific scene, avoiding interference from irrelevant data and thus accurately adapting to the needs of the target sparse scene. The pre-defined loss function provides a clear optimization objective for model training. It measures the difference between the noise predicted by the model and the actual added noise, guiding the model to adjust parameters through the backpropagation algorithm to minimize this difference. This ensures that the model learns in the right direction during fine-tuning, effectively updating model parameters and thereby improving model performance and generation quality.

[0036] Secondly, this application provides a dataset generation system for sparse scenes, including:

[0037] The detection box output module is used to input the real-world image of the sparse target scene into the target detection model to obtain the detection boxes for the target foreground and background;

[0038] The image segmentation module is used to use a preset segmentation model to perform model inference by using the detection boxes of the target foreground and background as prompt words, and to obtain the masks of the target foreground and background respectively, and to obtain the target foreground image and background region coordinate information based on the masks of the target foreground and background.

[0039] The adaptive fusion module is used to integrate the target foreground image and background region coordinate information using adaptive image fusion technology to generate a new target sparse scene image, and to add the newly generated target sparse scene image to the training dataset of the target detection model.

[0040] Thirdly, this application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the sparse scene dataset generation method of the first aspect or any corresponding embodiment described above.

[0041] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the sparse scene dataset generation method of the first aspect or any corresponding embodiment described above.

[0042] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the sparse scene dataset generation method of the first aspect or any corresponding embodiment described above.

[0043] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 is a schematic diagram of a scene where an indoor fire escape is blocked;

[0046] Figure 2 is a flowchart illustrating a method for generating a dataset in a sparse scene according to an embodiment of this application;

[0047] Figure 3 is a flowchart illustrating another sparse scene dataset generation method according to an embodiment of this application;

[0048] Figure 4 is a structural block diagram of a sparse scene dataset generation system according to an embodiment of this application;

[0049] Figure 5 is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Specific Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] With the acceleration of urbanization, high-rise buildings are becoming increasingly common, highlighting the growing importance of fire safety. Indoor fire escape routes are crucial for evacuation and firefighting during a fire, and their unobstructed access directly impacts people's lives. However, due to various reasons, such as accumulated debris and damaged fire-fighting facilities, as shown in Figure 1, blocked indoor fire escape routes are frequently encountered, posing a serious threat to fire safety. Currently, the main methods for addressing blocked indoor fire escape routes include regular inspections, public education campaigns, and video surveillance. However, these methods still have limitations in practical application. For example, inspectors may not be able to detect all blockages in a timely manner; the effectiveness of public education campaigns is affected by various factors, making it difficult to ensure practical results. Furthermore, data acquisition in some key areas is difficult due to challenges in sensor deployment, high data collection costs, and unstable data quality. At the same time, traditional data collection methods are often costly and constrained by user data privacy issues, making comprehensive data collection difficult. Therefore, how to obtain indoor fire escape route data in a low-cost and efficient manner has become an urgent problem to be solved in the field of urban governance.

[0052] Image fusion technology can integrate image data from different sources to generate an image with richer information and higher resolution. This method not only improves data quality but also reduces data acquisition costs. Existing image fusion technologies typically use fixed parameters when processing images from different sources and of different qualities. This may lead to inaccurate or incomplete datasets in some cases. For example, when the source images have significant differences in quality or features, fixed-parameter fusion methods may not adequately account for these differences, resulting in unsatisfactory fusion results. To ensure the quality of the generated image, the adaptive image fusion technology proposed in this application automatically adjusts the fusion parameters based on the characteristics of images from different sources and of different qualities, thereby generating a more comprehensive and accurate dataset. This method not only improves data quality but also solves the challenge of acquiring data in key areas, providing strong support for urban governance.

[0053] According to an embodiment of this application, a method for generating a dataset in a sparse scene is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0054] This embodiment provides a method for generating a dataset for sparse scenes. Figure 2 is a flowchart of the method for generating a dataset for sparse scenes according to an embodiment of this application. As shown in Figure 2, the process includes the following steps:

[0055] S101, Input the real image of the sparse target scene into the target detection model to obtain the detection boxes of the target foreground and background.

[0056] This application uses an indoor fire escape obstruction scenario as an example. An image of the obstructed indoor fire escape is input into a target detection model to obtain the target obstruction (target foreground) and the indoor staircase (background). The above target detection model is trained through the following process:

[0057] A1 is used to annotate the foreground and background of a small number of images collected from real-world scenes with sparse targets.

[0058] Specifically, for example, in an existing real-world dataset, 100 images can be selected for annotation, with annotation categories including indoor stairs and target obstructions. This is just an example and is not a limitation.

[0059] A2 normalizes the labeled data and combines it with data augmentation techniques to generate a training set.

[0060] Specifically, for example, mosaicking can be used to increase the diversity of training data and expand the dataset. Mosaicking creates new images by stitching together parts of different image categories, allowing the model to learn the relationships between features of different object categories, thereby improving classification accuracy. Since the new image is composed of parts of different images, the model is exposed to more diverse combinations of object features, allowing it to see combinations of objects that are unlikely to occur simultaneously in natural scenes, thus improving the model's ability to recognize complex scenes.

[0061] A3 is trained using a pre-defined machine learning model based on the training set. The trained model is then used as the object detection model, and the object detection model outputs a detection box to detect the foreground and background of the target.

[0062] This application uses the YOLO model as the initial object detection model, and obtains the object detection model after iterative training. There are multiple versions of YOLO; for example, YOLOv5 can be selected, which features small model size, fast training speed, and high accuracy. Before object detection, the input labeled image is preprocessed, typically including image scaling and normalization. The preprocessed image is then input into the model for detection, and the model outputs information such as the category of the detected target object, the coordinates of the detection box, and the confidence score. Reliable object detection results can be filtered based on a confidence threshold; for example, only targets with a confidence score greater than 0.5 are retained.

[0063] This application embodiment only requires processing a small number of images collected from real-world scenarios with sparse targets. Compared to large-scale data collection, this significantly reduces the manpower, material resources, and time costs required in the data collection phase. By labeling the target foreground and background, a clear learning objective is provided for the model. The model can accurately understand which areas are the target foreground and which are the background based on the labeled information, effectively improving the accuracy, efficiency, and generalization ability of target detection.

[0064] S102, using a preset segmentation model, the detection boxes of the target foreground and background are used as prompt words for model inference to obtain the masks of the target foreground and background respectively, and the target foreground image and background region coordinate information are obtained based on the masks of the target foreground and background.

[0065] Specifically, there are currently several excellent large-scale image segmentation models available, such as Mask R-CNN, U-Net and its variants, FastSAM, and the DeepLab series. This application uses the FastSAM model as the large-scale segmentation model. Prepared prompts containing target foreground and background detection box information are provided as input to the large-scale segmentation model, and the target foreground and background masks are extracted from the model's output. To obtain the target foreground image, the original image can be processed using the mask information. The original image is multiplied element-wise with the foreground mask, ensuring that only pixels in the foreground region are retained. The result is then converted into a suitable image data type. The coordinate information of the background region can be obtained based on the background mask by traversing the pixels in the mask to find the pixel coordinates with a mask value of a specific value (e.g., 1, indicating a background region).

[0066] This application embodiment utilizes an object detection model to provide the approximate location of the target by outputting a detection box. By segmenting the image using a large segmentation model, the segmentation can be further refined to the pixel level, generating a mask to accurately divide the target foreground and background. This combination allows for a deeper understanding of the image and enables better application to subsequent processing tasks.

[0067] S103, using adaptive image fusion technology, integrate the target foreground image and background region coordinate information to generate a new target sparse scene image, and add the newly generated target sparse scene image to the training dataset of the target detection model.

[0068] Specifically, adaptive image fusion technology aims to dynamically adjust the fusion strategy based on the characteristics and needs of different image regions to achieve a more natural and target-compliant image synthesis effect. When integrating the coordinate information of the target foreground image and background region, it can rationally determine the fusion method based on the characteristics of the background region and the features of the foreground image, generating a new target sparse scene image. This newly generated image is then added to the training dataset according to its format and storage method. If the image is stored in folders and annotation files, the filename of the new image must first conform to the dataset's naming conventions before being copied or moved to the corresponding training image folder. Simultaneously, corresponding annotation files need to be generated for the new image, with the annotation content determined based on the new image's role and features within the entire training task.

[0069] The sparse scene dataset generation method provided in this application, through object detection technology, can accurately identify and locate target objects in images, providing effective cue words for large-scale model segmentation. Combined with large-scale model segmentation technology, it can accurately segment both target objects and background. Furthermore, it can adaptively and flexibly adjust the fusion method based on images from different sources and of different qualities, improving the realism of the fused images. It requires only a small number of target object and background images to generate a large number of new high-quality images, solving the problem of scarce scene data and providing rich data resources for subsequent model training.

[0070] This embodiment provides a method for generating a dataset for sparse scenes. Figure 3 is a flowchart of the method for generating a dataset for sparse scenes according to an embodiment of this application. As shown in Figure 3, the process includes the following steps:

[0071] S201, input the real-world image of the sparse target scene into the target detection model to obtain the detection bounding boxes of the target foreground and background; see the relevant description of step S101 for details, which will not be repeated here.

[0072] S202, using a preset segmentation model, the detection boxes of the target foreground and background are used as prompt words for model inference to obtain the masks of the target foreground and background respectively, and the target foreground image and background region coordinate information are obtained based on the masks of the target foreground and background; specifically including the following steps:

[0073] S2021, using a pre-defined segmentation model, the detection boxes of the target foreground and background are used as prompt words for model inference, and the masks of the target foreground single-channel image and the background single-channel image are obtained respectively.

[0074] Specifically, the preceding object detection steps (e.g., using the YOLO model) yielded bounding boxes for the foreground and background of the target. This bounding box information is typically provided in coordinate form; for example, a bounding box might have top-left corner coordinates (x1, y1) and bottom-right corner coordinates (x2, y2), along with the corresponding target category. The bounding box coordinates and related categories are then organized into a suitable tensor or data structure and input into the FastSAM model to obtain the mask m of the target obstruction image. obj and the mask m for indoor stairs bkg It should be noted that a mask is a binary image (usually with pixel values ​​of only 0 and 1) or a grayscale image (pixel values ​​within a certain range, such as 0 to 255), where each pixel corresponds to a specific position in the original image. In a binary mask, regions with a pixel value of 1 typically represent regions of interest, i.e., regions that require specific processing (such as extraction, analysis, or modification); regions with a pixel value of 0 represent regions that do not require this processing. The mask obtained in this embodiment is a binary image.

[0075] S2022, the mask of the single-channel image of the target foreground is expanded to mask data with the same number of channels as the original real-world image, and the segmented target foreground image is obtained based on the original real-world image and the mask data with the same number of channels as the original real-world image, and the width and height values ​​of the target object in the target foreground image are calculated.

[0076] Specifically, the single-channel mask data m of the target obstruction obj Copy and concatenate to the original mask m obj Above, we obtain mask data m′ with the same number of channels as the original, real-world image. obj The pixel value of the target blockage at each pixel is obtained using the following formula: Target blockage = m′ obj ×Original, authentically captured image

[0077] The target obstruction is then segmented from the original image to obtain a segmented foreground image, and the width w of the target object is calculated. o and high h o .

[0078] This application's embodiments expand the single-channel image mask of the target foreground into mask data with the same number of channels as the original, realistically acquired image. Then, based on the original image and this mask data, a segmented target foreground image is obtained. This allows for accurate extraction of the target foreground from the original image while preserving the original channel information. For example, for an RGB color image, the expanded mask ensures the correct extraction of the RGB pixel values ​​of the target foreground portion, thus obtaining a complete target foreground image, facilitating subsequent independent analysis and processing of the target foreground.

[0079] S2023, obtain the minimum bounding rectangle corresponding to the background single-channel image, and obtain the center coordinates, width value and height value of the minimum bounding rectangle.

[0080] Specifically, information about the minimum bounding rectangle helps in analyzing the spatial relationship between the background and the target foreground. For example, by comparing the position and size of the minimum bounding rectangles of the target foreground and the background, we can assess the target's relative position in the background (whether it is in the center or at the edge) and its proportion (the size ratio of the target relative to the background).

[0081] This application embodiment obtains a single-channel background image and an indoor staircase mask. bkg The coordinate set (X, Y) with a median value of 1 encloses the area of ​​the stairs in the original image. Then, the minimum bounding rectangle of this coordinate set is calculated, and the center coordinates (x, y) of this rectangle are obtained. c ,y c ) and width wb and high h b If the width or height of the target blockage is greater than the width or height of the indoor staircase, the merged blockage will exceed the original image. Therefore, the size of the target blockage needs to be scaled, and the width w should be updated after scaling. o and high h o The scaling factor is calculated as follows: scale = max(w b / w o ,w o / w b )

[0082] This application's embodiments scale the target object based on a scaling factor, ensuring that the target object can be reasonably placed within the background area. Scaling the target object makes the proportion between the target and the background more harmonious. In a dataset containing multiple target and background combinations, if the size of the target object is too large, it may lead to data inconsistency. By scaling the target object, all data can be made more uniform in size, facilitating unified processing and analysis.

[0083] S203 utilizes adaptive image fusion technology to integrate the coordinate information of the target foreground image and the background region, generating a new sparse scene image of the target, and then adds the newly generated sparse scene image of the target to the training dataset. Specifically, it includes the following steps:

[0084] S2031, perform data augmentation processing on the target foreground image to introduce new features and obtain a target foreground image set; specifically, for example, perform data augmentation processing operations such as horizontal flipping, adjusting brightness and adding artificial noise on the target image to introduce new features into the image, thereby enriching the data of the target blockage.

[0085] S2032, the fusion position range is determined randomly based on the center coordinates of the minimum bounding rectangle;

[0086] In this embodiment, the coordinates of the fusion position (the center coordinates of the smallest bounding rectangle) (x, y) are selected randomly, within the following range: x b +w o / 2≤x≤x b +w b -w o / 2 y b +h o / 2≤y≤y b +h b -h o / 2

[0087] Where, x b and y b The horizontal and vertical coordinates of the background area are represented respectively. The random selection method above is only an example and is not a limitation.

[0088] S2033, based on the fusion location range, the target foreground image set is superimposed and fused with the real-acquired background image to obtain a fused image set, and the divergence value of each image in the fused image set in the fusion region is calculated.

[0089] Specifically, before overlaying and fusion, some preprocessing operations may be needed on the target foreground image set and the background image to ensure they are more compatible in terms of size, color space, etc., facilitating better fusion. Based on the determined fusion location range, each image in the target foreground image set is overlaid and fused with the background image. Taking a rectangular fusion region as an example, assuming the target foreground image is I... f The background image is I b The top-left corner of the fusion region is (x1, y1), and the bottom-right corner is (x2, y2). Overlay fusion can be achieved through the following steps:

[0090] First, create an all-zero image with the same dimensions as the background image as a temporary image I for fusion. temp Then the target foreground image I f Copy the pixel values ​​of the corresponding blending region to I temp At the same position; finally, it will be compared with the background image I. b Perform element-wise addition (or other operations such as weighted addition depending on the specific fusion strategy) to obtain the fused image f.

[0091] Furthermore, the divergence value of each fused image f in the fused image set within the fused region S is calculated using the following formula: △f(x,y)=f(x-1,y)+f(x+1,y)+f(x,y-1)+f(x,y+1)-4f(x,y)

[0092] Divergence is a measure of the degree of divergence of a vector field. In the context of image fusion, it reflects the variation of pixel values ​​within the fusion region. The magnitude and distribution of divergence values ​​play a crucial role in indicating the effectiveness of image fusion. When the divergence value in the fusion region is low, it usually means that the pixel value variation is relatively flat, indicating a more natural transition between the foreground and background in the fusion region.

[0093] S2034, Images with a divergence value less than or equal to the first threshold in the fused image set are used as the first fused image set.

[0094] In one embodiment, for example, if the first threshold is 1, and the divergence value of the fused image is less than or equal to 1, it indicates that the fusion effect is relatively good and no further processing is required.

[0095] S2035, the images with a divergence value greater than the first threshold and less than the second threshold in the fused image set are processed by a preset fusion method and used as the second fused image set.

[0096] In one embodiment, for example, if the second threshold is 10, it indicates that the quality of the fused image in step S2033 is poor and further processing is needed to improve the image quality. This embodiment uses Poisson fusion to make the transition between the fused region and the surrounding environment in terms of color and texture very natural, without obvious splicing, color difference, or other problems. Poisson fusion is based on the Poisson equation to achieve image fusion. Its core idea is to ensure that the boundary conditions of the fused region remain unchanged, so that the gradient field of the fused image matches the source and target images as closely as possible, thereby achieving a natural fusion effect.

[0097] S2036, the images with a divergence value greater than the second threshold in the fused image set are input into the fine-tuned diffusion model to obtain the third fused dataset.

[0098] In this embodiment, images with a divergence value greater than 10 from the fused image set are input into a fine-tuned diffusion model to obtain a third fused dataset. The process of fine-tuning the diffusion model includes:

[0099] B1, annotates the foreground and background of a small number of images (e.g., using 100 real-world labeled images) of the sparse target scene; gradually adds Gaussian noise to the labeled images until the final pure noise image is obtained.

[0100] This step is the forward process of the diffusion model: the forward diffusion process is defined as q(x) t |x t-1 This involves gradually adding Gaussian noise to the image until pure noise is obtained. A series of coefficients are defined: 0 < β1 < ... < 1.

[0101] in, It follows a normal distribution.

[0102] B2. Input the image with added noise into the preset noise prediction network, and train the model using the preset loss function to obtain the fine-tuned diffusion model.

[0103] Specifically, this process is the reverse of the diffusion model. In this embodiment, the UNet model is used as the noise prediction network. The UNet network needs to predict the noise data; by subtracting the network-predicted noise from the image with applied noise, the true image can be generated. The noise output by the UNet network is ∈ θ (x t The actual noise is taken from a normal distribution. The loss function is then defined as:

[0104] The training mechanism of diffusion models is typically based on learning and restoring noise. By adding Gaussian noise until a purely noisy image is obtained, a complete noise learning path is provided for the model. This helps the model understand the process of an image going from its original state to being completely destroyed by noise, thus better learning how to reverse-engineer high-quality images. It should be noted that the diffusion model is a relatively mature model, and the embodiments in this application have fine-tuned it for high-quality image generation in sparse scenes.

[0105] This application embodiment utilizes a small, pre-defined number of images realistically captured from the target sparse scene for annotation, ensuring that the training data closely revolves around the target scene. This ensures that the diffusion model focuses on learning the features of the target foreground and background within that specific scene, avoiding interference from irrelevant data and thus accurately adapting to the needs of sparse target scenes. A pre-defined loss function provides a clear optimization objective for model training; it measures the difference between the noise predicted by the model and the actual added noise, guiding the model to adjust parameters through backpropagation to minimize this difference. This ensures that the model learns in the correct direction during fine-tuning, effectively updating model parameters and thereby improving model performance and generation quality.

[0106] S2037, the first fused image set, the second fused image set, and the third fused image set are added to the training dataset as newly generated target sparse scene images.

[0107] Specifically, it depends on the format and storage method of the training dataset. For example, if the training dataset is stored in image folders and annotation files, the new images need to be copied or moved to the corresponding folders and appropriate annotation files need to be generated for them. If a specific data format is used (such as TensorFlow's TFRecord format, PyTorch's Dataset format, etc.), the new images and their related information need to be encoded and added to the existing data files according to the corresponding specifications.

[0108] Through the above steps, the entire process from image fusion and divergence value calculation to adaptive image fusion technology processing based on different divergence value ranges is completed, and finally the processed image set is added to the training dataset. This helps to further optimize the image fusion effect and enrich the content of the training dataset.

[0109] This embodiment also provides a dataset generation system for sparse scenes, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0110] This embodiment provides a dataset generation system for sparse scenes, as shown in Figure 4, including:

[0111] The detection box output module 401 is used to input the image of the target sparse scene into the target detection model to obtain the detection boxes of the target foreground and background;

[0112] The image segmentation module 402 is used to use a preset segmentation model to perform model inference by using the detection boxes of the target foreground and background as prompt words, to obtain the masks of the target foreground and background respectively, and to obtain the target foreground image and background region coordinate information based on the masks of the target foreground and background.

[0113] The adaptive fusion module 403 is used to integrate the target foreground image and background region coordinate information using adaptive image fusion technology to generate a new target sparse scene image, and to add the newly generated target sparse scene image to the training dataset of the target detection model.

[0114] In some optional implementations, the target detection model in the detection box output module 401 is trained through the following process:

[0115] Annotate the foreground and background of a small number of images realistically captured from sparse target scenes;

[0116] The labeled data is normalized and combined with data augmentation techniques to generate a training set.

[0117] Based on the training set, a preset machine learning model is trained, and the trained model is used as an object detection model. The object detection model outputs a detection box to detect the foreground and background of the target.

[0118] In some alternative implementations, the image segmentation module 402 includes:

[0119] The segmentation large model inference unit is used to use the detection boxes of the target foreground and background as prompt words to perform model inference using a preset segmentation large model, and to obtain the mask of the target foreground single-channel image and the mask of the background single-channel image respectively.

[0120] The target object size acquisition unit is used to expand the mask of the single-channel image of the target foreground into mask data with the same number of channels as the original real-acquired image, and obtain the segmented target foreground image based on the original real-acquired image and the mask data with the same number of channels as the original real-acquired image, and calculate the width and height values ​​of the target object in the target foreground image;

[0121] The background region coordinate information acquisition unit is used to obtain the minimum bounding rectangle corresponding to the background single-channel image, and to obtain the center coordinates, width value and height value of the minimum bounding rectangle.

[0122] In some optional implementations, when the width or height of the target object is greater than the width or height of the background, the size of the target object is scaled based on a scaling factor, and the width and height of the target object are updated after scaling.

[0123] In some alternative implementations, the adaptive fusion module 403 includes:

[0124] The target foreground image set acquisition unit is used to perform data augmentation processing on the target foreground image to introduce new features and obtain the target foreground image set.

[0125] The fusion location range acquisition unit is used to determine the fusion location range randomly based on the center coordinates of the minimum bounding rectangle.

[0126] The divergence value calculation unit is used to overlay and fuse the target foreground image set with the real-acquired background image based on the fusion location range to obtain a fused image set, and calculate the divergence value of each image in the fused image set in the fusion region.

[0127] The first fused image set acquisition unit is used to select images in the fused image set whose divergence value is less than or equal to a first threshold as the first fused image set;

[0128] The second fused image set acquisition unit takes the images in the fused image set whose divergence values ​​are greater than the first threshold and less than the second threshold, and processes them using a preset fusion method to obtain the second fused image set.

[0129] The third fusion image set acquisition unit inputs images with a divergence value greater than the second threshold into the fine-tuned diffusion model to obtain the third fusion dataset.

[0130] The training dataset expansion unit is used to add the first fused image set, the second fused image set, and the third fused image set as newly generated target sparse scene images to the training dataset.

[0131] In some alternative implementations, the process of fine-tuning the diffusion model includes:

[0132] A small number of images of a sparse target scene are collected and labeled with target foreground and background. Gaussian noise is gradually added to the labeled images until the final image with pure noise is obtained.

[0133] The image with added noise is input into a preset noise prediction network, and the model is trained using a preset loss function to obtain a fine-tuned diffusion model.

[0134] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0135] In this embodiment, the sparse scene dataset generation system is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0136] This application also provides a computer device having a dataset generation system for the sparse scene shown in Figure 4 above.

[0137] Please refer to Figure 5, which is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application. As shown in Figure 5, the computer device includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 uses one processor 10 as an example.

[0138] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0139] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0140] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0141] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0142] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0143] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor central control systems, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0144] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0145] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification. Although embodiments of this application have been described in conjunction with the accompanying drawings, various modifications and variations can be made by those skilled in the art without departing from the spirit and scope of this application, and such modifications and variations fall within the scope defined by the appended claims.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating datasets for sparse scenes, characterized in that, include: The target detection model is input into the image of the sparse target scene to obtain the detection bounding boxes of the target foreground and background; Using a pre-defined segmentation model, the detection boxes of the target foreground and background are used as prompt words for model inference to obtain the masks of the target foreground and background respectively, and the target foreground image and background region coordinate information are obtained based on the masks of the target foreground and background. Adaptive image fusion technology is used to integrate the target foreground image and background region coordinate information to generate a new target sparse scene image, and the newly generated target sparse scene image is added to the training dataset of the target detection model.

2. The method according to claim 1, characterized in that, The target detection model is trained through the following process: Annotate the foreground and background of a small number of images realistically captured from sparse target scenes; The labeled images are normalized and combined with data augmentation techniques to generate a training set; Based on the training set, a preset machine learning model is trained, and the trained model is used as an object detection model. The object detection model outputs a detection box to detect the foreground and background of the target.

3. The method according to claim 1, characterized in that, The process involves using a pre-defined segmentation model to perform model inference on the detection boxes of the target foreground and background as prompt words, obtaining masks for the target foreground and background respectively, and obtaining the target foreground image and background region coordinate information based on the masks, including: Using a pre-defined segmentation model, the detection boxes of the target foreground and background are used as prompt words for model inference, and the masks of the single-channel images of the target foreground and the single-channel images of the background are obtained respectively. The mask of the single-channel image of the target foreground is expanded to mask data with the same number of channels as the original real-world image. Based on the original real-world image and the mask data with the same number of channels as the original real-world image, the segmented target foreground image is obtained, and the width and height values ​​of the target object in the target foreground image are calculated. Obtain the minimum bounding rectangle corresponding to the background single-channel image, and obtain the center coordinates, width, and height of the minimum bounding rectangle.

4. The method according to claim 3, characterized in that, The method further includes: When the width or height of the target object is greater than the width or height of the background, the size of the target object is scaled based on a scaling factor, and the width and height of the target object are updated after scaling.

5. The method according to claim 3, characterized in that, The method of integrating the target foreground image and background region coordinate information using adaptive image fusion technology to generate a new target sparse scene image, and adding the newly generated target sparse scene image to the training dataset of the target detection model, includes: The target foreground image is subjected to data augmentation processing to introduce new features, resulting in a target foreground image set. The fusion position range is determined randomly based on the center coordinates of the minimum bounding rectangle. Based on the fusion location range, the target foreground image set is superimposed and fused with the real-acquired background image to obtain a fused image set, and the divergence value of each image in the fused image set in the fusion region is calculated. Images with a divergence value less than or equal to the first threshold in the fused image set are used as the first fused image set; Images with a divergence value greater than the first threshold and less than the second threshold in the fused image set are processed using a preset fusion method and used as the second fused image set. The images with a divergence value greater than the second threshold in the fused image set are input into the fine-tuned diffusion model to obtain the third fused dataset. The first fused image set, the second fused image set, and the third fused image set are added to the training dataset of the target detection model as newly generated target sparse scene images.

6. The method according to claim 5, characterized in that, The process of fine-tuning the diffusion model includes: Annotate the foreground and background of a small number of images realistically captured from sparse target scenes; Gaussian noise is gradually added to the labeled image until the final image with only noise is obtained; The image with added noise is input into a preset noise prediction network, and the model is trained using a preset loss function to obtain a fine-tuned diffusion model.

7. A dataset generation system for sparse scenes, characterized in that, include: The detection box output module is used to input the real-world image of the sparse target scene into the target detection model to obtain the detection boxes for the target foreground and background; The image segmentation module is used to use a preset segmentation model to perform model inference by using the detection boxes of the target foreground and background as prompt words, and to obtain the masks of the target foreground and background respectively, and to obtain the target foreground image and background region coordinate information based on the masks of the target foreground and background. The adaptive fusion module is used to integrate the target foreground image and background region coordinate information using adaptive image fusion technology to generate a new target sparse scene image, and to add the newly generated target sparse scene image to the training dataset of the target detection model.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the dataset generation method for sparse scenes according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the dataset generation method for sparse scenes according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the dataset generation method for sparse scenes according to any one of claims 1-6.

Images