Method for selecting training data for training deep learning model, and training data selection device using same

Abstract

The present invention relates to a method for selecting training data for training a deep learning model, the method comprising: with reference to a bipartite graph in which each of multiple training images stored in a data pool and individual types corresponding thereto are matched, performing a cycle of selecting some training images matching all of the individual types by repeating a process of (i) identifying, among the individual types, the number of corresponding individual types matching each of the training images, and selecting a specific training image having the largest number of corresponding individual types, and (ii) identifying, among individual types remaining after excluding the corresponding individual types, the number of the remaining corresponding individual types matching each of the training images, and selecting another specific training image having the largest number of the remaining corresponding individual types; and repeating the process of performing the cycle until n training images for training the deep learning model are selected from among training images remaining after excluding some training images selected from among the multiple training images.

Description

Method for selecting training data for training a deep learning model and device for selecting training data using the same

[0001] The present invention relates to a method for selecting training data for training a deep learning model evenly from all training data stored in a data pool without data bias and imbalance, and a training data selection device using the same.

[0002] In general, deep learning models recognize complex patterns in images, text, sound, and other data to generate accurate insights and predictions, and are being applied in various fields such as computer vision, speech recognition, autonomous vehicles, robotics, natural language processing, and medical image analysis.

[0003] In order for such deep learning models to accurately perform the intended task, deep learning models must be trained using a large amount of training data.

[0004] Conventional methods for selecting training data for training a deep learning model from a collected data pool include random sampling, which selects a target number of training data from the entire training data stored in the data pool, and vector quantization, which clusters and groups vectors representing each of the entire training data generated through embedding extraction and selects representative values ​​for each group of the grouped vectors.

[0005] However, when selecting training data using conventional methods, there is a problem of type-specific bias and imbalance.

[0006] For example, if 1 million training images are stored in a data pool, and among the 1 million images, 70% are training images related to clear weather, 20% are training images related to cloudy weather, 5% are training images related to foggy weather, and 5% are training images related to snow and / or rain, then when 10,000 training images are randomly sampled, only about 500 training images are probabilistically selected out of a total of 50,000 training images related to snow and / or rain, so bias and imbalance occur in the training images selected according to weather-related types.

[0007] Furthermore, when selecting training images using vector quantization, type-specific bias and imbalance among training images selected by embedding extraction and clustering may be somewhat mitigated, but problems related to data bias and imbalance cannot be fundamentally prevented.

[0008] Accordingly, the applicant intends to propose a method to enable the uniform selection of training data for training a deep learning model from the entire training data stored in a data pool, without bias or imbalance by type.

[0009] The present invention aims to solve all of the aforementioned problems.

[0010] Another objective of the present invention is to enable the selection of training data for training a deep learning model from a data pool to be unbiased toward a specific type and to be evenly distributed by type.

[0011] Another objective of the present invention is to enable the equal selection of training data suitable for various scenarios according to the types of training data in a data pool.

[0012] The characteristic configuration of the present invention for achieving the objectives of the present invention as described above and realizing the characteristic effects of the present invention described below is as follows.

[0013] According to one embodiment of the present invention, a method for selecting training data for training a deep learning model comprises: (a) a training data selection device, wherein each of at least one individual attribute corresponding to each of a plurality of training images stored in a data pool, and a bipartite graph that matches each of the plurality of training images with the individual attributes; and (b) a method comprising the step of the learning data selection device performing a cycle to select some learning images that match all of the individual types by repeating the process of (i) checking the number of corresponding individual types that match each of the learning images among the individual types by referring to the bipartite graph, and selecting a specific learning image that has the largest number of corresponding individual types, and (ii) checking the number of remaining corresponding individual types that match each of the learning images among the remaining individual types excluding the corresponding individual types, and selecting another specific learning image that has the largest number of remaining corresponding individual types, and performing the process of performing the cycle on the remaining learning images excluding the selected some learning images from the plurality of learning images until n learning images for training a deep learning model are selected - wherein n is the target number of learning images for training the deep learning model and is an integer representing a plurality of numbers; is provided.

[0014] In the above embodiment, in step (b), the individual types include a first_1 individual type to a first_x individual type corresponding to a first type of each of the training images, wherein x is an integer greater than or equal to 1, and a second_1 individual type to a second_y individual type corresponding to a second type of each of the training images, wherein y is an integer greater than or equal to 1, and the training data selection device may select the n training images such that the number of the first_1 individual types to the first_x individual types corresponding to the first type of the training images and the number of the second_1 individual types to the second_y individual types corresponding to the second type of the training images matched to the n training images are within the threshold deviation, the number of the first_1 individual types to the first_x individual types are within the first threshold deviation, and the number of the second_1 individual types to the second_y individual types are within the second threshold deviation.

[0015] In the above embodiment, in step (b), the learning data selection device may select one of the multiple learning images as a first criterion to select the specific learning image when there are multiple learning images with the largest number of corresponding individual types, and select one of the multiple remaining learning images as a second criterion to select the other specific learning image when there are multiple learning images with the largest number of remaining individual types.

[0016] In the above embodiment, in step (a), the learning data selection device may transmit the learning images to a labeler terminal to enable a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the learning images.

[0017] In the above embodiment, in step (a), the learning data selection device performs a process of performing a first embedding operation on each of the learning images to generate first scene vectors corresponding to each of the learning images and clustering the first scene vectors to generate first scene clusters, or a process of performing a k-th embedding operation on each of the learning images—where k is an integer greater than or equal to 1—to generate k-th scene vectors corresponding to each of the learning images and clustering the k-th scene vectors to generate k-th scene clusters, and can generate individual types corresponding to the learning images by referencing the first scene clusters to the k-th scene clusters.

[0018] In the above embodiment, in step (a), the learning data selection device can generate the individual types corresponding to each of the learning images by checking each of the metadata included in each of the learning images and further referring to each of the shooting times included in each of the metadata.

[0019] In the above embodiment, in step (a), the learning data selection device may (i) perform a specific embedding operation on each of the learning images to generate specific scene vectors corresponding to each of the learning images, and cluster the specific scene vectors to generate specific scene clusters, (ii) check the shooting times of each of the learning images by referring to each of the metadata included in each of the learning images, and (iii) generate the individual types corresponding to each of the learning images by referring to the specific scene clusters and the shooting times.

[0020] According to another embodiment of the present invention, a training data selection device for selecting training data for training a deep learning model comprises: a memory storing instructions for selecting training data for training a deep learning model; and a processor for performing an operation to select training data for training a deep learning model according to the instructions stored in the memory. The process comprises: (I) acquiring at least one individual attribute corresponding to each of a plurality of training images stored in a data pool, and generating a bipartite graph that matches each of the plurality of training images with the individual attributes; and (II) (i) referring to the bipartite graph, identifying the number of corresponding individual attributes among the individual attributes that match each of the training images, and selecting a specific training image with the largest number of corresponding individual attributes, and (ii) repeating the process of identifying the number of remaining corresponding individual attributes among the remaining individual attributes that match each of the training images, and selecting another specific training image with the largest number of remaining corresponding individual attributes, thereby performing a cycle to select some training images that match all of the individual attributes, and performing the cycle on the remaining training images excluding the selected some training images from the plurality of training images. A training data selection device is provided that performs a process of repeating until a training image is selected.

[0021] In the other embodiment above, the processor can select the n training images such that, in the process (II), the individual types include a first_1 individual type to a first_x individual type corresponding to a first type of each of the training images, wherein x is an integer greater than or equal to 1, and a second_1 individual type to a second_y individual type corresponding to a second type of each of the training images, wherein y is an integer greater than or equal to 1, and the number of the first_1 individual types to the first_x individual types corresponding to the first type and the number of the second_1 individual types to the second_y individual types corresponding to the second type of the training images matched to the n training images are within the threshold deviation, the number of the first_1 individual types to the first_x individual types are within the first threshold deviation, and the number of the second_1 individual types to the second_y individual types are within the second threshold deviation.

[0022] In the other embodiment above, the processor may, in the process (II), if there are multiple training images with the largest number of corresponding individual types, select one of the multiple training images as a first criterion to select the specific training image, and if there are multiple remaining training images with the largest number of remaining individual types, select one of the multiple remaining training images as a second criterion to select the other specific training image.

[0023] In the other embodiment above, the processor may, in the process (I), transmit the training images to a labeler terminal so that a labeler corresponding to the labeler terminal may generate at least one individual type corresponding to each of the training images.

[0024] In the other embodiment above, the processor may, in the process (I), perform a first embedding operation on each of the training images to generate first scene vectors corresponding to each of the training images and cluster the first scene vectors to generate first scene clusters, or perform a k-th embedding operation on each of the training images—where k is an integer greater than or equal to 1—to generate k-th scene vectors corresponding to each of the training images and cluster the k-th scene vectors to generate k-th scene clusters, and may generate individual types corresponding to the training images by referencing the first scene clusters to the k-th scene clusters.

[0025] In the other embodiment above, the processor can, in the process (I), check each meta-information included in each of the training images and further refer to each shooting time included in each of the meta-information to generate the individual types corresponding to each of the training images.

[0026] In the other embodiment above, the processor may, in the process (I), (i) perform a specific embedding operation on each of the training images to generate specific scene vectors corresponding to each of the training images, and cluster the specific scene vectors to generate specific scene clusters, (ii) check the shooting times of each of the training images by referring to each of the metadata included in each of the training images, and (iii) generate the individual types corresponding to each of the training images by referring to the specific scene clusters and the shooting times.

[0027] In addition to this, a computer-readable recording medium for recording a computer program for executing the method of the present invention is further provided.

[0028] The present invention enables the selection of training data for training a deep learning model from a data pool evenly by type without being biased toward a specific type.

[0029] The present invention enables the uniform selection of training data suitable for various scenarios according to the types of training data in a data pool.

[0030] The drawings attached below for use in describing embodiments of the present invention are merely some of the embodiments of the present invention, and other drawings can be obtained based on these drawings without inventive work by a person skilled in the art to which the present invention pertains (hereinafter "person skilled in the art").

[0031] FIG. 1 schematically illustrates a training data selection device for selecting training data for training a deep learning model according to an embodiment of the present invention, and

[0032] FIG. 2 schematically illustrates a method for selecting training data for training a deep learning model according to a first embodiment of the present invention, and

[0033] FIG. 3 schematically illustrates an example of generating individual types of training data in the first embodiment of the present invention, and

[0034] FIG. 4 schematically illustrates another example of generating individual types of training data in the first embodiment of the present invention, and

[0035] FIG. 5 exemplarily illustrates a bipartite graph matching each of the training data with individual types in the first embodiment of the present invention, and

[0036] FIGS. 6a to 6d schematically illustrate the process of selecting training data by referring to a bipartite graph in the first embodiment of the present invention, and

[0037] FIG. 7 schematically illustrates a method for selecting training data for training a deep learning model according to a second embodiment of the present invention.

[0038] FIG. 8 schematically illustrates a method for selecting training data for training a deep learning model according to a third embodiment of the present invention, and

[0039] FIG. 9 schematically illustrates a method for selecting training data for training a deep learning model according to a fourth embodiment of the present invention.

[0040] The following detailed description of the present invention refers to the accompanying drawings, which illustrate specific embodiments in which the present invention can be practiced in order to clarify the objects, technical solutions, and advantages of the present invention. These embodiments are described in sufficient detail to enable a person skilled in the art to practice the present invention.

[0041] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the invention is limited only by the appended claims, including all equivalents to those claimed therein, provided appropriately described. Similar reference numerals in the drawings refer to the same or similar functions across various aspects.

[0042] Hereinafter, in order to enable a person skilled in the art to easily practice the present invention, various preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0043] FIG. 1 schematically illustrates a training data selection device for selecting training data for training a deep learning model according to an embodiment of the present invention. The training data selection device (100) may include a memory (110) in which instructions for selecting training data for training a deep learning model are stored, and a processor (120) that performs an operation to select training data for training a deep learning model according to the instructions stored in the memory (110).

[0044] Specifically, the learning data selection device (100) may achieve desired system performance by utilizing a combination of a computing device (e.g., a device that may include components of a computer processor, memory, storage, input device and output device, and other conventional computing devices; an electronic communication device such as a router, switch, etc.; an electronic information storage system such as a Network Attached Storage (NAS) and a Storage Area Network (SAN)) and computer software (i.e., instructions that cause the computing device to function in a specific way), but is not limited thereto.

[0045] Additionally, the processor (120) of the learning data selection device (100) may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Additionally, the network controller (100) may further include software configurations such as an operating system and an application for a specific purpose.

[0046] However, this does not exclude the case where the learning data selection device (100) includes an integrated processor in which a medium, a processor, and a memory are integrated for implementing the present invention.

[0047] Meanwhile, the processor (120) of the training data selection device (100) may perform a process of generating at least one individual type (attribute) corresponding to each of the multiple training data included in the entire training data stored in the data pool according to instructions stored in the memory (110), and generating a bipartite graph that matches each of the multiple training data included in the entire training data with the individual types. Furthermore, the processor (120) of the training data selection device (100) may perform a process of selecting n training data that match the individual types from the entire training data by referring to the bipartite graph according to instructions stored in the memory (110), and selecting n training data for training a deep learning model such that the number of each of the individual types matched to the n training data is within a predetermined threshold deviation. The above n may be a target number of training data for training a deep learning model and may be an integer representing a multiple number.

[0048] A method for selecting training data for training a deep learning model is described as follows with reference to a training data selection device according to one embodiment of the present invention configured as described above.

[0049]

[0050] [1st Example]

[0051] FIG. 2 schematically illustrates a method for selecting training data for training a deep learning model according to a first embodiment of the present invention. Referring to FIG. 2, the first embodiment of the present invention is described as follows.

[0052] First, the training data selection device (100) can acquire (S110) each at least one individual type (attribute) corresponding to each of the multiple training data included in the entire training data stored in the data pool.

[0053] At this time, the individual types may include 1_1 individual types to 1_x individual types corresponding to a 1st type having each of the multiple training data, and 2_1 individual types to 2_y individual types corresponding to a 2nd type having each of the multiple training data. Each of x and y may be an integer greater than or equal to 1. However, the present invention is not limited thereto and may have three or more types depending on the characteristics of the training data.

[0054] For example, if the training data is text data, it can be composed of various individual types corresponding to various types, such as genre-related types corresponding to individual types like essays, novels, reports, travelogues, and letters; posting-related types corresponding to individual types like blogs, newspapers, and social media posts; language-related types corresponding to individual types like Korean, English, and German; and field-related types corresponding to individual types like economic terms, social terms, and mathematical terms.

[0055] As another example, if the training data is audio data, it can be composed of various individual types corresponding to various types, such as type types related to time periods corresponding to individual types like day, night, dawn, and evening; type types related to places corresponding to individual types like city, countryside, mountains, and sea; and type types related to sound sources corresponding to individual types like car sounds, human sounds, bird sounds, and wind sounds.

[0056] As another example, when the training data is image data, it can be composed of various individual types corresponding to various types, such as type types related to time periods corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as clear, cloudy, rain, and fog; and type types related to objects corresponding to individual types such as cars, people, and animals.

[0057] In addition, for each of the multiple training data, at least one individual type (attribute) corresponding to each of the multiple training data can be automatically extracted by a labeler labeling each of the multiple training data with respect to predefined types, or by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0058] For example, a learning data selection device (100) can transmit a plurality of learning data to a labeler terminal so that a labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each of the plurality of learning data.

[0059] As another example, the training data selection device (100) can automatically generate various individual types corresponding to various types of the entire image that each of the training images, which are the entire training data, has.

[0060] That is, referring to FIG. 3, the learning data selection device (100) may perform a process of generating first scene vectors corresponding to each of the learning images by performing a first embedding operation (10_1) on each of the learning images and generating first scene clusters by clustering (20) the first scene vectors, or a process of generating k-th scene vectors corresponding to each of the learning images by performing a k-th embedding operation (10_k) on each of the learning images and generating k-th scene clusters by clustering (20) the k-th scene vectors. The k may be an integer greater than or equal to 1. In addition, the learning data selection device (100) may generate individual types corresponding to the learning images by referencing the first scene clusters to the k-th scene clusters.

[0061] At this time, the training data selection device (100) may use visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to k-th embedding operations, and each of the first embedding model for the first embedding operation to the k-th embedding operation may be a different embedding model that generates scene vectors corresponding to each of the first to k-th type types, which are different types. Also, the training data selection device (100) may use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to k-th scene vectors.

[0062] In addition, although the above method involves performing embedding operations and clustering on each of the training images to generate at least one individual type for each of the training images, alternatively, individual types may be generated using metadata included in each of the training images. For example, when k is 1, that is, when only individual types for one type corresponding to each of the training images are generated through a single embedding operation, individual types for a type different from the one type resulting from the embedding operation may be additionally generated using metadata of each of the training images.

[0063] In other words, by checking the individual metadata included in each of the training images and further referring to the individual shooting times included in each of the metadata, individual types corresponding to each of the training images can be generated.

[0064] For example, in order to generate individual types according to time zone types such as day, night, evening, dawn, etc., the learning data selection device (100) groups the learning images according to preset time zones by referring to the shooting times of each of the learning images, and can set individual types related to the time zone for each group of the grouped learning images.

[0065] In addition, when the learning data selection device (100) generates individual types corresponding to each of the learning images by further referring to the shooting times of each of the learning images, it may generate individual types corresponding to one of the types using the specific embedding model and individual types corresponding to another type using meta-information by using only a specific embedding model which is any one of the first embedding model to the k-th embedding model.

[0066] That is, the learning data selection device (100) can generate specific scene vectors corresponding to each of the learning images by performing a specific embedding operation on each of the learning images, and generate specific scene clusters by clustering the specific scene vectors. In addition, the learning data selection device (100) can check the shooting times of each of the learning images by referring to the respective metadata included in each of the learning images. Subsequently, the learning data selection device (100) can generate individual types corresponding to each of the learning images by referring to the specific scene clusters and the shooting times.

[0067] As another example, the learning data selection device (100) may generate individual types corresponding to objects included in the image, rather than generating various individual types corresponding to various types for the entire image.

[0068] That is, with reference to FIG. 4, the training data selection device (100) performs object detection (1) on each of the training images, which are the entire training data, to detect at least one object in each of the training images, generates cropped images by cropping (2) the regions corresponding to the bounding boxes of each of the detected objects in each of the training images, performs an embedding operation (10) on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters (20) the object vectors to generate object clusters, and generates individual types corresponding to the training images by referencing the object clusters. At this time, to perform object detection on each of the training images, the training data selection device (100) may use an Open-World object detection model such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), or YOLO (You Only Look Once)-World.

[0069] At this time, the learning data selection device (100) may generate individual types corresponding to the learning images by additionally referencing the scene clusters generated according to the above description in addition to the object clusters.

[0070] That is, the learning data selection device (100) can perform a process of generating first scene vectors corresponding to each of the learning images by performing a first embedding operation on each of the learning images and generating first scene clusters by clustering the first scene vectors, or a process of generating k-th scene vectors corresponding to each of the learning images by performing a k-th embedding operation on each of the learning images and generating k-th scene clusters by clustering the k-th scene vectors. In addition, the learning data selection device (100) can perform object detection on each of the learning images to detect at least one object in each of the learning images, generate cropped images by cropping the regions corresponding to the bounding boxes of each of the detected objects in each of the learning images, perform an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, and generate object clusters by clustering the object vectors. Afterwards, the learning data selection device (100) can generate individual types corresponding to the learning images by referring to the first to k scene clusters and object clusters.

[0071] Meanwhile, although the above method generated cropped images after performing object detection on each of the training images, in a different case, if each of the training images has ground truths labeled on it, cropped images can be generated by referencing the ground truths of each of the training images.

[0072] That is, the learning data selection device (100) can generate cropped images by cropping regions corresponding to bounding boxes of objects in each of the learning images by referring to ground truth information included in each of the learning images, generate object vectors corresponding to each of the cropped images by performing an embedding operation on each of the cropped images, generate object clusters by clustering the object vectors, and generate individual types corresponding to the learning images by referring to the object clusters.

[0073] At this time, the learning data selection device (100) may generate individual types corresponding to the learning images by additionally referencing the scene clusters generated according to the above description in addition to the object clusters.

[0074] That is, the learning data selection device (100) can perform a process of generating first scene vectors corresponding to each of the learning images by performing a first embedding operation on each of the learning images and generating first scene clusters by clustering the first scene vectors, or a process of generating k-th scene vectors corresponding to each of the learning images by performing a k-th embedding operation on each of the learning images and generating k-th scene clusters by clustering the k-th scene vectors. In addition, the learning data selection device (100) can generate cropped images by cropping regions corresponding to the bounding boxes of each object in each of the learning images by referring to each ground truth information included in each of the learning images, generate object vectors corresponding to each of the cropped images by performing an embedding operation on each of the cropped images, and generate object clusters by clustering the object vectors. Afterwards, the learning data selection device (100) can generate individual types corresponding to the learning images by referring to the first to k scene clusters and object clusters.

[0075] Next, referring again to FIG. 2, the training data selection device (100) can generate a bipartite graph (S120) that matches each of the multiple training data included in the entire training data with individual types.

[0076] For example, referring to FIG. 5, the bipartite graph matches the relationship between the training data group and the individual type group, and it can be seen that training data 1 has individual type 1 and individual type 6, training data 2 has individual type 1, individual type 5, and individual type 6, training data 3 has individual type 2 and individual type 7, training data 4 has individual type 1, training data 5 has individual type 3, individual type 6, and individual type 8, training data 6 has individual type 4, and training data 7 has individual type 2, individual type 5, individual type 7, and individual type 8. Meanwhile, FIG. 5 illustrates some training data and some individual types in the bipartite graph.

[0077] Next, referring again to FIG. 2, the training data selection device (100) may select n training data that match individual types from the entire training data by referring to a bipartite graph, and select n training data for training a deep learning model such that the number of each individual type that matches the n training data is within a predetermined threshold deviation (S130). The n is the target number of training data for training a deep learning model and may be an integer representing a multiple number.

[0078] At this time, when individual types include 1_1 individual types to 1_x individual types corresponding to a 1st type type each of the multiple training data and 2_1 individual types to 2_y individual types corresponding to a 2nd type type each of the multiple training data, the training data selection device (100) can ensure that the number of 1_1 individual types to 1_x individual types corresponding to a 1st type type and the number of 2_1 individual types to 2_y individual types corresponding to a 2nd type type matched to n training data are within a threshold deviation. In addition, the number of 1_1 individual types to 1_x individual types is within a 1st threshold deviation, and the number of 2_1 individual types to 2_y individual types is within a 2nd threshold deviation. That is, the training data selection device (100) can select training data such that the number of training data for each individual type is as even as possible for all individual types of each type.

[0079] Meanwhile, the learning data selection device (100) may use at least some of heuristic algorithms, optimization algorithms, etc., to select the number of learning data for each individual type so that they are as even as possible.

[0080] First, the process of the learning data selection device (100) selecting learning data using a heuristic algorithm is described as follows.

[0081] The training data selection device (100) can check the number of corresponding individual types that match each of the total training data among the individual types by referring to the bipartite graph, and select the specific training data with the largest number of corresponding individual types.

[0082] And, the learning data selection device (100) can perform the process of checking the number of remaining individual types that match each of the total learning data among the remaining individual types excluding the above-mentioned individual types, and selecting another specific learning data in which the number of remaining individual types is the largest.

[0083] In this way, by repeating the process of selecting training data with the largest number of remaining individual types among the individual types remaining after excluding the selected individual types until all individual types are selected, the training data selection device (100) can perform a cycle of selecting some training data that matches all individual types.

[0084] Afterward, the learning data selection device (100) can repeat the process of performing the above cycle on the remaining learning data, excluding some selected learning data that matches all individual types from the entire learning data, until n learning data are selected.

[0085] For example, referring to FIGS. 6a to 6d, since training data 7 in FIG. 6a is matched to the four individual types with the most, the training data selection device (100) can select training data 7 from training data 1 to training data 7.

[0086] And, if individual types 2, 5, 7, and 8 matched to training data 7 are excluded from the bipartite graph of Fig. 6a, individual types 1, 3, 4, and 6 remain as shown in Fig. 6b.

[0087] Then, since training data 1, training data 2, and training data 5 in FIG. 6b are matched to the two individual types with the most, the training data selection device (100) can select training data 1, which is one of training data 1, training data 2, and training data 5. At this time, if there are multiple training data such as training data 1, training data 2, and training data 5, which have the largest number of corresponding individual types, the training data selection device (100) can select training data 1, which is a specific training data, by selecting one of the multiple training data, training data 1, training data 2, and training data 5, as a first criterion. And, the first criterion can be set in various ways, such as selecting one of the multiple training data by a random selection algorithm, or selecting one of the multiple training data according to a sequential ranking of the multiple training data.

[0088] And, if individual type 1 and individual type 6 matched to training data 1 are excluded from the bipartite graph of Fig. 6b, individual type 3 and individual type 4 remain as shown in Fig. 6c.

[0089] Then, since training data 5 and training data 6 in FIG. 6c are matched to the one individual type with the largest number, the training data selection device (100) can select either training data 5 or training data 6. At this time, if there are multiple remaining training data such as training data 5 and training data 6, which have the largest number of remaining corresponding individual types, the training data selection device (100) can select a specific training data, training data 5, by selecting one of the multiple remaining training data, training data 5 and training data 6, as a second criterion. The second criterion can be set in various ways, such as selecting one of the multiple training data by a random selection algorithm or selecting one of the multiple training data according to a sequential ranking, and the second criterion can be set to be the same as or different from the first criterion above.

[0090] And, if individual type 3 matched to training data 5 is excluded from the bipartite graph of Fig. 6c, individual type 4 remains as shown in Fig. 6d.

[0091] Then, since the training data 6 in FIG. 6d is matched to the one individual type with the most, the training data selection device (100) selects the training data 6, and accordingly, it is possible to perform one cycle of selecting selected training data that matches all individual types.

[0092] Afterwards, the learning data selection device (100) can select n learning data, which is the target number, by repeating the above cycle from the remaining learning data, excluding some learning data selected from the entire learning data, namely learning data 1, learning data 5, and learning data 7.

[0093] Meanwhile, although the above description explains the selection of some training data that matches all individual types, training data can be selected by further referring to the types to which each individual type belongs. That is, some training data that matches all individual types belonging to all types can be selected. For example, when selecting some training data from the day and night time zone types and the car and person object types, it is possible to select some training data that matches the individual types of day and night, but is matched evenly; that matches the individual types of cars and people within the individual types of day, but is matched evenly; and that matches both the individual types of cars and people within the individual types of night, but is matched evenly.

[0094] Next, the process of the learning data selection device (100) selecting learning data using an optimization algorithm is described as follows.

[0095] The training data selection device (100) can select a specific subset with the smallest number of training data among subsets consisting of a predetermined number of training data including all individual types by referring to a bipartite graph and using an optimization algorithm, calculate the remaining training data excluding a specific number of training images included in the specific subset, and repeat the process of selecting at least one other specific subset consisting of a predetermined number of training data including all individual types for the remaining training data by using an optimization algorithm, so that the number of selected training data is n or more. At this time, ensuring that the number of selected training data is n or more, which is the target number, does not mean that the number of selected training data is unconditionally n or more, but may mean that the total number of training data finally selected does not become less than n depending on the number of the last selected specific subset. That is, when the number of training data selected so far is less than n, and the total number of training data selected by summing the number of specific subsets selected so far becomes n or more, the process of selecting training data is terminated, and the total number of training data can be generated as a training data set for training a deep learning model.

[0096] For example, a training data selection device (100) can select a specific subset of training data corresponding to specific selection fitness variables having a minimum sum of selection fitness variables among the selection fitness variables of a Q-dimensional vector in which the sum of selection fitness variables has a minimum value, and the total fitness sums in the P-dimensional vector generated by matrix multiplying a PxQ binary matrix corresponding to P individual types and Q training data in a bipartite graph and a Q-dimensional vector representing selection fitness variables for each of the Q training data in each of the P individual types are greater than or equal to 1, and the selection fitness variables in the Q-dimensional vector satisfy being greater than or equal to 0 and less than or equal to 1, and can calculate the remaining training data from the Q training data excluding the specific training data included in the specific subset. The P-dimensional vector may represent the total fitness sums of the Q training data belonging to each of the P individual types. And, the learning data selection device (100) can repeat the process of selecting at least one other specific subset by linear programming for the remaining learning data so that the number of selected learning data is n or more.

[0097] At this time, linear programming can be expressed as follows.

[0098] Objective: min 1 T x

[0099] Condition: Ax ≥ 1,

[0100] 0 ≤ x ≤ 1

[0101] In the above, A is a PxQ binary matrix, and A[j,i] can indicate whether training data i matches individual type j, and

[0102] x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable of training data i, and

[0103] Ax is the matrix product of A and x, and Ax[j] can represent the sum of the goodness-of-fit of the training data belonging to individual types j.

[0104] Additionally, the learning data selection device (100) may select a specific subset by a dual linear programming method that applies at least one of merging, separating, and changing the sign of the constraints in the linear programming method.

[0105] The dual linear programming method modified from the above linear programming method can be represented as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the above linear programming method.

[0106] Objective: min 1 T x

[0107] Condition: Bx ≥ b,

[0108] x ≥ 0

[0109] At this time, B=[A;-I], (P+Q)xQ binary matrix,

[0110] B[j,i]=A[j,i],

[0111] B[P+i,i] = -1 for all i ∈ [1,2,캜,Q],

[0112] B[P+i,j] = 0 for all j≠i

[0113] b=[1 P ;-1 P ], (P+Q)-dimensional vector,

[0114] b[j] = 1 for all j ∈ [1,2,j,P],

[0115] b[P+i] = -1 for all i ∈ [1,2,x,Q]

[0116] It can be represented as follows.

[0117] As another example, a learning data selection device (100) can select a specific subset of learning data corresponding to specific selection variables having a minimum sum of selection variables among the selection variables of a Q-dimensional vector in which the selection quantities are 1 or greater and the selection variables in the Q-dimensional vector satisfy 0 or 1, by matrix multiplying a PxQ binary matrix corresponding to P individual types and Q learning data in a bipartite graph and a Q-dimensional vector representing selection variables of each of the Q learning data in each of the P individual types through integer programming as an optimization algorithm, and can produce the remaining learning data excluding the specific learning data included in the specific subset from the Q learning data. The P-dimensional vector may represent the selection quantities of learning data belonging to each of the P individual types. And, the learning data selection device (100) can repeat the process of selecting at least one other specific subset by integer programming for the remaining learning data so that the number of selected learning data is n or more.

[0118] At this point, integer programming can be expressed as follows.

[0119] Objective: min 1 T x

[0120] Condition: Ax ≥ 1,

[0121] x[i] ∈ {0,1} for all i

[0122] In the above, A is a PxQ binary matrix, and A[j,i] can indicate whether training data i matches individual type j, and

[0123] x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of training data i, and

[0124] Ax is the matrix product of A and x, and Ax[j] can represent the selection quantity of training data belonging to individual type j.

[0125] Additionally, the learning data selection device (100) may select a specific subset by a dual linear programming method that applies at least one of merging, separating, and changing the sign of the constraints in the integer programming method.

[0126] According to the first embodiment of the present invention, it is possible to select training data for each type so that it is as even as possible for all individual types. In addition, if the entire training data stored in the data pool is not labeled with ground truth, the ground truth only needs to be labeled for the training data selected to be as even as possible for each type, thereby reducing the cost of generating training data sets.

[0127]

[0128] [2nd Example]

[0129] FIG. 7 schematically illustrates a method for selecting training data for training a deep learning model according to a second embodiment of the present invention. The second embodiment of the present invention is described below with reference to FIG. 7. For reference, detailed descriptions of parts that can be easily understood from the description of the first embodiment are omitted in the following description.

[0130] First, the learning data selection device (100) can acquire at least one individual type corresponding to each of the multiple learning images stored in the data pool (S210).

[0131] At this time, the individual types may include 1_1 individual types to 1_x individual types corresponding to the 1st type of each of the multiple training images, and 2_1 individual types to 2_y individual types corresponding to the 2nd type of each of the multiple training images. Each of x and y may be an integer greater than or equal to 1. However, the present invention is not limited thereto and may have three or more types depending on the characteristics of the training images.

[0132] For example, it can be composed of various individual types corresponding to various types, such as type types related to time periods corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as clear, cloudy, rain, and fog; and type types related to places corresponding to individual types such as city, countryside, and highway.

[0133] In addition, for each of the multiple training images, at least one individual type (attribute) corresponding to each of the multiple training images can be automatically extracted by a labeler labeling each of the multiple training images with respect to predefined types, or by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0134] For example, a learning data selection device (100) can transmit a plurality of learning images to a labeler terminal to enable a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the plurality of learning images.

[0135] As another example, the learning data selection device (100) can automatically generate various individual types corresponding to various types of the entire image that each of the multiple learning images has.

[0136] That is, referring again to FIG. 3, the learning data selection device (100) may perform a process of generating first scene vectors corresponding to each of the learning images by performing a first embedding operation (10_1) on each of the learning images and generating first scene clusters by clustering (20) the first scene vectors, or a process of generating k-th scene vectors corresponding to each of the learning images by performing a k-th embedding operation (10_k) on each of the learning images and generating k-th scene clusters by clustering (20) the k-th scene vectors. The k may be an integer greater than or equal to 1. And, the learning data selection device (100) may generate individual types corresponding to the learning images by referencing the first scene clusters to the k-th scene clusters.

[0137] At this time, the training data selection device (100) may use visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to k-th embedding operations, and each of the first embedding model for the first embedding operation to the k-th embedding operation may be a different embedding model that generates scene vectors corresponding to each of the first to k-th type types, which are different types. Also, the training data selection device (100) may use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to k-th scene vectors.

[0138] In addition, although the above method involves performing embedding operations and clustering on each of the training images to generate at least one individual type for each of the training images, alternatively, individual types may be generated using metadata included in each of the training images. For example, when k is 1, that is, when only individual types for one type corresponding to each of the training images are generated through a single embedding operation, individual types for a type different from the one type resulting from the embedding operation may be additionally generated using metadata of each of the training images.

[0139] In other words, by checking the individual metadata included in each of the training images and further referring to the individual shooting times included in each of the metadata, individual types corresponding to each of the training images can be generated.

[0140] For example, in order to generate individual types according to time zone types such as day, night, evening, dawn, etc., the learning data selection device (100) groups the learning images according to preset time zones by referring to the shooting times of each of the learning images, and can set individual types related to the time zone for each group of the grouped learning images.

[0141] In addition, when the learning data selection device (100) generates individual types corresponding to each of the learning images by further referring to the shooting times of each of the learning images, it may generate individual types corresponding to one of the types using the specific embedding model and individual types corresponding to another type using meta-information by using only a specific embedding model which is any one of the first embedding model to the k-th embedding model.

[0142] That is, the learning data selection device (100) can generate specific scene vectors corresponding to each of the learning images by performing a specific embedding operation on each of the learning images, and generate specific scene clusters by clustering the specific scene vectors. In addition, the learning data selection device (100) can check the shooting times of each of the learning images by referring to the respective metadata included in each of the learning images. Subsequently, the learning data selection device (100) can generate individual types corresponding to each of the learning images by referring to the specific scene clusters and the shooting times.

[0143] Next, referring again to FIG. 7, the training data selection device (100) can generate a bipartite graph (S220) that matches each of the multiple training images with individual types.

[0144] Next, the training data selection device (100) can perform a cycle (S230) of selecting some training images that match all individual types from among a number of training images by referring to a bipartite graph.

[0145] That is, the learning data selection device (100) can check the number of corresponding individual types that match each of the multiple learning images among the individual types by referring to the bipartite graph, and select a specific learning image that has the largest number of corresponding individual types.

[0146] And, the learning data selection device (100) can perform the process of checking the number of remaining individual types that match each of the multiple learning images among the remaining individual types excluding the corresponding individual types, and selecting another specific learning image that has the largest number of remaining individual types.

[0147] In this way, by repeating the process of selecting the training image with the largest number of remaining individual types among the individual types remaining after excluding the selected individual types until all individual types are selected, the training data selection device (100) can perform a cycle of selecting some training images that match all individual types.

[0148] At this time, the learning data selection device (100) can select one of the multiple learning images as a first criterion when there are multiple learning images with the largest number of corresponding individual types, and select another specific learning image when there are multiple remaining learning images with the largest number of remaining corresponding individual types, by selecting one of the multiple remaining learning images as a second criterion.

[0149] Afterwards, the learning data selection device (100) can repeat the process of performing the cycle on the remaining learning images, excluding some selected learning images that match all individual types from the multiple learning images, until n learning images are selected (S240).

[0150] At this time, when individual types include 1_1 individual types to 1_x individual types corresponding to a 1st type type that each of the multiple training images has, and 2_1 individual types to 2_y individual types corresponding to a 2nd type type that each of the multiple training images has, the training data selection device (100) can ensure that the number of 1_1 individual types to 1_x individual types corresponding to the 1st type type and the number of 2_1 individual types to 2_y individual types corresponding to the 2nd type type matched to n training images are within a threshold deviation. In addition, the number of 1_1 individual types to 1_x individual types is within a 1st threshold deviation, and the number of 2_1 individual types to 2_y individual types is within a 2nd threshold deviation. That is, the training data selection device (100) can select training images such that the number of training images for each individual type is as even as possible for all individual types for each type. For example, when selecting some training images from the day and night time zone types and the car and person object types, some training images may be selected that are matched equally to the individual types of day and night, matched equally to the individual types of cars and people in the individual types of day, and matched equally to both the individual types of cars and people in the individual types of night.

[0151]

[0152] [3rd Example]

[0153] FIG. 8 schematically illustrates a method for selecting training data for training a deep learning model according to a third embodiment of the present invention. Referring to FIG. 8, the third embodiment of the present invention is described as follows. For reference, in the following description, detailed descriptions of parts that can be easily understood from the description of the first embodiment are omitted.

[0154] First, the learning data selection device (100) can acquire at least one individual type corresponding to each of the multiple learning images stored in the data pool (S310).

[0155] At this time, the individual types may include 1_1 individual types to 1_x individual types corresponding to the 1st type of each of the multiple training images, and 2_1 individual types to 2_y individual types corresponding to the 2nd type of each of the multiple training images. Each of x and y may be an integer greater than or equal to 1. However, the present invention is not limited thereto and may have three or more types depending on the characteristics of the training images.

[0156] For example, it can be composed of various individual types corresponding to various types, such as type types related to time periods corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as clear, cloudy, rain, and fog; and type types related to places corresponding to individual types such as city, countryside, and highway.

[0157] In addition, for each of the multiple training images, at least one individual type (attribute) corresponding to each of the multiple training images can be automatically extracted by a labeler labeling each of the multiple training images with respect to predefined types, or by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0158] For example, a learning data selection device (100) can transmit a plurality of learning images to a labeler terminal to enable a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the plurality of learning images.

[0159] As another example, the learning data selection device (100) can automatically generate various individual types corresponding to various types of the entire image that each of the multiple learning images has.

[0160] That is, referring again to FIG. 3, the learning data selection device (100) may perform a process of generating first scene vectors corresponding to each of the learning images by performing a first embedding operation (10_1) on each of the learning images and generating first scene clusters by clustering (20) the first scene vectors, or a process of generating k-th scene vectors corresponding to each of the learning images by performing a k-th embedding operation (10_k) on each of the learning images and generating k-th scene clusters by clustering (20) the k-th scene vectors. The k may be an integer greater than or equal to 1. And, the learning data selection device (100) may generate individual types corresponding to the learning images by referencing the first scene clusters to the k-th scene clusters.

[0161] At this time, the training data selection device (100) may use visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to k-th embedding operations, and each of the first embedding model for the first embedding operation to the k-th embedding operation may be a different embedding model that generates scene vectors corresponding to each of the first to k-th type types, which are different types. Also, the training data selection device (100) may use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to k-th scene vectors.

[0162] In addition, although the above method involves performing embedding operations and clustering on each of the training images to generate at least one individual type for each of the training images, alternatively, individual types may be generated using metadata included in each of the training images. For example, when k is 1, that is, when only individual types for one type corresponding to each of the training images are generated through a single embedding operation, individual types for a type different from the one type resulting from the embedding operation may be additionally generated using metadata of each of the training images.

[0163] In other words, by checking the individual metadata included in each of the training images and further referring to the individual shooting times included in each of the metadata, individual types corresponding to each of the training images can be generated.

[0164] For example, in order to generate individual types according to time zone types such as day, night, evening, dawn, etc., the learning data selection device (100) groups the learning images according to preset time zones by referring to the shooting times of each of the learning images, and can set individual types related to the time zone for each group of the grouped learning images.

[0165] In addition, when the learning data selection device (100) generates individual types corresponding to each of the learning images by further referring to the shooting times of each of the learning images, it may generate individual types corresponding to one of the types using the specific embedding model and individual types corresponding to another type using meta-information by using only a specific embedding model which is any one of the first embedding model to the k-th embedding model.

[0166] That is, the learning data selection device (100) can generate specific scene vectors corresponding to each of the learning images by performing a specific embedding operation on each of the learning images, and generate specific scene clusters by clustering the specific scene vectors. In addition, the learning data selection device (100) can check the shooting times of each of the learning images by referring to the respective metadata included in each of the learning images. Subsequently, the learning data selection device (100) can generate individual types corresponding to each of the learning images by referring to the specific scene clusters and the shooting times.

[0167] Next, referring again to FIG. 8, the training data selection device (100) can generate a bipartite graph (S320) that matches each of the multiple training images with individual types.

[0168] Next, the training data selection device (100) can select a specific subset with the smallest number of training images among a predetermined number of training images that include all individual types through an optimization algorithm by referring to a bipartite graph, and can produce the remaining training images excluding a specific number of training images included in the specific subset (S330).

[0169] For example, a learning data selection device (100) can select a specific subset containing specific learning images corresponding to specific selection fitness variables having a minimum sum of selection fitness variables among the selection fitness variables of the Q-dimensional vector, which are generated by matrix multiplying a PxQ binary matrix corresponding to P individual types and Q learning images in a bipartite graph and a Q-dimensional vector representing selection fitness variables for each of the Q learning images in each of the P individual types, such that the total sums of fitness in the P-dimensional vector are 1 or greater and the selection fitness variables in the Q-dimensional vector satisfy 0 or greater and 1 or less, and the remaining learning images excluding the specific learning images included in the specific subset from the Q learning images. The P-dimensional vector may represent the total sums of fitness of Q learning images belonging to each of the P individual types.

[0170] At this time, linear programming can be expressed as follows.

[0171] Objective: min 1 T x

[0172] Condition: Ax ≥ 1,

[0173] 0 ≤ x ≤ 1

[0174] In the above, A is a PxQ binary matrix, and A[j,i] can indicate whether a training image i is matched to an individual type j, and

[0175] x is a Q-dimensional vector, and x[i] can represent the selection fitness variable for training image i, and

[0176] Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness of training images belonging to individual type j.

[0177] Additionally, the learning data selection device (100) may select a specific subset by a dual linear programming method that applies at least one of merging, separating, and changing the sign of the constraints in the linear programming method.

[0178] The dual linear programming method modified from the above linear programming method can be represented as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the above linear programming method.

[0179] Objective: min 1 T x

[0180] Condition: Bx ≥ b,

[0181] x ≥ 0

[0182] At this time, B=[A;-I], (P+Q)xQ binary matrix,

[0183] B[j,i]=A[j,i],

[0184] B[P+i,i] = -1 for all i ∈ [1,2,캜,Q],

[0185] B[P+i,j] = 0 for all j≠i

[0186] b=[1 P ;-1 P ], (P+Q)-dimensional vector,

[0187] b[j] = 1 for all j ∈ [1,2,j,P],

[0188] b[P+i] = -1 for all i ∈ [1,2,x,Q]

[0189] It can be represented as follows.

[0190] As another example, a learning data selection device (100) can select a specific subset containing specific learning images corresponding to specific selection variables having a minimum sum of selection variables among the selection variables of a Q-dimensional vector in which the selection quantities in the P-dimensional vector are 1 or greater and the selection variables in the Q-dimensional vector satisfy 0 or 1, by matrix multiplying a PxQ binary matrix corresponding to P individual types and Q learning images in a bipartite graph and a Q-dimensional vector representing selection variables of each of the Q learning images in each of the P individual types through integer programming as an optimization algorithm, and can produce the remaining learning images excluding the specific learning images included in the specific subset from the Q learning images. The P-dimensional vector may represent the selection quantities of learning images belonging to each of the P individual types.

[0191] At this point, integer programming can be expressed as follows.

[0192] Objective: min 1 T x

[0193] Condition: Ax ≥ 1,

[0194] x[i] ∈ {0,1} for all i

[0195] In the above, A is a PxQ binary matrix, and A[j,i] can indicate whether a training image i is matched to an individual type j, and

[0196] x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of training image i, and

[0197] Ax is the matrix product of A and x, and Ax[j] can represent the selection quantity of training images belonging to individual type j.

[0198] Additionally, the learning data selection device (100) may select a specific subset by a dual linear programming method that applies at least one of merging, separating, and changing the sign of the constraints in the integer programming method.

[0199] Afterwards, the training data selection device (100) can repeat (S340) the process of selecting at least one other specific subset consisting of a predetermined number of training images including all individual types through an optimization algorithm for the remaining training images until n training images for training a deep learning model are selected.

[0200] At this time, when individual types include 1_1 individual types to 1_x individual types corresponding to a 1st type type that each of the multiple training images has, and 2_1 individual types to 2_y individual types corresponding to a 2nd type type that each of the multiple training images has, the training data selection device (100) can ensure that the number of 1_1 individual types to 1_x individual types corresponding to the 1st type type and the number of 2_1 individual types to 2_y individual types corresponding to the 2nd type type matched to n training images are within a threshold deviation. In addition, the number of 1_1 individual types to 1_x individual types is within a 1st threshold deviation, and the number of 2_1 individual types to 2_y individual types is within a 2nd threshold deviation. That is, the training data selection device (100) can select training images such that the number of training images for each individual type is as even as possible for all individual types for each type. For example, when selecting some training images from the day and night time zone types and the car and person object types, some training images may be selected that are matched equally to the individual types of day and night, matched equally to the individual types of cars and people in the individual types of day, and matched equally to both the individual types of cars and people in the individual types of night.

[0201]

[0202] [Fourth Example]

[0203] FIG. 9 schematically illustrates a method for selecting training data for training a deep learning model according to a fourth embodiment of the present invention. Referring to FIG. 9, the fourth embodiment of the present invention is described as follows. For reference, in the following description, detailed descriptions of parts that can be easily understood from the description of the first embodiment are omitted.

[0204] First, the training data selection device (100) can acquire (S410) each at least one individual type (attribute) corresponding to each at least one object included in each of the multiple training images stored in the data pool.

[0205] At this time, for each of the at least one individual type (attribute) corresponding to the objects, a labeler can label each of the multiple training images for predefined types, or the individual types present in each of the multiple training images can be automatically extracted by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0206] For example, a learning data selection device (100) can transmit a plurality of learning images to a labeler terminal to enable a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each object included in each of the plurality of learning images.

[0207] As another example, the learning data selection device (100) can automatically generate at least one individual type corresponding to each object included in each of the multiple learning images.

[0208] That is, referring again to FIG. 4, the training data selection device (100) performs object detection (1) for each of the training images to detect at least one object in each of the training images, generates cropped images by cropping (2) the regions corresponding to the bounding boxes of each of the detected objects in each of the training images, performs an embedding operation (10) on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters (20) the object vectors to generate object clusters, and generates individual types corresponding to the training images by referencing the object clusters. At this time, to perform object detection for each of the training images, the training data selection device (100) may use an Open-World object detection model such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), or YOLO (You Only Look Once)-World.

[0209] At this time, the learning data selection device (100) may, in addition to the object clusters, check the respective metadata included in each of the learning images and further refer to the respective shooting times included in each metadata to generate individual types corresponding to each of the learning images.

[0210] Meanwhile, although the above method generated cropped images after performing object detection on each of the training images, in a different case, if each of the training images has ground truths labeled on it, cropped images can be generated by referencing the ground truths of each of the training images.

[0211] That is, the learning data selection device (100) can generate cropped images by cropping regions corresponding to bounding boxes of objects in each of the learning images by referring to ground truth information included in each of the learning images, generate object vectors corresponding to each of the cropped images by performing an embedding operation on each of the cropped images, generate object clusters by clustering the object vectors, and generate individual types corresponding to the learning images by referring to the object clusters.

[0212] Next, referring again to FIG. 9, the training data selection device (100) can generate a bipartite graph (S420) that matches each of the multiple training images with individual types.

[0213] Next, the training data selection device (100) can select a specific subset with the smallest number of training images among a predetermined number of training images that include all individual types through an optimization algorithm by referring to a bipartite graph, and can produce the remaining training images excluding a specific number of training images included in the specific subset (S430).

[0214] For example, a learning data selection device (100) can select a specific subset containing specific learning images corresponding to specific selection fitness variables having a minimum sum of selection fitness variables among the selection fitness variables of the Q-dimensional vector, which are generated by matrix multiplying a PxQ binary matrix corresponding to P individual types and Q learning images in a bipartite graph and a Q-dimensional vector representing selection fitness variables for each of the Q learning images in each of the P individual types, such that the total sums of fitness in the P-dimensional vector are 1 or greater and the selection fitness variables in the Q-dimensional vector satisfy 0 or greater and 1 or less, and the remaining learning images excluding the specific learning images included in the specific subset from the Q learning images. The P-dimensional vector may represent the total sums of fitness of Q learning images belonging to each of the P individual types.

[0215] At this time, linear programming can be expressed as follows.

[0216] Objective: min 1 T x

[0217] Condition: Ax ≥ 1,

[0218] 0 ≤ x ≤ 1

[0219] In the above, A is a PxQ binary matrix, and A[j,i] can indicate whether a training image i is matched to an individual type j, and

[0220] x is a Q-dimensional vector, and x[i] can represent the selection fitness variable for training image i, and

[0221] Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness of training images belonging to individual type j.

[0222] Additionally, the learning data selection device (100) may select a specific subset by a dual linear programming method that applies at least one of merging, separating, and changing the sign of the constraints in the linear programming method.

[0223] The dual linear programming method modified from the above linear programming method can be represented as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the above linear programming method.

[0224] Objective: min 1 T x

[0225] Condition: Bx ≥ b,

[0226] x ≥ 0

[0227] At this time, B=[A;-I], (P+Q)xQ binary matrix,

[0228] B[j,i]=A[j,i],

[0229] B[P+i,i] = -1 for all i ∈ [1,2,캜,Q],

[0230] B[P+i,j] = 0 for all j≠i

[0231] b=[1 P ;-1 P ], (P+Q)-dimensional vector,

[0232] b[j] = 1 for all j ∈ [1,2,j,P],

[0233] b[P+i] = -1 for all i ∈ [1,2,x,Q]

[0234] It can be represented as follows.

[0235] As another example, a learning data selection device (100) can select a specific subset containing specific learning images corresponding to specific selection variables having a minimum sum of selection variables among the selection variables of a Q-dimensional vector in which the selection quantities in the P-dimensional vector are 1 or greater and the selection variables in the Q-dimensional vector satisfy 0 or 1, by matrix multiplying a PxQ binary matrix corresponding to P individual types and Q learning images in a bipartite graph and a Q-dimensional vector representing selection variables of each of the Q learning images in each of the P individual types through integer programming as an optimization algorithm, and can produce the remaining learning images excluding the specific learning images included in the specific subset from the Q learning images. The P-dimensional vector may represent the selection quantities of learning images belonging to each of the P individual types.

[0236] At this point, integer programming can be expressed as follows.

[0237] Objective: min 1 T x

[0238] Condition: Ax ≥ 1,

[0239] x[i] ∈ {0,1} for all i

[0240] In the above, A is a PxQ binary matrix, and A[j,i] can indicate whether a training image i is matched to an individual type j, and

[0241] x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of training image i, and

[0242] Ax is the matrix product of A and x, and Ax[j] can represent the selection quantity of training images belonging to individual type j.

[0243] Additionally, the learning data selection device (100) may select a specific subset by a dual linear programming method that applies at least one of merging, separating, and changing the sign of the constraints in the integer programming method.

[0244] Afterwards, the training data selection device (100) can repeat (S440) the process of selecting at least one other specific subset consisting of a predetermined number of training images including all individual types through an optimization algorithm for the remaining training images until n training images for training a deep learning model are selected.

[0245] The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0246] Although the present invention has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and variations from this description.

[0247] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims set forth below, as well as the claims described below, shall be considered to fall within the scope of the concept of the present invention.

Claims

1. Regarding the method of selecting training data for training a deep learning model, (a) A training data selection device generates at least one individual attribute corresponding to each of a plurality of training images stored in a data pool, and generates a bipartite graph that matches each of the plurality of training images with the individual attributes; and (b) The training data selection device performs a cycle of selecting some training images that match all of the individual types by repeating the process of (i) checking the number of corresponding individual types that match each of the training images among the individual types by referring to the bipartite graph, and selecting a specific training image that has the largest number of corresponding individual types, and (ii) checking the number of remaining corresponding individual types that match each of the training images among the remaining individual types excluding the corresponding individual types, and selecting another specific training image that has the largest number of remaining corresponding individual types, and performing the process of performing the cycle on the remaining training images excluding the selected some training images from the plurality of training images until n training images for training the deep learning model are selected - where n is the target number of training images for training the deep learning model and is an integer representing a plurality of numbers; A method including 2. In Paragraph 1, In step (b) above, The above individual types include 1_1 individual types to 1_x individual types corresponding to a 1 type having each of the above training images, wherein x is an integer greater than or equal to 1, and 2_1 individual types to 2_y individual types corresponding to a 2 type having each of the above training images, wherein y is an integer greater than or equal to 1, and The above-described training data selection device is a method for selecting the n training images such that the number of the first_1 individual types to the first_x individual types corresponding to the first type and the number of the second_1 individual types to the second_y individual types corresponding to the second type matched to the n training images are within the threshold deviation, the number of the first_1 individual types to the first_x individual types are within the first threshold deviation, and the number of the second_1 individual types to the second_y individual types are within the second threshold deviation.

3. In Paragraph 1, In step (b) above, The above-described learning data selection device selects one of the multiple learning images by a first criterion when there are multiple learning images with the largest number of corresponding individual types, thereby selecting the specific learning image, and selects one of the multiple remaining learning images by a second criterion when there are multiple learning images with the largest number of the remaining individual types, thereby selecting the other specific learning image.

4. In Paragraph 1, In step (a) above, The above-described training data selection device transmits the training images to a labeler terminal, thereby causing a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the training images.

5. In Paragraph 1, In step (a) above, The above-described training data selection device performs a process of generating first scene vectors corresponding to each of the training images by performing a first embedding operation on each of the training images and generating first scene clusters by clustering the first scene vectors, or a process of generating k-th scene vectors corresponding to each of the training images by performing a k-th embedding operation on each of the training images—where k is an integer greater than or equal to 1—and generating k-th scene clusters by clustering the k-th scene vectors, and a method of generating individual types corresponding to the training images by referencing the first scene clusters to the k-th scene clusters.

6. In Paragraph 1, In step (a) above, The above-described learning data selection device is a method for generating the individual types corresponding to each of the learning images by checking each of the metadata included in each of the learning images and further referring to each of the shooting times included in each of the metadata.

7. In Paragraph 1, In step (a) above, The above-described training data selection device comprises: (i) performing a specific embedding operation on each of the training images to generate specific scene vectors corresponding to each of the training images, and clustering the specific scene vectors to generate specific scene clusters; (ii) checking the shooting times of each of the training images by referring to each of the metadata included in each of the training images; and (iii) generating the individual types corresponding to each of the training images by referring to the specific scene clusters and the shooting times.

8. A training data selection device for selecting training data for training a deep learning model, A memory storing instructions for selecting training data for training a deep learning model; and A processor that performs an operation to select training data for training the deep learning model according to the instructions stored in the memory; Includes, The processor comprises: (I) a process of generating at least one individual attribute corresponding to each of a plurality of training images stored in a data pool, and generating a bipartite graph that matches each of the plurality of training images with the individual attributes; and (II) a process of (i) checking the number of corresponding individual attributes that match each of the individual attributes among the individual attributes and selecting a specific training image with the largest number of corresponding individual attributes, by referring to the bipartite graph, and (ii) repeating the process of checking the number of remaining corresponding individual attributes that match each of the training images among the remaining individual attributes excluding the corresponding individual attributes and selecting another specific training image with the largest number of remaining corresponding individual attributes, thereby performing a cycle to select some training images that match all of the individual attributes, and performing the cycle on the remaining training images excluding the selected some training images from the plurality of training images, wherein n training images for training the deep learning model—where n is the target number of training images for training the deep learning model and is an integer representing a plurality of numbers— A training data selection device that performs a process that repeats until selected.

9. In Paragraph 8, The above processor is, In the above process (II), the individual types include 1_1 individual types to 1_x individual types corresponding to a 1 type type having each of the training images, wherein x is an integer greater than or equal to 1, and 2_1 individual types to 2_y individual types corresponding to a 2 type type having each of the training images, wherein y is an integer greater than or equal to 1, and A learning data selection device for selecting n learning images such that the number of individual types from 1_1 to 1_x corresponding to the first type and the number of individual types from 2_1 to 2_y corresponding to the second type matched to the n learning images are within the threshold deviation, the number of individual types from 1_1 to 1_x is within the first threshold deviation, and the number of individual types from 2_1 to 2_y is within the second threshold deviation.

10. In Paragraph 8, The above processor is, A learning data selection device that, in the above process (II), if there are multiple learning images with the largest number of corresponding individual types, selects one of the multiple learning images as a first criterion to select the specific learning image, and if there are multiple remaining learning images with the largest number of remaining individual types, selects one of the multiple remaining learning images as a second criterion to select the other specific learning image.

11. In Paragraph 8, The above processor is, A training data selection device that, in the above process (I), transmits the training images to a labeler terminal so that a labeler corresponding to the labeler terminal generates at least one individual type corresponding to each of the training images.

12. In Paragraph 8, The above processor is, A learning data selection device that, in the above process (I), performs a process of generating first scene vectors corresponding to each of the learning images by performing a first embedding operation on each of the learning images and generating first scene clusters by clustering the first scene vectors, or performs a process of generating k-th scene vectors corresponding to each of the learning images by performing a k-th embedding operation - where k is an integer greater than or equal to 1 - on each of the learning images and generating k-th scene clusters by clustering the k-th scene vectors, and generates individual types corresponding to the learning images by referencing the first scene clusters to the k-th scene clusters.

13. In Paragraph 6, The above processor is, A learning data selection device that, in the above process (I), checks each meta-information included in each of the learning images and further refers to each shooting time included in each of the meta-information to generate the individual types corresponding to each of the learning images.

14. In Paragraph 8, The above processor is, A learning data selection device that, in the above process (I), (i) performs a specific embedding operation on each of the learning images to generate specific scene vectors corresponding to each of the learning images, and clusters the specific scene vectors to generate specific scene clusters, (ii) checks the shooting times of each of the learning images by referring to each of the metadata included in each of the learning images, and (iii) generates the individual types corresponding to each of the learning images by referring to the specific scene clusters and the shooting times.

Images