Data generation device and data generation method

Abstract

According to the present invention, a data generation device that generates training data for an image recognition model that outputs recognition results for inputted images is a computer that has a computation device that executes a program and a storage device that can be accessed by the computation device. The computation device acquires scene designation information that designates environmental elements that constitute a scene and objects that are included in the scene, calculates reality constraints that indicate image conditions to which the training data is to conform on the basis of a traffic situation description that describes a traffic situation in a prescribed format and camera settings that are setting information for a camera, inputs the scene designation information into an image generation model to cause the image generation model to generate a training data candidate image for which the camera is the viewpoint, determines a reality agreement score that indicates the agreement between the generated training data candidate image and the reality constraints, and determines whether to update training data designation information that designates the training data in accordance with the reality agreement score.

Description

Data generation device and data generation method Incorporation by reference

[0001] This application claims the priority of Japanese Patent Application No. 2024-193769, filed on November 5, 2024, and incorporates its content by reference.

[0002] The present invention relates to a data generation device and a data generation method, and particularly to a technology for generating machine learning data for an AI model for autonomous driving.

[0003] With the development of AI (Artificial Intelligence) technology, the realization of autonomous driving of automobiles is approaching. Currently, the autonomous driving technology in operation is autonomous driving with a person on board, and in places where the system has difficulty coping, it is switched to manual operation for safety. Autonomous driving without a person on board is limited operation in depopulated areas or events, and an expansion of autonomous driving without a person on board is desired. In expanding autonomous driving without a person on board, it is necessary to improve the AI model so that it can handle situations (corner cases) that occur only rarely.

[0004] The input to the AI model for autonomous driving accurately recognizes the distance and direction between the host vehicle and surrounding vehicles, people, structures, etc. from the data acquired by cameras and three-dimensional sensors attached to the vehicle.

[0005] The AI model learns using learning data in which examples and correct answers are paired. It is known that the accuracy of the AI model improves as the number of learning data increases, but it requires a large cost to collect data on corner cases in the actual environment. Current general in-vehicle AI models recognize vehicles, people, and bicycles, but do not recognize corner cases (for example, obstacles such as debris on the road), and preventing accidents by accurately recognizing such obstacles is an issue.

[0006] To address the aforementioned challenges, Patent Document 1 discloses a method for collecting corner case training data using three-dimensional computer simulation. Patent Document 1 describes a three-dimensional virtual space generation and rendering means for generating and rendering objects in a three-dimensional virtual space using three-dimensional computer graphics (3D CG), an AI recognition target and information generation and rendering means for generating and rendering information about recognition targets of an AI model in the three-dimensional virtual space using 3D CG, a three-dimensional virtual space scene generation and rendering means for generating and rendering various scenes by changing the settings of at least one of the three-dimensional virtual space and the recognition target, a CG image including the recognition target generated by the three-dimensional virtual space scene generation and rendering means, and the information about the recognition target generated by the AI ​​recognition target and information generation and rendering means for machine learning of the AI ​​model. A machine learning system is disclosed comprising: a machine learning data generation means for generating necessary training data; an AI model machine learning means for training the AI ​​model using the training data generated by the machine learning data generation means; and an AI machine learning-trained model execution means for executing processing by the AI ​​model using the trained model trained by the AI ​​model machine learning means. The AI ​​recognition target object / information generation and drawing means is characterized by having a 3D spatial information generation function for generating 3D spatial information of feature points of the recognition target object in the 3D virtual space; a 3D spatial information placement and drawing function for placing and drawing the 3D spatial information of the recognition target object at a specified position in the 3D virtual space by referring to the 3D spatial information; and a 2D image position information generation function for generating position information on a 2D image of the recognition target object drawn by the 3D spatial information placement and drawing function.

[0007] Furthermore, Non-Patent Document 1 discloses a method for generating an arbitrary image from a string of characters.

[0008] Patent Document 1: Japanese Unexamined Patent Publication No. 2023-146041 Non-Patent Document 1: Tim Brooks, et al., "Video generation models as world simulators", [online], February 15, 2024, [February 29, 2024], Internet<URL: https: / / openAI.com / research / video-generation-models-as-world-simulators>

[0009] The invention described in Patent Document 1 allows for the reproduction of corner cases in a computer-generated three-dimensional space, and enables the theoretical calculation of "two-dimensional images" and "distances to various objects" from any position in the three-dimensional space, thereby enabling the collection of low-cost training data. Furthermore, since roads and the surrounding environment can be easily modified in computer simulations, a wide variety of training data can be collected. While the invention described in Patent Document 1 enables the collection of a wide variety of training data at low cost, the resolution of the training data remains a challenge. To reproduce a real environment with high resolution in three-dimensional space, complex three-dimensional models of vehicles, people, and the surrounding environment are necessary. On the other hand, the cost of creating complex three-dimensional models is high, so it is necessary to simplify the three-dimensional models. The technology described in Non-Patent Document 1 enables the low-cost creation of training data images that closely resemble the real environment of a corner case from string-based instructions (prompts). While the technology described in Non-Patent Document 1 can create training data with lower cost and higher resolution than computer simulations, its quality remains a challenge. Image generation using prompts makes it difficult to give accurate instructions to the AI ​​model, and inconsistencies, such as distorted roads, are likely to occur in the generated images.

[0010] The present invention aims to provide a data generation device and a data generation method for collecting high-quality training data at low cost.

[0011] A typical example of the invention disclosed in this application is as follows: A data generation device for generating training data for an image recognition model that outputs recognition results of an input image, comprising a computer having a computing unit for executing a program and a storage device accessible by the computing unit, wherein the computing unit acquires scene specification information that specifies environmental elements constituting a scene and objects included in the scene, calculates reality constraints that indicate the conditions on the image that the training data must follow based on a traffic situation description that describes the traffic situation in a predetermined format and camera settings which are camera setting information, inputs the scene specification information to the image generation model, generates candidate training data images with the camera as the viewpoint, determines a reality consistency score that indicates the consistency between the generated candidate training data images and the reality constraints, and determines whether or not to update the training data specification information that specifies the training data according to the reality consistency score.

[0012] According to one aspect of the present invention, the quality of training data images can be improved. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

[0013] This is a diagram showing the configuration of a data generation device according to the first embodiment of the present invention. This is a flowchart of the processing performed by the data generation device according to the first embodiment of the present invention. This is a diagram showing the processing performed by the reality constraint calculation unit according to the first embodiment of the present invention. This is a diagram showing the processing performed by the reality consistency score determination unit according to the first embodiment of the present invention. This is a diagram showing the prompt text update unit according to the first embodiment of the present invention. This is a flowchart of the processing performed by the prompt text update unit according to the first embodiment of the present invention. This is a diagram showing an example of a user interface screen output by the data generation device according to the first embodiment of the present invention. This is a block diagram showing the physical configuration of the computer constituting the data generation device according to the first embodiment of the present invention. This is a diagram showing an example of a prompt generated by the data generation device according to the first embodiment of the present invention. This is a diagram showing the configuration of a data generation device according to the second embodiment of the present invention. This is a flowchart of the processing performed by the data generation device according to the second embodiment of the present invention.

[0014] The following embodiments are merely examples for illustrating the present invention, and the present invention is not limited to these embodiments. It is clear that various modifications and alterations are possible within the scope of the technical idea of ​​the present invention, in addition to the components, means, or combinations thereof described in each of the above embodiments.

[0015] <First Embodiment> Figure 1 is a configuration diagram of the data generation device 100 according to the first embodiment.

[0016] The data generation device 100 includes a scene specification unit 101, a training data candidate generation unit 103, a domain-specific knowledge specification unit 108, a reality constraint calculation unit 111, a reality consistency score determination unit 107, a data selection unit 106, and a prompt text update unit 102.

[0017] The scene specification unit 101 sets the scene specification information for the generated image. The scene specification information represents the scene to be generated and the location, type, quantity, behavior, etc. of objects in natural language or a combination of natural language and images. Scene specification information in natural language is also called a prompt. The scene specification information includes, for example, information (first specification information) that constitutes the scene, such as road structure, time of day, and streetscape. Furthermore, the scene specification information may also include object information (second specification information) that specifies the number, type, and behavior of objects. Specifically, the scene specification information may include, for example, scenes such as "urban environment," "highway," and "general road," and object types such as "vehicle" and "pedestrian." Furthermore, the scene specification information may include, for example, the location of objects such as "preceding vehicle" and "pedestrian on crosswalk." Furthermore, the scene specification information may include, for example, the quantity of objects such as "two preceding vehicles" and "five pedestrians," and the behavior of objects such as "pedestrian crossing crosswalk." The above scene specification information is illustrative and not limited to these examples.

[0018] The training data candidate generation unit 103 receives scene specification information set by the scene specification unit 101, inputs the received scene specification information into a large-scale AI model (hereinafter also referred to as an image generation model), obtains candidate images for training data (hereinafter also referred to as training data candidate images) and correct values, and outputs them to the data selection unit 106 or the reality consistency score determination unit 107. The training data candidate generation unit 103 may have a large-scale AI model internally, or it may utilize a large-scale AI model provided outside the data generation device 100.

[0019] The image generation unit 104 generates candidate images for training data based on the scene specification information set by the scene specification unit 101. The image generation unit 104 can generate images of the driving environment from the perspective of a camera installed on the vehicle. The candidate images for training data generated by the image generation unit 104 may be a single image, images from multiple perspectives assuming multiple cameras mounted on the vehicle, or multiple images that change over time. Furthermore, the candidate images for training data generated by the image generation unit 104 may be images from various cameras according to the scene specification information, such as panoramic images or fisheye images. The image generation unit 104 can use an image generation model (e.g., stable diffusion) that generates images having the features described in the input text.

[0020] The ground truth value generation unit 105 takes the candidate training data images generated by the image generation unit 104 as input and generates ground truth values ​​for the candidate training data images. For example, the ground truth values ​​may be obtained by recognizing the environment of the generated image using an environment recognition AI model that can derive ground truth values ​​with high accuracy, such as a machine learning model that is more multilayered, has more dimensions and a larger number of parameters than the target in-vehicle environment recognition AI model, or a machine learning model that has been trained on more data than the target in-vehicle environment recognition AI. Alternatively, the image generation and the predicted ground truth values ​​may be output simultaneously using a diffusion model or the like. By using an environment recognition AI model that can derive ground truth values ​​with high accuracy, the position of the object on the image (where the IOU, which is the overlap rate between the ground truth value and the bounding box, is high), the distance to the object (the difference between the distance from the camera associated with the pixel and the ground truth value), and the type of object can be accurately derived.

[0021] The ground truth generation unit 105 outputs an object class representing the type of object in the image, such as a roadside, white lines, vehicles, or animals, and position information indicating the position of the object in the image. The ground truth generation unit 105 may also calculate other ground truth values ​​in the environmental recognition AI model, such as parallax / depth information and drivable area information.

[0022] The domain-specific knowledge specification unit 108 acquires the traffic situation description 109 and the camera settings 110. Details of the traffic situation description 109 and the camera settings 110 will be described later.

[0023] The reality constraint calculation unit 111 obtains the traffic situation description 109 and camera settings 110 from the domain-specific knowledge specification unit 108 and calculates the reality constraints. The reality constraints refer to the conditions on the image that the training data 112 must follow. The processes executed by the reality constraint calculation unit 111 will be described later with reference to Figure 3.

[0024] The reality consistency score determination unit 107 receives the candidate learning data images and correct values ​​output from the candidate learning data generation unit 103, as well as the reality constraints calculated by the reality constraint calculation unit 111, and calculates the difference between the generated candidate learning data images and the reality constraints as the reality consistency score. The processing performed by the reality consistency score determination unit 107 will be described later with reference to Figure 4.

[0025] The data selection unit 106 takes the reality consistency score as input and decides whether to save the candidate images as training data 112 in memory or discard them as unsuitable data, and then saves or discards them according to the decision.

[0026] The prompt text update unit 102 updates the prompt text used for scene specification so that the reality consistency score of the training data candidate images is improved. The processes executed by the prompt text update unit 102 will be described later with reference to Figures 5 and 6.

[0027] Figure 8 is a block diagram showing the physical configuration of the computer constituting the data generation device 100 according to the first embodiment.

[0028] The data generation device 100 in this embodiment is composed of a computer having a processor (CPU) 1, memory 2, auxiliary storage device 3, and communication interface 4. The data generation device 100 may also have an input interface 5 and an output interface 6.

[0029] Processor 1 is an arithmetic unit that executes programs stored in memory 2. By executing various programs, Processor 1 realizes the various functional units of the data generation device 100 (for example, the scene specification unit 101, the prompt text update unit 102, the learning data candidate generation unit 103, the data selection unit 106, the reality consistency score determination unit 107, the domain-specific knowledge specification unit 108, the reality constraint calculation unit 111, etc.). Note that some of the processing performed by Processor 1 when executing programs may be executed by other arithmetic units (for example, hardware such as a GPU, ASIC, FPGA, etc.).

[0030] Memory 2 is a storage device that includes ROM, a non-volatile memory element, and RAM, a volatile memory element. ROM stores immutable programs (e.g., BIOS). RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by processor 1 and data used during program execution.

[0031] The auxiliary storage device 3 is, for example, a high-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device 3 also stores data used by the processor 1 when executing a program and the program that the processor 1 executes. In other words, the program is read from the auxiliary storage device 3, loaded into memory 2, and executed by the processor 1, thereby realizing each function of the data generation device 100.

[0032] Communication interface 4 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0033] Input interface 5 is an interface to which input devices such as a keyboard 7 and a mouse 8 are connected and to receive input from the operator. Output interface 6 is an interface to which output devices such as a display device 9 and a printer (not shown) are connected and to output the program execution results in a format that can be viewed by the operator.

[0034] The program executed by processor 1 is provided to data generation device 100 via removable media (CD-ROM, flash memory, etc.) or a network, and stored in non-volatile auxiliary storage device 3, which is a non-temporary storage medium. For this reason, it is preferable for data generation device 100 to have an interface for reading data from removable media.

[0035] The data generation device 100 is a computer system that operates on a single physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer built on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple units may be combined and operate on a single physical or logical computer.

[0036] Figure 2 is a flowchart of the process performed by the data generation device 100 according to the first embodiment.

[0037] First, in step S202, the scene specification unit 101 sets the scene specification information (prompt). The scene specification unit 101 sets the scene specification information based on, for example, natural language input received from the user via the input interface 5. In addition to natural language, it is also possible to set the scene specification information using an image input by the user via the input interface 5. Scene specification information in natural language may be, for example, "In an urban environment, there are two preceding vehicles and five pedestrians are crossing a crosswalk." When adding an image as scene specification information, the changes from the image are specified in natural language. For example, if there are two preceding vehicles in the image, specifying "There are three preceding vehicles" in natural language will cause the scene specification unit 101 to generate a scene with one more preceding vehicle added while maintaining the style of the input image.

[0038] In step S203, the domain-specific knowledge specification unit 108 acquires the traffic situation description 109 and camera settings 110. The traffic situation description 109 and camera settings 110 may be received from the user via the input interface 5, or they may be stored in memory 2 in advance. Here, the traffic situation specified by the scene specification information and the traffic situation description 109 may be different. Also, if the scene specification information is set in natural language and images, and there is no difference in the number of objects or the shape of roads and white lines between the scene specified by natural language and the scene specified by images, the input image may be converted into a traffic situation description 109 to create the traffic situation description 109.

[0039] In step S204, the reality constraint calculation unit 111 calculates the reality constraints.

[0040] In step S205, the reality consistency score determination unit 107 generates candidate learning data images and correct values ​​using the learning data candidate generation unit 103.

[0041] In step S206, the reality consistency score determination unit 107 calculates a reality consistency score between the training data candidate images and the reality constraints.

[0042] In step S207, the data selection unit 106 determines whether to use the learning data candidate image as the learning data 112 or discard it based on whether the real consistency score between the learning data candidate image and the real constraint is below a predetermined reference value. When the real consistency score is lower than the predetermined reference value, the data selection unit 106 discards the learning data candidate, and the prompt sentence update unit 102 updates the prompt in step S208. On the other hand, when the real consistency score generated by the real consistency score determination unit 107 exceeds the predetermined reference value, the data selection unit 106 stores the learning data candidate image as the learning data 112 in the memory in step S209. The learning data 112 stored in the memory can be used, for example, as the learning data of an image recognition model in an image processing device mounted on a vehicle not shown. By learning an image recognition model using the learning data 112, for example, an image recognition model that outputs the recognition result of an input image acquired by an in-vehicle camera can be created. Although the description of the learning of the image recognition model using the learning data 112 is omitted, it may be executed inside the data generation device 100 or by an external information processing device.

[0043] FIG. 3 is a diagram showing the process executed by the real constraint calculation unit 111 according to the first embodiment.

[0044] The real constraint calculation unit 111 has a 3D model engine and calculates real constraints based on the traffic situation description 109 and the camera setting 110 acquired from the domain specialization knowledge specification unit 108.

[0045] The traffic situation description 109 is represented by the overhead view information 301 that describes the traffic situation specified by the domain specialization knowledge specification unit 108. The overhead view information 301 is information in a form also called BEV (Bird's Eye View), and is provided by an overhead view from the perspective of looking down on the surroundings of the vehicle. It is an all-round expression showing the class, position, and the position of the host vehicle of the object. For example, the overhead view information 301 is input to the real constraint calculation unit 111 in XML format, CSV format, or the like.

[0046] The camera setting 110 is the setting information of the camera specified by the domain-specific knowledge specifying unit 108, includes information such as the mounting position, focal length, type, and distortion of the camera, and is input to the reality constraint calculation unit 111 in a file of the same format as the aerial view information 301.

[0047] The reality constraint calculation unit 111 calculates the reality constraint 304 based on the traffic situation description 109 based on the aerial view information 301 and the camera setting 110. The calculated reality constraint 304 is represented by a simple diagram of lines and objects in the space represented by the traffic situation description 109, and is the coordinate, size, and class information of the object when shooting and rendering the space according to the camera setting 110. The class of the object may be, for example, a white line, a road edge, and others. Other objects may include any dropped objects on the road such as vehicles, people, motorcycles, animals, and furniture.

[0048] The reality constraint calculation unit 111 can calculate the viewing angle of the image, the coordinates of the road edge and the white line at the viewing angle, the position coordinates and size of other objects as the reality constraint 304, and output the reality constraint 304 as visual image information.

[0049] The reality constraint calculation unit 111, for example, uses a simulator to generate a terrain in a three-dimensional environment from the information of the road edge, white line, and intersection in the traffic situation description 109, and arranges boxes on the environment using the position and size information of the objects. A virtual camera is arranged in the three-dimensional environment using the position of the host vehicle in the traffic situation description 109 and the camera setting 110, and the scene in the three-dimensional environment is rendered from the viewpoint of the arranged virtual camera to generate an image. The same environmental recognition AI model as the correct value generation unit 105 is applied to the image generated by rendering to estimate the position coordinates and size of the white line, road edge, and other objects in the image, and output visual image information.

[0050] FIG. 4 is a diagram showing the process executed by the reality consistency score determination unit 107 according to the first embodiment.

[0051] The reality consistency score determination unit 107 includes an object position score calculation unit 404 that calculates an object position score using the correct values ​​403 generated by the correct value generation unit 105, and an object class score calculation unit 405 that calculates an object class score using the correct values ​​403. The reality consistency score determination unit 107 determines the reality consistency score based on the calculated object position score and object class score. The reality consistency score may be the sum of the object position score and object class score or a weighted average. The method for calculating the reality consistency score is not limited to this.

[0052] The object position score calculation unit 404 calculates the error between the object positions included in the ground truth value 403 and the object positions specified by the real-world constraints 304 as the object position score. The object position score calculation unit 404 calculates conversion parameters to match the positions of the generated ground truth value 403 and the real-world constraints 304, for example using the least squares method, and calculates the error ratio between the generated ground truth value 403 and the object positions in the image under the real-world constraints 304. The error ratio is expressed, for example, as the value obtained by subtracting the overlap rate (IOU) of the bounding boxes of the two from 1. Alternatively, the IOU may be calculated using the ratio of pixels that are white lines in the image of the generated ground truth value 403 that are also white lines in the real-world constraints 304.

[0053] In the ground truth generation unit 105, the environment recognition unit 402 estimates the object position and class from the training data candidate images 401 generated by the image generation unit 104 using an environment recognition AI model, and converts the training data candidate images 401 into ground truth values ​​403 of image representations that conform to the reality constraints 304.

[0054] The object position score calculation unit 404 may calculate the percentage of pixel error for the white line class, the percentage of pixel error for the drivable area class, and the percentage of position error for other object position detection, normalize each class error to information between 0 and 1, and then calculate the average.

[0055] The object class score calculation unit 405 calculates the object class score as the difference between the number of objects of each class in the image generated by the image generation unit 104 and the number of objects of each class specified by the reality constraint 304.

[0056] Figure 5 shows the prompt text update unit 102 according to the first embodiment.

[0057] The prompt text update unit 102 includes a scene specification update prompt creation unit 501 and an image generation prompt generation unit 502. The prompt text update unit 102 receives scene specification information and reality consistency score used in the generation of the previous training data candidate images by the image generation unit 104. The scene specification update prompt creation unit 501 receives the scene specification information and reality consistency score used in the generation of the previous training data candidate images and creates a scene specification update prompt.

[0058] The image generation prompt generation unit 502 can utilize machine learning models such as generative AI trained to solve general problems, inputs a prompt for updating scene specification to a large-scale AI model (language model), acquires updated scene specification information, and outputs it. For example, the image generation prompt generation unit 502 outputs updated scene specification information that improves the reasons for consistency degradation in order to increase the reality consistency score. The image generation prompt generation unit 502 may have a large-scale AI model internally, or it may utilize a large-scale AI model provided outside the data generation device 100.

[0059] The scene specification update prompt creation unit 501 integrates the scene specification information used in the previous image generation, the reality consistency score, a sentence explaining the reason for the deterioration of object positions, and a sentence instructing the creation of an image generation prompt that reduces object class errors, and creates a scene specification update prompt. It has a deterioration reason generation function that inputs the generated prompt into a large-scale AI model to generate the reason for the deterioration of object positions.

[0060] Figure 6 is a flowchart of the processes executed by the prompt text update unit 102 and the scene specification unit 101 according to the first embodiment.

[0061] Steps S602 to S606 describe the processing flow in the scene specification update prompt creation unit 501 of the prompt text update unit 102. In step S602, the scene specification information and reality consistency score used to generate the previous training data candidate image are obtained from the image generation unit 104 and added to the scene specification update prompt. In step S603, the prompt text update unit 102 determines whether the object class score is 1 or greater. If it is determined in step S603 that the object class score is 1 or greater, in step S604, the prompt text update unit 102 adds a class error suppression prompt to the scene specification update prompt.

[0062] An example of a class error suppression prompt might be: "A class score of 0 indicates that the object has been created correctly. A class score of 0 or greater indicates that an extra object has been added or that the object class has not been created. Based on this, explain the reason for the degraded class score, such as currently having one car missing. Furthermore, generate a prompt that will reduce the class error."

[0063] In step S605, the prompt text update unit 102 determines whether the object position score is above a predetermined threshold. If it is determined in step S605 that the object position score is above a predetermined threshold, in step S606, the large-scale AI model is used to generate quantitative error causes for the object position based on the generated training data candidate images, reality constraints, and object position score, and an object position error suppression prompt is added to the scene specification update prompt to suppress the error.

[0064] An example of an object position error suppression prompt might be: "Based on the training data candidate images, reality constraints, and object position score, quantitatively tell us whether there are errors in each item, such as the left-right error of the object, the object's scale error, and defects in the road structure, for example, if the road is shifted approximately 0.4m to the left. Furthermore, generate prompts that reduce the object position error."

[0065] On the other hand, if it is determined in step S605 that the object position score is smaller than a predetermined threshold, the object position error suppression prompt is not set, and the process proceeds to step S607.

[0066] In step S607, the image generation prompt generation unit 502 obtains a prompt for updating the scene specification, provides the prompt to the large-scale AI model to generate scene specification information, and outputs it to the scene specification unit 101. An example of the final updated scene specification information is "Delete one vehicle in the front right, slide the road to the right (approximately 0.4m)".

[0067] The scene specification information updated by the prompt text update unit 102 is input again to the learning data candidate generation unit 103 along with the previously generated learning data candidate image. The learning data candidate generation unit 103 then generates a new learning data candidate image that matches the scene specification information from the input scene specification information and the previously generated learning data candidate image, and outputs it to the reality consistency score determination unit 107. The reality consistency score determination unit 107 calculates a reality consistency score that represents the consistency between the generated new learning data candidate image and the scene specification information, and outputs it to the prompt text update unit 102. In this way, the data generation device 100 of this embodiment repeatedly generates images that match the scene specification information while updating the scene specification information.

[0068] Figure 9 shows an example of a scene specification update prompt input to the large-scale AI model by the image generation prompt generation unit 502. The scene specification update prompt consists of the previous image generation prompt, the previous training data candidate image, reality constraints, object class score, object position score, and instructions to the large-scale AI model. In addition to these, the scene specification update prompt may also include information on previous image generation prompts and their rejection rates. The processing shown in Figure 6 is executed by the data generation device 100, for example, by the prompt shown in Figure 9. The format and language of the prompt may be other than those shown in Figure 9. That is, according to the prompt shown in Figure 9, the previous prompt, the training data candidate image, object class score, and object position score are provided, and a prompt is generated that explains the reason for degradation as an instruction, and reduces the object class error and object position error. This makes it possible to generate training data with high consistency (high quality) with real-world driving scenes by giving accurate instructions to the image generation unit 104. Furthermore, by calculating a reality consistency score that indicates consistency with the reality constraints 304, which are constraints that the images generated as training data must adhere to, it is possible to appropriately evaluate whether or not inconsistencies occur in the generated images, such as distortion of roads.

[0069] Figure 7 shows an example of a user interface screen 701 output by the data generation device 100 according to the first embodiment.

[0070] The user interface screen 701 includes an image display area, a discrimination threshold input area 702, and a consistency check result display area 703. The image display area displays candidate training data images 401, reality constraints 304, and correct values ​​403, visualizing the generated data. The consistency check result display area 703 changes its display content depending on the input discrimination threshold. The consistency check result display area 703 includes scenario identification information, generated image identification information, consistency check results, identification information of the generative model used for image generation, and prompts used for image generation.

[0071] As described above, according to the embodiments of the present invention, in the generation of training data images by prompts, a score indicating whether the generated images are realistic is calculated, the prompts are modified based on the calculated score, and the training data images are regenerated, thereby improving the quality of the training data images.

[0072] <Second Embodiment> Next, a second embodiment will be described using Figures 10 and 11. Note that the common points with the first embodiment will not be explained.

[0073] Figure 10 is a configuration diagram of the data generation device 200 according to the second embodiment.

[0074] The data generation device 200 according to the second embodiment differs from the data generation device 100 according to the first embodiment in that it has a camera constraint calculation unit 801 and a learning data candidate generation adjustment unit 802.

[0075] In the data generation device 100 according to the second embodiment, if the camera settings of the images used during training of the image generation model used in the training data candidate generation unit 103 differ from the camera settings 110 specified in the domain-specific knowledge specification unit 108, even if the reality consistency score exceeds the standard value, there is a possibility that reality consistency based on the camera settings (hereinafter also referred to as the camera constraint score) will not be satisfied. Camera settings include the focal length, resolution, and pose of the camera. Reality consistency based on camera settings (camera constraint score) indicates the degree to which the training data satisfies the constraints related to the camera settings. For example, when generating an image pair with a stereo camera as the viewpoint as training data 112, the camera settings of the stereo camera include the field of view and resolution of the right and left cameras, the relative position and orientation between the right and left cameras, and the baseline length. In this case, the image pair generated by the training data candidate generation unit 103 must satisfy reality consistency based on the camera settings of the stereo camera. That is, the camera constraint score of the training data must be equal to or greater than the standard value. However, if the camera settings of the images used during training of the image generation model used in the training data candidate generation unit 103 differ from the camera settings 110 specified in the domain-specific knowledge specification unit 108, and pixels in one image do not match corresponding pixels located horizontally in the other image, then real-world consistency based on camera settings cannot be satisfied. In this case, if the camera settings of the images used during training of the image generation model used in the training data candidate generation unit 103 differ from the camera settings 110 specified in the domain-specific knowledge specification unit 108, the image generation model is adjusted so that the camera constraint score of the generated training data 112 satisfies the standard value, thereby generating training data 112 that satisfies real-world consistency based on camera settings.

[0076] In step S209 of Figure 2, after the training data 112 is stored in memory, the camera constraint calculation unit 801 calculates the camera constraint score for each training data 112 based on the camera settings of each training data 112. For example, the camera constraint calculation unit 801 generates warping data (a third training data image) by performing a geometric transformation on one of the two training data images, the first training data image, using the camera settings. Warping data can be generated, for example, using epipolar geometry through a pinhole camera model or depth estimation. Alternatively, if the image generation model used in the training data candidate generation unit 103 is a free-view generation model, the warping data may be generated using point cloud data generated using Multi-View Stereo technology from the image group used during training of the free-view generation model. Next, the camera constraint calculation unit 801 can calculate the camera constraint score by finding the difference between the generated warping data (third training data image) and the other second training data image. The camera constraint score can be obtained, for example, from the squared error of the difference between the warping data and the corresponding training data 112. However, the method for generating the warping data and the method for calculating the camera constraint score are not limited to this. If the camera constraint calculation unit 801 calculates the camera constraint score in this way and it falls below a reference value, it instructs the training data candidate generation adjustment unit 802 to adjust the training data candidate generation unit 103. If the camera constraint score exceeds the reference value, the camera constraint calculation unit 801 may terminate the process without updating the training data 112.

[0077] The training data candidate generation adjustment unit 802 adjusts the output of the image generation model used by the training data candidate generation unit 103 so that the camera constraint score of the training data 112 exceeds a reference value. One way to adjust the output of the image generation model is to update the parameters of the image generation model. The parameters to be updated are one or more parameters that can be adjusted during inference of the image generation model, and are also called hyperparameters. Parameter updates can be performed using known deep learning methods. The training data candidate generation unit 103 generates training data candidates using the image generation model whose parameters have been updated by the training data candidate generation adjustment unit 802, and the data selection unit 106 saves the generated training data candidates in memory as training data 112. The training data candidate generation adjustment unit 802 may repeatedly update the parameters of the image generation model until the camera constraint score of the training data 112 exceeds a reference value, and if the camera constraint score of the training data 112 exceeds a reference value, the data selection unit 106 may save the images generated by the image generation model with the updated parameters set in memory as training data 112. Alternatively, the baseline value of the camera constraint score may be changed sequentially, and the above process may be repeated.

[0078] Figure 11 shows the processing flow according to the second embodiment. Steps S202 to S209 are the same as in the first embodiment, so their explanation is omitted.

[0079] In step S901, the camera constraint calculation unit 801 calculates the camera constraint score for each piece of training data 112 based on the camera settings of that piece of training data 112, after the training data 112 has been saved.

[0080] In step S902, the camera constraint calculation unit 801 determines whether the calculated camera constraint score is below a reference value. If the camera constraint score is above the reference value, the training data 112 is saved to memory and the flow ends. If the camera constraint score is below the reference value, the process proceeds to step S903.

[0081] In step S903, the parameters of the image generation model used in the training data candidate generation unit 103 are updated.

[0082] In step S904, the training data candidate generation unit 103 generates training data candidates again using the image generation model after the parameter update.

[0083] In step S905, the data selection unit 106 stores the generated training data candidates in memory as training data 112. Alternatively, after generating training data candidates in step S904, the camera constraint calculation unit 801 may calculate the camera constraint score and re-determine whether the camera constraint score of the training data candidates generated by the image generation model after parameter updates meets the criteria value.

[0084] As described above, according to the second embodiment of the present invention, the image generation model is updated so that the training data generated by the image generation model satisfies the actual camera settings, thereby improving the quality of the training data images.

[0085] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the configurations described. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0086] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0087] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

[0088] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A data generation device for generating training data for an image recognition model that outputs recognition results of an input image, comprising a computer having an arithmetic unit for executing a program and a storage device accessible by the arithmetic unit, wherein the arithmetic unit acquires scene specification information specifying environmental elements constituting a scene and objects included in the scene, calculates reality constraints indicating image conditions to which the training data must follow based on a traffic situation description in a predetermined format and camera settings which are camera setting information, inputs the scene specification information to an image generation model, generates candidate training data images with the camera as the viewpoint, determines a reality consistency score indicating the consistency between the generated candidate training data images and the reality constraints, and decides whether or not to update the training data specification information specifying the training data according to the reality consistency score.

2. A data generation device according to claim 1, wherein the arithmetic unit stores the candidate learning data images whose reality consistency score is equal to or greater than a reference value as learning data in the storage device.

3. A data generation device according to claim 1, wherein the arithmetic unit, when it decides to update the learning data designation information, updates the learning data designation information based on the reality consistency score, inputs the updated learning data designation information to the image generation model to generate new learning data candidate images with the camera as the viewpoint, determines a new reality consistency score indicating the consistency between the new learning data candidate images and the reality constraints, and stores the new learning data candidate images whose new reality consistency score is equal to or greater than a reference value as learning data in the storage device.

4. A data generation device according to claim 1, wherein the arithmetic unit calculates an object position score indicating the ratio of the position error of an object in the training data candidate image with respect to the reality constraints, calculates an object class score indicating the ratio of the class error of an object for each class in the training data candidate image with respect to the reality constraints, and determines the reality consistency score based on the object position score and the object class score.

5. A data generation device according to claim 1, wherein the computing device inputs a scene specification update prompt to a language model that instructs the generation of an image with a good reality consistency score, and obtains updated training data specification information.

6. A data generation device according to claim 5, wherein the arithmetic unit generates reasons for deterioration of the object's position by inputting a prompt to a language model that is generated based on the reality constraints and the object position information of the candidate learning data images, and that prompts the language model to explain the reasons for deterioration of the object's position; obtains updated scene specification information by inputting scene specification information and a prompt to the language model that instructs the language model to change the scene specification information so that the generated reasons for deterioration improve; and updates the first specification information based on the obtained updated learning data specification information.

7. A data generation device according to claim 6, wherein the calculation device generates prompts for a language model to generate explanations for the reasons for deterioration of at least one of the following: left-right displacement of each object including a road and a white line, a scale error, and a defect in the road structure.

8. A data generation device according to claim 1, wherein the arithmetic unit calculates a camera constraint score, which is the degree of satisfaction of the learning data with respect to constraints relating to the camera settings, based on the learning data stored in the storage device, and adjusts the parameters of the image generation model so that the camera constraint score is equal to or greater than the reference value if the camera constraint score is less than or equal to a reference value.

9. A data generation device according to claim 8, wherein the training data includes a first image and a second image which are a pair of images viewed from a stereo camera, and the computing device generates a third image by geometrically transforming the first image based on the camera settings, and calculates the camera constraint score by finding the difference between the second image and the third image.

10. A data generation method for generating training data for an image recognition model that outputs recognition results of an input image, wherein the data generation device is composed of a computer having an arithmetic unit that executes a program and a storage device accessible by the arithmetic unit, and is capable of deriving an answer using a language model that executes a task according to an input prompt, and the data generation method comprises: a step of the arithmetic unit acquiring scene specification information that specifies environmental elements constituting a scene and objects included in the scene; a step of the arithmetic unit calculating reality constraints that indicate conditions on an image to which the training data follows, based on a traffic situation description in which traffic conditions are described in a predetermined format and camera settings which are camera setting information; a step of the arithmetic unit inputting the scene specification information to an image generation model and causing it to generate candidate training data images with the camera as the viewpoint; a step of the arithmetic unit determining a reality consistency score that indicates the consistency between the generated candidate training data images and the reality constraints; and a step of the arithmetic unit determining whether or not to update the training data specification information that specifies the training data according to the reality consistency score.

Images