Travel scene extraction system and travel scene extraction method

The driving scene extraction system addresses the limitations of existing AD/ADAS verification by identifying problematic scenes through data analysis and generative AI, enhancing the accuracy and efficiency of automated driving systems.

WO2026239658A1PCT designated stage Publication Date: 2026-11-19ASTEMO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/042606
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-12
Filing Date
2025-12-05
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing methods for extracting difficult driving scenes for AD/ADAS function verification are inadequate as they rely on unclear tagging, fail to identify issues when recognition is correct but judgment fails, and disrupt the logic balance under low reliability conditions, and do not capture obstructed scenes as recognition failures.

Method used

A driving scene extraction system utilizing a computer with an arithmetic unit and storage device, including a vehicle data collection unit, first and second logic execution units, a correct/incorrect determination unit, and a communication unit, to identify problematic scenes without tagging by analyzing vehicle data and utilizing generative AI for correctness determination.

Benefits of technology

Efficiently extracts a wide range of situations where logic is weak, improving AD/ADAS functions by identifying incorrect judgments and obstructed scenes, enhancing system performance and operational efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025042606_19112026_PF_FP_ABST
    Figure JP2025042606_19112026_PF_FP_ABST
Patent Text Reader

Abstract

This travel scene extraction system is characterized in that: a vehicle data collection unit collects vehicle data of a host vehicle and vehicle peripheral environment measurement data obtained by measuring a vehicle peripheral environment; a first logic execution unit acquires the collected data, executes a vehicle control function, and records a recognition result and a determination result; a second logic execution unit refers to relationship data defining a relationship of processes included in a first logic in the vehicle control function, and creates, by tracking the relationship defined in the relationship data, a prompt including at least two processes and an instruction sentence for determining whether or not a determination result of the first logic is correct; a correctness determination unit uses the created prompt to acquire a result of determination of correctness of the determination result of the first logic, and classifies a scene used for the determination as a difficult scene if the determination result of the first logic is incorrect; and a communication unit outputs the determined difficult scene.
Need to check novelty before this filing date? Find Prior Art

Description

Driving Scene Extraction System, and Driving Scene Extraction Method Incorporation by Reference

[0001] This application claims priority to Japanese Patent Application No. 2025-79787 filed on May 12, 2025, the entire content of which is incorporated herein by reference.

[0002] The present invention relates to a driving scene extraction system, and particularly to a technology for extracting specific driving scenes suitable for improving the efficiency of AD / ADAS function verification.

[0003] In recent years, various initiatives have been advanced toward the popularization of automated driving. To improve the existing logic of automated driving functions and advanced driver assistance system functions (AD / ADAS functions), the accumulation of actual driving data for use in function verification and improvement is under consideration. For efficient function verification, it is necessary to extract and present specific driving scenes (difficult scenes) that the existing logic is not good at. To extract difficult scenes for the existing logic from accumulated driving data, a common general method is to pre-tag driving scenes (for example, with characteristics of weather, characteristics of location, etc.) and perform searches based on the tags.

[0004] However, when the difficult scenes of the existing logic are unknown, the tags required for search are unclear, making it difficult to set appropriate tags in advance. Accordingly, there is a demand for a method of extracting difficult scenes that does not require tagging.

[0005] The following prior art exists as background technology for this field. For example, Patent Document 1 (Japanese Patent Application Publication No. 2019-21201) discloses a learning server comprising: a data acquisition unit that acquires photographic data to which evaluations are attached in vehicle image recognition processing and which is associated with additional information including information on the date and time of shooting and location information of the vehicle; a data holding unit that stores the photographic data and stores a dataset used for learning parameters used in the image recognition processing; a data determination unit that determines whether or not it is a learning timing when the photographic data is acquired from the vehicle; a server-side learning unit that, if it is a learning timing, selects data to be used for learning from the data holding unit by referring to the additional information associated with the photographic data and performs learning; and an update unit that notifies the vehicle of the learning results.

[0006] Furthermore, Patent Document 2 (Japanese Patent Application Publication No. 2021-43622) discloses a recognition model distribution system for distributing recognition models that recognize the external world to a vehicle, comprising: a data analysis unit that acquires a recognition model suitable for the vehicle from a plurality of recognition models, compares it with data from an external world recognition unit provided in the vehicle to determine locations where recognition of the traffic scene in the acquired recognition model has failed, and reflects the locations where recognition has failed in the acquired recognition model to create a three-dimensional computer graphics image; a parameter generation unit that generates parameters for generating a plurality of traffic scenes similar to the three-dimensional computer graphics image created by the data analysis unit; a three-dimensional object generation unit that executes a traffic flow simulation using a traffic flow model created with the parameters generated by the parameter generation unit to generate a three-dimensional traffic object; a training data generation unit that creates training data necessary for learning the recognition model based on the three-dimensional traffic object generated by the three-dimensional object generation unit; a learning unit that executes learning of the recognition model based on the training data generated by the training data generation unit; and a recognition model distribution unit that distributes a recognition model suitable for the vehicle from the plurality of recognition models.

[0007] When detecting and controlling lane changes of other vehicles, the process proceeds in the following order: other vehicle recognition / lane recognition → lane change determination → control content determination. Therefore, it is necessary to identify not only cases where other vehicle recognition / lane recognition is difficult, but also cases where lane change determination is difficult, as problematic scenarios.

[0008] The method described in Patent Document 1 has the following problems: (1) While problematic data is extracted when recognition fails, problematic data is not extracted when recognition is performed correctly but lane change judgment fails. For example, if an object and lane are recognized correctly, but the lane change judgment is incorrect, the problematic data is not extracted based solely on the reliability of the recognition. (2) Focusing only on specific problematic data under low reliability conditions disrupts the balance of the overall logic and reduces the generalizability of the logic.

[0009] Furthermore, in the technology described in Patent Document 2, when an object is obstructed and cannot be recognized, the scene that should be extracted is not extracted as a recognition failure point. For example, if a lane is obstructed and the recognition model cannot recognize the lane, the lane will not be visible in the reconstructed video. In this case, there is no difference between the in-vehicle camera video and the reconstructed video, so it is not extracted as a recognition failure point.

[0010] This invention has been made in view of these problems, and aims to extract a wide range of situations where logic is weak without the need for tagging.

[0011] A representative example of the invention disclosed in this application is as follows: In other words, the driving scene extraction system is comprised of a computer having an arithmetic unit that performs predetermined processing and a storage device accessible by the arithmetic unit, and comprises a vehicle data collection unit, a first logic execution unit, a second logic execution unit, a correct / incorrect determination unit, and a communication unit, wherein the vehicle data collection unit collects vehicle data of its own vehicle and vehicle surrounding environment measurement data that measures the surrounding environment of the vehicle, and stores the collected data in the storage device, the first logic execution unit acquires the data collected from the storage device and executes a vehicle control function, and records the recognition result and the determination result in the storage device, the second logic execution unit refers to relationship data that defines the relationships between processes included in the first logic in the vehicle control function, and creates a prompt that includes at least two processes and an instruction statement that determines the correctness of the determination result of the first logic by following the relationships defined in the relationship data, the correct / incorrect determination unit uses the created prompt to acquire the correctness of the determination result of the first logic, and if the determination result of the first logic is incorrect, it classifies the scene used for the determination as a problematic scene, and the communication unit outputs the determined problematic scene.

[0012] According to one aspect of the present invention, it is possible to extract scenes where logic is weak without tagging. Problems, configurations, and effects other than those mentioned above will be clarified by the following description of the embodiments.

[0013] This figure shows a use case of an embodiment of the present invention. This figure shows another use case of an embodiment of the present invention. This figure shows the configuration of the system of the present invention. This figure shows the details of the user communication unit of this embodiment. This figure shows the details of the first logic improvement effect confirmation unit of this embodiment. This figure shows an example of a vehicle recognition result stored in the recognition result storage unit of this embodiment. This figure shows an example of a lane recognition result stored in the recognition result storage unit of this embodiment. This figure shows an example of a lane change judgment result stored in the judgment result storage unit of this embodiment. This figure shows an example of a knowledge graph stored in the first logic specification storage unit of this embodiment. This figure shows an example of processing node information of a knowledge graph stored in the first logic specification storage unit of this embodiment. This figure shows an example of data node information of a knowledge graph stored in the first logic specification storage unit of this embodiment. This figure shows an example of script information stored in the first logic specification storage unit of this embodiment. This figure shows an example of a candidate list of effective prompt configurations created by the effective prompt configuration selection unit of this embodiment. This figure shows an example of an effective prompt configuration stored in the effective prompt configuration storage unit of this embodiment. This figure shows an example of a prompt statement generated by the effective prompt creation unit of this embodiment. This figure shows an example of a list of difficult scenes stored in the difficult scene storage unit of this embodiment. This figure shows an example of a list of difficult scene aggregation results stored in the aggregation result storage unit of this embodiment. This figure shows an example of test data for confirming improvement effects stored in the test data storage unit for confirming improvement effects of this embodiment. This figure shows an example of first logic accuracy stored in the first logic accuracy storage unit of this embodiment. This figure shows an example of a normal screen displayed on the user terminal of this embodiment. This figure shows an example of an instruction statement setting / update screen displayed on the user terminal of this embodiment. This figure shows an example of a logic specification setting / update screen displayed on the user terminal of this embodiment. This figure shows an example of a difficult scene confirmation screen displayed on the user terminal of this embodiment. This figure shows an example of an improvement effect confirmation screen displayed on the user terminal of this embodiment. This figure shows an example of a test data addition screen displayed on the user terminal of this embodiment. This is a flowchart of the instruction statement setting / update process of this embodiment. This is a flowchart of the logic specification setting / update process of this embodiment. This is a flowchart of the process executed by the user communication unit of this embodiment.This is a flowchart of the test data addition process in this embodiment. This is a flowchart detailing the process from steps S102 to S103 in Figure 19. This is a flowchart detailing step S1023 in Figure 20. This is a flowchart of the valid prompt configuration selection process in this embodiment. This is a flowchart of the improvement effect confirmation process in this embodiment. This is a block diagram showing the physical configuration of the server in this embodiment.

[0014] <Example 1> Figure 1A shows a use case of an embodiment of the present invention.

[0015] As shown in Figure 1A, vehicle 1 is equipped with multiple types of sensors 11 and cameras 12, and vehicle data is collected using the sensors 11 and cameras 12. The collected vehicle data is transmitted to server 2 via the network.

[0016] The vehicle data is basically transmitted to server 2 in real time via the network. Video data may be stored in the internal storage of the camera 12 mounted on the vehicle, and sensor data may be temporarily stored in a NAS (Network Attached Storage) installed inside the vehicle. In this case, when vehicle 1 returns to the base after driving, the stored data may be uploaded to server 2 in bulk using Wi-Fi or a wired network.

[0017] With the aforementioned configuration, vehicle data can be transferred using a combination of real-time and non-real-time transmission, enabling flexible data management and transmission tailored to the operating environment. Furthermore, Server 2 stores vehicle data collected from multiple vehicles.

[0018] The data stored on server 2 can be used for various purposes. For example, user 4 can use user terminal 3 to perform data analysis, develop autonomous driving algorithms, and verify AD / ADAS functions.

[0019] Server 2 connects to the external generating AI 5 via the API of the generating AI service, sends a question (prompt) to the generating AI, and receives a response. Specifically, this is done in the following steps. Generating AI 5 learns from information available on the internet. For example, if the logic is to determine lane changes, it is a machine learning model that has learned from image data tagged with either "lane change" or "no lane change".

[0020] First, Server 2 connects to Generation AI 5 using the API provided by Generation AI 5. Then, Server 2 configures a prompt to send to Generation AI 5 and sends the configured prompt to Generation AI 5. This prompt explicitly includes the question and the target of analysis. Server 2 utilizes the content of the response returned by Generation AI 5 for data analysis and decision-making.

[0021] By utilizing an external AI generation service, Server 2 can achieve more advanced data processing, improve the accuracy of vehicle data analysis, and enhance operational efficiency.

[0022] For example, Server 2 creates a prompt containing detailed information about the lane change judgment result in a driving scene using the logic under development, and sends it to Generator AI 5 to determine whether the lane change judgment result in the driving scene is correct or incorrect. If the response returned by Generator AI 5 is "the lane change judgment result of the logic is incorrect", Server 2 determines that the corresponding driving scene is a scene in which the logic is not well-suited.

[0023] This configuration allows us to efficiently identify problematic scenarios and improve the logic by utilizing the Generative AI 5, thereby contributing to an overall improvement in system performance.

[0024] Figure 1B shows another use case of an embodiment of the present invention.

[0025] The vehicle 1, server 2, user terminal 3, and user 4 shown in Figure 1A are the same in Figure 1B, but the configuration shown in Figure 1B differs in that the server 2 does not connect to an external AI generation service and has a large language model (LLM) 6 inside the server 2.

[0026] The configuration shown in Figure 1B allows data analysis and logic evaluation to be performed within Server 2 without relying on external services.

[0027] Figure 2 shows the configuration of the driving scene extraction system in this embodiment, and mainly shows the logical configuration of Server 2 in Figure 1A. There are two ways to operate the server, as shown in Figures 1A and 1B, but Figure 2 explains the use case (Figure 1A) in which the server connects to an externally generated AI service via an API.

[0028] As shown in Figure 2, the server 2 includes a vehicle data collection unit 201, a vehicle data storage unit 202, a first logic execution unit 203, a first logic improvement effect confirmation unit 204, a recognition result storage unit 205, a judgment result storage unit 206, an effective prompt configuration selection unit 207, a first logic specification storage unit 208, a second logic execution unit 209, an instruction statement storage unit 210, a difficult scene storage unit 211, an aggregation processing unit 212, an aggregation result storage unit 213, a user communication unit 214, an effective prompt configuration storage unit 215, and a first logic accuracy storage unit 216.

[0029] In the first logic execution unit 203, the first logic specification storage unit 208, and the first logic improvement effect confirmation unit 204, the "first logic" is any logic that performs AD (Automated Driving) or ADAS (Advanced Driver-Assistance System) functions. In this specification, a logic that recognizes lanes and vehicles based on video data and determines lane changes based on the recognition results will be described as an example.

[0030] The vehicle data collection unit 201 communicates with the vehicle 1 and stores the received vehicle data (e.g., video data, vehicle speed, etc.) in the vehicle data storage unit 202.

[0031] The vehicle data storage unit 202 stores video data and vehicle speed data collected from the vehicle 1. The data stored in the vehicle data storage unit 202 is acquired by the first logic execution unit 203 and other logic execution units and used as input data when executing logic. This configuration allows for efficient management of data collected from the vehicle 1 and appropriate utilization in the execution of various logics.

[0032] The first logic execution unit 203 retrieves the script for the first logic from the first logic specification storage unit 208, retrieves the necessary data from the vehicle data storage unit 202, and executes the retrieved script for the first logic using the retrieved data as parameters. The first logic execution unit 203 stores the recognition result, which is the execution result of the first logic, in the recognition result storage unit 205, and stores the judgment result, which is the execution result of the first logic, in the judgment result storage unit 206. For example, if the first logic recognizes lanes and vehicles from video data and determines a lane change based on the lane recognition result and vehicle recognition result, the first logic execution unit 203 retrieves the video data from the vehicle data storage unit 202, stores the vehicle recognition result and lane recognition result, which are the execution results, in the recognition result storage unit 205, and stores the lane judgment result in the judgment result storage unit 206.

[0033] The first logic improvement effect verification unit 204 retrieves the script for each version of the first logic from the first logic specification storage unit 208, executes the script for the first logic using pre-tagged test data as input, and calculates the accuracy rate of the first logic. The first logic improvement effect verification unit 204 stores the calculated accuracy rate and the version of the first logic in the first logic precision storage unit 216. Details of the first logic improvement effect verification unit 204 will be explained with reference to Figure 4.

[0034] The recognition result storage unit 205 and the judgment result storage unit 206 store the recognition result and judgment result of the first logic, respectively. The valid prompt creation unit 2091 of the second logic execution unit 209 retrieves the stored recognition result and judgment result and creates a prompt for the generated AI 5. Details of the recognition result storage unit 205 will be explained with reference to Figure 5. Details of the judgment result storage unit 206 will be explained with reference to Figure 6.

[0035] The active prompt configuration selection unit 207 acquires knowledge graph information related to the first logic stored in the first logic specification storage unit 208 and acquires all data items necessary to execute the first logic. Subsequently, the active prompt configuration selection unit 207 generates a list of candidate prompt configurations by combining the presence or absence of the acquired data items. For example, if the first logic is lane change judgment logic, the data items necessary to execute the first logic are video data, lane recognition result, vehicle recognition result, and lane change judgment result.

[0036] Furthermore, the active prompt configuration selection unit 207 has pre-prepared correct tagged data (for example, video data + recognition result + judgment result + correct answer) and can acquire this data. In addition, the active prompt configuration selection unit 207 acquires instruction sentences from the instruction sentence storage unit 210 and creates prompt sentences corresponding to each prompt configuration candidate based on the prompt configuration.

[0037] The active prompt configuration selection unit 207 inputs the created prompt sentence multiple times into the external generation AI service 5 and evaluates whether any of the multiple responses obtained from the multiple inputs include an incorrect response. If an incorrect response is output from the generation AI, the active prompt configuration selection unit 207 removes the corresponding prompt configuration candidate from the configuration candidate list. This evaluation process is performed for all candidates. After evaluation of all candidates is complete, if multiple configuration candidates remain in the configuration candidate list, it is preferable to select the configuration with the most input data elements (i.e., the most information) as the active prompt configuration. The active prompt configuration selection unit 207 stores the finally selected active prompt configuration in the active prompt configuration storage unit 215.

[0038] This process allows for the efficient and accurate selection of the optimal prompt configuration for executing the first logic.

[0039] The second logic execution unit 209 includes an effective prompt creation unit 2091 and a correct / incorrect determination unit 2092. The effective prompt creation unit 2091 obtains the effective prompt configuration necessary to determine the correctness of the latest version of the first logic execution result from the effective prompt configuration storage unit 215. Furthermore, the second logic execution unit 209 obtains instruction statements from the instruction statement storage unit 210 and a knowledge graph related to the first logic from the first logic specification storage unit 208. Then, based on the effective prompt configuration, the second logic execution unit 209 obtains the necessary data from the corresponding storage units (for example, the vehicle data storage unit 202, the recognition result storage unit 205, and the determination result storage unit 206) and creates an effective prompt.

[0040] Here, the effective prompt configuration is described in the format of, for example, "instruction statement + summary + input data (video data + vehicle recognition result + lane recognition result + lane change judgment result)". Specific examples of the effective prompt configuration and details of the effective prompt statements created based on the effective prompt configuration will be explained with reference to Figures 7, 8, 9, and 10.

[0041] The correct / incorrect judgment unit 2092 connects to the external generation AI service 5 via the generation AI's API and inputs the prompt created by the valid prompt creation unit 2091 to the generation AI 5. The correct / incorrect judgment unit 2092 also checks the response output by the generation AI 5 and determines the correctness of the execution result of the first logic based on that response.

[0042] Furthermore, if the correct / incorrect judgment unit 2092 determines that the execution result of the first logic is incorrect, it stores the corresponding scene as a weak scene in the weak scene storage unit 211. With this configuration, information necessary for performance evaluation and improvement of the first logic can be efficiently managed by utilizing the generated AI.

[0043] The instruction message storage unit 210 stores the updated instruction message sent by the user terminal 3. The basic instruction message is overwritten when it is updated. The active prompt creation unit 2091 of the active prompt configuration selection unit 207 and the second logic execution unit 209 uses the instruction message stored in the instruction message storage unit 210 when creating a prompt.

[0044] The weak scene storage unit 211 stores the weak scenes extracted by the correctness determination unit 2092 of the second logic execution unit 209. Details of the data items and data examples of the weak scenes will be described with reference to FIG. 11.

[0045] The aggregation processing unit 212 classifies the weak scenes. For example, the classification may be performed by a specific rule-based method such as classification using keywords. The keywords may be extracted, for example, from text indicating the determination reason included in the response from the generative AI. After the classification processing, the value of the total number of weak scenes for each classified feature is stored in the aggregation result storage unit 213 as the aggregation result for each classification.

[0046] The aggregation result storage unit 213 stores the aggregation processing result of the aggregation processing unit 212. Details of the data items and data examples of the aggregation result will be described with reference to FIG. 12.

[0047] The user communication unit 214 receives data and instructions input by the user 4 to the user terminal 3, and communicates with the functional blocks of the server 2 corresponding to the received instructions. Details of the user communication unit 214 will be described with reference to FIG. 3.

[0048] The effective prompt configuration storage unit 215 stores the optimal effective prompt configuration for each version of the first logic, in order to determine the correctness of the first logic of each version selected by the effective prompt configuration selection unit 207. The data items and data examples of the effective prompt configuration in the effective prompt configuration storage unit 215 will be described with reference to FIG. 9.

[0049] The first logic accuracy storage unit 216 stores the correct answer rate (that is, accuracy) of the first logic of each version, which is calculated by the first logic improvement effect confirmation unit 204. The data items and data examples of the first logic accuracy storage unit 216 will be described with reference to FIG. 14.

[0050] FIG. 24 is a block diagram showing the physical configuration of the server 2 of the present embodiment.

[0051] In this embodiment, Server 2 is comprised of a computer having a processor (CPU) 2001, memory 2002, auxiliary storage device 2003, and communication interface 2004. Server 2 may also have an input interface 2005 and an output interface 2008.

[0052] The processor 2001 is an arithmetic unit that executes programs stored in the memory 2002. By executing various programs, the processor 2001 enables the functions of each functional unit of the server 2 (for example, the vehicle data collection unit 201, the vehicle data storage unit 202, the first logic execution unit 203, the first logic improvement effect confirmation unit 204, the effective prompt configuration selection unit 207, the second logic execution unit 209, the aggregation processing unit 212, and the user communication unit 214). Note that some of the processing performed by the processor 2001 when executing programs may be performed by other arithmetic units (for example, hardware such as ASICs and FPGAs).

[0053] Memory 2002 includes ROM, a non-volatile memory element, and RAM, a volatile memory element. ROM stores immutable programs (e.g., BIOS). RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 2001 and data used during program execution.

[0054] The auxiliary storage device 2003 is, for example, a high-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device 2003 also stores data used by the processor 2001 when executing a program (for example, the recognition result storage unit 205, the judgment result storage unit 206, the first logic specification storage unit 208, the instruction statement storage unit 210, the problematic scene storage unit 211, the aggregation result storage unit 213, the effective prompt configuration storage unit 215, and the first logic precision storage unit 216), as well as the program executed by the processor 2001. In other words, the program is read from the auxiliary storage device 2003, loaded into memory 2002, and executed by the processor 2001 to realize the various functions of the server 2.

[0055] The communication interface 2004 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0056] The input interface 2005 is an interface to which input devices such as a keyboard 2006 and a mouse 2007 are connected and to receive input from the operator, and constitutes the user communication unit 214. The output interface 2008 is an interface to which output devices such as a display device 2009 and a printer (not shown) are connected and to output the program execution results in a format that the user can see, and constitutes the user communication unit 214. In addition, a user terminal 3 connected to the server 2 via a network may provide the input and output devices. In this case, the server 2 may have the functionality of a web server, and the user terminal may access the server 2 using a predetermined protocol (e.g., http).

[0057] The program executed by processor 2001 is provided to server 2 via removable media (CD-ROM, flash memory, etc.) or a network, and stored in non-volatile auxiliary storage device 2003, which is a non-temporary storage medium. For this reason, server 2 should have an interface for reading data from the removable media.

[0058] Server 2 is a computer system that consists of one physical computer or multiple logically or physically configured computers, and may operate on a virtual computer built on multiple physical computer resources. For example, multiple programs that implement the functions of Server 2 may each run on separate physical or logical computers, or multiple programs may be combined and run on a single physical or logical computer.

[0059] Figure 3 shows the details of the user communication unit 214 in this embodiment.

[0060] The user communication unit 214 includes a communication unit 2141 for the effective prompt configuration selection unit that communicates with the effective prompt configuration selection unit 207, a communication unit 2142 for the first logic execution unit that communicates with the first logic execution unit 203, a communication unit 2143 for the difficult scene memory unit that communicates with the difficult scene memory unit 211, a communication unit 2144 for the aggregated result memory unit that communicates with the aggregated result memory unit 213, a communication unit 2145 for the first logic improvement effect confirmation unit that communicates with the first logic improvement effect confirmation unit 204, a communication unit 2146 for the first logic precision memory unit that communicates with the first logic precision memory unit 216, a communication unit 2148 for the first logic specification memory unit that communicates with the first logic specification memory unit 208, and a communication unit 2147 for the user terminal that communicates with the user terminal 3.

[0061] Specifically, the communication unit 2141 for the active prompt configuration selection unit sends a request to the active prompt configuration selection unit 207 to start selecting the active prompt configuration and receives notification of the completion of the selection. The communication unit 2142 for the first logic execution unit sends a request to the first logic execution unit 203 to start executing the first logic. The communication unit 2143 for the difficult scene memory unit retrieves the difficult scene list 2110 from the difficult scene memory unit 211. The retrieved difficult scene list 2110 is sent to the user terminal 3 by the communication unit 2147 for the user terminal. The communication unit 2144 for the aggregation result memory unit retrieves the aggregation result from the aggregation result memory unit 213. The retrieved aggregation result is sent to the user terminal 3 by the communication unit 2147 for the user terminal. The communication unit 2145 for the first logic improvement effect confirmation unit sends a request to the first logic improvement effect confirmation unit 204 to start confirming the improvement effect. The communication unit 2146 for the first logic accuracy memory unit obtains a list of accuracy rates for each version of the first logic from the first logic accuracy memory unit 216. The obtained list of accuracy rates is sent to the user terminal 3. The communication unit 2148 for the first logic specification memory unit stores the update contents of the first logic received from the user terminal 3 (for example, a knowledge graph related to the first logic and a script for the first logic) in the first logic specification memory unit 208.

[0062] Figure 4 shows the details of the first logic improvement effect confirmation unit 204 in this embodiment.

[0063] The first logic improvement effect verification unit 204 includes an improvement effect verification test data storage unit 2041 and an accuracy rate calculation unit 2042. The improvement effect verification test data storage unit 2041 stores a dataset used to evaluate the accuracy of the first logic, and this dataset is tagged with correct answers by the user. Details of this dataset will be explained with reference to Figure 13.

[0064] The accuracy calculation unit 2042 retrieves the specifications and scripts of the first logic for each version from the first logic specification storage unit 208 as needed, and also retrieves input data from the improvement effect confirmation test data storage unit 2041. Then, the first logic improvement effect confirmation unit 204 executes the first logic using the acquired data. The first logic improvement effect confirmation unit 204 compares the execution result of the first logic with the correct answer tag stored in the improvement effect confirmation test data storage unit 2041, and determines that the execution result is correct if it matches the correct answer tag. After comparing all the test data with the correct answer tag, the first logic improvement effect confirmation unit 204 calculates the accuracy rate and stores the calculated accuracy rate (for example, in the format "70%") and the associated version of the first logic in the first logic precision storage unit 216.

[0065] Figures 5A and 5B show examples of the data structure and data content of the results of the recognition processing of the first logic (vehicle recognition results and lane recognition results in this embodiment) stored in the recognition result storage unit 205 of this embodiment.

[0066] The recognition result storage unit 205 stores, for example, the vehicle recognition result 20501 shown in Figure 5A and the lane recognition result 20502 shown in Figure 5B.

[0067] Each record of the vehicle recognition result 20501 shown in Figure 5A includes video data 2051, frame 2052, tracking ID 2053, class 2054, and coordinate (bbox) 2055 data.

[0068] The video data 2051 contains identification information, such as the video file name, for identifying the video data that is the target of the vehicle recognition process.

[0069] Frame 2052 stores the temporal division unit of the video data. The video data contains multiple still images (frames), and this item is used to indicate a specific image to which vehicle recognition processing has been applied. Each frame can be identified by an index value that indicates its chronological position within the video, and the index value clearly identifies which timing in the video corresponds to the vehicle recognition result.

[0070] Tracking ID 2053 stores identification information assigned to an object (e.g., a vehicle) detected in the video during the tracking process. This allows the same object to be consistently identified across multiple frames.

[0071] Class 2054 stores classification information to identify the type of object being recognized. Objects detected by the vehicle recognition process may belong to classes such as "passenger car," "bus," or "truck."

[0072] The coordinate (bbox) 2055 stores the coordinates of a rectangular area indicating the position of an object within the video. These coordinates are represented by the coordinate values ​​of the rectangle surrounding the object, and the size and position information of the object can be accurately determined using these coordinates.

[0073] Each record of the lane recognition result 20502 shown in Figure 5B includes video data 2056, frame 2057, lane number 2058, and coordinate 2059 data.

[0074] The video data 2056 contains identification information, such as the video file name, for identifying the video data that is subject to lane recognition processing.

[0075] Frame 2057, like frame 2052, stores an index value that indicates its chronological position within the video.

[0076] Lane number 2058 stores identification information assigned to identify the recognized lane. Since roads often have multiple lanes, each lane is assigned a number to distinguish it. This identification information is represented numerically, allowing for the clear identification of a specific lane within the video.

[0077] Coordinate 2059 stores the coordinates of a set of points that represent the position of each recognized lane. Each coordinate point functions as fundamental data for the lane and is used to accurately depict the shape and position of the lane. The coordinates are expressed in pixels on the screen, for example, and each point indicates its position in two-dimensional space (e.g., x-coordinate, y-coordinate). Note that the number of points in the set used to depict the lane may also differ because the shape of the detected lane may vary.

[0078] It is preferable to manage the vehicle recognition result 20501 and the lane recognition result 20502 data separately. Furthermore, depending on the logic, other recognition results (e.g., pedestrian recognition results) may be stored in the recognition result storage unit 205. The lane recognition result and vehicle recognition result are associated with the video data and frames. That is, the lane change determination process determines whether a lane change has occurred based on the lane recognition result and vehicle recognition result if the data stored in video data 2051 and video data 2056 are the same, and the data stored in frame 2052 and frame 2057 are the same.

[0079] Figure 6 shows an example of the data structure and data content of the result of the first logic's judgment process (in this embodiment, the lane change judgment result) stored in the judgment result storage unit 206 of this embodiment.

[0080] The judgment result storage unit 206 stores, for example, the lane change judgment result 2060 shown in Figure 6. Each record of the lane change judgment result 2060 includes video data 2061, frame 2062, tracking ID 2063, and data indicating whether or not a lane change occurred 2064.

[0081] The video data 2061 contains identification information, i.e., the video file name, for identifying the video data that will be used for vehicle recognition processing and lane recognition processing.

[0082] Frame 2062 stores an index value that indicates the chronological position within the video, similar to frame 2052 and frame 2057 of the lane recognition result 20502 (Figure 5B).

[0083] Tracking ID 2063, like Tracking ID 2053 of Vehicle Recognition Result 20501 (Figure 5A), stores identification information assigned to the object detected in the video (e.g., a vehicle) during the tracking process.

[0084] The lane change status 2064 stores the result of determining whether the vehicle described in tracking ID 2063 has changed lanes in the image of the frame described in frame 2062 of the video file described in video data 2061. If it is determined that the vehicle has not changed lanes, the value stored is "None". If it is determined that the vehicle has changed lanes, the value stored is the lane number that the vehicle is crossing. This lane number is one of the lane numbers described in lane number 2058 in the frame in which the lane change was determined.

[0085] Figures 7A, 7B, and 7C show examples of the data structure and data contents of the knowledge graph for each version of the first logic stored in the first logic specification storage unit 208 of this embodiment.

[0086] Figure 7A is a diagram showing an example of a knowledge graph that illustrates the input / output data of each process stored in the first logic specification storage unit 208 of this embodiment, and the input / output relationships between each process. The knowledge graph includes data nodes, processing nodes, and edges that show the relationships between these nodes.

[0087] In Figure 7A, the system consists of four processing nodes: measurement processing, lane recognition processing, object recognition processing, and lane change determination processing; data nodes indicating the input and output of each processing node; and edges connecting the nodes. Furthermore, the thick lines in Figure 7A represent the knowledge graph related to the lane change determination logic, which is explained as an example in this embodiment. The knowledge graph may also include processing nodes and data nodes related to logic other than the lane change determination logic. However, in this embodiment, since the lane change determination logic is explained as the first logic, the knowledge graph related to the lane change determination logic will be described.

[0088] This configuration allows for a visual understanding of the data flow and interrelationships between each processing node, enabling efficient management of the primary logic specifications.

[0089] Figure 7B is a diagram showing the data structure and data example of the information of the processing node of the knowledge graph stored in the first logic specification storage unit 208 of this embodiment, and shows the information of the processing node of the knowledge graph related to the lane change determination logic shown in Figure 7A.

[0090] Each record in the processing node information includes data for version 2081, processing name 2082, input data items 2083, output data items 2084, processing summary 2085, and input / output conditions 2086.

[0091] Version 2081 stores version information of the first logic that includes the relevant processing node. This version information allows for the unique identification of which version of the logic the processing node is being used with. From the basic version of the first logic onward, information on each processing node and data node in the knowledge graph is stored in the first logic specification storage unit 208.

[0092] The processing name 2082 stores the name of the processing node. For example, it is preferable to record a specific name of the processing, such as "lane recognition processing" or "lane change determination processing".

[0093] Input data item 2083 stores the input data required when the corresponding processing node is executed. For example, the input data required when executing the lane change detection process includes the lane recognition result and the object recognition result. If there are multiple input data items, each input data item is listed.

[0094] Output data item 2084 stores the data items of the data generated as a result of the execution of the corresponding processing node. For example, the output data item for the lane change detection process records that it is the lane change detection result. If there are multiple output data items, each output data item is listed.

[0095] The processing summary 2085 stores a concise description of the processing content to be performed by the relevant processing node, including the purpose of the processing, the execution procedure, and / or characteristic behavior of the processing. The processing summary 2085 is used by the active prompt configuration selection unit 207 and the active prompt creation unit 2091 when creating prompts. Specifically, when the summary statement is automatically generated on a template basis, the information in the processing summary 2085 is preferably used.

[0096] Input / output condition 2086 stores the temporal range of input / output data required when executing the corresponding processing node. The input / output condition 2086 should be written in a way that indicates "which moment or time period of input data is needed to obtain output data for a given moment or time period." For example, the input condition for object recognition processing might be "data from the past minute," and the output condition might be "object recognition result for a given moment based on data from the past minute." In this case, specifically, to obtain output data between 08:30 and 08:31, input data from 08:29 to 08:31, starting one minute before the time period being determined, is required.

[0097] This type of description clarifies the temporal dependencies of the data required for the execution of processing nodes, ensuring consistency in data management and processing flow across the entire driving scene extraction system.

[0098] Note that the time range may vary depending on the specifications of the first logic, and this example is merely one example. When the specifications of the first logic are changed, the input / output conditions should be changed accordingly.

[0099] Figure 7C is a diagram showing the data structure and data example of the data node information of the knowledge graph stored in the first logic specification storage unit 208 of this embodiment, and shows the data node information of the knowledge graph related to the lane change determination logic shown in Figure 7A.

[0100] Each record in the data node information contains data for version 2087, data item 2088, and data storage location 2089.

[0101] Version 2087 stores version information of the logic to which the corresponding data node belongs, similar to version 2081 of the first logic specification storage unit 208 (Figure 7B). This version information allows identification of which logic version the data node is associated with.

[0102] Data item 2088 stores the specific data represented by the corresponding data node, such as the name of the data node. Specifically, data items such as video data, lane recognition results, object recognition results, and lane change judgment results are recorded. Data item 2088 clearly indicates the information handled by the data node. The method of describing data item 2088 must be consistent with the method of describing input data item 2083 and output data item 2084 of the first logic specification storage unit 208 shown in Figure 7B. Such a consistent description method ensures consistency between data nodes and guarantees consistency in data management and processing flow.

[0103] The data storage location 2089 stores identification information for the physical or logical storage location where the relevant data node is stored. For example, it may record the record location in the database, the URL of cloud storage, or the file path on the file system. The data storage location 2089 allows for efficient referencing and retrieval of the necessary data.

[0104] Figure 8 shows an example of the data structure and data content of the script information for each version of the first logic stored in the first logic specification storage unit 208 of this embodiment.

[0105] Each record of information regarding the first logic for each version includes data for version 20810, script storage location 20811, logic summary 20812, and script 20813.

[0106] Version 20810 stores information to identify the version of the primary logic. Version 20810 identifies the version of the primary logic, allowing for proper management of the primary logic versions even if multiple versions exist.

[0107] The script storage location 20811 stores identification information for the physical or logical storage location where the script for the corresponding first logic is stored. For example, it may record the record location in a database, the URL of cloud storage, or the file path on the file system. The script storage location 20811 allows for efficient referencing and retrieval of the script.

[0108] Logic Summary 20812 stores a concise description of the function and processing content of the corresponding primary logic, as well as explanations of the differences between versions of the primary logic. For example, it is good to record information that helps to understand the overall picture of the logic, such as its purpose, basic operation, and design intent.

[0109] Script 20813 stores the program code for executing the corresponding first logic. For example, it should contain the specific code needed to implement the logic. The script recorded in script 20813 mainly implements the logic's algorithm and processing procedures, and directly controls the logic's operation.

[0110] Figure 9A shows an example of a list of candidate configurations for an effective prompt configuration, which is created based on a knowledge graph when the effective prompt configuration selection unit 207 of this embodiment selects an effective prompt configuration.

[0111] Here, the lane change determination logic will be explained as the first logic. First, the active prompt configuration selection unit 207 acquires a knowledge graph related to the lane change process and identifies data nodes related to the lane change determination logic by tracing back from the data node of the lane change determination result to the upstream node. In this process, "lane change determination result," "lane recognition result," "vehicle recognition result," and "video data" are identified as data nodes related to the lane change determination logic.

[0112] The candidate valid prompt configurations created by the valid prompt configuration selection unit 207 are in the format of "instruction statement" + "summary" + "input data". However, the input data considers all possible combinations of presence or absence of data nodes related to the identified lane change determination logic and lists them in a list. As a result, a total of 16 candidate configurations are generated based on the combinations of presence or absence of input data.

[0113] This allows for the optimization of interactions with the generating AI in order to identify the areas where the first logic struggles.

[0114] The effective prompt configuration candidate 2071 shown in Figure 9A exemplifies 16 configuration candidates when the first logic is lane change determination logic. The effective prompt configuration selection unit 207 can select the optimal prompt configuration candidate from the created list of configuration candidates in order to extract the problematic scenes of the first logic.

[0115] Figure 9B shows an example of the data structure and data contents of the valid prompt configuration stored in the valid prompt configuration storage unit 215 for each version in this embodiment.

[0116] Each record in the active prompt configuration storage unit 215 includes data for logic version 2151 and active prompt configuration 2152.

[0117] Logic version 2151 stores the version of the first logic to which the corresponding active prompt configuration is associated. Logic version 2151 allows identification of which logic version the active prompt configuration is used in. This provides a foundation for appropriately tracking prompt configurations for each version when managing multiple versions. Active prompt configuration 2152 stores details of the prompt configuration selected by the active prompt configuration selection unit 207 as optimal for the corresponding logic version. This item should ideally contain the specific configuration details of the prompt, for example, in the format of "instruction text + summary + input data (lane change judgment result + lane recognition result + vehicle recognition result + video data)". This ensures clear management of configuration information for optimizing interaction with the generating AI.

[0118] Figure 10 shows an example of a prompt statement generated by the effective prompt creation unit 2091 of this embodiment, illustrating the case where the effective prompt configuration selected by the effective prompt configuration selection unit 207 for extracting scenes where the lane change judgment logic is weak is "instruction statement + summary + input data (video data + lane recognition result + vehicle recognition result + lane change judgment result)".

[0119] The instruction text portion is copied directly from the instruction text storage unit 210.

[0120] The summary section describes the processing and input / output data necessary to execute the lane change detection logic. This description should be automatically generated by the active prompt creation unit 2091 based on a template such as ("Processing" input is "Processing Input Node". Output is "Processing Output Node". See "Processing Summary" for details). "Processing" is the processing name 2082 (Figure 7B) of the processing node information in the knowledge graph of the lane change detection logic, "Processing Input Node" is the input data item 2083 (Figure 7B) of the corresponding processing, "Processing Output Node" is the output data item 2084 (Figure 7B) of the corresponding processing, and "Processing Summary" is the processing summary 2085 (Figure 7B). This automatically generates a text that represents the processing flow of the first logic. Note that this template is applied to all processing nodes related to the lane change detection logic, and the summary text is generated.

[0121] In the input data section, first, the time range of the final judgment result, the lane change judgment result, is specified (for example, 08:30 to 08:31). Next, based on the knowledge graph of the lane change judgment logic, the system traces back to each processing node in the upstream nodes, checks the input / output conditions 2086 (Figure 7B) of the processing node's information, and identifies the time range of the output data and input data for each processing node. For example, to obtain a lane change judgment result for one minute, the input / output conditions 2086 (Figure 7B) state that corresponding one minute of time-synchronized lane recognition results and vehicle recognition results are required. Based on this information, it can be determined that to obtain a lane change judgment result for 08:30 to 08:31, vehicle recognition results and lane recognition results for 08:30 to 08:31 are required.

[0122] Furthermore, it is determined that in order to obtain vehicle recognition results for 08:30 to 08:31, video data from 08:29 to 08:31 and video data from 08:30 to 08:31 are required. In this case, the temporal range of the required video data is determined to be the broader range of 08:29 to 08:31.

[0123] Ultimately, in order to obtain the lane change detection result for 08:30 to 08:31, it is determined that the vehicle recognition result for 08:30 to 08:31, the lane recognition result for 08:30 to 08:31, and video data for 08:29 to 08:31 are required.

[0124] Next, the data storage location for each data item is checked from the data node information data storage location 2089 (Figure 7C), and the necessary data is retrieved based on the specified time range. The retrieved data is added as input data for the prompt.

[0125] Furthermore, if there are limitations on the maximum amount of data that can be input to the generating AI (for example, a limit of 60 images), when adding one minute of video data to the prompt, it is best to provide consecutive images at 1 fps or 0.5 fps. By considering such constraints, it is possible to efficiently input the necessary data into the generating AI and achieve appropriate processing.

[0126] Figure 11 shows an example of the data structure and data content of the list of difficult scenes 2110 stored in the difficult scene memory unit 211 of this embodiment.

[0127] The difficult scene memory unit 211 stores the difficult scenes extracted by the second logic execution unit 209 after determining the correctness of the first logic execution result as a difficult scene list 2110. Each record in the difficult scene list 2110 includes data for the first logic version 2111, scene ID 2112, storage location 2113, and extraction reason 2114.

[0128] The first logic version, 2111, stores identification information to identify the first logic version, i.e., the version number of the logic related to the extraction of problematic scenes.

[0129] Scene ID 2112 stores identification information to identify individual scenes that have been extracted as problematic scenes. Defining Scene ID 2112 to represent specific scenes facilitates the management and referencing of scenes within the driving scene extraction system.

[0130] The storage location 2113 stores identification information for the physical or logical storage location where the video data for the relevant scene is stored. For example, it may record the record location in the database, the URL of cloud storage, or the file path on the file system. The storage location 2113 for the video data allows for the identification of the storage location of the video file or the location of the media storage, enabling efficient access to and retrieval of the video data.

[0131] The extraction reason 2114 stores an explanatory text describing why the second logic execution unit 209 extracted the problematic scenes of the first logic. Specifically, it should describe the basis for extracting them as problematic scenes.

[0132] Figure 12 shows an example of the data structure and data content of the summary result list 2130 of difficult scenes stored in the summary result storage unit 213 of this embodiment.

[0133] Each record in the summary results list 2130 contains data for features 2131 and count 2132.

[0134] Feature 2131 records information that serves as the basis for classifying difficult scenes. For example, keywords and related attributes extracted from the responses generated by the AI ​​may be stored as features. This makes it possible to explicitly show the characteristics of each difficult scene in the aggregated results.

[0135] The record of 2132 entries records the frequency of occurrence of difficult scenes corresponding to specific characteristics. For example, the total number of difficult scenes for each characteristic classified by the aggregation process can be stored as a numerical value. The record of 2132 entries allows us to identify characteristics that frequently occur as difficult scenes.

[0136] Figure 13 shows an example of the data structure and data content of the improvement effect confirmation test data 20410 stored in the improvement effect confirmation test data storage unit 2041 of this embodiment.

[0137] The test data 20410 for verifying improvement effects is test data with correct tags attached to verify the improvement effect of the first logic. Each record in the test data 20410 for verifying improvement effects includes the data of number 20411, data file name 20412, and correct tag 20413.

[0138] The number 20411 stores identification information for identifying each record in the test data 20410 used to verify the improvement effect.

[0139] The data file name 20412 stores identification information (e.g., the file name) to identify the data file used as test data for verifying the improvement effect. If the first logic determines a lane change for each image, it will be stored in a format such as test001.jpg or test02.jpg.

[0140] The correct answer tag 20413 stores the correct label assigned to each data point in the test data used to confirm the improvement effect. The correct answer tag 20413 is the reference data necessary to evaluate the accuracy of the first logic. For example, if the first logic is a logic that determines lane changes, the correct answer tag will be represented as either "lane change" or "no lane change".

[0141] The test data 20410 for confirming improvement effects is created by user 4, uploaded to server 2 via user terminal 3, and stored in the test data storage unit 2041 for confirming improvement effects. For example, the test data 20410 for confirming improvement effects may be created by visually checking the difficult scenes stored in the difficult scene storage unit 211 when new difficult scenes are stored, assigning correct answer tags, and compiling them into a file (for example, in xlsx format). Alternatively, difficult scenes may be randomly extracted at a certain ratio and visually checked. In addition, the file (for example, an image file) described in the data file name 20412 is also uploaded to server 2 by the user and stored in the test data storage unit 2041 for confirming improvement effects.

[0142] Figure 14 shows an example of the data structure and data contents of the first logic accuracy 2160 stored in the first logic accuracy storage unit 216 of this embodiment.

[0143] The first logic accuracy 2160 indicates the accuracy of each version of the first logic. Each record of the first logic accuracy 2160 includes data for logic version 2161 and accuracy rate 2162.

[0144] Logic version 2161 stores the version information of the first logic. Logic version 2161 may be, for example, "1.1", "1.2", "1.3", etc.

[0145] The accuracy rate 2162 stores the accuracy rate of the first logic, which is calculated by aggregating the number of correct answers. The accuracy rate 2162 is usually expressed in the form of a percentage (%), and may be recorded as, for example, "70%" or "85%". The accuracy rate 2162 is calculated when the first logic improvement effect confirmation unit 204 executes each version of the first logic, determines whether the execution result of the first logic matches the correct answer data stored in the improvement effect confirmation test data storage unit 2041, and aggregates the number of correct answers. The accuracy rate 2162 is used as an indicator to evaluate how accurately the first logic operated with respect to the input data.

[0146] Figures 15A to 15F show examples of screens displayed on the user terminal 3 in this embodiment, with Figure 15A showing an example of a normal screen 310 displayed on the user terminal 3.

[0147] On the normal screen, users can select the process they want to perform from the following five items: 1. Setting / updating instruction statements 2. Setting / updating the first logic specifications 3. Checking difficult scenes and the results of the difficult scene aggregation 4. Checking the improvement effect (correctness rate of the first logic) 5. Adding test data for checking the improvement effect

[0148] User 4 can select their desired item from the pull-down menu 312 by operating the "▽" icon 311 on the screen. After selecting an item, User 4 operates the "Confirm" button 313 located at the bottom of the screen, and the screen on User Terminal 3 transitions to the screen corresponding to the item selected by User 4 (Figures 15B to 15F).

[0149] Figure 15B shows an example of the instruction text setting / update screen 320 that is displayed on the user terminal 3 when user 4 selects "1. Instruction text setting / update".

[0150] Above the instruction setting / update screen 320, instructions 321 regarding setting and updating instruction texts are displayed for user 4.

[0151] In the center of the instruction setting / update screen 320, there is an input field 322 for entering instruction text. For example, when the first logic determines a lane change, the text shown in the input field 322 is entered as an example of an instruction text.

[0152] Furthermore, by specifying the output format of the answer in the input field 322, the answer generated by the generating AI 5 can include the "reason for judgment". If the "reason for judgment" included in the answer generated by the generating AI 5 is written to the extraction reason 2114 (Figure 11) of the list of difficult scenes 2110, the list of difficult scenes 2110 can be automatically completed. Figure 15B shows an example of the instruction text setting / update screen 320, but the instruction text can be freely updated based on the user's judgment.

[0153] After user 4 enters the instruction text, when user 4 operates the "Confirm" button 313 located at the bottom of the screen, the screen of user terminal 3 returns to the normal screen 310 (Figure 15A).

[0154] Figure 15C shows an example of the logic specification setting / update screen 330 displayed on the user terminal 3 when user 4 selects "2. Setting / updating the first logic specification".

[0155] Above the logic specification setting / update screen 330, instructions 331 for user 4 regarding the setting and updating of the first logic specification are displayed. In the central part of the logic specification setting / update screen 330, a knowledge graph area 332 is provided to display the current knowledge graph. Below the knowledge graph area 332, when updating the knowledge graph, an operation area 333 for updating or adding processing nodes and an operation area 334 for updating or adding data nodes are provided. Below the logic specification setting / update screen 330, an input area 335 is provided for entering the script of the first logic to be updated.

[0156] After user 4 inputs the first logic specification, when user 4 operates the "Confirm" button 313 located at the bottom of the screen, the screen of user terminal 3 returns to the normal screen 310 (Figure 15A).

[0157] Figure 15D shows an example of the difficulty scene confirmation screen 340 that is displayed on the user terminal 3 when user 4 selects "3. Confirmation of difficult scenes and summary results of difficult scenes".

[0158] At the top of the difficulty scene confirmation screen 340, a message 341 is displayed to user 4 instructing them to check the difficulty scenes and their aggregated results. Additionally, a message is displayed stating, "If you would like to see details of the difficulty scenes, please click on the scene ID."

[0159] In the center of the disliked scene confirmation screen 340, the aggregated results stored in the aggregated results storage unit 213 and the disliked scene list 342 stored in the disliked scene storage unit 211 are displayed. Furthermore, when user 4 selects the ID of a specific disliked scene to check its details, the disliked scene details, including the video of the selected disliked scene, are displayed. Note that the reason for extraction 2114 for each disliked scene may not be displayed in the disliked scene list 2110, but only in the disliked scene details section.

[0160] After user 4 has identified the scenes they find difficult, they can operate the "Confirm" button 313 located at the bottom of the screen, and the screen on user terminal 3 will return to the normal screen 310 (Figure 15A).

[0161] Figure 15E shows an example of the improvement effect confirmation screen 350 that is displayed on the user terminal 3 when user 4 selects "4. Confirmation of improvement effect (accuracy rate of the first logic)".

[0162] In the center of the improvement effect confirmation screen 350, the accuracy rate 351 of each version of the first logic stored in the first logic accuracy storage unit 216 is displayed.

[0163] After confirming the improvement effect, user 4 operates the "Confirm" button 313 located at the bottom of the screen, and the screen of user terminal 3 returns to the normal screen 310 (Figure 15A).

[0164] Figure 15F shows an example of the test data addition screen 360 displayed on the user terminal 3 when user 4 selects "5. Add test data to confirm improvement effect".

[0165] At the top of the test data addition screen 360, an instruction 361 prompts the user to attach a file containing additional test data. Instructions regarding the file format and the necessary data items within the file are also displayed (for example, a table is displayed at the bottom of the instruction). Furthermore, an input area 363 is provided in the center of the test data addition screen 360 for entering the file containing the additional test data.

[0166] After user 4 adds test data, when user 4 operates the "Confirm" button 313 located at the bottom of the screen, the screen of user terminal 3 returns to the normal screen 310 (Figure 15A).

[0167] Figure 16 is a flowchart of the instruction setting / update process executed by the user communication unit 214 when "1. Setting / updating instruction" is selected on the user terminal 3 of this embodiment and an instruction is received.

[0168] The process shown in Figure 16 is executed when the user terminal communication unit 2147 of the user communication unit 214 receives an instruction message entered into the instruction message setting / update screen 320 from the user terminal 3.

[0169] First, the communication unit 2147 for the user terminal sends the command "set / update instruction text" and the received instruction text to the communication unit 2141 for the active prompt configuration selection unit. The communication unit 2141 for the active prompt configuration selection unit stores the received instruction text in the instruction text storage unit 210 (step S11).

[0170] Next, the communication unit 2141 for the active prompt configuration selection unit sends a request to the active prompt configuration selection unit 207 to start the active prompt configuration selection (step S12).

[0171] Next, the communication unit 2141 for the active prompt configuration selection unit determines whether it has received a notification from the active prompt configuration selection unit 207 that the configuration selection is complete (step S13). If it has not received a notification from the active prompt configuration selection unit 207 that the configuration selection is complete (NO in step S13), the communication unit 2141 for the active prompt configuration selection unit waits for a predetermined time (for example, 10 seconds) (step S15), then returns to step S13 and re-determines whether it has received a notification from the active prompt configuration selection unit 207 that the configuration selection is complete (step S13). If it has received a notification from the active prompt configuration selection unit 207 that the selection is complete (YES in step S13), the communication unit 2141 for the active prompt configuration selection unit notifies the communication unit 2142 for the first logic execution unit that the active prompt configuration selection is complete.

[0172] Then, the communication unit 2142 for the first logic execution unit sends a request to the first logic execution unit 203 to start execution of all the data of the latest version of the first logic (step S14).

[0173] Figure 17 is a flowchart of the logic specification setting and update process executed by the user communication unit 214 when "2. Setting / Updating First Logic Specification" is selected on the user terminal 3 of this embodiment and the first logic specification and the script for the first logic are received.

[0174] The process shown in Figure 17 is executed when the user terminal communication unit 2147 of the user communication unit 214 receives the first logic specification and the script for the first logic input from the user terminal 3 to the logic specification setting / update screen 330.

[0175] First, the communication unit 2147 for the user terminal sends the command "Setting / updating the first logic specification" and the update content, which is the first logic specification and the script for the first logic, to the communication unit 2148 for the first logic specification storage unit. The communication unit 2148 for the first logic specification storage unit stores the update content (first logic specification and script for the first logic) in the first logic specification storage unit 208 (step S21).

[0176] Next, the communication unit 2147 for the user terminal sends a request to the active prompt configuration selection unit 207 to start the active prompt configuration selection (step S22).

[0177] Next, the communication unit 2147 for the user terminal determines whether it has received a notification from the active prompt configuration selection unit 207 that the configuration selection is complete (step S23). If it has not received a notification from the active prompt configuration selection unit 207 that the configuration selection is complete (NO in step S23), the communication unit 2147 for the user terminal waits for a predetermined time (for example, 10 seconds) (step S26), then returns to step S23 and re-determines whether it has received a notification from the active prompt configuration selection unit 207 that the configuration selection is complete (step S13). If it has received a notification from the active prompt configuration selection unit 207 that the selection is complete (YES in step S23), the communication unit 2147 for the user terminal notifies the communication unit 2142 for the first logic execution unit that the active prompt configuration selection is complete.

[0178] Then, the communication unit 2142 for the first logic execution unit sends a request to the first logic execution unit 203 to start execution of all the data of the latest version of the first logic (step S14).

[0179] Next, the communication unit 2145 for the first logic improvement effect verification unit sends a request to the first logic improvement effect verification unit 204 to start the verification of the improvement effect of the latest version of the first logic (step S25).

[0180] Note that the order of steps S24 and S25 in the process shown in Figure 17 may be reversed, or they may be executed in parallel.

[0181] Figure 18 is a flowchart of the test data addition process executed by the user communication unit 214 when "5. Addition of test data to confirm improvement effect" is selected on the user terminal 3 of this embodiment and additional test data is received.

[0182] The process shown in Figure 18 is executed when the user terminal communication unit 2147 of the user communication unit 214 receives test data for confirming improvement effects entered into the test data addition screen 360 from the user terminal 3.

[0183] First, the user communication unit 2147 transmits test data for confirming the improvement effect to the first logic improvement effect confirmation unit communication unit 2145. The first logic improvement effect confirmation unit communication unit 2145 stores the received test data for confirming the improvement effect in the improvement effect confirmation test data storage unit 2041 (step S51).

[0184] Next, the communication unit 2145 for the first logic improvement effect verification unit sends a request to the first logic improvement effect verification unit 204 to start the verification of the improvement effect of the first logic for all versions (step S52).

[0185] Figure 19 is a flowchart of the process for extracting the problematic scenarios of the first logic in this embodiment.

[0186] The process shown in Figure 19 is executed under one of the following conditions: 1. When a request for difficult scene extraction is received from the user communication unit 214 because the data stored in the first logic specification storage unit 208 or the instruction statement storage unit 210 has been updated. 2. When a certain amount of data (for example, 24 hours' worth) is newly collected and stored in the vehicle data storage unit 202.

[0187] The first logic execution unit 203 obtains the latest version of the first logic script and the first logic specifications (e.g., knowledge graph) from the first logic specification storage unit 208, obtains the necessary vehicle data from the vehicle data storage unit 202, and executes the first logic using the obtained data. The recognition result and judgment result of the first logic are stored in the corresponding storage unit. The first logic execution unit 203 sends a request to the second logic execution unit 209 to start the execution of the second logic (step S101). For example, if the first logic is lane change judgment logic, the first logic execution unit 203 stores the lane recognition result and vehicle recognition result in the recognition result storage unit 205 and the lane change judgment result in the judgment result storage unit 206.

[0188] Next, the valid prompt creation unit 2091 of the second logic execution unit 209 obtains the valid prompt configuration from the valid prompt configuration storage unit 215, and based on the obtained valid prompt configuration, obtains the necessary data from the corresponding storage units (for example, the recognition result storage unit 205 and the judgment result storage unit 206) to create a valid prompt (step S102). Details of step S102 will be explained with reference to Figure 20.

[0189] Next, the correct / incorrect determination unit 2092 of the second logic execution unit 209 connects to the generation AI 5 via a predetermined API and inputs the created prompt to the generation AI 5. Then, the correct / incorrect determination unit 2092 determines whether the first logic execution result is correct or incorrect based on the response from the generation AI, and stores the correct / incorrect determination result in the difficulty scene storage unit 211 (step S103). The details of step S103 will be explained with reference to Figure 20.

[0190] Next, the aggregation processing unit 212 aggregates the difficult scenes and stores the aggregation results in the aggregation result storage unit 213 (step S104). The aggregation processing may classify the difficult scenes based on a rule-based method (for example, classification using keywords). Here, the keywords may be extracted, for example, from sentences indicating the reason for the judgment included in the response from the generating AI.

[0191] Figure 20 is a flowchart detailing the process executed by the second logic execution unit 209 in steps S102 to S103 of Figure 19.

[0192] First, the active prompt creation unit 2091 obtains an instruction from the instruction storage unit 210 (step S1021), and then obtains information from the first logic specification storage unit 208 regarding the knowledge graph of the first logic (step S1022).

[0193] Subsequently, the active prompt creation unit 2091 extracts the final output result of the first logic (for example, the lane change judgment result by the lane change judgment logic) from the knowledge graph. Starting from the final output result of the first logic, it traverses the upstream nodes and obtains information on processes related to the first logic (for example, recognition processes, judgment processes), and the input and output data of those processes. It also obtains the active prompt configuration from the active prompt configuration storage unit 215 and creates an active prompt from the obtained input data, output data, and active prompt configuration (step S1023). Details of the active prompt creation in step S1023 will be explained with reference to Figure 21.

[0194] Next, the valid prompt creation unit 2091 sends the created valid prompt to the correct / incorrect determination unit 2092 (step S1024).

[0195] The correct / incorrect judgment unit 2092 connects to the external generation AI service 5 via the generation AI's API and sends the prompt created by the valid prompt creation unit 2091 to the generation AI 5 (step S1031). Next, the correct / incorrect judgment unit 2092 obtains the answer from the external generation AI 5 (step S1032).

[0196] The correct / incorrect determination unit 2092 determines whether the response obtained from the generating AI 5 is in the specified output format (step S1033). If the response from the generating AI 5 is different from the specified output format (NO in S1033), the correct / incorrect determination unit 2092 resends a valid prompt to the generating AI 5 (step S1036). Then, it obtains a response from the external generating AI 5 again (step S1032). This process is repeated until the response from the generating AI is in the specified output format.

[0197] On the other hand, if the output of the generating AI is in the specified output format (YES in S1033), the correct / incorrect determination unit 2092 checks whether the correct / incorrect determination result based on the output of the generating AI 5 is False (step S1034). If the correct / incorrect determination result is not False, the process proceeds to the next aggregation process (step S104).

[0198] On the other hand, if the correct / incorrect judgment result is False, the correct / incorrect judgment unit 2092 stores the corresponding scene as a difficult scene in the difficult scene storage unit 211 (step S1035). After that, the process transitions to the aggregation process (step S104).

[0199] Figure 21 is a flowchart detailing step S1023 of Figure 20.

[0200] First, the active prompt creation unit 2091 extracts the final output result of the first logic from the knowledge graph of the first logic specification storage unit 208. Then, the active prompt creation unit 2091 traces upstream nodes starting from the output result of the first logic and obtains the processes related to the first logic (e.g., recognition process, judgment process), and the data items that are the input and output of those processes (video data, recognition result, judgment result). In addition, the active prompt creation unit 2091 obtains the necessary elements of the prompt input data and the active prompt configuration from the active prompt configuration storage unit 215 (step S10231).

[0201] Next, based on the connections of a series of processing nodes necessary to obtain the final result of the first logic extracted from the knowledge graph in S10231, an overview of the first logic is generated based on a predetermined template for the processing flow (step S10232).

[0202] The summary should ideally be generated based on the following template. Template example: "The input for 'Processing' is the 'Processing Input Node'. The output is the 'Processing Output Node'. Details are in the 'Processing Summary'." Here, the specific contents of each item are as follows: "Processing": Input the processing name 2082 from the processing node information of the knowledge graph related to the lane change judgment logic. "Processing Input Node": Enter the input data using the input data item 2083 of the corresponding processing node. "Processing Output Node": Enter the output data using the output data item 2084 of the corresponding processing node. "Processing Summary": Enter detailed information using the processing summary 2085 of the corresponding processing node.

[0203] By embedding these elements into the template, it is possible to generate text that explains the processing flow of the lane change detection logic. This template is applied to all processing nodes related to the lane change detection logic, and a summary text is generated.

[0204] Next, the active prompt creation unit 2091 obtains all the data necessary to obtain the final result of the first logic from the dataset, based on the connections of a series of processing nodes necessary to obtain the final result of the first logic extracted from the knowledge graph in S10231 and the input / output data items of those processing nodes (step S10233).

[0205] The active prompt creation unit 2091 should, starting from the node that shows the final result of the first logic in the knowledge graph, traverse upstream nodes and extract the range of data to be acquired based on the "input / output data conditions" of each node so as to satisfy the range (temporal range) of input information required by the downstream node (for example, if a downstream node uses information from the last minute, and an upstream node uses information from the past minute for processing at a certain moment, the unit should extract two minutes of information necessary to prepare the one minute of information for the downstream node as input data for the upstream node).

[0206] One specific method for adding input data to a prompt is to specify a time range for the final judgment result, which is the lane change judgment result (for example, 08:30 to 08:31). Next, the active prompt creation unit 2091 checks the input / output conditions 2086 by going back to each processing node in the upstream node based on the knowledge graph of the lane change judgment logic, and identifies the time range for the output data and input data of each processing node.

[0207] For example, input / output condition 2086 states that in order to obtain the lane change judgment result for one minute, corresponding one-minute data of time-synchronized lane recognition results and vehicle recognition results are required. Based on this information, in order to obtain the lane change judgment result for 08:30 to 08:31, vehicle recognition results and lane recognition results for 08:30 to 08:31 are required.

[0208] Furthermore, it is determined that in order to obtain vehicle recognition results from 08:30 to 08:31, video data from 08:29 to 08:31 is required, and in order to obtain lane recognition results from 08:30 to 08:31, video data from 08:30 to 08:31 is required. In this case, the required time range for video data becomes the wider 08:29 to 08:31.

[0209] Ultimately, in order to obtain the lane change detection result for 08:30 to 08:31, the vehicle recognition result for 08:30 to 08:31, the lane recognition result for 08:30 to 08:31, and the video data for 08:29 to 08:31 are required.

[0210] Next, the active prompt creation unit 2091 checks the location where each data is stored from the data storage location 2089 recorded in the data node information, and retrieves the necessary data based on the identified time range. This retrieved data is added as input data for the prompt.

[0211] Furthermore, given the constraint that the maximum number of images that can be input to the generating AI 5 is 60, when adding one minute of video data to the prompt, it is best to pass consecutive images at 1 fps or 0.5 fps. By considering these constraints, the necessary data can be efficiently input to the generating AI 5, enabling appropriate processing.

[0212] Finally, the active prompt creation unit 2091 creates an active prompt using the "instruction text" obtained from the instruction text storage unit 210, the "summary" automatically generated in step S10232, and the input data obtained in step S10233 (step S10234). After that, the process proceeds to step S1024.

[0213] Figure 22 is a flowchart of the process by which the active prompt configuration selection unit 207 in this embodiment selects an active prompt configuration. The active prompt configuration selection process is executed when a request to start active prompt configuration selection is received from the user communication unit 214.

[0214] First, the active prompt configuration selection unit 207 obtains a knowledge graph related to the first logic from the first logic specification storage unit and identifies data items 2088 of all data nodes related to the lane change determination process in the obtained knowledge graph (step S201).

[0215] Next, the active prompt configuration selection unit 207 creates a candidate list of active prompt configurations from the instruction text + summary + input data (combination of data items) (step S202). For example, if the data items of the data node related to the lane change determination process are lane change determination result, lane recognition result, vehicle recognition result and video data, the candidate list shown in Figure 9A is created.

[0216] Next, the active prompt configuration selection unit 207 creates a prompt using pre-prepared data with correct answer tags based on the configuration candidate list, sends it to the generating AI multiple times, and receives a response from the generating AI (step S203). The prompt text is created by the process shown in step S1023 (Figure 21). However, the input data is not obtained from each recognition unit, but rather from pre-prepared data with correct answers within the active prompt configuration selection unit 207.

[0217] Note that n starts from 1, and its maximum value is the number of candidates in the configuration candidate list. In the configuration candidate list shown in Figure 9A, n is 16. The number of times prompt messages are sent to the generation AI can be freely set according to the server's computing power, etc.

[0218] Next, the active prompt configuration selection unit 207 determines whether each of the answers received from the generated AI 5 is correct (step S204). If some of the answers are incorrect (NO in S204), the active prompt configuration selection unit 207 removes the prompt configuration from the candidate configuration list (step S205). On the other hand, if all of the answers are correct (YES in S204), the process proceeds to step S206.

[0219] Next, the valid prompt configuration selection unit 207 adds 1 to the value of n (step S206), and returns to step S203 to execute the process. The process from step S203 to step S206 is repeated for each candidate configuration (in the candidate configuration list shown in Figure 9A, n = 1 to 16). That is, the process from step S203 to step S206 is executed until it is determined whether the prompts generated for all candidates in the candidate configuration list are valid. After determining all candidate configurations in the candidate configuration list, the process proceeds to step S207.

[0220] The active prompt configuration selection unit 207 determines whether there are any prompt configuration candidates remaining in the configuration candidate list (step S207). If there are no prompt configuration candidates remaining in the configuration candidate list (NO in S207), the active prompt configuration selection unit 207 selects a configuration that includes all data items as the active prompt configuration (step S208). For example, in the case of lane change judgment logic, candidate number 1 shown in Figure 9A, "instruction text + summary + input data (video data + vehicle recognition result + lane recognition result + lane change judgment result)", is selected as the active prompt configuration. The active prompt configuration selection unit 207 then stores the selected active prompt configuration in the active prompt configuration storage unit 215 (step S212).

[0221] On the other hand, if a prompt configuration candidate remains in the configuration candidate list (YES), the valid prompt configuration selection unit 207 determines whether there are multiple remaining prompt configuration candidates (step S209). If there is only one remaining prompt configuration candidate (NO in S209), the valid prompt configuration selection unit 207 selects the remaining single prompt configuration as the valid prompt configuration (step S210) and stores the selected valid prompt configuration in the valid prompt configuration storage unit 215 (step S212).

[0222] On the other hand, if multiple candidates remain in the candidate list (YES in S209), the valid prompt configuration selection unit 207 selects the prompt configuration candidate with the most data items as the valid prompt configuration (step S211), and stores the selected valid prompt configuration in the valid prompt configuration storage unit 215 (step S212).

[0223] Through this process, the active prompt configuration selection unit 207 can efficiently select the optimal prompt configuration for the first logic in cooperation with the generating AI 5.

[0224] Figure 23 is a flowchart of the improvement effect confirmation process in this embodiment, in which the first logic improvement effect confirmation unit 204 confirms the improvement effect of the first logic.

[0225] The improvement effect verification process is executed when the data stored in the improvement effect verification test data storage unit 2041 is updated, when the data stored in the first logic specification storage unit 208 is updated, or when a request to start the first logic improvement effect verification is received from the user communication unit 214.

[0226] Furthermore, if the data stored in the test data storage unit 2041 for verifying improvement effects is updated, the first logic improvement effect verification unit 204 calculates the accuracy rate of all first logics. When the data stored in the first logic specification storage unit 208 is updated, the first logic improvement effect verification unit 204 calculates the accuracy rate of the latest version of the first logic.

[0227] First, the first logic improvement effect verification unit 204 obtains the required version of the first logic from the first logic specification storage unit 208 and executes the first logic on the test data with correct answer tags obtained from the improvement effect verification test data storage unit 2041 (step 301).

[0228] Next, the first logic improvement effect confirmation unit 204 compares the execution result of the first logic with the correct test data for checking the improvement effect. If the execution result of the first logic matches the correct data, it determines that it is correct and calculates the accuracy rate. Then, the first logic improvement effect confirmation unit 204 stores the accuracy rate and the version of the first logic in the first logic accuracy storage unit 216 (step S302).

[0229] As described above, according to the embodiment of the present invention, it is possible to comprehensively and efficiently extract scenarios where logic is weak without the need for tagging. Therefore, the efficiency of verifying AD / ADAS functions can be improved.

[0230] More specifically, the valid prompt creation unit 2091 obtains the valid prompt configuration necessary to determine the correctness of the latest version of the first logic execution result from the valid prompt configuration storage unit 215, obtains an instruction from the instruction storage unit 210, and obtains a knowledge graph related to the first logic from the first logic specification storage unit 208. Then, based on the valid prompt configuration, the valid prompt creation unit 2091 obtains the necessary data from the corresponding storage units (e.g., vehicle data storage unit 202, recognition result storage unit 205, judgment result storage unit 206) and creates a valid prompt. The correctness determination unit 2092 connects to the external generation AI service 5 via the generation AI API, inputs the prompt created by the valid prompt creation unit 2091 to the generation AI 5, checks the response returned from the generation AI 5, and determines the correctness of the first logic execution result based on that response. Furthermore, if the correctness determination unit 2092 determines that the first logic execution result is incorrect, it extracts the corresponding scene as a difficult scene and stores the extracted difficult scene in the difficult scene storage unit 211. With this configuration, in addition to the judgment result of the first logic, vehicle data and information on the recognition result of the first logic are also added to the prompt message, and the generated AI 5 is used to determine whether the execution result of the first logic is correct or incorrect. As a result, both scenes in which recognition fails and scenes in which judgment fails can be extracted as problematic scenes, improving the comprehensiveness of problematic scene extraction.

[0231] Furthermore, the active prompt configuration selection unit 207 acquires knowledge graph information related to the first logic stored in the first logic specification storage unit 208 and acquires all data items necessary to execute the first logic. Subsequently, it generates a list of candidate prompt configurations corresponding to each of the total combinations of the presence or absence of the acquired data items. For example, if the first logic is lane change judgment logic, the data items necessary to execute the first logic include video data, lane recognition result, vehicle recognition result, and lane change judgment result. The active prompt configuration selection unit 207 also acquires data with correct answers tagged in advance (for example, video data + recognition result + judgment result + correct answer). Furthermore, it acquires instruction sentences from the instruction sentence storage unit 210 and creates prompt sentences corresponding to each prompt configuration candidate based on the prompt configuration. The created prompt sentences are input to the external generation AI 5 multiple times. If at least one of the answers from the generation AI 5 is not correct, the corresponding prompt configuration candidate is removed from the configuration candidate list. After performing this process for all candidates and completing the evaluation, if multiple configuration candidates remain in the candidate list, the configuration with the most input data elements (i.e., the most information) is selected as the valid prompt configuration. Finally, the selected valid prompt configuration is stored in the valid prompt configuration storage unit 215. This process allows for the efficient and accurate selection of the optimal prompt configuration for extracting scenes where the first logic is weak.

[0232] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all the configurations described above. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0233] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0234] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

[0235] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A driving scene extraction system comprising a computer having an arithmetic unit that performs predetermined processing and a storage device accessible by the arithmetic unit, comprising a vehicle data collection unit, a first logic execution unit, a second logic execution unit, a correct / incorrect determination unit, and a communication unit, wherein the vehicle data collection unit collects vehicle data of its own vehicle and vehicle surrounding environment measurement data measuring the surrounding environment of the vehicle, and stores the collected data in the storage device, the first logic execution unit acquires the data collected from the storage device and executes a vehicle control function, and records the recognition result and the determination result in the storage device, the second logic execution unit refers to relationship data that defines the relationships between processes included in the first logic in the vehicle control function, and creates a prompt that includes at least two processes and an instruction statement for determining the correctness of the determination result of the first logic by following the relationships defined in the relationship data, the correct / incorrect determination unit uses the created prompt to obtain a determination result of the correctness of the determination result of the first logic, and if the determination result of the first logic is incorrect, classifies the scene used for the determination as a problematic scene. The communication unit is characterized by outputting the determined problematic scenes, making it a driving scene extraction system.

2. A driving scene extraction system according to claim 1, characterized in that the relational data is represented in a knowledge graph that includes a processing node which is a process executed by the first logic, a data node which consists of data output from the process or data input to the process, and an edge which represents the relationship between the input and output of data to the process by the connection between the processing node and the data node.

3. A driving scene extraction system according to claim 2, wherein the second logic execution unit refers to the knowledge graph and identifies a first processing node that outputs a determination result by the first logic, and a data node for output data and a data node for input data connected to the first processing node; refers to the knowledge graph and identifies a second processing node connected to the input side of the first processing node that outputs a recognition result by the first logic, and a data node for output data and a data node for input data connected to the second processing node; refers to the knowledge graph and identifies a third processing node connected to the input side of the second processing node that outputs a measurement result by the first logic, and a data node for output data connected to the third processing node; and creates a prompt that includes an instruction statement for determining whether the determination result of the first logic is correct or incorrect using the processing of the identified first processing node, second processing node, and third processing node, and the data of the data nodes connected to these processing.

4. A driving scene extraction system according to claim 3, wherein the second logic execution unit refers to the knowledge graph to trace back through all processing nodes related to the first logic and identifies the input / output conditions of each processing node; refers to the identified input / output conditions to identify the time range of the output data and input data of each processing node; and creates a prompt that includes the data within the identified time range as input data.

5. A driving scene extraction system according to claim 3, wherein the effective prompt configuration selection unit acquires knowledge graph information relating to the first logic, acquires all data items necessary to execute the first logic, generates prompt configuration candidates that include each of all combinations of the presence or absence of the acquired data items, creates a prompt based on the generated prompt configuration candidates, inputs the created prompt multiple times into a machine learning model, deletes the prompt configuration candidate that derived the incorrect answer if at least one of the answers from the machine learning model is incorrect, and selects the prompt configuration candidate with the most data items among the remaining prompt configuration candidates as the effective prompt configuration.

6. A driving scene extraction system according to claim 5, wherein the effective prompt configuration selection unit selects a prompt configuration candidate that includes all data items as the effective prompt configuration if no prompt configuration candidates remain.

7. A driving scene extraction system according to claim 1, comprising a first logic improvement effect confirmation unit that obtains scripts for each version of the first logic, executes the first logic using test data to which correct answer tags have been assigned, and calculates the accuracy rate of the first logic.

8. A driving scene extraction method to be performed by a driving scene extraction system, wherein the driving scene extraction system is composed of a computer having a computing device that performs predetermined processing and a storage device accessible by the computing device, the driving scene extraction method is characterized in that the computing device collects vehicle data of its own vehicle and vehicle surrounding environment measurement data that measures the environment around the vehicle, and stores the collected data in the storage device, the computing device obtains the data collected from the storage device and performs a vehicle control function, and records the recognition result and the judgment result in the storage device, the computing device refers to relationship data that defines the relationships between processes included in the first logic in the vehicle control function, and creates a prompt that includes at least two processes and an instruction statement that determines whether the judgment result of the first logic is correct or incorrect by following the relationships defined in the relationship data, the computing device uses the created prompt to obtain a judgment result of whether the judgment result of the first logic is correct or incorrect, and if the judgment result of the first logic is incorrect, it classifies the scene used for the judgment as a problematic scene, and the computing device outputs the determined problematic scene.