Method for evaluating performance of autonomous driving algorithm and electronic device supporting same

Abstract

A method for evaluating the performance of an autonomous driving algorithm according to one embodiment of the present disclosure may comprise the steps of: setting a driving scenario for at least some driving sections in a road environment; acquiring execution data of the autonomous driving algorithm executed on the basis of the driving scenario in the at least some driving sections; setting at least one evaluation item for evaluating the performance of the autonomous driving algorithm on the basis of the execution data; and providing performance evaluation information of the autonomous driving algorithm, based on the at least one evaluation item.

Description

Method for evaluating the performance of an autonomous driving algorithm and an electronic device supporting the same

[0001] Various embodiments of the present disclosure relate to a method for evaluating the performance of an autonomous driving algorithm and an electronic device supporting the same.

[0002] The autonomous driving system of a mobile device executes autonomous driving algorithms to perceive and determine the driving environment of the device, and controls the device's driving based on this. In other words, systematic verification and evaluation of the performance of autonomous driving algorithms are required to realize stable autonomous driving of the device. However, since autonomous driving testing of mobile devices involves a vast amount of simulation, there are practical limitations to quantitatively evaluating the related execution data by collecting and analyzing it in real time.

[0003] The foregoing is provided as background technology for the purpose of aiding understanding of the present disclosure, and no claim or determination is made as to whether it can be applied as prior art related to the present disclosure.

[0004] The present disclosure provides a method for evaluating the performance of an autonomous driving algorithm and an electronic device supporting the same, for solving the above-mentioned problems.

[0005] The technical problems that this disclosure aims to solve are not limited to those described above, and other unmentioned problems will be clearly understood by those skilled in the art from the various embodiments described below.

[0006] The present disclosure may be implemented in various ways, including methods, electronic devices, and / or computer programs stored on a readable recording medium.

[0007] A method for evaluating the performance of an autonomous driving algorithm according to one embodiment of the present disclosure may include the steps of: setting a driving scenario for at least a portion of a driving section within a road environment; obtaining execution data of an autonomous driving algorithm executed based on the driving scenario in at least a portion of the driving section; setting at least one evaluation item for evaluating the performance of the autonomous driving algorithm based on the execution data; and providing performance evaluation information of the autonomous driving algorithm based on at least one evaluation item.

[0008] According to one embodiment of the present disclosure, the step of setting at least one evaluation item may include the step of displaying a first graphical user interface (GUI) including at least one recommended evaluation item related to a driving scenario through a user interface of a designated application.

[0009] According to one embodiment of the present disclosure, the step of displaying a first GUI may include identifying at least one recommended evaluation item associated with a driving scenario from a database.

[0010] According to one embodiment of the present disclosure, the step of displaying a first GUI may include identifying the history of evaluation items set in relation to a driving scenario and determining at least one recommended evaluation item based on the history.

[0011] According to one embodiment of the present disclosure, the step of displaying a first GUI may include changing at least a portion of at least one recommended evaluation item based on user input.

[0012] According to one embodiment of the present disclosure, the step of setting at least one evaluation item may include the step of displaying a second GUI comprising at least one of a calculation formula associated with at least one evaluation item, a user-defined value compared with the result value of the calculation formula, or a mathematical formula condition used for comparison between the result value and the user-defined value.

[0013] According to one embodiment of the present disclosure, the step of providing performance evaluation information may include the step of displaying a third GUI comprising at least one of a video of an autonomous driving simulation executed based on a driving scenario in at least a portion of the driving section or a progress bar associated with the playback of the video of the autonomous driving simulation.

[0014] According to one embodiment of the present disclosure, a method for evaluating the performance of an autonomous driving algorithm may further include the step of displaying a list of a plurality of evaluation items available for setting at least one evaluation item through a user interface of a designated application.

[0015] According to one embodiment of the present disclosure, a computer program stored on a computer-readable recording medium may be provided for executing a method for evaluating the performance of an autonomous driving algorithm on a computer.

[0016] An electronic device according to one embodiment of the present disclosure includes a memory for storing instructions and at least one processor, and when the instructions are executed by at least one processor, the electronic device may set a driving scenario for at least a portion of a driving section within a road environment, acquire execution data of an autonomous driving algorithm executed based on the driving scenario in at least a portion of the driving section, set at least one evaluation item for evaluating the performance of the autonomous driving algorithm based on the execution data, and provide performance evaluation information of the autonomous driving algorithm based on at least one evaluation item.

[0017] According to various embodiments of the present disclosure, a mechanism capable of evaluating the performance of an autonomous driving algorithm in real time and providing evaluation information may be provided.

[0018] According to various embodiments of the present disclosure, user convenience can be enhanced by recommending evaluation items referenced in the performance evaluation of an autonomous driving algorithm based on driving scenarios of a mobile device.

[0019] According to various embodiments of the present disclosure, the user experience can be enhanced by intuitively providing various information related to the performance evaluation of an autonomous driving algorithm through graphic elements.

[0020] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims.

[0021] Various embodiments of the present disclosure will be described with reference to the drawings below, and identical or similar reference numbers may be assigned to identical or corresponding components in connection with the description of the drawings.

[0022] FIG. 1 is a drawing illustrating an example of a first user interface according to one embodiment of the present disclosure.

[0023] FIG. 2 is a drawing illustrating an example of an electronic device in a network environment according to one embodiment of the present disclosure.

[0024] FIG. 3 is a drawing illustrating an example of a first GUI of a second user interface according to one embodiment of the present disclosure.

[0025] FIG. 4 is a drawing illustrating an example of a second GUI of a second user interface according to one embodiment of the present disclosure.

[0026] FIG. 5 is a drawing illustrating an example of a third GUI of a second user interface according to one embodiment of the present disclosure.

[0027] FIG. 6 is a drawing illustrating an example of a third user interface according to one embodiment of the present disclosure.

[0028] FIG. 7 is a drawing illustrating examples of a fourth GUI and a fifth GUI of a third user interface according to one embodiment of the present disclosure.

[0029] FIG. 8 is a drawing illustrating an example of a fourth user interface according to one embodiment of the present disclosure.

[0030] FIG. 9 is a diagram illustrating an example of a method for evaluating the performance of an autonomous driving algorithm according to one embodiment of the present disclosure.

[0031] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0032] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0033] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0034] The terms used in this disclosure will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this disclosure have been selected to be as generally used as possible, taking into account their functions within the disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0035] In this disclosure, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout this disclosure, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0036] As used in this disclosure, the terms “module” or “part” refer to software or hardware components, and the “module” or “part” performs certain roles. However, the meaning of “module” or “part” is not limited to software or hardware. The “module” or “part” may be configured to reside in an addressable storage medium or configured to run on one or more processors. Accordingly, as an example, the “module” or “part” may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the “module” or “part” may be combined into a smaller number of components and “modules” or “parts,” or further separated into additional components and “modules” or “parts.”

[0037] According to one embodiment, a 'module' or 'part' may be implemented as a processor and memory. The term 'processor' should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term 'processor' may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term 'memory' should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Marked Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0038] The terms 1, 2, A, B, (a), (b), etc. used in this disclosure are used merely to distinguish one component from another, and the essence, order, or sequence of said component is not limited by such terms.

[0039] Where in this disclosure it is stated that one component is 'connected,' 'coupled,' or 'connected' to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be 'connected,' 'coupled,' or 'connected' between each component.

[0040] As used in this disclosure, “comprising” and / or “comprising” does not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0041] FIG. 1 is a drawing illustrating an example of a first user interface according to an embodiment of the present disclosure. Referring to FIG. 1, an electronic device according to an embodiment (e.g., the electronic device (100) of FIG. 2) can evaluate the performance of an autonomous driving algorithm (hereinafter referred to as "algorithm") related to the autonomous driving of a mobile device (e.g., a vehicle). For example, the electronic device may obtain execution data of an algorithm related to the autonomous driving of a mobile device from a mobile device (or a server and / or system related to the mobile device) that actually drives at least a portion of a driving section within a specific road environment, and evaluate the performance of the algorithm using at least a portion of the execution data. Alternatively, the electronic device may obtain execution data of the algorithm from an autonomous driving simulation program that is executed to drive at least a portion of a driving section in a digital twin virtual environment that reproduces a specific road environment, and evaluate the performance thereof. According to various embodiments, the algorithm may include at least one of a rule-based algorithm, an artificial neural network (machine learning) algorithm, or a combination thereof related to the autonomous driving of a mobile device (or a virtual object corresponding to the mobile device). In various embodiments, the execution data of the algorithm may include at least one of output data (or result data) resulting from the execution of the algorithm, parameter data related to the execution of the algorithm, or secondary data derived by a predefined formula, function, or learned artificial neural network model based on at least one of these data. Additionally, the execution data of the algorithm may further include log data of an autonomous driving test mapped to a time stamp containing the aforementioned data and information on the time of generation (or acquisition) of said data.According to one embodiment, the electronic device can visually provide (or display) various information related to the performance evaluation of an algorithm through the execution screen of a specified application (e.g., a web browser).

[0042] In relation to the performance evaluation of an algorithm and the provision of evaluation information, the electronic device may set at least a portion of a driving section within a specific road environment where a mobile device (or a virtual object corresponding to the mobile device) will perform autonomous driving. For example, the electronic device may set at least a portion of a driving section (11) by running an autonomous driving simulation program to provide a map (10) or a digital twin virtual environment corresponding to a specific road environment and receiving a user input specifying at least a portion of an autonomous driving path set in the map (10) or the digital twin virtual environment. In this regard, at least a portion of the driving section (11) may be defined by a user input specifying the range of the driving section on the autonomous driving path, or by a user input selecting at least a portion of a plurality of driving sections that are pre-separated for the autonomous driving path.

[0043] Additionally, the electronic device may set a driving scenario for a mobile device (or a virtual object corresponding to the mobile device) to be reproduced in at least some driving sections (11) within a specific road environment. For example, the electronic device may provide (or display) a list in which a plurality of driving scenarios are listed, and receive user input to select a specific driving scenario based on a road type (e.g., straight lanes, intersections, tunnels, toll gates and / or overpasses) and / or traffic infrastructure (e.g., lanes, stop lines, traffic lights, signs and / or crosswalks) associated with at least some driving sections (11). Alternatively, the electronic device may obtain information about a driving scenario associated with at least some driving sections (11) from a memory (e.g., memory (110) of FIG. 2). In this regard, the memory may store mapping information associated with a specific driving scenario for each of the plurality of driving sections that are pre-separated for an autonomous driving path. For example, memory may store a database in which driving scenarios valid for executing autonomous driving tests of a mobile device are mapped according to the road type and / or traffic infrastructure of each of multiple driving sections.

[0044] In one embodiment, the electronic device may provide data regarding at least a set portion of driving sections (11) and driving scenarios to a mobile device. For example, the electronic device may transmit data to a mobile device (or a server and / or system associated with the mobile device) that actually drives in a specific road environment. Alternatively, the electronic device may load data into an autonomous driving simulation program that runs a digital twin virtual environment associated with a specific road environment.

[0045] In one embodiment, the electronic device may acquire execution data of an algorithm executed based on a driving scenario in at least a portion of a driving section (11) within a specific road environment in response to the transmission or loading of data. For example, the electronic device may acquire execution data of the algorithm from a mobile device (or a server and / or system associated with the mobile device) that actually drives in the specific road environment or from an autonomous driving simulation program that runs a digital twin virtual environment. According to various embodiments, the acquisition of algorithm execution data by the electronic device may mean that an autonomous driving test has been performed on a mobile device or a virtual object corresponding to the mobile device that has autonomously driven in at least a portion of a driving section (11) according to a driving scenario.

[0046] In one embodiment, the electronic device may evaluate the performance of the algorithm based on the execution data of the algorithm and provide the performance evaluation information. For example, the electronic device may process the information represented by the execution data of the algorithm based on at least one evaluation item (or key performance indicator (KPI)) related to the performance evaluation of the algorithm, and provide (or display) the performance evaluation information representing the processing result through the execution screen of a designated application (e.g., a web browser).

[0047] In one embodiment, the electronic device may execute an application to provide performance evaluation information of an algorithm based on user input. For example, the electronic device may execute the application based on user authentication of a specified method. According to one embodiment, the electronic device may provide a plurality of tabs (21, 23, and 25) related to the performance evaluation of an algorithm through an application execution screen (20). For example, the electronic device may output an application execution screen (20) comprising a first tab (21) that supports a series of processes for performing performance evaluation of an algorithm (e.g., KPI measurement), a second tab (23) that provides information on a plurality of evaluation items that can be used for performance evaluation of an algorithm (e.g., KPI list), and a third tab (25) that supports history inquiry regarding the performance evaluation of an algorithm (e.g., history inquiry). In one embodiment, the plurality of tabs (21, 23, and 25) may be provided by default on the application execution screen (20). For example, a plurality of tabs (21, 23, and 25) may be provided identically for a plurality of pages output by the application execution screen (20). According to various embodiments, at least some of the plurality of tabs (21, 23, and 25) may be omitted, or additional tabs may be included in the plurality of tabs (21, 23, and 25).

[0048] According to one embodiment, a first page (e.g., a main page and / or a dashboard) output by an application execution screen (20) may include a first user interface (30). For example, an electronic device may provide a first user interface (30) that includes at least one of a first content (31) (e.g., recent KPI evaluation history) representing performance evaluation information of a recently performed algorithm on the first page of the application, a second content (33) (e.g., previous evaluation list) representing history information regarding the performance evaluation of the algorithm, or a third content (35) (e.g., recent test list) representing history information regarding the autonomous driving test of a mobile device. According to various embodiments, depending on the history of the performance evaluation of the algorithm and / or the history of the autonomous driving test of the mobile device, at least some of the first content (31), the second content (33), or the third content (35) may be omitted from the first user interface (30).

[0049] According to various embodiments, the layout of the first user interface (30) included in the application execution screen (20) (or the first page) may be varied. For example, at least one of the location, arrangement, type, or size of the occupied area of ​​at least one content included in the first user interface (30) may be varied. Hereinafter, it should be understood that at least one of the location, arrangement, type, or size of the at least one content and / or at least one graphical user interface (GUI) included in the user interface mentioned in various embodiments may also be varied.

[0050] FIG. 2 is a drawing illustrating an example of an electronic device in a network environment according to one embodiment of the present disclosure. Referring to FIG. 2, an electronic device (100) according to one embodiment may include various types of devices capable of storing, providing, and executing computer-executable programs and data related to a data processing service (e.g., a service for evaluating the performance of an autonomous driving algorithm related to autonomous driving of a mobile device and providing performance evaluation information). For example, the electronic device (100) may include at least one of a system device, a server device, a cloud service-based distributed computing device, or a mobile communication device.

[0051] In one embodiment, the electronic device (100) can communicate with at least one external electronic device (200) through a network (300). In various embodiments, the network (300) may be configured, depending on the installation environment, as a wired network including at least one of Ethernet, a wired home network (power line communication), a telephone line communication device, or RS-serial communication, a mobile communication network, a wireless network including at least one of a WLAN (wireless LAN), Wi-Fi, Bluetooth, or ZigBee, or a combination thereof. In various embodiments, the method of communication between the electronic device (100) and at least one external electronic device (200) is not limited and may include a communication method utilizing a communication network (e.g., a mobile communication network, a wired internet, a wireless internet, a broadcasting network and / or a satellite network) that the network (300) can support, as well as a short-range wireless communication method between the electronic device (100) and at least one external electronic device (200).

[0052] In one embodiment, the data processing service provided by the electronic device (100) may be provided to a user of the electronic device (100) through a data processing application (e.g., a web browser) included in the electronic device (100). Additionally, the data processing service provided by the electronic device (100) may be provided to a user of at least one external electronic device (200) through a data processing application included in at least one external electronic device (200).

[0053] According to one embodiment, the electronic device (100) may include components related to the provision of data processing services. For example, the electronic device (100) may include a memory (110), a display (120), a communication module (130), and at least one processor (140). According to various embodiments, some of the components of the electronic device (100) described above may be omitted, or the electronic device (100) may additionally include at least one other component. For example, the electronic device (100) may additionally include a speaker device, a receiver device, a haptic device, a camera device, a touchscreen display device, and / or various sensors. At least some of the components included in the electronic device (100) may be connected to each other via a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface) to exchange signals (e.g., commands or data).

[0054] In one embodiment, the memory (110) may include a non-transient recording medium readable by any computer. According to one embodiment, the memory (110) may include a non-perishable permanent mass storage device such as ROM (read-only memory), a disk drive, a solid-state drive (SSD), and / or flash memory. In various embodiments, a non-perishable permanent mass storage device such as ROM, an SSD, flash memory, and / or a disk drive may be included in the electronic device (100) as a separate permanent storage device distinct from the memory (110). Additionally, the memory (110) may store an operating system, at least one application, and / or at least one program code. In various embodiments, these software components may be loaded from a computer-readable recording medium distinct from the memory (110). Such a recording medium may include a recording medium that can be directly connected to the electronic device (100), and may include, for example, a floppy drive, a disk, a tape, a DVD / CD-ROM drive, and / or a memory card. Alternatively, software components may be loaded into memory (110) via a communication module (130) rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (110) based on a computer program installed by files provided by a file distribution system via a network (300). In one embodiment, memory (110) may store instructions related to the functional operation of components of the electronic device (100). For example, memory (110) may store instructions that cause components of the electronic device (100) to perform defined functional operations when executed by at least one processor (140).

[0055] In one embodiment, the display (120) may be exposed to the outside of the electronic device (100) in at least a portion to visually provide various content. For example, the display (120) may output (or display) an execution screen and / or user interface of at least one application included in the electronic device (100). Additionally, the display (120) may output a digital twin virtual environment that reproduces a specific road environment. In this regard, the display (120) may include a display driving integrated circuit that receives and processes a driving signal corresponding to image information from at least one processor (140). Additionally, the display (120) may include a touch panel (or touch sensing circuit) and may detect various types of user input based on the touch panel and output a corresponding electrical signal.

[0056] In one embodiment, the communication module (130) may provide a configuration or function for the electronic device (100) to communicate with at least one external electronic device (200) through a network (300). For example, the communication module (130) may establish communication (or a communication channel) with at least one external electronic device (200) according to a defined communication protocol and perform the transmission and reception of signals and / or data (or data packets) through said communication. According to various embodiments, the communication module (130) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module and / or a GNSS (global navigation satellite system) communication module) and a wired communication module (e.g., a LAN (local area network) communication module and / or a power line communication module). Each of these wireless communication modules and wired communication modules may be composed of separate chips or integrated into a single chip.

[0057] In one embodiment, at least one processor (140) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. According to various embodiments, instructions of a computer program may be provided to at least one processor (140) by memory (110) and / or a communication module (130). For example, at least one processor (140) may be configured to execute instructions received according to program code stored in memory (110) and / or instructions received from at least one external electronic device (200) through the communication module (130) and a network (300).

[0058] According to one embodiment, at least one processor (140) may be configured to execute a data processing application (e.g., a web browser) for providing data processing services. For example, at least one processor (140) may execute a data processing application by executing computer program code (or instructions) loaded into memory (110). In various embodiments, at least one processor (140) may receive signals, data, and / or information transmitted from at least one external electronic device (200) through a communication module (130) while executing the application, and may process said signals, data, and / or information and store them in memory (110). Conversely, at least one processor (140) may transmit signals, data, and / or information generated while executing the application to at least one external electronic device (200) through the communication module (130).

[0059] Various embodiments of the present disclosure described below with reference to the drawings may be implemented by at least one processor (140) executing computer program code stored in memory (110).

[0060] FIG. 3 is a drawing illustrating an example of a first GUI of a second user interface according to an embodiment of the present disclosure. FIG. 4 is a drawing illustrating an example of a second GUI of a second user interface according to an embodiment of the present disclosure. Referring to FIG. 3, an electronic device according to an embodiment (e.g., the electronic device (100) of FIG. 2) may receive user input for a first tab (21) included in an execution screen (20) of an application and, in response, output a second page of an application that provides a second user interface (40). For example, the electronic device may provide a second user interface (40) that includes a fourth content (41) that supports a query on an autonomous driving test of a mobile device (or a virtual object corresponding to a mobile device) performed within a specific period range, and a fifth content (43) that indicates a list of autonomous driving tests according to the query results within the specific period range.

[0061] In one embodiment, the list represented by the fifth content (43) may list at least one autonomous driving test in which execution data of the algorithm was obtained (e.g., sorted in ascending or descending order based on the date the autonomous driving test was performed). According to various embodiments, at least one autonomous driving test listed in the list may include at least one of the following: date information on which the autonomous driving test was performed, driving section (or zone) information, scenario information, type information of the autonomous driving test (e.g., VIL simulation (vehicle in the loop simulation), SIL simulation (software in the loop simulation), HIL simulation (hardware in the loop simulation), DIL simulation (driver in the loop simulation), PIL simulation (processor in the loop simulation), MIL simulation (model in the loop simulation), space occupancy test, or infrastructure occupancy test), or file format information of the autonomous driving test, but the present disclosure is not limited thereto. For example, at least one autonomous driving test listed in the list may further include various information of the autonomous driving test expressed in text, images, symbols and / or combinations thereof (e.g., information on the time the autonomous driving test was performed, information on the moving device and / or sensor information of the moving device).

[0062] According to various embodiments, a space occupancy test exemplified as at least part of the autonomous driving test type information may be referred to as testing the autonomous driving of a mobile device (or a virtual object corresponding to the mobile device) within a region of a two-dimensional or three-dimensional space. In this regard, the two-dimensional or three-dimensional space used for the space occupancy test may be defined or described by a set of spatial coordinates or a regional name, etc. Additionally, an infrastructure occupancy test exemplified as at least part of the autonomous driving test type information may be referred to as a test performed by the mobile device (or a virtual object corresponding to the mobile device) using at least one traffic infrastructure (e.g., a road, sidewalk, tunnel, bridge, lane, road marking, safety sign, traffic light, guardrail, and / or RSU (road side unit)) or interacting with at least one traffic infrastructure while performing autonomous driving.

[0063] In one embodiment, the electronic device may receive user input selecting a specific autonomous driving test from a list represented by the fifth content (43). For example, the electronic device may receive user input selecting the specific autonomous driving test from the list to evaluate the performance of an algorithm executed during the performance of the specific autonomous driving test. In one embodiment, in response to receiving user input, the electronic device may provide a first GUI (45) on at least a portion of the second user interface (40) of the application. For example, the electronic device may display the first GUI (45) so as to overlap with at least a portion of the fifth content (43).

[0064] According to one embodiment, the electronic device may recommend at least one evaluation item (e.g., TTS (time to collision), DST (deceleration to safety time), and / or safety distance compliance rate) that can be used to evaluate the performance of an algorithm related to an autonomous driving test selected by user input through the first GUI (45). For example, the electronic device may identify at least one evaluation item associated with a driving scenario of an autonomous driving test selected by user input from a memory (e.g., memory (110) of FIG. 2) and recommend the at least one evaluation item. In this regard, the memory may store a database in which each of a plurality of driving sections pre-separated for the autonomous driving path of a moving device (or a virtual object corresponding to the moving device) and a driving scenario valid for executing an autonomous driving test in the corresponding driving section and at least one evaluation item valid for evaluating the performance of the algorithm during the execution of the corresponding driving scenario are mapped. Alternatively, the electronic device may recommend at least one evaluation item based on the performance evaluation history of the algorithm. For example, based on log data related to the performance evaluation of an algorithm, the electronic device can identify at least one evaluation item previously set for a driving scenario of an autonomous driving test selected by user input, and determine the at least one evaluation item as a recommended evaluation item.

[0065] According to one embodiment, the electronic device may change at least a portion of at least one recommended evaluation item based on user input to the first GUI (45). In this regard, the electronic device may provide a visual object that supports the deletion of at least one recommended evaluation item or the addition of at least one other evaluation item to the first GUI (45). For example, the electronic device may provide a first visual object (45a) that allows the evaluation item to be excluded from the setting target in an area adjacent to the at least one recommended evaluation item. Additionally, the electronic device may provide a second visual object (45b) (e.g., 'Custom') that allows at least one evaluation item other than the at least one recommended evaluation item to be added to the setting target. In various embodiments, in response to receiving user input to the second visual object (45b), the electronic device may provide a list in which at least one evaluation item other than the at least one recommended evaluation item is listed, and add and display at least one evaluation item selected by user input from the list to the first GUI (45). In one embodiment, the electronic device may, in response to receiving user input for a third visual object (e.g., 'Next') provided to the first GUI (45), set at least one recommendation evaluation item displayed on the first GUI (45) as an evaluation item to be used for evaluating the performance of the algorithm.

[0066] Referring to FIG. 4, an electronic device according to one embodiment may provide a second GUI (47) to at least a portion of a second user interface (40) of an application in response to at least one evaluation item to be used for evaluating the performance of an algorithm being set (or determined). For example, the electronic device may stop (or disable) the display of the first GUI (45) and display a second GUI (47) that overlaps with at least a portion of the fifth content (43).

[0067] In one embodiment, the electronic device may provide at least one of calculation formula information (47a), user-defined value information (47b) (e.g., operands such as user-defined value or threshold), or mathematical expression condition information (47c) (e.g., condition element or comparison operator) related to at least one evaluation item through the second GUI (47). For example, the electronic device may provide information through the second GUI (47) regarding at least one of a calculation formula to which at least a portion of the execution data of the algorithm is applied to evaluate the performance of the algorithm for at least one evaluation item set through the first GUI (45), a user-defined value to be compared with the result value of the calculation formula related to at least one evaluation item, or a mathematical expression condition (e.g., inequality sign) used for comparison between the result value of the calculation formula and the user-defined value. In this regard, a database in which user-defined values ​​and mathematical expression conditions are mapped by user input may be stored in memory for each of a plurality of evaluation items that can be used to evaluate the performance of the algorithm.

[0068] In various embodiments, the user-defined value or mathematical expression condition provided through the second GUI (47) may be changed. For example, the electronic device may receive user input entering a new user-defined value or a new mathematical expression condition through the second GUI (47). Alternatively, the electronic device may receive user input selecting a display area for a specific user-defined value or a specific mathematical expression condition in the second GUI (47), and in response, provide a pop-up window that allows changing the user-defined value or mathematical expression condition.

[0069] In one embodiment, when the electronic device receives user input for a fourth visual object (e.g., 'save') provided to the second GUI (47), it can evaluate the performance of an algorithm for at least one evaluation item based on at least one set evaluation item and a calculation formula, a user-defined value, and a mathematical condition related to that evaluation item. For example, the electronic device may obtain information (or a value) required by a calculation formula related to at least one evaluation item from the execution data of the algorithm and apply it to the calculation formula, and compare the result value obtained from the calculation formula with a user-defined value related to that evaluation item based on the mathematical condition related to that evaluation item. For example, assuming that the result value obtained from the calculation formula related to a specific evaluation item is A, the user-defined value is B, and the mathematical condition is less than or equal to, the electronic device may determine that the performance of the algorithm for that evaluation item is satisfied (e.g., pass) if the result value A is less than or equal to the user-defined value B. Conversely, if result value A is greater than user-defined value B, the electronic device may determine that the algorithm's performance is not satisfactory for the corresponding evaluation item (e.g., fail).

[0070] FIG. 5 is a diagram illustrating an example of a third GUI of a second user interface according to an embodiment of the present disclosure. Referring to FIG. 5, an electronic device according to an embodiment (e.g., the electronic device (100) of FIG. 2) may provide performance evaluation information of an algorithm in response to the completion of a performance evaluation of an algorithm for at least one set evaluation item. For example, the electronic device may provide (or display) a third GUI (49) containing performance evaluation information of an algorithm to a second user interface (40) provided by an application execution screen (20) (or a second page of the application).

[0071] According to one embodiment, the electronic device may provide at least one content related to algorithm performance evaluation information through the third GUI (49). For example, the electronic device may provide the third GUI (49) including at least one of the following: a sixth content (42) providing an autonomous driving simulation video (or autonomous driving test video) of a mobile device (or a virtual object corresponding to a mobile device) that performed autonomous driving based on a driving scenario in at least a set driving section; a seventh content (44) providing a timeline of the autonomous driving simulation video; or an eighth content (48) providing evaluation result information regarding the performance evaluation of the algorithm. According to various embodiments, the electronic device may further display a ninth content (46) through the third GUI (49) providing information regarding at least one evaluation item, user-defined value, or mathematical expression condition used for the performance evaluation of the algorithm.

[0072] In one embodiment, the seventh content (44) may include at least one of a progress bar (44a) indicating the playback status of the autonomous driving simulation video, at least one marker (44b) indicating the time of occurrence of at least one event that occurred when the autonomous driving simulation was executed on the progress bar (44a), or a seek bar (44c) configured to be movable on the progress bar (44a) based on user input. According to one embodiment, the playback screen of the autonomous driving simulation video provided by the sixth content (42) may be changed based on user input regarding at least a part of the seventh content (44). For example, in response to receiving user input that changes the position of the seek bar (44c) on the progress bar (44a), the electronic device may provide the autonomous driving simulation video as a playback screen at a playback time corresponding to the position of the seek bar (44c).

[0073] According to various embodiments, the time of occurrence of at least one event indicated by the marker (44b) may mean the time of occurrence of an action of at least one surrounding dynamic object (e.g., a pedestrian and / or a non-playable character vehicle (NPC) device) interacting with the moving device (or a virtual object corresponding to the moving device) in the autonomous driving simulation, or the time of occurrence of an action of the moving device triggered at the time of such action. In this regard, the action of at least one surrounding dynamic object may be defined as at least part of a driving scenario (e.g., a driving scenario in the ASAM OpenSCENARIO standard format). In various embodiments, at least one event may be associated with at least one evaluation item. For example, at least one event may mean an action of the moving device or surrounding dynamic object capable of measuring (or evaluating) at least one set evaluation item.

[0074] In one embodiment, the evaluation result information of the algorithm provided by the 8th content (48) may include at least one of the following: time information of an event that occurred during the execution of the algorithm, result value information calculated from a calculation formula of an evaluation item related to the event, or performance judgment information (e.g., pass or fail) determined by comparing the result value with a user-defined value based on mathematical formula conditions.

[0075] According to one embodiment, the electronic device may generate a file containing performance evaluation information of an algorithm provided through the second user interface (40). For example, the electronic device may generate a file containing data regarding at least one of the sixth content (42), the seventh content (44), the eighth content (48), or the ninth content (46) and store it in memory.

[0076] FIG. 6 is a drawing illustrating an example of a third user interface according to one embodiment of the present disclosure. FIG. 7 is a drawing illustrating examples of a fourth GUI and a fifth GUI of a third user interface according to one embodiment of the present disclosure. Referring to FIG. 6, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 2) may output a third page of an application that provides a third user interface (50) in response to receiving user input for a second tab (23) included in an execution screen (20) of an application. For example, the electronic device may provide a third user interface (50) that includes a list (51) in which a plurality of evaluation items that can be used to evaluate the performance of an algorithm and calculation formulas related to said evaluation items are listed. In this regard, the memory of the electronic device (e.g., the memory (110) of FIG. 2) may store a database in which a plurality of evaluation items and calculation formulas associated with said evaluation items are mapped, which can be used to evaluate the performance of an algorithm in an autonomous driving process of a mobile device (or a virtual object corresponding to a mobile device) performed based on various driving scenarios in various driving sections. Although various types of evaluation items and related calculation formulas are illustrated in FIG. 6, the present disclosure is not limited thereto. For example, at least a portion of the calculation formula of a specific evaluation item illustrated in FIG. 6 may be modified. Additionally, at least one evaluation item and related calculation formula not illustrated in FIG. 6 may be further stored in the memory (or database) of the electronic device, and examples of such evaluation items and related calculation formulas may be referenced in Table 1.

[0077] Evaluation Criteria Calculation Formula Minimum Forward Safe Distance (Own Vehicle Speed ​​- Front Vehicle Speed) X Own Vehicle Reaction Time + (Own Vehicle Speed ​​- Front Vehicle Speed)^2 / (2 X Own Vehicle Deceleration) + Front Vehicle Speed ​​X Collision Avoidance Time Factor Minimum Rear Safe Distance (Rear Vehicle Speed ​​- Own Vehicle Speed) X Own Vehicle Reaction Time + (Rear Vehicle Speed ​​- Own Vehicle Speed)^2 / (2 X Own Vehicle Deceleration) + Own Vehicle Speed ​​X Collision Avoidance Time Factor Minimum Lateral Safe Distance Own Vehicle Width + (Own Vehicle Speed ​​X Safety Factor) Driver Experience (Degree of Change in Lateral Acceleration During Lane Change) Own Vehicle Speed^2 / Radius of Curvature of Lane Change Path ≤ A (excellent) ≤ B (very good) ≤ C (good) ≤ D (so so) ≤ E (bad) ≤ F (very bad) ≤ G (terrible) Driver Experience (Degree of Change in Acceleration) Change in Own Vehicle Acceleration / Change in Time Degree of Impact on Surrounding Vehicles Own Vehicle Deceleration < A (good) ≥ A (bad)

[0078] Referring to FIG. 7, the electronic device may provide (or display) a fourth GUI (53) in a part of the third user interface (50). For example, the electronic device may provide a fourth GUI (53) that supports the addition of at least one evaluation item to a list (51). According to one embodiment, the electronic device may receive user input selecting the fourth GUI (53) and, in response, display a fifth GUI (55) in a part of the third user interface (50). For example, the electronic device may provide a fifth GUI (55) that allows input of identification information (e.g., KPI name) and calculation formula information for at least one evaluation item to be added to the list (51). According to various embodiments, the electronic device may store information regarding at least one evaluation item received through the fifth GUI (55) in a database in memory (e.g., a database in which a plurality of evaluation items and calculation formulas related to the evaluation items are mapped).

[0079] FIG. 8 is a drawing illustrating an example of a fourth user interface according to an embodiment of the present disclosure. Referring to FIG. 8, an electronic device according to an embodiment (e.g., the electronic device (100) of FIG. 2) may output a fourth page of an application that provides a fourth user interface (60) in response to receiving user input for a third tab (25) included in an execution screen (20) of the application. For example, the electronic device may provide a fourth user interface (60) that includes a tenth content (61) that supports a lookup of an algorithm performance evaluation file stored within a specific period range, and a eleventh content (63) that indicates a list of performance evaluation files according to the lookup results within the specific period range. According to various embodiments, the electronic device may provide a specific performance evaluation file based on user input regarding at least a portion of the eleventh content (63). For example, when the electronic device receives user input regarding a fifth visual object (e.g., a download icon) included in the eleventh content (63), it may support the download of a performance evaluation file associated with said fifth visual object. According to various embodiments, in an operation that supports the download of a performance evaluation file, the electronic device may determine the download format of the performance evaluation file (e.g., PDF and / or excel) based on user input.

[0080] FIG. 9 is a diagram illustrating an example of a method for evaluating the performance of an autonomous driving algorithm according to an embodiment of the present disclosure. The steps of the method for evaluating the performance of an autonomous driving algorithm (900) described in the embodiment of FIG. 9 may be performed sequentially or non-sequentially. For example, the order of execution of the steps described in the embodiment of FIG. 9 may be changed, some steps may be performed repeatedly, or at least two steps may be performed in parallel.

[0081] Referring to FIG. 9, in step S910, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 2) may set a driving scenario for at least a portion of a driving section within a specific road environment. In one embodiment, the electronic device may set at least a portion of a driving section and a driving scenario for said driving section based on user input. For example, the electronic device may receive user input specifying at least a portion of a driving section for an autonomous driving path set in a map or digital twin virtual environment corresponding to a specific road environment, and may receive user input selecting a specific driving scenario from a list in which a plurality of driving scenarios are listed. Alternatively, the electronic device may set at least a portion of a driving section based on user input and obtain information of a driving scenario associated with said at least a portion of the driving section from a memory (e.g., the memory (110) of FIG. 2).

[0082] In one embodiment, the electronic device may transmit data regarding at least some set driving sections and driving scenarios to a mobile device (or a server and / or system associated with the mobile device) that actually drives in a specific road environment, or load it into an autonomous driving simulation program that runs a digital twin virtual environment associated with a specific road environment.

[0083] In step S920, an electronic device according to one embodiment may obtain execution data of an autonomous driving algorithm executed based on a driving scenario in at least a portion of the driving section. For example, the electronic device may obtain execution data for an autonomous driving algorithm related to the autonomous driving of a mobile device from a mobile device (or a server and / or system associated with the mobile device) that actually drives in a specific road environment. Alternatively, the electronic device may obtain execution data for an autonomous driving algorithm from an autonomous driving simulation program that runs in a digital twin virtual environment that reproduces a specific road environment.

[0084] In step S930, an electronic device according to one embodiment may set at least one evaluation item for evaluating the performance of the autonomous driving algorithm based on execution data of the autonomous driving algorithm. In this regard, the electronic device may provide a GUI through a user interface of a specified application (e.g., a web browser). For example, the electronic device may provide a GUI in which at least one evaluation item that can be used to evaluate the performance of the autonomous driving algorithm is recommended (or displayed).

[0085] In one embodiment, the electronic device may identify at least one evaluation item to be recommended via a GUI from a memory (e.g., memory (110) of FIG. 2). In this regard, the memory may store a database in which each of a plurality of pre-separated driving sections for an autonomous driving path of a mobile device (or a virtual object corresponding to the mobile device), a driving scenario valid for executing an autonomous driving test in the corresponding driving section, and at least one evaluation item valid for evaluating the performance of an autonomous driving algorithm during the execution of the corresponding driving scenario are mapped. The electronic device may refer to the database in the memory to identify at least one evaluation item related to the execution data of the acquired autonomous driving algorithm and recommend the at least one evaluation item via a GUI. Alternatively, the electronic device may identify at least one evaluation item to be recommended via a GUI based on the performance evaluation history of the autonomous driving algorithm. For example, the electronic device may identify at least one evaluation item previously set for the driving scenario of the autonomous driving test related to the execution data of the acquired autonomous driving algorithm based on log data related to the performance evaluation of the autonomous driving algorithm and recommend the at least one evaluation item via a GUI.

[0086] In one embodiment, in response to receiving user input for a specific visual object included in a GUI, the electronic device may set at least one evaluation item recommended through the GUI as an evaluation item to be used for evaluating the performance of an autonomous driving algorithm from which execution data has been obtained.

[0087] In step S940, an electronic device according to one embodiment may provide performance evaluation information of an autonomous driving algorithm based on at least one set evaluation item. For example, the electronic device may display a GUI containing performance evaluation information of an autonomous driving algorithm through a user interface of a specified application.

[0088] According to one embodiment, the electronic device may provide at least one piece of content related to performance evaluation information of an autonomous driving algorithm through a GUI. For example, the electronic device may display at least one of the following: content providing an autonomous driving simulation video (or autonomous driving test video) of a mobile device (or a virtual object corresponding to a mobile device) that performed autonomous driving based on a driving scenario in at least a set portion of the driving section; content providing a timeline of the autonomous driving simulation video; or content providing evaluation result information regarding the performance evaluation of the algorithm.

[0089] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0090] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0091] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0092] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0093] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or marked data storage device, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0094] When implemented in software, the techniques described above may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available media accessible by a computer. As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transfer or store desired program code in the form of instructions or data structures. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0095] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disks and discs include CDs, laser discs, optical discs, DVDs (digital versatile discs), floppy disks, and Blu-ray discs, wherein disks usually play data magnetically, whereas discs play data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0096] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0097] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0098] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to the present disclosure.

Claims

1. A method for evaluating the performance of an autonomous driving algorithm executed by at least one processor, A step of setting a driving scenario for at least a portion of a driving section within a road environment; A step of obtaining execution data of an autonomous driving algorithm executed based on the driving scenario in at least some of the driving sections; A step of setting at least one evaluation item for evaluating the performance of the autonomous driving algorithm based on the above execution data; and A step of providing performance evaluation information of the autonomous driving algorithm based on at least one evaluation item. A method for evaluating the performance of an autonomous driving algorithm, including 2. In Paragraph 1, The step of setting at least one evaluation item above is, A step of displaying a first GUI (graphical user interface) including at least one recommended evaluation item related to the driving scenario through the user interface of a designated application. A method for evaluating the performance of an autonomous driving algorithm, including 3. In Paragraph 2, The step of displaying the first GUI above is, Step of identifying at least one recommended evaluation item associated with the driving scenario from a database A method for evaluating the performance of an autonomous driving algorithm, including 4. In Paragraph 2, The step of displaying the first GUI above is, A step of identifying the history of evaluation items set in relation to the above driving scenario; and Step of determining at least one recommended evaluation item based on the above history A method for evaluating the performance of an autonomous driving algorithm, including 5. In Paragraph 2, The step of displaying the first GUI above is, A step of changing at least a portion of the above-mentioned at least one recommendation evaluation item based on user input A method for evaluating the performance of an autonomous driving algorithm, including 6. In Paragraph 1, The step of setting at least one evaluation item above is, A step of displaying a second GUI comprising at least one of a calculation formula related to at least one evaluation item, a user-defined value compared with the result value of the calculation formula, or a mathematical formula condition used for comparison between the result value and the user value. A method for evaluating the performance of an autonomous driving algorithm, including 7. In Paragraph 1, The step of providing the above performance evaluation information is, A step of displaying a third GUI including at least one of an image of an autonomous driving simulation executed based on the driving scenario in at least some of the driving sections above, or a progress bar related to the playback of the image of the autonomous driving simulation above. A method for evaluating the performance of an autonomous driving algorithm, including 8. In Paragraph 1, A step of displaying a list of multiple evaluation items available for setting at least one evaluation item through the user interface of a designated application. A method for evaluating the performance of an autonomous driving algorithm, further comprising 9. A computer program stored on a computer-readable recording medium for executing a method for evaluating the performance of an autonomous driving algorithm according to any one of paragraphs 1 through 8 on a computer.

10. In an electronic device, Memory for storing instructions; and At least one processor Includes, When the above instructions are executed by the at least one processor, the electronic device, Set driving scenarios for at least a portion of the driving section within the road environment, and Acquire execution data of an autonomous driving algorithm executed based on the driving scenario in at least some of the driving sections above, and Based on the above execution data, at least one evaluation item is set to evaluate the performance of the autonomous driving algorithm, and An electronic device that provides performance evaluation information of the autonomous driving algorithm based on at least one evaluation item.

Images