Method for generating training data and apparatus for performing same

Abstract

Disclosed is a method for generating training data. The method according to an embodiment may comprise the operations of: collecting a navigation history; generating navigation information on the basis of a driving log of a vehicle; and generating a dataset on the basis of the navigation history and the navigation information.

Description

Method for generating training data and device for performing the same

[0001] The following disclosure relates to a method for generating training data and an apparatus for performing the same.

[0002] An end-to-end model is a type of machine learning model that derives output data by performing operations directly on input data without intermediate processing steps or separate components. Based on information obtained from sensors of an autonomous driving vehicle (ADV), an end-to-end model can generate a path for the vehicle to drive (e.g., a target trajectory).

[0003] The background technology described above is possessed or acquired by the inventor in the process of deriving the content of the disclosure of the present application, and cannot necessarily be considered as prior art disclosed to the general public prior to the filing of this application.

[0004] One embodiment may provide a technique for generating data for training an end-to-end model.

[0005] One embodiment can generate a dataset based on navigation information and map data.

[0006] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist.

[0007] A method for generating a dataset for training a model according to one embodiment may include collecting a navigation history, generating navigation information based on a vehicle's driving log, and generating the dataset based on the navigation history and the navigation information.

[0008] The above driving record may be a record of the vehicle's driving recorded chronologically on an SD (standard definition) map.

[0009] The above navigation usage record may include information regarding a starting point, a destination, and a driving route from the starting point to the destination.

[0010] The above navigation usage record may be collected from at least one of a navigation device and a navigation server installed in a vehicle.

[0011] The operation of generating the above navigation information may include the operation of extracting the starting point and destination of the vehicle from the driving record of the vehicle, and the operation of inputting the starting point and the destination into a navigation path search algorithm to generate the navigation information including a path to the destination.

[0012] The above model may be an end-to-end model included in a vehicle's autonomous driving system.

[0013] The operation of generating the above dataset may include the operation of acquiring path data based on the navigation usage record and the navigation information, and the operation of encoding the path data into SD map-based geometry information.

[0014] The operation of generating the above dataset may include the operation of extracting turn-by-turn (TBT) information from the navigation usage record and the navigation information, and the operation of encoding the path data and the TBT information to generate the above dataset.

[0015] According to one embodiment, a computer-readable recording medium storing one or more computer programs may include instructions for performing the method in a processor.

[0016] An apparatus for generating a dataset for training a model according to one embodiment may include at least one processor and a memory including instructions. Based on the instructions being executed individually or collectively by the at least one processor, the apparatus may be able to collect a navigation history, generate navigation information based on a vehicle's driving log, and generate the dataset based on the navigation history and the navigation information.

[0017] The above driving record may be a record of the vehicle's driving recorded chronologically on an SD (standard definition) map.

[0018] The above navigation usage record may include information regarding a starting point, a destination, and a driving route from the starting point to the destination.

[0019] The above navigation usage record may be collected from at least one of a navigation device and a navigation server installed in a vehicle.

[0020] Based on the above instructions being executed individually or collectively by at least one processor, the device may be configured to extract a starting point and a destination of the vehicle from the vehicle's driving record, input the starting point and the destination into a navigation path search algorithm, and generate navigation information including a path to the destination.

[0021] The above model may be an end-to-end model included in a vehicle's autonomous driving system.

[0022] Based on the above instructions being executed individually or collectively by at least one processor, the device may acquire route data based on the navigation usage record and the navigation information, and encode the route data into SD map-based geometry information.

[0023] Based on the above instructions being executed individually or collectively by at least one processor, the device may be able to extract turn-by-turn (TBT) information from the navigation usage record and the navigation information, and to encode the path data and the TBT information to generate the dataset.

[0024] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0025] FIG. 1 is a drawing for explaining an autonomous driving system according to one embodiment.

[0026] FIG. 2 is a diagram illustrating an end-to-end based path generation framework according to one embodiment.

[0027] FIG. 3a is a diagram illustrating a dataset generation and verification process according to one embodiment.

[0028] FIG. 3b is a diagram illustrating a learning process according to one embodiment.

[0029] FIG. 3c is a diagram illustrating an inference process according to one embodiment.

[0030] FIG. 4 is a diagram illustrating a dataset generation device according to one embodiment.

[0031] FIG. 5 is a diagram illustrating a data processing process according to one embodiment.

[0032] FIG. 6 is a diagram illustrating the network structure of a model according to one embodiment.

[0033] FIG. 7 illustrates an example of a dataset according to one embodiment.

[0034] FIGS. 8A and FIGS. 8B are drawings for illustrating an example of a dataset according to one embodiment.

[0035] FIGS. 9a to 9c are drawings for illustrating a predicted path according to one embodiment.

[0036] FIG. 10 is a diagram illustrating a dataset verification process according to one embodiment.

[0037] FIGS. 11a and FIGS. 11b are drawings for explaining verification results according to one embodiment.

[0038] FIG. 12 is a flowchart illustrating a method according to one embodiment.

[0039] FIG. 13 is a schematic block diagram of an electronic device according to one embodiment.

[0040] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0041] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0042] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0043] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0044] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0045] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0046] As used in this document, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, that performs certain roles. However, "part" is not limited to software or hardware. "Part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. For example, "part" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. Additionally, '~part' may include one or more processors.

[0047] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0048]

[0049]

[0050] FIG. 1 is a drawing for explaining an autonomous driving system according to one embodiment.

[0051] Referring to FIG. 1, according to one embodiment, an autonomous driving system (e.g., autonomous driving system (10)) may be a system that generates a driving path and controls the driving of a vehicle (e.g., vehicle (100)) based on sensing data about the surroundings of the vehicle (e.g., vehicle (100)). The autonomous driving system (10) may include an autonomous driving device (100), a vehicle (110), and a server (130). The autonomous driving system (10) may generate a driving path of the vehicle (110) based on an end-to-end model. The end-to-end model may be a model that performs motion planning by directly processing input data without complex intermediate steps. The end-to-end model may be included in the autonomous driving device (100) and / or the server (130) to generate a driving path of the vehicle (110). The autonomous driving device (100) is mounted inside the vehicle (110) and can control the driving of the vehicle (110). For example, the autonomous driving device (100) may be a software module implemented on a processor (not shown) of the vehicle (110). Some components of the autonomous driving device (100) may be located outside the vehicle (110) (e.g., a server (130)). For example, a module that generates a driving path of the autonomous driving device (100) may be implemented on the server (130) to generate a driving path and transmit the generated driving path to the vehicle (110). The vehicle (110) means a vehicle that carries people and / or goods and may include a vehicle such as an automobile. The vehicle (110) may be an autonomous vehicle.

[0052] The autonomous driving device (100) can obtain information about the surroundings of the vehicle (110) (e.g., scene images) and driving information of the vehicle (110) (e.g., driving records) from a plurality of sensors mounted on the vehicle (110). For example, the autonomous driving device (100) can obtain information such as the surroundings of the vehicle (110) and driving information of the vehicle (110) from various sensors such as an image sensor, a LiDAR sensor, a radar sensor, an event sensor, an illuminance sensor, a GPS device, and an acceleration sensor. The autonomous driving device (100) can generate a driving path of the vehicle (110) based on information about the surroundings of the vehicle (110). For example, the autonomous driving device (100) can generate a driving path of the vehicle (110) based on information such as sensing data about the surroundings of the vehicle (110), the destination of the vehicle (110), and the location of the vehicle (110).

[0053] The autonomous driving device (100) may include a model (e.g., an end-to-end model). The model may be trained using a dataset generated based on the driving records of the vehicle (110). The training of the model may be performed internally within the autonomous driving device (100), or the autonomous driving device (100) may use a model trained in a separate device. The autonomous driving device (100) may generate a driving path for the vehicle (110) using the trained model.

[0054] The autonomous driving device (100), vehicle (110), and server (130) can communicate using a network (not shown). For example, the network may include a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. The network is a comprehensive data communication network that enables the autonomous driving device (100) and the server (130) to communicate smoothly with each other, and may include wired internet, wireless internet, and mobile wireless communication networks. Additionally, the wireless communication network may include, for example, Wi-Fi, Bluetooth, Bluetooth Low Energy, Zigbee, Wi-Fi Direct (WFD), Ultra-Wideband (UWB), Infrared Data Association (IrDA), Near Field Communication (NFC), but is not limited thereto.

[0055]

[0056] FIG. 2 is a diagram illustrating an end-to-end based path generation framework according to one embodiment.

[0057] Referring to FIG. 2, according to one embodiment, an end-to-end based path generation framework may include a dataset generation and verification process (210), a training process (220), and an inference process (230). An autonomous driving system (e.g., the autonomous driving system (10) of FIG. 1) can generate a driving path for a vehicle (e.g., the vehicle (110) of FIG. 1) through the end-to-end based path generation framework.

[0058] The dataset generation and verification process (210) may be a process for generating a dataset to train a model (e.g., an end-to-end model) and verifying the generated dataset. The generation of the dataset may be performed by a dataset generation device (e.g., the dataset generation device (300) of FIG. 3a). The operation of the dataset generation device (300) generating the dataset will be described in detail later with reference to FIG. 4. The verification of the dataset may be performed by a dataset verification device (e.g., the dataset verification device (330) of FIG. 3a). The verification of the dataset performed by the dataset verification device (330) will be described in detail later with reference to FIG. 10 to FIG. 11b.

[0059] The data generation device (300) and the dataset verification device (330) may be a learning device (not shown) or a device separate from the learning device (not shown). For example, the dataset generation and verification process (210) and the learning process (220) may be performed on different devices. The learning device (not shown) may receive the dataset generated through the dataset generation and verification process (210) and perform the learning process (220).

[0060] The learning process (220) may be a process of training a model (e.g., an end-to-end model) to generate an optimal driving path. The learning process (220) may be performed in a learning device (not shown). The learning device (not shown) may be an autonomous driving device (100) or a device separate from the autonomous driving device (100). For example, the training of the model may be performed in a separate learning device (not shown), and the autonomous driving device (100) may generate a driving path using the trained model.

[0061] The inference process (230) may be a process for generating a driving path of a vehicle (e.g., vehicle (110)) using a learned model. In the inference process (230), an autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) may generate a driving path based on data input from a plurality of sensors of the vehicle (110). The inference process (230) will be described in detail later with reference to FIG. 6.

[0062]

[0063] FIG. 3a is a diagram illustrating a dataset generation and verification process according to one embodiment.

[0064] Referring to FIG. 3a, according to one embodiment, a dataset generating device (300) can generate a dataset for training a model (e.g., an end-to-end model) that generates a driving path of a vehicle (e.g., the vehicle (110) of FIG. 1). The model may be an end-to-end model included in an autonomous driving system (e.g., the autonomous driving system (10) of FIG. 1). The dataset generating device (300) can transmit the generated dataset to a dataset verification device (330). The dataset verification device (330) can verify whether the dataset was properly generated. For example, the dataset verification device (330) can verify whether navigation information generated retrospectively based on driving records was properly generated.

[0065] The dataset generation device (300) and the dataset verification device (330) may be implemented as the same device or as different devices. For example, the dataset generation device (300) and the dataset verification device (330) may be implemented as software modules running on a single device (e.g., the autonomous driving device (100) and server (130) of FIG. 1) or as separate devices.

[0066]

[0067] FIG. 3b is a diagram illustrating a learning process according to one embodiment.

[0068] Referring to FIG. 3b, according to one embodiment, a learning process (e.g., the learning process (220) of FIG. 2) may be performed by a learning device (not shown). The learning process (220) may be a process of training a model so that a desired output (e.g., a driving path) can be generated based on input data (e.g., sensing data). The model (350) may be an end-to-end model. The learning device (not shown) may generate a learned model (370) by training the model (350) using a dataset generated by a dataset generating device (e.g., the dataset generating device (300) of FIG. 3a). For example, the learning device (200) may generate a learned model (370) by calculating the difference between the predicted value and the actual value (ground truth) using a loss function and adjusting the weights of the model (350) with an optimization algorithm based on the loss value.

[0069]

[0070] FIG. 3c is a diagram illustrating an inference process according to one embodiment.

[0071] Referring to FIG. 3c, according to one embodiment, an inference process (e.g., the inference process (230) of FIG. 2) may be a process of inputting input data (e.g., sensing data) into a trained model (370) to generate a driving path in real time. The trained model (370) can generate a driving path in real time using the sensing data in an actual driving situation of a vehicle (e.g., the vehicle (110) of FIG. 1).

[0072]

[0073] FIG. 4 is a diagram illustrating a dataset generation device according to one embodiment.

[0074] Referring to FIG. 4, according to one embodiment, a dataset generating device (400) (e.g., the dataset generating device (300) of FIG. 3a) can generate a dataset for training of a model (e.g., an end-to-end model). The dataset generating device (400) can collect a navigation history (430). The navigation history (430) may include information about a starting point, a destination, and a driving route from the starting point to the destination. The navigation history (430) may be a record (e.g., a trajectory) of a vehicle driving from a starting point (e.g., the vehicle's starting location) to a destination using a navigation device. For example, the navigation history (430) may be a driving log containing the driving route from the vehicle's starting point to the destination. The navigation usage record (430) may be collected from at least one of a navigation device and a navigation server (e.g., the server (130) of FIG. 1) installed in a vehicle (e.g., vehicle (110)). For example, information regarding the starting point, destination, and driving route from the starting point to the destination included in the navigation usage record (430) may be collected from at least one of a navigation device and a navigation server installed in the vehicle (110).

[0075] The driving record (410) may be a driving record that does not include navigation information. For example, the driving record (410) may include the driving trajectory of the vehicle (110) driven without using a navigation device. The driving record (410) may be a record of the vehicle's driving in a time-series format on a standard definition (SD) map. An SD map may be a general map containing static geographical information. Compared to a high-definition (HD) map, an SD map contains less information and can use fewer resources. A dataset generating device (400) may generate navigation information based on the driving record (410). A navigation information generating module (420) may be a module that generates navigation information retrospectively from a record of the vehicle driving without using a navigation device (e.g., driving record (410)).

[0076] The navigation information generation module (420) can extract the vehicle's starting point and destination from the driving record (410). For example, the navigation information generation module (420) can extract the point where the vehicle started driving as the starting point and the point where the vehicle ended driving as the destination from the driving record (410). The navigation information generation module (420) can input the extracted starting point and destination into a navigation path search algorithm to generate navigation information including a path to the destination. For example, the navigation information generation module (420) can input the extracted vehicle's starting point and destination into a path search algorithm to generate navigation information retrospectively.

[0077] The dataset generation device (400) can generate a dataset based on navigation information generated retrospectively from navigation usage records (430) and driving records (410). The dataset generation device (400) can generate a dataset by using navigation information and SD map information together. The dataset generation device (400) can encode navigation information generated retrospectively from navigation usage records (430) and driving records (410) into various types of datasets using an encoder (470). The dataset generation device (400) can obtain route data based on navigation usage records (430) and navigation information. The encoder (470) can encode route data based on navigation usage records (430) and navigation information into SD map-based geometry information.

[0078] The dataset generation device (400) can extract turn-by-turn (TBT) information from the navigation usage record (430) and navigation information. The dataset generation device (400) can generate a dataset by encoding route data based on the navigation usage record (430) and navigation information together with the TBT information. The dataset generated by the dataset generation device (400) will be described in detail later with reference to FIGS. 7 to 8b.

[0079]

[0080] FIG. 5 is a diagram illustrating a data processing process according to one embodiment.

[0081] Referring to FIG. 5, according to one embodiment, a dataset generated by a dataset generating device (e.g., the dataset generating device (300) of FIG. 3) may include path data encoded with geometric information based on an SD map. For example, the path data may be at least part of a dataset generated based on navigation information based on a global SD map. The path data may include a plurality of vectors (e.g., a plurality of segments) partitioned into predetermined time units. If the time unit of the path data is too long, the path data may become complex, and if it is too short, sufficient context information may not be provided, so it may be necessary to set an appropriate interval. For example, it should be noted that the optimal time unit of the path data that includes sufficient context information without making the data complex may be 12 seconds, but is not limited thereto.

[0082] Since driving situations consist mostly of straight sections with relatively few turn sections, path data may primarily contain samples related to straight-line travel. Consequently, data imbalance may affect model training. It may be necessary to process the path data before using the dataset for model training.

[0083] A processing unit (not shown) can detect turning data from path data. A processing unit (not shown) can extract vector pairs from multiple vectors constituting the path data. For example, a processing unit (not shown) can iteratively extract vector pairs from multiple vectors. Based on the vector pairs, a processing unit (not shown) can determine whether a path section corresponding to a vector pair is a turning section. A processing unit (not shown) can perform a vector product operation on the vector pairs. For example, a processing unit (not shown) can perform a dot product or a cross product on vector (510) and vector (520). Based on the result of the vector product operation, a processing unit (not shown) can determine whether a path section corresponding to a vector pair is a turning section. For example, if the value of the cross product between vector (510) and vector (520) is equal to or greater than 0 degrees, and the arc cosine value between vector (510) and vector (520) is greater than 80 degrees and less than 100 degrees, the path section corresponding to vector (510) and vector (520) may be determined to be a left turn section. For example, if the value of the cross product between vector (510) and vector (520) is less than 0 degrees, and the arc cosine value between vector (510) and vector (520) is greater than 80 degrees and less than 100 degrees, the path section corresponding to vector (510) and vector (520) may be determined to be a right turn section. A processing device (not shown) can determine the part of the path data corresponding to the vector pair as direction change data in response to the determination that the path section corresponding to the vector pair is a direction change section (e.g., left turn section, right turn section).

[0084] A processing unit (not shown) can upsample direction change data to construct a subset. For example, the processing unit (not shown) can construct a balanced dataset by forming a mini-batch using the detected direction change data. The processing unit (not shown) can upsample the direction change data. The processing unit (not shown) can process path data by constructing the upsampled direction change data into a subset (e.g., a mini-batch). The processed path data can be used to generate a driving path for a vehicle. The process of generating a driving path will be explained in detail later with reference to FIG. 6. The path data processing unit (not shown) may be a learning unit (not shown) or a separate unit.

[0085]

[0086] FIG. 6 is a diagram illustrating the network structure of a model according to one embodiment.

[0087] Referring to FIG. 6, according to one embodiment, FIG. 6 may illustrate the network structure of a model. The model may be the learned model (370) described with reference to FIG. 3b. The model may obtain sensing data regarding the surrounding conditions of the vehicle (e.g., the vehicle (110) of FIG. 1) from a sensor (e.g., sensor (601)) mounted on the vehicle (e.g., the vehicle (110) of FIG. 1). The sensing data may be various types of sensing data (e.g., multi-modality sensing data) that can be obtained from the sensor (e.g., sensor (601)) mounted on the vehicle (110). The model may receive the sensing data, path data (603) expressed in SD geometric information, and a driving trajectory (605) to generate a driving path. The driving trajectory (605) may include a record of the path traveled by the vehicle (110) and information regarding the state of the vehicle (110) (e.g., the state of the vehicle (110) in a driving situation, such as speed, acceleration, and steering). The sensing data, path data (603) expressed in SD geometric information, and driving trajectory (605) may be processed by element-wise projection and converted into tokens. An autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) may input the sensing data, processed path data (e.g., path data (603)), and driving trajectory (605) respectively into a corresponding first mixer. For example, the sensing data, path data (603), and driving trajectory (605) converted into tokens may be input into mixers (611) to (615), respectively. The first mixer (e.g., mixer (611) to mixer (615)) may be an intra mixer. The mixer (611) to mixer (615) may convert data of different dimensions (e.g., sensing data, path data (603), and driving trajectory (605)) into the same dimension.For example, mixers (611) to (615) can encode data of different dimensions (e.g., sensing data, path data (603), and driving trajectory (605)) so that they can be converted into the same dimension. The output of the first mixer (e.g., mixer (611) to (615)) can be input to the second mixer (e.g., mixer (630)). For example, the sensing data, path data, and driving trajectory, which are input to the mixers (611) to (615) respectively and converted into the same dimension, can be input to the mixer (630). The second mixer (e.g., mixer (630)) may be an inter-mixer that performs interaction between different types of data. The second mixer (e.g., mixer (630)) can generate an accurate driving path based on various types of data by performing encoding between different types of data. The model (e.g., the trained model (370) of FIG. 3b) can generate a driving path in which interactions between data (e.g., fusion) are performed with better performance by inputting data of different dimensions into corresponding mixers and processing them primarily.

[0088]

[0089] FIG. 7 illustrates an example of a dataset according to one embodiment.

[0090] Referring to FIG. 7, according to one embodiment, FIG. 7 may represent a dataset in which path data obtained based on navigation information is encoded into SD map-based geometric information. The dataset may be generated based on navigation information generated retrospectively from a driving record. The dataset may consist of a plurality of frames at predetermined time intervals. A dataset generating device (e.g., the dataset generating device (300) of FIG. 3a) may obtain path data based on navigation information generated retrospectively from a driving record (e.g., the driving record (410) of FIG. 4) and a navigation usage record (e.g., the navigation usage record (430) of FIG. 4). The dataset generating device (300) may generate a dataset by encoding the path data into SD map-based geometric information. The path data may include a plurality of vectors of predetermined time units. The dataset generating device (300) may request SD data regarding information about surrounding conditions (e.g., scene images) or driving information (e.g., driving trajectory) obtained from a vehicle (e.g., vehicle (110) of FIG. 1). The dataset generating device (300) may interpolate each vector at intervals of a predetermined distance (e.g., 1 meter). The dataset generating device (300) may interpolate data by finding the point closest to the location of the ego-vehicle (e.g., vehicle (110)) in each frame included in the dataset. For example, the dataset generating device (300) may determine the time that becomes the interpolation interval based on the speed limit at that location. The dataset may consist of nA+nB+1 frames, which is the sum of nA points corresponding to a past time (e.g., A seconds), 1 point corresponding to the current time, and nB points corresponding to a future time (e.g., B seconds in the future). Note that the interpolation interval may be set to 1 / n, but is not limited thereto.

[0091]

[0092] FIGS. 8A and FIGS. 8B are drawings for illustrating an example of a dataset according to one embodiment.

[0093] Referring to FIGS. 8a and 8b, according to one embodiment, a dataset generating device (e.g., the dataset generating device (300) of FIG. 3) can extract turn-by-turn (TBT) information from navigation usage records and navigation information. The TBT information may represent information regarding turn-by-turns in a driving path from a starting point to a destination. The dataset generating device (300) may encode path data and TBT information to generate a dataset. For example, TBT information (811) to TBT information (815) may be data in which turn-by-turn information at points (801) to (805) is encoded. The TBT information (e.g., TBT information (811) to TBT information (815)) may include an offset, which is the distance from the starting point to each turn-by-turn point, a type of turn, a turn location expressed in x and y coordinates, and confidence of the information. FIG. 8b may be another example of a dataset generated by encoding TBT information. Graphs (830) to (850) may each represent the distance from the starting point, the speed of the vehicle at the time corresponding to the frame, and the yaw angle of the vehicle. The dataset may include TBT information extracted from navigation usage records and navigation information.

[0094]

[0095] FIGS. 9a to 9c are drawings for illustrating a predicted path according to one embodiment.

[0096] Referring to FIGS. 9a through 9c, according to one embodiment, a model (e.g., the learned model (370) of FIG. 3b) can predict and output a driving path of a vehicle (e.g., an image taken of the front of the vehicle (110)) based on an image (e.g., an image taken of the front of the vehicle (110)) obtained from a sensor (e.g., an image sensor) mounted on the vehicle (e.g., the vehicle (110) of FIG. 1). For example, an autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) can generate a right-turn path based on an image taken of a situation where the vehicle (110) needs to make a right turn. The image (910) may be an image taken of a situation where the vehicle (110) needs to make a right turn with the generated right-turn path overlaid on it. Since the driving path is determined based on the destination of the vehicle (110), different driving paths can be generated based on the same image (e.g., the image shown in FIG. 9b and FIG. 9c). For example, the autonomous driving device (100) can generate a left-turn path based on an image of the front of the vehicle (110) shown in FIG. 9b when the vehicle (110) needs to turn left to reach a destination. The image (920) may be an image of the front of the vehicle (110) with the generated left-turn path overlaid. For example, the autonomous driving device (100) can generate a straight-line path based on an image of the front of the vehicle (110) shown in FIG. 9c when the vehicle needs to go straight to reach a destination. The image (930) may be an image of the front of the vehicle (110) with the generated straight-line path overlaid. Images (910) to (930) may be an image of the front of the vehicle with a driving path (e.g., a driving path generated by the autonomous driving device (100)) overlaid on an image of the front of the vehicle. The driving path may be the same as the predicted path generated by the model (370). The predicted path may be generated by the model (370) using the SD navigation path as input.

[0097]

[0098] FIG. 10 is a diagram illustrating a dataset verification process according to one embodiment.

[0099] Referring to FIG. 10, according to one embodiment, the dataset verification process may include a verification image generation process (1010) and a navigation information verification process (1030). A dataset verification device (e.g., the dataset verification device (330) of FIG. 3a) can verify whether the dataset generated by the dataset generation device (e.g., the dataset generation device (300) of FIG. 3a) is appropriate. For example, the dataset verification device (330) can verify whether the navigation information generated retrospectively from a driving record (e.g., the driving record (410) of FIG. 4) by the navigation information generation module (e.g., the navigation information generation module (420) of FIG. 4) of the dataset generation device (300) is appropriately generated.

[0100] In the verification image generation process (1010), the dataset verification device (330) may generate multiple frames containing driving records and navigation information. Each of the multiple frames may include a driving trajectory at a point in time corresponding to each of the multiple frames in the driving records and a path segment corresponding to the driving trajectory in the navigation information. The navigation information may be generated by inputting the vehicle's starting point and destination extracted from the driving records into a navigation path search algorithm. The generated verification images will be described in detail later with reference to FIGS. 11a and FIGS. 11b. The dataset verification device (330) may extract a representative frame from the generated multiple frames. Since navigation information does not change in real time, verifying all of the multiple frames may be inefficient, so the dataset verification device (330) may extract a representative frame. The dataset verification device (330) can verify navigation information based on the difference between the driving record and navigation information included in the representative frame. The dataset verification device (330) can determine the navigation information as failure data in response to the difference between the driving record and navigation information being greater than a predetermined threshold. The failure data is organized into a separate list and may not be used for training a model that generates a driving path (e.g., the trained model (370) in FIG. 3b). The failure data may be used in the navigation information generation algorithm of the dataset generation device (300) after additional classification is performed. The dataset verification device (330) can determine the navigation information as success data in response to the difference between the driving record and navigation information being less than or equal to a predetermined threshold.

[0101]

[0102] FIGS. 11a and FIGS. 11b are drawings for explaining verification results according to one embodiment.

[0103] Referring to FIGS. 11a and 11b, according to one embodiment, a dataset verification device (e.g., the dataset verification device (330) of FIG. 3a) can extract a representative frame from a plurality of generated frames. The representative frame may include at least one frame among the initial frame, middle frame, and last frame of the plurality of frames. Since navigation information generated retrospectively from a driving record may have ambiguity at the starting and ending points of the driving, it may be a suitable target for verifying whether the navigation information was properly generated. The dataset generation device (330) can verify navigation information based on the difference between the generated navigation information (e.g., SD data (1105)) and the driving record (e.g., past trajectory (1101), future trajectory (1103)). For example, in FIG. 11a, the dataset verification device (330) can determine the navigation information (e.g., SD data (1105)) as successful data because the difference between the subsequently generated navigation information (e.g., SD data (1105)) and the driving record (e.g., past trajectory (1101), future trajectory (1103)) is below a predetermined threshold value. For example, in FIG. 11b, the dataset verification device (330) can determine the difference between the subsequently generated navigation information (e.g., SD data (1105)) and the driving record (e.g., past trajectory (1101), future trajectory (1103)) is below a predetermined threshold value. Since it is greater than the threshold value, navigation information (e.g., SD data (1105)) can be determined as failure data.

[0104]

[0105] FIG. 12 is a flowchart illustrating a method according to one embodiment.

[0106] Referring to FIG. 12, according to one embodiment, operations 1210 to 1250 may be operations performed by the dataset generating device (300) of FIG. 3a described with reference to FIG. 1 to FIG. 11b.

[0107] According to one embodiment, operations 1210 to 1250 may be understood to be performed in a processor (e.g., processor (1330) of FIG. 13) of a dataset generating device (300) (e.g., electronic device (1300) of FIG. 13) described with reference to FIG. 1 to 11b.

[0108] In operation 1210, the dataset generating device (300) can collect navigation usage records. The navigation usage records may be records of a vehicle driving by entering a destination through a navigation device.

[0109] In operation 1230, the dataset generating device (300) can generate navigation information based on the vehicle's driving record. The dataset generating device (300) can generate navigation information retrospectively from the vehicle's driving record using a navigation information generating module (e.g., the navigation information generating module (420) of FIG. 4). The driving record may be a record of the vehicle's driving recorded chronologically on an SD map. The driving record may include a general driving trajectory that does not include navigation information.

[0110] In operation 1250, the dataset generating device (300) can generate a dataset based on navigation usage records and navigation information. The dataset generating device (300) can generate a dataset by using navigation information and SD map information together.

[0111] Operations 1210 through 1250 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel.

[0112]

[0113] FIG. 13 is a schematic block diagram of an electronic device according to one embodiment.

[0114] Referring to FIG. 13, according to one embodiment, an electronic device (1300) (e.g., autonomous driving device (100) of FIG. 1, dataset generation device (300) of FIG. 3a, dataset verification device (330), processing device (not shown)) may include a memory (1310) and a processor (1330).

[0115] The memory (1310) can store instructions (or programs) executable by the processor (1330). For example, the instructions may include instructions for executing the operation of the processor (1330) and / or the operation of each component of the processor (1330).

[0116] The memory (1310) may include one or more computer-readable storage media. The memory (1310) may include non-volatile storage devices (e.g., magnetic hard disc, optical disc, floppy disc, flash memory, EPROM (electrically programmable memories), EEPROM (electrically erasable and programmable)).

[0117] The memory (1310) may be a non-transitory medium. The term "non-transitory" may indicate that the storage medium is not implemented by a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted as meaning that the memory (1310) is immobile.

[0118] The processor (1330) can process data stored in memory (1310). The processor (1330) can execute computer-readable code (e.g., software) stored in memory (1310) and instructions triggered by the processor (1330).

[0119] The processor (1330) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program.

[0120] For example, a data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

[0121] The processor (1330) can cause the electronic device (1300) to perform one or more operations by executing code and / or instructions stored in memory (1310). The operations performed by the electronic device (1300) may be substantially the same as the operations performed by the dataset generating device (300) described with reference to FIGS. 1 through 13. Such redundant descriptions are omitted.

[0122]

[0123] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0124] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0125] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0126] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0127] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0128] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. Regarding the method of generating a dataset for training a model, Collecting navigation history; Generating navigation information based on the vehicle's driving log; and The operation of generating the dataset based on the above navigation usage record and the above navigation information A method including 2. In Paragraph 1, The above driving record is, A method in which the driving of the vehicle is logged in time-series on an SD (standard definition) map.

3. In Paragraph 1, The above navigation usage record is, Information regarding a starting point, a destination, and a driving route from the starting point to the destination A method including 4. In Paragraph 3, The above navigation usage record is, A method of collecting from at least one of a navigation device and a navigation server installed in a vehicle.

5. In Paragraph 1, The operation of generating the above navigation information is, The operation of extracting the starting point and destination of the vehicle from the driving record of the vehicle; and The operation of inputting the above starting point and the above destination into a navigation path search algorithm to generate the navigation information including the path to the above destination. A method including 6. In Paragraph 1, The above model is, A method that is an end-to-end model included in a vehicle's autonomous driving system.

7. In Paragraph 1, The operation of generating the above dataset is, An operation to acquire route data based on the above navigation usage record and the above navigation information; and Encoding the above path data into SD map-based geometry information A method including 8. In Paragraph 7, The operation of generating the above dataset is, The operation of extracting TBT (turn-by-turn) information from the above navigation usage record and the above navigation information; and The operation of encoding the path data and the TBT information to generate the above dataset A method including 9. A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 through 8.

10. An apparatus for generating a dataset for training a model, At least one processor; and memory that stores instructions Includes, Based on the above instructions being executed individually or collectively by the at least one processor, the device, Collect navigation history, and Based on the vehicle's driving log, navigation information is generated, and A device for generating the dataset based on the navigation usage record and the navigation information.

11. In Paragraph 10, The above driving record is, A device in which the driving of the vehicle is logged in time-series on an SD (standard definition) map.

12. In Paragraph 10, The above navigation usage record is, Information regarding a starting point, a destination, and a driving route from the starting point to the destination A device including 13. In Paragraph 12, The above navigation usage record is, A device that collects from at least one of a navigation device and a navigation server installed in a vehicle.

14. In Paragraph 10, Based on the above instructions being executed individually or collectively by the at least one processor, the device, Extract the starting point and destination of the vehicle from the driving record of the vehicle, and A device that inputs the above-mentioned starting point and the above-mentioned destination into a navigation path search algorithm to generate the above-mentioned navigation information including a path to the above-mentioned destination.

15. In Paragraph 10, The above model is, A device that is an end-to-end model included in a vehicle's autonomous driving system.

16. In Paragraph 10, Based on the above instructions being executed individually or collectively by the at least one processor, the device, Based on the above navigation usage records and the above navigation information, route data is obtained, and A device for encoding the above path data into SD map-based geometry information.

17. In Paragraph 16, Based on the above instructions being executed individually or collectively by the at least one processor, the device, TBT (turn-by-turn) information is extracted from the above navigation usage records and the above navigation information, and A device for encoding the path data and the TBT information to generate the above dataset.

Images